Middleware Pipeline Overview

Quantum Server is a pass-through system. Whenever a client sends a request, the server routes the request to an external data source (a Database, MongoDB, JSON service) and sends back the back-end response to the client.

  • Quantum Server does not generate responses; however, it receives the external source's result and sends to the client. During this stage, the Server must process each request before it is sent to the external data source and each response before it is sent to the client.
  • The middleware pipeline is divided into different layers, and each layer has its functionality, which applies transformations required for request and response objects.
  • Each layer is responsible for transforming requests and responses.
  • Each request and response object is propagated through these layers sequentially to apply transformations, as shown in the following flow diagram.

Middleware Pipeline Flow Diagram

The following middleware flow diagram details all the layers involved in the request and response transformation process:

 

  • A client request passes through a series of different middleware layers before the request is sent to the data source.

    Request pass-through process system

     

    Client

    Start

    1

    2

    3

    4

    5

    End

    Request

    Filters

    Data Controller

    Data Processor

    Connector

    Data Provider

    Data Source

 

  • A data source response passes through a series of different middleware layers before the response is sent to the client.

    IMPORTANT: The sequence of layers processed for the response is in the request's opposite direction.

    Response pass-through process system

     

    Client

    End

    5

    4

    3

    2

    		 

    1

    Start

    Request

    Filters

    Data Controller

    Data Processor

    Connector

    		Response

    Data Provider

    Data Source

 

Layers in Middleware Pipeline

Request

  • The Filters intercept a client request and perform pre-processing before passing them to Middleware Pipeline.

    For example, you can use custom filters to transform request objects before the request is sent to the data source, such as validation checks, security checks, and logging metrics.

Response

  • Filters are used to intercept the responses and perform post-processing before passing it to the client.

The following table details about different filters and functionality involved in the Filters layer.

Filter Functionality
DiagnosticContextFilter
  • Implements diagnostic (Logging) Context for the thread which helps trace the issues.
  • Adds X_KONY_REQUESTID, X_KONY_DEVICE_SESSIONID into org.apache.logging.log4j.ThreadContext
  • Adds DEVICE_SESSIONID to HttpServletRequest
MetricsFilter Logs request details to metrics queue like environment details, custom metrics data, duration for processing request, remoteAddress from which request originated etc.
MWTenantFilter Adds TenantBean, takes care of tenant throttling  and  also adds cache director to request (CacheDirector: retrieve or destroy cache object which holds the metadata/properties)
AddAdditionalResponseHeaderAttribute Verifies if CORS (cross origin resource sharing) is enabled and if enabled, adds “Access-Control-Allow-Headers” and “Access-Control-Allow-Methods” headers to response.
FrontendResolutionFilter If request is not middleware service and if frontendURL  is configured then frontEndUriContext is created and added to request.
MWValidationAndAuthenticationFilter
  • Invalid characters like '<' or '>' filtered out for security reasons
  • Validates security level, supported verbs, request method for a request.
  • Authenticates request using MBAASGateWayWrapper. 
SessionsFilter Appends SessionID to sessionManager.
PayloadParsingFilter
  • Adds ServiceInfo to Metrics Bean.
  • Creates requestWrapper and responseWrapper and populates params(X_KONY_API_VERSION, CURRENT_APP_ID, CURRENT_API_VERSION, CURRENT_SERVICE_ID,RequestPassThrough, ResponsePassThrough)
  • Checks for invalid characters like '<' or '>' in requestUri, serverName, parameter names, header names.  
EventsAutoFilter If request event and response event topics are present in request context then event data object is created and notified.
RecordingFilter If response is instance of RecordingResponseWrapper then details like httpMethod, requestURL, queryString, payload are stored in a file.
MiddlewareMemCacheDCFilter extends DCFilter
  • If device request log bean or response log bean is empty then request is populated with new DeviceRequestData, DeviceResponseData.
  • If character encoding in request is null then UTF8 is set. 
AuditLogFilter If debugging is enabled then, serviceid, operationId, objectID, requestedMethod, appVersion, clientIP, userId are logged before passing the request to next filter

1.a. MW Servlet

  1. Requests from the Filters layer are routed to MWServlet that initiates post (HttpServerletRequest) calls.
  2. Responses from the filters layer are routed to MWServlet that initiates get (HttpServerletResponse) calls.
  3. And the MW Servlet calls the Data Controller layer and passes the request and response objects.

    Sample Code - Request from filters layer is routed to MWServlet

For Requests

  1. Data Controller layer transforms HTTP servlet requests into Data Controller request objects for processing.
    • Sets CACHE_ID, TENANT_BEAN, CACHE_DIRECTOR, APP_CACHE, GATEWAY_WRAPPER_OBJECT, MetricsBean to the data controller request object.
    • Sends data controller request object to service delegator.
  2. It encapsulates data controller request objects.

    IMPORTANT: Data controller request objects help you configure additional parameters required in subsequent layers in the pipeline to process requests.

For Responses

  1. Data Controller layer transforms the data controller response objects into HTTP Serverlet responses for processing.
  2. Sets CACHE_ID, TENANT_BEAN, CACHE_DIRECTOR, APP_CACHE, GATEWAY_WRAPPER_OBJECT, MetricsBean to the data controller request object.
  3. Sends HTTP servlet responses to the Filters layer.
  4. It encapsulates data control response objects.

    IMPORTANT: Data controller response objects help you configure additional parameters required in subsequent layers in the pipeline to process responses.

    Datacontroller response
    • String responseData;
    • String charsetEncoding;
    • int statusCode;
    • List<Cookie> cookies;
    • List<Cookie> deviceCookies;
    • Map<String, String> headers;
    • Map<String, String> deviceHeaders;
    • String location;


After a request is passed through the Data Controller, the request enters the Data Processor layer containing two different entities, the Service Delegator and JSON Processor. These entities have different functionality.

Service Delegator functionality

  • Identifies appropriate data source connector based on the service configuration parameters.
  • Invokes data source connectors which eventually make a service call to the external data source.

  • After a request passed through the Data Controller layer, the Service Delegator is invoked. And the Service Delegator passes the request object to the Connector layer.

JSON Processor functionality

passes through the Data Controller layer, the Service Delegator is invoked. And the Service Delegator passes the request to the Connect layer.

  1. After the Connector layer passes a result/response object to Data Processor layer, the JSON Processor is invoked.

    NOTE: At this stage, the result/response objects are in the form of Data Controller requests and responses.

    • The JSON Processor converts the response object data structure to JSON string and sends it to the Data Controller layer. At this stage, the response is in the form of a Data Controller response.
  2. The Data Controller layer sends back the response to the device.

    NOTE: When you have multiple back-end sources, different responses must be captured in a standard way. The Data Models play a vital role in encapsulating these multiple responses from different back-end sources.



    Data Models encapsulating data source response

    The Data Models contain the following data structures:

    • Result
      • Abstraction of the result of a back-end service call.
      • Collection of Params, Datasets, and Records.
    • DataSet
      • list of records identified using a unique id.
    • Record
      • A record is an item in the Dataset.
      • A record, in turn, can have Records or Datasets.
      • A record can also contain Params.
    • Param
      • Param represents a parameter or variable part of the Result or Record object.
      • It has parameter name, parameter value along with meta-data like data type, data format, and so on.
  1. Transforms a request object.

    The Service Delegator chooses a connector based on the service configurations. The Connector layer helps transforms multiple data sources into expected data formats. A Pre-processor utility contains custom logic to transform the request object.

    • For example, you have an integration service type of XML service configuration. The Service Delegator selects the XML connector. And the Connect layer transforms data controller requests into the XML format. After that, a connector layer invokes pre-processor utility if configured, and the pre-processor executes classes and transforms the request object. A Pre-preprocessors utility allows developers to choose logic for data transformation.

      The following base connectors are supported for pre-processor utility configuration:

    Base Connectors /

    Technology Adapters    		Pre-processorBusiness Connectors /

    Data Adapter Connector

    • XML

    • SOAP
    • JSON
    • JavaScript
    • API Proxy
    • MongoDB
    		Executes the custom logic to transform the request object.

    Mock Data

    Relational Database

    Salesforce

    Open API (Swagger)

    IBM MQ

    RAML

    AWS API Gateway

    MuleSoft

    SAP JCO

  1. Sends the request object to the Data Provider layer.
  2. Makes a call to the external data source and gets the response, and the response is in the form of a response object.
  3. The Post-processor checks for custom logic and executes it to transform the response object. Transforms the response object based on the custom logic specified in the Post-processor utility, if any.
  4. Sends the transformed response object to the JSON Processor in the previous layer.

Quantum Server provides out-of-the-box support for standard interfaces to back-end data sources.

NOTE: The functionality of the Connector layer is to transform the request objects, make a call to the Data Source layer, and get the response. The response is in the form of a response object.

The Data Provider layer establishes a connection to the external Data Source layer. For example, connection throttling and connection timeouts.

The Data Provider interface allows you to provide a custom implementation for connection establishment and generating the response.

  • Default Data Provider
  • Rules Data Provider
  • Stub Data Provider

NOTE: The functionality of the Connector layer is to transform the request objects, make a call to the Data Source layer, and get the response. The response is in the form of a response object.