| An action performed by an end user, whether through an enterprise portal application or a user-facing client application, often results in complex requests that need to query multiple backend systems. This process is referred to as a federated query. As we have seen, the Forward Cache pattern and its variations can make this process much more efficient. 11.5.1 Cache Push Versus Pull Patterns The patterns examined so far have been based on a push model, where data is proactively identified to be pushed to a cache. Sometimes, however, the data in the backend system can't reasonably be pushed to a cache. The volume of data may be too great, or it may just not be technically or economically viable to trap all the updates necessary to maintain the forward cache. Sometimes the data can't reasonably be cached at all, due to the rate of data change in the backend systems. Querying cached data that is frequently outdated can lead to data inconsistency and provide incorrect results. The federated query pattern has a cache pull variant that can address this issue. Implementing the federated query real-time request pattern and its variations requires going out to the multiple backend systems and querying them in real time. The federated query patterns are based on the request/reply and reply-forward patterns, which were discussed in Chapter 5. The federated query pattern comes in at least two variants: real-time request and long-duration request. Which one you use depends upon the duration of the query. 11.5.2 The Real-Time Request Pattern The federated query could be addressed without an ESB by utilizing multiple concurrent request/reply messaging patterns. A portal server (or an application) can use multithreading techniques to fire off simultaneous requests to reduce the overall latency associated with querying multiple backend applications and data sources. However, this approach is not tolerant of failure conditions. An ESB can address concurrency and reliability issues by invoking the backend data sources as service requests using reliable publish-and-subscribe messaging channels (Figure 11-21). Figure 11-21. A portal request initiates multiple simultaneous requests to backend applications using publish-and-subscribe messaging across an ESB The responses are sent back to the portal server directly from each remote data source, which runs as an event-driven service invocation, as illustrated in Figure 11-22. Because the portal server itself is plugged into the bus, it is accessible from every other process that is plugged into the bus. If the underlying implementation of the service invocation is mapped onto a point-to-point message queue channel, the portal server can have an MDB interface that listens on the point-to-point message queue. Figure 11-22. The response is sent back to the portal server as a direct service invocation across the bus, using a message queue to an MDB to receive the response In the cache pull variant of the federated query, the request will first go to the cache to see if there is any nonexpired data that can satisfy the request. If not, the federated query will be initiated. The replies from the federated query can be routed to an ESB process that updates the cache and then forwards it on to the portal server (Figure 11-23). Figure 11-23. The responses go to a process that updates the cache, then forwards to the MDB 11.5.3 The Long-Duration Request Pattern The real-time request pattern works well, assuming that all of the data sources are immediately accessible and that the response to the query will complete within a reasonable amount of time. In practice, requests may take a very long time to complete. Here are some reasons why: Human input (a step in a process may involve manual intervention, such as approval) Requests deferred to off-peak hours Offline system or application Network outage
The long-duration request pattern builds on the real-time request pattern, but assumes that responses from services on the bus will return over a much less predictable period of time. This allows the portal user to browse to other areas of the web site while responses asynchronously aggregate in the user's session. In this scenario, a user logs into the application and initiates an action that launches a query to the ESB. The ESB routes the request to the appropriate systems on the bus, and delivers the responses back to the same application or portal session. The user is then free to use other features of the application while the requests are accumulated. If the application is an enterprise portal, it requires the portal to have a user session feature that allows the user to log in at a later time and check for responses. The XML persistence service that is used for the cache service may also be responsible for collecting the multiple responses and then forwarding them on to the portal. In that sense it acts as an aggregation point as well as a cache. All of these techniques require you to configure a "rule" that defines when the result set is complete, which in turn triggers the response to the original query request. Normally the rule is simply an XPath or XQuery returning a Boolean. The trigger may be a CBR step or may be built into the XML service. 11.5.4 Federated Query Pattern Variations There are several variations of the federated query patterns. The real-time and long-duration patterns can be combined. When the federated query is issued in this scenario, the portal waits for a fixed amount of time and then returns the aggregated results to the user. Any late responses are asynchronously aggregated by the portal and are available on subsequent requests by the user. The "response" event that gets generated from the aggregation service to the portal could be implemented as a single message that carries all of the data, or it could be an empty message that signals the portal server to issue another request to extract the data from the XML aggregation service. Before forwarding to the portal server, the multiple responses could be run through a VETO pattern that could consolidate the data and reformat it using XSLT into a form that is more suitable for consumption by the portal.
11.5.4.1 Federated query using itinerary processes Variations of the federated query can be accomplished using ESB process itineraries rather than publish-and-subscribe messaging channels, as illustrated in Figure 11-24 and Figure 11-25. This gives you more control over how each subquery is handled, but increases the latency for the federated query if the subqueries are executed sequentially. To execute subqueries in parallel with an itinerary process, you need a specialized join service that can intelligently collect the multiple responses. Figure 11-24. A real-time federated query using ESB process itineraries Figure 11-25. A cache pull federated query using ESB process itineraries 11.5.4.2 Federated query using BPEL4WS and an orchestration service An orchestration service is well suited for the branching out of multiple requests and for waiting for the responses to come back, as illustrated in Figure 11-26. Rather than using process itineraries and fan-out services, a process orchestration service can orchestrate more complex interactions such as splits and joins based on transition conditions. More information on the merits of an orchestration service versus process itineraries can be found in Chapter 7. Figure 11-26. A federated query using an orchestration service that is plugged into the ESB |