Enterprise Service Bus: Theory in Practice

There are ways to enhance an ESB with specialized value-added services that can help solve some of the more complex integration problems.

7.6.1 Routing Patterns Using Itineraries and Services

One of the attractive characteristics of an ESB is the way in which services can be constructed together to solve common integration problems in ways that are profoundly simple. This section will examine some of the common patterns that are used in ESB integration solutions.

Multi-itinerary splitter pattern

Sometimes you need a more sophisticated means of controlling process flow. More complex business processes might involve multiple paths of execution to happen in parallel. These multiple paths can be handled through a combination of process itineraries and special services by placing a fan-out service within an itinerary that splits the process into two parallel subprocesses (Figure 7-10).

Figure 7-10. In the multi-itinerary splitter pattern, a fan-out/splitter service routes messages to parallel execution paths of multiple itineraries

Performing a simple join operation using itineraries means waiting for specific messages to arrive at a service. In this simple case, you can use an aggregator service along with some special design patterns that may require coding techniques. An example of this is discussed in Chapter 11 in the "Federated Query Patterns" section. Figure 7-11 shows a pattern sketch of this multi-itinerary splitter pattern.

Figure 7-11. Pattern sketch of parallel execution paths using a combination of itineraries and splitter/aggregator services

Multi-itinerary CBR pattern

A CBR service may need to choose between multiple possible paths, and at some point these parallel paths may need to converge. This is commonly known as a join operation, where a business process blocks and waits for one or more conditions to be satisfied before proceeding to the next step (Figure 7-12). This is also accomplished by some coding techniques in an aggregator service.

Figure 7-12. In a multi-itinerary CBR pattern, a CBR service chooses between multiple possibilities of itinerary paths

Figure 7-13 shows a pattern sketch of the multi-itinerary CBR pattern.

7.6.2 Sophisticated Process Flow Using an Orchestration Service (BPEL4WS)

There are join conditions under which simply blocking and waiting for one or more responses to arrive at a service is not enough. For more sophisticated process management situations, you need a specialized ESB capability known as an orchestration service. An orchestration service is a special processing engine that can be plugged into the bus to coordinate other services that are on the bus, such as a BPEL4WS engine (Figure 7-14).

An orchestration service can manage stateful information about a business process that spans a length of time. For example, a step in a business process may require human interaction, such as manual approval of a purchase order. Itineraries can carry state, and recover from failure if the messages carrying the itineraries are being transported over reliable messaging. However, if a message is traveling through an itinerary and there are no failures, the business process simply runs its course and finishes to completion. There is no automatic way to temporarily suspend an itinerary-based process and have it wait for an external event to occur. If a stateful conversation will be carried out over a long duration, with pauses and resumes that are separated by time and triggered by external events, then an orchestration service is a better choice than a process itinerary.

Figure 7-15 shows an illustration of multiple states being managed by an orchestration service, represented by four distinct state transitions. An orchestration service can work in conjunction with an XML persistence service to keep track of process state between state transitions.

Even if no conditional joins of multiple processes occur, a business process can be set up to remain blocked until a certain transition condition is met. A transition condition blocks the process flow until a state variable in the process reaches a certain value or a Boolean expression evaluates to true. The release of a transition condition can be triggered by an event resulting from a person clicking an Approve button, or by the arrival of a message that the process is waiting for.

7.6.2.1 Recovery of process state

An orchestration service can better manage the scenarios of failure and recovery of stateful long-duration conversations. An example of stateful information could be the current values of join conditions themselves. If a join is waiting for multiple conditions to be satisfied, the current state of the process is that two out of three of those conditions have already been met. If the service goes down, the knowledge that two of the three transition conditions have been met will be restored upon recovery.

7.6.2.2 Correlation of asynchronous conversations

In a long-duration conversation that involves asynchronous requests and responses, the responses can arrive in any order over an extended period of time. An orchestration service can manage multiple conversations by correlating the requests with the appropriate responses. If using itineraries alone, it is conceivable that long-duration processes could be managed by letting messages build up in a queue, and then removing them from the queue when the join condition is met. However, it's not easy to take things out of a queue in any arbitrary order.

7.6.2.3 BPEL4WS for scripting of orchestration services

BPEL4WS is intended to be a portable way of abstractly describing a business process. An orchestration server may support BPEL4WS as the scripting language for describing the process definition, as well as supporting other scripting languages that are proprietary. It is expected that initial adopters of BPEL4WS will provide import/export capabilities into their existing BPM tools, but still use something proprietary as an internal representation. Microsoft BizTalk 4.0 takes this approach, for example. More details on BPEL4WS can be found in Chapter 12.

7.6.3 XML Storage and Caching Services

Another type of specialized service is an XML storage service. An XML storage service can be used to capture and store XML messages as they travel through the bus between services. This can help enable the retrieval of real-time data as it flows across the bus. Specialized aggregation services can be strategically arranged across the bus as a means of capturing important business data. Data can be collected regularly and retrieved using XQuery, and then used in reporting scenarios where meaningful data about the state of a business is reported and analyzed (Figure 7-16). For example, a common retail function is to collect and retrieve the details of the on-hand inventory in the warehouse, in the stores, and even in transit. This data can then be accessed via in-store kiosks to quickly locate out-of-stock items.

7.6.3.1 Data caching

Chapter 11 discusses the use of an XML persistence service for use in the forward cache and federated query patterns. In the forward cache pattern, data is proactively pushed out from backend data sources across the ESB to be held in a cache that is coresident with the application that needs to access the data. In the federated query pattern, an XML cache is used to collect asynchronous responses from multiple messages that are fired off at different backend data sources to gather data.

7.6.3.2 Aggregation and reporting

An XML persistence service can be used to collect data from multiple data sources, making the data much more valuable than it is when lying dormant in its constituent parts. This can enable BAM functionality by collecting data that can be fed into visual reporting tools. Aggregated data can have XSLT transformations applied to it, which can make it more meaningful to a business analyst.

The implication is that applications can participate in providing BAM-like functionality without any modifications. Existing applications can continue to post the same bits and pieces of data as usual, and that data can be routed through business processes, transformed, and sent to an XML aggregation service to be fed into reporting infrastructures.

7.6.3.3 Implementation choices

An XML persistence service could potentially be implemented using a relational database, with some XML to relational mapping technology built on top of it. This involves a process known as "shredding" and "unshredding" an XML document into rows and columns to meet the requirements of a relational database table structure.

An XML storage service is best implemented as a native XML database that knows how to store and retrieve XML as an in-memory parsed representation. This allows the use of XQuery to access individual nodes of an XML document without running the entire document through a parser just to access an individual element.

If the XML is managed by the XML persistence service as a preparsed XML document, then multiple processing steps that access the same XML document can be coresident within the service and linked together. These multiple steps can be a series of XQuery statements that are executed sequentially as related actions, forming an XML processing pipeline. Figure 7-17 shows a series of operations on a shared XML document, which is accessible across multiple steps in a single service container and is shared between multiple service containers.

The advantage of this is that you can optimize for multiple steps operating on the same instance of an XML document that is stored in the same XML persistence service. The shared XML document is parsed only once, as opposed to being parsed, serialized, and reparsed between each step in the process. If the process spans physical service containers, a message containing a token, or a key representing the shared XML document that is managed by the XML persistence service, is sent to the next container through the bus. This is called an XML pass-by-reference model.

The XML persistence service manages the transactional integrity of the document while changes are made to it during the XML pipelining process.

7.6.4 Convergence in Process Modeling

In this chapter we have examined three different ways of managing the steps in a business process: itinerary-based routing, sophisticated process orchestration using an orchestration service that is layered on top of the ESB, and the XML processing pipeline in an XML persistence service, also a layered service on top of the ESB. Each method has its merits and its optimizations, and each should be used in the situation that it is best suited for. Table 7-1 shows a summary of the advantages of each approach.

The disparity in the underlying processing techniques can be converged in a unified view of a process model. Regardless of the form you use for a given situation, the means for expressing the business process should be the same. An ESB can provide a single visual tool interface for describing processes. Using a visual drag-and-drop tool that visualizes business processes using UML diagramming techniques, the business process can be defined. The tool itself can generate BPEL4WS as the scripting language to describe the abstract process. Ideally, the visual tool and the runtime infrastructure of the ESB can make the optimal choice of underlying implementation, whether it is based upon itineraries, process orchestrations, or XML processing pipelines.