Reducing data inconsistencies with Red Hat Process Automation Manager

Managing data reconciliation through a specific process is a common necessity for projects that require Digital Process Automation (formerly known as Business Process Management), and Red Hat Process Automation Manager helps to address such a requirement. This article provides good practices and a technique for satisfying data reconciliation in a structured and clean way.

Red Hat Process Automation Manager was formerly known as Red Hat JBoss BPM Suite, so it's worth mentioning that jBPM is the upstream project that fuels Process Automation Manager. The blog post From BPM and business automation to digital automation platforms explains the reasons behind the new name and shares exciting news for this major release.

Data reconciliation

During a process's life, data is collected from systems and humans, mixed and enriched, and then transferred to other systems. Along the way, a system might not be aligned with other systems and a specific call might end up in an exception state that has to be handled in order to successfully complete the process instance.

There are many reasons for data inconsistencies. Among others, the most common are:

• User input is inaccurate
• The source of the information is out of date or erroneous
• The destination of the information is outdated or erroneous
• The destination system already contains the inserted data

Sometimes, it's sufficient to manage the exception automatically and unwind the process with a set of actions to preserve the data integrity. In other cases, the user must decide how to solve the inconsistency. This approach is known as data reconciliation—when the system catches an exception, it creates a new user task, and the user has three options: retry, change the data and retry, or re-throw the exception.

This is the basic requirement, but sometimes an organization requires a sophisticated process to handle the exception. For example, the organization might want to manage a "triage" process which requires that when the first error detection occurs, the error is routed to the specific information stakeholder who can decide which resolution strategy is correct. For example, the user might understand that the failing system needs an update before retrying to insert the data again. Conversely, the information stakeholder might know that the error comes from another system or by a human mistake, so he can change the data (the request payload) in order to successfully complete the process.

With the last option, the user thinks that the process is unrecoverable, so the best action is to re-throw the exception to the process engine. In this case, it's important that the process is designed to handle the exception and gracefully unwind the process instance by calling the compensation logic and leaving a clean situation.

Exception handling with decorum

The usual approach is to wrap the external system interaction with a sub-process that manages the exceptional situation. In this way, whenever the "business process" has to deal with an external system where data reconciliation is expected, the designer has to call the sub-process that engages the user for the resolution.

Even if this approach works, it has the big disadvantage of creating a new process instance for each system interaction that requires reconciliation in case of errors. For example, when a process is completed that contains three calls to systems that require such extra care, the process administrator will see four process instances. Even if the sub-process life is really short, it represents an overhead and leaves the process history cluttered.

With Red Hat Process Automation Manager, it's possible to address this problem with a really elegant solution: the Exception Handling Decorators.

In general, the decorator pattern is a way to add a new behavior to an existing class. It's similar to inheritance but has the benefit that it can be added at runtime. In this specific use case, it's used to manage the work item exception. Since any remote call to an external system is managed by a Work Item Handler, we can leverage the decorator to catch the exception and create on the fly a process to manage it when needed.

The idea is quite simple and the flexibility of the runtime enables it. In the following repository, you will find an implementation that works pretty well, but you are free to extend and improve it: decorator project.

The decorator in action

To understand how the decorator works in practice, you have to compile and install it in a Maven repository that is reachable by the Process Server (sometimes known as the Kie server). If you have a standalone installation in your local machine, it's enough to issue the following command:

mvn install

In the following repository, you will find a sample Process Automation Manager project that leverages the decorator to implement the data reconciliation strategy: decorator usage project.

Clone it in your Business Central repository to inspect the content and deploy it.

The project proc-decorator-usage defines the decorator as a dependency in its own pom.xml file, and then the configuration relies on the work-item-handlers section of the kie-deployment-descriptor.xml file. For the sake of simplicity, in this sample, the decorator replaces the standard REST Work Item Handler, but if you want to restrict the influence of the decorator to a subset of REST calls, you can create another Work Item Handler definition (for example, ReconciledRest).

The following screenshot shows the Business Central view where the decorator is defined:

In the Work Item Handler definition, there is the name that uniquely identifies the reusable service task and the constructor initialization (using an MVEL expression): new example.ProcessTaskHandlerDecorator( org.jbpm.process.workitem.rest.RESTWorkItemHandler.class, runtimeManager)

The first parameter is the class of the Work Item Handler that the decorator will surround. In this example, there is the REST Work Item Handler, but the same decorator can be used for any kind of Work Item Handler: the Web Services one or a custom one. The second parameter is the instance of Runtime Manager used internally by the decorator.

The decorator has other constructors:

• ProcessTaskHandlerDecorator( Class<? extends WorkItemHandler> originalTaskHandlerClass, RuntimeManager runtimeManager, String processId ). This constructor requires an extra parameter: the process ID of the reconciliation process (for example, src.exception-handling). With this constructor, the reconciliation process is defined once instead of passing it in every task, but the side effect is that the reconciliation process is the same for all tasks.
• ProcessTaskHandlerDecorator( RuntimeManager runtimeManager ) This construct omits the originalTaskHandlerClass and uses the REST one by default.

main-proc is a fictitious process with two service calls: Other System and Failing System. The latter is the REST call that is configured to fail and will trigger the decorator logic to manage the reconciliation logic. This chain of calls is wrapped with exception handling logic. If the exception reaches this level, it means that the reconciliation process was not able to repair the failure and the only option is to give up that instance and abort everything. Technically, it's a managed throwing back of the original exception to the main process. It's a good practice to design the process to manage the unexpected such that the process is gracefully unwound: the goal is to leave a clean situation. This is often managed with compensation logic such as this: cancel the reservation, invalidate a record, and send an email to warn that the process failed. Here is a diagram of main-proc:

Failing System is a standard REST call with all the parameters that the REST Work Item Handler expects except that the processId parameter is used by the decorator to identify the process to start when a reconciliation is required. The request is designed to fail, because the URL is http://localhost. So unless the host has an HTTP server listening on port 80, it will raise an exception and the decorator will show its behavior.

exception-handling is the process in charge of the reconciliation. The decorator will trigger it when an exception is raised by the Failing System, which is always in our test case, but in real cases, the main process completes its normal life without engaging the reconciliation process. The following is a diagram of the sample reconciliation process.

exception-handling is the simplest possible process for managing the reconciliation: one human task and a gateway to the normal end or to the error end. The human task form contains the following fields:

• Url to change the endpoint of the REST Request. In the test case context, it is useful to switch the request on a working endpoint and simulate a successful retry.
• Information to change the request payload and ask for a retry.
• Exception flagging. This is when the user closes the reconciliation process with a failure. It means that it's not possible to find a solution and the original error is sent back to the originator process (main-proc).
• Retry if the previous field is false. This controls the retry behavior. If this field is true, the call to the external system is issued again using the parameters changed above in the form. If this field is false, the Failing System task is flagged as done and the main process can continue its life. From a reconciliation point of view, this means that the user has fixed the problem with other tools (outside of Process Automation Manager).

The decorator transfers all the input parameters of the task to the exception-handling process, so the user can inspect and change the input parameters that caused the exception. In fact, when the exception handling process is completed successfully and a retry is requested, the decorator executes again the original task logic with the updated parameters (the REST call, in this case). By the way, if the second execution fails, a new reconciliation process is instantiated and so forth.

The following steps show the runtime behavior:

1. Build and deploy the proc-decorator-usage project.
2. Launch main-proc in the Process Definition view. Fill in the process form with any data and click submit.
3. The process instance page is opened. Click Diagram; there you can notice that the Failing System is highlighted. The fact that a process is waiting on a REST task denotes that the decorator is working. The default REST task behavior is synchronous. The external request is issued and the result must come back within the timeout limits; otherwise, an exception is raised and the process handles the exception immediately. In the case of error, the decorator changes the normal REST task behavior to asynchronous behavior; the process execution is suspended and waits for the completion of the reconciliation process. The following is a diagram of the main process waiting for reconciliation.
4. Switch to the Process Instances view. You should find two active process instances: main-proc and exception-handling. The latter was instantiated by the decorator. This screenshot shows the process instances list.
5. Switch to the Tasks view. The exception task is created for the person in charge of the reconciliation. In this sample, all users can work on this task but in real cases, it should be routed to the people responsible for the Failing System. It's possible to extend this sample by adding an input parameter to the failing task to identify the users in charge of the reconciliation for that task; then, that parameter can be used to dynamically assign the potential owners of the human task.
6. Open, claim, and start the task. You can test three possible routes:
   • don't retry: Just complete the task. This will close successfully the reconciliation process and flag as completed the Failing System task. Then even the main-proc will complete.
   • retry: Flag retry, change the information fields and provide a URL to match an existing HTTP POST endpoint. Eventually, use SoapUI or another similar tool to mock up a REST service and check the modified payload. If the REST call will succeed (get back an HTTP 200-level response), the processes will even be completed successfully. Otherwise, the first exception-handling process will be completed and a new one is started while main-proc still waits.
   • re-throw exception: Finally, flag the exception field to simulate the reconciliation failure. The decorator will throw the original exception to main-proc. Check in the log for the graceful unwind script execution. The main-proc process will complete, whereas the exception-handling process is aborted, because it reaches the error end node.

There's a final consideration: in this sample, the exception form is bound to a specific data object (Information). It's possible to create a user interface for the reconciliation task that is able to render a generic object and, in such a way, to have one reconciliation process to handle different payloads.

Analyzing the decorator implementation

This section provides a brief description of the decorator implementation. The source code contains some interesting examples of Process Automation Manager API usage.

ProcessTaskHandlerDecorator

It extends the org.jbpm.bpmn2.handler.AbstractExceptionHandlingTaskHandler class that is provided by the Process Automation Manager libraries and implements the decorator pattern for Work Item Handlers.

• handleExecuteException. This method is called when an exception happens in the decorated Work Item Handler while running the executeWorkItem method. It starts the reconciliation process and attaches a completion listener implemented by the ErrorHandlingCompletion class. All input parameters of the decorated Work Item Handler are transferred to the reconciliation process.
• handleAbortException. This method is called when an exception happens in the decorated Work Item Handler while running the abortWorkItem method. The decorator just logs the situation.
• rethrowException. This method re-throws the original exception to the process containing the failing Work Item Handler. It's called by the ErrorHandlingCompletion class.

ProcessCompletionListener

This is an abstract class that listens to a specific process instance completion. The sub-class has to implement two methods: processCompleted, which is called upon successful completion, and processAborted, which is called when the process reaches an error end node. This abstract class is designed to be reusable even in other contexts.

ErrorHandlingCompletion

It extends the previous abstract class and handles the reconciliation process completion.

• processCompleted is executed when the reconciliation process completes normally. If the retry variable is true, the process variables override the original input parameters of the decorated Work Item Handler and it is executed again. Otherwise, the decorated Work Item Handler is closed as aborted (abortWorkItem). It's arguable if a not-executed task should be considered aborted or completed, but in both cases, the containing process continues its normal execution.
• processAborted is executed when the reconciliation process completes with an error end node. The original error is thrown back to the main process.

Conclusion

Red Hat Process Automation Manager is a powerful technology for reducing data inconsistencies across different systems, which is one of the key benefits provided by a Digital Process Automation initiative. This article proposes a general approach and a reusable asset that streamlines process design.