This article is about a real problem I faced where the timezone on a Java application server (in my case it was JBoss) changed unexpectedly during the run time of the server. It was hard to find any pattern or the reason for the change, as it was triggered by a HTTP request. To debug this scenario, I used the Byteman tool and injected the script into the JVM. This helped me to identify the root cause of the issue and come up with a few Do's and Don'ts for a shared JVM (like on Java application servers).
Any application server is considered a shared JVM. There are multiple applications deployed on the JVM and they share the same resources. In such a scenario, there are some precautions which need to be taken care of. One of them is dealing with the JVM's timezone.
Byteman is a tool that makes it easy to trace, monitor, and test the behavior of Java applications and the JDK runtime code. It injects Java code into your application APIs or into Java runtime methods without the need for you to recompile, repackage, or even redeploy your application. Injection can be performed at startup or in running code.
The simplest use of Byteman is to inject print statements that trace what your application is doing, identifying control flow through your code and displaying the values of static or instance data. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you avoid the expensive performance overheads, which normally arise when you switch on debug or product trace.
There are many industries like finance and insurance which rely on the Java applications deployed on the application servers to run their day to day business, which also includes coupon calculations, premium calculation, etc. All these business calculations are mainly based on date and time. In case one of the applications accidentally changes any of the date and time attributes, specially the timezone on a shared JVM (like any application server), this could cause huge business losses or anomalies in the data, which could be hard to reconcile.
Usually on a JVM the timezone is always set at the start-up. Then the question arises how could it change during the run time and affect all the deployed applications?
There could be business cases where you need to manipulate the timezone temporarily for a calculation. The normal practice is to manipulate it in a Calendar object:
Calendar.getInstance().setTimeZone(TimeZone.getTimeZone("UTC"));
e.g.:
Calendar cal = Calendar.getInstance();
cal.setTimeZone(TimeZone.getTimeZone("America/Chicago"));
SimpleDateFormat sdt = new SimpleDateFormat("dd/MM/yyyy HH:mm");
sdt.setTimeZone(cal.getTimeZone());
While doing this, the timezone is only set for this calendar object and nothing gets disturbed over the JVM.
However, sometimes unknowingly (or due to developer's mistake), the code could be written as:
TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
This line of code could override the timezone on the JVM. After execution of this line, the application will have timezone set as UTC (as per example). Therefore all the date/time-based calculations will produce the erroneous data.
If these changes are performed in a code base which is deployed on the server as war/jar file, and if the source code is available for review, we can check the code. However, if this line of code is in a library file for which source is not available, its hard to find the root cause.
To check that we can follow the following approach:
1. Download the latest Byteman binary from here .
2. Extract it on the same node where shared JVM exists (i.e. JBoss, etc.).
3. Once extracted, create a byteman script as mentioned below and keep it in the classpath of the application server.
RULE check setDefault
CLASS java.util.TimeZone
METHOD setDefault(TimeZone)
AT ENTRY
IF TRUE
DO traceStack("XXX attempting to get change the default timezone "+ ": parameter : " +$1 + " : detail : " + $1.getDisplayName())
ENDRULE
4. Configure the byteman (as below) in your application server (JBoss for example), so you can make these changes in standalone.conf where "extracted path" is the path where you extract the byteman zip file, and scriptpath is the path where your TEMPhas is kept in your byteman script:
JAVA_OPTS="$JAVA_OPTS -javaagent:<extractedpath>/lib/byteman.jar=script:<scriptpath>/examplescript.btm,sys:<extractedpath>/lib/byteman.jar"
5. Make the above changes and restart the server.
6. Whenever the method "TimeZone.setDefault (TimeZone zone)" is called, it will print a stack trace to show the call. This is how you can trace the call and reaches to the code which is causing the issue on the shared JVM.
This example/troubleshooting is from a real use case where the premium calculations were disturbed due to the change in timezone. In the original issue, the code was not calculating the premium as after changing the timezone (e.g. UTC). The premium calculation was getting delayed by a day. This was causing loss of one day to the organization.
Similarly Byteman can help in debugging issues on JVM caused by the deployed application without any changes over the server in deployments. As explained above you can debug the issue and find the code/library which is causing the issue. As a best practice the developer should be aware of the date and time APIs provided by Java and use them wisely in the code. Otherwise, the introduced bug in the application could cause huge losses to the business for which the application was developed.
