Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Diagnosing Java applications on the fly with Byteman

November 6, 2018
Marko Myllynen Andrew Dinn
Related topics:
JavaDeveloper Tools

Share:

    Production being affected by software issues is always an unwanted scenario. Diagnosing production issues, however, should never be an unplanned activity. Structured testing and QA efforts would ideally prevent any software bugs from entering production. So the dilemma is how to prepare for something unexpected in production that was not considered during the earlier testing and QA phases.

    This article discusses Byteman, a tool that leverages the Java Instrumentation API to inject Java code into methods without the need to recompile, repackage, or even redeploy the application.

    To troubleshoot production issues, the answer lies with adaptive tools that allow for dynamically diagnosing selected functionality with minimal disruption for the rest of the system. Think of a recently introduced new software component or function that, as a whole, is not consistently performing as expected, and its existing instrumentation is not providing the needed insight about the possible root cause of the issue. Since merely restarting a complex application might be unfeasible in production or make the issue go away for a certain period, you need to be able to safely modify running code on the fly to avoid time-consuming delays while diagnosing the issue at hand.

    In more concrete terms, you could start diagnosis efforts by adding timings for selected application methods and monitoring their error rates when the system is still running and showing issues. This would allow you to narrow down the exact problem area in an iterative manner without affecting other parts of the application. Given that these retrospective code changes would be done in production, the modifications should be absolutely certain not to introduce additional and possibly more severe problems. This means the changes need to be done using generic, proven tools but in an application-specific manner.

    The Java Instrumentation API allows modification of the bytecodes of methods on the Java Virtual Machine (JVM) at runtime. While technically this would make it possible to implement any wanted changes to an application to gain insight into its behavior, approaching an unclear production issue at a bytecode level would be the complete opposite of the high-level iterative approach described above.

    Byteman to the rescue

    Byteman, unlike many other bytecode transformers, operates at the level of Java, not bytecode. You give Byteman one or more rules that specify the Java code you want to be executed and the location in methods where you want it to be injected. Byteman works out how to rewrite the bytecode so it behaves as if the original Java code included the source-level changes you requested. Byteman also does the needed type checking and type inference, which are an absolute must for the safety of transformations.

    Below is a simple example of Byteman rules that would help you understand how often an exception is thrown during a method's execution by keeping a tally of the exception count and printing it at a certain interval:

    RULE Count exits via exceptions
    CLASS com.example.SomeClass
    METHOD someMethod
    AT EXCEPTION EXIT
    IF true
    DO incrementCounter("exceptions");
    ENDRULE
    
    RULE Print exception exit count
    CLASS com.example.SomeClass
    METHOD someMethod
    AT ENTRY
    BIND exceptions = readCounter("exceptions");
    IF exceptions % 10 == 0
    DO trace("Exception exit count for someMethod - ");
    trace("" + new java.sql.Timestamp(System.currentTimeMillis()));
    traceln(": " + readCounter("exceptions"));
    ENDRULE

    The combination of CLASS, METHOD, and location (AT ENTRY, etc.) identify where the Java code provided in the rule gets injected and executed. All expressions appearing in the BIND, IF, or DO clauses are just plain old Java code. An IF expression must always be provided to determine when the DO actions get run. The calls to the built-in convenience methods incrementCounter, readCounter, trace, and traceln actually invoke corresponding methods of a Byteman-provided helper class (aptly named Helper). The first pair enables counting of events during execution, and the second pair simply wraps calls which print to System.out.

    The IF expression is always type-checked to ensure it is boolean. By contrast, the type of rule variable exceptions is inferred to be int (using the known signature of readCounter) and is used later to check type-correctness at points of use, such as the modulo (%) or String concat (+) operations.

    With Byteman helper scripts, Java statements corresponding to these rules can be dynamically injected into a running Java application without affecting any other parts of the application except for this one particular method. Later these modifications could be removed with a Byteman helper script as well. In a way, this already provides previously unavailable information from the running application and allows you to see whether the method is executing as expected.

    Obviously, there are few downsides with this initial approach. Writing rules manually for several methods for different kinds of tracing purposes would be tedious and error-prone. Creating rules for keeping track of, for example, method execution times would be easier if you utilized custom Java classes and methods. Last, but not least, statistics should not be written to stdout but instead be available over the standard JMX interface so that they can be consumed by commonly used monitoring tools.

    Byteman automation tool

    To make diagnosing issues on the fly with Byteman easier and to address the above downsides of the manual approach, a Byteman automation tool was recently introduced.

    The tool automates the generation of Byteman rules to provide statistics from unmodified Java applications for metrics such as the number of calls per method, the execution times of methods, the exception exit count per method, the number of instances of a class, and the instance lifetimes. Any set of these statistics can be enabled and disabled dynamically and then monitored with standard tools using JMX.

    The tool only requires defining the target methods in a simple text file and then generating the wanted rules with a selected set of command-line options. Below is an example that would create rules to provide metrics for method executions times and exception exit count for three different methods:

    $ cat targets.txt
    com.example.SomeClass#methodOne
    com.example.SomeClass#methodTwo
    com.example.SomeClass#methodThree
    $ java \
        -jar ./target/proftool-1.0.jar \
          --input-file targets.txt \
          --register-class com.example.SomeClass \
          --register-method 'methodOne' \
          --call-exectimes-min \
          --call-exectimes-avg \
          --call-exectimes-max \
          --call-exit-except \
          --output-file rules.btm
    $ wc -l rules.btm
    108
    $ tail -n 9 rules.btm
    RULE Exits via exceptions from method: com.example.SomeClass - methodThree
    CLASS com.example.SomeClass
    METHOD methodThree
    AT EXCEPTION EXIT
    HELPER org.jboss.byteman.automate.proftool.JMXHelper
    COMPILE
    IF true
    DO incrementMethodExitExceptCount($CLASS, $METHOD);
    ENDRULE

    The generated rule employs a dedicated helper class part of the tool, JMXHelper. The call to its instance method, incrementMethodExitExceptCount(), updates a class/method-specific counter and makes the value available via JMX. $CLASS and $METHOD are special rule variables provided by Byteman, identifying the method the rule was injected into. That is, in this case, their values will be com.example.SomeClass and methodThree when the injected code runs.

    Now we can use Byteman helper scripts to inject all the needed Java code into a running application on a JVM to start gathering these statistics:

    $ bminstall <pid-of-jvm>
    $ bmsubmit -s proftool-1.0.jar
    $ bmsubmit -l rules.btm

    (In case JMX metrics were not enabled on the JVM on startup, a simple utility is available in the repo for enabling JMX on the fly on a JVM.)

    The above is all that is needed to see how long each of the three methods take to execute at a minimum, on average, and at a maximum and to see how often they exit due to an exception. Tools like JConsole or Prometheus could then be used for analyzing the situation and determining the next steps.

    Conclusion

    In this article, we saw how the Byteman automation tool provides a quick way to provide additional instrumentation for unmodified Java applications without even the need for restarts. This information may then be used as a basis for further troubleshooting and error correction.

    If further details are needed, Byteman supports a lot of alternative "AT" locations for injecting code to track and respond to application events like countdowns, flags, and timers. The Byteman Programmer's Guide and other resources listed below provide full details on how to use these additional capabilities.

    Byteman Resources

    • Byteman homepage
    • Byteman documentation
    • Byteman automation tool
    • Byteman Programmer's Guide
    • Byteman automation tutorial

    Other Byteman articles

    • Using Byteman to find out why the TimeZone changed on a Java app server
    • Enabling Byteman scripts with Red Hat JBoss Fuse and AMQ – Part 1
    • Enabling Byteman scripts with Red Hat JBoss Fuse and AMQ – Part 2

    Also see these Java articles on the Red Hat Developer blog and the Red Hat Developer Enterprise Java page.

    Recent Posts

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    • Assessing AI for OpenShift operations: Advanced configurations

    • OpenShift Lightspeed: Assessing AI for OpenShift operations

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue