JDK Flight Recorder support for GraalVM Native Image: The journey so far

July 23, 2021
Jie Kang
Related topics:
JavaKubernetesQuarkus
Related products:
Red Hat build of OpenJDK

Over the past year, Oracle and Red Hat engineers have worked together to bring JDK Flight Recorder (JFR) support to GraalVM Native Image. Prototype pull requests were first introduced by Red Hat and Oracle in late 2020 and early 2021. Work then continued in a shared repository, with plans to contribute a new pull request incorporating work from all parties. In June 2021, we merged the first pull request that introduces the infrastructure for supporting JDK Flight Recorder on GraalVM with OpenJDK 11 upstream. It is available in GraalVM's 21.2 release. This article shares some of the details behind that story: What native images are, why we worked to add JDK Flight Recorder, some of the technical challenges we faced, and what we're looking to do next.

Performance profiling for GraalVM Native Image

GraalVM is a high-performance runtime that includes support for producing native images of Java applications. This feature takes Java class files and produces a binary executable for a target platform. The resulting native executable incorporates a runtime system written in Java, called SubstrateVM, which includes components such as memory management and thread scheduling. This is similar to how OpenJDK uses the HotSpot virtual machine (VM) to execute Java applications. The native executable is generated via iterative points-to-analysis and heap population, followed by ahead-of-time compilation. The native executables produced by GraalVM ideally have a significantly faster startup time compared to a Java application running on a traditional JVM like HotSpot.

Binary executables promising better performance still need an ecosystem of supporting tools for monitoring and performance profiling. At this time, however, most of the monitoring features typically used for Java applications on OpenJDK are not available for GraalVM Native Image on GraalVM. Diagnosing application performance problems is difficult without having tools to examine the application execution and runtime execution layers. This omission is critical for applications deployed in production, whether they are in containers or on bare metal.

Why JDK Flight Recorder?

JDK Flight Recorder is a profiling system in the Hotspot VM that delivers JVM internal data and custom, application-specific data through an events-based logging system. JDK Flight Recorder is designed to collect critical data in production environments with minimal performance overhead. Traditionally, the output is a JFR file (.jfr) that can be read by tools like JDK Mission Control. With JDK 14 and later, the data can also be streamed through a Java API.

Having JDK Flight Recorder support for GraalVM Native Image will provide users a strong performance profiling tool that is similar to the HotSpot VM experience. Ideally, developers who use JDK Flight Recorder to profile their Java applications on OpenJDK would have the same experience using it for native executables. JFR for native images will be a profiling system in SubstrateVM that delivers SubstrateVM internal data, and custom, application-specific data with the same JFR file and streaming capabilities found in HotSpot.

JFR for GraalVM Native Image would provide internal data on SubstrateVM systems, mimicking much of the kinds of data seen in HotSpot. This includes information about garbage collection operations, thread state, monitor state, exceptions, safepoints, object allocations, file or socket I/O events, and more.

Profiling with internal and custom events

JDK Flight Recorder and other profiling tools in HotSpot help us answer a wide variety of questions. JDK Flight Recorder in GraalVM Native Image aims to do the same. Some of these questions include:

  • How much execution time is consumed by the garbage collector?
  • How much time is spent in safepoint, stop-the-world operations?
  • Are there bottlenecks in the execution flow and where are they?
  • What are the hot methods in my application?
  • How much time is spent on I/O processing?
  • What about synchronization issues like deadlocks and operations waiting on a lock?

Along with internal events, JDK Flight Recorder has an API for developers to add their own events to the system. This lets users add even more data for analysis that benefits from the existing low-overhead, production-oriented JFR infrastructure. Custom events can take advantage of the common pool for runtime constants (class names, method names, strings, and so on) for overall reduced output cost compared to separate, custom, event or metric emission agents.

Solving performance problems

Events by themselves are not enough to diagnose performance issues. Fortunately, the existing JFR format lets us quickly integrate with visualization and analysis tools like JDK Mission Control. We can use these tools together to solve performance problems in our native image applications. Any of the events in SubstrateVM that are one-to-one matches to events in HotSpot will already be understood without any changes to the existing analysis tools.

Note: JDK Flight Recorder for HotSpot has a plethora of resources, some of which I’ve listed at the end of the article. Please take a look to learn more about JDK Flight Recorder in general and the benefits it brings to solving performance problems.

Implementing JDK Flight Recorder support for GraalVM Native Image

Prior to our work, there was no JDK Flight Recorder support for GraalVM Native Image, and we've encountered many challenges in implementing that support. While the Java API is accessible, there is no mechanism to manage the JDK Flight Recorder system. Additionally, existing code that uses the JDK Flight Recorder Java API directly, for example, to start a recording, fails with an unsatisfied link error. The failure happens because the JDK Flight Recorder native API is not present. This particular error occurs because the native JDK Flight Recorder code found in the HotSpot library, libjvm.so, is not included in the SubstrateVM system. Unfortunately, the native code isn’t suitable for inclusion because of its deep ties to HotSpot internals.

To add JDK Flight Recorder to GraalVM Native Image, we reimplemented the native infrastructure for JDK Flight Recorder in the HotSpot VM entirely in Java code in SubstrateVM. However, we were not able to perform a one-to-one translation of the HotSpot code, written in C++, to a SubstrateVM equivalent in Java. This section explains some of the reasons why.

Class transformation

First, the event class in HotSpot is nominally empty and class transformation is done lazily at runtime to add implementations to the methods for enabled events. The methods have empty bodies and are marked final, preventing extends. See the code in EventInstrumentation to see what the implementations look like when the event is transformed.

The transformed event class is needed in SubstrateVM to emit events but the class transformation cannot be done at runtime for native images. OpenJDK JDK Flight Recorder has the boolean flag retransform to instrument classes via the Java Virtual Machine Tool Interface (JVMTI). The default value is true. To get around this, we can set the value to false for the JVM being run for SubstrateVM analysis, for example via


$ mx native-image -J-XX:FlightRecorderOptions=retransform=false ...

The resulting event classes that SubstrateVM sees will contain the transformed implementations as if events were enabled and JDK Flight Recorder was running.

Another solution is to transform the event classes before compilation by using the internal API retransformClasses in jdk.jfr.internal.JVM. This workaround also relies on implementation details of HotSpot JDK Flight Recorder, but it allows for more fine-grained control over exactly which classes to transform.

Uninterruptible safepoints

By default, native code (HotSpot) cannot be interrupted for a safepoint, whereas Java code (SubstrateVM) can. Fortunately, SubstrateVM has the uninterruptible annotation that can be used to mark method blocks that will not be interrupted for a safepoint. It’s not ideal to mark all the "native" portions as uninterruptible, as performance will be impacted by these sections. As well, some JDK Flight Recorder code paths in HotSpot are commented with “can safepoint here” (and marked appropriately to allow safepointing), so we still need to pay careful attention to the SubstrateVM implementation to ensure consistency is maintained.

Using the Uninterruptible annotation also comes with a restriction: No object allocation is allowed. This proves quite problematic when implementing familiar code patterns in Java, an object-oriented programming language. Even logging can be difficult as any toString method that allocates is not usable in an uninterruptible method call. This has a follow-on in making debugging a bit more painful than normal, especially when dealing with concurrency issues where simple debugger strategies don’t see the same execution flow as the issue being hunted.

Additionally, some critical parts of SubstrateVM, such as the garbage collector, have methods that are marked uninterruptible. We must be able to trigger the event emission infrastructure in these locations, and so its components also need to be uninterruptible. This requires that they do not allocate Java objects. Fortunately, SubstrateVM provides methods to malloc, or free memory, that are not managed by garbage collection. These instances do not extend Object, though, and so any existing Java API, like ArrayList, cannot be used to manipulate the data. These data structures must be reimplemented if they are needed.

The JFR Recorder thread

Some critical work for writing data is given to a single thread in HotSpot, the JFR Recorder thread, which is itself excluded from the JDK Flight Recorder system. This exclusion cannot be mimicked in SubstrateVM because any thread can be appropriated to perform a virtual machine operation such as garbage collection. The garbage collector could emit relevant JDK Flight Recorder events and exclusion would result in missing event data from such operations.

It is even possible for the closing of a chunk—where events are written to disk, the epoch is transitioned, and metadata and constant pools are emitted—to be paused for a garbage collection operation. This action could then write garbage collection events to the buffer. The tasks here need to be analyzed and marked appropriately as uninterruptible to keep consistency for these garbage collection events and their constant pool data. The write operation to close the chunk, which I described earlier, needs to be carefully written as a safepoint operation with uninterruptible portions to account for the possibility of appropriation for garbage collection.

Conclusion

JDK Flight Recorder is an important feature of the HotSpot VM, giving users an impressive amount of data about the JVM’s execution with low overhead suitable for continuously running in production environments. It will be useful for developers to have access to the same tool when running native image applications with SubstrateVM.

The initial merge for JFR infrastructure is complete but there is a long road ahead before the system can provide a view into native executables produced by GraalVM that is similar to what is possible for HotSpot. Up next is the work to add events for garbage collection, threads, exceptions, and other useful locations in SubstrateVM. The RJMX API is not currently implemented for SubstrateVM, so there is no API to remotely manage JDK Flight Recorder. Alongside this, the JFR system is still being significantly improved and enhanced in OpenJDK and HotSpot. We will need to account for major changes that affect the API in the SubstrateVM implementation to properly support the different underlying OpenJDK versions. At this time, the codebase only targets OpenJDK 11.

For now, you can try out the latest GraalVM and test JDK Flight Recorder by passing the flag -H:+AllowVMInspection to the native-image process at build time. After you've done that, you may add flags such as -XX:+FlightRecorder -XX:StartFlightRecording="filename=recording.jfr" to the application binary at runtime.

Alongside GraalVM, Mandrel, the community distribution for Quarkus, will naturally inherit JFR support. As we continue to improve and enhance JDK Flight Recorder for GraalVM Native Image, please look for more announcements and articles about using JFR with native images in Quarkus and Mandrel environments.

Learn more about JDK Flight Recorder

The following resources are available for learning more about JDK Flight Recorder and using it to solve performance problems:

Acknowledgment

The author thanks Alina Yurenko and Oracle for their time in reviewing this article.

Last updated: September 19, 2023