Inject custom JDK Flight Recorder events in containerized applications

The JDK Mission Control (JMC) agent is a powerful tool that allows users to inject custom JDK Flight Recorder (JFR) events at runtime without needing to restart the Java virtual machine. Just as the JMC agent plugin simplifies the process of using the agent in a non-containerized environment, the Cryostat agent plugin does the same for containerized environments.

JMC agent support is now merged into Cryostat, and Cryostat supports various API handlers for using the JMC agent in a containerized environment. This article introduces the Cryostat agent and its API handlers.

Note: To get up to speed on the technologies discussed in this article, check out JVM performance monitoring with JMC agent and Introduction to Cryostat: JDK Flight Recorder for containers.

Prerequisites and setup

Cryostat agent support is available with the Cryostat 2.1 release. To get started, simply download and install Cryostat 2.1, and the agent support will work out of the box. Instructions for installing Cryostat can be found in the getting started guide.

In order to use the JMC agent integration with Cryostat, you must ensure that its JAR is present in the same container as the application. Then, run the application with the javaagent flag specifying the JAR:

-javaagent:target/org.openjdk.jmc.agent-1.0.0-SNAPSHOT.jar

Note: You can download the JMC agent from Adoptium's JMC overrides releases.

Using the Cryostat JMC agent integration

The Cryostat agent offers a set of API handlers that interact with the JMC agent and manage templates for custom events. The ProbeTemplate handlers allow users to post or delete probe templates. These templates are XML files that describe the custom events to be injected; they are stored with Cryostat and can be applied to any valid targets with the agent running, or deleted when they are no longer needed. Similarly, the TargetProbePost, Delete, and Get handlers facilitate adding, removing, and retrieving custom event configurations from target JVMs.

Custom event monitoring

For an example workflow, suppose the Cryostat agent is currently running with Cryostat itself as the target application. Now, you want to add custom events to Cryostat to monitor how it's running. To start, you need a probe template to work with, so consider the following:

<jfragent>
    <!-- Global configuration options -->
    <config>
        <!-- This is the prefix to use when generating event class names. -->
        <classprefix>__JFREvent</classprefix>
        <!-- Will allow the recording of arrays and object parameters as Strings.
             This will cause toString to be called for array elements and objects other
             than strings, which in turn can cause trouble if the toString method is badly
             implemented. Use with care. -->
        <allowtostring>true</allowtostring>
         <!-- Allows converters to be used.
              See the org.openjdk.jmc.agent.converters package for more information. -->
        <allowconverter>true</allowconverter>
    </config>
    <events>
        <event id="demo.cryostat.jfr">
            <label>CryostatDemoEvent</label>
            <description>Event for the agent plugin demo</description>
            <path>demo</path>
            <stacktrace>true</stacktrace>
            <class>io.cryostat.net.web.http.generic.HealthGetHandler</class>
            <method>
                <name>handle</name>
                <descriptor>(io/vertx/ext/web/RoutingContext;)V</descriptor>
            </method>
            <!-- location {ENTRY, EXIT, WRAP}-->
            <location>ENTRY</location>
        </event>
    </events>
</jfragent>

This is a simple configuration that adds a custom event to the entry point of the handle method. Once injected, this configuration will emit a custom CryostatDemoEvent every time the handle method is called.

The <config> section of the template determines the prefix used for the custom event classes, as well as whether any advanced functionality is required. The Cryostat agent supports the capture of fields and method parameters to be emitted with the event. It then utilizes user-defined converter methods to convert those fields and parameters into content types that JDK Flight Recorder can use. However, this functionality must be specifically enabled in the <config> section.

Following the <config> section, you can specify any number of events to be injected, you only need to provide an event ID, label, and description. The <path> element corresponds to the path that the event will appear under when viewed in a graphical tool like JDK Mission Control. The <stacktrace> element determines if stack traces should be recorded with the event.

The <class>, <method>, and <location> elements determine precisely where the event should be injected. You must provide the formal descriptor for the method here; this takes the form: (Parameter Descriptor*)Return Descriptor. For example, for the method int isAlive(), the method descriptor would be ()Z. See the JVM specification for more about the method descriptor format.

Injecting and recording custom events

Once you have a probe template, all you need to do is send a few API requests to Cryostat to upload it and apply it to the target. (Remember that, in this case, the target is Cryostat itself.)

$ curl -F probeTemplate=@cryostat-probe.xml http://0.0.0.0:8181/api/v2/probes/

This first request calls out to the ProbeTemplateUpload handler (api/v2/probes). This handler expects the ProbeTemplate to be uploaded in a form with the name probeTemplate.

Now you just have to send one more request:

$ curl -X POST localhost:8181/api/v2/targets/localhost/probes/cryostat-probes

The second request calls out to the TargetProbePost handler (api/v2/targets/:targetID/probes/:probeTemplate). When you provide this handler with a target and a probe template, it will apply the probe template to the target, injecting the custom events. The next time a recording is run, the custom events will be recorded.

Conclusion

The JMC agent is a powerful tool for creating and recording custom events without the need to manually write those events or rebuild an application. The Cryostat agent brings the same advantages to a containerized environment. With JMC agent support being merged upstream, it is now easier than ever to take advantage of these capabilities.

Last updated: September 20, 2023