Profiling Java workloads with the Cryostat Agent

In Cryostat 2.3, we deliver the first iteration of a long-requested feature: a JDK Instrumentation Agent for profiling Java workloads using JFR.

In this article we will look at a brief overview or refresher of what an Instrumentation Agent is, what specifically the Cryostat Agent does, reasons you should consider instrumenting your containerized applications with the Cryostat Agent, and finally a brief example of how to include the Cryostat Agent into a Quarkus application.

What is an Agent?

The JDK Instrumentation API allows additional software packages, called Agents, to be included with or attached to applications, as optional modular components. Agents can be specified by the application’s startup command line with the -javaagent flag, or they can dynamically attach to a running application process. In either scenario, the Agent can transform the host application’s bytecode - for example, to inject observability hooks or to patch application behaviour - or extend the application with additional functionality.

This latter mechanism is interesting for Cryostat, since it allows an Agent to act as a sort of plugin to applications, enhancing them with additional capabilities by simply including the Agent rather than needing to directly implement those capabilities. Specifically, the Cryostat Agent:

1. Implements the Cryostat Discovery Plugin interface. The SmallRye config property cryostat.agent.callback must be set [1]. This value defines the HTTP URL where the Agent instance can be located on the network, and the Agent uses this URL to talk to the Cryostat server instance and inform Cryostat of where it can find this Agent. For example, set the environment variable CRYOSTAT_AGENT_CALLBACK=https://myapp.example.com:8080/.
2. Provides a readonly HTTP API alternative for Cryostat functionality, removing the requirement for target applications to expose a JMX port. The cryostat.agent.callback URL above is also used as the Agent HTTP API URL if the config property cryostat.agent.registration.prefer-jmx is false (which is the default). If the application JVM does not have JMX enabled then the Agent will always publicize itself to Cryostat using the readonly HTTP API URL. If JMX is enabled then the Agent will still prefer to publicize its HTTP API URL, but this config property can be used to force publication of a read/write JMX Service URL instead.
3. Can be configured to collect JFR data from the application JVM and periodically push it to the Cryostat backend. This is conceptually similar to Cryostat’s Automated Rules feature, but using an Agent-initiated data push model, rather than Automated Rules’ Cryostat-initiated data pull model. This JFR harvesting also allows the Agent to collect the “tail end” of JFR data as the application shuts down and push that to Cryostat as an on-exit data dump, so cases of abnormal application shutdown can be detected and investigated from the Cryostat console.

[1] SmallRye config properties are most conveniently set by JVM system property, ex. -Dcryostat.agent.callback="http://myapp.pod.mycluster:9999", or by environment variable transformation ex. CRYOSTAT_AGENT_CALLBACK="http://myapp.pod.mycluster:9999"

Why should you use the Cryostat Agent?

So why use the Cryostat Agent with your applications? Let’s start with a brief recap of why to use Cryostat:

1. To discover and map out your Java applications and their deployment topology
2. To gather JDK Flight Recorder data from your applications
3. To store that JFR data in an in-cluster volume
4. To perform online analysis of that JFR data without the raw data leaving the cluster

As alluded to previously, Cryostat has historically required target applications to expose a JMX port. Cryostat talks to the application JVM over this JMX port to start and stop JFR recordings and to pull JFR files over the network, enabling the ability to store and analyze this JFR data. But, JMX has a very large surface area, can be difficult to secure, and may be undesirable even behind the curtains of an OpenShift internal network service.

Since the Cryostat Agent exposes a small, readonly HTTP API, it can be much easier to audit and secure than a JMX port. The agent does not allow for remote dynamic start/stop of JFR recordings, nor does it allow for remote pulling of JFR data. Instead, SmallRye config properties must configure a Cryostat server URL to communicate with and a schedule for pushing collected JFR recordings. The Cryostat Agent knows how to register itself with the Cryostat server as a Discovery Plugin, how to publish a description of itself to the server, how to respond to requests for information about what JFR event types, event templates, and recordings are present in the application JVM, and how to secure its HTTP API so that only the Cryostat server it has registered with is able to request that information. This way, the Cryostat console can still be used to visualize the deployment topology including Cryostat Agent instances. The JFR data pushed by Agents is properly associated with the Agent that pushed it, and that data is stored in the same volume that Cryostat normally uses. Of course, Cryostat’s online analysis tools can be used on this JFR data regardless of whether it was pulled over JMX or pushed over HTTP.

How do you use the Cryostat Agent?

So how does one get started using the Cryostat Agent? Let’s use a Quarkus application as an example. Applications built on frameworks other than Quarkus, or with tools other than Maven, should be similar. First, we will modify the application pom.xml to add a dependency on the Cryostat Agent:

 <groupId>org.acme</groupId>
  <artifactId>code-with-quarkus</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <repositories>
    <repository>
    <id>github</id>
    <url>https://maven.pkg.github.com/cryostatio/cryostat-agent</url>
    </repository>
  </repositories>
   …
  <properties>
   …
    <dependency-plugin.version>3.3.0</dependency-plugin.version>

    <io.cryostat.agent.version>0.2.0-SNAPSHOT</io.cryostat.agent.version>
  </properties>
   …
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>${dependency-plugin.version}</version>
        <executions>
      	  <execution>
            <phase>prepare-package</phase>
            <goals>
              <goal>copy</goal>
            </goals>
            <configuration>
            <artifactItems>
              <artifactItem>
              <groupId>io.cryostat</groupId>
              <artifactId>cryostat-agent</artifactId>
              <version>${io.cryostat.agent.version}</version>
              </artifactItem>
            </artifactItems>
            <stripVersion>true</stripVersion>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

These three sections define the GitHub Packages repository to pull the upstream Cryostat Agent build from, define the Agent version to use and the version of a build plugin for packaging it with our application, and finally configure the build plugin to actually include the Agent in our application build output as a separate JAR.

Next, let’s modify the Quarkus Dockerfile.jvm:

FROM registry.access.redhat.com/ubi8/openjdk-17:1.14

ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'

# We make four distinct layers so if there are application changes the library layers can be re-used
COPY --chown=185 target/quarkus-app/lib/ /deployments/lib/
COPY --chown=185 target/quarkus-app/*.jar /deployments/
COPY --chown=185 target/quarkus-app/app/ /deployments/app/
COPY --chown=185 target/quarkus-app/quarkus/ /deployments/quarkus/

# add the Cryostat Agent JAR to the application image
COPY --chown=185 target/dependency/cryostat-agent.jar /deployments/app/

USER 185
# Add the -javaagent flag so that the JVM loads the Cryostat Agent on startup
ENV JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -javaagent:/deployments/app/cryostat-agent.jar"
ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"

The last COPY ensures that the Cryostat Agent JAR is included in the application image, and the JAVA_OPTS sets the default launch command line to include a flag to load the Agent at application startup. When we deploy this application, we must also set the SmallRye config properties cryostat.agent.callback and cryostat.agent.baseuri separately as mentioned above.

Congratulations, you have successfully bundled your application with the Cryostat Agent and can now enjoy the many benefits of doing so:

• providing a simple readonly HTTP API for retrieving recording data without exposing a potentially unsafe JMX port
• allowing Cryostat to discover your application
• enabling additional features on your application such as pushing collected JFR data to Cryostat on application shutdown
• pushing collected JFR data to Cryostat on a fixed schedule for monitoring