Automating JDK Flight Recorder in containers

This article is part of a series of hands-on guides to using Cryostat 2.0, or JDK Flight Recorder for containers. This article introduces Cryostat's new API for automated rules. We'll walk through two use cases highlighting the API's compact but powerful rule definitions. You'll see how to use rule definitions to specify a match expression for one or more target Java applications, and how to configure the type of flight recording you want to start on these targets.

Once you've created a rule, Cryostat immediately matches it against all existing discovered targets and starts your flight recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.

The automated rules API is brand new in Cryostat 2.0, and we haven't yet developed the user interface (UI) for it. For now, we'll use curl to interact with the Cryostat HTTP API directly.

Read all of the articles in this series:

• Part 1: Get started with Cryostat 2.0
• Part 2: Configuring Java applications to use Cryostat
• Part 3: Java monitoring for custom targets with Cryostat
• Part 4: Automating JDK Flight Recorder in containers
• Part 5: Creating Custom JFR event templates with Cryostat 2.0

Note: The Red Hat build of Cryostat 2.0 is now widely available in technology preview. Cryostat 2.0 introduces many new features and improvements, such as automated rules, a better API response JSON format, custom targets, concurrent target JMX connections, WebSocket push notifications, and more. The Red Hat build includes the Cryostat Operator to simplify and automate Cryostat deployment on OpenShift.

Use case 1: Continuous monitoring in a containerized JVM

Previously, if we wanted to enable always-on continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat's automated rules, we can enable JDK Flight Recorder at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.

We can also adjust the continuous monitoring event template at runtime, with no downtime. First, we create a rule with the template we want to use. Then, we create a new template and a new rule with the new or updated event template. Finally, we delete the original rule with the clean=true parameter. Here's the source:

$ export CRYOSTAT=https://cryostat.example.com # replace this URL with your actual Cryostat instance URL

$ curl -F name="firstRule" -F matchExpression=”target.alias==‘com.example.MainClass’” -F eventSpecifier=”template=Profiling,type=TARGET” $CRYOSTAT/api/v2/rules # Create a Rule named firstRule using the Profiling template

$ curl $CRYOSTAT/api/v1/targets/$serviceUri/templates/Profiling/type/TARGET -o profiling.jfc # Download the Profiling template to your computer

$ $EDITOR profiling.jfc # Edit the Profiling template to update the event definitions inside, and to rename it from Profiling to Demo

$ mv profiling.jfc demo.jfc # Rename the file to demo.jfc so it's more identifiable as the Demo profile

$ curl -F template=@demo.jfc $CRYOSTAT/api/v1/templates # Upload the new Demo profile to Cryostat

$ curl -F name="secondRule" -F matchExpression=”target.alias==‘com.example.MainClass’” -F eventSpecifier=”template=Demo,type=CUSTOM” $CRYOSTAT/api/v2/rules # Create a second Rule using the Demo profile

$ curl -X DELETE $CRYOSTAT/api/v2/rules/firstRule?clean=true # Delete the first Rule that used the Profiling template and clean up any active recordings it created

Use case 2: Custom monitoring with Kubernetes labels or annotations

We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here's an example in JSON notation:

{
  "name": "k8sMonitoring",
  "description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
  "matchExpression": "target.annotations.platform[‘jfrMonitoring’]==’enabled’",
  "eventSpecifier": "template=Demo,type=CUSTOM",
  "archivalPeriodSeconds": 300,
  "preservedArchives": 12
}

Once we've created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a recording started automatically using the custom Demo template from our first use case. It will take an archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.

With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.

As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application's behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these archived recordings in Cryostat’s Grafana dashboard.

Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.

How to use match expressions in Cryostat

The matchExpression in a rule definition is a Java-like snippet of code that Cryostat interprets and uses to determine if a rule should be applied to any given target.  matchExpressions should thus evaluate to a boolean value. The simplest matchExpressions would be the booleans true or false; if we use true, the rule will apply to every target. The expression has a target object in global scope, with the following form in JSON notation:

{
  “alias”: “myAppAlias”,
  “connectUrl”: “service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi”,
  “labels”: {
    “com.example/service”: “customer-login”,
  },
  “annotations”: {
    “platform”: {
      “io.kubernetes/annotation”: “annotated”
    },
    “cryostat”: {
      “PORT”: 9091,
      “HOST”: “cryostat”,
      “NAMESPACE”: “myproject”
    }
  }
}

The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The labels and platform annotations may be empty—in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty.

Here are some examples of matchExpressions:

target.alias == ’com.example.MainClass’

target.alias == ’myAlias’

target.labels[‘com.example/service’] == ’customer-login’

target.labels[‘com.example/service’] != ’customer-login’

target.annotations.cryostat.PORT > 3000

target.annotations.cryostat.PORT > 3000 && target.annotations.platform[‘io.kubernetes/annotation’] == ‘enabled’

!!target.annotations.platform[‘io.kubernetes/annotation’]

/^customer-login[0-9]*$/.test(target.alias)

Conclusion

In this article, you've learned how to use match expressions to create automated rules for Cryostat monitoring. Once you've set up your automated rules, Cryostat will continuously monitor applications that meet the criteria defined in those rules, with no need to restart or redeploy those applications. Visit Cryostat.io for further details.