Cryostat 2.2's new JMX credentials keyring

An important security enhancement for Java applications is now available through Java Management Extensions (JMX), using open source JDK Flight Recorder (JFR) management with Cryostat. (Cryostat is a container-native JVM application that provides a secure API for profiling and monitoring containers with JFR.) This article explains how the new JMX credentials keyring offers an alternative credential management system.

How the JMX credentials keyring works

If applications are configured to require JMX authentication for incoming connections, Cryostat must know what those credentials are for any application it needs to connect to. Cryostat connects to the target application in order to start, list, or stop flight recordings, copy JFR data to a local file, get information about JFR event types and templates, etc.

Before Cryostat 2.2, connections to an application through JMX would include an X-JMX-Authorization header. This header's value would be the JMX credentials required by the target application, so that Cryostat could pass the authentication challenge and connect to the target. The credentials passed in this way weren't stored by Cryostat persistently; they were held in memory just long enough to establish the JMX connection. If you set up your Cryostat installation to use HTTPS and configured your target application to use JMX over SSL/TLS, these credentials were always encrypted while being transmitted.

Cryostat 2.2 adds a keyring to store JMX credentials for your applications. Credentials stored in this way are encrypted at rest using a user-provided passphrase supplied to Cryostat via the CRYOSTAT_JMX_CREDENTIALS_DB_PASSWORD environment variable.

When attempting to open a JMX connection to a target, Cryostat first checks for the X-JMX-Authorization header. If there is none, Cryostat checks the keyring database for credentials matching the target, decrypts them using the passphrase, and establishes the JMX connection. Again, if your Cryostat installation is set up with HTTPS and your target uses JMX over SSL/TLS, these JMX credentials are always encrypted in flight, and now they're encrypted at rest within the keyring.

Advantages of the keyring for microservices

You might be wondering what I meant by "credentials matching this target" in the previous paragraph. Older versions of Cryostat required the user to define credentials on a per-target basis. Each JMX credential username/password pair was tied to one specific target connection URL. This works fine if your applications are deployed as long-lived "pets" with stable IP addresses, but is not very useful if your applications are deployed as microservice "cattle," where the URL to one specific replica or instance is not particularly meaningful.

For this reason, credentials stored in Cryostat 2.2's JMX keyring are no longer tied to specific target connection URLs. Instead, they reuse the same match expression concept that was previously introduced for Cryostat automated rules. To recap, match expressions are small snippets of Java-like code, where a representation of a target application is passed and available as a target object reference. The expression evaluates various properties of the target object to determine whether the automated rule or the JMX credentials should apply to that target.

The most reliable way to make sure JMX credentials work using this system is to copy and paste the match expressions from your automated rules definitions and apply them to the JMX credentials, so that the rules and credentials apply to the same set of target applications.

Additional considerations for using JMX credentials with Cryostat

Because the connection asking for the JMX credentials challenges the target applications to which it connects, and not Cryostat itself, Cryostat is required to store the whole credential and not just a salted hash of it. Storage operates much like the login keyring you might be familiar with on your personal computer, or a password manager you use for storing your logins for online accounts.

An additional enhancement in Cryostat 2.2 lets you define your JMX credentials after creating an automated rule. Previously, if an automated rule failed to activate for a target application—due to JMX authentication failure, for example—the rule's actions wouldn't trigger for that target application at all. To make the connection work, after you added the credentials, you had to either delete and recreate the rule or restart the application. Now, if you add JMX credentials to the keyring after the fact, your automated rules will re-trigger and attempt to perform their actions against the relevant target applications.