Many Java programmers use Cryostat to monitor and report statistics on their applications. Since its early days, Cryostat has performed the discovery of target Java Virtual Machine (JVM) applications in various ways. This article demonstrates how you can fully customize your application selection with Cryostat 2.2.
Previous options for application discovery
By default, Cryostat detects the platform it is running on at startup. Users can set an environment variable to force a specific platform.
Once Cryostat starts up and selects a platform, it uses a hardcoded mechanism specific to that platform to discover target JVM applications. In Red Hat OpenShift, for instance, Cryostat queries the OpenShift API server for Endpoints objects within the same namespace as Cryostat, then filters the objects to choose ones where the port either is named jfr-jmx or has the number 9091. Each pair of IP addresses and port numbers is assumed to be a JVM application exposing a JMX port from within a pod, so Cryostat maps these Endpoints objects to its internal representation and exposes them through its API and web user interface.
Unfortunately, this way of finding applications is inflexible, and problems are quickly apparent. What if your application deployment already uses port 9091 for something other than JMX, and you want to add Cryostat now? Or what if something else in your system depends on a port naming convention, and you can't use jfr-jmx?
The past several major versions of Cryostat have also supported custom targets, allowing you to make an API request with a JMX Service URL pointing to a JVM application and a human-friendly alias to identify it. This mechanism can help make discovery accurate by informing Cryostat that you have target applications it isn't discovering on its own.
But using custom targets does not prevent Cryostat from discovering things that are not JVM applications or aren't ones that you care to interact with through Cryostat. Custom targets also miss out on more recent Cryostat versions' awareness of the deployment scenario: In OpenShift, for instance, Cryostat knows that the endpoint represents a JVM within a pod, which belongs to a deployment, which is within a namespace. In contrast, a custom target definition is just a member of a flat list outside this hierarchy. Cryostat also applies annotations to nodes in this hierarchical tree structure and copies OpenShift annotations and labels into them, which would also be missed when you define a custom target.
And finally, custom target definitions can't accommodate the disappearance or reappearance of a target. If the target is reachable when the definition is created, Cryostat accepts the definition but never rechecks the target's reachability and never deletes the definition if the target becomes unreachable.
The discovery plugin customizes application discovery
What is the solution? It is the new Cryostat 2.2 discovery plugin. The old platform-specific mechanisms still exist. Internally, they are wrapped in and registered as a built-in discovery plugin. The discovery plugin API opens up this system for extension by external clients. By registering a discovery plugin and disabling the built-ins (which you can do by setting the environment variable CRYOSTAT_DISABLE_BUILTIN_DISCOVERY), you can fully customize Cryostat's mechanism for discovering target applications and tailor it to suit your exact deployment needs.
Using the Discovery Plugin is fairly simple through Cryostat's API. The API exposes the following new endpoints to control discovery.
Register the plugin with Cryostat
You must register the plugin with Cryostat before publishing information about target applications. Use the following request in your plugin to register it:
POST /api/v2.2/discovery
The request must include a POST body in JSON form. Include an item named realm that identifies the kind of plugin (say, myproject or servicemesh-bridge) and an item named callbackUrl that points back to the plugin instance. Cryostat uses the callbackUrl to verify that the plugin is reachable and that communication is open in both directions.
The plugin must also pass an Authorization header with the request and pass a standard Cryostat platform authz check. You might choose to create a separate OpenShift service account for authorization in production.
Cryostat's response to this request is a JSON object containing an id and a token. The id is a unique identifier specific to this plugin instance and is required for follow-up API requests. The token is a JWT token with a relatively short expiration date, which the plugin must use for follow-up API requests in place of the Authorization header.
Application discovery
Denote the applications you want to be discovered through a request in the following format:
POST /api/v2.2/discovery/:<id>?token=:<token>
The <id> path parameter here is the id that was supplied to the plugin in the previous step. Likewise, the <token> query parameter is the token from the previous step.
The plugin should send a JSON array as the body of this request. The elements within the array are JSON objects representing either target JVM applications directly or some hierarchical node above the target, such as a pod or deployment. For more specifics about the format of these objects, refer to the upstream project documentation. Cryostat inserts this array into the total discovery scenario tree that Cryostat tracks, with a node representing the plugin and its realm as one of the high-level subtrees and the published array as the children of that realm node.
Removing a plugin
To stop monitoring the applications requested by the plugin, remove it from Cryostat as follows:
DELETE /api/v2.2/discovery/:<id>?token=:<token>
Again, the <id> and <token> are the ones supplied from registration. The plugin uses this request to deregister itself from Cryostat and perform a graceful shutdown.
If the plugin shuts down without issuing this request, Cryostat eventually notices the missing plugin when trying to ping the plugin's callbackUrl. When that ping fails, Cryostat internally deregisters the plugin.
Plugins can also time out. Cryostat uses the callbackUrl to inform the plugin that its token is going to expire soon. The plugin can then re-register itself to obtain a refreshed token. For more information on this process, please refer to the upstream project documentation.
A sample Quarkus JVM mode application
I have provided a general description but not much practical implementation or demonstration of this plugin. To partly fill the gap, I have prepared a small sample application built with Quarkus that implements the Cryostat discovery plugin API contract outlined in the previous section. The example is a Quarkus JVM mode application that specifies where it is and how to find it rather than letting Cryostat use the platform (the OpenShift API server) to locate the application. You can find the implementation on my GitHub repository. You might be surprised at how small and simple the implementation is, compared to the specification in the previous section.
My sample application is not ready for widespread use yet and is not part of the downstream Red Hat build of the Cryostat distribution, but work is in progress upstream on a Cryostat agent. This project is a standard JVM tool interface (JVM TI) agent that will be published as a simple JAR to be attached to workload applications.
This agent will require a few configuration options to indicate the location of its Cryostat backend instance, but the intent is for the agent to register itself as a discovery plugin and notify Cryostat of its presence.
By using this discovery plugin agent, developers can enjoy the flexibility of the discovery API within their deployment setup without having to write a custom discovery plugin and encode domain-specific knowledge of that deployment setup. They can simply add the Cryostat discovery agent to your container image builds and redeploy your workloads.
The agent works similarly to the Quarkus sample application, except the Cryostat discovery plugin API requests are extracted into a pluggable JVM TI agent that you can bundle with your existing application without writing a single additional line of code. The next Cryostat 2.3 release will include the Cryostat Agent six months from this publication. So keep an eye out for it.