Manage JFR across instances with Cryostat and GraphQL

Cryostat manages the monitoring of Java applications using Java Flight Recorder (JFR) in the cloud. Cryostat 2.1 includes support for GraphQL to control flight recordings on multiple applications, containers, and Kubernetes pods, with powerful filtering capacities. This article discusses the motivation for adding GraphQL support, shares some examples of queries along with expected results, and takes a look at the underlying web requests on the GraphQL endpoint.

Why use GraphQL with Cryostat?

Previous versions of Cryostat exposed flight recordings with a simple HTTP REST API that limited each request to a single conceptual action—starting one recording on one replica instance, for example. But Cryostat serves developers who create microservices architectures running large numbers of instances in multiple scaled replicas, and each replicated instance can have many flight recordings running for various purposes.

Thus, a developer who wanted to start an identical recording on 12 replicas of a simple container needed to make 12 API requests. Even worse, a developer starting an identical recording on 10 replicas across 10 microservices—a common use case—needed to make 10x10=100 API requests.

Clearly, Cryostat needs a flexible and powerful API for managing all these conditions. GraphQL is a simple but capable language for issuing queries on groups of carefully chosen instances. Reducing multiple actions to a single API request improves overall performance significantly, due to reduced network traffic overhead. As an additional benefit, developers can write much shorter and simpler queries to perform complex actions, rather than writing custom clients that parse API JSON responses and perform actions iteratively on response data.

Queries for target JVMs and the active or archived recordings that belong to them, as well as recordings present in the general Cryostat archives, can combine with mutations to start, stop, archive, and delete active or archived recordings to create powerful queries for building automation around Cryostat and JDK Flight Recorder.

Example queries

Cryostat accepts queries formatted in GraphQL at its /api/beta/graphql endpoint. If you are completely unfamiliar with GraphQL, I suggest you take a brief look at its documentation to gain a better understanding of which parts in the examples in this article are just GraphQL syntax and concepts, versus Cryostat-specific behavior.

Here is our first simple query:

query {
    targetNodes {
            name
            nodeType
            labels
            target {
                alias
                serviceUri
            }
    }
}

This asks Cryostat for the targetNodes result, which is a query that returns all of the JVM targets that Cryostat is aware of. The response is an array of those objects. Since the query specifies various fields like name and nodeType, the objects within the array contain only those fields.

Here is another, more advanced query:

query {
environmentNodes(filter: { name: "my-app-pod-abcd1234" }) {
            descendantTargets {
                doStartRecording(recording: {
                    name: "myrecording",
                template: "Profiling",
                templateType: "TARGET",
                        duration: 30
                   }) {
                      name
                      state
                }
            }
    }
}

This asks Cryostat for the environmentNodes result, which is a query that returns all of the nodes in the deployment graph that Cryostat is aware of but that are not JVM target applications. In a Red Hat OpenShift context, results would include the Deployments or DeploymentConfigs, for example, and the Pods that belong to them.

The previous query used a name-based filter, selecting only the nodes with the name my-app-pod-abcd1234. Let's assume that this string matches one pod within our project namespace. Upon that pod, the query performs the nested query descendantTargets, which yields an array of JVM target objects much like our previous example query.

Upon each of those JVM targets, we perform the doStartRecording mutation. As its name suggests, this mutation causes Cryostat to start a new JDK Flight Recording on each of the JVM targets using the configuration provided. The response will contain simply the name and state of each of these recordings, which we can anticipate to be myrecording and RUNNING.

The previous example shows the power and utility of the GraphQL API, where a single request can make Cryostat perform the complex action of communicating with OpenShift to determine all of the Cryostat-compatible JVMs that belong to the specific pod and start a recording on each JVM.

Here is one final sample query:

query {
    targetNodes(filter: { annotations: "PORT = 9093" }) {
            recordings {
                    active(filter: { labels: "mylabel = redhatdevelopers" }) {
                    doArchive {
                            name
                    }
                }
            }
    }
}

If you are familiar with OpenShift, the labels filter here might look familiar to you. This label selector uses the same syntax as OpenShift's label selectors, and applies to Cryostat's recording labels as well as to the labels and annotations in the deployment graph.

You can use expressions like mylabel = redhatdevelopers, env in (prod, stage), or !mylabel. You can also create a logical AND conjunction of multiple label selectors by passing them as an array in the filter:

active(filter: { labels: ["mylabel = redhatdevelopers", "env in (prod, stage)"] })

To better understand what Cryostat sees in your deployment graph for your particular namespace, check the Discover API endpoint with a command like:

$ curl https://my-cryostat.openshift.example.com/api/v2.1/discovery | jq

There are several different kinds of filters, which can be combined and applied to various nested queries. So a large number of possible semantic requests can be pieced together. Refer to Cryostat's GraphQL queries.graphqls and types.graphqls repositories to see all of the implemented queries and mutations, and learn which ones accept which kinds of filters. The names should be self-explanatory: Types ending in FilterInput are filters, and type fields starting with do are actually nested mutations that perform some action like starting a recording or copying a recording to the archives.

API endpoint details

The examples in this article show typical query payloads of a GraphQL endpoint, the behaviors that you can perform, and what the expected responses should be. In this section, we take a closer look at the specific details of how to make these requests.

The primary GraphQL endpoint for Cryostat 2.1 is POST /api/beta/graphql. The POST requests sent to this path include a body string formatted like { query: "THE_QUERY" }, replacing THE_QUERY with the text of the query as given in the previous section's examples. Here's a concrete example of a request and its metadata:

POST /api/beta/graphql HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 171
Content-Type: application/json
Host: localhost:8181
User-Agent: HTTPie/3.1.0

{
    "query": "query {\n    targetNodes {\n        name\n        nodeType\n        labels\n        target {\n            alias\n            serviceUri\n        }\n    }\n}\n"
}

You can also use a GET /api/beta/graphql request with the same results:

"/api/beta/graphql?query={targetNodes{name nodeType labels target{alias serviceUri}}}"
HTTP/1.1 200 OK
content-encoding: gzip
content-length: 247
content-type: application/json

Taking GraphQL further

In the previous section's examples, we explored some possible query filters. To play with the power of the GraphQL interface in the context of your environment, try the following exercise:

1. Craft a GraphQL query that snapshots all of your JVM applications at once and copies the resulting recordings into the Cryostat archives.
2. Execute your query by sending the request to Cryostat using cURL or your favorite HTTP client.
3. Write a script wrapping around this API request and add it to your crontab to run at 5:00 PM every day.
4. Combine this script with a Cryostat automated rule that automatically starts a new continuous monitoring recording on your JVM applications whenever they appear.

Each of these pieces is simple in isolation, but together they form a powerful automation workflow.

One final note: The standard API in older Cryostat versions still exists in 2.1, and there are even further additions to this API for actions and capabilities that are not suitable for GraphQL—new endpoints for downloading JDK Flight Recorder files using JWT tokens rather than Authorization headers.