OpenJDK has long been a top pick for real-world applications and workloads, chosen for its blend of performance, compatibility, reliability, and observability. For many years, JDK Flight Recorder (JFR) and JDK Mission Control (JMC) have contributed to OpenJDK's success. Until recently, both were commercial features, however, available only for certain users and workloads.
In 2018, JDK Mission Control and JDK Flight Recorder were open-sourced. JDK Flight Recorder is now built into the Java Virtual Machine (JVM) for later releases of OpenJDK 8 and all versions from OpenJDK 11 onward. Open-sourcing these tools brings their power—always-on, near-zero overhead production profiling and monitoring, application-specific custom events, and unified-core JDK analytical tooling—to all JDK users. On the downside, JDK Mission Control and JDK Flight Recorder have emerged into a world rapidly moving toward containerization, which is not the paradigm that they were designed for.
The desktop-only JDK Mission Control application requires developers and administrators to access flight recordings on the local disk. Otherwise, one resorts to a complex and potentially insecure setup to connect directly to applications over Java Management Extensions (JMX) in the cloud. Similarly, the bare-metal-focused JDK Flight Recorder allows JVMs to dump recordings into the local filesystem, but not when the application runs inside a container. In that case, the filesystem is not easily accessible from the outside world, and it isn't possible to retrieve and analyze recordings.
This article introduces ContainerJFR, a young project on the way to becoming a Red Hat product. ContainerJFR seeks to bridge the gaps between JDK Flight Recorder in the cloud and end-users at their workstations.
Manual ContainerJFR installation and setup
Installing ContainerJFR manually is straightforward using the available images on Quay.io. Run the following for a basic installation and product demonstration:
$ podman run -it --rm -p 8181 -e CONTAINER_JFR_WEB_HOST=0.0.0.0 quay.io/rh-jmc-team/container-jfr:latest
For a more full-fledged demonstration, you can clone the ContainerJFR repository and run its smoketest.sh. The following sets up a few containers in a Podman pod for testing and demonstration:
$ git clone https://github.com/rh-jmc-team/container-jfr $ cd container-jfr $ sh smoketest.sh
ContainerJFR's credentials, in this case, are smoketest:smoketest. The other application’s credentials are admin:adminpass123.
Deploying ContainerJFR on Red Hat OpenShift
If you have access to Red Hat OpenShift or another Kubernetes cluster, you can use the ContainerJFR Operator to deploy ContainerJFR on your cluster:
$ git clone https://github.com/rh-jmc-team/container-jfr-operator $ cd container-jfr-operator $ oc login # ensure you are logged in to your running OpenShift cluster first $ make deploy # to directly deploy the Operator in your cluster along with a ContainerJFR CR, required ServiceAccount and Role/RoleBindings, etc. $ make catalog # alternative to make deploy. This will add a custom CatalogSource to your cluster, allowing you to install the Operator from your Administrator view’s OperatorHub panel
Note: If you don’t already have access to an OpenShift or Kubernetes cluster, try Red Hat CodeReady Containers.
Figure 1 shows ContainerJFR in the OpenShift OperatorHub after we've issued a $ make catalog to add a custom catalog source.
Figure 2 shows a ContainerJFR instance installed in the default namespace.
After you've installed ContainerJFR, create a custom resource for it, as shown in Figure 3. Choose any name you like and leave the minimal setting on false for now.
After a short time, the Operator finishes deploying ContainerJFR and its services. You can use any view that shows the exposed route URLs to see the ContainerJFR web client. Figure 4 shows ContainerJFR in OpenShift's Topology view.
Next, we'll take a look at ContainerJFR's major features, and I'll show you how to configure them for your container-managed OpenJDK applications.
Discovery with ContainerJFR
ContainerJFR is a containerized JVM that runs as a “sidecar” alongside your OpenJDK applications. Depending on the runtime environment, it automatically selects the best strategy for discovering your JMX-enabled applications. For applications running with docker-compose or podman-compose, ContainerJFR would use the Java Discovery Protocol (JDP). For applications running on Kubernetes or OpenShift, it would use endpoints. These are only examples of the platform implementations ContainerJFR currently provides. It is easily extensible if you need customized support for a different container platform.
Java Management Extensions
Ensure that your applications have JMX enabled and that the JMX port is published and reachable by ContainerJFR. In practical terms, this means passing the following JVM flags when you start the application:
-Dcom.sun.management.jmxremote.port=9091 -Dcom.sun.management.jmxremote.rmi.port=9091
Then, expose the port using whatever mechanism your container platform uses. In OpenShift or Kubernetes, you would create a service for your deployment and then add an exposed port to the service. By default, ContainerJFR uses port 9091 for JMX, but you can use any port number given the port is named jfr-jmx.
Figure 5 shows ContainerJFR with two sample applications in the OpenShift Topology view.
Java Discovery Protocol
If you are running with Podman or Docker, or if you are running a local JVM process directly on your host machine, you should also enable Java Discovery Protocol (JDP):
-Dcom.sun.management.jmxremote.autodiscovery=true
Event templates in ContainerJFR
ContainerJFR supports JDK Flight Recorder event templates, which pre-set the event types and configuration properties to be supported. Using event templates simplifies the task of capturing meaningful data for your applications. ContainerJFR also includes a view that displays all of the event types registered with the JDK Flight Recorder framework for a target JVM. This view is useful when creating or modifying your own event templates.
The Event Templates view in Figure 7 displays pre-set event templates that you can download to your local machine to examine or modify. After you've modified a template, you can upload it for reuse. You can also create recordings from templates.
The Event Types view in Figure 8 displays all of the event types registered with the selected target JVM. You can use this view to search for events by category, keyword, or provider.
Editing templates
You can use ContainerJFR to download a template from a target JVM to your local machine, then open and inspect the template XML document with your favorite text editor. You can even import and edit the template using JDK Mission Control. Figure 9 shows the dialog to download an event template.
Custom templates
After you have created a custom template or modified an existing one, you can re-upload it to ContainerJFR, where it will be retained for future use. You will be able to apply the template whenever you create new recordings across your JVM applications.
You can open and edit event templates using any plaintext editor. Another option is to use JDK Mission Control's graphical template editor to import, edit, and export a template. Figure 10 shows an event template in a plaintext editor.
When you upload a new or modified template, ContainerJFR validates it, as shown in Figure 11.
Figure 12 shows all of the available templates for a ContainerJFR instance.
All of these features also work with JDK Flight Recorder’s Custom Events API. You can create application-specific event types while developing your application, then create a custom event template including these events, and tailor your own continuous production recordings.
Recordings in ContainerJFR
ContainerJFR offers several ways to capture and preserve recordings, including custom recordings, snapshots, and archives.
Custom recordings
Figure 13 shows the configuration properties to be defined when you set up a new custom recording.
First is the name of the recording, which ContainerJFR uses to enforce uniqueness per target. You will also configure the duration before the recording is automatically stopped or if it should run continuously until it is manually stopped. You will need to configure the event specifier string or template for the events you want the recording to capture. Advanced properties include “to disk," “max size," and “max age." See the JDK Flight Recorder documentation to learn more about these properties.
Snapshots
Figure 14 shows the dialog to create a new snapshot recording. A snapshot is an overview of all of the information captured by other recordings.
If you have multiple custom recordings running in a target at once, you could use a snapshot to create a new recording file at that point in time. The snapshot contains the merged data from all of your other recordings. Snapshots can also be useful for preserving data from a single, continuous recording at a particular point in time.
Data flow
When you create a recording, you ask ContainerJFR to send instructions to your target JVM to start a flight recording. No data is transferred outside of your application at this point, only the name, state, and start time of your recordings, along with other basic metadata. The recording lives only in the memory of your target application, within its container.
Archives
Archiving streams a snapshot of a recording out of your application and into ContainerJFR. ContainerJFR immediately writes the snapshot to its local disk (or a persistent volume in OpenShift or Kubernetes) and drops it from memory. Even if your application is scaled down or otherwise goes away, you will still be able to access the recording for analysis. If you accidentally delete a .jfr file, you can re-upload it from your workstation’s local disk into the archives. This also works if you remove ContainerJFR from the cluster and re-install it later.
The archived recording list in Figure 16 displays all of the recordings saved to ContainerJFR's persistent storage, which is common across all target JVMs.
Automated analysis with ContainerJFR
ContainerJFR lets you run an automated analysis on your flight recordings within your cloud deployment without ever needing to transfer data to your local machine or outside of the cluster. You can use this feature to check your applications' health from a hotel with a slow connection or an airport using only your phone or tablet.
Expanding the list of active and archived recordings in Figure 17 reveals an automated analysis generated within the cluster.
When you expand a recording, ContainerJFR uses its JDK Mission Control back end to generate an automated analysis report, alerting you to any apparent or probable issues with your application. You can also save reports in HTML format for future reference.
Using Grafana for analysis
If the automated analysis report doesn’t contain enough information for you, or if it points out a problem that you want to inspect more closely, you can send the recording to the jfr-datasource exporter within the ContainerJFR pod. From there, you can view the data using Grafana. Figure 18 shows the recording list item action menu, which you can use to send a recording to the Grafana dashboard.
ContainerJFR provides a pre-configured dashboard with time-series data, but you are encouraged to create your own dashboards with the metrics that matter to you. Note, again, that none of your data leaves the cluster. The jfr-datasource that provides the translation from a .jfr file to Grafana metrics is hidden within the ContainerJFR pod, and the Grafana dashboard instance is secured with generated credentials (stored in an OpenShift or Kubernetes secret) before being exposed to the outside world. It is easy to retrieve those generated credentials using the following commands:
$ oc get secret containerjfr-grafana-basic -o json | jq -crM .data.GF_SECURITY_ADMIN_USER | base64 -d $ oc get secret containerjfr-grafana-basic -o json | jq -crM .data.GF_SECURITY_ADMIN_PASSWORD | base64 -d
Once you have the credentials, you can plug them into the Grafana dashboard's login page and start viewing your metrics, as shown in Figure 19.
Using JDK Mission Control for analysis
Finally, if you need even more detail, you can download a recording file from ContainerJFR to your local disk and open it with the full-featured offline JDK Mission Control desktop application. This is the only scenario where your recording actually leaves the cluster.
The JDK Mission Control desktop application offers a wealth of features and capabilities, but I will leave that discussion for another article.
Secure authentication with ContainerJFR
JDK Flight Recorder captures a tremendous amount of data with no significant runtime overhead. Keeping the data secure and ensuring its integrity is vital. As shown in Figure 21, ContainerJFR does not require your application to open its JMX connections to the world—only to connections from inside your OpenShift namespace or your Docker or Podman network.
Once ContainerJFR has received a copy of your Java Flight Recorder data—which it does only upon your instruction—that data is accessible only through secured API requests. The secured API requests support JMX authentication to connect to your application and another authentication layer to connect to ContainerJFR.
When running in OpenShift, all sensitive API requests require a user account token for authentication, as shown in Figure 22. Note that requests from the client to ContainerJFR over HTTP or WebSocket and requests from ContainerJFR to targets over JMX all support and enable the Secure Socket Layer or Transport Layer Security (SSL/TLS) protocol by default.
Future plans for ContainerJFR
ContainerJFR is still a young project, and the worlds of containers and monitoring are always expanding, so there is a lot on our horizon. In the future, we hope to make the following changes and improvements to ContainerJFR:
- Implement better and more flexible ways for ContainerJFR to identify, group, and label target applications. One example is the ability to examine OpenShift or Kubernetes labels and annotations.
- Add support for batched operations, used to manage recordings across a group of targets with a single request.
- Add a Trigger feature to allow recordings to be automatically started and stopped on a target or group of targets when a predefined event occurs. For example, when a new target appears, automatically start a recording with a predefined template.
- Embed Grafana views and other visualizations directly into the ContainerJFR web client.
- Provide integration or deep linking to the desktop JDK Mission Control application.
Visit the ContainerJFR repository for future updates.
Last updated: January 21, 2021