Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Introduction to Cryostat: JDK Flight Recorder for containers

January 25, 2021
Andrew Azores
Related topics:
ContainersJavaKubernetes
Related products:
Red Hat OpenShift

Share:

    OpenJDK has long been a top pick for real-world applications and workloads, chosen for its blend of performance, compatibility, reliability, and observability. For many years, JDK Flight Recorder (JFR) and JDK Mission Control (JMC) have contributed to OpenJDK's success. Until recently, both were commercial features, however, available only for certain users and workloads.

    Note: This article has been updated to reflect the Cryostat project's name change from ContainerJFR to Cryostat.

    In 2018, JDK Mission Control and JDK Flight Recorder were open-sourced. JDK Flight Recorder is now built into the Java Virtual Machine (JVM) for later releases of OpenJDK 8 and all versions from OpenJDK 11 onward. Open-sourcing these tools brings their power—always-on, near-zero overhead production profiling and monitoring, application-specific custom events, and unified-core JDK analytical tooling—to all JDK users. On the downside, JDK Mission Control and JDK Flight Recorder have emerged into a world rapidly moving toward containerization, which is not the paradigm that they were designed for.

    The desktop-only JDK Mission Control application requires developers and administrators to access flight recordings on the local disk. Otherwise, one resorts to a complex and potentially insecure setup to connect directly to applications over Java Management Extensions (JMX) in the cloud. Similarly, the bare-metal-focused JDK Flight Recorder allows JVMs to dump recordings into the local filesystem, but not when the application runs inside a container. In that case, the filesystem is not easily accessible from the outside world, and it isn't possible to retrieve and analyze recordings.

    This article introduces Cryostat, a young project on the way to becoming a Red Hat product. Cryostat seeks to bridge the gaps between JDK Flight Recorder in the cloud and end-users at their workstations.

    About Cryostat: Formerly named ContainerJFR, Cryostat is a container-native JVM application that acts as a JMX bridge to other containerized JVMs. Cryostat exposes a secure API for producing, analyzing, and retrieving JDK Flight Recorder data from cloud workloads.

    Manual Cryostat installation and setup

    Installing Cryostat manually is straightforward using the available images on Quay.io. Run the following for a basic installation and product demonstration:

    $ podman run -it --rm -p 8181 -e CRYOSTAT_WEB_HOST=0.0.0.0 quay.io/cryostat/cryostat:latest

    For a more full-fledged demonstration, you can clone the Cryostat repository and run its smoketest.sh. The following sets up a few containers in a Podman pod for testing and demonstration:

    $ git clone https://github.com/cryostatio/cryostat
    
    $ cd cryostat
    
    $ sh smoketest.sh

    Cryostat's credentials, in this case, are smoketest:smoketest. The other application’s credentials are admin:adminpass123.

    Deploying Cryostat on Red Hat OpenShift

    If you have access to Red Hat OpenShift or another Kubernetes cluster, you can use the Cryostat Operator to deploy Cryostat on your cluster:

    
    $ git clone https://github.com/cryostatio/cryostat-operator
    
    $ cd cryostat-operator
    
    $ git checkout v1.0.0
    
    $ oc login # ensure you are logged in to your running OpenShift cluster first
    
    $ make cert_manager # cert-manager is used for SSL/TLS certificates to secure in-cluster traffic
    
    $ make deploy # to directly deploy the Operator in your cluster along with a Cryostat CR, required ServiceAccount and Role/RoleBindings, etc.
    
    $ make deploy_bundle # alternative to make deploy. This will deploy the Cryostat Operator Bundle to your cluster, allowing you to install the Operator from your Administrator view’s OperatorHub panel
    
    

    Note: If you don’t already have access to an OpenShift or Kubernetes cluster, try Red Hat CodeReady Containers.

    Figure 1 shows Cryostat in the OpenShift OperatorHub after we've issued a $ make deploy_bundle.

    Cryostat Operator in the OpenShift OperatorHub
    Figure 1: Cryostat in the OpenShift OperatorHub.

    Figure 2 shows a Cryostat instance installed in the default namespace.

    Cryostat installed in a project namespace
    Figure 2: Cryostat installed in a project namespace.

    After you've installed Cryostat, create a custom resource for it, as shown in Figure 3. Choose any name you like and leave the minimal setting on false for now.

    Create and configure the ContainerJFR custom resource
    Figure 3: Create and configure the Cryostat custom resource.

    After a short time, the Cryostat Operator finishes deploying Cryostat and its services. You can use any view that shows the exposed route URLs to see the Cryostat web client. Figure 4 shows Cryostat in OpenShift's Topology view.

    Cryostat in the OpenShift Topology View
    Figure 4: Cryostat in the OpenShift Topology view.

    Next, we'll take a look at Cryostat's major features, and I'll show you how to configure them for your container-managed OpenJDK applications.

    Discovery with Cryostat

    Cryostat is a containerized JVM that runs as a “sidecar” alongside your OpenJDK applications. Depending on the runtime environment, it automatically selects the best strategy for discovering your JMX-enabled applications. For applications running with docker-compose or podman-compose, Cryostat would use the Java Discovery Protocol. For applications running on Kubernetes or OpenShift, it would use endpoints. These are only examples of the platform implementations Cryostat currently provides. It is easily extensible if you need customized support for a different container platform.

    Java Management Extensions

    Ensure that your applications have JMX enabled and that the JMX port is published and reachable by Cryostat. In practical terms, this means passing the following JVM flags when you start the application:

    
    -Dcom.sun.management.jmxremote.port=9091 -Dcom.sun.management.jmxremote.rmi.port=9091
    

    Then, expose the port using whatever mechanism your container platform uses. In OpenShift or Kubernetes, you would create a service for your deployment and then add an exposed port to the service. By default, Cryostat uses port 9091 for JMX, but you can use any port number given the port is named jfr-jmx.

    Figure 5 shows Cryostat with two sample applications in the OpenShift Topology view.

    Topology view with Cryostat and two sample applications
    Cryostat with two sample applications in the topology view
    Figure 5: Cryostat with two sample applications.

    Java Discovery Protocol

    If you are running with Podman or Docker, or if you are running a local JVM process directly on your host machine, you should also enable Java Discovery Protocol:

    
    -Dcom.sun.management.jmxremote.autodiscovery=true
    

    Event templates in Cryostat

    Cryostat supports JDK Flight Recorder event templates, which pre-set the event types and configuration properties to be supported. Using event templates simplifies the task of capturing meaningful data for your applications. Cryostat also includes a view that displays all of the event types registered with the JDK Flight Recorder framework for a target JVM. This view is useful when creating or modifying your own event templates.

    The Event Templates view in Figure 6 displays pre-set event templates that you can download to your local machine to examine or modify. After you've modified a template, you can upload it for reuse. You can also create recordings from templates.

    Cryostat Event Templates
    The Templates view lists preset event templates that can be downloaded to your local machine and examined or modified. Modified templates can be uploaded for re-use. Recordings can be created from templates.
    Figure 6: Event templates in Cryostat.

    The Event Types view in Figure 7 displays all of the event types registered with the selected target JVM. You can use this view to search for events by category, keyword, or provider.

    Cryostat Event Types
    The Event Types view displays all of the event types registered with the selected target JVM and can be used to search for events by category, keyword, and provider.
    Figure 7: Event types in Cryostat.

    Editing templates

    You can use Cryostat to download a template from a target JVM to your local machine, then open and inspect the template XML document with your favorite text editor. You can even import and edit the template using JDK Mission Control. Figure 8 shows the dialog to download an event template.

    Event template download page.
    Figure 8: Download an event template to your local disk.

    Custom templates

    After you have created a custom template or modified an existing one, you can re-upload it to Cryostat, where it will be retained for future use. You will be able to apply the template whenever you create new recordings across your JVM applications.

    You can open and edit event templates using any plain-text editor. Another option is to use JDK Mission Control's graphical template editor to import, edit, and export a template. Figure 9 shows an event template in a plain-text editor.

    An event template in a plaintext editor.
    Templates can be opened and edited with any plaintext editor, or imported/edited/exported using JDK Mission Control's graphical template editor
    Figure 9: Use any editor to modify an event template in XML format.

    When you upload a new or modified template, Cryostat validates it, as shown in Figure 10.

    Event Template Ready for Upload
    Templates are validated when received by the server
    Figure 10: Templates are validated when the server receives them.

    Figure 11 shows all of the available templates for a Cryostat instance.

    Template Available for Use
    If the template is valid, it will be added to the list of available templates and available for later use
    Figure 11: A list of templates available for later use.

    All of these features also work with JDK Flight Recorder’s Custom Events API. You can create application-specific event types while developing your application, then create a custom event template including these events, and tailor your own continuous production recordings.

    Recordings in Cryostat

    Cryostat offers several ways to capture and preserve recordings, including custom recordings, snapshots, and archives.

    Custom recordings

    Figure 12 shows the configuration properties to be defined when you set up a new custom recording.

    Customized Recording Creation Form
    Creation of a customized recording
    Figure 12: Creating a custom recording.

    First is the name of the recording, which Cryostat uses to enforce uniqueness per target. You will also configure the duration before the recording is automatically stopped or if it should run continuously until it is manually stopped. You will need to configure the event specifier string or template for the events you want the recording to capture. Advanced properties include “to disk," “max size," and “max age." See the JDK Flight Recorder documentation to learn more about these properties.

    Snapshots

    Figure 13 shows the dialog to create a new snapshot recording. A snapshot is an overview of all of the information captured by other recordings.

    Snapshot Recording Creation View
    Creating a snapshot recording is a one-click operation
    Figure 13: Create a snapshot recording with one click.

    If you have multiple custom recordings running in a target at once, you could use a snapshot to create a new recording file at that point in time. The snapshot contains the merged data from all of your other recordings. Snapshots can also be useful for preserving data from a single, continuous recording at a particular point in time.

    Data flow

    When you create a recording, you ask Cryostat to send instructions to your target JVM to start a flight recording. No data is transferred outside of your application at this point, only the name, state, and start time of your recordings, along with other basic metadata. The recording lives only in the memory of your target application, within its container. Figure 14 shows the recordings present in a target JVM.

    Recording List View
    The Active list displays the recordings present in the selected target JVM
    Figure 14: Cryostat displays all of the recordings present in a selected target JVM.

    Archives

    Archiving streams a snapshot of a recording out of your application and into Cryostat. Cryostat immediately writes the snapshot to its local disk (or a persistent volume in OpenShift or Kubernetes) and drops it from memory. Even if your application is scaled down or otherwise goes away, you will still be able to access the recording for analysis. If you accidentally delete a .jfr file, you can re-upload it from your workstation’s local disk into the archives. This also works if you remove Cryostat from the cluster and re-install it later.

    The archived recording list in Figure 15 displays all of the recordings saved to Cryostat's persistent storage, which is common across all target JVMs.

    Archived Recording List View
    The Archived recording list displays the recordings which are saved to ContainerJFR's persistent storage, which is common across all target JVMs
    Figure 15: Recordings saved to Cryostat's persistent storage.

    Automated analysis with Cryostat

    Cryostat lets you run an automated analysis on your flight recordings within your cloud deployment without ever needing to transfer data to your local machine or outside of the cluster. You can use this feature to check your applications' health from a hotel with a slow connection or an airport using only your phone or tablet.

    Expanding the list of active and archived recordings in Figure 16 reveals an automated analysis generated within the cluster.

    Automated Analysis Report
    Active and Archived recording list items can be expanded to reveal an automated analysis generated within the cluster
    Figure 16: An automated analysis report for active and archived recordings.

    When you expand a recording, Cryostat uses its JDK Mission Control back end to generate an automated analysis report, alerting you to any apparent or probable issues with your application. You can also save reports in HTML format for future reference.

    Using Grafana for analysis

    If the automated analysis report doesn’t contain enough information for you, or if it points out a problem that you want to inspect more closely, you can send the recording to the jfr-datasource exporter within the Cryostat pod. From there, you can view the data using Grafana. Figure 17 shows the recording list item action menu, which you can use to send a recording to the Grafana dashboard.

    View in Grafana Action
    The recording list item action menu can be used to send a recording to the Grafana dashboard
    Figure 17: Send a recording to the Grafana dashboard for further analysis.

    Cryostat provides a pre-configured dashboard with time-series data, but you are encouraged to create your own dashboards with the metrics that matter to you. Note, again, that none of your data leaves the cluster. The jfr-datasource that provides the translation from a .jfr file to Grafana metrics is hidden within the Cryostat pod, and the Grafana dashboard instance is secured with generated credentials (stored in an OpenShift or Kubernetes secret) before being exposed to the outside world. It is easy to retrieve those generated credentials using the following commands:

    
    $ oc get secret cryostat-grafana-basic -o json | jq -crM .data.GF_SECURITY_ADMIN_USER | base64 -d
    $ oc get secret cryostat-grafana-basic -o json | jq -crM .data.GF_SECURITY_ADMIN_PASSWORD | base64 -d
    

    Once you have the credentials, you can plug them into the Grafana dashboard's login page and start viewing your metrics, as shown in Figure 18.

    Grafana Dashboard
    The preconfigured Grafana dashboard gives more detailed insights into your application's performance, batteries included and installed
    Figure 18: The preconfigured Grafana dashboard gives more detailed insights into your application's performance (batteries included and installed).

    Using JDK Mission Control for analysis

    Finally, if you need even more detail, you can download a recording file from Cryostat to your local disk and open it with the full-featured offline JDK Mission Control desktop application. This is the only scenario where your recording actually leaves the cluster. Figure 19 shows a recorded file open in the JDK Mission Control desktop application.

    JDK Mission Control
    The JDK Mission Control desktop application can be used for a deep-dive into all of the data collected from your cloud applications
    Figure 19: Use the JDK Mission Control desktop application for a deep-dive into the data collected from your cloud applications.

    The JDK Mission Control desktop application offers a wealth of features and capabilities, but I will leave that discussion for another article.

    Secure authentication with Cryostat

    JDK Flight Recorder captures a tremendous amount of data with no significant runtime overhead. Keeping the data secure and ensuring its integrity is vital. As shown in Figure 20, Cryostat does not require your application to open its JMX connections to the world—only to connections from inside your OpenShift namespace or your Docker or Podman network.

    Graph illustrating Cryostat deployment and relations between components
    Figure 20: Overview of a secure Cryostat deployment.

    Once Cryostat has received a copy of your JDK Flight Recorder data—which it does only upon your instruction—that data is accessible only through secured API requests. The secured API requests support JMX authentication to connect to your application and another authentication layer to connect to Cryostat.

    When running in OpenShift, all sensitive API requests require a user account token for authentication, as shown in Figure 21. Note that requests from the client to Cryostat over HTTP or WebSocket and requests from Cryostat to targets over JMX all support and enable the Secure Socket Layer or Transport Layer Security (SSL/TLS) protocol by default.

    Cryostat Login Page
    Cryostat detects that it is running in OpenShift and uses the OpenShift cluster's authentication server to authenticate users for all sensitive API requests.
    Figure 21: Cryostat uses the OpenShift cluster's authentication server for user authentication.

    Future plans for Cryostat

    Cryostat is still a young project, and the worlds of containers and monitoring are always expanding, so there is a lot on our horizon. In the future, we hope to make the following changes and improvements to Cryostat:

    • Implement better and more flexible ways for Cryostat to identify, group, and label target applications. One example is the ability to examine OpenShift or Kubernetes labels and annotations.
    • Add support for batched operations, used to manage recordings across a group of targets with a single request.
    • Add a Trigger feature to allow recordings to be automatically started and stopped on a target or group of targets when a predefined event occurs. For example, when a new target appears, automatically start a recording with a predefined template.
    • Embed Grafana views and other visualizations directly into the Cryostat web client.
    • Provide integration or deep linking to the desktop JDK Mission Control application.

    Visit the Cryostat repository for future updates.

    Last updated: January 12, 2024

    Related Posts

    • Get started with JDK Flight Recorder in OpenJDK 8u

    • Collect JDK Flight Recorder events at runtime with JMC Agent

    • JDK Flight Recorder support for GraalVM Native Image: The journey so far

    • Set up JDK Mission Control with Red Hat Build of OpenJDK

    Recent Posts

    • Expand Model-as-a-Service for secure enterprise AI

    • OpenShift LACP bonding performance expectations

    • Build container images in CI/CD with Tekton and Buildpacks

    • How to deploy OpenShift AI & Service Mesh 3 on one cluster

    • JVM tuning for Red Hat Data Grid on Red Hat OpenShift 4

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue