Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

A faster way to access JDK Flight Recorder data

 

November 23, 2021
Andrew Azores
Related topics:
ContainersJavaKubernetesMicroservices
Related products:
Red Hat build of OpenJDK

Share:

    This article introduces a special rule definition in Cryostat 2.0 that lets you access JDK Flight Recorder (JFR) data on the fly, without waiting for your application's normally scheduled archive process. We'll introduce Cryostat's new POST rule definition and show you how to use it to quickly diagnose performance problems in containerized applications running on Kubernetes and Red Hat OpenShift.

    Read the series:

    We're publishing a series of hands-on guides to using Cryostat 2.0, which is JDK Flight Recorder (JFR) for Kubernetes. Read all of the articles in this series:

    • Part 1: Get started with Cryostat 2.0
    • Part 2: Configuring Java applications to use Cryostat
    • Part 3: Java monitoring for custom targets with Cryostat
    • Part 4: Automating JDK Flight Recorder in containers
    • Part 5: Creating Custom JFR event templates with Cryostat 2.0

    Note: The Red Hat build of Cryostat 2.0 is now widely available in technology preview. The Red Hat build includes the Cryostat Operator to simplify and automate Cryostat deployment on OpenShift.

    Continuous monitoring for service disruptions

    Consider a scenario where a cluster administrator has deployed Cryostat in a project namespace and enabled continuous application monitoring using Cryostat's automated rules. Such a rule definition might look like this:

    
    {
      "name": "continuousMonitoring",
      "description": "Enable the Continuous template on all parts of our service",
      "matchExpression": "target.labels.platform[‘app’]==’myService’",
      "eventSpecifier": "template=Continuous,type=TARGET",
      "archivalPeriodSeconds": 3600,
      "preservedArchives": 4
    }
    
    

    This rule, called continuousMonitoring, creates a recording using the Continuous event template on all targets with the app=myService label. It then copies the recording data to archives once every hour and maintains four of these archived copies.

    In this scenario, the project contains many instances of microservices that together provide a larger external service to customers. The microservices make internal requests to each other while servicing customer traffic. Occasionally, for reasons the cluster administrator doesn't yet understand, a hiccup occurs, and the time to service a customer’s request suddenly spikes.

    Using the POST rule for OpenJDK flight data

    Let say an admin receives notification that a spike has just occurred, but the continuousMonitoring rule is not set to copy the JDK Flight Recorder data into archives for another 40 minutes. Obviously, the admin doesn't want to wait that long: They need to discover what caused the service latency spike and fix it right away. For that, they need an immediate way to capture the JDK Flight Recorder data.

    Cryostat 2.0 lets cluster admins capture OpenJDK flight data immediately by POSTing a rule definition with the following form:

    
    {
      "matchExpression": "target.annotations.platform[‘app’]==’myService’",
      "eventSpecifier": "archive"
    }
    

    Notice the eventSpecifier: archive property. This is a special case that instructs Cryostat that, rather than creating a new recording in all of the matching targets, it should immediately copy recording data from all of the matching targets to the Cryostat archives. Using the same matchExpression as the original rule definition ensures that the archival action applies to the same targets.

    As an example, if the admin suspected that the problem lived in a particular subset of the microservice targets, they could fine-tune the matchExpression to suit. For example, they could focus their search on the login service:

    
    {
        "matchExpression": "target.annotations.platform[‘app’]==’myService’ && /^my-app-login-service-/.test(target.alias)",
        "eventSpecifier": "archive"
    }
    
    

    After POSTing the new rule definition to Cryostat, the admin will immediately have access to a copy of each microservice replica’s recording data in the Cryostat archives. Let’s take a look:

    
    $ CRYOSTAT=https://cryostat.example.com
    $ curl -F name="continuousMonitoring" -F description=”Enable the Continuous template on all parts of our service” -F matchExpression=”target.annotations.platform[‘app’]==’myService’” -F eventSpecifier=”template=Continuous,type=TARGET” -F archivalPeriodSeconds=3600 -F preservedArchives=4 $CRYOSTAT/api/v2/rules
    $ # some time passes
    $ curl -F matchExpression=”target.annotations.platform[‘app’]==’myService’ && /^my-app-login-service-/.test(target.alias)” -F eventSpecifier=archive $CRYOSTAT/api/v2/rules
    $ curl $CRYOSTAT/api/v1/recordings
    
    

    From here, they can process the JSON response and take various actions. The response is an array of objects containing information about each recording in the archives. The recording names can be filtered to isolate the ones produced by our archive rule. The admin can then use the JSON objects’ reportUrl or downloadUrl properties to get an HTML document containing either an automated analysis or the full JDK Flight Recorder data dump.

    Conclusion

    Regularly archived JDK Flight Recorder data recordings are adequate for diagnostics purposes most of the time. But sometimes you need more immediate access to data. In this article, you've learned how to use Cryostat's new POST rule format to access JDK Flight Recorder data on the fly. As a cluster admin, you can respond to system disruptions by POSTing a specially formulated rule, then batch retrieve the data to diagnose disruptions in your containerized application's performance. Visit Cryostat.io for more about this and other new features in Cryostat 2.0.

    Last updated: September 20, 2023

    Related Posts

    • Announcing Cryostat 2.0: JDK Flight Recorder for containers

    • Automating JDK Flight Recorder in containers

    • Configuring Java applications to use Cryostat

    • Java monitoring for custom targets with Cryostat

    • Custom JFR event templates with Cryostat 2.0

    Recent Posts

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue