Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Explore new features in SystemTap 4.5.0

August 16, 2021
Stan Cox
Related topics:
C, C#, C++Developer ToolsLinux
Related products:
Red Hat Enterprise Linux

Share:

    SystemTap uses a command-line interface (CLI) and a scripting language to write instrumentation for a live, running kernel or a user-space application. A SystemTap script associates handlers with named events. When a specified event occurs, the default SystemTap kernel runtime runs the handler in the kernel like a quick subroutine and then resumes.

    This article lays out the new features in SystemTap 4.5.0. This version will appear in Red Hat Enterprise Linux 9.0. The features fall into three general categories: context variable access, aliasing, and the Berkeley Packet Filter (BPF) back-end.

    Context variable improvements

    These features deal with enumerator access, thread-local storage access, floating-point variable access, and context variable access within functions.

    Enumerator access

    Enumerator values can now be accessed as $context variables, specified through a dollar sign. For example, consider the following C program source code:

    typedef enum { at='@',sharp='#' } symbols;
    symbols symbol = at;

    Suppose you run the following probe handler statement:

    printf("symbol=%c\n",$symbol)

    The statement displays:

    symbol=@

    Only unscoped enumerators are currently supported; scoped enumerators such as the following are not:

    enum class Color { red, green = 20, blue };

    Implicit thread-local storage

    Implicit thread-local storage consists of variables that have a different instance in each thread and persist as long as the thread is alive. The following C declaration defines an implicit thread-local variable named tls:

     __thread unsigned long tls = 99;

    Each thread has its own instance of tls and can change it without affecting the variable of the same name in other threads. On GNU/Linux systems, you can obtain more information about thread-local variables through the command:

    info gcc 'C Extensions' Thread-Local

    If a probe handler is defined for the same module that defines tls, the handler can access the value via @var("tls"). If tls is defined in another module, e.g., the shared object libtls.so, the value of tls can be accessed via @var("tls","libtls.so").

    One common use of this feature is to read the errno value, which is set by many library functions. Take for example the following C code:

    infile = fopen (file_does_not_exist, "r");

    When a statement like this one tries to open a file that doesn't exist, the system returns an errno value of ENOENT. After issuing the fopen call, assume you run the following probe handler statement:

    printf("errno=%d %s\n",@errno,errno_str(@errno))}

    The statement displays:

    errno=2 ENOENT

    Support for tracking virtual memory addresses was added to the stapdyn back-end. That feature enables the SystemTap Dyninst back-end to access global and static variables, including thread-local variables.

    Implicit thread-local variables can be accessed as $context variables on the x86_64, PowerPC, and s390 architectures.

    Floating-point variables

    A SystemTap probe handler can now access floating-point variables as $context variables. Start with the following C declaration:

    float pi = 3.14159

    Run the probe handler statement:

    printf("%s\n",fp_to_string($pi,5))

    The statement displays:

    3.14159

    The %e and %f conversion specifiers of printf are not currently supported by the SystemTap printf statement. The fp_to_string function can be used instead. Other useful floating-point functions include:

    • string_to_fp (string)
    • long_to_fp (long)
    • fp_to_long (float)
    • fp_add (float1, float2)
    • fp_sub (float1, float2)
    • fp_mul (float1, float2)
    • fp_div (float1, float2)
    • fp_sqrt (float1, float2)
    • fp_eq (float1, float2)
    • fp_le (float1, float2)
    • fp_lt (float1, float2)

    32-bit floats are automatically widened to doubles.

    Access context variables inside functions

    Functions may now refer to $context variables and to operators that reference $context variables such as $$vars and $$locals. Previously these references could appear only inside probe handlers.

    Consider the following C code:

    int
    handle_object1 ()
    {
      int attr1;
      int attr2;
      attr1 = 9;
      attr2 = 99;
      types type = thing1;
      return 0;
    }
    
    int
    handle_object2 ()
    {
      int attr1;
      int attr2;
      attr1 = 8;
      attr2 = 88;
      types type = thing2;
      return 0;
    }

    Probe handlers can refer to a common binary_op_vals function that accesses the context variables attr1 and attr2, which are common to both the handle_object1 and handle_object2 functions. The binary_op_vals function can also refer to the $context variable operators $$vars and $$locals.

    function binary_op_vals ()
    {
    printf ("type = %d attribute 1 = %d attribute 2 = %d\n", $type, $attr1, $attr2);
    }
    
    probe process.statement("handle_object1@*:11")
    {
    binary_op_vals ();
    }
    
    probe process.statement("handle_object2@*:22")
    {
    binary_op_vals ();
    }

    The preceding probe handlers display:

    type = 0 attribute 1 = 9 attribute 2 = 99
    type = 1 attribute 1 = 8 attribute 2 = 88

    BPF back-end improvements

    Values in user space can now be accessed with functions such as user_string, user_int, and user_long. Consider the following C source code:

    long ipf = 12;
    long *ipfp = &ipf;

    Then run the following stap command from the SystemTap CLI:

    stap --bpf --disable-cache -e 'probe process.statement("main@*:22")
    {printf("%d\n",user_long($ipfp))}' -c ./tstustr

    The command displays:

    12

    Note the use of the -c option to specify the target program. The systemtap -c <command> option can now be used with the BPF back-end. This option sets the SystemTap target process to the process ID (PID) of the running command.

    Aliasing improvements

    An alias provides a mechanism for augmenting the handlers that are taken by another probe. The alias handlers can be invoked either before the handler, known as a prologue, or after the handler, known as an epilogue.

    It is now possible to define an alias with both a prologue and an epilogue. For example, a probe follows that has a prologue and an epilogue. The probe sets the global variable file_descriptors whenever the open or openat syscalls are invoked:

    private global filenames
    private global file_descriptors
    
    probe file_open = syscall.{open,openat} {
      delete filenames[tid()]
    },{
      if (! (tid() in filenames))
        filenames[tid()] = filename;
    }
    
    probe file_open_return = syscall.{open.return,openat.return} {
      if (tid() in filenames)
        file_descriptors[tid(),retval] = filenames[tid()]
    }
    
    // These alias definitions can be used to only record file descriptors
    // that match a given pathname.
    
    probe file_open
    {
    %($# > 0 %?
      if (strpos(filename,@1) == -1)
        next
    %)
    }
    
    probe file_open_return
    {
    }
    
    probe end
    {
      foreach ([t,f] in file_descriptors)
      {
        printf ("%d %d %s\n", t, f, file_descriptors[t,f])
      }
    }

    If this script is named alias.stp, you can execute it through:

    stap alias.stp /home/user

    The script displays output such as:

    2309651 101 "/home/user/path1"
    2310349 459 "/home/user/path2"

    Additionally, the @probewrite predicate can be used to determine whether a variable has already been written to. For example, the prologue just shown could be defined as follows, to check that filesnames has already been written to:

      if (@probewrite(filenames))
         delete filenames[tid()]

    Summary

    SystemTap in Red Hat Enterprise Linux 9 fills several gaps left in previous versions and provides many conveniences to developers investigating kernel and application behavior. These new features might inspire you to try moving to a lower level of the system through SystemTap.

    Last updated: November 7, 2024

    Related Posts

    • Automating the testing process for SystemTap, Part 1: Test automation with libvirt and Buildbot

    • Using the SystemTap Dyninst runtime environment

    • Making the Operation of Code More Transparent and Obvious with SystemTap

    • Analyzing and reducing SystemTap's startup cost for scripts

    • Speed up SystemTap scripts with statistical aggregates

    Recent Posts

    • Our top 10 articles of 2025 (so far)

    • The benefits of auto-merging GitHub and GitLab repositories

    • Supercharging AI isolation: microVMs with RamaLama & libkrun

    • Simplify multi-VPC connectivity with amazon.aws 9.0.0

    • How HaProxy router settings affect middleware applications

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue