Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

ISystemTap: An interactive SystemTap notebook

June 30, 2023
Ryan Goldberg
Related topics:
C, C#, C++Linux
Related products:
Red Hat Enterprise Linux

Share:

SystemTap is a tool that streamlines the gathering of real-time information about a running Linux system. It can probe and even modify running processes in both the user and kernel spaces, so you can easily monitor your programs without recompiling and installing tools on your system. It also provides many tapsets, or prewritten libraries that aid in these tasks.

But to a beginner, SystemTap can be overwhelming, and it can be said that with this great power comes an equally great learning curve. To solve this issue, we've introduced an interactive, beginner-friendly interface called ISystemTap.

About ISystemTap

ISystemTap is a Jupyter kernel that allows for running SystemTap scripts in an easy, incremental way within Jupyterlab, a lightweight web-browser-based client for executing IPython notebook format documents. It has been popularized with the IPython and IRkernel kernels for education, data science, and machine learning.

Getting started

ISystemTap can run right on your machine or from within a container. To get started, install the latest version of SystemTap, currently version 4.9. You will also need SSH to be running on your machine. If it's not running, start it using systemd:

sudo systemctl start sshd

The following setup will be done for the root user, but alternatively any user in the groups stapusr and stapdev can run SystemTap as if with root privileges. In this case, the following would not need to be run with sudo.

If you don't have an SSH key available for the root user, create one with the default name id_rsa:

sudo ssh-keygen -t rsa -b 4096

Then execute sudo stap-jupyter-container --run to start the container as the root user.

This method uses containers to hold Jupyter and related Python packages that are not included in our typical Linux distributions. Feel free to follow along with this article's examples in the RedHatBlog.ipynb or ISystemtap.ipynb notebook files.

Example: Hello ISystemTap

At its core, ISystemTap, like all Jupyter notebooks, is a collection of cells. We further introduce the concept of combining groups of cells under a common namespace, which allows cells to share information amongst themselves.

The best way to show this is using an example. Take the following 3 cells in the helloworld namespace:


%%edit helloworld



global ns



probe begin {



  println("Hello ISystemtap")



  ns = module_name()



}

%%edit helloworld



probe oneshot{



  printf("I am the namespace %s\n", ns)



}

%%run helloworld

Cell Output 1: Running the helloworld namespace:

Hello ISystemTap

I am the namespace helloworld

Executing the first 2 edit cells will add them to the helloworld namespace, with the first cell defining a begin probe-point and a global called ns, and the second defining a oneshot probe-point. Executing the run cell will stitch the edit cells together in the order in which they were executed and run them as a SystemTap script. Notice that ns was defined in the first cell but can be used in the second.

The power of Python

ISystemTap replaces SystemTap's ASCII histograms with rich inline figures, allowing the user to get interactive graphics right alongside the rest of the output. Take the following example, which also introduces the script cell, a cell that combines an edit cell and a run cell:


%%script histogram



global accumulator



global i = 0







probe timer.ms(100){



  printf("ping %d\n", i)



  accumulator <<< randint(i*64+32)



  if (i++ > 10)



    exit()



}







probe end{



  println("Printing Histogram")



  println(@hist_log(accumulator))



  print("All done\n")



}

The cell 2 output is illustrated in Figure 1.

Cell Output 2: Running the histogram namespace
Figure 1: Cell output 2: Running the histogram namespace.

ISystemTap also introduces the python cell, which takes the defined globals from a completed namespace and lets the user access them with Python code. This allows the user to pipeline system information directly into data manipulation tools like numpy and pandas as well as visualization tools like bqplot. The globals are defined as Python global variables and can be accessed in the same way as any other Python data:


%%python histogram



print(f'{accumulator = }')



print(f'{i = }')



print(f'{i == accumulator["@count"] = }')

Cell Output 3: Running a Python cell in the histogram namespace:

accumulator = {'@count': 11, '@min': 14, '@max': 390, '@sum': 2189, '@avg': 199}

i = 11

i == accumulator["@count"] = True

Monitoring the state

ISystemTap doesn't just provide cells for the user; it also provides a new interface for SystemTap in the form of a control widget. This widget gives the user data about the namespace's current state, such as its runtime and how much memory it is using (see Figure 2).

The ISystemTap control widget with memory information.
Figure 2: A control widget of a namespace that completed in 1 second.

Additionally, ISystemTap provides a panel for pausing some or all probes as well as resetting globals to their initial states. The control widget further lets the user see the current values of the various globals as well as some metrics about the probe points. Astute readers might recognize this as the SystemTap monitor mode, which is available without ISystemTap as stap --monitor.

Writing SystemTap scripts

Thus far, we've talked about running existing scripts, but now we'll discuss a new powerful feature for writing scripts: the SystemTap Language Server.

A language server aims to solve the MxN complexity problem of adding useful language development features, such as code completion, to various editors. Instead of each language having a unique editor-specific interface for its particular implementation (e.g., python-vscode, python-vim, python-emacs), there is one language server implementation that can communicate with many language server clients, one for each editor, reducing the complexity to M+N. The SystemTap Language Server supports code completion and has been tested with many popular editors such as VS Code, Vim, Emacs, and Jupyterlab.

SystemTap code completion covers many common actions one might need when writing scripts. It will complete probe points, functions, and globals from the current script and library tapsets. It will also complete macros, strings, and even context variables within relevant probe points, providing documentation as possible. It is worth noting that the language server is built upon a custom C++ SDK which can be used for the creation of other language servers in C++.

Installation is straightforward, with a comprehensive guide with the required manual configuration steps provided in the SystemTap documentation. ISystemTap's install script includes these steps.

Conclusion

Check out our interactive demo to try all of these features and a few others, including a new interface to the SystemTap example scripts and a widget that lists all the probe points that match a user input. Both ISystemTap and the language server are included in SystemTap 4.9.

If you encounter any issues or have any feedback about either of these, please feel free to reach out to us at systemtap@sourceware.org.

Last updated: August 14, 2023

Related Posts

  • Use a SystemTap example script to trace kernel code operation

  • Explore new features in SystemTap 4.5.0

  • Write a SystemTap script to trace code execution on Linux

  • How to trace application errors using SystemTap

  • Reducing the startup overhead of SystemTap monitoring scripts with syscall_any tapset

Recent Posts

  • Create and enrich ServiceNow ITSM tickets with Ansible Automation Platform

  • Expand Model-as-a-Service for secure enterprise AI

  • OpenShift LACP bonding performance expectations

  • Build container images in CI/CD with Tekton and Buildpacks

  • How to deploy OpenShift AI & Service Mesh 3 on one cluster

What’s up next?

systemd Commands cheat sheet card image

Users and administrators query and control systemd behavior through the systemctl command. The systemd Commands Cheat Sheet presents the most common uses of systemctl, along with journalctl for displaying information about systemd activities from its logs.

Get the cheat sheet
Red Hat Developers logo LinkedIn YouTube Twitter Facebook

Products

  • Red Hat Enterprise Linux
  • Red Hat OpenShift
  • Red Hat Ansible Automation Platform

Build

  • Developer Sandbox
  • Developer Tools
  • Interactive Tutorials
  • API Catalog

Quicklinks

  • Learning Resources
  • E-books
  • Cheat Sheets
  • Blog
  • Events
  • Newsletter

Communicate

  • About us
  • Contact sales
  • Find a partner
  • Report a website issue
  • Site Status Dashboard
  • Report a security problem

RED HAT DEVELOPER

Build here. Go anywhere.

We serve the builders. The problem solvers who create careers with code.

Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

Sign me up

Red Hat legal and privacy links

  • About Red Hat
  • Jobs
  • Events
  • Locations
  • Contact Red Hat
  • Red Hat Blog
  • Inclusion at Red Hat
  • Cool Stuff Store
  • Red Hat Summit
© 2025 Red Hat

Red Hat legal and privacy links

  • Privacy statement
  • Terms of use
  • All policies and guidelines
  • Digital accessibility

Report a website issue