SystemTap is a tool that streamlines the gathering of real-time information about a running Linux system. It can probe and even modify running processes in both the user and kernel spaces, so you can easily monitor your programs without recompiling and installing tools on your system. It also provides many tapsets, or prewritten libraries that aid in these tasks.
But to a beginner, SystemTap can be overwhelming, and it can be said that with this great power comes an equally great learning curve. To solve this issue, we've introduced an interactive, beginner-friendly interface called ISystemTap.
About ISystemTap
ISystemTap is a Jupyter kernel that allows for running SystemTap scripts in an easy, incremental way within Jupyterlab, a lightweight web-browser-based client for executing IPython notebook format documents. It has been popularized with the IPython and IRkernel kernels for education, data science, and machine learning.
Getting started
ISystemTap can run right on your machine or from within a container. To get started, install the latest version of SystemTap, currently version 4.9. You will also need SSH to be running on your machine. If it's not running, start it using systemd:
sudo systemctl start sshd
The following setup will be done for the root user, but alternatively any user in the groups stapusr and stapdev can run SystemTap as if with root privileges. In this case, the following would not need to be run with sudo.
If you don't have an SSH key available for the root user, create one with the default name id_rsa:
sudo ssh-keygen -t rsa -b 4096
Then execute sudo stap-jupyter-container --run to start the container as the root user.
This method uses containers to hold Jupyter and related Python packages that are not included in our typical Linux distributions. Feel free to follow along with this article's examples in the RedHatBlog.ipynb or ISystemtap.ipynb notebook files.
Example: Hello ISystemTap
At its core, ISystemTap, like all Jupyter notebooks, is a collection of cells. We further introduce the concept of combining groups of cells under a common namespace, which allows cells to share information amongst themselves.
The best way to show this is using an example. Take the following 3 cells in the helloworld namespace:
%%edit helloworld
global ns
probe begin {
println("Hello ISystemtap")
ns = module_name()
}
%%edit helloworld
probe oneshot{
printf("I am the namespace %s\n", ns)
}
%%run helloworld
Cell Output 1: Running the helloworld namespace:
Hello ISystemTap
I am the namespace helloworld
Executing the first 2 edit cells will add them to the helloworld namespace, with the first cell defining a begin probe-point and a global called ns, and the second defining a oneshot probe-point. Executing the run cell will stitch the edit cells together in the order in which they were executed and run them as a SystemTap script. Notice that ns was defined in the first cell but can be used in the second.
The power of Python
ISystemTap replaces SystemTap's ASCII histograms with rich inline figures, allowing the user to get interactive graphics right alongside the rest of the output. Take the following example, which also introduces the script cell, a cell that combines an edit cell and a run cell:
%%script histogram
global accumulator
global i = 0
probe timer.ms(100){
printf("ping %d\n", i)
accumulator <<< randint(i*64+32)
if (i++ > 10)
exit()
}
probe end{
println("Printing Histogram")
println(@hist_log(accumulator))
print("All done\n")
}
The cell 2 output is illustrated in Figure 1.
ISystemTap also introduces the python cell, which takes the defined globals from a completed namespace and lets the user access them with Python code. This allows the user to pipeline system information directly into data manipulation tools like numpy and pandas as well as visualization tools like bqplot. The globals are defined as Python global variables and can be accessed in the same way as any other Python data:
%%python histogram
print(f'{accumulator = }')
print(f'{i = }')
print(f'{i == accumulator["@count"] = }')
Cell Output 3: Running a Python cell in the histogram namespace:
accumulator = {'@count': 11, '@min': 14, '@max': 390, '@sum': 2189, '@avg': 199}
i = 11
i == accumulator["@count"] = True
Monitoring the state
ISystemTap doesn't just provide cells for the user; it also provides a new interface for SystemTap in the form of a control widget. This widget gives the user data about the namespace's current state, such as its runtime and how much memory it is using (see Figure 2).
Additionally, ISystemTap provides a panel for pausing some or all probes as well as resetting globals to their initial states. The control widget further lets the user see the current values of the various globals as well as some metrics about the probe points. Astute readers might recognize this as the SystemTap monitor mode, which is available without ISystemTap as stap --monitor.
Writing SystemTap scripts
Thus far, we've talked about running existing scripts, but now we'll discuss a new powerful feature for writing scripts: the SystemTap Language Server.
A language server aims to solve the MxN complexity problem of adding useful language development features, such as code completion, to various editors. Instead of each language having a unique editor-specific interface for its particular implementation (e.g., python-vscode, python-vim, python-emacs), there is one language server implementation that can communicate with many language server clients, one for each editor, reducing the complexity to M+N. The SystemTap Language Server supports code completion and has been tested with many popular editors such as VS Code, Vim, Emacs, and Jupyterlab.
SystemTap code completion covers many common actions one might need when writing scripts. It will complete probe points, functions, and globals from the current script and library tapsets. It will also complete macros, strings, and even context variables within relevant probe points, providing documentation as possible. It is worth noting that the language server is built upon a custom C++ SDK which can be used for the creation of other language servers in C++.
Installation is straightforward, with a comprehensive guide with the required manual configuration steps provided in the SystemTap documentation. ISystemTap's install script includes these steps.
Conclusion
Check out our interactive demo to try all of these features and a few others, including a new interface to the SystemTap example scripts and a widget that lists all the probe points that match a user input. Both ISystemTap and the language server are included in SystemTap 4.9.
If you encounter any issues or have any feedback about either of these, please feel free to reach out to us at systemtap@sourceware.org.
Last updated: August 14, 2023