SystemTap uses a command-line interface (CLI) and a scripting language to write instrumentation for a live, running kernel or a user-space application. A SystemTap script associates handlers with named events. When a specified event occurs, the default SystemTap kernel runtime runs the handler in the kernel like a quick subroutine and then resumes.
This article lays out the new features in SystemTap 4.5.0. This version will appear in Red Hat Enterprise Linux 9.0. The features fall into three general categories: context variable access, aliasing, and the Berkeley Packet Filter (BPF) back-end.
Context variable improvements
These features deal with enumerator access, thread-local storage access, floating-point variable access, and context variable access within functions.
Enumerator access
Enumerator values can now be accessed as $context variables, specified through a dollar sign. For example, consider the following C program source code:
typedef enum { at='@',sharp='#' } symbols;
symbols symbol = at;
Suppose you run the following probe handler statement:
printf("symbol=%c\n",$symbol)
The statement displays:
symbol=@
Only unscoped enumerators are currently supported; scoped enumerators such as the following are not:
enum class Color { red, green = 20, blue };
Implicit thread-local storage
Implicit thread-local storage consists of variables that have a different instance in each thread and persist as long as the thread is alive. The following C declaration defines an implicit thread-local variable named tls:
__thread unsigned long tls = 99;
Each thread has its own instance of tls and can change it without affecting the variable of the same name in other threads. On GNU/Linux systems, you can obtain more information about thread-local variables through the command:
info gcc 'C Extensions' Thread-Local
If a probe handler is defined for the same module that defines tls, the handler can access the value via @var("tls"). If tls is defined in another module, e.g., the shared object libtls.so, the value of tls can be accessed via @var("tls","libtls.so").
One common use of this feature is to read the errno value, which is set by many library functions. Take for example the following C code:
infile = fopen (file_does_not_exist, "r");
When a statement like this one tries to open a file that doesn't exist, the system returns an errno value of ENOENT. After issuing the fopen call, assume you run the following probe handler statement:
printf("errno=%d %s\n",@errno,errno_str(@errno))}
The statement displays:
errno=2 ENOENT
Support for tracking virtual memory addresses was added to the stapdyn back-end. That feature enables the SystemTap Dyninst back-end to access global and static variables, including thread-local variables.
Implicit thread-local variables can be accessed as $context variables on the x86_64, PowerPC, and s390 architectures.
Floating-point variables
A SystemTap probe handler can now access floating-point variables as $context variables. Start with the following C declaration:
float pi = 3.14159
Run the probe handler statement:
printf("%s\n",fp_to_string($pi,5))
The statement displays:
3.14159
The %e and %f conversion specifiers of printf are not currently supported by the SystemTap printf statement. The fp_to_string function can be used instead. Other useful floating-point functions include:
string_to_fp (string)long_to_fp (long)fp_to_long (float)fp_add (float1, float2)fp_sub (float1, float2)fp_mul (float1, float2)fp_div (float1, float2)fp_sqrt (float1, float2)fp_eq (float1, float2)fp_le (float1, float2)fp_lt (float1, float2)
32-bit floats are automatically widened to doubles.
Access context variables inside functions
Functions may now refer to $context variables and to operators that reference $context variables such as $$vars and $$locals. Previously these references could appear only inside probe handlers.
Consider the following C code:
int
handle_object1 ()
{
int attr1;
int attr2;
attr1 = 9;
attr2 = 99;
types type = thing1;
return 0;
}
int
handle_object2 ()
{
int attr1;
int attr2;
attr1 = 8;
attr2 = 88;
types type = thing2;
return 0;
}
Probe handlers can refer to a common binary_op_vals function that accesses the context variables attr1 and attr2, which are common to both the handle_object1 and handle_object2 functions. The binary_op_vals function can also refer to the $context variable operators $$vars and $$locals.
function binary_op_vals ()
{
printf ("type = %d attribute 1 = %d attribute 2 = %d\n", $type, $attr1, $attr2);
}
probe process.statement("handle_object1@*:11")
{
binary_op_vals ();
}
probe process.statement("handle_object2@*:22")
{
binary_op_vals ();
}
The preceding probe handlers display:
type = 0 attribute 1 = 9 attribute 2 = 99
type = 1 attribute 1 = 8 attribute 2 = 88
BPF back-end improvements
Values in user space can now be accessed with functions such as user_string, user_int, and user_long. Consider the following C source code:
long ipf = 12;
long *ipfp = &ipf;
Then run the following stap command from the SystemTap CLI:
stap --bpf --disable-cache -e 'probe process.statement("main@*:22")
{printf("%d\n",user_long($ipfp))}' -c ./tstustr
The command displays:
12
Note the use of the -c option to specify the target program. The systemtap -c <command> option can now be used with the BPF back-end. This option sets the SystemTap target process to the process ID (PID) of the running command.
Aliasing improvements
An alias provides a mechanism for augmenting the handlers that are taken by another probe. The alias handlers can be invoked either before the handler, known as a prologue, or after the handler, known as an epilogue.
It is now possible to define an alias with both a prologue and an epilogue. For example, a probe follows that has a prologue and an epilogue. The probe sets the global variable file_descriptors whenever the open or openat syscalls are invoked:
private global filenames
private global file_descriptors
probe file_open = syscall.{open,openat} {
delete filenames[tid()]
},{
if (! (tid() in filenames))
filenames[tid()] = filename;
}
probe file_open_return = syscall.{open.return,openat.return} {
if (tid() in filenames)
file_descriptors[tid(),retval] = filenames[tid()]
}
// These alias definitions can be used to only record file descriptors
// that match a given pathname.
probe file_open
{
%($# > 0 %?
if (strpos(filename,@1) == -1)
next
%)
}
probe file_open_return
{
}
probe end
{
foreach ([t,f] in file_descriptors)
{
printf ("%d %d %s\n", t, f, file_descriptors[t,f])
}
}
If this script is named alias.stp, you can execute it through:
stap alias.stp /home/user
The script displays output such as:
2309651 101 "/home/user/path1"
2310349 459 "/home/user/path2"
Additionally, the @probewrite predicate can be used to determine whether a variable has already been written to. For example, the prologue just shown could be defined as follows, to check that filesnames has already been written to:
if (@probewrite(filenames))
delete filenames[tid()]
Summary
SystemTap in Red Hat Enterprise Linux 9 fills several gaps left in previous versions and provides many conveniences to developers investigating kernel and application behavior. These new features might inspire you to try moving to a lower level of the system through SystemTap.