Recently, I have been working on enabling cooperation between SystemTap (a kernel profiling tool) and gprof (a tool that makes graphs from program profiles). This exercise has given me insight into meaningful topics only briefly touched upon at my university, such as kernel space, user space, and virtual memory. But these concepts are fundamental to the proper and safe execution of programs on any modern operating system. The trade off is some address translation when viewing memory from kernel space versus user space. In this article, you'll see how that translation can be handled.
A quick dive into user and kernel address spaces
User space and kernel space are kept separate in all modern OSes. This separation is an essential layer of security between information essential to the operation of the machine and the applications on that machine.
As the term suggests, user process address space is the area where user-run processes are stored and store information. Text pages are a portion of the process address space that stores the program’s code.
Consider a simple calculator program, of the sort installed on most OSes. If you ran this calculator program with the expression 5+5 and a web browser at the same time without any preventative measures, the browser might be able to access the mathematical expression located in the calculator’s address space and change it to a different expression, such as 6+6. Common sense dictates that this should not occur except under very unique circumstances. The operating system provides the program separation that the user desires by giving each process its own isolated address space.
The kernel address space is the realm of the kernel code and information that may be useful to other programs, such as user input/output and network data. Kernel space information is generally inaccessible except through functions known as system calls.
How virtual memory works
Virtual memory is a simple convenience in the form of an abstraction. Without virtual memory, applications would need to manage their physical memory space, coordinating with every other process running on the computer. Virtual memory leaves that management to the kernel by creating maps that allow translation between virtual and physical memory. Applications use virtual memory, and the kernel’s memory management unit (MMU) uses physical memory. The MMU is the mapping hardware that translates virtual memory addresses to physical memory addresses.
To understand how virtual memory works, suppose there are two running programs: a calculator with the expression 5+5 and an internet browser. The calculator may have the expression 5+5 stored in the address 0xdeadbeef, and the browser could store any of its data, such as an IP address, in 0xdeadbeef as well. Why doesn't one piece of information overwrite the other? The addresses where the programs are storing their data are located in virtual memory, and they refer to different locations in physical memory. Although the virtual addresses are the same, the OS handles the physical addresses, which are completely different.
The OS may also take a page from a process and write it to disk to make space for another process page, resulting in two operations at the same physical address.
SystemTap and gprof cooperation
I have been working on leveraging the data collected by SystemTap and outputting it into a gprof-readable format. SystemTap profiles programs that use shared libraries. The problem is that SystemTap can load those libraries anywhere in virtual memory. It must relativize their addresses and process them later.
umodaddr(), shown in the listing below, was created for the purpose of this translation:
function umodaddr:long (address:long)
%{ /* pragma:vma */
long vm_start = -1;
stap_find_vma_map_info(current, STAP_ARG_address,
&vm_start, NULL, NULL, NULL,NULL);
if(vm_start == -1)
STAP_ERROR("umodaddr 0x%llx unknown\n ", STAP_ARG_address);
STAP_RETURN(STAP_ARG_address - vm_start);
%}
This function takes the virtual address of the called function, found in STAP_ARG_address, and returns its offset from the beginning of the virtual memory. This function becomes a tool to ensure that SystemTap outputs addresses relative to the beginning of their virtual memory, enabling compatibility with gprof.
Conclusion
With this translation complete, a SystemTap script can use these translated addresses to create a file that can be interpreted by gprof. This file will be of the gmon.out format, so it will internally contain a histogram. A script can then automatically invoke gprof on all of these files, giving us the ability to leverage gprof's functionality to inspect a process's activity.
Understanding how virtual memory works made it possible to combine the extremely low-level capabilities of SystemTap with the informative functionality of gprof. This will allow users to gain insight into where precisely a program spends time and contribute to solving time-sensitive issues.
Last updated: June 15, 2022