Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Opening black boxes with statement tracing

August 4, 2021
Frank Eigler
Related topics:
C, C#, C++LinuxSecure Coding
Related products:
Red Hat Enterprise Linux

Share:

    Imagine you're a programmer with a problem: Your code is linked to a library that you're unfamiliar with. The code should work, but it doesn't. It almost works, but something is wrong inside the library. Another program works correctly with the same part of the same library. So, now what? It's probably a silly problem, but how will you locate it?

    In a scenario like this one, you could be in for some serious source code gazing, documentation digestion, and mailing list archaeology. If that fails, it's time to reach out to human experts and hope for the best. Or, you could try to save time with a clever tool. I was recently hit with several code traps like this one. Fortunately, I'm familiar with SystemTap.

    Statement tracing with SystemTap

    SystemTap is a general-purpose, systemwide probing and tracing tool that is just about to pass its sixteenth birthday. It can do many things, including performance monitoring, injecting print statements into running programs, and inspecting or even modifying data structures. It can probe any part of the kernel, or a running program or library in C, C++, assembler, Java, Python, and other languages. Given enough access to DWARF debuginfo, SystemTap can reveal the internal operation of the whole software stack.

    How does this help you today? By using tracing at the granularity of statements, you can find out which line of which function leads to a divergence of behavior between working and broken cases. Let's look at an example.

    Tracing a function in a shared library

    If you suspected divergence occurring in a particular function, myfn, of the shared library libfoo.so, you could trace it like this:

    
    probe process("/lib64/libfoo.so").statement("myfn@*:*") {
        println(pp(), " ", $$vars)
    
    }
    
    

    Here, you are asking SystemTap to intercept all instructions of the libfoo.so shared library corresponding to every source line statement of the myfn function. At each hit, print the current probe point (the file or line number) and pretty-print local variables. Plop that script into a file foo.stp. Set a $DEBUGINFOD_URLS environment variable if appropriate. Assuming the passing and failing tests are available as binaries, run the following script for the two tests:

    
    # stap foo.stp -c "./test_scenario1" | tee test-1.txt
    # stap foo.stp -c "./test_scenario2" | tee test-2.txt
    
    

    If this worked, you should have two text files containing streams of lines that record the history of each function-call invocation, plus a long list of variable names or values in scope:

    
    process("libfoo.so").statement("myfn@foo.c:1252") [...vars...] 
    process("libfoo.so").statement("myfn@foo.c:1253") [...vars...]  
    process("libfoo.so").statement("myfn@foo.c:1277") [...vars...] 
    
    

    Differential analysis: Comparing traces

    Next, open up the two files in an extra-wide text editor and compare them side-by-side. Segments should be much alike, maybe even the same sequence of line numbers. The key is to look for points where the line number sequences or data meaningfully diverge. A good editor like emacs, or classic Unix tools like perl, sed, and grep can help eliminate near-duplication by searching and replacing common content.

    Note: You can also add almost any conceivable filtering logic right into the SystemTap script to reduce or focus the trace.

    With practice and a keen eye, you could get a side-by-side view as specific as the one shown in Figure 1.

    A block of side-by-side statements in a text editor window.
    Figure 1: A side-by-side comparison of a filtered trace.

     

    In Figure 1, note the statement number divergence around line 1470. Note also the suggestive variable must_add_keep_alive changing its value to 0x1, but only in one of the divergent forks.

     

    Once you have the side-by-side comparison and analysis, you just need to pop open the library source code at the given file and line number, look at the variable, make the final inferences, and write a bug fix.

     

    Conclusion

     

    This sunny day scenario might sound too good to be true, but let me assure you it is not: It's based on not one but three real head-scratchers involving different bugs, programs, and libraries. Even as an experienced programmer, I benefited from a tool that let me cut right through the firehose of activity in an unfamiliar library and focus in on the most salient lines in the code.

     

    Differential, statement-level tracing is a powerful technique, and SystemTap is one of the few tools that does it well. Try it, and check out some of SystemTap's other related examples.

     

    Last updated: September 19, 2022

    Related Posts

    • Automating the testing process for SystemTap, Part 1: Test automation with libvirt and Buildbot

    • Automating the testing process for SystemTap, Part 2: Test result analysis with Bunsen

    • Introducing debuginfod, the elfutils debuginfo server

    Recent Posts

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue