Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Quickly analyze regression testing results with Bunsen

October 17, 2023
Keith Seitz
Related topics:
CI/CDLinuxOpen source
Related products:
Red Hat Enterprise Linux

Share:

    It is not altogether uncommon for software developers to do regression testing, comparing results of testing before and after a change. This is a common continuous integration strategy that I employ when doing releases of GDB for Red Hat Enterprise Linux (RHEL).

    What happens, though, when the results of testing are non-deterministic due to racy (thread) tests or "slow" or high-load machines that cause tests to timeout? This often requires human intervention and analysis of the test results.

    Preparing a release of GDB typically involves identifying patches to backport to fix specific code deficiencies identified in bugs reported by users. Once these backports are completed, we effectively have a "patch" (a quasi-release candidate, if you will), that we can test to verify that no regressions will be introduced into the release.

    For the GDB team, this "release candidate" is tested using an internal command-line tool called gdb-beaker, which, as the name suggests, uses an internal Beaker instance to reserve computers of every supported RHEL architecture and run regression tests. The beaker task associated with this tool will install a fresh copy of the operating system, check out the desired Git repository, install build dependencies, and build and test GDB before and after applying the patch to test.

    A Bunsen database is then created, and all test results are imported. This is where the fun begins.

    What is Bunsen and how does it help?

    From the project's homepage: "Bunsen is a test result storage and analysis toolkit that collects test result and build log files in a variety of formats (e.g., DejaGnu, Autoconf config.log, glibc), stores them in a ludicrously compact de-duplicated Git repo, parses and indexes the contents, and provides a toolkit for analyzing and browsing the indexed test results."

    Before Bunsen, the process of analyzing test results was very manual: inspect diffs of unpatched versus patched results and determine if a regression has occurred. We run regression testing against several DejaGnu "target boards," meaning that we actually run testing at least three times for every supported architecture—more on x86_64 because we support both 64- and 32-bit executables on that architecture. In total, a single release requires the analysis of 15 test runs—per actual release. Factor in GCC Toolset releases on both RHEL 8 and RHEL 9, and this amounts to a whopping 60 test runs to analyze!

    Staring at this many test diffs is exceptionally error-prone, but with Bunsen, there is a more accurate and much faster way to analyze this many results. The rest of this article will explain the approach I've implemented internally at Red Hat to shave many person-days off this process.

    Overview

    Bunsen stores test data in a Git repository, tagging each data set with a user-generated name. Therefore, the first order of business is to name our test result runs uniformly.

    There are a number of considerations for GDB. We have different target boards, different "bitness" (32- vs 64-bit), and "unpatched" versus "patched" test runs to name.

    I have chosen to name all Bunsen imports with (un)?patched-TARGET_BOARD(.-mBITNESS)?, where TARGET_BOARD is the DejaGnu board name (see below), and BITNESS, if needed, is either "32" or "64". For example, the Git tag that contains the results for the patched, 64-bit native-gdbserver test run would be patched-native-gdbserver.-m64.

    GDB's test suite is written in DejaGnu, and the GDB sources define a number of "target boards" meant to facilitate testing various hardware/software configurations. Table 1 summarizes the configurations that are tested for RHEL GDB releases. I refer to these throughout this article via a "target abbreviation," which I will use later to save my fingers from having to repeatedly type so much.

    Abbreviation

    DejaGnu –target_board/bitness

    Support architectures

    U

    unix/-m64

    All

    G

    native-gdbserver/-m64

    All

    E

    native-extended-gdbserver/-m64

    All

    u

    unix/-m32

    x86_64 only

    g

    native-gdbserver/-m32

    x86_64 only

    e

    native-extended-gdbserver/-m32

    x86_64 only

    Table 1: GDB DejaGnu target boards and target abbreviations.

    When the Bunsen database is populated by automated testing, it will create imports for each of these target boards both before ("unpatched") and after ("patched") patching the sources with the proposed changes for the release.

    A note on implementation

    For all of the following implementations, it is assumed that the environment has two variables defined that point at the Bunsen instance's Git repository (BUNSEN_REPO, e.g., $HOME/bunsendb) and the Bunsen database (BUNSEN_DB, e.g., $BUNSEN_REPO/bunsen.sqlite3).

    Note: The Bunsen Git repository is the Git repository created by importing results into Bunsen. It is not the actual source repository of the Bunsen project.

    The scripts

    The following scripts/shell functions are almost verbatim copies of what I use to analyze test results for RHEL GDB. All of the examples given in the rest of this article have been run on Red Hat Enterprise Linux 9 on x86_64, and while they work for me, they are neither "perfect" nor guaranteed to work for you. While all of the scripts in this article are actually shell functions, I will use the terms "script(s)" and "function(s)" interchangeably.

    For discussion purposes, I have intentionally introduced regressions into two test files, gdb.base/default.exp and gdb.cp/var-tag.exp, and I have limited the output of the scripts to these results. In later examples, we will discover what these regressions are using the methods described herein.

    tgt: Target abbreviation expansion

    This simple function eases the pain of typing native-extended-gdbserver/-m64 over and over. While the Bunsen database stores the "long form" of these board configurations, I don't like to type them all the time. The following shell function allows me to simply type tgt U. You can pass the output of this function directly to DejaGnu via the RUNTESTFLAGS variable, as indicated in the function's comments.

    # Function to convert one-letter target abbreviation to full target boards
    # strings.
    #
    # This function returns strings that can be passed to
    # RUNTESTFLAGS="--target_board $(tgt LTR)". LTR is either 'u', 'g', 'e',
    # or any of those capitalized. When testing 64- and 32-bit variations,
    # lower case will represent 32-bit boards and capitals will represent
    # 64-bit boards. If we only support one bitness, then both lower and
    # capital versions will be the same.
    #
    # For example, `u' on x86_64 will return "unix/-m32", but on aarch64, 'u'
    #  and `U' will just be "unix".
    
    function tgt() {
        if [ $# -ne 1 ]; then
            >&2 echo "error: tgt requires an argument"
            >&2 echo “tgt: usage: tgt [UuGgEe]"
            return 1
        fi
    
        case $(uname -m) in
        aarch64)
            case "$1" in
            U|u) echo "unix";;
            G|g) echo "native-gdbserver";;
            E|e) echo "native-extended-gdbserver" ;;
            *)
                >&2 echo "unknown letter' $1': must be U, u, G, g, E, or e"
                return 1
                ;;
            esac
            ;;
        ppc64le)
            # For historical reasons (to follow gdb.spec), we explicitly
            # use "-m64".
            case "$1" in
            U|u) echo "unix/-m64" ;;
            G|g) echo "native-gdbserver/-m64" ;;
            E|e) echo "native-extended-gdbserver/-m64" ;;
            *)
                >&2 echo "unknown letter' $1': must be U, u, G, g, E, or e"
                return 1
                ;;
            esac
            ;;
        s390x)
            # For historical reasons (to follow gdb.spec), we explicitly
            # use "-m64".
            case "$1" in
            U|u) echo "unix/-m64" ;;
            G|g) echo "native-gdbserver/-m64" ;;
            E|e) echo "native-extended-gdbserver/-m64" ;;
            *)
                >&2 echo "unknown letter' $1': must be U, u, G, g, E, or e"
                return 1
                ;;
            esac
            ;;
        x86_64)
            case "$1" in
            U) echo "unix/-m64";;
            u) echo "unix/-m32";;
            G) echo "native-gdbserver/-m64";;
            g) echo "native-gdbserver/-m32";;
            E) echo "native-extended-gdbserver/-m64";;
            e) echo "native-extended-gdbserver/-m32";;
            *)
                >&2 echo "unknown letter' $1': must be U, u, G, g, E, or e"
                return 1
                ;;
            esac
            ;;
        *)
            >&2 echo "unknown architecture: $(uname -m)"
            return 1
            ;;
        esac
    
        return 0
    }
    
    

    The following examples demonstrate how you can use this function, including how to use the tgt function to tell DejaGnu to use the specific target board for testing.

    $ tgt g
    native-gdbserver/-m32
    $ make check RUNTESTFLAGS=”--target_board $(tgt U)"
    [snip]

    btgt: Return a Git tag snippet for a target abbreviation

    Because Bunsen uses Git for underlying storage, we are necessarily limited to naming our imports with valid Git tag characters. This means translating any invalid characters to .. This is what btgt does.

    # Similar to tgt above, but outputs something suitable for use in
    # bunsen git tags, e.g., "unix.-m32".
    function btgt()
    {
        if [ $# -ne 1 ]; then
              >&2 echo "error: btgt requires an argument"
              >&2 echo “btgt: usage: btgt [UuGgEe]”
              return 1
        fi
    
        r=$(tgt $1)
        echo $r | sed s@/@.@g
        return 0
    }
    

    The following examples show how btgt converts the target board abbreviations g (native-gdbsever/-m32 ) and U (unix/-m64) into something suitable for use with Git. We will use this function's output later to construct the full Git tag used when importing results into our Bunsen instance.

    $ btgt g
    native-gdbserver.-m32
    $ btgt U
    unix.-m64

    btag: Create the Git tag used by Bunsen imports

    Using the above two scripts, we can define a simple shell function to output the complete name of any test result run (a.k.a. Bunsen import) with minimal typing. [See a theme? :-)]

    Note that like any of the functions defined in this article, any argument that designates a "patched" test run versus an "unpatched" one can simply use some subset of those words; i.e., u will expand to unpatched and pa will expand to patched.

    # Output a suitable tag for the test run's bunsen import.
    # These look like "(un)?patched-TARGET_BOARD.-mBITNESS"
    function btag()
    {
        if [ $# -ne 2 ]; then
            >&2 echo "error: btag requires two arguments"
            >&2 echo "btag: usage: btag  [UuGgEe] (un)?patched"
            return 1
        fi
    
        target=$1
        case "$2" in
            u*) ver="unpatched" ;;
            p*) ver="patched" ;;
            *)
                >&2 echo "error: unknown version, '$2'. Must be 'unpatched' or 'patched'"
                return 1
                ;;
        esac
        echo "$ver-$(btgt $target)"
        return 0
    }
    
    

    The following output lists the Git tags that we will use later to import the results for the target board configurations native-gdbserver/-m32 and unix/-m64:

    $ btag g u
    unpatched-native-gdbserver.-m32
    $ btag U p
    patched-unix.-m64

    btags: List all imported Bunsen test runs

    Bunsen creates a Git tag for each import. Therefore, a simple shell alias suffices to query the Bunsen Git repository for any stored results. This is usually the first thing I do to verify that testing completed successfully.

    alias btags="git --git-dir $BUNSEN_REPO/.git tag -l"

    In the following example, the output of btags shows that our Bunsen instance is loaded with patched and unpatched results for all the target boards and configurations tested on x86_64 RHEL.

    $ btags
    patched-native-extended-gdbserver.-m32
    patched-native-extended-gdbserver.-m64
    patched-native-gdbserver.-m32
    patched-native-gdbserver.-m64
    patched-unix.-m32
    patched-unix.-m64
    unpatched-native-extended-gdbserver.-m32
    unpatched-native-extended-gdbserver.-m64
    unpatched-native-gdbserver.-m32
    unpatched-native-gdbserver.-m64
    unpatched-unix.-m32
    unpatched-unix.-m64

    bsum: Summarize test results

    Finally, we get to the interesting scripts!

    bsum outputs the testing summary from a given test run that has been imported into Bunsen. Its output mimics DejaGnu's summary table. To output results that exactly imitate DejaGNU's output, use the –verbose option. This will explicitly list all non-passing tests.

    The function also optionally takes the name of a test file (as a glob expression) to limit the results to specific test file(s), as shown in the example that follows:

    # Summarize test results for a given bunsen commit.
    function bsum () {
        if [ $# -lt 2 ]; then
            >&2 echo "error: bsum requires at least two arguments"
            >&2 echo "bsum: usage: bsum [-v] TARGET_ABBREV (un)?patched [EXP_GLOB]"
            return 1
        fi
    
        # Get verbose output option.
        verbose=""
        if [ "$1" = "-v" ]; then
            verbose="-v"
            shift
        fi
    
        # Get target abbreviation and version.
        target=$1 # any valid target abbreviation (UuGgEe)
        ver=$2 # "unpatched" or "patched"
    
        # Get optional expfile glob.
        glob=""
        if [ $# -eq 3 ]; then
            glob="--expfile-glob $3"
        fi
    
        # Run sum script.
        tag=$(btag $target $ver)
        r-dejagnu-summary --git ${BUNSEN_REPO} --db ${BUNSEN_DB} $verbose \ $glob $tag
        return $?
    }
    
    In the following examples, the first command outputs the summary results for the entire test suite run of the U target board (unix/-m64) in the "unpatched" test suite run. The second outputs the summary for just GDB's C++-specific tests (in the gdb.cp folder) for the same test run.
    $ bsum U u
    # of expected passes        111673
    # of unexpected failures    10
    # of expected failures      83
    # of known failures         112
    # of untested testcases     30
    # of unsupported tests      559
    $ bsum U u 'gdb.cp/*.exp'
    # of expected passes        8585
    # of expected failures      14
    # of known failures         25
    # of untested testcases     3
    

    Note that the C++-specific summary did not output a field for "unexpected failures" because there were no such failures. This is consistent with how DejaGnu outputs results. (I often compare manual test run summaries with the data stored in Bunsen.)

    breg: Compute regressions

    The breg function will query Bunsen for any regressions discovered during testing. Its inputs include the target board abbreviation and an optional test file to which to limit results. Notice that unlike bsum, this function does not accept a glob expression.

    # Convenience function to find (possible) regressions between two testruns.
    # Optionally takes a test file name (expfile).
    function breg () {
        if [ $# -lt 1 ]; then
            >&2 echo "error: breg requires at least one arguments"
            >&2 echo "breg: usage: breg TARGET_ABBREV [EXPFILE]"
            return 1
        fi
    
        # Get target abbreviation.
        expfile=""
        target=$1 # any valid target abbreviation (UuGgEe)
    
        # Get optional expfile.
        if [ $# -eq 2 ]; then
            expfile="--dgexpfile $2"
        fi
    
        # Run diff script. Add "--regressions"?
        r-diff-testruns --git ${BUNSEN_REPO} --db ${BUNSEN_DB} \
                        $(btag $target u) $(btag $target p) $expfile
        return $?
    }
    In the following example, breg lists the regressions discovered between running the "unpatched" and "patched" test suite runs of the U target. For the sake of brevity, I have limited the output to the regressions I previously introduced to gdb.base/default.exp and gdb.cp/var-tag.exp:
    $ breg U
    2023-08-30 13:24:45,299:r-diff-testruns:INFO:opened git repo /root/bunsendb
    
        unpatched-unix.-m64
           patched-unix.-m64
    
    dejagnu diffs
    
    gdb.base/default.exp gdb.base/default.exp: call
        PASS    gdb.log:109690 gdb.sum:9435
           FAIL    gdb.log:109893 gdb.sum:9435
    
    gdb.base/default.exp gdb.base/default.exp: inspect
        PASS    gdb.log:111372 gdb.sum:9529
           FAIL    gdb.log:111575 gdb.sum:9529
    
    gdb.base/default.exp gdb.base/default.exp: print
        PASS    gdb.log:111453 gdb.sum:9551
           FAIL    gdb.log:111656 gdb.sum:9551
    
    gdb.base/default.exp gdb.base/default.exp: print "p" abbreviation
        PASS    gdb.log:111450 gdb.sum:9552
           FAIL    gdb.log:111653 gdb.sum:9552
    
    gdb.base/default.exp gdb.base/default.exp: ptype
        PASS    gdb.log:111459 gdb.sum:9554
           FAIL    gdb.log:111662 gdb.sum:9554
    
    gdb.base/default.exp gdb.base/default.exp: whatis
        PASS    gdb.log:112837 gdb.sum:9675
           FAIL    gdb.log:113040 gdb.sum:9675
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: before start: c++: ptype C
        PASS    gdb.log:424504 gdb.sum:62659
           FAIL    gdb.log:425341 gdb.sum:62685
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: before start: c: ptype C
        PASS    gdb.log:424626 gdb.sum:62689
           FAIL    gdb.log:425463 gdb.sum:62715
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: C::f: c++: ptype C
        PASS    gdb.log:425013 gdb.sum:62723
           FAIL    gdb.log:425850 gdb.sum:62750
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: C::f: c: ptype C
        PASS    gdb.log:425135 gdb.sum:62753
           FAIL    gdb.log:425972 gdb.sum:62780
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: main: c++: ptype C
        PASS    gdb.log:424761 gdb.sum:62783
           FAIL    gdb.log:425598 gdb.sum:62810
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: main: c: ptype C
        PASS    gdb.log:424883 gdb.sum:62813
           FAIL    gdb.log:425720 gdb.sum:62840

    The following example limits the list to those in the gdb.cp/var-tag.exp test file:

    $ breg U gdb.cp/var-tag.exp
    2023-08-30 13:26:30,565:r-diff-testruns:INFO:opened git repo /root/bunsendb
    
        unpatched-unix.-m64
           patched-unix.-m64
    
    dejagnu diffs
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: before start: c++: ptype C
        PASS    gdb.log:424504 gdb.sum:62659
           FAIL    gdb.log:425341 gdb.sum:62685
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: before start: c: ptype C
        PASS    gdb.log:424626 gdb.sum:62689
           FAIL    gdb.log:425463 gdb.sum:62715
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: C::f: c++: ptype C
        PASS    gdb.log:425013 gdb.sum:62723
           FAIL    gdb.log:425850 gdb.sum:62750
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: C::f: c: ptype C
        PASS    gdb.log:425135 gdb.sum:62753
           FAIL    gdb.log:425972 gdb.sum:62780
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: main: c++: ptype C
        PASS    gdb.log:424761 gdb.sum:62783
           FAIL    gdb.log:425598 gdb.sum:62810
    
    gdb.cp/var-tag.exp gdb.cp/var-tag.exp: main: c: ptype C
        PASS    gdb.log:424883 gdb.sum:62813
           FAIL    gdb.log:425720 gdb.sum:62840
    

    Now that Bunsen can tell us what test(s) might have regressed, the next step is to ascertain why a test might have failed. This normally means comparing the test output in the two log files, but with Bunsen, there is a much easier way.

    bdiff: Output a diff between two tests

    This script outputs a diff of the log output for a given test for a target board. This is where Bunsen can save ridiculous amounts of time by forgoing having to manually search log files for a given test and inspecting the results. With Bunsen, it's all automagic.

    The function requires two inputs: the target board (abbreviation) and the test name.

    # Convenience function to diff two test results between any two commits.
    function bdiff()
    {
        if [ $# -ne 2 ]; then
            >&2 echo "error: bdiff requires two arguments"
            >&2 echo "bdiff: usage: bdiff TARGET_ABBREV 'TEST'"
            return 1
        fi
    
        target=$1
        r-dejagnu-diff-logs --template=diff --git ${BUNSEN_REPO}  \
          --db ${BUNSEN_DB} $(btag $target unpatched) $(btag $target patched) \ 
          "$2"
        return 0
    }

    Now let's use bdiff to quickly figure out what happened to cause my (intentionally introduced) regressions.

    One of the regressing tests identified in the breg example output is gdb.base/default.exp: print. bdiff can show us how the output of this test changed between the "unpatched" and "patched" test runs:

    $ bdiff U 'gdb.base/default.exp: print'
    logfile: <class 'dict'> *********************
    *** unpatched-unix.-m64: gdb.base/default.exp: print
    --- patched-unix.-m64: gdb.base/default.exp: print
    ***************
    *** 1,3 ****
      print
    ! The history is empty.
    ! (gdb) PASS: gdb.base/default.exp: print
    --- 1,3 ----
      print
    ! The history is really empty.
    ! (gdb) FAIL: gdb.base/default.exp: print

    The diff clearly shows how I artificially regressed the test by adding the word "really" to the string The history is empty.

    Is that also what caused the subsequent tests in the file to regress? Asking for the Bunsen diff of the next test (gdb.base/default.exp: print "p" abbreviation) also clearly shows this to be the case:

    $ bdiff U 'gdb.base/default.exp: print "p" abbreviation'
    
    logfile: <class 'dict'> *********************
    *** unpatched-unix.-m64: gdb.base/default.exp: print "p" abbreviation
    --- patched-unix.-m64: gdb.base/default.exp: print "p" abbreviation
    ***************
    *** 1,3 ****
      p
    ! The history is empty.
    ! (gdb) PASS: gdb.base/default.exp: print "p" abbreviation
    --- 1,3 ----
      print
    ! The history is really empty.
    ! (gdb) FAIL: gdb.base/default.exp: print "p" abbreviation

    As for the subsequent failing tests, you can repeat this process to verify that the sudden appearance of the word "really" is the cause for all of these failures.

    Turning our attention to the other introduced regression in gdb.cp/var-tag.exp:

    $ bdiff U 'gdb.cp/var-tag.exp: in C::f: c++: ptype C'
    logfile: <class 'dict'> *********************
    *** unpatched-unix.-m64: gdb.cp/var-tag.exp: in C::f: c++: ptype C
    --- patched-unix.-m64: gdb.cp/var-tag.exp: in C::f: c++: ptype C
    ***************
    *** 1,12 ****
      ptype C
    ! type = class C {
    public:
    !  C::C1 C1;
    !  C::E1 E1;
    !  C::U1 U1;
    
    !  C(void);
       void global(void) const;
       int f(void) const;
      }
    ! (gdb) PASS: gdb.cp/var-tag.exp: in C::f: c++: ptype C
    --- 1,12 ----
      ptype C
    ! type = class CX {
    public:
    !  CX::C1 C1;
    !  CX::E1 E1;
    !  CX::U1 U1;
    
    !  CX(void);
       void global(void) const;
       int f(void) const;
      }
    ! (gdb) FAIL: gdb.cp/var-tag.exp: in C::f: c++: ptype C
    

    From the above output, it is obvious that I renamed a class in the source file from C to CX, causing the observed regression. Using bdiff on the other failing tests would show that this is the root cause of the regressions.

    Conclusion

    In this article, I have attempted to demonstrate how I use Bunsen to facilitate the analysis of automated regression testing for GDB releases on Red Hat Enterprise Linux. While there is still more automation and analysis that would yield further benefits, the process implemented above has already saved me countless hours of tedious and unrewarding work.

    If you are also doing manual regression testing analyses, I hope this article will inspire you to adopt Bunsen and save yourself and your organization time and money!

    Further reading

    • Automating the testing process for SystemTap, Part 2: Test result analysis with Bunsen by Serhei Makarov
    • Detecting nondeterministic test cases with Bunsen by Serhei Makarov
    • Store and analyze your test-suite logs with this open source tool by Frank Eigler
    • Uncover interesting test cases with AI/ML and Bunsen by Frank Eigler

    Special thanks

    I would like to thank Serhei Makarov, Frank Eigler, and the entire Bunsen community for all the work they've done over the past few years. You have saved my sanity!

    Related Posts

    • New C features in GCC 13

    • A leaner <iostream> in libstdc++ for GCC 13

    • Improvements to static analysis in the GCC 13 compiler

    • Use source-level annotations to help GCC detect buffer overflows

    • How to debug stack frames and recursion in GDB

    Recent Posts

    • How to run AI models in cloud development environments

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    • How to encrypt RHEL images for Azure confidential VMs

    • How to manage RHEL virtual machines with Podman Desktop

    What’s up next?

    Bash Commands feature image

    Bash is the default shell installed on GNU/Linux distributions and many other Unix-style systems, such as macOS. Download the Bash Commands cheat sheet to get started with Bash scripting.

    Get the cheat sheet
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue