Memory error checking in C and C++: Comparing Sanitizers and Valgrind

This article compares two tools, Sanitizers and Valgrind, that find memory bugs in programs written in memory-unsafe languages. These two tools work in very different ways. Therefore, while Sanitizers (developed by Google engineers) presents several advantages over Valgrind, each has strengths and weaknesses. Note that the Sanitizers project has a plural name because the suite consists of several tools, which we will explore in this article.

Memory-checking tools are for memory-unsafe languages such as C and C++, not for Java, Python, and similar memory-safe languages. In memory-unsafe languages, it is easy to mistakenly write past the end of a memory buffer or read memory after it has been freed. Programs containing such bugs might run flawlessly most of the time and crash only very rarely. Catching these bugs is difficult, which is why we need tools for that purpose.

Valgrind imposes a much higher slowdown on programs than Sanitizers. A program running under Valgrind could run 20 to 50 times slower than in regular production. This can be a showstopper for CPU-intensive programs. The slowdown for Sanitizers is generally 2 to 4 times worse than regular production. Instead of Valgrind, you can specify the use of Sanitizers during compilation.

This article is divided into the following sections:

A quick Sanitizers how-to

The following list of rules and recommendations sums up some of the information from this article, and can help readers who are familiar with Sanitizers:

  • Clang/GCC option: -fsanitize=address -fsanitize=undefined -fno-sanitize-recover=all -fsanitize=float-divide-by-zero -fsanitize=float-cast-overflow -fno-sanitize=null -fno-sanitize=alignment
    • For LLDB/GDB and to prevent very short stack traces and usually false leaks detection:
      $ export ASAN_OPTIONS=abort_on_error=1:fast_unwind_on_malloc=0:detect_leaks=0 UBSAN_OPTIONS=print_stacktrace=1
    • Fix false memory leak reporting when using glib2:
      $ export G_SLICE=always-malloc G_DEBUG=gc-friendly
    • Clang option to catch uninitialized memory reads: -fsanitize=memory. That option cannot be combined with -fsanitize=address.
    • rpmbuild *.spec files should additionally use: -Wp,-U_FORTIFY_SOURCE.

    Performance benefits of Sanitizers

    Valgrind uses dynamic instrumentation instead of static instrumentation at compile time, which leads to the high performance overhead that can be impractical for CPU-intensive applications. Sanitizers uses static instrumentation and allows for similar checks with a lower overhead.

    Table 1 presents a more complete comparison showing features and runtime slowdowns in both sets of tools.

    Table 1: Comparison between Sanitizers and Valgrind
    Sanitizers Valgrind
    ASAN: General runtime slowdown 2 times - 4 times 20 times - 50 times
    ASAN: Slowdown test loop 1 (=the same) 3.45 times (+23s=12 times)
    ASAN: Slowdown test malloc 17 times 101 times (+23s=256 times)
    ASAN: Slowdown test AVX2 1.05 (=5% slower) 53 times (+23s=60 times)
    TSAN: Slowdown test data races 9 times 340 times (+31s=404 times)
    Variable overruns (stack, global) Caught (ASAN) Missed (Memcheck)
    Uninitialized memory Caught (MSAN) Caught (Memcheck)
    Stack use after return Caught (ASAN) Missed (Memcheck)
    Undefined behavior Caught (UBSAN) Missed (Memcheck)
    Memory leaks Caught (LSAN) Caught (Memcheck)
    Data races Caught (TSAN) Caught (Helgrind)
    Recompilation Required Not needed

    In my tests, Valgrind spends 23 seconds during startup reading system debug info files. This overhead can be mitigated temporarily by renaming the /usr/lib/debug directory. Please do not forget to rename it back. Otherwise, system debugging information could be missing and at the same time no longer installable (as it is already installed):

    $ sudo mv /usr/lib/debug /usr/lib/debug-x; sleep 1h; sudo mv /usr/lib/debug-x /usr/lib/debug
    

    Slowdown tests were compiled through the following command:

    $ clang++ -g -O3 -march=native -ffast-math ...
    clang-11.0.0-2.fc33.x86_64
    valgrind-3.16.1-5.fc33.x86_64

    The commands used plain -fsanitize=address, -fsanitize=undefined, or -fsanitize=thread without any of the extra options I suggested in the Sanitizers how-to.

    This article uses the Clang compiler. GCC has the same support for Sanitizers, except that the compiler lacks support for -fsanitize=memory (discussed in the section MSAN: Uninitialized memory reads).

    Installing debuginfo

    The examples in this article assume that *-debuginfo.rpm files are already installed. You can install them using dnf debuginfo-install packagename. Use yum instead of dnf on some versions of Red Hat Enterprise Linux (RHEL). When you start the program in gdb it will tell you whether you are still missing some of the RPMs:

    $ cat >vector.cpp <<'EOF'
    // Here should be your program you are debugging,
    // this std::vector sample code is just an example.
    #include <vector>
    int main() {
      std::vector<int> v;
    }
    EOF
    $ clang++ -o vector vector.cpp -Wall -g
    $ gdb ./vector
    Reading symbols from ./vector...
    (gdb) start
    ...
    Missing separate debuginfos, use: dnf debuginfo-install glibc-2.32-2.fc33.x86_64
    ...
    Missing separate debuginfos, use: dnf debuginfo-install libgcc-10.2.1-9.fc33.x86_64 libstdc++-10.2.1-9.fc33.x86_64
    (gdb) quit
    $ sudo dnf debuginfo-install glibc-2.32-2.fc33.x86_64
    ...
    $ sudo dnf debuginfo-install libgcc-10.2.1-9.fc33.x86_64 libstdc++-10.2.1-9.fc33.x86_64
    ...
    

    Sample case: A buffer overrun

    The following program runs fine despite having a buffer overrun bug. While the bug here looks obvious, in real world programs such bugs are more hidden and difficult to find. C developers commonly use Valgrind Memcheck, which can catch most of these errors. A run of the program with Valgrind Memcheck shows:

    $ cat >overrun.c <<'EOF'
    #include <stdlib.h>
    int main() {
      char *p = malloc(16);
      p[24] = 1; // buffer overrun, p has only 16 bytes
      free(p); // free(): invalid pointer
      return 0;
    }
    EOF
    $ clang -o overrun overrun.c -Wall -g
    $ valgrind ./overrun
    ...
    ==60988== Invalid write of size 1
    ==60988==    at 0x401154: main (overrun.c:4)
    ==60988==  Address 0x4a52058 is 8 bytes after a block of size 16 alloc'd
    ==60988==    at 0x4839809: malloc (vg_replace_malloc.c:307)
    ==60988==    by 0x401147: main (overrun.c:3)
    

    Note: You can try out this code and view its output in Compiler Explorer.

    Because the program runs fine without Valgrind, one might think there is no bug. After adding code simulating a non-trivial program, the example crashes even when being run on its own. Please note that the message free(): invalid pointer comes from glibc (the GNU C standard system library) and not from any Sanitizers or Valgrind instrumentation:

    $ cat >overrun.c <<'EOF'
    #include <stdlib.h>
    int main() {
      char *p = malloc(16);
      char *p2 = malloc(16);
      p[24] = 1; // buffer overrun, p has only 16 bytes
      free(p2); // free(): invalid pointer
      free(p);
      return 0;
    }
    EOF
    $ clang -o overrun overrun.c -Wall -g
    $ ./overrun
    free(): invalid pointer
    Aborted
    

    Note: You can try out this code and view its output in Compiler Explorer.

    The tool in Sanitizers corresponding to the use of Valgrind for this memory error is AddressSanitizer. The difference is that, whereas Valgrind runs with regular executables, AddressSanitizer needs a recompilation of the code, which is then executed directly without any extra tool:

    $ clang -o overrun overrun.c -Wall -g -fsanitize=address
    $ ./overrun
    =================================================================
    ==61268==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000028 at pc 0x0000004011b8 bp 0x7fff37c8aa70 sp 0x7fff37c8aa68
    WRITE of size 1 at 0x602000000028 thread T0
        #0 0x4011b7 in main overrun.c:4
        #1 0x7f4c94a2d1e1 in __libc_start_main ../csu/libc-start.c:314
        #2 0x4010ad in _start (overrun+0x4010ad)
    
    0x602000000028 is located 8 bytes to the right of 16-byte region [0x602000000010,0x602000000020)
    allocated by thread T0 here:
        #0 0x7f4c94c7b3cf in __interceptor_malloc (/lib64/libasan.so.6+0xab3cf)
        #1 0x401177 in main overrun.c:3
        #2 0x7f4c94a2d1e1 in __libc_start_main ../csu/libc-start.c:314
    
    SUMMARY: AddressSanitizer: heap-buffer-overflow overrun.c:4 in main
    ...
    

    Note: You can try out this code and view its output in Compiler Explorer.

    ASAN: Out-of-bounds access in stack variables

    Because Valgrind does not require recompiling the program, it cannot detect some invalid memory accesses. One such bug is accessing memory out of the range of automatic (local) variables and global variables. (See the AddressSanitizer Stack Out of Bounds documentation.)

    Because Valgrind gets involved only at runtime, it does fence and track memory from malloc allocations. Unfortunately, allocation of variables on the stack is inherent to the already compiled program without calling any external functions such as malloc, so Valgrind cannot find out whether the access to stack memory is valid or is just an accidental runaway from a different stack object. Sanitizers, on the other hand, instruments all the code at compile time, when the compiler still knows which specific variable on the stack the program is trying to access and what the variable's correct stack boundaries are:

    $ cat >stack.c <<'EOF'
    int main(int argc, char **argv) {
      int a[100];
      return a[argc + 100];
    }
    EOF
    $ clang -o stack stack.c -Wall -g -fsanitize=address
    $ ./stack
    ==88682==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff54f500f4 at pc 0x0000004f4c51 bp 0x7fff54f4ff30 sp 0x7fff54f4ff28
    READ of size 4 at 0x7fff54f500f4 thread T0
        #0 0x4f4c50 in main /tmp/stack.c:3:10
        #1 0x7f9983c7e1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #2 0x41c41d in _start (/tmp/stack+0x41c41d)
    
    Address 0x7fff54f500f4 is located in stack of thread T0 at offset 436 in frame
        #0 0x4f4a9f in main /tmp/stack.c:1
    
      This frame has 1 object(s):
        [32, 432) 'a' (line 2) <== Memory access at offset 436 overflows this variable
    HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
          (longjmp and C++ exceptions *are* supported)
    SUMMARY: AddressSanitizer: stack-buffer-overflow /tmp/stack.c:3:10 in main
    ...
    $ clang -o stack stack.c -Wall -g
    $ valgrind ./stack
    ...
    (nothing found by Valgrind)
    

    Note: You can try out this code and view its output in Compiler Explorer.

    ASAN: Out-of-bounds access in global variables

    As with variables on the stack, Valgrind cannot detect a global variable overrun because it does not recompile the program. (See the AddressSanitizer Global Out of Bounds documentation.)

    I described earlier why Valgrind cannot catch such errors. Here are the outputs from AddressSanitizer and Valgrind:

    $ cat >global.c <<'EOF'
    int a[100];
    int main(int argc, char **argv) {
      return a[argc + 100];
    }
    EOF
    $ clang -o global global.c -Wall -g -fsanitize=address
    $ ./global
    =================================================================
    ==88735==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000000dcee74 at pc 0x0000004f4b04 bp 0x7ffd5292b580 sp 0x7ffd5292b578
    READ of size 4 at 0x000000dcee74 thread T0
        #0 0x4f4b03 in main /tmp/global.c:3:10
        #1 0x7fd416cda1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #2 0x41c41d in _start (/tmp/global+0x41c41d)
    
    0x000000dcee74 is located 4 bytes to the right of global variable 'a' defined in 'global.c:1:5' (0xdcece0) of size 400
    SUMMARY: AddressSanitizer: global-buffer-overflow /tmp/global.c:3:10 in main
    ...
    $ clang -o global global.c -Wall -g
    $ valgrind ./global
    ...
    (nothing found by Valgrind)
    

    Note: You can try out this code and view its output in Compiler Explorer.

    MSAN: Uninitialized memory reads

    AddressSanitizer does not detect reads of uninitialized memory. MemorySanitizer was developed for that. It needs a separate compilation and run. (See the MemorySanitizer documentation.) Why AddressSanitizer was not designed to include the functionality of MemorySanitizer is unclear to me, and I'm not the only one.

    A run of MemorySanitizer follows:

    $ cat >uninit.c <<'EOF'
    int main(int argc, char **argv) {
      int a[2];
      if (a[argc != 1])
        return 1;
      else
        return 0;
    }
    EOF
    $ clang -o uninit uninit.c -Wall -g -fsanitize=address -fsanitize=memory
    clang-11: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=memory'
    $ clang -o uninit uninit.c -Wall -g -fsanitize=memory
    $ ./uninit
    ==63929==WARNING: MemorySanitizer: use-of-uninitialized-value
        #0 0x4985a9 in main /tmp/uninit.c:3:7
        #1 0x7f93e232c1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #2 0x41c39d in _start (/tmp/uninit+0x41c39d)
    SUMMARY: MemorySanitizer: use-of-uninitialized-value /tmp/uninit.c:3:7 in main
    

    Catching this bug is easier with Valgrind, which reports uninitialized memory reads by default:

    $ clang -o uninit uninit.c -Wall -g
    $ valgrind ./uninit
    ...
    ==87991== Conditional jump or move depends on uninitialised value(s)
    ==87991==    at 0x401136: main (uninit.c:3)
    ...
    

    Note: You can try out this code and view its output in Compiler Explorer.

    ASAN: Stack use after return

    AddressSanitizer requires enabling ASAN_OPTIONS=detect_stack_use_after_return=1 at runtime, because this feature imposes extra runtime overhead. (See the AddressSanitizer Use After Return documentation.) The following is a sample program that runs without error by itself or with Valgrind, but reveals the error when run with AddressSanitizer:

    $ cat >uar.cpp <<'EOF'
    int *f() {
      int i = 42;
      int *p = &i;
      return p;
    }
    int g(int *p) {
      return *p;
    }
    int main() {
      return g(f());
    }
    EOF
    $ clang++ -o uar uar.cpp -Wall -g -fsanitize=address
    $ ./uar
    (nothing found by default)
    $ ASAN_OPTIONS=detect_stack_use_after_return=1 ./uar
    =================================================================
    ==164341==ERROR: AddressSanitizer: stack-use-after-return on address 0x7fb71a561020 at pc 0x0000004f78e1 bp 0x7ffc299184c0 sp 0x7ffc299184b8
    READ of size 4 at 0x7fb71a561020 thread T0
        #0 0x4f78e0 in g(int*) /home/lace/src/uar.cpp:7:10
        #1 0x4f790b in main /home/lace/src/uar.cpp:10:10
        #2 0x7fb71dbde1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
        #3 0x41c41d in _start (/home/lace/src/uar+0x41c41d)
    
    Address 0x7fb71a561020 is located in stack of thread T0 at offset 32 in frame
        #0 0x4f771f in f() /home/lace/src/uar.cpp:1
    
      This frame has 1 object(s):
        [32, 36) 'i' (line 2) <== Memory access at offset 32 is inside this variable
    HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
          (longjmp and C++ exceptions *are* supported)
    SUMMARY: AddressSanitizer: stack-use-after-return /home/lace/src/uar.cpp:7:10 in g(int*)
    ...
    $ clang++ -o uar uar.cpp -Wall -g
    $ valgrind ./uar
    ...
    (nothing found by Valgrind)
    

    UBSAN: Undefined behavior

    UndefinedBehaviorSanitizer protects code from computations that are forbidden by the language standard. (See the UndefinedBehaviorSanitizer documentation.) For performance reasons, some undefined computations might not be trapped at runtime, but nobody guarantees anything about the program if they are included. Most commonly, such numeric expressions just compute an unexpected result. UndefinedBehaviorSanitizer can detect and report such operations.

    UndefinedBehaviorSanitizer can be used together with the most common Sanitizer, the AddressSanitizer:

    $ cat >undefined.cpp <<'EOF'
    int main(int argc, char **argv) {
      return 0x7fffffff + argc;
    }
    EOF
    $ clang++ -o undefined undefined.cpp -Wall -g -fsanitize=undefined
    $ export UBSAN_OPTIONS=print_stacktrace=1
    $ ./undefined
    undefined.cpp:2:21: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
        #0 0x429269 in main /tmp/undefined.cpp:2:21
        #1 0x7f1212a3e1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #2 0x40345d in _start (/tmp/undefined+0x40345d)
    
    SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior undefined.cpp:2:21 in
    $ valgrind ./undefined
    ...
    (nothing found by Valgrind)
    

    Note: You can try out this code and view its output in Compiler Explorer.

    My personal preference is to abort the program at the first such occurrence, because otherwise it is difficult to find the bug. I therefore use -fno-sanitize-recover=all. I also prefer to extend the UndefinedBehaviorSanitizer coverage a bit by including: -fsanitize=float-divide-by-zero -fsanitize=float-cast-overflow.

    LSAN: Memory leaks

    LeakSanitizer reports allocated memory that has not been freed before the program finished. (See the LeakSanitizer documentation.) Such behavior is not necessarily a bug. But freeing all allocated memory makes it easier, for example, to catch real, unexpected memory leaks:

    $ cat >leak.cpp <<'EOF'
    #include <stdlib.h>
    int main() {
      void *p = malloc(10);
      return p == nullptr;
    }
    EOF
    $ clang++ -o leak leak.cpp -Wall -g -fsanitize=address
    $ ./leak
    =================================================================
    ==188539==ERROR: LeakSanitizer: detected memory leaks
    
    Direct leak of 10 byte(s) in 1 object(s) allocated from:
        #0 0x4bfcdf in malloc (/tmp/leak+0x4bfcdf)
        #1 0x4f7728 in main /tmp/leak.cpp:3:13
        #2 0x7fd5a7a781e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    
    SUMMARY: AddressSanitizer: 10 byte(s) leaked in 1 allocation(s).
    $ clang++ -o leak leak.cpp -Wall -g
    $ valgrind --leak-check=full ./leak
    ...
    ==188524== 10 bytes in 1 blocks are definitely lost in loss record 1 of 1
    ==188524==    at 0x4839809: malloc (vg_replace_malloc.c:307)
    ==188524==    by 0x401148: main (leak.cpp:3)
    ...
    

    Note: You can try out this code and view its output in Compiler Explorer.

    LSAN: Memory leaks with specific libraries (glib2)

    Some frameworks have custom memory allocators that prevent LeakSanitizer from doing its job. The following example uses such a framework, glib2 (not glibc). Other libraries may have other runtime or compile-time options. Output from LeakSanitizer and Valgrind follows:

    $ cat >gc.c <<'EOF'
    #include <glib.h>
    int main(void) {
        GHashTable *ht = g_hash_table_new(g_str_hash, g_str_equal);
        g_hash_table_insert(ht, "foo", "bar");
    //    g_hash_table_destroy(ht); // leak through glib2
        g_malloc(100); // direct leak
        return 0;
    }
    EOF
    $ clang -o gc gc.c -Wall -g $(pkg-config --cflags --libs glib-2.0) -fsanitize=address
    $ ./gc
    =================================================================
    ==233215==ERROR: LeakSanitizer: detected memory leaks
    
    Direct leak of 100 byte(s) in 1 object(s) allocated from:
        #0 0x4bfd2f in malloc (/tmp/gc+0x4bfd2f)
        #1 0x7f1fcf12b908 in g_malloc (/lib64/libglib-2.0.so.0+0x5b908)
        #2 0x7f1fced961e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    
    SUMMARY: AddressSanitizer: 100 byte(s) leaked in 1 allocation(s).
    $ clang -o gc gc.c -Wall -g $(pkg-config --cflags --libs glib-2.0)
    $ valgrind --leak-check=full ./gc
    ...
    ==233250== 100 bytes in 1 blocks are definitely lost in loss record 8 of 11
    ==233250==    at 0x4839809: malloc (vg_replace_malloc.c:307)
    ==233250==    by 0x48DF908: g_malloc (in /usr/lib64/libglib-2.0.so.0.6600.3)
    ==233250==    by 0x4011C5: main (gc.c:6)
    ==233250==
    ==233250== 256 (96 direct, 160 indirect) bytes in 1 blocks are definitely lost in loss record 9 of 11
    ==233250==    at 0x4839809: malloc (vg_replace_malloc.c:307)
    ==233250==    by 0x48DF908: g_malloc (in /usr/lib64/libglib-2.0.so.0.6600.3)
    ==233250==    by 0x48F71C1: g_slice_alloc (in /usr/lib64/libglib-2.0.so.0.6600.3)
    ==233250==    by 0x48C5A51: g_hash_table_new_full (in /usr/lib64/libglib-2.0.so.0.6600.3)
    ==233250==    by 0x401197: main (gc.c:3)
    ...
    

    The leaked hashtable is not reported by LeakSanitizer, whereas it is reported by Valgrind. That's because glib2 specifically detects Valgrind and, in Valgrind's presence, turns off its custom memory allocator (g_slice). One can force glib2 to be debugging-friendly even with LeakSanitizer, however:

    $ clang -o gc gc.c -Wall -g $(pkg-config --cflags --libs glib-2.0) -fsanitize=address
    # otherwise the backtraces would have only 2 entries:
    $ export ASAN_OPTIONS=fast_unwind_on_malloc=0
    # Show all glib2 memory leaks:
    $ export G_SLICE=always-malloc G_DEBUG=gc-friendly
    $ ./gc
    =================================================================
    ==233921==ERROR: LeakSanitizer: detected memory leaks
    
    Direct leak of 100 byte(s) in 1 object(s) allocated from:
        #0 0x4bfd2f in malloc (/tmp/gc+0x4bfd2f)
        #1 0x7f2a7c302908 in g_malloc ../glib/gmem.c:106:13
        #2 0x4f4b35 in main /tmp/gc.c:6:5
        #3 0x7f2a7bf6d1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #4 0x41c46d in _start (/tmp/gc+0x41c46d)
    
    Direct leak of 96 byte(s) in 1 object(s) allocated from:
        #0 0x4bfd2f in malloc (/tmp/gc+0x4bfd2f)
        #1 0x7f2a7c302908 in g_malloc ../glib/gmem.c:106:13
        #2 0x7f2a7c31a1c1 in g_slice_alloc ../glib/gslice.c:1069:11
        #3 0x7f2a7c2e8a51 in g_hash_table_new_full ../glib/ghash.c:1072:16
        #4 0x4f4b07 in main /tmp/gc.c:3:22
        #5 0x7f2a7bf6d1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #6 0x41c46d in _start (/tmp/gc+0x41c46d)
    
    Indirect leak of 32 byte(s) in 1 object(s) allocated from:
        #0 0x4bfd2f in malloc (/tmp/gc+0x4bfd2f)
        #1 0x7f2a7c302908 in g_malloc ../glib/gmem.c:106:13
        #2 0x7f2a7c317ce1  ../glib/gstrfuncs.c:392:17
        #3 0x7f2a7c317ce1 in g_memdup ../glib/gstrfuncs.c:385:1
        #4 0x7f2a7c2e8b65 in g_hash_table_ensure_keyval_fits ../glib/ghash.c:974:36
        #5 0x7f2a7c2e8b65 in g_hash_table_insert_node ../glib/ghash.c:1327:3
        #6 0x7f2a7c2e930f in g_hash_table_insert_internal ../glib/ghash.c:1601:10
        #7 0x7f2a7c2e930f in g_hash_table_insert ../glib/ghash.c:1630:10
        #8 0x4f4b28 in main /tmp/gc.c:4:5
        #9 0x7f2a7bf6d1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #10 0x41c46d in _start (/tmp/gc+0x41c46d)
    
    Indirect leak of 32 byte(s) in 1 object(s) allocated from:
        #0 0x4bfed7 in calloc (/tmp/gc+0x4bfed7)
        #1 0x7f2a7c302e20 in g_malloc0 ../glib/gmem.c:136:13
        #2 0x7f2a7c2e50ef in g_hash_table_setup_storage ../glib/ghash.c:592:24
        #3 0x7f2a7c2e8a90 in g_hash_table_new_full ../glib/ghash.c:1084:3
        #4 0x4f4b07 in main /tmp/gc.c:3:22
        #5 0x7f2a7bf6d1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #6 0x41c46d in _start (/tmp/gc+0x41c46d)
    
    Indirect leak of 32 byte(s) in 1 object(s) allocated from:
        #0 0x4c0098 in realloc (/tmp/gc+0x4c0098)
        #1 0x7f2a7c302f5f in g_realloc ../glib/gmem.c:171:16
        #2 0x7f2a7c2e50da in g_hash_table_realloc_key_or_value_array ../glib/ghash.c:380:10
        #3 0x7f2a7c2e50da in g_hash_table_setup_storage ../glib/ghash.c:590:24
        #4 0x7f2a7c2e8a90 in g_hash_table_new_full ../glib/ghash.c:1084:3
        #5 0x4f4b07 in main /tmp/gc.c:3:22
        #6 0x7f2a7bf6d1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #7 0x41c46d in _start (/tmp/gc+0x41c46d)
    
    SUMMARY: AddressSanitizer: 292 byte(s) leaked in 5 allocation(s).
    

    TSAN: Data races

    ThreadSanitizer reports data races where multiple threads access data without thread-race protection. (See the ThreadSanitizer documentation.) An example follows:

    $ cat >tiny.cpp <<'EOF'
    #include <thread>
    
    static volatile bool flip1{false};
    static volatile bool flip2{false};
    
    int main() {
      std::thread t([&]() {
        while (!flip1);
        flip2 = true;
      });
      flip1 = true;
      while (!flip2);
      t.join();
    }
    EOF
    $ clang++ -o tiny tiny.cpp -Wall -g -pthread -fsanitize=thread
    $ ./tiny
    ==================
    WARNING: ThreadSanitizer: data race (pid=4057433)
      Write of size 1 at 0x000000fb4b09 by thread T1:
        #0 main::$_0::operator()() const /tmp/tiny.cpp:9:11 (tiny+0x4cfc98)
        #1 void std::__invoke_impl<void, main::$_0>(std::__invoke_other, main::$_0&&) /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:60:14 (tiny+0x4cfc30)
        #2 std::__invoke_result<main::$_0>::type std::__invoke<main::$_0>(main::$_0&&) /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:95:14 (tiny+0x4cfb40)
        #3 void std::thread::_Invoker<std::tuple<main::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/thread:264:13 (tiny+0x4cfae8)
        #4 std::thread::_Invoker<std::tuple<main::$_0> >::operator()() /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/thread:271:11 (tiny+0x4cfa88)
        #5 std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_0> > >::_M_run() /usr/lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/thread:215:13 (tiny+0x4cf97f)
        #6 execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:80:18 (libstdc++.so.6+0xd65f3)
    
      Previous read of size 1 at 0x000000fb4b09 by main thread:
        #0 main /tmp/tiny.cpp:12:11 (tiny+0x4cf51f)
    
      Location is global 'flip2' of size 1 at 0x000000fb4b09 (tiny+0x000000fb4b09)
    
      Thread T1 (tid=4057435, running) created by main thread at:
        #0 pthread_create <null> (tiny+0x488b7d)
        #1 <null> /usr/src/debug/gcc-10.2.1-9.fc33.x86_64/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/x86_64-redhat-linux/bits/gthr-default.h:663:35 (libstdc++.so.6+0xd6898)
        #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ../../../../../libstdc++-v3/src/c++11/thread.cc:135:37 (libstdc++.so.6+0xd6898)
        #3 main /tmp/tiny.cpp:7:15 (tiny+0x4cf4f4)
    
    SUMMARY: ThreadSanitizer: data race /tmp/tiny.cpp:9:11 in main::$_0::operator()() const
    ==================
    ThreadSanitizer: reported 1 warnings
    $ clang++ -o tiny tiny.cpp -Wall -g -pthread
    $ valgrind --tool=helgrind ./tiny
    ...
    ==4057510== ----------------------------------------------------------------
    ==4057510==
    ==4057510== Possible data race during write of size 1 at 0x40406D by thread #1
    ==4057510== Locks held: none
    ==4057510==    at 0x4011DC: main (tiny.cpp:11)
    ==4057510==
    ==4057510== This conflicts with a previous read of size 1 by thread #2
    ==4057510== Locks held: none
    ==4057510==    at 0x4015F8: main::$_0::operator()() const (tiny.cpp:8)
    ==4057510==    by 0x4015DC: void std::__invoke_impl<void, main::$_0>(std::__invoke_other, main::$_0&&) (invoke.h:60)
    ==4057510==    by 0x40156C: std::__invoke_result<main::$_0>::type std::__invoke<main::$_0>(main::$_0&&) (invoke.h:95)
    ==4057510==    by 0x401544: void std::thread::_Invoker<std::tuple<main::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (thread:264)
    ==4057510==    by 0x401514: std::thread::_Invoker<std::tuple<main::$_0> >::operator()() (thread:271)
    ==4057510==    by 0x40148D: std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_0> > >::_M_run() (thread:215)
    ==4057510==    by 0x49575F3: execute_native_thread_routine (thread.cc:80)
    ==4057510==    by 0x4840737: mythread_wrapper (hg_intercepts.c:387)
    ==4057510==  Address 0x40406d is 0 bytes inside data symbol "_ZL5flip1"
    ==4057510==
    ==4057510== ----------------------------------------------------------------
    ==4057510==
    ==4057510== Possible data race during read of size 1 at 0x40406D by thread #2
    ==4057510== Locks held: none
    ==4057510==    at 0x4015F8: main::$_0::operator()() const (tiny.cpp:8)
    ==4057510==    by 0x4015DC: void std::__invoke_impl<void, main::$_0>(std::__invoke_other, main::$_0&&) (invoke.h:60)
    ==4057510==    by 0x40156C: std::__invoke_result<main::$_0>::type std::__invoke<main::$_0>(main::$_0&&) (invoke.h:95)
    ==4057510==    by 0x401544: void std::thread::_Invoker<std::tuple<main::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (thread:264)
    ==4057510==    by 0x401514: std::thread::_Invoker<std::tuple<main::$_0> >::operator()() (thread:271)
    ==4057510==    by 0x40148D: std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_0> > >::_M_run() (thread:215)
    ==4057510==    by 0x49575F3: execute_native_thread_routine (thread.cc:80)
    ==4057510==    by 0x4840737: mythread_wrapper (hg_intercepts.c:387)
    ==4057510==    by 0x4BD33F8: start_thread (pthread_create.c:463)
    ==4057510==    by 0x4CED902: clone (clone.S:95)
    ==4057510==
    ==4057510== This conflicts with a previous write of size 1 by thread #1
    ==4057510== Locks held: none
    ==4057510==    at 0x4011DC: main (tiny.cpp:11)
    ==4057510==  Address 0x40406d is 0 bytes inside data symbol "_ZL5flip1"
    ==4057510==
    ==4057510== ----------------------------------------------------------------
    ==4057510==
    ==4057510== Possible data race during write of size 1 at 0x40406E by thread #2
    ==4057510== Locks held: none
    ==4057510==    at 0x401613: main::$_0::operator()() const (tiny.cpp:9)
    ==4057510==    by 0x4015DC: void std::__invoke_impl<void, main::$_0>(std::__invoke_other, main::$_0&&) (invoke.h:60)
    ==4057510==    by 0x40156C: std::__invoke_result<main::$_0>::type std::__invoke<main::$_0>(main::$_0&&) (invoke.h:95)
    ==4057510==    by 0x401544: void std::thread::_Invoker<std::tuple<main::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (thread:264)
    ==4057510==    by 0x401514: std::thread::_Invoker<std::tuple<main::$_0> >::operator()() (thread:271)
    ==4057510==    by 0x40148D: std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_0> > >::_M_run() (thread:215)
    ==4057510==    by 0x49575F3: execute_native_thread_routine (thread.cc:80)
    ==4057510==    by 0x4840737: mythread_wrapper (hg_intercepts.c:387)
    ==4057510==    by 0x4BD33F8: start_thread (pthread_create.c:463)
    ==4057510==    by 0x4CED902: clone (clone.S:95)
    ==4057510==
    ==4057510== This conflicts with a previous read of size 1 by thread #1
    ==4057510== Locks held: none
    ==4057510==    at 0x4011E4: main (tiny.cpp:12)
    ==4057510==  Address 0x40406e is 0 bytes inside data symbol "_ZL5flip2"
    ==4057510==
    ==4057510== ----------------------------------------------------------------
    ==4057510==
    ==4057510== Possible data race during read of size 1 at 0x40406E by thread #1
    ==4057510== Locks held: none
    ==4057510==    at 0x4011E4: main (tiny.cpp:12)
    ==4057510==
    ==4057510== This conflicts with a previous write of size 1 by thread #2
    ==4057510== Locks held: none
    ==4057510==    at 0x401613: main::$_0::operator()() const (tiny.cpp:9)
    ==4057510==    by 0x4015DC: void std::__invoke_impl<void, main::$_0>(std::__invoke_other, main::$_0&&) (invoke.h:60)
    ==4057510==    by 0x40156C: std::__invoke_result<main::$_0>::type std::__invoke<main::$_0>(main::$_0&&) (invoke.h:95)
    ==4057510==    by 0x401544: void std::thread::_Invoker<std::tuple<main::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (thread:264)
    ==4057510==    by 0x401514: std::thread::_Invoker<std::tuple<main::$_0> >::operator()() (thread:271)
    ==4057510==    by 0x40148D: std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_0> > >::_M_run() (thread:215)
    ==4057510==    by 0x49575F3: execute_native_thread_routine (thread.cc:80)
    ==4057510==    by 0x4840737: mythread_wrapper (hg_intercepts.c:387)
    ==4057510==  Address 0x40406e is 0 bytes inside data symbol "_ZL5flip2"
    ...
    

    Note: You can try out this code and view its output in Compiler Explorer.

    Recompiling libraries

    AddressSanitizer automatically processes all calls to glibc. This is not the case for other system or user libraries. For AddressSanitizer to function best, one should also recompile such libraries with -fsanitize=address. This is not required with Valgrind.

    The following bug in the libuser.c library is still caught by AddressSanitizer thanks to the glibc interceptor, even though the library is not compiled with AddressSanitizer:

    $ cat >library.c <<'EOF'
    #include <string.h>
    void library(char *s) {
      strcpy(s,"string");
    }
    EOF
    $ cat >libuser.c <<'EOF'
    #include <stdlib.h>
    void library(char *s);
    int main(void) {
      char *s = malloc(1);
      library(s);
      free(s);
    }
    EOF
    $ clang -o library.so library.c -Wall -g -shared -fPIC
    $ clang -o libuser libuser.c -Wall -g ./library.so -fsanitize=address
    $ ./libuser
    =================================================================
    ==128657==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000011 at pc 0x000000484a6d bp 0x7fff13a4ace0 sp 0x7fff13a4a490
    WRITE of size 7 at 0x602000000011 thread T0
        #0 0x484a6c in __interceptor_strcpy.part.0 (/tmp/libuser+0x484a6c)
        #1 0x7fae9f53512b in library /tmp/library.c:3:3
        #2 0x4f4abe in main /tmp/libuser.c:5:3
        #3 0x7fae9f1be1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #4 0x41c42d in _start (/tmp/libuser+0x41c42d)
    
    0x602000000011 is located 0 bytes to the right of 1-byte region [0x602000000010,0x602000000011)
    allocated by thread T0 here:
        #0 0x4bfcef in malloc (/tmp/libuser+0x4bfcef)
        #1 0x4f4ab1 in main /tmp/libuser.c:4:13
        #2 0x7fae9f1be1e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
    
    SUMMARY: AddressSanitizer: heap-buffer-overflow (/tmp/libuser+0x484a6c) in __interceptor_strcpy.part.0
    ...
    

    In the following case, AddressSanitizer misses the memory corruption when the library has not been recompiled with AddressSanitizer:

    $ cat >library.c <<'EOF'
    void library(char *s) {
      const char *cs = "string";
      while (*cs)
        *s++ = *cs++;
      *s = 0;
    }
    EOF
    $ cat >libuser.c <<'EOF'
    #include <stdlib.h>
    void library(char *s);
    int main(void) {
      char *s = malloc(1);
      library(s);
      free(s);
    }
    EOF
    $ clang -o library.so library.c -Wall -g -shared -fPIC
    $ clang -o libuser libuser.c -Wall -g ./library.so -fsanitize=address
    $ ./libuser
    (nothing found by AddressSanitizer)
    

    Valgrind can find the bug without any recompilation:

    $ clang -o library.so library.c -Wall -g -shared -fPIC; clang -o libuser libuser.c -Wall -g ./library.so; valgrind ./libuser
    ...
    ==128708== Invalid write of size 1
    ==128708==    at 0x4849146: library (library.c:4)
    ==128708==    by 0x40116E: main (libuser.c:5)
    ==128708==  Address 0x4a57041 is 0 bytes after a block of size 1 alloc'd
    ==128708==    at 0x4839809: malloc (vg_replace_malloc.c:307)
    ==128708==    by 0x401161: main (libuser.c:4)
    ...
    

    AddressSanitizer can also find the bug, as long as we recompile the library with AddressSanitizer:

    $ clang -o library.so library.c -Wall -g -shared -fPIC -fsanitize=address
    $ clang -o libuser libuser.c -Wall -g ./library.so -fsanitize=address
    $ ./libuser
    =================================================================
    ==128719==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000011 at pc 0x7f7e4e68b269 bp 0x7ffc40c0dc30 sp 0x7ffc40c0dc28
    WRITE of size 1 at 0x602000000011 thread T0
        #0 0x7f7e4e68b268 in library /tmp/library.c:4:10
        #1 0x4f4abe in main /tmp/libuser.c:5:3
        #2 0x7f7e4e3141e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
        #3 0x41c42d in _start (/tmp/libuser+0x41c42d)
    
    0x602000000011 is located 0 bytes to the right of 1-byte region [0x602000000010,0x602000000011)
    allocated by thread T0 here:
        #0 0x4bfcef in malloc (/tmp/libuser+0x4bfcef)
        #1 0x4f4ab1 in main /tmp/libuser.c:4:13
        #2 0x7f7e4e3141e1 in __libc_start_main /usr/src/debug/glibc-2.32-20-g5c36293f06/csu/../csu/libc-start.c:314:16
    
    SUMMARY: AddressSanitizer: heap-buffer-overflow /tmp/library.c:4:10 in library
    ...
    

    Sanitizers' interaction with _FORTIFY_SOURCE

    By default, rpmbuild uses the -Wp,-D_FORTIFY_SOURCE=2 option, which implements its own kind of memory access sanity checking. Unfortunately, it disables some of the memory checks done by AddressSanitizer. This problem might be fixed in the future. Currently, to prepare for checking by Sanitizers, just disable _FORTIFY_SOURCE using -Wp,-U_FORTIFY_SOURCE (which is a more universal form of the simple -D_FORTIFY_SOURCE=0):

    $ cat >strcpyfrom.spec <<'EOF'
    Summary: strcpyfrom
    Name: strcpyfrom
    Version: 1
    Release: 1
    License: GPLv3+
    %description
    %build
    cat >strcpyfrom.c <<'EOH'
    #include <stdlib.h>
    #include <string.h>
    int main(void) {
      char *s = malloc(1);
      char d[0x1000];
      strcpy(d, s);
      return 0;
    }
    EOH
    gcc -o strcpyfrom strcpyfrom.c $RPM_OPT_FLAGS -fsanitize=address
    echo no error caught:
    ./strcpyfrom
    gcc -o strcpyfrom strcpyfrom.c $RPM_OPT_FLAGS -fsanitize=address -Wp,-U_FORTIFY_SOURCE
    echo error caught:
    ./strcpyfrom
    EOF
    $ rpmbuild -bb strcpyfrom.spec 
    Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.KTLr7c
    + umask 022
    + cd src/rpm/BUILD
    + cat
    + gcc -o strcpyfrom strcpyfrom.c -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fsanitize=address
    + echo no error caught:
    no error caught:
    + ./strcpyfrom
    + gcc -o strcpyfrom strcpyfrom.c -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fsanitize=address -Wp,-U_FORTIFY_SOURCE
    annobin: strcpyfrom.c: Warning: -D_FORTIFY_SOURCE defined as 0
    + echo error caught:
    error caught:
    + ./strcpyfrom
    =================================================================
    ==412157==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000011 at pc 0x7fe75d8b2075 bp 0x7ffccf5dd1e0 sp 0x7ffccf5dc990
    READ of size 2 at 0x602000000011 thread T0
        #0 0x7fe75d8b2074  (/lib64/libasan.so.6+0x52074)
        #1 0x4011be in main strcpyfrom.c:6
        #2 0x7fe75d6bd1e1 in __libc_start_main ../csu/libc-start.c:314
        #3 0x40127d in _start (strcpyfrom+0x40127d)
    
    0x602000000011 is located 0 bytes to the right of 1-byte region [0x602000000010,0x602000000011)
    allocated by thread T0 here:
        #0 0x7fe75d90b3cf in __interceptor_malloc (/lib64/libasan.so.6+0xab3cf)
        #1 0x4011b2 in main strcpyfrom.c:4
        #2 0x40200f  (strcpyfrom+0x40200f)
    
    SUMMARY: AddressSanitizer: heap-buffer-overflow (/lib64/libasan.so.6+0x52074) 
    ...
    error: Bad exit status from /var/tmp/rpm-tmp.KTLr7c (%build)
    
    RPM build errors:
        Bad exit status from /var/tmp/rpm-tmp.KTLr7c (%build)
    

    Conclusion

    If you are accustomed to Valgrind, give AddressSanitizer a try—just add the -fsanitize=address compilation plus linking parameter (that is, to all of the CFLAGS, CXXFLAGS, and LDFLAGS) as the first try. If you find it great, check the "A quick Sanitizers how-to" section for fine-tuning the experience.

Last updated: May 7, 2021