Improvements in memstomp

memstomp is an interposition library to detect cases where applications may exhibit undefined behaviour when calling routines within the C library (glibc).

The first version of memstomp was focused on detecting cases where source and destination memory regions passed to C library routines such as memcpy overlapped in ways not allowed by the ISO C standard.  Matt Newsome's blog post shows how to utilize memstomp to find that class of bugs.

For many years, GCC has attempted to eliminate unnecessary NULL pointer checks.  GCC 4.9 (now available in Developer Toolset 3) improves on this optimization by utilizing guarantees made by the ISO C standards about the validity of certain pointer arguments passed to library functions.  In particular the standard mandates that certain arguments to certain functions must have a non-NULL value.   For example, the source and destination pointers passed to memcpy must not be NULL, even if the count of bytes to copy is zero. So for example:

int copy (int* dest, int* src, size_t nbytes) {
  memcopy (dest, src, nbytes);
  if (src != NULL)
    return *src;
  return 0;
}

Invokes undefined behavior at runtime if "src" or "dest" is ever NULL. Thus, GCC can validly transform that function into:

int copy (int* dest, int* src, size_t nbytes) {
  memcpy (dest, src, nbytes);
  return *src;
}

During the development and early deployment of GCC 4.9, we encountered several instances where code violated the ISO C standards in this way. As a result we enhanced memstomp to detect these cases. As a trivial example consider:

#include <stdlib.h>
#include <string.h>

char *src = NULL;
size_t size = 0;
char *dest;
main() {
    dest = malloc (size);
    memcpy (dest, src, size);
}
[law ~]# gcc foo.c
[law ~]# memstomp -q ./a.out
memmove NULL pointer a.out(18279)
    /lib64/libmemstomp.so(memmove+0x7c) [0x7f0309539c3c]
    ./a.out() [0x4005d6]
    /lib64/libc.so.6(__libc_start_main+0xf4) [0x36fb021d64]
    ./a.out() [0x4004d8]

As desired, memstomp identified that this code passed a NULL pointer to memcpy and issued an error. What's particularly useful about memstomp is that it can be used on existing applications without recompiling them and that it's significantly faster than valgrind.

Just for fun, on my Fedora 21-Alpha machine:

[law ~]# ( memstomp -q vi foo.c > /dev/null ) > & ERR
:q
[law ~]# cat ERR
memmove NULL pointer vi(18283)
    /lib64/libmemstomp.so(memmove+0x7c) [0x7fb5caf2ac3c]
    vi() [0x433225]
    vi() [0x4337e6]
    vi() [0x433f00]
    vi() [0x42608b]
    vi() [0x4074e5]
    vi() [0x4039d4]
    /lib64/libc.so.6(__libc_start_main+0xf4) [0x36fb021d64]
    vi() [0x404bd8]

So clearly there's a bug in VIM that I need to get filed.

This version of memstomp is part of Fedora 21 and not in RHEL at this time.  As usual, we'd welcome your feedback!