One of the really useful tools provided by Red Hat Developer Toolset v2.x is "memstomp", which helps you identify a particularly nasty class of bug in applications built (directly or indirectly) from C/C++ code so you can then fix them before your customers experience problems. In this brief article, I'll explain the background for the tool, how to get it, how to use it yourself and briefly how it works.
Background
The memcpy() routine in the standard C library has a long and storied history, stretching back through POSIX and ISO standards into the early history of the modern computing era. It's a simple little routine that does one simple task required in a variety of scenarios - it copies a specified number of sequential bytes from one location in memory to another. You tell it where to start copying from, how many bytes to copy and where to copy to. Simple.
One thing that has been very well documented, but not always well understood by coders, however, has been the behaviour of memcpy() when the source and destination overlap. Take, for example, this little piece of code:
#include <string.h>
main() {
char src[] = "text-to-copy";
char *dest = src + 2;
memcpy((void*)dest,(void*)src,3);
return 0;
}
When we build and run it, we may or may not see a crash like this:
[mattn@rhel]$ gcc memcpy_bad.c -o memcpy_bad [mattn@rhel]$ ./memcpy_bad Segmentation fault (core dumped)
That's because the source and destination memory overlap, the behaviour of which is undefined in the various standards.
Note that we may not see a crash, as it depends. It depends on whether particular optimizations came into play, which often depends on which version of your operating system you're running and which hardware you're running it on. Why optimize? Well, as mentioned, memcpy() is a critical routine used by many applications even without knowing it. Pretty much everyone wants memcpy() to go as fast as it can to speed overall performance, so glibc's upstream developers, like many others, optimize memcpy() heavily to squeeze every inch of performance out of the system for you.
But you may also not see the crash because, in many cases, most of the time your code runs perfectly fine due to non-overlapping source and destination arguments, but once in a blue moon it happens. If you're unlucky, your testing may never find such use cases, but - and here's the important part - your customers could. At that point, you'll need to do a lot of work in a short space of time to reproduce the failure (which may itself be difficult), identify the cause of the problem, fix it, issue the update and so on. That can get pretty expensive.
Additionally, your application code may be fine - no memcpy() calls with overlapping arguments at all - but how about that third party library you're using - can you be sure that calls memcpy() as intended? Or how about the code generated to instantiate and copy your C++ ? Does it use memcpy() correctly? Maybe there's some software you know your customers use regularly with your own, and which they rely upon - are they checking their code?
Enter memstomp
Fixing the example I gave above is fairly straightforward, and you may find static analysis tools are able to spot this one and warn you. But most cases will only be seen at runtime. Wouldn't it be great, then, if there was a tool that can spot this happening at runtime, even if it didn't trigger a crash? Well, you're in luck.
In fact there are two tools you could apply to this problem. Valgrind is documented elsewhere and provides an excellent tool for this kind of analysis at the cost of some runtime performance loss. Let's assume we want to be as minimally invasive to runtime as possible. The tool you want in this case is memstomp and it's available as part of Red Hat Developer Toolset and Red Hat Enterprise Linux 7 Beta.
memstomp does one thing really well - it intercepts calls to memcpy() and checks the source and destination to see if they overlap. If they do, memstomp shouts loudly about it. Either way, it then lets the real memcpy() do its thing so your application can go about its normal business. This way, any calls to memcpy() will be highlighted, but otherwise normal application execution will occur, with negligible impact to performance. Here's what memcpy says when executing our errant program above:
[mattn@rhel]$ memstomp ./memcpy_bad memstomp: 0.1.4 successfully initialized for process memcpy_bad (pid 28195). memcpy(dest=0x7fff103cc1c2, src=0x7fff103cc1c0, bytes=3) overlap for memcpy_bad(28195) /opt/rh/devtoolset-2/root/usr/lib64/libmemstomp.so(+0x10a7) [0x7fdb4760a0a7] ./memcpy_bad(main+0x45) [0x400795] /lib64/libc.so.6(__libc_start_main+0xfd) [0x357fe1ed1d] ./memcpy_bad() [0x400669]
Note that memstomp doesn't just tell us that a call was made, but also where, so we can dig into the code and find the corresponding source line. And it doesn't just find errant memcpy() calls, but similar misuse of [w]memcpy, str[n]cat, wcs[n]cat, str[n]cpy, wcs[n]cpy, [w]mempcpy, memccpy and stp[n]cpy, too.
Under the hood - LD_PRELOAD
How does it work? Well, behind the scenes, memstomp is simply a script which invokes your specified string with LD_PRELOAD=libmemstomp.so prefixed. The LD_PRELOAD feature itself is well documented online, but essentially provides the libmemstomp.so symbols to your application in preference to the ones in your system or other user libraries. By ensuring it provides its own version of memcpy() to your application before anything else and see the arguments you are passing, memstomp can run some extra diagnostics before passing execution straight through to the normal memcpy().
Fixing the problem - various options
So, having identified the problem, let's now fix the problem. In this case we'll just change our call to memcpy() to a call to memmove(), which doesn't mind whether the source and destination overlaps. This comes with a performance penalty, so we might rewrite our memcpy() call to use non-overlapping arguments in the future. We'll continue with memmove() for now, though:
#include <string.h>
main() {
char src[] = "text-to-copy";
char *dest = src + 2;
memmove((void*)dest,(void*)src,3);
return 0;
}
We recompile:
[mattn@rhel]$ gcc memcpy_ok.c -o memcpy_ok [mattn@rhel]$ ./memcpy_ok [mattn@rhel]$
and for good measure, re-test with memstomp:
[mattn@rhel]$ memstomp ./memcpy_ok memstomp: 0.1.4 successfully initialized for process memcpy_ok (pid 28578). [mattn@rhel]$
All clean, we fixed it, great. So what if we wanted to change all our calls to memcpy() into memmove() calls - is there a shortcut so we can just test it maybe? Actually there is - using the LD_PRELOAD mechanism, you can write your own version of memcpy() that intercepts any calls in your application (or any libraries it uses) and instead calls memmove(). It's a neat way to at least quickly re-test your application and, because memstomp is open source, you can take a look at its own source code to see exactly how it works, reusing that in your own tool, subject to memstomp()'s own license terms.
If we want performance as well as standards-compliant code that avoids the risk of embarrassing crashes at runtime, we'll need to re-write our application's use of memcpy():
#include <string.h>
main() {
char src[] = "text-to-copy";
char *dest = src + 5; // no longer overlaps
memcpy((void*)dest,(void*)src,3);
return 0;
}
[mattn@rhel]$ gcc memcpy_fixed.c -o memcpy_fixed
[mattn@rhel]$ memstomp ./memcpy_fixed
memstomp: 0.1.4 successfully initialized for process memcpy_fixed (pid 31322).
[mattn@rhel]$
Where's memstomp?
So, where can you find memstomp? Well, it's currently available for Red Hat Enterprise Linux 5 and 6 users via Red Hat Developer Toolset v2 or later, and is part of Red Hat Enterprise Linux 7 Beta. The above examples are using Red Hat Enterprise Linux 7 Beta so when using memstomp from Red Hat Developer Toolset, remember to invoke the utility as follows (consistent with any software collection):
scl enable devtoolset-2 'memstomp ./your_application'
As usual, if you have any thoughts or feedback on Red Hat's developer tools, we'd love to hear from you - thanks!
Last updated: March 14, 2023