Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Find and fix nasty memory bugs with Developer Toolset's memstomp tool

March 4, 2014
Matt Newsome
Related topics:
Developer ToolsLinux
Related products:
Developer ToolsRed Hat Enterprise Linux

Share:

    One of the really useful tools provided by Red Hat Developer Toolset v2.x is "memstomp", which helps you identify a particularly nasty class of bug in applications built (directly or indirectly) from C/C++ code so you can then fix them before your customers experience problems. In this brief article, I'll explain the background for the tool, how to get it, how to use it yourself and briefly how it works.

    Background

    The memcpy() routine in the standard C library has a long and storied history, stretching back through POSIX and ISO standards into the early history of the modern computing era. It's a simple little routine that does one simple task required in a variety of scenarios - it copies a specified number of sequential bytes from one location in memory to another. You tell it where to start copying from, how many bytes to copy and where to copy to. Simple.

    One thing that has been very well documented, but not always well understood by coders, however, has been the behaviour of memcpy() when the source and destination overlap. Take, for example, this little piece of code:

    #include <string.h>
    main() {
     char src[] = "text-to-copy";
     char *dest = src + 2;
     memcpy((void*)dest,(void*)src,3);
     return 0;
    }

    When we build and run it, we may or may not see a crash like this:

    [mattn@rhel]$ gcc memcpy_bad.c -o memcpy_bad
    [mattn@rhel]$ ./memcpy_bad
    Segmentation fault (core dumped)

    That's because the source and destination memory overlap, the behaviour of which is undefined in the various standards.

    Note that we may not see a crash, as it depends. It depends on whether particular optimizations came into play, which often depends on which version of your operating system you're running and which hardware you're running it on. Why optimize? Well, as mentioned, memcpy() is a critical routine used by many applications even without knowing it. Pretty much everyone wants memcpy() to go as fast as it can to speed overall performance, so glibc's upstream developers, like many others, optimize memcpy() heavily to squeeze every inch of performance out of the system for you.

    But you may also not see the crash because, in many cases, most of the time your code runs perfectly fine due to non-overlapping source and destination arguments, but once in a blue moon it happens. If you're unlucky, your testing may never find such use cases, but - and here's the important part - your customers could. At that point, you'll need to do a lot of work in a short space of time to reproduce the failure (which may itself be difficult), identify the cause of the problem, fix it, issue the update and so on. That can get pretty expensive.

    Additionally, your application code may be fine - no memcpy() calls with overlapping arguments at all - but how about that third party library you're using - can you be sure that calls memcpy() as intended? Or how about the code generated to instantiate and copy your C++ ? Does it use memcpy() correctly? Maybe there's some software you know your customers use regularly with your own, and which they rely upon - are they checking their code?

    Enter memstomp

    Fixing the example I gave above is fairly straightforward, and you may find static analysis tools are able to spot this one and warn you. But most cases will only be seen at runtime. Wouldn't it be great, then, if there was a tool that can spot this happening at runtime, even if it didn't trigger a crash? Well, you're in luck.

    In fact there are two tools you could apply to this problem. Valgrind is documented elsewhere and provides an excellent tool for this kind of analysis at the cost of some runtime performance loss. Let's assume we want to be as minimally invasive to runtime as possible. The tool you want in this case is memstomp and it's available as part of Red Hat Developer Toolset and Red Hat Enterprise Linux 7 Beta.

    memstomp does one thing really well - it intercepts calls to memcpy() and checks the source and destination to see if they overlap. If they do, memstomp shouts loudly about it. Either way, it then lets the real memcpy() do its thing so your application can go about its normal business. This way, any calls to memcpy() will be highlighted, but otherwise normal application execution will occur, with negligible impact to performance. Here's what memcpy says when executing our errant program above:

    [mattn@rhel]$ memstomp ./memcpy_bad
    memstomp: 0.1.4 successfully initialized for process memcpy_bad (pid 28195).
    memcpy(dest=0x7fff103cc1c2, src=0x7fff103cc1c0, bytes=3) overlap for memcpy_bad(28195)
     /opt/rh/devtoolset-2/root/usr/lib64/libmemstomp.so(+0x10a7) [0x7fdb4760a0a7]
     ./memcpy_bad(main+0x45) [0x400795]
     /lib64/libc.so.6(__libc_start_main+0xfd) [0x357fe1ed1d]
     ./memcpy_bad() [0x400669]

    Note that memstomp doesn't just tell us that a call was made, but also where, so we can dig into the code and find the corresponding source line. And it doesn't just find errant memcpy() calls, but similar misuse of [w]memcpy, str[n]cat, wcs[n]cat, str[n]cpy, wcs[n]cpy, [w]mempcpy, memccpy and stp[n]cpy, too.

    Under the hood - LD_PRELOAD

    How does it work? Well, behind the scenes, memstomp is simply a script which invokes your specified string with LD_PRELOAD=libmemstomp.so prefixed. The LD_PRELOAD feature itself is well documented online, but essentially provides the libmemstomp.so symbols to your application in preference to the ones in your system or other user libraries. By ensuring it provides its own version of memcpy() to your application before anything else and see the arguments you are passing, memstomp can run some extra diagnostics before passing execution straight through to the normal memcpy().

    Fixing the problem - various options

    So, having identified the problem, let's now fix the problem. In this case we'll just change our call to memcpy() to a call to memmove(), which doesn't mind whether the source and destination overlaps. This comes with a performance penalty, so we might rewrite our memcpy() call to use non-overlapping arguments in the future. We'll continue with memmove() for now, though:

    #include <string.h>
    main() {
     char src[] = "text-to-copy";
     char *dest = src + 2;
     memmove((void*)dest,(void*)src,3);
     return 0;
    }

    We recompile:

    [mattn@rhel]$ gcc memcpy_ok.c -o memcpy_ok
    [mattn@rhel]$ ./memcpy_ok
    [mattn@rhel]$

    and for good measure, re-test with memstomp:

    [mattn@rhel]$ memstomp ./memcpy_ok
    memstomp: 0.1.4 successfully initialized for process memcpy_ok (pid 28578).
    [mattn@rhel]$

    All clean, we fixed it, great. So what if we wanted to change all our calls to memcpy() into memmove() calls - is there a shortcut so we can just test it maybe? Actually there is - using the LD_PRELOAD mechanism, you can write your own version of memcpy() that intercepts any calls in your application (or any libraries it uses) and instead calls memmove(). It's a neat way to at least quickly re-test your application and, because memstomp is open source, you can take a look at its own source code to see exactly how it works, reusing that in your own tool, subject to memstomp()'s own license terms.

    If we want performance as well as standards-compliant code that avoids the risk of embarrassing crashes at runtime, we'll need to re-write our application's use of memcpy():

    #include <string.h>
    main() {
     char src[] = "text-to-copy";
     char *dest = src + 5; // no longer overlaps
     memcpy((void*)dest,(void*)src,3);
     return 0;
    }
    [mattn@rhel]$ gcc memcpy_fixed.c -o memcpy_fixed
    [mattn@rhel]$ memstomp ./memcpy_fixed
    memstomp: 0.1.4 successfully initialized for process memcpy_fixed (pid 31322).
    [mattn@rhel]$

    Where's memstomp?

    So, where can you find memstomp? Well, it's currently available for Red Hat Enterprise Linux 5 and 6 users via Red Hat Developer Toolset v2 or later, and is part of Red Hat Enterprise Linux 7 Beta. The above examples are using Red Hat Enterprise Linux 7 Beta so when using memstomp from Red Hat Developer Toolset, remember to invoke the utility as follows (consistent with any software collection):

    scl enable devtoolset-2 'memstomp ./your_application'

    As usual, if you have any thoughts or feedback on Red Hat's developer tools, we'd love to hear from you - thanks!

    Last updated: February 26, 2024

    Recent Posts

    • Storage considerations for OpenShift Virtualization

    • Upgrade from OpenShift Service Mesh 2.6 to 3.0 with Kiali

    • EE Builder with Ansible Automation Platform on OpenShift

    • How to debug confidential containers securely

    • Announcing self-service access to Red Hat Enterprise Linux for Business Developers

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue