Memory access is one of the most basic operations in computer programs. It is also an unending source of program errors in C programs, because memory safety was never really a programming language goal in C. Memory-related issues also comprise a significant part of the top 25 security weaknesses that result in program vulnerabilities.
Memory access also plays an important role in performance, which makes memory management a prime target for performance tuning. It is natural, then, that dynamic memory management in the C runtime should have capabilities that allow fine-grained tracking and customizable actions on allocation events. These features allow users to diagnose memory issues in their programs and if necessary, override the C runtime allocator with their own to improve performance or memory utilization.
This article describes the clash between the quest for flexibility and introspection, on the one hand, and performance and security protections on the other. You'll learn why this clash ultimately led to a major change in how memory allocation (malloc) is implemented in the GNU C Library, or glibc. We'll also discuss how to adapt applications that depended on the old way of doing things, as well as the implications for future versions of Fedora and Red Hat Enterprise Linux (RHEL).
Debugging malloc in glibc
Until recently, the GNU C Library, which is the core runtime library for RHEL, provided diagnostic functionality in the form of function pointers that users could overwrite to implement their own allocation functions. These function pointers were collectively called malloc hooks. If a hook was set to a function address, glibc allocator functions would call the function instead of the internal implementations, allowing programmers to perform arbitrary actions. A programmer could even run a custom function and then, if necessary, call the glibc memory allocator function again (by momentarily setting the hook to NULL) to get the actual block of memory.
All of the malloc debugging features in glibc (i.e., mtrace, mcheck, and the MALLOC_CHECK_ environment variable) were implemented using these hooks. These debugging features, and the hooks in general, were very useful because they provided checking on a more lightweight basis than the memory checking done by full-fledged memory debugging programs such as Valgrind and sanitizers.
Malloc hooks in multi-threaded applications
As applications became increasingly multi-threaded, it was discovered that manipulating malloc hooks in such environments was fraught with risks. All of the debugging features in glibc malloc except MALLOC_CHECK_ were, and continue to be, unsafe in multi-threaded environments. Malloc hooks, the basis of the debugging features, were not the only way to override malloc, either; glibc always supported the interposition of malloc functions by preloading a shared library with those functions. Glibc itself always calls malloc functions through its procedure linkage table (PLT) so that it can invoke the interposed functions.
To make things worse, much of the debugging infrastructure was tightly integrated into system allocator functionality. This made the task of enhancing the allocator unnecessarily complex. Furthermore, there was always the possibility of corner cases inducing unexpected behavior. Finally, implementing debugging features in the system allocator created a minor but unnecessary performance overhead.
The key misfeature of the debugging hooks, though, was their presence as unprotected function pointers that were guaranteed to be executed at specific events. This made the hooks an easy exploit primitive in practically every program that ran on Linux distributions. A trivial search for __malloc_hook "house of" turns up a long list of exploit methods that use the hooks as either an intermediate step or the final goal for the exploit.
Malloc hooks had to go.
Excising malloc hooks from the main library
The last of the malloc hook variables were deprecated in glibc 2.32 and new applications were encouraged to use malloc interposition instead. The effect of deprecation was mild: Newer applications just got a warning during the build. In the interest of maintaining backward compatibility, the memory allocator continued to look for hooks and, if available, execute them. In glibc version 2.34 (August 2021), we finally bit the bullet and took support for malloc hooks out of the mainstream library.
The upstream glibc community agreed that malloc debugging features have no place in production. So we moved all debugging features into a separate library named libc_malloc_debug.so.0 that overrides system malloc behavior to enable debugging. Most importantly, we either moved unprotected hook function pointers into libc_malloc_debug.so.0 or removed them completely. Doing this eliminated a key exploit primitive from the library.
Debugging and hardening in a post-hook world
The new glibc without the problematic hooks will be available in future versions of Fedora and RHEL. With this glibc, malloc debugging features such as MALLOC_CHECK_, mtrace(), and mcheck() will no longer work by default. Users will need to preload libc_malloc_debug.so.0 to enable these debugging features. Additionally, the __after_morecore_hook, __default_morecore_hook, and __morecore function pointers are no longer read, and the system malloc uses the brk() and mmap() system calls to request memory from the kernel.
System administrators may also remove the library from the system and effectively disable malloc debugging and malloc hooks. This is useful hardening for production systems that have strong controls on what files are available on the system.
Conclusion
Separating malloc debugging from the main library is a significant security hardening improvement in glibc. It eliminates an exploit primitive from Linux distributions and adds an opportunity for hardening in both RHEL and Fedora. Simplifying system allocator code also sets the stage for improvements to malloc that may result in better security and performance. Watch out for more interesting changes to the malloc subsystem in future releases of glibc.
Last updated: September 12, 2022