GCC's new fortification level: The gains and costs

This article describes a new level of fortification supported in GCC. This new level detects more buffer overflows and bugs which mitigates security issues in applications at run time.

C programs routinely suffer from memory management problems. For several years, a _FORTIFY_SOURCE preprocessor macro inserted error detection to address these problems at compile time and run time. To add an extra level of security, _FORTIFY_SOURCE=3 has been in the GNU C Library (glibc) since version 2.34. I described its mechanisms in my previous blog post, Broadening compiler checks for buffer overflows in _FORTIFY_SOURCE. There has been compiler support for this builtin in Clang for some time. Compiler support has also been available for GCC since the release of version 12 in May 2022. The new mitigation should be available in GNU/Linux distributions with packaged GCC 12.

The following sections discuss two principal gains from this enhanced level of security mitigation and the resulting impact on applications.

2 principal gains:

1. Enhanced buffer size detection

2. Better fortification coverage

1. A new builtin provides enhanced buffer size detection

There is a new builtin underneath the new _FORTIFY_SOURCE=3 macro n GCC 12 named __builtin_dynamic_object_size. This builtin is more powerful than the previous __builtin_object_size builtin used in _FORTIFY_SOURCE=2. When passed a pointer, __builtin_object_sizereturns as a compile-time constant that is either the maximum or minimum object size estimate of the object that pointer may be pointing to at that point in the program. On the other hand, __builtin_dynamic_object_size is capable of returning a size expression that is evaluated at execution time. Consequently, the _FORTIFY_SOURCE=3 builtin detects buffer overflows in many more places than _FORTIFY_SOURCE=2.

The implementation of __builtin_dynamic_object_size in GCC is compatible with __builtin_object_size and thereby interchangeable, especially in the case of fortification. Whenever possible, the builtin computes a precise object size expression. When the builtin does not determine the size exactly, it returns either a maximum or minimum size estimate, depending on the size type argument.

This code snippet demonstrates the key advantage of returning precise values:

#include <string.h>
#include <stdbool.h>
#include <stdlib.h>

char *b;
char buf1[21];

char *__attribute__ ((noinline)) do_set (bool cond)
{
    char *buf = buf1;
    if (cond)
        buf = malloc (42);

    memset (buf, 0, 22);
    return buf;
}

int main (int argc, char **argv)
{
    b = do_set (false);
    return 0;
}

The program runs to completion when built with -D_FORTIFY_SOURCE=2:

gcc -O -D_FORTIFY_SOURCE=2 -o sample sample.c

But the program aborts when built with -D_FORTIFY_SOURCE=3 and outputs the following message:

*** buffer overflow detected ***: terminated
Aborted (core dumped)

The key enhancement stems from the difference in behavior between __builtin_object_size and __builtin_dynamic_object_size. _FORTIFY_SOURCE=2 uses __builtin_object_size and returns the maximum estimate for object size at pointer buf, which is 42. Hence, GCC assumes that the memset operation is safe at compile time and does not add a call to check the buffer size at run time.

However, GCC with _FORTIFY_SOURCE=3 invokes __builtin_dynamic_object_size to emit an expression that returns the precise size of the buffer that buf points to at that part in the program. As a result, GCC realizes that the call to memset might not be safe. Thus, the compiler inserts a call to __memset_chk into the running code with that size expression as the bound for buf.

2. Better fortification coverage

Building distribution packages with _FORTIFY_SOURCE=3 revealed several issues that _FORTIFY_SOURCE=2 missed. Surprisingly, not all of these issues were straightforward buffer overflows. The improved fortification also encountered issues in the GNU C library (glibc) and raised interesting questions about object lifetimes.

Thus, the benefit of improved fortification coverage has implications beyond buffer overflow mitigation. I will explain the outcomes of _FORTIFY_SOURCE=3 increased coverage in the following sections.

More trapped buffer overflows

Building applications with _FORTIFY_SOURCE=3 detected many simple buffer overflows, such as the off-by-one access in clisp issue. We expected these revelations, which strengthened our justification for building applications with _FORTIFY_SOURCE=3.

To further support the use of _FORTIFY_SOURCE=3 to improve fortification, we used the Fortify metrics GCC plugin to estimate the number of times _FORTIFY_SOURCE=3 resulted in a call to a checking function (__memcpy_chk, __memset_chk, etc.). We used Fedora test distribution and some of the Server package group as the sample, which consisted of 96 packages. The key metric is fortification coverage, defined by counting the number of calls to __builtin_object_size that resulted in a successful size determination and the ratio of this number taken to the total number of __builtin_object_size calls. The plugin also shows the number of successful calls if using __builtin_dynamic_object_size instead of __builtin_object_size, allowing us to infer the fortification coverage if all __builtin_object_size calls were replaced with __builtin_dynamic_object_size.

In this short study, we found that _FORTIFY_SOURCE=3 improved fortification by nearly 4 times. For example, the Bash shell went from roughly 3.4% coverage with _FORTIFY_SOURCE=2 to nearly 47% with _FORTIFY_SOURCE=3. This is an improvement of nearly 14 times. Also, fortification of programs in sudo went from a measly 1.3% to 49.57% — a jump of almost 38 times!

The discovery of bugs in glibc

The increased coverage of _FORTIFY_SOURCE=3 revealed programming patterns in application programs that tripped over the fortification without necessarily a buffer overflow. While there were some bugs in glibc, we had to either explain why we did not support it or discover ways to discourage those programming patterns.

One example is wcrtomb, where glibc makes stronger assumptions about the object size passed than POSIX allowed. Specifically, glibc assumes that the buffer passed to wcrtomb is always at least MB_CUR_MAX bytes long. In contrast, the POSIX description makes no such assumption. Due to this discrepancy, any application that passed a smaller buffer would potentially make wcrtomb overflow the buffer during conversion. Then the fortified version __wcrtomb_chk aborts with a buffer overflow, expecting a buffer that is MB_CUR_MAX bytes long. We fixed this bug in glibc-2.36 by making glibc conform to POSIX .

_FORTIFY_SOURCE=3 revealed another pattern. Applications such as systemd used malloc_usable_size to determine available space in objects and then used the residual space. The glibc manual discourages this type of usage, dictating that malloc_usable_size is for diagnostic purposes only. But applications use the function as a hack to avoid reallocating buffers when there is space in the underlying malloc chunk. The implementation of malloc_usable_size needs to be fixed to return the allocated object size instead of the chunk size in non-diagnostic use. Alternatively, another solution is to deprecate the function. But that is a topic for discussion by the glibc community.

Strict C standards compliance

One interesting use case exposed by _FORTIFY_SOURCE=3 raised the question of object lifetimes and what developers can do with freed pointers. The bug in question was in AutoGen, using a pointer value after reallocation to determine whether the same chunk extended to get the new block of memory. This practice allowed the developer to skip copying over some pointers to optimize for performance. At the same time, the program continued using the same pointer, not the realloc call result, since the old pointer did not change.

Seeing that the old pointer continued without an update, the compiler assumed that the object size remained the same. How could it know otherwise? The compiler then failed to account for the reallocation, resulting in an abort due to the perceived buffer overflow.

Strictly speaking, the C standards prohibit using a pointer to an object after its lifetime ends. It should neither be read nor dereferenced. In this context, it is a bug in the application.

However, this idiom is commonly used by developers to prevent making redundant copies. Future updates to GCC may account for this idiom wherever possible, but applications should also explicitly indicate object lifetimes to remain compliant. In the AutoGen example, a simple fix is to unconditionally refresh the pointer after reallocation, ensuring the compiler can detect the new object size.

The gains of improved security coverage outweigh the cost

Building with _FORTIFY_SOURCE=3 may impact the size and performance of the code. Since _FORTIFY_SOURCE=2 generated only constant sizes, its overhead was negligible. However, _FORTIFY_SOURCE=3 may generate additional code to compute object sizes. These additions may also cause secondary effects, such as register pressure during code generation. Code size tends to increase the size of resultant binaries for the same reason.

We need a proper study of performance and code size to understand the magnitude of the impact created by _FORTIFY_SOURCE=3 additional runtime code generation. However the performance and code size overhead may well be worth it due to the magnitude of improvement in security coverage.

The future of buffer overflow detection

_FORTIFY_SOURCE=3 has led to significant gains in security mitigation. GCC 12 support brings those gains to distribution builds. But the new level of fortification also revealed interesting issues that require additional work to support correctly. For more background information, check out my previous article, Enhance application security with FORTIFY_SOURCE.

Object size determination and fortification remain relevant areas for improvements in compiler toolchains. The toolchain team at Red Hat continues to be involved in the GNU and LLVM communities to make these improvements.