A binary may be subject to a wide range of attacks, but smashing the stack for fun and profit is one of the most venerable ones. Stack-based attacks are also the lowest-cost form of attack: Stack layouts are quite predictable because data containing return addresses can be overwritten to gain control over program execution.
Compiler developers have implemented a wide range of countermeasures in response. This article discusses the major stack protection mechanisms in the GNU Compiler Collection (GCC) and Clang, typical attack scenarios, and the compiler's attempts to prevent attacks.
Stack canary
A stack canary is the most rudimentary check for buffer overflows on the stack. The canary is an extra word of memory at the end of the stack frame with a value set at runtime. This value is presumably unknown to the attacker and checked for modification before jumping out of the function. A modification indicates the detection of a stack smashing followed by a termination routine.
Stack canaries are added by GCC and Clang through these flags:
-fstack-protector-fstack-protector-strong-fstack-protector-all-fstack-protector-explicit
SafeStack and shadow stack
This form of protection splits the stack into two distinct areas, storing precious variables and user variables in non-contiguous memory areas. The goal is to make it more difficult to smash one of the stacks from the other. This process occurs in hardware (e.g., x86_64 shadow stack) or in software such as LLVM's SafeStack.
GCC and Clang add shadow stack through this flag:
-mshstk
Clang adds SafeStack through this flag:
-fsanitize=safe-stack
Fortified source
The GNU C library (glibc) provides alternate implementations of some commonly used functions to smash the stack by copying a given amount of bytes from one address to another. These implementations typically use compiler support to check for memory bounds of objects. If an operation overflows these bounds, it would cause programs to terminate. This defense is called FORTIFY_SOURCE and the Red Hat blog has an excellent post on the subject.
GCC and Clang select fortified sources using the preprocessor flag -D_FORTIFY_SOURCE=<n> with an optimization level greater than -O0. A higher <n> corresponds to greater protection. However, there may be a tradeoff in performance, and some programs that conform to the standard may fail due to stricter security checks. GCC (version 12 and later) and Clang (version 9.0 and later) support <n> up to 3.
Control flow integrity
Return-oriented programming (ROP) uses an initial stack smash to take control of an indirect jump and then executes an arbitrary sequence of instructions. One countermeasure to this kind of attack is to ensure that jump addresses and return addresses are correct by using hardware support or pure software.
GCC and Clang can generate support code for Intel's Control-flow Enforcement Technology (CET) through this compiler flag:
-fcf-protection=[full]
GCC and Clang can also generate support code for Branch Target Identification (BTI) on AArch64 using:
-mbranch-protection=none|bti|pac-ret+leaf|pac-ret[+leaf+b-key]|standard
In addition to these flags, which require hardware support, Clang provides a software implementation of control flow integrity.
-fsanitize=cfi
Stack allocation control
The following options have an impact on the stack allocation. These options are not necessarily designed to provide extra security, but they may be a nice side-effect.
On an x86 target, GCC and Clang provide the ability to automatically allocate a discontiguous stack when running out of stack memory. The -fsplit-stack activates this behavior.
When using the GNU linker, it is possible to pass a stack limit to the program, reading from a symbol or a register. This process is supported by GCC using:
-fstack-limit-register-fstack-limit-symbol
These stack-allocated arrays are a common entry point for a stack-based attack. The Clang compiler makes it possible to reduce the attack surface by disallowing this pattern, subsequently preventing the use of stack-allocated arrays. The code -fno-stack-array enables this process.
Stack usage and statistics
Most stack usage consists of either call stack information or fixed-sized allocations in the form of local variables. There is, however, a facility to allocate dynamically sized objects on the stack by using the alloca() function or using variable-length arrays (VLAs). alloca() usage with sizes can be user-controlled. This is a common target for overflowing stacks or making them clash with other maps in memory. As a result, a well-behaved application must always keep track of them. Both GCC and Clang provide ways to control alloca() and stack usage to prevent clashes at the outset.
The following flags allow finer control over stack usage in applications and provide a warning when alloca() or VLA cross developer-defined thresholds:
-Wframe-larger-than-Walloca-Walloca-larger-than-Wvla -Wvla-larger-than
There are also recursion checks as well as higher-level checks on stack usage to help developers get better control of the stack behavior.
Here is a full list of warnings implemented in GCC:
-Wstack-usage-fstack-usage-Walloca-Walloca-larger-than-Wvla-Wvla-larger-than-Wframe-larger-than-mwarn-dynamicstack(s390x)-Wstack-protector-Wtrampolines-Winfinite-recursion
And here is a list of the warnings implemented in Clang:
-fstack-usage-Walloca-Wframe-larger-than-Wstack-usage
Summary
Both GCC and Clang provide a wide range of compiler flags to prevent stack-based attacks. Some of these flags relate to a specific kind of exploit. Others introduce generic protection. And some flags give feedback like warnings and reports to the user, providing a better understanding of the behavior of the stack program. Depending on the attack scenario, code size constraints, and execution speed, compilers provide a wide range of tools to address the attack.
Last updated: August 3, 2022