Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Diagnosing Function Pointer Security Flaws with a GCC plugin

March 17, 2017
Aldy Hernandez Martin Sebor
Related topics:
Security

Share:

    A few months ago, I had to write some internal GCC passes to perform static analysis on the GNU C Library (glibc). I figured I might as well write them as plugins since they were unlikely to see the light of day outside of my little sandbox. Being a long time GCC contributor, but having no experience writing plugins I thought it'd be a good way to eat our own dog food, and perhaps write about my experience.

    Unfortunately, I quickly realized that non-trivial plugins are basically GCC hacking with some boilerplate code, and the details of such were unlikely to benefit anyone but die-hard GCC contributors who were very much capable of contributing their own extensions to GCC.

    However, there was one particular pass which was squarely sitting in my sandbox, that could serve not only as an example of a non-trivial plugin for the curious but as a tool to pinpoint security flaws involving pointers to functions. Consequently, I'll make the code available and talk about the plugin use itself, not its implementation.

    It has been known for some time that dereferencing function pointers in code is a focus of exploits, yet they are frequently used in code everywhere. Any time a function pointer is stored in read-write memory, its contents can potentially be altered in such a way that the flow of the program is maliciously transferred elsewhere.

    A typical use of function pointers is a dispatch table:

    #include <stdio.h>
    
    typedef void (*callback)(void);
    
    void hello (void)
    {
      printf ("hello world\n");
    }
    
    callback dispatch[] = { hello };
    
    int main(int argc, char **argv)
    {
      callback cb = dispatch[0];
      // do some things
      cb ();
      return 0;
    }

    However, the dispatch table itself could be corrupted by any number of memory alterations. Even if the variable `cb' itself resides in memory, it could be altered between the reading from dispatch[0] and the execution of cb().

    One proposed solution to this problem has been to encrypt any function pointers residing in memory (the dispatch table in this case). For instance, we could XOR each item in the dispatch table by a random amount upon start-up, and at each indirect function execution point in the program, we could twiddle the bits back. This would ensure that any malicious change to the dispatch table behind our back would yield jumps to garbage.

    A simple decryptor for a function pointer use would be:

    #define PTR_DEMANGLE(var) \
      (var) = (__typeof(var)) ((uintptr_t) (var) ^ secret_random_number)
    
    int main(int argc, char **argv)
    {
      callback cb = dispatch[0];
      PTR_DEMANGLE (cb);
      cb ();
      return 0;
    }

    This is the gist of the problem at its simplest. Obviously, you probably don't want to store that number anywhere, but perhaps a thread local register. You would also likely want a fancier function than PTR_DEMANGLE above. For that matter, you may even want some fancy target specific assembly sequence that does this for you:

    __attribute__((decryptor)) // NOTE: `decryptor' attribute explained below.
    static callback asm_demangler (callback f)
    {
      asm ("#decrypt_operation \$2, \$0" : "=r" (f) : "0" (f));
      return f;
    }

    Other problematic uses of function pointers include:

    • Copying a decrypted function pointer to memory.  Note: Copying to a register is probably OK, inasmuch as that the register is not spilled to memory.
    • Passing a non-decrypted function pointer as an argument to a function, unless you are sure your function will decrypt things appropriately.
    • Comparing a non-decrypted function pointer to a constant (say NULL).
    • Instead, you should probably do:
    f = PTR_DEMANGLE (dispatch[n]);
      if (f != NULL)
        ok();
    • Copying a function pointer to memory:
      dispatch[i] = &printf;
    • Passing objects containing function pointers to functions:
      struct obj { callback f };
      void foo (void)
      {
        struct obj arg;
        bar(&arg);
      }
    • In C++, virtual functions can be the target of sophisticated exploits.

    Auditing every use of function pointers in something as large as the GNU C Library is cumbersome and error prone to say the least. That's why I thought getting a compiler plugin to do the work would be a better use of my time, not to mention fun.

    The plugin git sources, as well as instructions on how to build and use it, can be found here:

    https://pagure.io/funcp-encrypt

    To try this plugin on an existing code base, you can probably just set CC and CXX to use the plugin:

    $ export CC="your-gcc -fplugin=/path/to/plugin.so"
    $ export CXX="your-gcc -fplugin=/path/to/plugin.so"
    $ make YOUR_STUFF

    This should get you errors such as these while building with the plugin:

    foo.c: In function ‘init’:
    foo.c:49:21: warning: copy of a function pointer to memory
    __resume = resume;
    ~~~~~~~~~^~~~~~~~
    ...
    
    bar.c:88:8: warning: possible use of non decrypted function pointer
    (*__init [i]) (argc, argv, envp);
    ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

     

    So far, the plugin can diagnose the following cases:

    • Comparisons between encrypted function pointers and a constant.
    • Comparison between encrypted and unencrypted function pointers.
    • Passing non-decrypted function pointers as arguments.
    • Dereferencing non-decrypted function pointers.
    • Possible copies of function pointers to memory.

    For additional examples of what the plugin can do, you can look at the test case provided in test-funcp.c.

    Note: As an enhancement to existing code bases, I added a function attribute to be able to mark functions that were permitted to operate on function pointers without the usual restrictions (for instance, a function decryptor). With this plugin, a function with __attribute__((decryptor)) is considered an acceptable way to decrypt a function pointer, similar to PTR_DEMANGLE above. For example:

    __attribute__((decryptor))
    static callback asm_demangler (callback f)
    {
      asm ("#decrypt_operation \$2, \$0" : "=r" (f) : "0" (f));
      return f;
    }

    With this attribute, the plugin will allow the asm_demangler() function to accept function pointers and operate on them without issuing a warning.

    Paranoia aside, all these jumps and hoops are unlikely to yield any benefit in your financial report's callbacks, but keeping track of function pointers can be incredibly useful for library writers and other security-sensitive code.

    Feel free to comment on the plugin and/or function pointer usage!


    Build your first app with native GCC on RHEL 6 or 7.

    Last updated: March 14, 2017

    Recent Posts

    • Meet the Red Hat Node.js team at PowerUP 2025

    • How to use pipelines for AI/ML automation at the edge

    • What's new in network observability 1.8

    • LLM Compressor: Optimize LLMs for low-latency deployments

    • How to set up NVIDIA NIM on Red Hat OpenShift AI

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue