Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Build your own tool to search for code sequences in binary files

July 15, 2021
Nick Clifton
Related topics:
Open sourceSecurity
Related products:
Red Hat Enterprise Linux

Share:

    This article shows you how to create a scanning tool that can search for specific sequences of instructions inside binary files. Such searches are commonly required to verify that a compiled executable meets certain criteria, usually related to security. For example, Intel's Control-Flow Enforcement Technology (CET) extension mandates that all functions start with an ENDBR instruction. Verifying this requires a special tool specifically designed to search for instructions inside the binary.

    Simple searches with command-line tools

    Simple searches do not need special tools. Instead, you can do the job with a combination of command-line tools, possibly inside a script. For example:

    objdump -d FOO | grep -i endbr

    This searches the FOO program for any variation of the endbr instruction.

    Although searches like this are easy to construct, they are often insufficient. For example, the preceding command sequence doesn't verify that every function entry point starts with an ENDBR instruction. It just checks to see whether there are any occurrences of the instruction in FOO.

    Building a custom scanner

    To search for complex instructions or complex sequences of instructions, you will need a custom scanner. To this end, I'll show you how to modify the stack clash scanner that can be found in a branch of the annobin repository. Retrieve the scanner with:

    git clone -b stack-clash-scanner-branch git://sourceware.org/git/annobin.git

    This scanner uses the annocheck framework with a custom module. The scanner can check binary files, directories, and even RPMs.

    Note: The annocheck code is distributed under the GNU Public License v3 and is free for anyone to use and modify, but Red Hat does not provide any support for it.

    How the stack clash scanner works

    The basic stack clash scanner in the package looks for an AND instruction that takes the stack pointer and a large constant value as operands, for instance:

    and %rsp, 0x1000

    The point of this search is to find any stack pointer adjustments that are larger than a page size and that might therefore cause the stack to grow beyond a safe limit.

    The scanner works by disassembling the instructions in the binary and then searching the result for lines with the necessary features. The function is_affected_insn() in annocheck/stack-clash.c contains the necessary logic:

    /* Only AND instructions are affected. */
    if (strncmp (disas.buffer, "and", 3) != 0)
    return false;
    
    /* Must involve the stack pointer register. */
    if (strstr (disas.buffer, "%rsp") == NULL)
    return false;
    
    /* The instruction has to have a "large" immediate value.
    For now we take "large" to be 4K or more. */
    const char * const_start;
    if ((const_start = strstr (disas.buffer, "$0xff")) == NULL)
    return false;
    
    /* 4k = 4 * 1024 = 0x1000 */
    return strstr (const_start, "000") != NULL;

    Modifying the scanner

    You can modify the annocheck/stack-clash.c code to build a scanner that searches for other instruction sequences. The repository even includes a couple of other scanners adapted from the core code.

    For example, annocheck/jcc-scan.c contains code to locate conditional jump instructions that could be fused with previous instructions. These instructions were the subject of a potential security vulnerability a while back, which is why the scanner was created.

    The annocheck/retpoline.c file contains code that searches for a PAUSE instruction followed by a LFENCE instruction, which is an indicator of a compilation that supports the retpoline security feature.

    These examples all look for x86_64 instructions, but that is not a hard requirement. The scanners are linked against the opcodes library (part of the GNU Project binutils), which provides a disassembler for the host's architecture. The binutils package is usually installed on systems running Linux. So if the scanner is built on an AArch64 box, for example, it will have an AArch64 disassembler.

    It is also possible to link against a custom-built version of the opcodes library that has been configured to disassemble for another architecture. Thus, you could create a scanner that runs on an x86_64 box but examines PowerPC binaries.

    Building the custom scanner

    The scanner sources include a configure script which, when run, should populate a build directory with the necessary makefiles. Then just running make should build all three scanners I've mentioned. The scanners have the following dependencies:

    • The libbfd and libopcodes libraries, which are required by the binutils package.
    • The libelf library, which is provided by the elfutils-libelf package.
    • The libdw library, which is provided by the elfutils-libs package.
    • The libiberty library, which is provided by the binutils-devel package.

    Advanced scanning

    Sometimes the exact instructions you are searching for are unknown. Instead, you have to locate an effect or specific behavior. To handle these cases, you can extend the scanner to simulate the target binary file instead of just disassembling it. This process, of course, is much more complex.

    The scanner sources include two examples of this kind of advanced scanning, although neither is built by default. You can use the file annocheck/makefile.rop to build them, although you'll have to edit it to provide some necessary information. These scanners use the headers found in the binutils sources as well as the simulator code that is part of the GNU Debugger (GDB) project.

    The advanced scanners in the sources both have the same job: Examining binaries to see whether they are vulnerable to exploits via a return-oriented programming (ROP) attack. One scanner examines AArch64 binaries and the other examines x86_64 binaries. Multiple instruction sequences are vulnerable to this kind of attack, so the scanners simulate the execution of instructions and look for characteristics that are of use to an attacker. Since the attacker can, in theory, start execution at any point in the binary, the scanners have to run lots of simulations, looking for any possible vulnerable entry point.

    Conclusion

    Looking for instruction sequences in executable files is possible with today's tools. Although command-line tools will suffice for simple scans, a dedicated program is the best solution for complex needs. This article has shown you how to build a custom scanner that you can use for both common and advanced scanning scenarios.

    Last updated: October 6, 2022

    Related Posts

    • Annobin - Storing Extra Information in Binaries

    • Annocheck: Examining the contents of binary files

    • Stack Clash mitigation in GCC: Why -fstack-check is not the answer

    Recent Posts

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    • How to encrypt RHEL images for Azure confidential VMs

    • How to manage RHEL virtual machines with Podman Desktop

    • Speech-to-text with Whisper and Red Hat AI Inference Server

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue