Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Improve your code: Tales from confinement without a debugger

July 7, 2020
Aldy Hernandez
Related topics:
Linux

Share:

    I have always been impressed by developers who make do without a debugger, and have often wished I could be more like them. I vaguely recall a colleague saying he never used a debugger, favoring printf over gdb. Also, in my rookie years, I vividly recall chasing a kernel bug with a friend who was using objdump and the source of a much older kernel:

    "Richard, shouldn't you at least use the correct source?"
    "Meh ... they're close enough."

    I'm still impressed.

    Now that I have coded for some years, I've noticed that I have picked up some bad habits along the way. Over-dependence on the debugger is one of them. I often use it as a high-powered crutch, which frequently leads me waist-deep into stack traces, rarely stopping to think things through. I get lost inside 20 levels of recursion and wonder why an irrelevant variable is being tickled.

    Granted, there are many good uses for a debugger, but I'm at 40% on the good use scale. My uses usually start benign but then degrade into cancerous abstractions. So, for my 20th GNU Compiler Collection (GCC) hacking anniversary, I decided to give myself the challenge of one month without a debugger. Here is the tale.

    How I got here

    When I first started hacking GCC, newbies at Red Hat were put on old toolchain support duty. Nine times out of 10, those bugs had already been fixed upstream. I got quite adept at running two parallel gdbs, single-stepping until I found a difference in the codes, and eventually finding the patch that fixed the bug. My technique was effective, but taught me very little about the underlying problem that I was "fixing."

    Now, I'm a gray-bearded old fogey, and I can't count the number of times I have put a breakpoint on the garbage collector to find out who created a chunk of memory, just to save time analyzing the where of a given optimization. I even have an embarrassing gdb macro for doing it:

    define who_created_me
    break ggc-page.c:2683 if result == $arg0
    

    (Yes, I have different macros for different GCC versions. That source line inevitably changes.)

    I also have an assortment of strcmp variants in my local tree that I use for comparing GNU Compiler Collection internals (GIMPLE), assembly output, and other GCC internal states. I am frequently seen setting breakpoints for particular patterns because I'm trying awfully hard to avoid deep work. Here's an example:

    (gdb) break some_function if !strcmp_gimple(stmt, "var_3 = foo ();")
    

    Unfortunately, these cool party tricks seduce me into believing they're time savers. In reality, they quickly degrade into spelunking stack trace expeditions to nowhere. Frequently, the underlying reason for my expeditions is that I don't understand the code.

    Life without a debugger

    Initially, there was a lot of "Crap, what if I can't solve this?" What if my entire programming-fu is dependent on the debugger? What if this is the one bug that needs a debugger, and I have nothing but frustration to report in my weekly status report? After the initial panic, however, I got into a rhythm. Once that happened, I saw noticeable improvements in my thought process and my coding practice:

    • I noticed that a lot of my code was optimized for the debugging experience, not for readability or maintainability. I shied away from putting more than one statement in a line and broke things up to make it easier to set breakpoints. Doing this can sometimes, but not always, yield cleaner code.
    • I think it was Kernighan who said, "Debugging is twice as hard as writing a program in the first place." I was confronted by this cold, hard truth from the onset and realized what a timesaver gdb is for stepping through spaghetti code. I was also quick to notice, however, that I was spending so much time on specific problems because the code was horrible (which was not entirely my fault in a shared project). As a consequence, in my effort to debug less and fix more, my functions became smaller, less interdependent, and easier to understand. I also got quite good at refactoring other people's code in the compiler, to make things easier to understand. Overall, a win-win!
    • I judiciously added more asserts throughout the code to catch things earlier.
    • My dumps became more readable. It's surprising how many dumps in GCC are confusing at best, or useless at worst. Yes, I've contributed my share of unreadable dumps.
    • The keystrokes I saved navigating stack frames and setting breakpoints, went instead to writing carefully crafted printfs, which I made sure would persist into a dump file. Doing this saved me considerable time on subsequent "debugging" sessions.
    • Of the two times that I broke my promise and pulled out the debugger, both of them were because I was too tired to think the problem through. I hoped that randomly entering up, down, and print would magically solve my problem. Both times, what I needed was a break (no pun intended) and a fresh start the following day.

    Lessons learned

    In the interest of self-disclosure, I should say that I am not about to give up using debuggers. They are great tools in my arsenal. I might be more hesitant to pull out a debugger instinctively, at the first sign of trouble, but there are excellent reasons to use one. During my period of gdb-less confinement, I especially missed these uses for a debugger:

    • Debuggers are a boon for post-mortem analysis of why a process died. Going up and down the stack trace and examining variables is invaluable when inspecting a crash.
    • There is no real substitute for gdb-ing a hung process to see where it's stuck. I had to debug a few of those in the last month, and I don't wish that pain on anyone.
    • I draw the line at adding a bunch of printfs that are just emulating a gdb session. If you're reinventing a debugger through printfs in your code, for the love of Cthulhu, use the existing tools.
    • When you don't have the luxury of refactoring other people's spaghetti, a debugger can be a fast way of coming up to speed—just don't overuse it. At some point, you should sit down and understand the big picture.

    Conclusion

    I learned a lot during my month without a debugger. My code got smaller and easier to understand at a glance, and I got much better at sitting down and analyzing from a big-picture perspective. The fear of being stuck without a debugger led me to introduce fewer bugs in the first place. I thought more about what I wrote and depended less on gdb single-stepping sessions to analyze an algorithm. Overall, it was a good experience.

    If you spend more time noticing the forest rather than the tiny ant on the tree, you're more likely to fix the problem for good. Debuggers, by design, help you look at the ant on the tree. Don't get me wrong; that microfocus is useful. But it shouldn't be your primary way of analyzing a problem. The last month was scary, but it was time well spent.

    Last updated: July 1, 2020

    Recent Posts

    • How to use RHEL 10 as a WSL Podman machine

    • MINC: Fast, local Kubernetes with Podman Desktop & MicroShift

    • How to stay informed with Red Hat status notifications

    • Getting started with RHEL on WSL

    • llm-d: Kubernetes-native distributed inferencing

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue