Improve your code: Tales from confinement without a debugger

I have always been impressed by developers who make do without a debugger, and have often wished I could be more like them. I vaguely recall a colleague saying he never used a debugger, favoring printf over gdb. Also, in my rookie years, I vividly recall chasing a kernel bug with a friend who was using objdump and the source of a much older kernel:

"Richard, shouldn't you at least use the correct source?"
"Meh ... they're close enough."

I'm still impressed.

Now that I have coded for some years, I've noticed that I have picked up some bad habits along the way. Over-dependence on the debugger is one of them. I often use it as a high-powered crutch, which frequently leads me waist-deep into stack traces, rarely stopping to think things through. I get lost inside 20 levels of recursion and wonder why an irrelevant variable is being tickled.

Granted, there are many good uses for a debugger, but I'm at 40% on the good use scale. My uses usually start benign but then degrade into cancerous abstractions. So, for my 20th GNU Compiler Collection (GCC) hacking anniversary, I decided to give myself the challenge of one month without a debugger. Here is the tale.

How I got here

When I first started hacking GCC, newbies at Red Hat were put on old toolchain support duty. Nine times out of 10, those bugs had already been fixed upstream. I got quite adept at running two parallel gdbs, single-stepping until I found a difference in the codes, and eventually finding the patch that fixed the bug. My technique was effective, but taught me very little about the underlying problem that I was "fixing."

Now, I'm a gray-bearded old fogey, and I can't count the number of times I have put a breakpoint on the garbage collector to find out who created a chunk of memory, just to save time analyzing the where of a given optimization. I even have an embarrassing gdb macro for doing it:

define who_created_me
break ggc-page.c:2683 if result == $arg0

(Yes, I have different macros for different GCC versions. That source line inevitably changes.)

I also have an assortment of strcmp variants in my local tree that I use for comparing GNU Compiler Collection internals (GIMPLE), assembly output, and other GCC internal states. I am frequently seen setting breakpoints for particular patterns because I'm trying awfully hard to avoid deep work. Here's an example:

(gdb) break some_function if !strcmp_gimple(stmt, "var_3 = foo ();")

Unfortunately, these cool party tricks seduce me into believing they're time savers. In reality, they quickly degrade into spelunking stack trace expeditions to nowhere. Frequently, the underlying reason for my expeditions is that I don't understand the code.

Life without a debugger

Initially, there was a lot of "Crap, what if I can't solve this?" What if my entire programming-fu is dependent on the debugger? What if this is the one bug that needs a debugger, and I have nothing but frustration to report in my weekly status report? After the initial panic, however, I got into a rhythm. Once that happened, I saw noticeable improvements in my thought process and my coding practice:

• I noticed that a lot of my code was optimized for the debugging experience, not for readability or maintainability. I shied away from putting more than one statement in a line and broke things up to make it easier to set breakpoints. Doing this can sometimes, but not always, yield cleaner code.
• I think it was Kernighan who said, "Debugging is twice as hard as writing a program in the first place." I was confronted by this cold, hard truth from the onset and realized what a timesaver gdb is for stepping through spaghetti code. I was also quick to notice, however, that I was spending so much time on specific problems because the code was horrible (which was not entirely my fault in a shared project). As a consequence, in my effort to debug less and fix more, my functions became smaller, less interdependent, and easier to understand. I also got quite good at refactoring other people's code in the compiler, to make things easier to understand. Overall, a win-win!
• I judiciously added more asserts throughout the code to catch things earlier.
• My dumps became more readable. It's surprising how many dumps in GCC are confusing at best, or useless at worst. Yes, I've contributed my share of unreadable dumps.
• The keystrokes I saved navigating stack frames and setting breakpoints, went instead to writing carefully crafted printfs, which I made sure would persist into a dump file. Doing this saved me considerable time on subsequent "debugging" sessions.
• Of the two times that I broke my promise and pulled out the debugger, both of them were because I was too tired to think the problem through. I hoped that randomly entering up, down, and print would magically solve my problem. Both times, what I needed was a break (no pun intended) and a fresh start the following day.

Lessons learned

In the interest of self-disclosure, I should say that I am not about to give up using debuggers. They are great tools in my arsenal. I might be more hesitant to pull out a debugger instinctively, at the first sign of trouble, but there are excellent reasons to use one. During my period of gdb-less confinement, I especially missed these uses for a debugger:

• Debuggers are a boon for post-mortem analysis of why a process died. Going up and down the stack trace and examining variables is invaluable when inspecting a crash.
• There is no real substitute for gdb-ing a hung process to see where it's stuck. I had to debug a few of those in the last month, and I don't wish that pain on anyone.
• I draw the line at adding a bunch of printfs that are just emulating a gdb session. If you're reinventing a debugger through printfs in your code, for the love of Cthulhu, use the existing tools.
• When you don't have the luxury of refactoring other people's spaghetti, a debugger can be a fast way of coming up to speed—just don't overuse it. At some point, you should sit down and understand the big picture.

Conclusion

I learned a lot during my month without a debugger. My code got smaller and easier to understand at a glance, and I got much better at sitting down and analyzing from a big-picture perspective. The fear of being stuck without a debugger led me to introduce fewer bugs in the first place. I thought more about what I wrote and depended less on gdb single-stepping sessions to analyze an algorithm. Overall, it was a good experience.

If you spend more time noticing the forest rather than the tiny ant on the tree, you're more likely to fix the problem for good. Debuggers, by design, help you look at the ant on the tree. Don't get me wrong; that microfocus is useful. But it shouldn't be your primary way of analyzing a problem. The last month was scary, but it was time well spent.