In previous articles on Red Hat Developer, I introduced you to the GNU Debugger (GDB) and explained what every developer should know about debuginfo. This article begins a series that will cover the expansive topic of debug events—that is, all the events to which GDB can respond. These include signals, breakpoints, watchpoints, syscalls, exceptions, and more. I'll be covering the commands and convenience variables and functions that will aid you in your endeavors to stop GDB at the right place at the right time.
A breakpoint is the most useful of these events, so let me start there.
What is a breakpoint?
A breakpoint instructs the debugger to stop at a particular code location in the user's program, returning control of the debugger to them. The user may then inspect the application's state. Breakpoints are by far the most common type of debug event that developers use.
Under the covers, there are many ways to implement breakpoints, including inserting a trap or illegal instructions that cause the kernel to raise a signal which is then handled by the debugger. I'm not going to talk about the specifics of implementing breakpoints, but rather how to use breakpoints effectively.
In GDB, the break command is used to set a breakpoint:
(gdb) | help break | head -4
break, brea, bre, br, b
Set breakpoint at specified location.
break [PROBE_MODIFIER] [LOCATION] [thread THREADNUM]
[-force-condition] [if CONDITION]
Note: Here I've piped the output of help break to the shell program head. GDB supports a pipe command (| is an abbreviation) that allows users to pipe arbitrary output to any shell command. For more information, read about the pipe command in the GDB User's Manual, or consult GDB's online help (help pipe).
As you can see from this (truncated) help text, the break command accepts several arguments and has several abbreviations (which I will use interchangeably throughout this article). Let's quickly explore these. (I'll defer discussion of probes for now.)
Specifying a location
If you read the help text for the break command more closely, it explains that the LOCATION may be "a linespec, address, or explicit location." This argument is the one you will use the most. It is a conceptual "point" in your program at which you can instruct the debugger to stop your program and return control of the debugger to you. Let's take a look at each of these three types of location in turn.
Linespec locations
A linespec location is a colon-separated list that includes a source file name, source line number, and function and/or label names. Some examples of these easily illustrate how this works:
(gdb) # Set a breakpoint on the function main() (gdb) break main Breakpoint 1 at 0x4011a2: file factorial.c, line 21. (gdb) # Set a breakpoint on the twenty-third line of the file factorial.c (gdb) break factorial.c:23 Breakpoint 2 at 0x4011a8 file factorial.c<, line 23. (gdb) # Set a breakpoint at the label error_exit in the function factorial() (gdb) break factorial:error_exit Breakpoint 3 at 0x40118c: file factorial.c, line 14.
Address locations
An address location is an exact address in the inferior (your running program), and they are easily recognized by the use of an asterisk. Address locations accept any valid expression as arguments:
(gdb) # Set a breakpoint at the first instruction of the function main() (gdb) break *main Breakpoint 4 at 0x401193: file factorial.c, line 20. (gdb) # Set a breakpoint at the code address 0x401156 (gdb) b *0x401156 Breakpoint 5 at 0x401156 file factorial.c, line 6. (gdb) # Set a breakpoint at 16 bytes past main() (gdb) b *main + 16 Breakpoint 6 at factorial.c, line 21.
Notice the distinction between break main and break *main. The address location (*main) sets the breakpoint at the very first instruction of the function, which may be in a compiler-generated function prologue. The linespec location (main) sets the breakpoint after any function prologue—in other words, it sets the breakpoint at the first instruction of your code.
Explicit locations
Explicit locations allow users to unambiguously specify a location using key/value pairs. GDB's linespecs are ambiguous. When you type A:B:C, GDB does not know whether A represents a function or file name without doing file or function look-ups. When a linespec location is definitively parsed, GDB internalizes it into an explicit location, thereby removing all ambiguity and speeding location resolution. This can be very beneficial for large programs.
Users can also bypass the linespec parser by using explicit locations:
(gdb) # Set a breakpoint on the function main() (gdb) break -func main Breakpoint 7 at 0x4011a2: file factorial.c, line 21. (gdb) # Set a breakpoint on the twenty-third line of the file factorial.c (gdb) break -source factorial.c -line 23 Breakpoint 8 at 0x4011a8: file factorial.c, line 23. (gdb) # Set a breakpoint at the label error_exit in the function factorial() (gdb) break -func factorial -label error_exit Breakpoint 9 at 0x40118c: file factorial.c, line 14.
Limiting breakpoints to a specific thread
In multi-threaded programs, it is often desirable to limit breakpoints to a specific thread. This can be accomplished by supplying the thread keyword and GDB's thread-id after the location. GDB will then only stop on the breakpoint if it is hit in the specified thread:
(gdb) # Get a list of threads
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7ffff7da2740 (LWP 1016086) "threaded" main (argc=1, argv=0x7fffffffd918) at threaded.c:44
2 Thread 0x7ffff7da1640 (LWP 1016089) "threaded" 0x00007ffff7e7d5b5 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
3 Thread 0x7ffff75a0640 (LWP 1016090) "threaded" 0x00007ffff7e7d5b5 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
4 Thread 0x7ffff6d9f640 (LWP 1016091) "threaded" 0x00007ffff7e7d5b5 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
5 Thread 0x7ffff659e640 (LWP 1016092) "threaded" 0x00007ffff7e7d5b5 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
6 Thread 0x7ffff5d9d640 (LWP 1016093) "threaded" 0x00007ffff7e7d5b5 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
(gdb) # Set a breakpoint on function foo() in thread #3
(gdb) break foo thread 3
Breakpoint 2 at 0x4011a1: file threaded.c, line 9.
(gdb) continue
Continuing.
[Switching to Thread 0x7ffff75a0640 (LWP 3584523)]
Thread 3 "threaded" hit Breakpoint 2, foo (num=1) at threaded.c:9
9 printf("%d: here I am!\n", num);
(gdb) p $_thread
$1 = 3
Notice the use of a convenience variable to output the currently focused thread ($_thread). GDB defines a number of convenience variables and functions, and you can also define your own variables and functions (even in Python). Look for a future article discussing this useful feature.
Specifying a breakpoint condition
You can also specify that GDB should stop on a breakpoint only when some expression in the inferior evaluates as true. This is especially useful when a function is called many times during the execution of your program and you want to stop for one specific condition, such as when a variable (either a function parameter or a local variable) evaluates to a specified value.
Consider a function that computes the factorial of a number recursively, declared int factorial (int n);. Here's how you would tell GDB to stop whenever the parameter n is 2:
(gdb) # Stop when computing factorial (2) (gdb) b factorial if n == 2 Breakpoint 10 at 0x401161: file factorial.c, line 7. (gdb) r 8 Starting program: /home/keiths/factorial 8 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Breakpoint 10, factorial (n=2) at factorial.c:7 7 if (n > 10) (gdb) p n $1 = 2
To remove a condition from a breakpoint, simply set a new, blank condition on it.
The -force-condition flag tells GDB to set the condition even though it may not be evaluated at that time. This is useful, for example, if a variable used in the condition cannot be evaluated in the scope of the breakpoint's location but will in the future due to a library load.
Multi-location breakpoints and -qualified
By default, GDB will search all scopes to find a location when setting a breakpoint on a function. For example, break method will search all scopes to find a function or method named method. GDB will then set a breakpoint at every location found:
(gdb) b method Breakpoint 1 at 0x40110a: method. (9 locations) (gdb) info break Num Type Disp Enb Address What 1 breakpoint keep y <MULTIPLE> 1.1 y 0x000000000040110a in method() at ov.cc:57 1.2 y 0x00000000004011de in a<a<int> >::method() at ov.cc:10 1.3 y 0x00000000004011ea in a<b<int> >::method() at ov.cc:10 1.4 y 0x00000000004011f6 in c<c<int> >::method() at ov.cc:22 1.5 y 0x0000000000401202 in c<d<int> >::method() at ov.cc:22 1.6 y 0x000000000040123e in A<a<b<int> >, c<d<int> > >::method() at ov.cc:34 1.7 y 0x0000000000401250 in A<a<b<int> >, a<a<int> > >::method() at ov.cc:34 1.8 y 0x0000000000401262 in A<a<b<int> >, a<b<int> > >::method() at ov.cc:34 1.9 y 0x0000000000401274 in B<a<b<int> >, c<d<int> > >::method() at ov.cc:40
In the above example, GDB has set nine different breakpoints for the requested function method(), a method that is defined in several different classes/structures and one function. If you want to exclude one of these locations from triggering a break, simply disable it with disable break-id, where break-id is one of the numbered breakpoint locations listed in the above table—1.6, for example.
Whenever a condition is used when setting a breakpoint with multiple locations, only those locations in which the condition is valid will actually result in a breakpoint. All other locations will be ignored.
Use the -qualified flag to tell GDB that you are looking for an exact match:
(gdb) break -qualified method Breakpoint 2 at 0x40110a: file ov.cc, line 57. (gdb) info break Num Type Disp Enb Address What 2 breakpoint keep y 0x000000000040110a in method() at ov.cc:57
Other useful breakpoint information, commands, and variables
No discussion of breakpoints is complete without mentioning several adjacent commands and convenience variables associated with breakpoints.
Print the breakpoint table
The info breakpoints command prints the current list of user-defined breakpoints and other breakpoint-related information (conditions, thread, and ignore count):
(gdb) info break Num Type Disp Enb Address What 11 breakpoint keep y 0x0000000000401161 in factorial at factorial.c:7 stop only if n == 2 12 breakpoint keep y 0x00000000004011a2 in main at factorial.c:21 breakpoint already hit 1 time
If you want to output GDB's list of internal breakpoints, use maint info break instead.
Set a conditional breakpoint
cond break-id expression is an alternative form of break location if expression. If you forgot to set a conditional for your breakpoint, you can set it after the fact with the cond command.
Last created break-id
When you create a new breakpoint, its break-id will be saved in the convenience variable $bpnum. This can be passed to other commands, convenience functions, and so on.
(gdb) # These are equivalent (gdb) break factorial if n == 2 Breakpoint 13 at 0x401161: file factorial.c, line 7. (gdb) break factorial Note: breakpoint 13 also set at pc 0x401161. Breakpoint 14 at 0x401161: file factorial.c, line 7. (gdb) cond $bpnum n == 2 (gdb) info break Num Type Disp Enb Address What 13 breakpoint keep y 0x0000000000401161 in factorial at factorial.c:7 stop only if n == 2 14 breakpoint keep y 0x0000000000401161 in factorial at factorial.c:7 stop only if n == 2
Automatically execute commands at a breakpoint
The commands [break-id] command instructs GDB to execute a series of debugger commands when breakpoint break-id is hit. If no argument is supplied, this command acts on the last one set—that is, as if $bpnum had been specified. Commands to execute are entered, one per line, terminating with end. For example, the following will print a simple message to the terminal whenever a breakpoint at main() is hit:
(gdb) b main Breakpoint 4 at 0x4011a2: file factorial.c, line 21. (gdb) commands Type commands for breakpoint(s) 4, one per line. End with a line saying just "end". >printf "ARGC = %d\n", argc >end (gdb) r Starting program: /home/keiths/factorial [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Breakpoint 1, main (argc=1, argv=0x7fffffffd918) at factorial.c:21 21 if (argc != 2) ARGC = 1
As you may have gathered from this listing, GDB has its own built-in printf-like function (conveniently called printf). Semantically, this function behaves exactly the same as your beloved C version. For more information, see printf command in the GDB User's Manual, or consult GDB's online help (help printf).
Ignore the next set number of hits of a breakpoint
Notice the message breakpoint already hit 1 time in the output of info break above. GDB keeps a running count of the number of times a breakpoint has been hit. Along with the ignore command, this opens a powerful way to fine-tune when a breakpoint stops your application, which you can invoke with the ignore break-id count command. ignore tells GDB to simply ignore the next count hits of the breakpoint that has the ID break-id
Consider the following (very) contrived source code, which will abort when the global variable n reaches 1,000:
#include <stdlib.h>
static int n;
void
called_often (void)
{
if (n == 1000)
abort ();
}
int
main (void)
{
for (n = 0; n < 1001; ++n)
called_often ();
return EXIT_SUCCESS;
}
One way to stop at the desired call of called_often (at the right time) is to use the obvious breakpoint condition n == 1000. Why your application is crashing is seldom this easy to understand, though. That's why you are using a debugger after all!
A useful trick here is to use the commands and ignore commands together to stop you at the right spot. First, figure out what ignore count you need to use. Here, we silence the breakpoint so that we don't have to see 1,000 "breakpoint hit" messages.
(gdb) break called_often
Breakpoint 1 at 0x40112a: file ncrash.c, line 8.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>continue
>end
(gdb) r
Starting program: /home/keiths/ncrash
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Program received signal SIGABRT, Aborted.
0x00007ffff7e33c4c in __pthread_kill_implementation () from /lib64/libc.so.6
(gdb) info break
Num Type Disp Enb Address What
1 breakpoint keep y 0x000000000040112a in called_often at ncrash.c:8
breakpoint already hit 1001 times
silent
continue
Notice that GDB says breakpoint already hit 1001 times. So it was on the one-thousand-first time this breakpoint was hit that the application crashed. That means we want to ignore the first 1,000 hits of this breakpoint. GDB should then stop exactly before the "crash" happens:
(gdb) ignore 1 1000 Will ignore next 1000 crossings of breakpoint 1. (gdb) # We don't need or want the breakpoint to be silent anymore! (gdb) commands Type commands for breakpoint(s) 1, one per line. End with a line saying just "end". >end (gdb) r The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/keiths/ncrash [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Breakpoint 1, called_often () at ncrash.c:8 8 if (n == 1000) (gdb) p n $1 = 1000
Note: If you are stopped at a breakpoint, you can also give an ignore count to the continue command.
Delete a breakpoint by break-id or location
The delete breakpoints [break-id] and clear LOCATION commands both delete breakpoints. delete breakpoints (or simply delete) takes GDB's breakpoint ID as its argument. clear takes the event location (linespec, address, or explicit location) as its argument:
(gdb) info break Num Type Disp Enb Address What 1 breakpoint keep y 0x00000000004011a2 in main at factorial.c:21 2 breakpoint keep y 0x0000000000401161 in factorial at factorial.c:7 (gdb) delete 2 (gdb) clear factorial.c:21 Deleted breakpoint 1 (gdb) info break No breakpoints or watchpoints.
Set a breakpoint on a regular expression
rbreak regexp sets a breakpoint on each function that matches the given regular expression. Each of these essentially acts as if you had used break on the matching function name. This command may set multiple breakpoints with multiple locations. This is a very expensive operation to perform in large applications, so use it wisely!
Temporarily disable or enable a breakpoint
The disable and enable commands can temporarily disable or enable a breakpoint, respectively. A disabled breakpoint will simply be ignored by the debugger, and your application will not stop at it until it is re-enabled.
Tip: A useful debugging trick is to break at a function only after some other function has executed. For example, this is how you'd stop in the function common_function only after the function precursor has executed:
(gdb) # Set breakpoints at both locations (gdb) break common_function Breakpoint 1 at 0x40110a: file common.c, line 4. (gdb) break precursor Breakpoint 2 at 0x401111: file common.c, line 10. (gdb) # Disable the commonly called function (gdb) dis 1 (gdb) # When the break at `precursor' is hit, have it re-enable the other breakpoint (gdb) # We'll not silence this so that we can see that the breakpoint was hit. (gdb) commands 2 Type commands for breakpoint(s) 2, one per line. End with a line saying just "end". >enable 1 >continue >end (gdb) # Now GDB will only stop in `common_function' after `precursor' is executed. (gdb) # To re-disable Breakpoint 1 whenever re-running, use a hook: (gdb) define hook-run Type commands for definition of "hook-run". End with a line saying just "end". >disable 1 >end (gdb) r The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/keiths/common [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Breakpoint 2, precursor () at common.c:10 10 return; Breakpoint 1, common_function () at common.c:4 4 return;
Note that I've used a hook here to "automate" the disabling of Breakpoint 1. Since that breakpoint was left enabled, it would stop the debugger at the first hit when we re-run the application. So we need to ensure that it is re-disabled. Any command in GDB can be "hooked" in this fashion. For more information, see User-defined command hooks in the GDB User's Manual.
Save breakpoints to a file
If you've set up a bunch of breakpoints with conditions, ignore counts, and/or commands, it can be a real nuisance to manually write out a script file to restore your session. Let GDB do it for you! save breakpoints filename will save all of your breakpoints into the given named file, which will be a simple GDB script. Whenever you would like to restore your breakpoints, simply source this file into GDB.
Set a temporary breakpoint
tbreak LOCATION is similar to the normal break command, but it sets a temporary breakpoint at the specified location (with possible condition, thread, and so on). The first time the breakpoint is hit, GDB will automatically delete it.
Next up
I've attempted to present the basics of GDB's breakpoint facilities, including some tips and tricks on how to streamline your time getting your application stopped at exactly the right place and time.
In the next article in this series, I will continue to dive into debug events by discussing how GDB handles signals and how you can tweak this handling to suit your needs.
Do you have a suggestion or tip related to breakpoints or a suggestion for a future topic about how to use GDB? Leave a comment on this article and share your idea with us.
Last updated: June 7, 2023