The strace
command traces system calls and signals, deciding them and their corresponding arguments into a symbolic form. A frequent debugging request from developers is the ability to allow strace
to trace system calls for a program that is also being debugged by GDB, like this:
% gdb --args test-program (gdb) b main Breakpoint 1 at 0x40128e: file test-program.c, line 22. (gdb) run Starting program: test-program Breakpoint 1, main (argc=3, argv=0x7fffffffdb98) at test-program.c:22 22 int thread_count = 2; (gdb)
In another terminal window, we invoke strace
on the same process GDB is debugging:
% strace -p $(pgrep -f test-program) strace: attach: ptrace(PTRACE_SEIZE, 27882): Operation not permitted
The culprit here is that the ptrace
system call, which is used by both GDB and strace
to control the execution of programs, and does not allow both strace
and GDB to control the same process.
One solution is to use gdbserver
to support both the gdb
client and the strace
client.
Extending strace
to support gdbserver
The first step in this process is to extend strace
to support gdbserver
. The strace
command is implemented using the ptrace
call to intercept system calls. Gdbserver is also able to intercept system calls. To enable strace
to use gdbserver
, it is necessary to add a new back end that uses gdbserver
instead of ptrace
to do the system call interception. This gdbserver
back end provides similar functionality to the ptrace
back end.
Now, let's look at some examples.
Example of strace
using a gdbserver
An example of running strace
with an alternative gdbserver
back end is:
% strace -G '|/usr/bin/gdbserver --once --multi stdio' test-program
This example says to:
- Connect to
gdbserver
using the-G
option. - Run
gdbserver
only once. - Communicate with strace with stdio.
- Use gdbserver to trace
test-program
.
When run, the output provides (some output omitted):
... [pid 15603] close(-1) = -1 EBADF (Bad file descriptor) [pid 15603] chroot(".") = -1 EPERM (Operation not permitted) [pid 15603] pipe([3, 4]) = 0 [pid 15603] write(4, "a\0", 2) = 2 [pid 15603] read(3, "a\0", 2) = 2 [pid 15603] madvise(0x7ffff75bb000, 8368128, MADV_DONTNEED) = 0 [pid 15595] --- stopped by 255 --- [pid 15595] +++ exited with 0 +++
Example of strace
using a standalone gdbserver
Alternately strace
can use a standalone gdbserver
:
- Start
gdbserver
:
% gdbserver --multi :65432
- Second, start the
strace
client and communicate withgdbserver
using the TCP port 65432. - Tell
strace
to trace the test program:
% strace -G localhost${i} test-program
Example of strace
using a remote gdbserver
An example of running strace
with a standalone gdbserver
on a remote machine is to first start gdbserver
on the remote machine:
% gdbserver --once :65432 test-program
Second, start the strace
client and communicate with gdbserver
on the remote host using the TCP port given in the gdbserver
command:
% strace -G remote-host.org.com:65432
Note that we give neither a -p PID
option nor a test program option. That information was specified by gdbserver
, and strace
will inherit that test program connection:
... brk(NULL) = 0x63a000 arch_prctl(0x3001 /* ARCH_??? */, 0x7fffffffdb40) = -1 EINVAL (Invalid argument) --- stopped by 255 --- +++ exited with 0 +++
Extending gdbserver
to support both a gdb
and an strace
client
Gdbserver currently handles a single client connection. To support both a gdb
and strace
client, it is necessary to enable gdbserver
to handle multiple client connections. To do this, gdbserver
must alternate between:
- The
gdb
client waits. - Gdbserver sends a
syscall
packet tostrace
. - Strace continues running via
gdbserver
. - Gdbserver interacts with the
gdb
client while thestrace
client waits.
Gdbserver currently keeps state information for a single client. Making gdbserver
multiple client-aware requires adding state information for both the gdb
client and the strace
client—for example, the file descriptor for the connection and state information regarding the packets sent over the connection.
As is the case with time-sharing, one client will always be the active client. Client packet requests (continue, step, get memory, etc.) and the current client state (active or waiter) determine the client's next state. For instance:
- State: GDB client is active,
strace
client is waiting. - Request: GDB client makes the next request over a syscall.
- State: GDB client is waiting,
strace
client is active. - Request: Strace receives syscall and continues running via
gdbserver
. - State: GDB client is active,
strace
client is waiting.
Example of a gdbserver
and strace
client
This example says to start gdbserver
and use TCP port 65432 to communicate with clients:
- Start
gdbserver
:
% gdbserver --multi :65432
- Do the following for GDB:
- Start the
gdb
client in another terminal window. - Tell GDB to communicate with the
gdbserver
using the TCP port 65432. - Set a breakpoint and run the test program:
- Start the
% gdb (gdb) file test-program (gdb) target extended-remote localhost:65432 (gdb) set remote exec-file test-program (gdb) b thread_worker (gdb) run (gdb) Thread 1 "test-program" hit Breakpoint 1, thread_worker () ... 52 pthread_barrier_wait (&barrier);
- Start the
strace
client in another terminal window. - Do the following for
strace
:
- Tell
strace
to communicate withgdbserver
using the TCP port 65432. - Tell strace to trace the test program that was started by the
gdb
client.
% strace -G localhost:65432 -p $(pgrep test-program)
- The following would be displayed in the strace window:
... brk(0x426000) = 0x426000
- Tell
- [GDB window] Tell the
gdb
client to use thread two, set a breakpoint, and advance the test program:
(gdb) thread 2 [Switching to thread 2 (Thread 7464.7490)] #0 thread_worker () at test-program.c:59 (gdb) b 52 (gdb) continue 59 close (-1); (gdb) next 61 chroot ("."); (gdb) next
- [Strace window] Notice that
strace
has traced theclose
andchroot
system calls:
[pid 7490] close(-1) = -1 EBADF (Bad file descriptor) [pid 7490] chroot(".") = -1 EPERM (Operation not permitted)
- [GDB window] Advance the test program:
(gdb) next 63 pipe (fd); (gdb) next 65 write (fd[1], buf1, sizeof (buf1)); (gdb) next 67 read (fd[0], buf2, sizeof (buf2)); (gdb) next
- [Strace window] Notice that
strace
has traced thepipe
,write
, andread
calls:
[pid 7490] pipe([3, 4]) = 0 [pid 7490] write(4, "a\0", 2) = 2 [pid 7490] read(3, "a\0", 2) = 2
- [GDB window] Continue the test program to completion:
(gdb) continue [Inferior 1 (process 7464) exited normally] (gdb) quit Remote connection closed
- [Strace window] Notice that the attached process has exited:
[pid 7490] madvise(0x7ffff7475000, 8368128, MADV_DONTNEED) = 0 [pid 7464] --- stopped by 255 --- [pid 7464] +++ exited with 0 +++ strace: Process 7490 detached
Current tool status
Versions of the GDB remote protocol, strace
, and corresponding gdbserver
are currently under review. Initial versions of the gdbserver
and strace
tools are available for experimentation at gdbserver_copr and strace_copr.
Summary
Extending strace
to additionally support gdbserver
provides an additional means to run strace
. For example, to trace a program that is running on a remote machine. Additionally, this extended strace
, along with a corresponding extended gdbserver
, enables tracing system calls in a program while a gdb
client is debugging the same program.