How lazy debuginfo loading improves GDB and Valgrind

Debugging information, a.k.a debuginfo, relates the machine instructions in an executable or shared library with human-readable information from the binary's source files. This information includes the names of functions and variables as well as the source file and line numbers where these symbols are declared and referenced. Debuggers such as GDB, the GNU debugger, and profilers such as Valgrind use debuginfo to provide developers with richly detailed views into the occurrence of memory errors in a running process or the exact cause of a crash recorded in a core file. To learn more about debuginfo, check out the following articles: Debuginfo is not just for debugging programs and The GDB developer’s GNU Debugger tutorial Part 2.

Debuginfod

Debuginfo is very useful for developers but it can come with a few downsides. One problem is that it can be quite large. For example, libwebkitgtk debuginfo is well over 3 GB. Acquiring the precise version of a binary's debuginfo and having the necessary permissions to install it and make it available to your debugger can be inconvenient or even impossible, depending on your particular debugging environment.

We developed the debuginfod tool for developer tools to automatically download the exact debuginfo files they require for maximally detailed debugging and profiling. Debuginfod is an HTTP file server and client library that enables tools to easily download debugging resources, including debuginfo and source files. Debugging tools, including GDB and Valgrind, can use Fedora's public debuginfod server to download the precise set of debuginfo and source files needed to debug or profile any executable packaged for Fedora since Fedora 32. Other Linux distributions also provide public debuginfod servers. Refer to the documentation for a complete list of supported distros. Debuginfod is part of the elfutils project. For more information, read my 2019 article introducing debuginfod.

The problem with eagerly loading debuginfo

A straightforward implementation of a debuginfod client for GDB and Valgrind simply downloads shared library debuginfo as soon as  each shared library is linked to the running process being debugged or referenced in a core file. However just because a shared library is linked doesn't mean that it's relevant to every debugging or profiling session. This eager downloading of debuginfo for every library associated with a process or core file can result in longer download times as well as wasted space taken up by the unnecessary debuginfo. This problem is also relevant even when the tool is not using debuginfod but is instead accessing debuginfo that has been installed locally. Decoding and reading all of the symbols, strings, addresses, etc. in every library's debuginfo can be time consuming. If this information is never used during the debugging or profiling session, then time was wasted. The extent and significance of this waste of time depends on factors such as the state of the debugged process, the size of all relevant debuginfo, the speed of your network, and the amount of available storage space, etc.

Improving GDB and Valgrind with lazy debuginfo loading

To improve the efficiency of debuginfo reading and downloading in GDB and Valgrind, we have been implementing lazy loading of debuginfo. In contrast to eager loading, lazy loading defers downloading or parsing debuginfo until the tool has a specific need to do so. GDB leverages debuginfod's ELF/DWARF section downloading to implement lazy loading. Instead of downloading a shared library's entire debuginfo file, GDB can instead download only the gdb-index section of the debuginfo. These indices contain information regarding the shared library's symbols that GDB can use until full debuginfo is actually needed. The size of these indices is around 7% of the size of the entire debuginfo file, so this can significantly reduce the total amount of debuginfo downloaded in a given debugging session.

Here are a few comparisons of lazy loading download sizes to eager loading download sizes. Disclaimer: the following numbers can change depending on the exact versions of the executables and shared libraries involved, as well as debuginfo that you have already installed.

• $ gdb -ex ‘break do_main’ -ex ‘run’ /usr/lib64/firefox/firefox

  • 94% reduction (235 MB vs 3.8 GB)

• $ gdb -ex ‘backtrace’ --pid [/usr/bin/virt-manager process]

  • 82% reduction (57 MB vs 323 MB)

• $ gdb --core libreoffice-writer.core

  • 87% reduction (204 MB vs 1.6 GB)

• $ gdb -ex ‘run’ /usr/local/bin/gdb

  • 86% reduction (11 MB vs 80 MB)

Valgrind's lazy loading implementation differs from GDB's by not requiring any indices. Valgrind simply postpones downloading and reading of shared library debuginfo until it needs specific address or symbol information from a shared library. There are cases where Valgrind ends up requiring debuginfo for most shared libraries linked to some process. In that case, the performance of eager and lazy loading will be similar. In these cases, we can at least be sure that Valgrind is actually making use of every debuginfo it downloads. In other cases, we see significant reductions in the amount of debuginfo downloaded with lazy loading.

• $ valgrind /usr/bin/gnome-calendar

  • 94% reduction (279 MB vs 4.8 GB)

• $ valgrind /usr/local/bin/gdb

  • 41% reduction (47 MB vs 80 MB)

• $ valgrind /usr/bin/virt-manager

  • 15% reduction (273 MB vs 323 MB)

Lazy loading is now supported in Valgrind since Fedora 37. GDB's lazy loading is currently in development and will be coming to Fedora once complete.