New Delve features in RHEL 9.2

Delve, the Go debugger, ships with the go-toolset package in Red Hat Enterprise Linux (RHEL). The go-toolset package in RHEL 9.2 contains Delve version 1.9.1, which contains many improvements and features.

Delve 1.9.1 is also the last release to contain the old versioning scheme; new releases of Delve will mirror the corresponding Go release <major>.<minor> versioning, only diverging for point releases. So, for example, the latest release of Delve is 1.20.x which corresponds to the 1.20.x release of Go. Delve releases a new minor version when the Go project releases a new version's first RC (Release Candidate). Once that version of Go is released in RHEL, the corresponding version of Delve is also released.

New features

Each new release of Delve boasts many new features, fixes, and improvements. Let's take a moment to highlight some of the new features in the 1.9.x cycle.

In cycle 1.9.x, several additions, fixes, and changes were made to Delve, such as:

• support for empty strings and exact matches in the substitute path config;
• support for gnu_debuglink section to improve support for external debug information;
• the ability to show disassembly instead of source code when single-stepping CPU instructions; and
• the option to use -per-g-hitcount for breakpoint conditions.

One other major feature is the implementation of function call injection on the ARM64 architecture. This enables developers to call a function in the program being debugged during a debugging session, a feature already available to users on the AMD64 architecture.

This feature is very powerful, but it can be tricky to implement. The next section will dig into the feature in more detail.

Function call injection

This feature requires support from the Go runtime to ensure that Delve can safely inject function calls into the Go program being debugged. Injecting a function call into a Go program is very difficult to do safely to ensure the program is not disrupted or put into a bad state. If not for coordination with the Go runtime, we would run into potential deadlocks from the garbage collector, issues with stack space in the Go routine you'd like to use to call the function, and more.

However, as mentioned above, thanks to coordination with the Go runtime, function call injection can be done safely and provide enhanced insight and interactivity during your debug session.

The coordination between Delve and the Go runtime for injecting function calls happens at a very low level. Delve and the Go runtime communicate via setting specific values in specific CPU registers and reading and writing memory in the address space of the program being debugged. The Go runtime provides this via a private function called `debugCallV2` (in current Go releases). As shown in the name, this function is versioned to allow for changing the function injection protocol in ways that may not be backward compatible.

The basic flow for function call injection on AMD64 is as follows (taken from the Go runtime documentation):

1. Check that the goroutine is in state _Grunning and that at least 256 bytes are free on the stack.
2. Push the current PC on the stack (updating SP).
3. Write the desired argument frame size at SP-16 (using the SP after step 2).
4. Save all machine registers (including flags and XMM registers) so the debugger can restore them later.
5. Set the PC to debugCallV2 and resume execution.

Once Delve invokes debugCallV2, the runtime takes over and checks that the program is at a safe location to initiate a function call, and ensures there is enough space on the stack for the function being called to execute. Once the Go runtime has completed its checks and allocated stack space for the function call, it sets the appropriate register state to communicate with Delve. Then it notifies Delve by executing a breakpoint instruction, transferring control back to Delve. The debugger then reads the program memory and register values and continues with the function call injection protocol.

If the Go runtime reports no issues or errors, Delve will continue with the function call, running to completion and obtaining the return values to display to the user. If there are errors, Delve will either remedy them if they are recoverable from the debuggers perspective or abandon the function call notifying the user as to the reason why.

Summary

Delve, the Go debugger, has introduced new features and improvements in its 1.9.x cycle. Notable features include support for empty strings and exact matches in the substitute path configuration, support for the gnu_debuglink section to enhance external debug information support, the ability to display disassembly instead of source code during CPU instruction single-stepping, and the option to use -per-g-hitcount for breakpoint conditions. The ARM64 architecture also received a significant addition with implementing function call injection. This feature allows developers to call functions within the program being debugged, enhancing interactivity and insight during debugging sessions.

The coordination between Delve and the Go runtime ensures the safe injection of function calls by leveraging low-level communication and utilizing the runtime's private function, debugCallV2. The Go runtime performs necessary checks and stack allocations before transferring control back to Delve for further processing. Function call injection in Delve provides developers with advanced debugging capabilities while maintaining the stability of the program being debugged.

A solid, reliable debugger is an invaluable tool for developers. We ship Delve alongside Go with the go-toolset package, so you have everything you need to be a productive Go developer. Be on the lookout for new features and improvements as we continue to develop and improve both the Go toolchain and Delve debugger.

Last updated: August 14, 2023