Continuous integration is a hot topic these days, and the GNU Project Debugger is keeping up with the trend. Who better to serve as a role model for tracking and exterminating bugs than a debugger?
The GDB Buildbot started as a pet project back in 2014 but is now an integral part of the development process. It provides an infrastructure to test new commits pushed to the official repository, as well as a service (which we call try builds) for developers to submit their proposed changes. In this article, I share the story of our Buildbot instance, where we are right now in terms of functionality, and the plans (and challenges) for the future.
Continuous integration
First things first: It is important to define what continuous integration is. According to Wikipedia:
[...] continuous integration (CI) is the practice of merging all developers' working copies to a shared mainline several times a day. [...]
Not very clear. In other words, CI is an automated system that continuously compiles and tests a project's codebase. It is common for CI systems to test every single commit in a project. This way, if something breaks, the developers are able to fix the problem much faster. Usually, the CI test is done before the commit is actually integrated into the codebase. The GDB project, however, uses a different approach: CI is performed after the code has been pushed upstream.
Once upon a time
Back in 2014, the GDB project did not have a continuous integration tool. Developers kindly provided testsuite results and reported regressions in the code, often using their own machines. However, these developers had limited resources and could not test various architectures simultaneously. Compilation failures were often not caught in systems that are not widely used. Ultimately, this issue caused delays and annoyances during the release process (or in the worst cases) after GDB was released.
In an attempt to mitigate this problem, the GDB Buildbot was set up. Only GNU/Linux running on Intel/AMD 32 and 64-bit was supported at the beginning, but the community quickly started to contribute toward support other machines and architectures. The initial setup compiled and tested the code using common configure flags, but developers still needed to consult the web page in order to know the results.
Over time, the instance has been improved and new features were added, including email notifications whenever a commit introduced a compilation failure, and email notifications to the gdb-testers mailing list containing the results of each testsuite run.
Perhaps one of the most useful features was the try build system.
And try builds for all
The community started noticing the benefits of having a CI system, but we wanted more. Buildbot can test patches even before they are pushed to the master branch in the repository, which can be a useful tool for the developer. For example, a developer may not have access to a lot of different architectures to test a patch on. For this reason, we enabled the try build feature on Buildbot. Access to try builds is restricted to GDB developers who have push access to the repository; several of them already use this service daily.
When a developer submits a patch to the try build service, Buildbot will build, test and send the developer email containing the results. This setup is a convenient way to quickly know if the change breaks something in the project.
Challenges: A Schrödinger test
Perhaps the biggest challenge we have right now is what we call racy (or non-deterministic) tests. These tests can pass or fail depending on several factors, such as race conditions in their code, system load, and parallelism. As a community, we have been fixing these problems for a long time now. Even though we have made a lot of progress, there is a long way ahead. These tests cause the GDB Buildbot results to be unpredictable, which ultimately undermines the developers' trust in the CI system.
The difficulty in obtaining machines for architectures and systems that are not so widely used is another challenge. At the time of this writing, we have machines building for Intel/AMD 32 and 64-bit, S/390x, SPARC v9, AArch64, and ARMhf. These machines are testing GNU/Linux, FreeBSD, and other operating systems.
Last, but not least, we can mention the fact that the infrastructure is maintained by one person only (yours truly). This setup has always been this way, but it can become a problem if I am too busy with more pressing work.
A look at the horizon
The GDB Buildbot instance has proved to be an important part of the community infrastructure. Once the problem with racy tests is solved, we will be able to:
- Send email notifications with test results directly to our main mailing list, gdb-patches.
- Send email notifications with test results directly to commit authors.
- Implement a way to perform the CI tests before the commit is integrated into our codebase.
Another interesting project being developed is Bunsen. Its purpose is to provide a better way to analyze the testsuite results generated by the DejaGNU framework, which is used by GDB. In fact, Bunsen is tested through GDB Buildbot's own test results. Once Bunsen is considered mature enough, this tool will certainly be integrated into our instance.
Conclusion
The GDB Buildbot is an ongoing effort to improve code quality and help developers spend less time looking for regressions. There are many possibilities for enhancing the service and lots of ideas for new features. If you are looking for a way to contribute to GDB, helping with the CI system is certainly a good way to start.
Last updated: June 29, 2020