EfficiOS is pleased to announce it is now providing LTTng packages for Red Hat Enterprise Linux 7, available today as part of its Enterprise Packages portal.
EfficiOS specialises in the research and development of open source performance analysis tools. As part of its activities, EfficiOS develops the Linux Tracing Toolkit: next generation for which it provides enterprise support, training and consulting services.
What is tracing?
Tracing is a technique used to understand the behaviour of a software system. In this regard, it is not far removed from logging. However, tracers and loggers are designed to accommodate very different use cases.
Tracing specifically targets debugging, performance analysis, and monitoring use cases. Hence, tracers are typically designed from the ground up to handle the throughput associated with the capture of very low-level events that occur much more frequently than typical log messages. Examples of such events include context switches, virtual filesystem events and page allocations.
On the other hand, logging backends tend to be designed to accommodate the recording of infrequent high-level events such as user accesses, errors and warnings, database transactions, instant messaging communications, etc. Therefore, typical logging solutions are rarely able to handle finely grained logging without significantly degrading application performance.
What is LTTng?
As its name suggests, the Linux Trace Toolkit: next generation is a tracing framework that provides tracers for both user space applications and the Linux kernel. These tracers produce kernel and user space traces that can be correlated; this makes it easy for developers and system engineers to gain insight into their system’s behavior by tracking the interactions between applications and the Linux kernel.
Since LTTng is often used to track down performance and latency problems on production machines, its design is geared towards ensuring it can do so while incurring very little overhead. This focus on low intrusiveness also means that race conditions and other timing-dependant issues can generally be reproduced even when tracing is activated. LTTng’s non-blocking trace extraction also makes it production safe; faults in the LTTng infrastructure never affect the instrumented applications or the kernel.
Analyzing Traces
LTTng produces CTF traces (Common Trace Format), an open format developed in collaboration with the Multicore Association and the Linux community. This format is tailored for fast production and analysis of large datasets. Being an open format, a number of tools allowing the analysis of CTF traces have been made available. These range from Babeltrace, a command line trace reader, to full-fledged graphical viewers such as the Trace Compass project.
While CTF traces can be used as regular log files, a number of powerful performance analyses have been developed by the LTTng community over the years; most of them having been developed using the Babeltrace project’s Python library.
A number of blog posts showcasing the use of these tools are available on the official LTTng blog.
Finding the root cause of a web request latency
Full stack system call latency profiling
