Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

A guide to the open source distributed tracing landscape

May 1, 2019
Juraci Paixão Kröhling
Related topics:
Microservices
Related products:
Developer Tools

Share:

    Getting started with distributed tracing can be a daunting task. There are many new terms, frameworks, and tools with apparently overlapping capabilities, and it's easy to get lost or sidetracked. This guide will help you navigate the open source distributed tracing landscape by describing and classifying the most popular tools.

    Although tracing and profiling are closely related disciplines, distributed tracing is typically understood as the technique that is used to tie the information about different units of work together—usually executed in different processes or hosts—in order to understand a whole chain of events. In a modern application, this means that distributed tracing can be used to tell the story of an HTTP request as it traverses across a myriad of microservices.

    Most of the tools listed here can be classified as an instrumentation library, a tracer, an analysis tool (backend + UI), or any combination thereof. The article "The difference between tracing, tracing, and tracing" is a great resource in describing these three faces of distributed tracing.

    For the purposes of this guide, we'll define instrumentation as the library that is used to tell what to record, tracer as the library that knows how to record and submit this data, and analysis tool as the back end that receives the trace information. In the real world, these categories are fluid, with the distinction between instrumentation and tracer not always being clear. Similarly, the term analysis tool might be too broad, as some tools are focused on exploring traces and others being complete observability platforms.

    This guide lists only open source projects, but there are several other vendors and solutions worth checking out, such as AWS X-Ray, Datadog, Google Stackdriver, Instana, LightStep, among others.

    Apache SkyWalking

    InstrumentationTracerAnalysis tool
    ✗✗✓

    Apache SkyWalking was initially developed in 2015 as a training project to understand distributed systems. It has since become prevalent in China and aims to be a complete Application Performance Monitoring platform (APM), focusing heavily on automatic instrumentation via agents and integration with existing tracers, such as Zipkin's and Jaeger's, or with infrastructure components like service meshes. SkyWalking was recently promoted to a top-level project at the Apache Foundation.

    Apache (Incubating) Zipkin

    InstrumentationTracerAnalysis tool
    ✓✓✓

    Apache (Incubating) Zipkin was initially developed at Twitter and open sourced in 2012. It's one of the most mature open source tracing systems and has inspired pretty much all of the modern distributed tracing tools. It's a complete tracing solution, including the instrumentation libraries, the tracer, and the analysis tool. The propagation format, called B3, is the current lingua franca in distributed tracing, as well as its data format, which is natively supported by other tools, such as Envoy Proxy on the producing side and other tracing solutions on the consuming side. One of Zipkin's strengths is the number of high-quality framework instrumentation libraries.

    Haystack

    InstrumentationTracerAnalysis tool
    ✗✓✓

    Haystack is a tracing system with APM-like capabilities, such as anomaly detection and trend visualization. Originally developed at Expedia, the architecture has a clear focus on high-availability. Haystack leverages OpenTracing as its main instrumentation library and add-on components like Pitchfork can be used to ingest data in other formats.

    Jaeger

    InstrumentationTracerAnalysis tool
    ✗✓✓

    Jaeger was initially developed at Uber, open sourced in 2017, and moved to the Cloud Native Computing Foundation (CNCF) soon after. The inspiration from Dapper and Zipkin can be seen in Jaeger's original architecture, data model, and nomenclature, but it has evolved beyond that. For the instrumentation part, Jaeger leverages the OpenTracing API, which has been a first-class citizen since the beginning. The analysis tool is very lightweight, making it ideal for development purposes and for highly elastic environments (e.g., multi-tenant Kubernetes clusters), and it is the default tracer for tools like Istio.

    OpenCensus

    InstrumentationTracerAnalysis tool
    ✓✓✗

    Initially developed at Google based on its internal tracing platform, OpenCensus is both a tracer and an instrumentation library. Its tracer can be connected to "exporters," sending data to open source analysis tools such as Jaeger, Zipkin, and Haystack, as well as to vendors in the area, such as Instana and Google Stackdriver. In addition to the tracer, an OpenCensus Agent is available that can be used as an out-of-process exporter, allowing the instrumented applications to be completely agnostic from the analysis tool where the data ends up. Tracing is one side of OpenCensus, with metrics completing the picture. It's not as rich in terms of framework instrumentation libraries yet, but that will probably change once the merge with the OpenTracing project is completed.

    OpenTracing

    InstrumentationTracerAnalysis tool
    ✓✗✗

    If there's something close to a standard on the instrumentation side of distributed tracing, it's OpenTracing. This project, hosted at the CNCF was started by people implementing distributed tracing systems in a variety of scenarios: as vendors, as users, or as developers of in-house implementations. On one side of the project, there are many framework instrumentation libraries such as for JAX-RS, Spring Boot, or JDBC. On the other side, several tracers fully support the OpenTracing API, including Jaeger and Haystack, as well as for well-known vendors in the area, such as Instana, LightStep, Datadog, and New Relic. Compatible implementations also exist for Zipkin.

    OpenTracing + OpenCensus

    InstrumentationTracerAnalysis tool
    ?/✓?/✓✗

    It was recently announced that OpenTracing will be merging efforts with OpenCensus. While it's still not clear what the future tool will look like, or even how it will be named, this is certainly something to keep on the radar. A tentative roadmap has been published along with some concrete proposals in terms of code, showing the direction this new tool will follow.

    Pinpoint

    InstrumentationTracerAnalysis tool
    ✓✓✓

    Pinpoint was initially developed at Naver in 2012 and open sourced in 2015. It contains APM capabilities, featuring network topology, JVM telemetry graphs and trace views. Instrumentation is done exclusively via agents and can be extended via plugins. The upside of this approach is that instrumentation does not require code changes; but on the downside, it lacks support for explicit instrumentation. Pinpoint works with PHP and JVM-based applications, where it has broad support for frameworks and libraries.

    Veneur

    InstrumentationTracerAnalysis tool
    ✓✓✗

    The Veneur project was started by Stripe, and it is described as a pipeline for observability data. It deviates from almost all other tools in this guide in that it's very opinionated about what observability should be about: spans. It comes with a set of local agents (called "sinks") that are able to receive spans, extracting or aggregating data from them, sending the outcome to external systems like Kafka. To better achieve that, Veneur comes with its own data format, SSF. Metrics can either be embedded into the spans, or synthesized/aggregated based on "regular" span data.

    Dapper

    The Dapper distributed tracing solution originated at Google and is described in a paper from 2010. It is a common ancestor to most of the tools listed here, including Zipkin, Jaeger, Haystack, OpenTracing, and OpenCensus. Although Dapper doesn't exist as a solution you can download and install, the paper is still a good reference of the primitives used in modern distributed tracing solutions, as well as the reasoning behind some of the design decisions.

    W3C Trace Context

    One of the big problems with the current distributed tracing ecosystem is interoperability between applications instrumented using different tracers. To solve this problem, the Distributed Tracing Working Group was formed at the World Wide Web Consortium (W3C) to work on the Trace Context recommendation for the propagation format.

    Overview of all projects

    ProjectInstrumentationTracerAnalysis tool
    Apache SkyWalking✗✗✓
    Apache (Incubating) Zipkin✓✓✓
    Haystack✗✓✓
    Jaeger✗✓✓
    OpenCensus✓✓✗
    OpenTracing✓✓✗
    OpenTracing + OpenCensus?/✓?/✓✗
    Pinpoint✓✓✓
    Veneur✓✓✗

    Juraci Paixão Kröhling will present "What are my microservices doing?" at Red Hat Summit, Thursday, May 9, 11:00 a.m.-11:45 a.m. This talk will look at some of the challenges presented by microservices architecture, including the observability problem, where it is hard to know which services exist, how they interrelate, and the importance of each one.

    If you haven’t registered yet, visit Red Hat Summit to sign up. See you in Boston!

    Last updated: June 4, 2024

    Recent Posts

    • GuideLLM: Evaluate LLM deployments for real-world inference

    • Unleashing multimodal magic with RamaLama

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue