Introducing Tempo Monolithic mode

Starting with Red Hat OpenShift distributed tracing 3.2 (tempo-operator v0.10.0), the Tempo operator supports an additional mode of deploying Tempo: the Tempo Monolithic mode, as a tech preview feature.

In this article, we introduce Tempo Monolithic mode, describe the pros and cons, and present example deployment manifests.

What is Tempo Monolithic mode?

In Tempo Monolithic mode all core components such as compactor, distributor, ingester, querier, and query-frontend of Tempo are contained in a single binary, in a single container. This vastly simplifies the deployment, as only a single pod is created, and avoids potential issues arising from distributed deployments such as connectivity issues between pods or nodes, scheduling issues, etc. Additionally, this mode supports storing traces in-memory, and in a Persistent Volume. However, this mode comes at the expense of scalability: this mode does not scale horizontally. To scale your Tempo deployment horizontally, continue using the Tempo Microservices deployment via the TempoStack Custom Resource (CR).

The Monolithic mode is ideal for small deployments, demo and test setups, and is the recommended migration path of the Red Hat OpenShift distributed tracing platform (Jaeger) all-in-one deployment.

Migrating between TempoStack and TempoMonolithic

For Tempo deployments using object storage, you can switch between TempoStack and TempoMonolithic by using the same storage secret (don't forget to stop the previous deployment). Migration from/to a TempoMonolithic instance using in-memory or Persistent Volume storage is not supported.

Example Deployment using in-memory storage

The following manifest creates a TempoMonolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a 2 GiB tmpfs volume (in-memory storage) and exposing Jaeger UI via a Route:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: sample
spec:
  storage:
    traces:
      backend: memory
      size: 2Gi
  jaegerui:
    enabled: true
    route:
      enabled: true

Once the pod is ready, you can send traces to tempo-sample:4317 (OTLP/gRPC) and tempo-sample:4318 (OTLP/HTTP) inside the cluster. The Tempo API is available at tempo-sample:3200 inside the cluster.

Jaeger UI is available at the location of the tempo-sample-jaegerui Route (Navigate to the OpenShift Console → Networking → Routes).

To persist the traces in a Persistent Volume instead, use the following options in the storage block:

spec:
  [...]
  storage:
    traces:
      backend: pv
      size: 10Gi

Configuring retention

Tempo supports retention based on time. You can configure the maximum block age using the following spec:

spec:
  [...]
  extraConfig:
    tempo:
      compactor:
        compaction:
          block_retention: 24h

Note that this does not translate to the time of any trace or span. Internally, the ingester component of Tempo batches traces up to max_block_bytes (default: 500MB) or max_block_duration (default: 30m) into a block and flushes it to the configured storage. Another component, the compactor, combines (merges) multiple blocks in a time window of compaction_window (default: 1h) to a new block, and marks the input blocks for deletion. The block_retention (default: 14d) setting specifies at which block age (last modification time of a block) blocks are deleted.

Therefore, the block_retention is only indirectly related to the last received trace—due to (possibly multiple) compaction cycles, traces may be stored longer than defined by the block_retention setting. For more information about the available settings, refer to these docs.

Authentication and authorization

Tempo can be configured in single-tenancy (default) or multi-tenancy mode.

For single-tenancy, the Jaeger UI route is protected by an oauth-proxy sidecar. By default, any user or service account with permissions to list pods in the namespace of the Tempo instance, can access Jaeger UI. To change this, you can set the Subject Access Review (sar) option of the spec.jaegerui.authentication.sar field:

spec:
  [...]
  jaegerui:
    authentication:
      sar: '{"namespace": "<my_namespace>", "resource": "pods", "verb": "get"}'

For details on the SAR setting format, please refer to these docs.

For multi-tenancy, the access rules can be configured via ClusterRole objects. The following example CR deploys a TempoMonolithic instance with multi-tenancy and two tenants, dev and prod:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: multitenant
spec:
  jaegerui:
    enabled: true
    route:
      enabled: true
  multitenancy:
    enabled: true
    mode: openshift
    authentication:
    - tenantName: dev
      tenantId: 1610b0c3-c509-4592-a256-a1871353dbfa
    - tenantName: prod
      tenantId: 1610b0c3-c509-4592-a256-a1871353dbfb

The tenantName should be a human-readable identifier of the tenant, and the tenantId must be a unique identifier. Tempo uses the tenantId to distinguish between tenants in the trace storage. If no storage is specified in the TempoMonolithic CR, the operator defaults to in-memory storage.

To grant users and ServiceAccounts permission to read and write traces, please follow the instructions in the documentation: Configuring Multitenancy. Configuring role-based access control (RBAC) in TempoMonolithic deployments is identical to configuring RBAC in TempoStack deployments.