Make Kubernetes logs easier to correlate with traces using the log normalizer operator

The log normalizer operator for Red Hat OpenShift injects a sidecar into selected pods, captures application output without requiring application code changes, detects the input format, reassembles multiline entries, and emits a consistent structured record. The normalized record remains available through kubectl logs and can also be exported to an OpenTelemetry collector over OTLP.

Modern Kubernetes applications rarely produce logs in a consistent format. A single namespace can contain JSON and structured logs from a service, golang logfmt from a component, klog from Kubernetes libraries, Zap console output from an operator, and plain-text stack traces from a Python or Java dependency. That variety becomes a practical observability problem when an engineer just wants know what logs were produced by a trace. Tools like Korrel8r can navigate between related observability signals, but correlation depends on signals sharing useful identifiers and queryable resource information. If every component uses different field names, or if the trace and Kubernetes identity are missing from the log record, there is little for a correlation tool to match. The log normalizer operator addresses this problem at the Kubernetes boundary.

Use case: One logging contract across many components

A team can standardize logging in application code, but that approach is not always realistic. Some workloads are third-party images. Some components use different languages or logging libraries. Others already emit structured logs, but use different attribute names for the timestamp, severity, message, or trace identifiers.

Without a boundary normalizer, teams commonly choose 1 of 3 approaches:

• Maintain per-component parsing rules in the collector
• Fork or wrap upstream logging libraries and keep those changes current
• Accept inconsistent logs and lose cross-component queries and correlation

All 3 approaches create ongoing maintenance. Collector rules are coupled to the exact input format and can break when a dependency changes its output. Forks are expensive to rebase. Unstructured logs make it difficult to group recurring errors, identify new patterns, or connect a log event to the trace that caused it.

The sidecar approach moves normalization close to the workload while keeping the application unchanged. A namespace label controls the scope. A LogNormalizerConfig controls the behavior. A pod-specific annotation can opt an individual workload out.

How log normalizer works

The operator, webhook, wrapper, and normalizer sidecar form a flow illustrated in figure 1:

1. The operator selects the namespaces

The cluster administrator creates a LogNormalizerConfig with a namespace selector. The default selector used by the sample matches:

lognormalizer.opendatahub.io/inject: "enabled"

The operator reconciles that configuration into a MutatingWebhookConfiguration. The webhook is configured for pod creation in matching namespaces, so existing pods are not changed retroactively. Restart or recreate them after enabling injection.

2. The webhook adds a native sidecar and wrapper

At admission time, the webhook adds a shared emptyDir volume, an init container that copies the wrapper binary into that volume, and a native sidecar running the normalizer. The normalizer uses restartPolicy: Always, which requires a Kubernetes version that supports native sidecars. The project documents Kubernetes 1.33+ or OpenShift 4.18+ for this deployment model.

The webhook also rewrites each application container's command so the wrapper can tee both stdout and stderr. A copy still reaches the normal container output, while another copy is written to a named pipe mounted from the shared volume. This avoids log-file tailing and lets the sidecar process records as they are emitted.

For containers that do not specify a command, the webhook can resolve the image's ENTRYPOINT and CMD from the registry before applying the wrapper. If resolution fails, the pod is allowed to start without that container being wrapped.

3. The sidecar detects and normalizes the input

The normalizer identifies common formats using lightweight checks:

JSON

• Typical example: {"level":"info","msg":"request completed"}
• Normalized result: Maps common timestamp, level, and message field names

klog

• Typical example: I0722 15:04:05.000000 ...] message
• Normalized result: Extracts severity, timestamp, and message

Zap console

• Typical example: Timestamp, level, and message separated by tabs
• Normalized result: Handles controller-runtime and collector-style output

logfmt

• Typical example: level=info msg="started" ts=...
• Normalized result: Preserves key/value fields in structured

Plain text

• Typical example: Processing batch 42 of 100
• Normalized result: Supplies a timestamp and inferred severity

The emitted JSON record has stable top-level fields such as timestamp, level, message, and source_format. When available, it also includes a template_id, trace_id, span_id, Kubernetes resource attributes, and the original parsed fields under structured.

4. Multiline entries are reassembled before parsing

Stack traces and pretty-printed output are not useful when each physical line becomes a separate event. The normalizer buffers lines until a new entry begins, the configured maximum is reached, or the flush timeout expires.

Optional consolidators recognize common continuation patterns for Go panics and goroutine dumps, Java stack traces, and Python tracebacks. After reassembly, the complete entry is parsed and emitted as one log record. This is especially useful for preserving the relationship between an exception message and its stack frames.

5. Drain learns recurring message templates

The normalizer uses a Drain-style template miner to cluster structurally similar lines. Variable tokens such as IDs, paths, and changing values can become wildcards in the learned template. Each cluster receives a template ID and a count.

The template ID is not a replacement for the original message. It is additional metadata that can help an observability pipeline group recurring patterns, suppress repetitive noise, or highlight new and rare patterns. The similarity threshold and maximum cluster count are configurable through the custom resource.

How this enables Korrel8r

Korrel8r is useful when an investigation moves between signals, for example, from a trace span to the logs emitted during that span, or from a log event back to the trace that initiated it. The log normalizer operator does not install or configure Korrel8r. Instead, it prepares log records so the rest of the observability stack has consistent data to correlate.

There are 2 important parts to that relationship:

1. Trace identity: The normalizer looks for common trace and span field names, including trace_id, traceID, traceId, trace.id, otel.trace_id, and dd.trace_id. It promotes matching values to top-level trace_id and span_id fields and uses them when constructing the OTLP span context.
2. Kubernetes identity: The webhook supplies pod name, namespace, node name, and pod UID through the Downward API. The normalizer attaches those values as OpenTelemetry resource attributes when exporting the record.

With those fields available, the OTel collector can receive normalized logs and route them with the rest of the telemetry pipeline. Korrel8r can then use the shared trace and resource context when querying the configured log and trace backends. The exact backend, retention policy, and Korrel8r configuration remain deployment-specific.

The normalizer exports to two destinations when spec.otlpEndpoint is configured: Structured JSON remains on stdout for normal Kubernetes log access, and the same record is sent over OTLP gRPC to the configured collector. The normalizer also filters OTel collector debug-exporter feedback so a collector pod does not create a feedback loop by re-normalizing its own exported logs.

Set up the operator on a Red Hat OpenShift cluster

The following is a deployment pattern for a cluster where you build and push the operator and normalizer images to a registry accessible by Red Hat OpenShift.

Prerequisites

• OpenShift 4.18 or later, or Kubernetes 1.33 or later
• oc or kubectl configured for the target cluster
• A registry accessible from the cluster
• An OTel collector gRPC endpoint if you want OTLP export and Korrel8r correlation through the collector

Build and push the images

Set image names for your registry, then build and push both images:

export REGISTRY=quay.io/example
export IMG_OPERATOR=${REGISTRY}/log-normalizer-operator:latest
export IMG_NORMALIZER=${REGISTRY}/log-normalizer:latest

make docker-build IMG_OPERATOR="$IMG_OPERATOR" IMG_NORMALIZER="$IMG_NORMALIZER"
make docker-push IMG_OPERATOR="$IMG_OPERATOR" IMG_NORMALIZER="$IMG_NORMALIZER"

Use an image tag and registry policy appropriate for your environment. The images must be pullable by the operator and by injected workload pods.

Install the CRD and create the configuration

Install the custom resource definition first:

make install

Create the default configuration before starting the operator. This prevents the operator bootstrap routine from creating a default configuration with an unusable placeholder image:

apiVersion: lognormalizer.opendatahub.io/v1alpha1
kind: LogNormalizerConfig
metadata:
  name: default
spec:
  namespaceSelector:
    matchLabels:
      lognormalizer.opendatahub.io/inject: "enabled"
  sidecarImage: quay.io/example/log-normalizer:latest
  resources:
    requests:
      cpu: 50m
      memory: 32Mi
    limits:
      cpu: 200m
      memory: 128Mi
  outputFormat: json
  drain:
    similarityThreshold: 0.4
    maxClusters: 1000
    treeDepth: 4
  multiline:
    enabled: true
    maxLines: 100
    flushTimeoutMs: 2000
  consolidators:
    - go
    - java
    - python
  # Optional: send normalized logs to collector over OTLP gRPC.
  # otlpEndpoint: otel-collector.observability.svc:4317

Save the manifest as lognormalizerconfig.yaml, replace the example image and optional collector endpoint, and apply it:

oc apply -f lognormalizerconfig.yaml

The CRD defines the configuration fields and defaults. The sample in config/samples/lognormalizerconfig_default.yaml is the reference manifest shipped with the project.

Deploy the operator

Deploy the operator with the image you pushed:

make deploy IMG_OPERATOR="$IMG_OPERATOR"
oc -n log-normalizer-system rollout status deployment/log-normalizer-operator

The kustomize deployment creates the log-normalizer-system namespace, RBAC, webhook service, CRD resources, and operator deployment. On startup, the operator creates the webhook TLS secret if it does not already exist, registers the pod admission handler, and reconciles the webhook configuration for the default custom resource.

Check the operator and configuration status:

oc -n log-normalizer-system get pods
oc get lognormalizerconfig default -o yaml
oc get mutatingwebhookconfiguration log-normalizer-injector

The custom resource should report a Ready phase after reconciliation. The webhook uses failurePolicy: Ignore, so a temporary operator outage does not prevent new pods from starting; those pods simply start without injection while the webhook is unavailable.

Enable injection for a namespace

Label the namespace before creating or restarting the workloads you want to normalize:

oc label namespace my-app \
  lognormalizer.opendatahub.io/inject=enabled \
  --overwrite

Create or restart the workloads. Then inspect a pod to confirm the injected containers and annotation:

oc get pod -n my-app
oc get pod -n my-app <pod-name> -o \
  jsonpath='{.metadata.annotations.lognormalizer\.opendatahub\.io/injected}{"\n"}'
oc get pod -n my-app <pod-name> -o jsonpath='{.spec.initContainers[].name}{"\n"}'

To opt out a particular pod, add this annotation to its metadata:

metadata:
  annotations:
    lognormalizer.opendatahub.io/inject: "false"

Verify normalized output and OTLP delivery

Use oc logs on the injected log-normalizer sidecar to confirm that normalized records are available through the normal Kubernetes logging path:

oc logs -n my-app <pod-name> -c log-normalizer

The sidecar output should contain JSON records with fields such as timestamp, level, message, and source_format. The application container continues to expose the original output. If an OTLP endpoint is configured, check the sidecar logs for exporter initialization and inspect the collector's received logs. From there, validate that the trace backend and log backend expose the same trace and Kubernetes resource context to Korrel8r.

Operational considerations

The sidecar is additive. It doesn't replace the OTel collector. The sidecar normalizes and optionally emits OTLP, while the collector remains responsible for routing, batching, filtering, and exporting telemetry to backends.

The demo measurements for this project showed approximately 1m CPU and 18Mi memory for an individual sidecar under the tested workload. Treat those figures as observations, not capacity guarantees. Set spec.resources for the workloads and traffic levels in your cluster, and monitor the impact of normalization and collector export.

If the webhook is unavailable, failurePolicy: Ignore preserves workload availability but also means a pod can start without the sidecar. If a namespace is labeled after its pods already exist, restart those pods so admission runs again. If a container has no explicit command and the image entrypoint cannot be resolved, then that container may not be wrapped. Check the operator log for the resolution message.

Production FAQ

Here are some common questions, and answers.

How much CPU and memory does the sidecar use?

In a live-cluster measurement summarized by the structured-logging architecture proposal, an individual sidecar averaged about 1m CPU and 18Mi memory across 5 tested pods. The operator used about 1m CPU and 35Mi memory. The combined measurement for the operator and five sidecars was 6m CPU and 123Mi memory.

These are workload-specific observations, not capacity guarantees. The same measurement showed pod memory deltas of about 17Mi to 33Mi for different monitoring components. Set spec.resources on the LogNormalizerConfig, measure representative log volume, and leave headroom for bursts. The example configuration requests 50m CPU and 32Mi memory, with limits of 200m CPU and 128Mi memory.

Does the normalizer write log files or add disk I/O?

No. The wrapper writes a copy of stdout and stderr to a named pipe on a shared emptyDir volume. The normalizer reads that pipe and emits records to stdout, with optional OTLP export. The architecture measurements reported no disk I/O from the normalizer path.

Does normalization add noticeable latency?

The architecture proposal measured sub-millisecond processing latency per line in its test environment. Treat that as a benchmark observation rather than a service-level objective. Validate latency and CPU use with the formats, multiline patterns, and log rates used by each production workload.

What happens if the operator or collector is unavailable?

The admission webhook uses failurePolicy: Ignore. If the operator is unavailable when a pod is created, the pod can start without the sidecar. Pods that already have an injected sidecar continue to run independently of the operator.

OTLP export is optional. If the sidecar cannot initialize its OTLP exporter, it reports the initialization error and continues with stdout output. Applications should not depend on the collector, Loki, or another log backend for health. Logging remains additive: Workloads write to stdout, and collection can degrade without stopping the workload.

Should every component use the sidecar?

The architecture proposal recommends a compliance model in which every component follows the common logging contract, while components may use the sidecar to meet it. Components owned by the platform and already using Go can standardize fields natively and still use the sidecar for Kubernetes resource context and template clustering. Python, vendor, and upstream components are good candidates for the sidecar because it avoids modifying or forking their logging code.

This is a deployment policy decision, not an enforcement rule implemented by the operator. The namespace selector controls where injection occurs.

Which fields should production teams standardize?

The proposed contract calls for an RFC 3339 timestamp, a canonical severity, a message body, and a logger or instrumentation name. Trace and span IDs should be present when tracing is active. Kubernetes controllers should also expose service identity such as service.name and service.namespace as resource attributes.

The current normalizer output uses names such as timestamp and message, while the architecture proposal discusses ts and msg for the target standard. Align the normalizer, collector pipeline, and backend queries before relying on field names in dashboards or Korrel8r rules. The proposal also identifies service.name injection as integration work still requiring a platform decision.

Does the normalizer redact sensitive data or sample high-volume logs?

No production redaction or sampling policy is defined by this operator. The normalizer preserves parsed structured fields and the original message, so teams must decide where to remove personal or secret data and how to control high-volume sources. Define PII and sensitive-data rules at the application or collector boundary, and consider sampling guidance for high-throughput components before enabling injection broadly.

What else should be checked before production rollout?

Pin both images to approved versions and verify that every injected workload can pull the normalizer image. Set CPU and memory requests and limits, test multiline behavior with representative failures, and add a logs pipeline to the OTel collector before configuring OTLP export. Confirm that the collector, log backend, trace backend, and Korrel8r agree on resource and trace field names.

The operator generates a self-signed webhook certificate when the TLS secret is absent. The current implementation creates certificates with a 365-day validity period, so platform owners should include certificate lifecycle and rotation in their operational runbook. Finally, confirm that the cluster supports native sidecars and that the admission failure policy matches the availability requirements for the workloads being selected.

Conclusion

The log normalizer operator provides a practical compatibility layer for heterogeneous Kubernetes logs. It lets teams normalize output at pod admission time, preserve multiline events, learn recurring templates, attach Kubernetes identity, and promote trace context without forking upstream components or writing a parser for every application.

That makes the operator complementary to Korrel8r: The normalizer supplies consistent, richly attributed log records, while the observability stack and Korrel8r use that context to connect logs with traces and other signals during an investigation.

For implementation details, configuration defaults, and the latest deployment manifests, see the log normalizer operator repository.