What 429 chaos experiments taught us about Kubernetes operator resilience

In my article Why killing pods is not enough: Testing operator reconciliation with operator-chaos, I ran operator-chaos against cert-manager. Eighteen experiments. Eighteen Resilient verdicts. The reconciliation loop detected every failure and restored the desired state within seconds. That was the good news, but here's what happened next.

We ran operator-chaos against twenty-two Kubernetes operators with 429 CLI experiments. Most of them passed the standard tests: PodKill, NetworkPartition, LabelStomping. These are the tests that traditional chaos tools run, and they produce comforting green results because they are mostly testing the built-in Kubernetes Deployment controller, not the operator itself.

Then we tried the deeper injection types: DeploymentScaleZero. ImageCorrupt. CrashLoopInject. ResourceDeletion. These tests ask whether the operator, not Kubernetes, actively enforces the desired state. The results changed. We found operators that silently stop functioning after a network partition, controllers that let their own Deployments stay at zero replicas permanently, and webhook servers that never come back after being scaled down.

Then we went further still. We built chaos-instrumented operator images with the SDK, injecting faults directly into each operator's HTTP transport layer. This found four new bugs (nil pointer panics, incorrect status reporting, cert-rotator failures) plus two CSV RBAC gaps that had already been fixed in a later release, in code paths that no external CLI experiment can reach.

The test matrix

Twenty-two operators. Twenty failure injection types across four categories. 429 total CLI experiments.

The top-line numbers look encouraging. Most operators passed most tests. But the interesting story is in the failures, because each one reveals a different class of reconciliation gap.

Finding #1: PodKill passes, DeploymentScaleZero reveals the truth

Most operators pass PodKill. When you delete a pod, the Kubernetes Deployment controller recreates it. That's core Kubernetes functionality. You're not testing the operator, you're testing the ReplicaSet controller.

DeploymentScaleZero is a different story. It patches the Deployment's spec.replicas to zero. The pod goes away, but the Deployment controller doesn't bring it back, because spec.replicas says zero is the desired state. The only thing that can restore it is whatever controller originally set the replica count.

This is where Strimzi showed its first crack. Seven experiments passed: PodKill, CrashLoopInject, ImageCorrupt, NetworkPartition, LeaderElectionDisrupt, LabelStomping, PDBBlock. All Resilient.

Then we ran DeploymentScaleZero:

[PRE-CHECK] Steady state verified (1/1 checks passed)
[INJECT]    DeploymentScaleZero: scaled Deployment amq-streams-cluster-operator-v3.2.0-8
            to 0 replicas (original: 1, stored in annotation)
[OBSERVE]   Waiting for recovery (timeout: 3m0s)
[OBSERVE]   No replica count change detected after 30s
[OBSERVE]   No replica count change detected after 60s
[OBSERVE]   No replica count change detected after 120s
[OBSERVE]   Recovery timeout reached (180s)
[POST-CHECK] Steady state failed (0/1 checks passed)
[VERDICT]   Degraded
            Recovery time: N/A (did not recover)
            Reconcile cycles: 0
            Deviations: spec.replicas=0 (expected: 1)

Degraded. The Deployment stayed at zero replicas permanently. This is expected behavior for operators that rely solely on OLM for Deployment lifecycle management. OLM creates the Deployment from the CSV at install time but does not detect or revert replica count changes to zero. The CSV stays in the Succeeded phase even with the operator completely offline.

The operators that pass this test (cert-manager, Prometheus Operator, ArgoCD, Tekton) all actively enforce replica counts in their own reconciliation loops (figure 1). They don't rely on the installer to also be the enforcer.

Finding #2: How a network partition permanently killed the Spark operator

The Spark operator finding was the one that kept us up at night. Seven experiments passed: PodKill, CrashLoopInject, ImageCorrupt, LeaderElectionDisrupt, LabelStomping, PDBBlock, webhook PodKill. Then experiment 8, NetworkPartition, ran.

[PRE-CHECK] Steady state verified (1/1 checks passed)
[INJECT]    NetworkPartition: created deny-all NetworkPolicy
[OBSERVE]   T+12s: Deployment spark-operator-controller: Available=True
[OBSERVE]   T+60s: NetworkPolicy removed, network connectivity restored
[OBSERVE]   T+75s: Deployment spark-operator-controller: Available=True
[OBSERVE]   T+90s: Deployment spark-operator-controller: Available=False
[OBSERVE]   T+120s: Recovery timeout reached
[POST-CHECK] Steady state failed (0/1 checks passed)
[VERDICT]   Failed
            Recovery time: N/A (did not recover)
            Deviations: Deployment Available=False, 0 ready replicas

Failed. Not Degraded, but Failed.

The Spark operator's liveness probe checks a health endpoint that responds independently of the informer cache state. The endpoint returns 200 as long as the Go process is running and the HTTP server is accepting connections. It does not check whether the informer cache is connected to the API server. It does not check whether the controller is actually reconciling anything.

Kubelet sees a "healthy" pod. The liveness probe passes. There's no restart. The controller sits there alive, but useless, with no reconciliation, no event processing. It's a zombie controller (figure 2).

The readiness probe eventually fails, removing the pod from Service endpoints. But readiness probe failures never trigger a pod restart, only liveness probe failures do. Because the liveness probe keeps passing, the pod stays in limbo permanently.

The fix

The liveness probe needs to check informer cache status, not just HTTP server health:

mgr.AddHealthzCheck("informer-sync", func(req *http.Request) error {
    if !mgr.GetCache().WaitForCacheSync(req.Context()) {
        return fmt.Errorf("informer cache not synced")
    }
    return nil
})

This is not done by default in controller-runtime, so every operator that uses it should add it explicitly.

Finding #3: SDK testing finds bugs the CLI cannot reach

The 429 CLI experiments are black-box tests. You inject a failure at the resource level, then observe whether the operator recovers. This catches real bugs. But it only exercises failures that happen to the operator's managed resources. It never touches the operator's own API calls.

SDK testing flips the model (Figure 3).

Instead of breaking resources from the outside, you inject faults inside the operator's HTTP transport, so every API call the reconciler makes has a configurable chance of failing. This is white-box chaos: You're testing error-handling paths inside the reconciliation loop itself.

We deployed chaos-instrumented operator images on Red Hat OpenShift Container Platform 4.21 cluster running Open Data Hub (ODH) v3.5.0-ea.1. Seven standalone component operators with transport-level fault injection. 40% failure rate on update/patch calls, 20% on create/delete. Each operator ran under chaos for more than 20 minutes.

Five operators came through clean. Two did not. The odh-model-controller hit a nil pointer panic in FindSupportingRuntimeForISvc when a ServingRuntime lookup failed under chaos. The training-operator's cert-rotator timed out on cache sync when transport failures delayed its startup sequence. Both are real bugs that would manifest in production under API server pressure or network instability (figure 4).

The SDK also supports action-level injection (failing specific reconciler actions). This found the DSC controller reporting Ready=True even when reconciler actions were silently failing, because the status conditions were being removed rather than set to degraded. DSPO hit a nil pointer panic in ExtractParams during SDK test setup, before any chaos injection even started. Six issues total, filed as Jiras where applicable. Four are confirmed bugs: DSC false-Ready (fix merged), DSPO nil pointer, odh-model-controller nil pointer, and training-operator cert-rotator timeout. Two were CSV RBAC gaps already fixed in ODH ea.2.

Finding #4: The dependency conflict problem

Getting the SDK into these operators was not trivial. Four out of seven operators had pinned k8s.io dependency versions that were incompatible with the chaos SDK's controller-runtime dependency. You cannot just go get a module that pulls in a different controller-runtime version when your operator has replaced directives pinning every k8s.io package.

The solution was building a zero-dependency sub-module called chaostransport. Its go.mod has zero dependencies on controller-runtime or any k8s.io library. It only uses the Go standard library. It slots into any operator's rest.Config without touching the dependency tree:

cfg := ctrl.GetConfigOrDie()ct := chaostransport.NewChaosTransport(chaostransport.NewFaultConfig(nil))cfg.WrapTransport = ct.WrapTransport

For Trustyai, where even the sub-module import caused version conflicts, we inlined the chaos types directly (about 250 lines of Go) to avoid any dependency change at all. The point: Transport-level injection is enough to find real bugs, and you can always make it fit.

Three patterns that predict resilience

After testing twenty-two operators across 429 experiments, plus SDK-level fault injection, three patterns emerged:

• Full spec enforcement: The operators that pass everything compare current state against desired state on every reconciliation cycle. They don't just check whether a Deployment exists, and instead compare replicas, images, labels, resource limits against the intended spec and correct any drift. Operators that use server-side apply consistently pass more experiments.
• Health checks that reflect operational state: The operators that pass NetworkPartition have liveness probes wired to their actual operational state, not just process health. The Spark Operator's zombie-controller failure is the direct consequence of checking HTTP server health instead of informer cache state.
• Resource graph awareness: Resilient operators watch their own supporting resources (Services, Leases, ConfigMaps), not just the primary custom resources they manage. When a supporting resource disappears, they detect it and recreate it immediately.

How to test your operator

Here are my recommendations for testing your operator.

Experiment 1: PodKill (baseline)

This should always pass.

$ operator-chaos init --component controller --operator my-operator \
--type PodKill > podkill.yaml
$ operator-chaos run podkill.yaml --knowledge knowledge.yaml -v

Experiment 2: DeploymentScaleZero (tests replica enforcement)

If your operator is installed uing OLM or Helm without a managing controller, this returns the Degraded status.

$ operator-chaos init --component controller --operator \
my-operator --type DeploymentScaleZero > scale-zero.yaml
$ operator-chaos run scale-zero.yaml --knowledge knowledge.yaml -v

Experiment 3: NetworkPartition (tests informer reconnection)

If you see Failed here, then your liveness probe is not wired to the informer cache.

$ operator-chaos init --component controller --operator my-operator \
--type NetworkPartition > netpart.yaml
$ operator-chaos run netpart.yaml --knowledge knowledge.yaml -v

Experiment 4: SDK transport injection (tests internal error handling)

Add the chaostransport sub-module and inject failures at the API call level:

cfg := ctrl.GetConfigOrDie()
ct := chaostransport.NewChaosTransport(chaostransport.NewFaultConfig(nil))
cfg.WrapTransport = ct.WrapTransport

Run the operator with this transport for more than 20 minutes and watch for panics, stuck reconciliation, or incorrect status reporting.

How to fix what you find

Once you've found problems, the next step is to fix it. Here are some common issues and how to repair them.

• DeploymentScaleZero failures: Add spec enforcement to your reconciliation loop. Compare the current Deployment spec against the desired spec on every cycle and correct any drift. Consider using server-side apply (SSA).
• NetworkPartition failures: Wire your informer cache status into your liveness probes using mgr.AddHealthzCheck(). This is a 10-line change that prevents the zombie-controller scenario entirely.
• SDK-found bugs: Add nil checks on all API call return values. Never assume a Get, List, or Update will succeed. If FindSupportingRuntimeForISvc returns nil, handle it before dereferencing.

In every case, add operator-chaos experiments to your CI pipeline:

$ operator-chaos suite experiments/ --knowledge knowledge.yaml \
--recursive --report-dir results/

Combined results summary

Conclusion

PodKill tells you how the Kubernetes Deployment controller works. DeploymentScaleZero tells you whether your operator works. NetworkPartition tells you whether your health checks are honest. SDK transport injection tells you whether your error handling paths are safe. Each layer finds bugs the previous one cannot reach, and most chaos test suites only run the first one.

To learn more, check out these resources:

• Documentation
• GitHub
• Full experiment library (22 operators, 429 experiments)
• Red Hat Developer Sandbox