For years, the gold standard of infrastructure engineering and chaos testing was just uptime: Either a system survived a disruption and passed, or it crashed and failed. In the era of massive cloud-native deployments, however, this binary pass or fail metric no longer tells the whole story. A Kubernetes cluster or virtualized workload can technically remain operational through an experiment, yet silently drop packets, breach internal service level objectives (SLO), or suffer catastrophic performance degradation that ruins the end-user experience.
To bridge this gap and move past superficial green checkmarks, Krkn is shifting the focus of chaos validation away from black-and-white outcomes and toward deeply detailed, actionable downtime telemetry. By unifying macro-level metrics like the new Resiliency Score feature with continuous, micro-level data from HTTP Health Checks and KubeVirt Checks, Krkn gives engineering teams a high-fidelity look into the exact duration, location, and severity of degradation during an experiment. This multidimensional approach allows teams to pinpoint the true architectural blast radius of any failure, transforming chaos testing from a guessing game into a precise, version-controlled quality gate.
Quantifying stability: Resiliency Score
Currently in beta, Krkn's Resiliency Score acts as a macro-health metric ranging from 0 to 100%. Instead of guessing how well your cluster weathered a storm, Krkn tracks live Prometheus data across your specific chaos time window and evaluates it against pre-defined SLOs. If a Prometheus alert query triggers during the test, that SLO is marked as failed. Because not all failures carry the same weight, Krkn calculates the final percentage using a weighted scoring model. By default, standard warnings are worth 1 point and critical outages are worth 3 points, but engineers have the freedom to assign custom weights to individual SLOs and add SLOs that meet their needs. This ensures that business-critical components—like authentication or payment gateways—heavily influence the final score, giving teams a nuanced calculation of actual blast radius rather than a binary green or red light.
A telemetry JSON snippet
{
"telemetry": {
"run_uuid": "717c8135-2aa0-47c9-afdf-3a6fe855c535",
"job_status": false,
"overall_resiliency_report": {
"scenarios": {
"<scenario_1>": 97,
"<scenario_2>": 97
},
"resiliency_score": 97,
"passed_slos": 26,
"total_slos": 27
},
}
}An example resiliency-report.json snippet:
{
"scenarios": [
{
"name": "<scenario_1>",
"window": {
"start": "2026-03-18T15:36:14",
"end": "2026-03-18T15:36:25"
},
"score": 97,
"weight": 1,
"breakdown": {
"total_points": 45,
"points_lost": 1,
"passed": 26,
"failed": 1
},
"slo_results": {
"10 minutes avg. 99th etcd fsync latency on {{$labels.pod}} higher than 10ms. {{$value}}s": true,
"..."
},
"health_check_results": {}
},
{
"name": "<scenario_2>",
"window": {
"start": "2026-03-18T15:36:27",
"end": "2026-03-18T15:36:19"
},
"score": 97,
"weight": 1,
"breakdown": {
"total_points": 45,
"points_lost": 1,
"passed": 26,
"failed": 1
},
"slo_results": {
"10 minutes avg. 99th etcd fsync latency on {{$labels.pod}} higher than 10ms. {{$value}}s": false,
"..."
},
"health_check_results": {}
}
]
}Real-time micro-visibility: HTTP Health Checks
The Resiliency Score provides a broad assessment, but understanding the precise user impact requires real-time insight. This is where Krkn's continuous HTTP Health Checks come in. Configured right inside your config.yaml, the system probes application endpoints at user-defined intervals (down to the second) throughout the entire lifecycle of the chaos run. Rather than just reporting that a service failed, the health check telemetry captures the exact start time, end time, and duration of the downtime, alongside HTTP status codes. It supports complex setups, including bearer tokens and credential tuples, allowing you to monitor protected API endpoints. This exact duration tracking means teams can transition from wondering whether their application recovered to measuring how many seconds it took to self-heal.
health_checks:
interval: <time_in_seconds> # Defines the frequency of health checks, default value is 2 seconds
config: # List of application endpoints to check
- url: "https://example.com/health"
bearer_token: "hfjauljl..." # Bearer token for authentication if any
auth:
exit_on_failure: True # If value is True exits when health check failed for application, values can be True/False
verify_url: True # SSL Verification of URL, default to true
- url: "https://another-service.com/status"
bearer_token:
auth: ("admin","secretpassword") # Provide authentication credentials (username , password) in tuple format if any, ex:("admin","secretpassword")
exit_on_failure: False
verify_url: False
- url: http://general-service.com
bearer_token:
auth:
exit_on_failure: False
verify_url: False Bridging the infra gap: KubeVirt virtual machine checks
Modern cloud-native environments don't just run containers. Many rely on cloud-native virtualization. To ensure that legacy workloads and virtualized infrastructure are held to the same rigorous resilience standards, Krkn includes built-in KubeVirt Checks (Virt Checks). Running continuously from the moment Krkn starts until the post-chaos wait durations finish, Virt Checks actively monitor a virtual machine instance (VMI) within targeted namespaces. By attempting real-time SSH connectivity using virtctl (or directly using worker nodes in disconnected environments), Krkn logs detailed telemetry for every VM. The output records whether a VM lost connectivity, the node it was running on, its changing IP addresses if it was rescheduled, and the exact down-time duration. Crucially, a final post Virt Check runs after the chaos injection completes, catching hidden failures by explicitly flagging any VMs that remain unreachable after the experiment concludes.
Example telemetry output:
"virt_checks": [
{
"node_name": "000-000",
"namespace": "runner",
"vm_name": "windows-vm-1",
"ip_address": "0.0.0.0",
"status": false,
"start_timestamp": "2025-07-22T13:41:53.461951",
"end_timestamp": "2025-07-22T13:42:30.696498",
"duration": 37.234547,
"new_ip_address": "0.0.0.2",
},
{
"node_name": "000-000",
"namespace": "runner",
"vm_name": "windows-vm-0",
"ip_address": "0.0.0.1",
"status": true,
"start_timestamp": "2025-07-22T13:41:49.346861",
"end_timestamp": "2025-07-22T13:43:17.949613",
"duration": 88.602752,
"new_ip_address": ""
},
{
"node_name": "000-000",
"namespace": "runner",
"vm_name": "windows-vm-1",
"ip_address": "0.0.0.2",
"status": true,
"start_timestamp": "2025-07-22T13:42:36.260780",
"end_timestamp": "2025-07-22T13:43:17.949613",
"duration": 41.688833,
"new_ip_address": ""
}
],
"post_virt_checks": [
{
"node_name": "000-000",
"namespace": "runner",
"vm_name": "windows-vm-4",
"ip_address": "0.0.0.3",
"status": false,
"start_timestamp": "2025-07-22T13:43:30.461951",
"end_timestamp": "2025-07-22T13:43:30.461951",
"duration": 0.0,
"new_ip_address": "",
}
]Catching version regressions in CI/CD
Bringing these 3 layers of telemetry together—SLO scoring, endpoint uptime, and VM connectivity—creates an active quality gate for your deployment pipeline. As an application evolves through refactors and package updates, its failure modes change. Architecture that safely handled a node failure in version 1.1 might completely fall apart in version 1.2 due to a hidden connection pool bottleneck or an improper timeout configuration.
By running identical Krkn chaos scenarios against successive application versions, development teams can use these granular telemetry reports as a standardized baseline. If version 2.0 maintains a 95% Resiliency Score with 0 seconds of HTTP endpoint failure, but version 2.1 drops to a 70% score with a 45-second VM outage, then you have immediate, quantifiable proof of an architectural regression. Instead of hunting through endless logs, developers can look at the telemetry to see exactly which alert triggered and which component stayed down, ensuring code velocity never compromises cluster stability.
Engineering a culture of continuous resilience
Ultimately, building a resilient system isn't a one-time achievement—it's a continuous practice of observation and refinement. By moving away from binary pass/fail testing and adopting Krkn's trifecta of telemetry —the Resiliency Score, HTTP Health Checks, and KubeVirt Checks—you gain the exact insights needed to understand how your applications actually behave under stress. Krkn gives you the data-driven confidence to push your infrastructure to its limits, whether you're hunting down subtle performance degradations, tracking improvements across application versions in your CI/CD pipeline, or verifying virtualized workloads.
Don't wait for a production outage to find your architectural breaking points. Check out the Krkn documentation today, set up your scoring profiles, and start quantifying your resilience.
