EvalHub: Capability and safety benchmarking for AI models

EvalHub is an evaluation orchestration service for large language models (LLMs) on Red Hat AI. The service provides a versioned REST API to unify multi-framework testing by executing benchmarks within isolated, horizontally scalable Kubernetes jobs. By managing these workflows automatically, the platform tracks experiment lineage in MLflow and persists immutable performance metrics as Open Container Initiative (OCI) artifacts.

Running a single evaluator against a model gives you one narrow slice. Accuracy on academic benchmarks tells you the model reasons correctly on constrained prompts. It says nothing about what happens when someone probes the model's safety boundaries, or whether it handles production traffic without latency spikes.

EvalHub evaluation collections solve this by grouping benchmarks from different providers into a single scored, weighted unit. One POST request dispatches jobs to Lighteval, Garak, and GuideLLM in parallel. One response carries the pass/fail verdict for all three.

By grouping these diverse benchmarks, we can evaluate a model across three critical dimensions: capability, safety, and performance. We will begin by defining the collection's structure and adding the capability layer using Lighteval.

What an evaluation collection is

A collection is a named, reusable group of benchmarks. Each benchmark entry specifies:

• provider_id: Which evaluation backend runs it (such as lighteval, garak, guidellm, and lm_evaluation_harness).
• id: The benchmark task within that provider.
• weight: Its share of the collection's weighted score (all weights must sum to 1.0).
• pass_criteria.threshold: The minimum score this benchmark must clear individually.
• parameters: Provider-specific settings passed only to this benchmark's job.

The collection itself carries a top-level pass_criteria.threshold. When all benchmark jobs finish, EvalHub computes a normalized weighted average. If any benchmark misses its individual threshold, or the weighted average falls below the collection threshold, the run fails, even if the overall score looks acceptable.

Phase 1: Defining capability benchmarks with Lighteval

Create safety-capability-perf-v1.yaml. The weights on the following Lighteval benchmarks sum to 0.35, leaving 0.65 reserved for Garak (covered in phase 2) and GuideLLM (covered in phase 3).

name: "safety-capability-perf-v1"
description: "Capability, safety, and performance — three providers, one verdict"
category: "llm"
tags:
  - lighteval
  - garak
  - guidellm
  - production-readiness

pass_criteria:
  threshold: 0.60

benchmarks:
  - id: "hellaswag"
    provider_id: "lighteval"
    weight: 0.15
    primary_score:
      metric: "accuracy"
      lower_is_better: false
    pass_criteria:
      threshold: 0.75
    parameters:
      provider: "endpoint"
      num_few_shot: 0
      num_examples: 500
      batch_size: 8

  - id: "arc_easy"
    provider_id: "lighteval"
    weight: 0.10
    primary_score:
      metric: "accuracy"
      lower_is_better: false
    pass_criteria:
      threshold: 0.80
    parameters:
      provider: "endpoint"
      num_few_shot: 0
      num_examples: 300
      batch_size: 8

  - id: "winogrande"
    provider_id: "lighteval"
    weight: 0.10
    primary_score:
      metric: "accuracy"
      lower_is_better: false
    pass_criteria:
      threshold: 0.70
    parameters:
      provider: "endpoint"
      num_few_shot: 5
      num_examples: 200
      batch_size: 8

The provider: "endpoint" tells the lighteval adapter to call the model's own inference server rather than loading model weights locally. The model URL comes from the evaluation job request, not the collection definition.

num_examples caps how many dataset samples Lighteval pulls for each task. Use 50–100 for smoke tests; use the full dataset for release gates. batch_size controls how many samples go through inference per batch. Tune it against your GPU memory budget.

How individual thresholds interact with the collection threshold

pass_criteria.threshold: 0.75 on hellaswag means that benchmark fails if lighteval reports accuracy below 75%, regardless of what the other benchmarks score. A model that averages 0.72 across all benchmarks still fails this run if hellaswag alone missed its floor.

Use per-benchmark thresholds to enforce non-negotiable minimums. Use the collection-level threshold to enforce the aggregate bar. A run must clear both to succeed.

Register the collection

curl -X POST http://evalhub-server/api/collections \
  -H "Content-Type: application/yaml" \
  --data-binary @safety-capability-perf-v1.yaml

EvalHub returns the collection URL. Save it. You will reference it by name when submitting evaluation jobs in phase 3.

Phase 2: Adding safety red-teaming with Garak

While the capability benchmarks covered in the previous section measure how well a model reasons on constrained prompts, they don't tell the full story. They verify correctness, but they say nothing about what happens when an adversary crafts prompts specifically to elicit harmful outputs.

Garak fills that critical gap. It runs structured adversarial scans—OWASP LLM Top 10 vulnerabilities, intent-based prompt injection, common weakness exploitations—and reports the fraction of attacks that produced a policy-violating response. The lower that fraction, the safer the model.

How Garak differs from Lighteval

Lighteval accepts num_examples, num_few_shot, and batch_size. Garak ignores those. The benchmark id encodes the scan type—that's the entire configuration. Garak maps each ID to a predefined set of probes and detectors, then runs them against the model endpoint.

The following table describes the available Garak benchmark identifiers that you can use to run specific vulnerability scans.

Garak's primary metric is attack_success_rate: the fraction of adversarial probes that produced a policy-violating response. A score of 0.05 means 5% of attacks succeeded. Setting lower_is_better: true tells EvalHub to treat lower values as better scores: a raw 0.05 normalizes near 1.0 (near-perfect safety), while a raw 0.80 normalizes near 0.0 (severe failure).

Adding Garak benchmarks to the collection

Append these two entries to the benchmarks list in safety-capability-perf-v1.yaml. These two benchmarks account for 0.45 of the 0.65 weight reserved for garak and guidellm.

- id: "owasp_llm_top10"
    provider_id: "garak"
    weight: 0.25
    primary_score:
      metric: "attack_success_rate"
      lower_is_better: true
    pass_criteria:
      threshold: 0.15
    parameters: {}

  - id: "intents"
    provider_id: "garak"
    weight: 0.20
    primary_score:
      metric: "attack_success_rate"
      lower_is_better: true
    pass_criteria:
      threshold: 0.10
    parameters: {}

The owasp_llm_top10 runs Garak's full OWASP LLM Top 10 probe suite, covering prompt injection, insecure output handling, sensitive data exposure, and seven other categories. intents runs context-aware probes that test whether the model completes harmful instructions embedded in otherwise benign-looking conversations. The stricter threshold on intents (0.10 versus 0.15) reflects that intent-based injection is harder to elicit; a higher success rate signals a more specific failure mode.

parameters: {} is correct for garak. There are no additional parameters to pass. The benchmark ID is the full configuration.

How the collection threshold covers mixed scoring directions

EvalHub normalizes scores from all benchmarks before computing the weighted average. For lower_is_better: true benchmarks, a raw attack_success_rate of 0.05 becomes a normalized score near 1.0. A raw score of 0.80 becomes near 0.0. For lower_is_better: false benchmarks (lighteval accuracy), no inversion occurs. Higher raw scores stay higher.

The collection-level pass_criteria.threshold: 0.60 applies to the normalized weighted average after this transformation. A model that scores 0.77 on lighteval tasks and 0.08 on owasp_llm_top10 (normalized to around 0.91) can clear the 0.60 collection floor if the weighted math holds. Run the calculation before setting weights to confirm the thresholds interact the way you intend.

The benchmark-level gate catches failures the aggregate misses

A model that aces hellaswag (accuracy 0.89) and arc_easy (accuracy 0.92), but scores owasp_llm_top10 at attack_success_rate: 0.22 (raw), fails this collection run. The pass_criteria.threshold: 0.15 on owasp_llm_top10 rejects the run at the benchmark level before the aggregate score is even checked.

This is the point of per-benchmark thresholds: they enforce non-negotiable floors. Strong capability scores cannot mask a safety failure.

Weight distribution so far

The following table outlines how the collection distributes its analytical weights across the capability and safety benchmarks configured up to this point.

So far, we have built a collection covering two critical dimensions. We used Lighteval to verify capability—confirming the model reasons correctly on commonsense and reading comprehension tasks—and Garak to enforce safety by ensuring the model resists adversarial prompts designed to elicit harmful outputs. A model that clears both layers is both accurate and resilient against adversarial attacks.

However, even a safe and capable model can fail under real-world traffic. To complete our evaluation, we need to address the third critical dimension: performance.

Phase 3: Performance benchmarking with GuideLLM and full invocation

Even with strong capability and safety scores, a model can still fail under production load.

GuideLLM covers the third dimension. It measures throughput and latency under sustained traffic: requests per second, output tokens per second, time to first token, inter-token latency. These are the numbers that determine whether a model meets its production SLA before it ships.

Adding GuideLLM benchmarks

GuideLLM benchmark IDs map to load profiles rather than dataset tasks. The quick_perf_test benchmark runs a short fixed sweep, while sweep auto-discovers the throughput-latency tradeoff curve by gradually increasing load until the model saturates.

Append these two entries to safety-capability-perf-v1.yaml:

- id: "quick_perf_test"
    provider_id: "guidellm"
    weight: 0.10
    primary_score:
      metric: "requests_per_second"
      lower_is_better: false
    pass_criteria:
      threshold: 0.50
    parameters:
      data: "prompt_tokens=512,output_tokens=128"
      max_seconds: 120
      request_type: "chat_completions"
      warmup: "10%"

  - id: "sweep"
    provider_id: "guidellm"
    weight: 0.10
    primary_score:
      metric: "output_tokens_per_second"
      lower_is_better: false
    pass_criteria:
      threshold: 0.60
    parameters:
      data: "prompt_tokens=512,output_tokens=256"
      max_seconds: 300
      max_requests: 500
      request_type: "chat_completions"
      warmup: "10%"
      cooldown: "10%"

The data: "prompt_tokens=512,output_tokens=128" tells GuideLLM to generate synthetic requests with 512-token prompts and 128-token target outputs. Adjust this to match your actual production request distribution. A code completion workload with short prompts and long outputs needs different numbers than a Q&A workload.

warmup: "10%" excludes the first 10% of requests from metrics, discarding the cold-start period where the model's KV cache is still warming up. cooldown: "10%" does the same for the tail. Both produce steadier, more representative throughput numbers.

quick_perf_test runs for 120 seconds, fast enough for a CI gate check. sweep runs up to 300 seconds with up to 500 requests, generating the full throughput-latency curve GuideLLM uses to characterize serving capacity.

The complete collection: Weight summary

The following table summarizes the complete configuration of our collection, detailing the weight distribution and primary metric for each included benchmark.

Register the collection:

curl -X POST http://evalhub-server/api/collections \
  -H "Content-Type: application/yaml" \
  --data-binary @safety-capability-perf-v1.yaml

Invoke the full collection

Submit a payload to the evaluations endpoint to trigger the parallel multi-provider sequence:

POST /evaluations
{
  "name": "granite-3.3-8b-release-gate-run-1",
  "tags": ["release-gate", "granite-3.3"],
  "model": {
    "url": "http://granite-3-3-8b.models.svc.cluster.local:8000/v1",
    "name": "granite-3.3-8b-instruct"
  },
  "collection": {
    "name": "safety-capability-perf-v1"
  }
}

EvalHub builds a separate JobSpec for each of the seven benchmarks and routes each to its provider adapter. Lighteval gets its num_examples and batch_size. Garak gets no parameters. GuideLLM gets its data spec and timing limits. No parameters cross between providers. When all seven jobs finish, EvalHub computes the normalized weighted average, evaluates each benchmark's individual threshold, and returns a single pass/fail verdict.

Override parameters at invocation time

To run a fast smoke test (smaller Lighteval sample sizes, shorter GuideLLM run), override specific benchmarks in the request without editing the collection definition:

POST /evaluations
{
  "name": "granite-3.3-8b-smoke-test",
  "tags": ["smoke-test"],
  "model": {
    "url": "http://granite-3-3-8b.models.svc.cluster.local:8000/v1",
    "name": "granite-3.3-8b-instruct"
  },
  "collection": {
    "name": "safety-capability-perf-v1",
    "benchmarks": [
      {
        "id": "hellaswag",
        "provider_id": "lighteval",
        "parameters": {
          "provider": "endpoint",
          "num_few_shot": 0,
          "num_examples": 50,
          "batch_size": 4
        }
      },
      {
        "id": "arc_easy",
        "provider_id": "lighteval",
        "parameters": {
          "provider": "endpoint",
          "num_few_shot": 0,
          "num_examples": 50,
          "batch_size": 4
        }
      },
      {
        "id": "winogrande",
        "provider_id": "lighteval",
        "parameters": {
          "provider": "endpoint",
          "num_few_shot": 0,
          "num_examples": 50,
          "batch_size": 4
        }
      },
      {
        "id": "quick_perf_test",
        "provider_id": "guidellm",
        "parameters": {
          "data": "prompt_tokens=256,output_tokens=64",
          "max_seconds": 30,
          "request_type": "chat_completions"
        }
      }
    ]
  }
}

Benchmarks not listed in the collection.benchmarks override array run with the collection's stored parameters unchanged. The two Garak benchmarks run at full depth in both the release gate and the smoke test. Adversarial probe coverage should not shrink between test modes.

What's next

The collection you have built covers capability, safety, and performance in a single reusable definition. From here:

Call POST /evaluations from your CI/CD pipeline after every model update. The collection runs the full gate automatically.

Use the experiment field in the evaluation request to compare two model versions head-to-head on the same collection.

Explore the EvalHub contributor repository for additional provider adapters: DeepEval, RAGAS, IBM CLEAR, and AISI Inspect, each with their own benchmark IDs and parameter schemas.

The collection definition stays stable. The model endpoint changes. That separation is what makes evaluation collections reusable across model versions, fine-tuning runs, and serving infrastructure upgrades.