Language models offer impressive capabilities, but they're not always consistent. The same prompt can produce different answers, different reasoning paths, or different tool choices across runs. This variability might be acceptable in casual use, but it matters much more when models are used in products, enterprise workflows, and agentic systems. In those settings, reliability matters as much as raw capability.
Inference-time scaling (ITS) offers a practical way to address this inconsistency. Instead of accepting a single generation, ITS uses additional compute at runtime to generate multiple candidates, compare reasoning paths, and select the most accurate output before returning a final result. No retraining, no fine-tuning, no changes to model weights. The core insight: accuracy is an inference problem, not just a training problem.
That's the idea behind its_hub, Red Hat AI's open source framework for making these techniques usable in real systems. It supports methods such as best-of-N and self-consistency, as well as more advanced approaches like particle filtering and beam search, all in a common framework developers can apply incrementally.
What inference-time scaling is
ITS works by doing more during inference rather than relying on a single model call. An ITS system can generate several candidates, follow multiple reasoning paths, score outputs, or iteratively refine intermediate steps before returning a final answer.
This distinction matters because many model failures aren't caused by a lack of capability. Often the model can solve the task, but a single sampled response is brittle. A reasoning step drifts. The model returns a weaker answer over a more accurate one. It selects a plausible but wrong tool. An agent might choose the incorrect API call on one pass even though it was capable of the right choice from the start. Your models already know the answer; they don't always give it to you on the first try. Additional inference-time compute helps recover better outcomes by broadening the search space or improving the selection process.
This capability makes ITS applicable across a range of settings: improving convergence on correct answers for reasoning tasks, surfacing stronger candidates for generation tasks, and reducing the impact of any single noisy decision in agentic workflows. The tradeoff is real, but it's one that teams can choose to make selectively, applying more compute only where the problem justifies it.
The enterprise case for inference-time scaling
Inference-time scaling matters now because the market has validated it and enterprise adoption demands it. Every major provider has shipped thinking-budget controls: OpenAI, Google, Anthropic, DeepSeek, and NVIDIA all offer ways to allocate more inference-time compute for harder tasks. Inference-time compute scaling is a product category, not a research curiosity.
However, every provider's implementation is model-level: baked into one specific model, controlled by the provider's API. Customers who want ITS on their own models, on their own infrastructure, have no production option. That's the gap its_hub fills.
The need is especially urgent for agentic workloads. Fewer than 10% of organizations have moved AI agents to production, and a primary blocker is reliability. Agents fail not because the underlying models lack capability, but because tool calling is unreliable. A single wrong tool call cascades through the entire workflow. ITS with consensus-based voting addresses this issue directly, turning unreliable tool selection into dependable automation.
ITS also gives teams a way to match compute to the demands of each task. Easy questions stay fast and cheap. Hard questions get the compute they deserve. This approach makes inference a configurable budget rather than a fixed cost, and means teams can improve accuracy without retraining, without API lock-in, and without changing the model itself.
How its_hub brings ITS into practice
its_hub packages practical inference-time scaling techniques into a consistent framework so teams can experiment, compare, and integrate them without building everything from scratch. This flexibility matters because ITS isn't monolithic. Different tasks call for different strategies, and its_hub supports this range rather than locking developers into a single pattern.
Today, its_hub offers two integration paths:
- A Python SDK for direct programmatic use inside application code, which works with any OpenAI-compatible model endpoint, including vLLM.
- Inference-as-a-Service through an OpenAI-compatible HTTP API, allowing existing workflows to adopt ITS by pointing requests to the
its_hubendpoint without changing their call structure.
Both paths support reward-hub integration for feedback-driven scoring and selection, allowing developers to define custom evaluation functions and plug them directly into inference-time workflows.
Looking ahead, Red Hat is working to bring ITS into the Red Hat AI gateway as an infrastructure-level capability. The goal is to apply ITS to any model on any deployment through a single API header or parameter, with no code changes required. This work is underway and will be covered in future posts.
Techniques available in its_hub
Self-consistency generates multiple reasoning paths and selects the answer appearing most frequently across them, as shown in Figure 1. Rather than trusting a single trajectory, it lets agreement across runs influence the final result, which makes it particularly effective for reasoning tasks where the correct answer tends to recur even when individual traces vary.
Best-of-N generates multiple candidate responses and uses a scoring mechanism to select the most accurate one, illustrated in Figure 2. Unlike self-consistency, which relies on consensus, best-of-N depends on an explicit evaluation signal, whether that's a large language model (LLM)-as-a-judge, a custom reward model, a validator, or other task-specific scoring logic.
Hierarchical voting extends self-consistency to tool-calling workflows. When an agent needs to select and invoke tools, hierarchical voting samples multiple candidates in parallel and votes in stages: first on which tool to call, then on the arguments to pass. This two-level approach is especially effective for agentic tasks where the critical decision is not what text to generate, but what action to take.
Reward models act as lightweight evaluators scoring candidates based on criteria such as truthfulness, relevance, or domain fit. In best-of-N, they identify the top candidate; in self-consistency, they're optional but useful for resolving ties. Through reward-hub integration, developers can define custom scoring functions and plug them directly into inference-time workflows without touching the underlying model.
For workflows where intermediate reasoning steps matter, its_hub also supports particle filtering and beam search via its process reward model path. A process reward model scores each intermediate reasoning step rather than only the final output, making it possible to identify and prune weak trajectories before they reach a conclusion. These methods are better suited to complex multi-step tasks where early errors compound.
Why enterprises should care
For enterprise teams, the value of ITS is not benchmark improvement. It's control.
Instead, the most immediate value of ITS lies in agentic workflows where models need to choose the right action rather than simply generate text. Tool calling is the critical link: when an agent selects the wrong tool or passes the wrong arguments, the entire downstream workflow breaks. A single bad decision cascades.
To validate this capability, the Red Hat AI Innovation Team applied ITS to Red Hat OpenShift Lightspeed, an AI assistant helping operators troubleshoot and manage OpenShift clusters. OpenShift Lightspeed uses tool calling to interact with the cluster: listing pods, reading logs, inspecting network policies, and diagnosing failures across multiple steps. When tool selection is unreliable, the agent investigates the wrong resource and never recovers.
Using hierarchical voting with its_hub, the team evaluated 11 troubleshooting scenarios and found ITS improved overall pass rates from 10% to 20% on various models tested. The biggest gains came on the hardest scenarios, where log analysis tasks requiring multi-step search went from near-zero success to consistent resolution. Scenarios already reliable at baseline stayed reliable, and the most challenging ones saw the largest improvements.
These results point to a broader principle. ITS is a multiplier for model capability, not a substitute for it. It works best when the model is capable of solving the task but doesn't do so reliably on every pass. For teams building agentic systems on Red Hat AI, this reliability gap is exactly the problem ITS is designed to close.
Getting started with its_hub
its_hub supports two integration paths depending on how you want to use it.
Option 1: Python SDK
Use this option for direct programmatic control inside application code.
from its_hub import ScalingAlgorithm
from its_hub.lms import OpenAIModel
model = OpenAIModel("gpt-4")
scaler = ScalingAlgorithm("self_consistency")
response = scaler.infer(model, prompt="What is the probability of rolling a sum of 7 with two dice?")
print(response)Option 2: Inference-as-a-Service
its_hub can also run as a local or remote HTTP endpoint with an OpenAI-compatible API, allowing existing workflows to adopt it by pointing requests to the its_hub endpoint without changing their call structure.
its-iaas launch
curl http://localhost:8000/v1/chat/completions \
-d '{"model":"gpt-4", "messages":[{"role":"user","content":"..."}]}'Closing thought
Language models will always involve some degree of uncertainty. The question is not whether variability exists, but how we respond to it.
A single model call treats variability as something to accept. Inference-time scaling treats it as something to explore. This shift, from accepting one answer to searching for a better one, is what its_hub makes practical for real systems.
Explore the project
- its_hub on GitHub: Source code, documentation, and the Python SDK
- Interactive walkthrough: See ITS techniques in action with guided scenarios
- Companion demo: Try ITS with live API calls and adjustable compute budgets
- OpenShift Lightspeed experiment setup: Configuration details and code for testing ITS with OpenShift Lightspeed
- Smarter enterprise AI with inference-time scaling: An introduction to ITS concepts and early results on Red Hat AI
- How small models beat the big ones: A no-math introduction to inference-time scaling concepts
