Red Hat AI

ai-ml
Article

Smarter data generation for faster Speculator training

Shubhra Pandit +1

Learn how smarter data generation strategies can reduce the cost and time needed to train high-quality speculator models for speculative decoding. This post shares findings on cross-distillation, training efficiency, and production inference gains that deliver faster LLM serving with no loss in output quality.

Red Hat AI Inference Server
Article

Beyond regex: Harvesting security logic with LLMs

Itay Krispin +1

Learn how Red Hat's SastAI initiative, in collaboration with NVIDIA, automates false positive identification in static application security testing (SAST) using generative AI. By employing an agentic, multi-stage research workflow, SastAI reduces noise and improves triage efficiency. Discover the pattern harvesting methodology that greatly enhances the SastAI solution, now offering a tighter solution with better knowledge and reasoning.

A stylized illustration representing an artificial neural network, set against a dark purple background within a slightly rounded, darker purple square icon shape. The neural network consists of multiple layers of interconnected nodes, depicted as glossy, spherical red orbs. Lines connect these red orbs, forming a complex web. White arrow shapes extend horizontally from the left side, pointing towards the network, suggesting input or data flowing into the system.
Article

How speculative decoding delivers faster LLM inference

Sawyer Bowerman

Learn how speculative decoding can improve the performance of large language models (LLMs) in production by using a small, fast model to generate tokens speculatively and a large model to verify them.

A stylized illustration representing an artificial neural network, set against a dark purple background within a slightly rounded, darker purple square icon shape. The neural network consists of multiple layers of interconnected nodes, depicted as glossy, spherical red orbs. Lines connect these red orbs, forming a complex web. White arrow shapes extend horizontally from the left side, pointing towards the network, suggesting input or data flowing into the system.
Article

Intelligent inference scheduling with llm-d on Red Hat AI

Madhu Goutham Reddy Ambati +1

Learn how llm-d routes each inference request to the GPU that already has the relevant data cached, cutting down on time-to-first-token, and doubling throughput without changing hardware. Discover how Red Hat's stack packages this neatly into a single Kubernetes resource.

Red Hat AI
Article

Bring your own evaluation framework to EvalHub

William Caban Babilonia +2

Learn how to onboard a custom evaluation framework into EvalHub using one class, one method, and a container image. This guide covers the contract, data structures, and a complete minimal adapter.

Red Hat AI
Article

Understanding evaluation collections in EvalHub

William Caban Babilonia +2

Learn how to read an existing system collection, understand its threshold logic, and build your own collection that encodes your actual measurement strategy with thresholds that mean something.

Featured image for vLLM interference article.
Article

Speculators v0.5.0: DFlash support and online training

Helen Zhao +2

Speculators v0.5.0 introduces DFlash support, enabling single-pass draft token generation with block diffusion for more efficient speculative decoding workflows. The release also adds unified online and offline training through vLLM’s native hidden states extraction system, improving training flexibility, version stability, and production readiness.