Red Hat AI Inference

Double Nemotron 3.5 Super VL inference throughput on NVIDIA Hopper GPUs. Learn how FP8 quantization and DSpark speculative decoding boost speed and accuracy.

Explore how Red Hat AI Inference 3.5 MaaS brings multi-tenant API governance, API key authentication, and token rate limiting to LLM endpoints on AKS.

Learn how to use the open source scanner garak to test AI models for

Optimize LLM inference performance with a 71% latency reduction on NVIDIA H200 GPUs. Learn how to cut TTFT from 995ms to 287ms with smarter request routing.

Implement a regression detector for Red Hat AI Inference, catching performance regressions in CI/CD before release. Learn how in this guide.

Learn how to reduce Llama 3.1 8B Instruct model size and improve performance with W8A8 INT8 quantization.

Improve large language model inference speed with Speculators 0.6.0's FastMTP-style fine-tuning.

Cut Llama 3.1 8B VRAM by 46% without losing accuracy. Master the mechanics of INT8 W8A8 quantization, SmoothQuant, and GPTQ using llm-compressor.

Learn how P-EAGLE in Speculators v0.6.0 uses parallel drafting to reduce LLM latency. Train and deploy custom draft models with this step-by-step guide.

Learn how to prioritize mixed workloads with Red Hat AI Inference 3.5's GPU-shared flow control

Confidently deploy LLMs with Red Hat support: Learn how to determine if your model is supported by Red Hat's vLLM community.

Explore Kubernetes resources for Red Hat AI Inference on Amazon EKS, enabling intelligent routing for your model serving.

Improve model reliability at inference time with its_hub: Learn how to select accurate outputs without retraining.

Learn how to compress, serve, and benchmark LLMs with vLLM.

Explore a demo of serving a multimodal model (Qwen3-Omni) with vLLM-Omni on a single hardware accelerator.

Learn how to implement GPU-as-a-Service on Red Hat OpenShift using Kueue, NVIDIA MIG, and a custom dashboard plug-in for self-service GPU resource booking.

Learn how to optimize deployment of vLLM for various traffic shapes, including high-concurrency chat, long-context RAG, high-throughput batch, and distributed AI-grid.

Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.

Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.

Look inside Red Hat AI Inference on Amazon EKS to understand its core architectural components and Kubernetes resources.

Learn when to use llama.cpp and vLLM for local inference of large language models (LLMs). Discover the key differences, benchmarks, and use cases for each engine.

Learn how llm-d routes each inference request to the GPU that already has the relevant data cached, cutting down on time-to-first-token, and doubling throughput without changing hardware. Discover how Red Hat's stack packages this neatly into a single Kubernetes resource.

Red Hat and DeepLearning.AI have released a free hands-on course on the full LLM

Learn how to deploy and serve large language models (LLM) on Rebellions ATOM NPUs using Red Hat OpenShift AI and a certified vLLM container image on the Red Hat AI Inference Server. This post walks through the steps to set up the joint solution between Red Hat and Rebellions, including installing the Node Feature Discovery operator, the Rebellions NPU operator, creating the ATOM hardware profile in OpenShift AI, and creating the vLLM RBLN ServingRuntime.

Learn how to combine KServe and llm-d to optimize generative AI inference, improve performance, and reduce infrastructure costs. This article demonstrates the integration architecture and provides practical guidance for AI platform teams.