Red Hat AI is releasing optimized FP8 and DSpark speculative decoding checkpoints for Nemotron 3.5 Super VL, a 120B multimodal mixture-of-experts (MoE) model built for long-running agents. These checkpoints are designed to double inference throughput on NVIDIA Hopper GPUs without compromising output accuracy.
These optimizations combine FP8 quantization using LLM Compressor with a DSpark speculative decoding model trained using Speculators, delivering substantial inference performance gains while preserving model quality.
FP8 quantization
To accelerate inference for Nemotron 3.5 Super VL, we are releasing an optimized FP8-quantized checkpoint. FP8 quantization reduces the precision of model weights and activations from BF16 to 8-bit floating point, cutting model weight memory and storage requirements in half while preserving model quality.
FP8 quantization is particularly effective on NVIDIA Hopper GPUs, which feature native FP8 Tensor Core support and deliver approximately 2× higher theoretical matrix multiplication throughput compared to BF16. Combined with reduced memory bandwidth requirements, this enables higher inference throughput and improved GPU efficiency, especially for large-scale models.
We generated the quantized checkpoints using the LLM Compressor library and its model-free post-training quantization (PTQ) workflow, leveraging the calibration-free nature of FP8 Dynamic quantization.
After evaluating several quantization configurations, we settled on quantizing the attention, expert, and Mamba projection layers of the main model, while keeping the shared experts, routers, normalization layers, and lm_head in their original precision. For the multi-token prediction (MTP) layers, we quantized the attention, expert, and shared expert layers, while leaving the normalization layers and eh_proj unquantized.
The quantization process is fully reproducible using the creation script provided in the model card.
To assess the impact of quantization on model accuracy, particularly in reasoning-intensive tasks and agentic workflows, we evaluate both the BF16 and FP8 models across a broad suite of representative benchmarks. To ensure a fair and consistent comparison, both models are evaluated under identical conditions: non-greedy sampling, four NVIDIA H200 GPUs, vLLM 0.31.0, and the Inspect AI and Harbor evaluation frameworks. The evaluation results are presented in Figure 1. The FP8 checkpoint consistently preserves model accuracy across all evaluated benchmarks.
DSpark speculator
The performance gains from quantized checkpoints can be further extended by enabling speculative decoding with parallel drafting DSpark models. Speculative decoding accelerates inference by using a smaller, faster draft model to propose multiple tokens ahead of time, which the LLM then verifies in parallel.
While the target LLM natively includes MTP layers for multi-token prediction, DSpark takes a different approach. DSpark’s speedup mainly benefits from a parallel-drafting technique. Instead of drafting tokens autoregressively, it proposes 8-16 tokens in a single forward pass.
This gain is not completely free: parallel drafting means we now lose the causality between tokens. DSpark resolves this issue by extending the parallel drafting with a lightweight Markov head to introduce dependencies between speculated tokens, allowing each token in the draft sequence to build on previous predictions rather than generating all tokens independently. As shown in Figure 2, this architecture helps DSpark to produce higher-quality candidate sequences and increase overall inference throughput.
Performance benchmarking
Quantization and speculative decoding are complementary, orthogonal inference acceleration techniques that can be combined to achieve compounded performance gains. To demonstrate their individual and combined impact, we conducted a series of benchmarks using vLLM, highlighting the performance improvements enabled by each optimization and the additional benefits of combining them.
The following plots (Figure 3, Figure 4, and Figure 5) demonstrate performance benchmarking, comparing the performance gains achieved through swapping MTP speculative decoding with our trained DSpark speculator as well as swapping the BF16 checkpoint for our FP8 quantized checkpoint. All benchmarks were performed on four H100s across workloads with prompts from various datasets. Each chart shows per-user streaming speed (interactivity) on the x-axis versus total output throughput (tok/s) on the y-axis, with each point representing a different concurrency level N. The HumanEval prompts come from OpenAI's HumanEval coding benchmark, while the math_reasoning and RAG prompts come from Spec-Bench, the ACL 2024 speculative decoding benchmark suite.
Each upgrade delivers meaningful gains on its own, and they compose cleanly. Swapping MTP for our trained DSpark speculator at BF16 primarily boosts single-user streaming speed, while swapping the BF16 checkpoint for FP8 primarily boosts aggregate throughput under load, where its reduced compute and memory footprint matters most. Combining them outperforms either intermediate configuration at every concurrency level on both workloads.
Nemotron 3.5 Super VL models are available for inference through Red Hat Inference Hub on Red Hat AI Factory with NVIDIA.
