Large language models generate text autoregressively: one token at a time. Every generated token requires a full forward pass—and that serial, per-token cost is the fundamental bottleneck in LLM serving, especially for large models. As models get larger and deployments scale, the gap between what hardware can theoretically do and what autoregressive generation actually delivers becomes increasingly expensive to close.
Speculative decoding is one of the most effective techniques available today for closing that gap—delivering two to three times the throughput gains and dramatically lower latency across a wide range of production workloads, with no changes to the model and no loss in output quality. The key idea is to use a small, fast “speculator” model to propose several tokens at once, then let the large “verifier” model validate them all in a single parallel pass. Accepted tokens are almost free.
However, the speedup provided by speculative decoding depends entirely on one thing: how well the speculator model is aligned with the verifier model. A well aligned speculator yields more accepted tokens per step, whereas a poorly aligned speculator wastes compute on rejected proposals.
This brings us to a critical question: How do you train a good speculator model as fast and efficiently as possible?
One of the biggest bottlenecks in the training process is obtaining high-quality data that aligns closely with the distribution generated by the verifier model. The standard approach is to run the verifier model on a set of prompts and collect its responses, but for massive verifier models this process is slow and expensive. We wanted to see whether smarter choices about training data and training duration could reduce that cost without sacrificing speculator quality. This post covers our findings as we tackle two main questions:
- Can you avoid expensive response generation by training on outputs from distinct LLMs?
- How many training epochs does a speculator actually need?
We also look at how well these findings generalize across reasoning and instruct variants, and share real production inference numbers. We conducted all training using the Speculators library, an open source framework built by the Red Hat team for training speculator models. This framework supports multiple algorithms, including EAGLE3, DFlash, and P-EAGLE. We ran inference benchmarks using GuideLLM, and speculators deploy directly into vLLM.
Experimental setup and key metrics
Before diving into the data, it's worth reinforcing a vital property of speculative decoding: it is completely lossless. Because the verifier model validates the output, the output distribution matches what the model would have produced on its own. You gain massive speedups while giving up nothing in output quality.
And because the gains come entirely from how many draft tokens the verifier model accepts, the key metric throughout this post is mean acceptance length—the average number of draft tokens accepted per step. For example, an acceptance length of 2.5 at k=5 means the verifier model accepts 2.5 out of every five proposed tokens on average, effectively more than doubling useful output per step.
Most of our benchmarks use EAGLE3, a speculative decoding algorithm that generates draft tokens recursively. We also display an experiment with DFlash, a newer parallel drafting algorithm designed for higher compute utilization. Both of these state-of-the-art algorithms achieve high prediction accuracy by extracting and fusing a snapshot of the verifier model’s intermediate hidden states.
Methodology overview
To test how these data strategies perform in practice, we set up a controlled environment across several verifier models, hardware configurations, and real-world workloads.
- Verifier models evaluated: gpt-oss-20b, gpt-oss-120b, Qwen3-30B-A3B, Qwen3-32B, and Laguna-XS.2.
- Training dataset: Training data combined 500K prompt-response pairs: 300K from Magpie and 200K from UltraChat.
- Hardware setup: We conducted training on either 8xA100 or 8xH100 GPUs. Initial runs went for six epochs, though later optimized runs were shortened to two to four epochs based on our convergence findings.
- Evaluation workloads: We evaluated the models across seven datasets representing real-world production use cases: HumanEval (coding), math reasoning, QA (multi-sentence factual), question answering (concise/direct), RAG, summarization, and translation.
Self-distillation vs. cross-distillation: Can we skip expensive data generation?
The standard way to train a speculator is self-distillation: run the verifier model on a set of prompts, collect its responses, and train the speculator to predict them. This ensures your training data perfectly mirrors the target model's distribution.
The problem is cost. In practice, we typically train with roughly 500K examples; generating that many responses from a massive verifier model is a major compute expense. Depending on hardware, prompt/output lengths, and serving configuration, this step can take multiple days, and it must be repeated for every new model you want to accelerate. We wanted to know whether that expensive step could be reduced or skipped entirely by cross-distillation: using (pre-existing) responses generated by another model.
We tested two cross-distillation scenarios:
- Same-family: Distilling from a larger model within the same architecture family (for example, using Qwen3-235B-A22B responses to train a Qwen3-30B-A3B speculator)
- Cross-family: Distilling from a larger model from a different architecture family (for example, Qwen3-235B-A22B responses to train a gpt-oss-120b speculator)
Same-family distillation (large model → small model)
Let’s first take a look at the same-family architecture scenario. We trained EAGLE3 speculators for the reasoning and instruct variants of Qwen3-30B-A3B and Qwen3-32B, comparing self-distillation and cross-distillation with responses from Qwen3-235B-A22B.
Result: As displayed in Figures 1 through 4, speculators trained with data from the larger model consistently outperform self-distillation, sometimes by wide margins.
We observed the same pattern on a completely different model family, training a gpt-oss-20b speculator using responses from gpt-oss-120b (see figure 5).
It is not surprising that responses produced from a large model can be used to train effective speculators for smaller models within the same family, as smaller family variants are often distilled from larger models during initial training. However, the fact that cross-distillation can lead to higher acceptance rates is a major win.
The takeaway: For any given model family, you only need to perform expensive response generation once on the largest model. Those same responses can then be reused to train highly effective speculators across the entire family variation spectrum, generating significant compute savings.
Despite the savings, extracting responses from the largest (and most expensive) model within a family can still be expensive. What if we try to reuse data across different model families? Let’s look at that next.
Cross-family distillation
We tested exactly this scenario by training an EAGLE3 speculator for gpt-oss-120b and a DFlash speculator for Laguna-XS.2 using both self-distillation and data from Qwen3-235B-A22B. In this setup, the teacher is larger than the verifier models but has distinct architecture and training ancestry. The resulting acceptance lengths are displayed in Figures 6 and 7.
These results show that cross-distillation across model families is a bit more nuanced. While it can introduce a persistent performance gap in some targets, it can also produce highly competitive or better speculators in others. For gpt-oss-120b, cross-distillation resulted in significantly lower acceptance rates than self-distillation. For Laguna-XS.2, the cross-distilled DFlash model is competitive overall and ahead on many workloads. Since these two examples use different speculator algorithms (EAGLE3 for gpt-oss-120b and DFlash for Laguna-XS.2), we should interpret them as practical outcomes on two targets, not as a direct algorithm-to-algorithm comparison.
So if cross-family distillation can leave a gap and self-distillation is expensive, the practical question becomes: how much of that gap can a mixed strategy recover? Let’s see if a hybrid approach will work.
The hybrid approach
A natural middle ground between pure cross-family distillation and pure self-distillation is mixing data from both approaches. We tested this scenario by starting from the cross-distillation speculator for gpt-oss-120b from above and further training it on self-distillation data. Figure 8 shows how the mean acceptance length (averaged across all the use cases) shifts as a function of the number of self-distillation samples.
Adding a modest amount of self-distillation data (up to 50K samples) delivers a stark positive jump in acceptance rates. However, the cross-distillation data has a lasting impact on the speculator behavior. Even when the speculator was eventually trained on a full 500K self-distillation samples, it never quite matched the peak acceptance length achieved by training purely on self-distillation data from the start.
The data strategy verdict
Taken together across all three results:
- Generating responses from the largest model within the verifier family results in the highest acceptance rates. This is expensive, but the cost is amortized if the data is used to train speculators for multiple models within the same family.
- Adding a limited number of self-distillation samples (around 50K) to pre-existing data generated by other models is a relatively cheap and effective alternative, although it may not result in the same quality.
How many training epochs are needed?
With the data strategy settled, the next practical question is how long to train. For our particular data mix (300K samples from Magpie and 200K samples from UltraChat) all models show the same convergence pattern: the bulk of improvement arrives in the first two to three epochs, and the value of additional training drops sharply after that.
As shown in Figure 9, acceptance gains for the gpt-oss-120b EAGLE3 speculator flatten out rapidly regardless of data strategy. For self-distillation, the mean acceptance length (out of five draft tokens) only crawls from 2.49 at epoch 1 to 2.65 at epoch 6—a modest 6.4% gain—with two-thirds of that gain arriving in the first two epochs. Cross-distillation follows the same dynamics: rapid gains early, flattening sharply past epoch 3. Both strategies converge by epoch 4.
Figure 10 shows the acceptance rate per dataset for Qwen3-30B-A3B trained with cross-distillation data. Acceptance rates rise steadily and flatten together, with the mean settled around epoch 4.
Practical recommendation: When using our recommended data mix, three to four epochs is the optimal sweet spot for stopping speculator training runs.
What does this look like in production?
Training better speculator models pays off directly at serving time. The checkpoints produced by our training runs slot directly into vLLM's speculative decoding pipeline and substantially reduce inter-token latency with no changes to the serving stack.
Figure 11 shows Inter-Token Latency (ITL) measured with a GuideLLM sweep across a range of request rates. Two things stand out.
First, speculative decoding lowers the ITL floor dramatically:
- Approximately three times for gpt-oss-120b on math reasoning (using EAGLE3)
- Approximately 3.2 times for Laguna-XS.2 on coding (using DFlash).
Second, the reduction in total request time through speculative decoding allows the system to delay saturation. This shift enables the infrastructure to handle a higher volume of requests per second before performance begins to decline. Ultimately, this leads to accelerated token generation across all levels of demand and increases the operational ceiling before the system reaches its limit.
Conclusion
Training data strategy is the biggest lever for speculator quality: same-family cross-distillation wins where a larger sibling model's responses are available, a hybrid cross-family distillation with limited self-distillation is a viable alternative, and three to four epochs is enough for either. The result is two to three times throughput and dramatically lower latency with no changes to the verifier model.
The training framework used for all experiments in this post is the open source library Speculators. You can find the best-performing speculators for Llama, Qwen, gpt-oss, and Gemma (including EAGLE3, DFlash and P-EAGLE) at our Red Hat AI Speculator Models collection. More experiments are underway, including exploring broader training datasets and the effect of mixing responses from different models.
Questions and contributions are welcome in the vLLM Community Slack at #speculators and #feat-spec-decode.
