On reasoning versus inference-time scaling

This is the third update on our journey to reproduce R1-like reasoning in small LLMs. Catch up on the previous installments here:

• Lessons on reproducing R1-like reasoning in small LLMs
• How particle filtering makes small LLMs think big
• Granite, LIMO, and small LLM reasoning

Let’s start with the results

Last week, we shared some exciting findings from our paper on a particle-filtering-based inference-time scaling method. We demonstrated that using our method, even a 7B model could match the performance of a significantly larger reasoning model, o1-preview, on the Math500 dataset. You can check out that LinkedIn post.

While all models showed strong performance on Math500 when scaled at inference time using particle filtering, none were able to reach o1-preview’s performance on the much more challenging AIME-2024 dataset.

This week, we pushed our method further by scaling up with a larger 32B Qwen-instruct model, paired with an even larger 72B Qwen PRM. The results? We’re thrilled to share that without any additional training—meaning absolutely no distillation from R1 or similar techniques—we outperformed o1-preview by one question!

What are “reasoning” models really doing?

If you’ve spent enough time analyzing R1’s output traces, as we have, you’ve probably noticed that its so-called “reasoning” often looks more like a search process. The model starts by generating an initial response, then critiques or evaluates its own draft, and if necessary, backtracks or restarts. This iterative search process continues until the model decides it has found the best answer to return.

Interestingly, if you apply particle filtering, you’ll see strikingly similar patterns, with some differences. Our method starts by generating multiple draft responses, then uses a PRM (preference ranking model) score to evaluate and critique each partial draft. The promising responses continue, while weaker ones are eliminated, repeating until the model arrives at a final answer.

This raises an intriguing question: Is reasoning just a form of learned search?

Training to reason: Amortizing inference-time scaling

Inference-time scaling can be computationally expensive, requiring both the transition model and the PRM—sometimes even multiple PRMs. A natural next question is:

Can we use reinforcement learning (RL) or other training methods to make the transition model itself act as a PRM?

In other words, can we train the model to not only generate responses but also judge and refine its own drafts? This could effectively amortize the cost of inference-time scaling by training the model to perform this reasoning process upfront. Moving forward, we’ll be focusing our efforts on testing this hypothesis and will continue sharing our findings. Stay tuned!