Skip to main content
Redhat Developers  Logo
  • Products

    Platforms

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat AI
      Red Hat AI
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • View All Red Hat Products

    Featured

    • Red Hat build of OpenJDK
    • Red Hat Developer Hub
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat OpenShift Local
    • Red Hat Developer Sandbox

      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Secure Development & Architectures

      • Security
      • Secure coding
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • Product Documentation
    • API Catalog
    • Legacy Documentation
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

How particle filtering makes small LLMs think big

Lessons on reproducing R1-like reasoning in small LLMs without using DeepSeek-R1-Zero (or its derivatives)

February 6, 2025
Akash Srivastava Isha Puri Kai Xu Shivchander Sudalairaj Mustafa Eyceoz Oleg Silkin Abhishek Bhandwaldar Aldo Pareja GX Xu
Related topics:
Artificial intelligence
Related products:
Red Hat AI

Share:

    This is the first update on our journey to reproduce R1-like reasoning in small large language models (LLMs). You can find the original article here: Lessons on reproducing R1-like reasoning in small LLMs

    Today was mostly about organizing results, evaluating new checkpoints, and making sense of all the numbers. We also kicked off a fresh experiment to test the impact of data quality on reasoning in small LLMs—but more on that later.

    Granite + particle filtering = big gains

    We already knew from our earlier experiments that particle filtering works well across multiple small models. But as we were compiling today’s results, we found something even more exciting: Granite models also benefit significantly from our method!

    Here’s how Granite 8B performed on the key benchmarks:

    • MATH-500: 0.78 (Granite 8B)
    • AIME 2024: 16.6 (Granite 8B)

    This is huge—our particle filtering method actually makes Granite better than GPT-4o on Math-500 and AIME 2024! 

    More cool results

    Across the board, introducing reasoning—using all the methods we talked about earlier—led to consistent performance gains on the Math-500 and AIME 2024 benchmarks. Here’s a giant results table summarizing where we stand.

    modeldatasetckpt(expected) aime@1aime@8note
    deepseek-ai/DeepSeek-R1-Distill-Llama-8B--33.7566.67baselines/aime/DeepSeek-R1-Distill-Llama-8B
    meta-llama/Llama-3.1-8B-Instruct--2.9210.00baselines/aime/Llama-3.1-8B-Instruct (This is without a specific prompt)
    Llama-3.1-8B-InstructBespoke-promptllama-r1-bmo-bespoke-system-numinamath/samples_8191505.8316.67 
    Llama-3.1-8B-InstructBespoke-prompt + grponew_grpo_llama_solo/ckpt-1057.0820.00 
    Llama-3.1-8B-InstructBespoke-prompt + grponew_grpo_llama_solo/ckpt-1207.5020.00Our best llama
    Llama-3.1-8B-InstructBespoke-prompt + grponew_grpo_llama_solo/ckpt-1355.8320.00 
    Llama-3.1-8B-InstructBespoke-prompt + grponew_grpo_llama_solo/ckpt-1506.2520.00 
          
    ibm-granite/granite-3.1-8b-instruct--1.253.33 (was 10.00) 
          
          
          
    microsoft/phi-4--19.1743.33baselines/aime/phi-4
    Phi-4Bespoke-prompt(add sft here)   
    Phi-4Bespoke-prompt + grpophi-r1-test-new-checkpoint-7518.3336.67 
    Phi-4Bespoke-prompt + grponew_grpo_phi/ckpt-9015.4233.33 
    Phi-4Bespoke-prompt + grponew_grpo_phi/ckpt-10513.7530.00 
    Phi-4Backtracknuminamath-phi4-traj-8x1-0.8-backtracked_numinamath_phi_4/samples_20912816.6736.67 
    Phi-4Backtrack + grpo (beta 0.01)grpo_backtrack_phi_beta_01/ckpt-16512.0826.67 
    Phi-4Backtrack + grpo (beta 0.01)grpo_backtrack_phi_beta_01/ckpt-25514.5830.00 
    Phi-4Backtrack + grpo (beta 0.01)grpo_backtrack_phi_beta_01/ckpt-28513.3326.67 
    Phi-4Backtrack + grpo (beta 0.01)grpo_backtrack_phi_beta_01/ckpt-31516.2543.33 
    Phi-4Backtrack + grpo (beta 0.01)grpo_backtrack_phi_beta_01/ckpt-40515.0026.67 
    Phi-4Backtrack + grpo (beta 0.01)grpo_backtrack_phi_beta_01/ckpt-45015.8323.33 
    Phi-4Backtrack + grpo (beta 0.01)grpo_backtrack_phi_beta_01/ckpt-46516.2533.33 
    Phi-4But-Waitbut_wait_numinamath_phi_4/samples_12151917.0836.67 
    Phi-4But-Wait + grpo (beta 0.01)grpo_but_wait_phi_beta_01/ckpt-15018.7538.46 
    Phi-4But-Wait + grpo (beta 0.01)grpo_but_wait_phi_beta_01/ckpt-21017.0836.67 
    Phi-4But-Wait + grpo (beta 0.01)grpo_but_wait_phi_beta_01/ckpt-27020.0046.67Our best phi
    Phi-4But-Wait + grpo (beta 0.01)grpo_but_wait_phi_beta_01/ckpt-36016.6740.00 
    Phi-4Direct evolution GRPOgrpo_evolution_phi_beta_01/ckpt-1513.7536.67 
    Phi-4Direct evolution GRPOgrpo_evolution_phi_beta_01/ckpt-3015.8326.67 
    Phi-4Direct evolution GRPO MINIgrpo_evolution_mini_phi_beta_01/ckpt-515.4230.00 
    Phi-4Direct evolution GRPO MINIgrpo_evolution_mini_phi_beta_01/ckpt-1017.0826.67 
    Phi-4Direct evolution GRPO MINIgrpo_evolution_mini_phi_beta_01/ckpt-1516.2533.33 
    Phi-4LIMO- no system prompt. Used <thinking><thinking> and <answer><answer> on the training.limo_phi_4_lr_6e-6/samples_1`115932.0048.00LIMO has 817 samples. LIMO has AIME-ish and MATH-ish data.

    Data quality: A game-changer?

    We came across this fascinating paper, which dives deep into the importance of data quality in reasoning. The results are wild—they trained a Qwen-32B model on just ~800 high-quality reasoning samples and got O1/R1-level performance on MATH-500 and AIME24.

    Naturally, we had to try it out ourselves. And guess what? It worked! Applying the same strategy to Phi-4 gave us Phi-LIMO, which is the best performing model so far (investigating the evaluation script as the numbers on the second run turned out to be lower).

    Most interesting takeaway of the day

    Our synthetic data-based reasoning methods actually resulted in a Phi-4 model that reasons better than vanilla Phi-4—and it shows its reasoning in the process. That’s a big win for using synthetic data to enhance reasoning capabilities.

    What’s still running?

    Most of our compute is tied up with existing runs, so today we’re launching just two more experiments:

    • Testing the LIMO dataset on Granite: Can a really small model develop reasoning with just ~800 high-quality examples? We’ll let you know.
    • Generating synthetic data using particle filtering on LIMO dataset questions—will this further enhance reasoning abilities?

    This is funny, so I have to mention it—GPT4-o just told me the following:

    Did you know? The human brain makes approx. 35,000 decisions per day, many of them involving subconscious “particle filtering” to evaluate possible outcomes. Teaching LLMs to backtrack and refine their reasoning is, in a way, mimicking our own decision-making process.

    What? 🤯

    Read the next update here: Granite, LIMO, and small LLM reasoning

    Last updated: May 15, 2025

    Related Posts

    • Lessons on reproducing R1-like reasoning in small LLMs

    • Open source AI coding assistance with the Granite models

    • Llama 4 herd is here with Day 0 inference support in vLLM

    • Granite, LIMO, and small LLM reasoning

    • Deploy Llama 3 8B with vLLM

    • Deployment-ready reasoning with quantized DeepSeek-R1 models

    Recent Posts

    • Cloud bursting with confidential containers on OpenShift

    • Reach native speed with MacOS llama.cpp container inference

    • A deep dive into Apache Kafka's KRaft protocol

    • Staying ahead of artificial intelligence threats

    • Strengthen privacy and security with encrypted DNS in RHEL

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue