Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning
News Release Summary
Retrieval-augmented language models usually fetch examples that look similar to the current prompt, but the most helpful example for a hard reasoning problem may instead share its solution pattern. RA-RFT trains a retriever to select demonstrations for their expected reasoning benefit and then uses those analogies during reinforcement fine-tuning. The authors find that this reasoning-aware retrieval surfaces complementary solution strategies and improves mathematical reasoning over standard reinforcement fine-tuning. On AIME 2025, the reported average@32 gain over GRPO is 7.1 points for Qwen3-1.7B and 2.8 points for Qwen3-4B, suggesting that better example selection can complement advances in rewards and training curricula.
abstract
Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern. We propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap, and then fine-tunes the policy model via reinforcement fine-tuning methods with retrieved analogous demonstrations, so the model learns to leverage reasoning traces under verifiable outcome rewards. We further analyze the diversity of retrieved contexts and find that reasoning-aware retrieval surfaces complementary solution strategies that provide distinct reasoning scaffolds for individual problems. Across challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning methods. For example, it improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively -- suggesting that reasoning-aware retrieval is a complementary axis of improvement and orthogonal to advances in reward design or training curricula.
citation
@inproceedings{xiao2026learning,
title = {Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning},
author = {Xiao, Zilin and Ma, Qi and Chen, Chun-cheng Jason and Chen, Xintao and Atreya, Avinash and Chen, Hanjie and Ordonez, Vicente},
year = {2026},
booktitle = {Findings of the Conference on Empirical Methods in Natural Language Processing. EMNLP 2026},
url = {https://arxiv.org/abs/2606.13680},
}
automatically generated questions, main contributions and limitations of this paper
Questions this paper helps answer
- What is RA-RFT? RA-RFT is a post-training framework that combines reasoning-aware retrieval with reinforcement fine-tuning so language models can learn from analogous solution demonstrations.
- How does its retrieval differ from conventional RAG? Instead of ranking examples mainly by lexical or semantic similarity, it trains a retriever to rank contexts by their expected benefit to the target reasoning process.
- How is the retriever trained? The paper uses gold-relevance distillation to teach the retriever which demonstrations are useful for solving a reasoning problem.
- What results are reported? Across mathematical reasoning benchmarks, RA-RFT outperforms standard reinforcement fine-tuning; on AIME 2025 average@32, it exceeds GRPO by 7.1 points for Qwen3-1.7B and 2.8 points for Qwen3-4B.
Main contributions
- RA-RFT frames retrieval for reasoning as finding analogous solution strategies rather than merely semantically similar text.
- It combines a gold-relevance-distilled retriever with reinforcement fine-tuning on retrieved demonstrations and verifiable outcome rewards.
- The paper analyzes retrieved-context diversity and reports that reasoning-aware retrieval provides complementary reasoning scaffolds.
- Reported mathematical-reasoning results show improvements over standard reinforcement fine-tuning for Qwen3-1.7B and Qwen3-4B.
Limitations and cautions
- The reported evaluation focuses on mathematical reasoning benchmarks, so testing analogous retrieval on coding, science, and open-ended tasks is an important next step.
- The method relies on gold-relevance distillation, and future work could study how the quality and availability of that supervision affect transfer to new domains.
- The study emphasizes outcome accuracy; deployment-oriented work could also quantify retrieval cost, corpus sensitivity, and the effect of noisy demonstrations.
- The benchmark gains do not establish that every reasoning task benefits equally from analogy, but they provide strong evidence that retrieval choice is a useful complementary training axis.
How to read this result
RA-RFT is best read as a strong demonstration that retrieval can improve reasoning not just by adding information but by supplying the right analogies, with clear empirical gains on challenging math benchmarks.