From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

This research introduces a hierarchical latent selection model to explain how large language models develop robust reasoning through post-training. The authors argue that supervised fine-tuning (SFT) provides the essential building blocks, while reinforcement learning (RL) decomposes these traces into reusable atomic skills and routing mechanisms. By isolating these components, RL enables models to solve out-of-distribution problems through novel combinations of learned modules. Controlled experiments on synthetic tasks prove that training on compositional traces is superior to learning isolated skills. The study concludes that an ideal training protocol uses SFT to ensure broad module coverage and RL to explore unseen...