Direct Reasoning Optimization for LLMs

This document introduces Direct Reasoning Optimization (DRO), a novel reinforcement learning framework designed to enhance the reasoning abilities of Large Language Models (LLMs) in open-ended, long-form tasks. The core innovation is the Reasoning Reflection Reward (R3), a self-contained reward signal that allows LLMs to internally assess and refine their reasoning processes without requiring external human feedback or reward models. DRO also incorporates a dynamic data filtering strategy based on R3 to improve training efficiency and performance. The authors demonstrate DRO's effectiveness across diverse tasks, including paragraph revision and financial question answering, showcasing its versatility and cost-reduction benefits compared to existing methods.