Reasoning Arena: 검증 불가능한 보상 극복하는 추론 토너먼트 프레임워크
Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 12
한국어 요약
검증 가능한 보상 기반 강화학습에서 동일 보상을 받는 샘플들의 학습 신호 부족 문제를 해결하기 위해 Reasoning Arena을 제안했다. 추론 과정을 토너먼트 방식으로 비교하여 세밀한 상대적 선호도를 추출하고, Bradley-Terry 모델로 효율적으로 순위를 매겨 수학·코딩 벤치마크에서 7.6% 성능 향상과 27~41% 학습 가속을 달성했다.
원문 발췌 · English
Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models through outcome-based supervision. However, verifiable rewards frequently become uninformative at the group level: when all sampled traces of a given prompt receive identical rewards, group-relative advantage estimation provides no gradient signal, even though the traces may differ substantially in reasoning quality. We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them. Beyond examining the final answer, Reasoning Arena constructs trace tournaments, where reasoning traces are compared head-to-head to expose finer-grained preferences within the group, converting reasoning quality into rich relative reward signals. To make reward estimation efficient, rather than exhaustively comparing every pair, each new trace is evaluated against a small, dynamically updated pool of previously generated traces as anchors to efficiently establish a relative ranking. We then fit a Bradley-Terry model on the incomplete comparison graph, enabling scalable RL integration without quadratic pairwise comparisons. Empirical results demonstrate that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks. By converting otherwise wasted zero-advantage samples into useful gradient updates, our method accelerates training by 27% to 41%, saving nearly 50% of generation compute, and substantially improves overall reasoning performance.