R^2Orch, 자기 개선 루프로 에이전트 오케스트레이션 자동화
R^2Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 16
한국어 요약
에이전트 오케스트레이션 학습의 과제인 수작업 커리큘럼, 이점 붕괴, 신뢰도 낮은 평가를 해결하기 위해 자기 플레이 내 루프와 개선 방식 개선 외 루프로 구성된 R^2Orch 패러다임을 제안했다. 12개 데이터세트에서 질문 응답, 수학, 코드, 상호작용 태스크 전반에 걸쳐 기존 모델을 일관되게 상회했다.
원문 발췌 · English
In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R^2Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R^2Orch consistently outperforms baselines across question answering, math, code, and interactive tasks. Our code is available at https://github.com/beita6969/R2Orch.