Reframe, 에이전트 시각 추론을 위한 관찰 정렬 증류 프레임워크
Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning
- 매체
- Hugging Face
- 게시
- 중요도
- 2/5
- 화제 지수
- 18
한국어 요약
시각 에이전트가 도구 사용 시 관찰을 간과하거나 근거 미흡 행동으로 제한받는 문제를 다룬 사후훈련 프레임워크 Reframe을 제안했다. Ground-Drop-GRPO로 증거 인식 보상과 시각 입력 확률적 제거로 훈련하고, Observation-Gated-MOPD로 롤아웃 중 부실 도구 행동을 표준화된 것으로 교체 후 복구된 관찰에서 증류를 재개한다. 14개 이미지·영상 벤치마크에서 Mix-RL과 기본 MOPD를 모두 능가했다.
원문 발췌 · English
Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.