MERL, 진화하는 상상력으로 VLA 정책 학습 개선
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
- 매체
- arXiv
- 게시
- 중요도
- 2/5
- 화제 지수
- 2
한국어 요약
시뮬레이터 신뢰도 기반 정책 최적화 프레임워크 MERL을 제시했다. 월드 모델과 보상 프록시를 포함한 온라인 업데이트 시뮬레이터를 결합하며, 단계별로 시뮬레이터 신뢰도가 상상된 데이터 비율과 롤아웃 수평을 제어한다. 변형된 LIBERO-PRO 작업에서 79.7% 평균 성공률을 달성했고, 같은 기준 모델 없는 제어보다 고임계값 학습 속도가 10.9% 향상했다.
원문 발췌 · English
Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability--efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.