DreamX, 동영상 확산 모델로 범용 잠재 액션 학습
DreamX: Video Diffusion Model is a Universal Latent Action Learner
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 19
한국어 요약
연구팀이 사전학습된 동영상 확산 모델을 범용 잠재 액션 러너로 재용도하는 DreamX를 제안했다. 각 프레임을 300배 압축(1,200→4 토큰)한 콤팩트 잠재 액션을 추출하며, 시각적 외형에서 분리된 액션 동역학을 캡처한다. LARY-Bench에서 최고 성능을 기록했고 인간과 로봇 간, 환경 간, 액션 파라미터화 간 제로샷 액션 전이를 지원한다.
원문 발췌 · English
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300times (1{,200 rightarrow 4} tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.