SALA, 인간-로봇 행동 의미 정렬로 구체화 기초 모델 이중축 확장
SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models
- 매체
- Takara TLDR
- 게시
- 중요도
- 4/5
- 화제 지수
- —
한국어 요약
구체화 기초 모델의 핵심은 로봇 상호작용 데이터인데, 수집 비용이 많이 든다. 본 논문은 SALA를 제안해 인간과 로봇 영상의 언어 설명으로 의미 유사성을 추정하고, 인간 행동을 로봇 잠재 행동에 의미론적으로 정렬하는 방식으로 해결했다. 공유 잠재 행동 공간을 통해 구체화 기초 모델을 인간과 로봇 영상에서 모두 학습할 수 있게 하며, 인간 영상 사전학습과 작업별 영상 미세조정 두 축으로 확장 가능하게 했다.
원문 발췌 · English
Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.