DexForesight, 손가락 조작 로봇에 공간 예측 능력 주입하는 프레임워크
DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
- 매체
- Hugging Face
- 게시
- 중요도
- 4/5
- 화제 지수
- 16
한국어 요약
DexForesight는 역학 인식 세계 표현을 정교한 정책의 잠재 공간에 주입해 공간 예측을 암시적으로 학습하는 프레임워크다. 오프라인 장면 깊이, 손 기하학, 시연 궤적에서 동작 조건부 세계 모델을 학습해 손가락 동작과 장면 변화를 예측한다. 11개 DexJoCo 시뮬레이션 과제에서 기본 정책 성공률을 49.4%에서 60.5%로 개선했으며, 3가지 실제 로봇 손가락 조작 과제에서 58.3% 성공률을 달성했다.
원문 발췌 · English
Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/