NEO-VLA, 인코더 없는 비전-언어-액션 모델 제안
From Pixels to Actions - Towards Native Vision-Language-Action Policies
- 매체
- Hugging Face
- 게시
- 중요도
- 4/5
- 화제 지수
- 27
한국어 요약
시각 신호 처리 시 인코더로 압축하는 전통적 구조의 한계를 지적하고, 이미지를 경량 패치 컨볼루션으로 토큰화한 후 백본 전체에서 언어와 함께 진화하는 NEO-VLA를 제시했다. 조작 태스크 전반과 분포 이동 상황에서 인코더 기반 모델보다 더 강한 제어 성능을 보였다.
원문 발췌 · English
Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.