AI Daybook
AI Daybook
오늘의 지면
논문연구·논문

V-JEPA-LM, 비디오의 역학 정보를 읽는 비디오-언어 모델

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

매체
Hugging Face
게시
중요도
3/5
화제 지수
11

한국어 요약

V-JEPA 같은 월드 모델 인코더로부터 역학 정보(방향, 속도, 순서, 상태)를 활용하도록 언어 모델을 훈련하는 방법을 제시했다. 픽셀 변환으로 만든 반사실적 감독 신호를 사용해 1B~8B 파라미터 모델(Qwen3, Llama 기반)을 단일 단계로 학습했을 때, 기존 오픈 소스 비디오 LLM보다 11개 벤치마크에서 앞섰다. 4B 모델이 72.1점으로 기존 최고 4B 모델의 61.1점, 최고 8B 모델의 64.8점을 초과했다.

원문 발췌 · English

World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: https://github.com/kkt94/Llama-V-JEPA-LM.

#world model#vision-language model#video understanding
원문 기사 읽기
전체 보기 →