코사인 유사도는 오도한다: 보조 손실이 VLM의 잠재 변수가 아닌 모델을 재구성
Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 3
한국어 요약
시각-언어 모델의 잠재 시각 추론(LVR)에서 코사인 유사도 기반 정렬이 정확도와 음의 상관관계(r=-0.94)를 보인다는 연구 결과가 발표됐다. PRISM 진단 도구를 통해 감독된 잠재 토큰이 실제로는 우회되며, 보조 목적함수가 잠재 변수가 아닌 공유 파라미터를 통해 언어 모델을 재구성함을 밝혔다.
원문 발췌 · English
Latent visual reasoning (LVR) inserts supervised latent tokens between perception and answer generation in vision-language models (VLMs). The field uses alignment between these latents and their visual targets, i.e., cosine similarity or mean squared error (MSE), as both the training loss and the quality metric, assuming that better alignment yields a better answer. We test this with a designed matrix of five LVR variants and find the assumption inverted: cosine alignment is negatively correlated with accuracy across all five (r=-0.94). To explain this, we introduce PRISM, a pair of inference-time diagnostics: a linear probe that asks where the answer is decodable, and a corruption test that asks whether the latent is load-bearing. The supervised latents are largely bypassed. Corrupting them shifts accuracy by at most four points. The answer is decodable downstream of the latent but not at it, and the size of this decodability gap predicts how much each variant relies on its latent under perturbation. Consistent with an Information Bottleneck reading of the loss, the auxiliary objective reshapes the language model via shared parameters rather than via the latent variable it nominally optimizes.