Cosine alignment is negatively correlated with latent visual reasoning accuracy
Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
Hugging FaceTrending 3
Summary
The authors find that cosine similarity between latent visual tokens and visual targets is negatively correlated with accuracy (r=-0.94) across five latent visual reasoning variants. Using linear probes and corruption tests, they show that supervised latents are largely bypassed, and the auxiliary loss reshapes the language model via shared parameters rather than optimizing the latent variable itself.