LOWN, 비전 언어 확산 모델 추론 35% 수준까지 경량화
Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 17
한국어 요약
LOWN은 이미지 토큰 처리 여부를 매 스텝에서 동적으로 판단해 Diffusion 비전 언어 모델의 디코딩을 최적화한다. LLaDA-V 성능 99.6%를 유지하면서 평균 35.3% 수준의 FLOP으로 10개 벤치마크에서 검증했다.
같은 소식, 다른 매체 · 11곳
- Hy-Nomas: Towards Free-form Interactive World Generation
- A-RSI: Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement
- ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning
- CodeCap-Bench: Can Code Serve as a Good Video Caption?
- The Trinity:Capability-Driven Recursive Self-Improvement for Multimodal Agents
- WorkFactory: Turning Professional Intents into Executable Work for Scalable Agent Training
- What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents
- Can Test-Time Memory Evolution Outperform Test-Time Training for Scientific Discovery?
- Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
- A Broader Look into MLLM's Retrieval Heads
- OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning
원문 발췌 · English
Diffusion vision-language models (DVLMs) process their entire input at every denoising step. Since visual tokens make up 89-97% of this input on the benchmarks we study, most of the decoding computation is spent repeatedly processing the same image. Visual token pruning methods partially relieve this burden by removing part of the visual tokens, but they lose the removed information and still process the remaining image at every step. To measure how often the image is actually needed, we compare LLaDA-V's predictions with and without the image at every decoding step and find that removing the image changes a token being unmasked at only 22.8% of steps on average and at as few as 7.9%. We therefore propose LOWN, a decoding method that decides at every step whether to process the image and how many tokens to unmask. LOWN predicts the masked tokens without the image and processes the image only when this prediction disagrees with the last image-conditioned prediction. Agreement certifies the earlier prediction for reuse, and the number of certified tokens sets how many are unmasked in that step. We further observe that the need for the image differs across benchmarks, so LOWN's adaptive decision is needed rather than a fixed schedule. Without training or a tuned threshold, LOWN retains 99.6% of LLaDA-V's score on ten benchmarks at 35.3% of the decoding FLOPs on average. LOWN also generalizes across LLaDA-based DVLMs, composes orthogonally with visual token pruning, and runs under key-value caching for further efficiency gains. Code and project page are available at https://hyeon9698.github.io/LOWN/