CHIAR-Former: 스펙트럼 엔트로피 기반 동적 라우팅 트랜스포머
Chiaroscuro Attention: Spending Compute in the Dark
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 3
한국어 요약
스펙트럼 엔트로피를 신호로 각 토큰을 DCT 혼합, RBF 커널 혼합, 자기주의 중 하나로 라우팅하는 CHIAR-Former이 제안됐다. WikiText-103에서 기존 자기주의 대비 45% 성능 향상을 달성하면서 주의 연산량을 62.5% 감소시켰으며, 대규모 자연어 텍스트에서 스펙트럼 라우팅의 효과를 입증했다.
원문 발췌 · English
Standard transformers apply self-attention uniformly at every layer and token, regardless of whether the input requires dynamic cross-token interaction. We propose CHIAR-Former (Chiaroscuro Attention), a 4-layer hybrid transformer that routes each token to one of three operators - DCT spectral mixing, RBF kernel mixing, or full self-attention - based on per-token spectral entropy, a theoretically justified complexity signal. Through systematic ablation on WikiText-103, we discover routing collapse: the router consistently rejects RBF in favour of DCT and attention, revealing that spectral mixing and dynamic attention are complementary and sufficient. A purpose-designed DCT+Attention-only variant achieves Val PPL 36.54 on WikiText-103 - a 45% improvement over a full-attention baseline (PPL 66.62) at 62.5% fewer attention FLOPs. We extend evaluation to WikiText-2, IMDB sentiment classification, and synthetic ListOps operations, establishing a clear operating regime: CHIAR-Former excels on large-scale naturalistic text where token diversity supports spectral specialisation, while full attention retains an edge on small datasets and synthetic pattern-matching tasks. These findings - both the wins and the losses - together define when and why spectral routing earns its keep.