CHIAR-Former routes tokens via spectral entropy to reduce attention compute by 62.5%

Summary

CHIAR-Former is a hybrid transformer that routes each token to DCT spectral mixing, RBF kernel mixing, or self-attention based on per-token spectral entropy. On WikiText-103, it achieves 45% improvement in perplexity (36.54 vs 66.62) while reducing attention FLOPs by 62.5% compared to full-attention baselines, with strong performance on large-scale naturalistic text.