Sparse autoencoders reduce Whisper hallucinations from 72.63% to 14.11%
Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
Hugging FaceTrending 3
Summary
The authors show that Whisper hallucinations can be detected and mitigated through hidden representations. Using sparse autoencoder (SAE) latent-space steering, they reduce hallucination rate from 72.63% to 14.11% for Whisper small and from 86.88% to 27.33% for Whisper large-v3, with minimal word error rate degradation on speech data.
#whisper#asr#hallucination#sparse autoencoder#representation steering