WaveGen, 외부 모듈 없이 텍스트에서 파형까지 직접 생성하는 TTS
WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 11
한국어 요약
신경 오디오 코덱·멜 스펙트로그램 보코더·외부 정렬 모듈 같은 중간 표현에 의존하지 않고, 단일 모델 내에서 스펙트럼 강제를 통해 스펙트럼 궤적과 파형 궤적을 동시에 학습하는 완전 엔드투엔드 확산 기반 텍스트-파형 생성 프레임워크 WaveGen을 제안했다. 0.1B~0.6B 파라미터 범위에서 Emilia 데이터세트에서 최첨단 성능을 달성했으며(Seed-en 벤치마크에서 WER 1.47, SIM 0.75) 소스코드와 체크포인트를 공개했다.
원문 발췌 · English
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral v-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen