FOAM, LLM 사전학습에서 SOAP과 같은 성능을 Adam 수준 속도로 달성
Factored One-sided Adam-Moment for Practical and Scalable SOAP
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 13
한국어 요약
SOAP의 이점을 Adam/AdamW에 가까운 효율로 얻기 위한 경량 최적화 기법 FOAM을 제시했다. 입력 전용 사전처리로 메모리를 절감하고 출력 측 Kronecker 인수를 제거하며, SOAP의 조밀한 회전 2차 모멘트를 행렬 기반 Adam-모멘트 추정기로 대체한다. 124M~760M 파라미터 LLM 사전학습에서 SOAP과 Muon 기법 수준의 검증 손실을 유지하면서 SOAP 메모리의 절반을 사용하고 Muon 수준의 벽시계 시간에 작동한다.
같은 소식, 다른 매체 · 1곳
원문 발췌 · English
Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam