OmniOpt benchmarks over 100 optimizer methods across model training regimes

Summary

OmniOpt is a unified survey and benchmark of optimizers for large-scale model training. It treats every optimizer update as a five-stage transformation, uses linear minimization oracles to unify methods, and provides a dual-dimension taxonomy across language model pretraining to image classification to help researchers select optimizers under explicit assumptions.