시퀀스 모델의 학습 전략: 특화 vs 인컨텍스트 학습의 경쟁
In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
- 매체
- Takara TLDR
- 게시
- 중요도
- 2/5
- 화제 지수
- —
한국어 요약
시퀀스 모델은 훈련 데이터에서 본 구조를 외삽하거나 프롬프트로 새 작업에 적응하는 두 가지 학습 전략으로 일반화한다. 이론 분석 결과 사전학습 데이터의 질과 양에 따라 인컨텍스트 예측은 일찍 나타나고 특화 학습은 데이터량이 모델 크기를 초과할 때 나타난다.
원문 발췌 · English
Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary. However, there is no learning theory for this competition, hence no fundamental understanding of \textit{why} one solution might be favoured over another. Here we provide such a theory in a solvable topic model---documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data. The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training. Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process. Trained transformers reproduce both scales and the separating crossover in remarkable agreement with the theory, with no fitted parameters.