다중 교사 온-정책 증류에서 그래디언트가 학습에 미치는 영향 분석
From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
- 매체
- Hugging Face
- 게시
- 중요도
- 2/5
- 화제 지수
- 9
한국어 요약
연구팀이 다중 교사 온-정책 증류(MOPD)에서 교사 신호가 파라미터 변화에 미치는 영향을 분석했다. 손실 평균화, Adam 옵티마이저, BF16 반올림 등이 그래디언트와 학습 곡선에 미치는 영향을 실증적으로 규명했으며, 평균화 방식에 따라 수학 정확도가 2.1~2.6점 차이를 보였다.
원문 발췌 · English
Multi-teacher on-policy distillation (MOPD) aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. We study Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, optimizer updates, and task learning curves, with additional SmolLM3-3B diagnostics. We find that several factors influence teacher signals. First, loss averaging implicitly weights responses: token averaging favors longer responses, and equalizing domain contributions retains this weighting within domains. Second, Adam's first moment reduces differences in parameter updates: the cosine similarity is 0.83 between teachers and 0.96 between averaging rules, despite differences in raw gradients. Third, BF16 rounding hides small changes: about 97\% of FP32 master weights differ from initialization, but only 7--11\% of BF16 weights do. Finally, the top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, but the effect on task performance depends on averaging: mathematics accuracy is 2.6 points higher than with sampled-token policy-gradient (PG) under response averaging and 2.1 points lower under global token averaging.