PLatour, 대형·소형 언어모델 협력 프레임워크 공개
On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer
- 매체
- Hugging Face
- 게시
- 중요도
- 2/5
- 화제 지수
- 10
한국어 요약
대형 모델의 프롬프트 표현과 소형 모델의 효율적인 생성을 결합해 성능과 추론 비용을 균형 맞추는 Prefill-to-Autoregressive Cross-model Transfer(PACT) 기법을 평가하는 오픈소스 프레임워크 PLatour를 제안했다. PLatour는 경량 선형 다리로 동결 모델들을 연결하고 작업 보상으로 직접 학습하며 하이브리드 어텐션 아키텍처를 지원한다. 수학과 코딩 벤치마크에서 기존 오픈소스 기준을 크게 능가했으나, 통제 실험 결과 일부 작업의 성능 향상은 무정보 신호나 자기통신으로도 재현 가능했다.
원문 발췌 · English
Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.