AI Daybook
AI Daybook
오늘의 지면
논문연구·논문

SPLIT, 검증기 불완전성 하에서 추론 시간 계산 최적 배분

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

매체
arXiv
게시
중요도
3/5
화제 지수
3

한국어 요약

추론 시간에 여러 후보를 생성해 검증기가 선택하는 방식 하에서, 비완전 검증기의 실수와 상관 오류를 고려하여 샘플링·모델·검증 예산을 최적 배분하는 방법 SPLIT을 제시했다. 수학·과학·코드·소프트웨어 에이전트 등 7개 벤치마크 26개 모델 175개 에이전트 시스템에서 사후 최적 대비 0.02~0.8 퍼센트포인트 범위 내 성능을 달성했다.

원문 발췌 · English

A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.

#llm#agent#verification
원문 기사 읽기
전체 보기 →