Terminal-Bench 2.0 감사, 벤치마크 결함 30% 발견·연속 검증 워크플로우 제안
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
- 매체
- Takara TLDR
- 게시
- 중요도
- 2/5
- 화제 지수
- —
한국어 요약
코딩 에이전트 평가 벤치마크의 '유지보수 부채' 문제를 지적하는 연구로, 모델 개선에 따라 기존 검증자가 거부하는 유효한 해결책이 나타나거나 인프라 변경으로 평가 가정이 무효화될 수 있음을 보였다. Terminal-Bench 2.0의 약 30%에서 결함을 발견했고, 수정된 Terminal-Bench 2.1을 공개했으며, 수리 후 정확도는 10포인트 이상 향상되고 리더보드 순위는 최대 3계급 변동했다. 자원 할당(시간 제한 등)도 평가 결과에 크게 영향을 미쳐 측정 사양의 일부로 검증되어야 함을 보였다.
원문 발췌 · English
Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers reject, while changes to evaluation infrastructure can invalidate assumptions embedded in tasks and tests. We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure. We introduce a continuous-validation workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. We apply this workflow to Terminal-Bench 2.0, a benchmark that underwent extensive human and LM-assisted review and red-teaming before release. Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1. These repairs change evaluation results: accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958). We evaluate how resource allocation shapes benchmark conclusions. We show that fixed wall-clock can materially change agent performance: For example, extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp. Benchmark maintenance is therefore not only about repairing task artifacts: resource budgets are part of the measurement specification and must be validated alongside the tasks themselves. More broadly, benchmark maintenance debt is not just accumulated implementation bugs: our results show that as models improve, both evaluation assumptions and resource settings may need to be revalidated.