에이전트 하네스 진화의 함정, 더 높은 점수가 더 나은 성능을 보장하지 못한다
Beneath the Surface: What We Overlook When We Talk About Agent Evolution
- 매체
- Hugging Face
- 게시
- 중요도
- 2/5
- 화제 지수
- 11
한국어 요약
에이전트 하네스 진화 기법이 모델 가중치 수정 없이 자가 개선을 가능하게 하지만, 높은 벤치마크 점수가 신뢰성·효율·전이 가능성 있는 개선을 보장하지 못함을 보였다. 네 가지 방법을 평가한 결과 진화한 16개 에이전트 모두가 새로운 과제는 얻으면서 기존 과제는 잃었으며, 신뢰성 향상도 배포 비용 증가를 초래했다.
원문 발췌 · English
Agent harness evolution enables recursive self-improvement (RSI) without updating model weights. We examine whether higher benchmark scores yield reliable, efficient, and transferable improvement. We evaluate four methods across harnesses, general and coding tasks. Repeated evaluations, controlled agent expansion, component interventions, and admission replays reveal recurring patterns. All sixteen evolved agents gain previously uncovered tasks while losing previously covered ones. In one general-task case, reliability rises by 5.2 points despite losing ten previously covered tasks. Most reliability gains also increase deployment cost. Specific changes usually outperform generic reminders, but their benefits can change when combined. The highest-scoring agent does not yield the largest mean improvement in any of our four pools, and stricter admission can discard useful alternatives. Transfer selectively preserves gains: coverage usually falls on the general-task target across both harness types and paradigms, but rises on the coding target with the outer-layer-only harness. These findings motivate preservation-aware evaluation, tested decision rules, and explicit applicability conditions. As a constructive follow-up, EvoAudit turns these diagnostics into revisions of select, analyze, mutate, and gate, achieving the best performance.