Switch-SWE, 코딩 에이전트의 작업 전환 성능 평가 벤치마크
Switch-SWE: Do Coding Agents Hold Focus When Switching Between Unfinished Tasks?
- 매체
- Hugging Face
- 게시
- 중요도
- 2/5
- 화제 지수
- 9
한국어 요약
코딩 에이전트가 한 저장소 내에서 여러 미완료 작업 사이를 전환할 때 초점을 유지하는지를 측정하는 벤치마크 Switch-SWE를 소개했다. 19개 Python 저장소에서 200개의 초점 작업으로 구성되며, 7개 에이전트를 평가한 결과 작업 간 전환 시 평균 11.5 퍼센트 포인트의 성능 격차가 발생했다. 작업 관련 히스토리와 상태 요약을 관리하는 모듈로 5.5~15.5 퍼센트 포인트 개선 가능함을 보였다.
원문 발췌 · English
Software development sessions often involve several unfinished tasks rather than one issue carried through to completion. As coding agents take on repository-level development, they need to recover an earlier task's requirements and partial progress after intervening work. Existing repository-level benchmarks typically score one issue-to-patch episode and do not directly measure whether an agent can resume a task after advancing other unfinished work in the same repository. We introduce \ourmethod, a benchmark for focus-task resumption in sessions that interleave unfinished tasks drawn from the same source repository, with a persistent checkout for each task. Each focus task is evaluated in matched uninterrupted and switched sessions with identical focus requirements and executable verifiers. The switched session interleaves ordered stages from other unfinished tasks in the same repository, while only the final focus patch is scored. \ourmethod~contains 200 focus tasks from 19 Python repositories, reconstructed from issue, pull-request, and review histories and rendered as 400 matched session specifications. Across 7 coding agents, interleaved unfinished work produces an average focus-task resolution gap of 11.5 percentage points, ranging from 6.0 to 15.0 percentage points. Trajectory analysis identifies cross-task requirement confusion and loss of task state as recurrent resumption behaviors. We introduce a plug-in context-management module that selects task-relevant history and maintains concise task-state summaries. On OpenCode, the module improves focus-task completion under same-repository task switching by 5.5 to 15.5 percentage points across the evaluated models. Code is available at github.com/Miraclecsc/Switch-SWE.