AI Daybook
AI Daybook
오늘의 지면
논문에이전트

HEELO, 에이전트 네이티브 프리미티브로 현실 업무 자동화

The Last Mile to the Real World: Agent-Native Primitives as Reusable Building Blocks for Solving Real-World Tasks

매체
Hugging Face
게시
중요도
2/5
화제 지수
12

한국어 요약

에이전트가 실제 업무를 끝까지 처리하기 위해 리소스에 도달한 후 적용하는 단계(마지막 마일)가 병목인 문제를 다룬다. 에이전트 네이티브 프리미티브(상주 LLM이 있는 재사용 가능한 유닛)와 하네스 엔지니어링 프레임워크 HEELO를 제안했다. Terminal-Bench 4.0에서 해상도 37.3%에서 55.5%로 향상되었고, heelo.dev 플랫폼의 46개 실제 업무에서 Claude Opus 5.5와 GPT-6.1 Sol이 단독으로는 각 9개씩만 완료했으나 HEELO 연동 시 39개, 35개를 완료했으며 완료당 비용도 1.55배 낮았다.

원문 발췌 · English

How can an agent handle a real-world task end to end? Frontier models with their own harnesses still fall short on real-world benchmarks: on Terminal-Bench 4.0, GPT-5.6 Sol with Codex resolves 37.3% of the tasks and Claude Opus 5 with Claude Code 51.8%. These tasks fail at the step from reaching a resource to applying it, which we call the last mile to the real world. Existing approaches put the resource within reach but leave the competence to apply it with the caller, so each harness rebuilds that competence. We introduce agent-native primitives, reusable units that equip a resource with a resident LLM that applies it in context and communicates with other primitives, and HEELO (Harness Engineering around Evolving Agent-Native Primitives for the Last Mile to Real-WOrld Tasks), a harness engineering framework that builds each task's harness from a library of such primitives. HEELO composes the primitives a task needs into a task-specific graph, verifies the result against evidence from the environment, revises the primitives a failure implicates, and admits new or improved primitives after validation. We instantiate HEELO with eight primitive families, each built for one kind of task, and evaluate them on more than twenty benchmarks. Dev-Primitives raise Terminal-Bench 4.0 resolution from 37.3% under Codex to 55.5% on the same backbone, Harness Primitives composed per task raise Terminal-Bench 2.0 Pass@1 from 60.0% for the seed harness to 72.2%, Tool Primitives raise ToolBench pass rate from 55.8% to 75.1%, and Navigation Primitives raise GAIA-Text-103 accuracy from 45.6% to 58.3%. On the 46 real-world tasks of our heelo.dev platform, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex complete 9 each on their own and 39 and 35 when connected to HEELO through our MCP server, with 1.9times and 2.3times less time and 1.55times lower cost per completed task.

#agent#benchmark#lmm
원문 기사 읽기
전체 보기 →