Sentry, LLM 에이전트 실패 복구 시스템
Sentry: Learning to Recover from LLM Agent Failures at Test Time
- 매체
- arXiv
- 게시
- 중요도
- 2/5
- 화제 지수
- 2
한국어 요약
LLM 에이전트의 실패를 관리하는 계층인 Sentry는 실패 감지 시 외부 플레이북에서 교훈을 검색해 복구를 유도하고 검증하여 새 교훈을 저장한다. 여러 에이전트 벤치마크에서 기존 기법 대비 평균 37~39% 성능 향상을 보였고 학습한 교훈이 미학습 작업으로 이전된다.
원문 발췌 · English
LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability. We find that how failure knowledge reaches the agent matters as much as what it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37\% on average, and the strongest context-evolution baseline by 39\% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.