금융시장 기반 LLM 에이전트 능력 평가 벤치마크 제시
LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets
- 매체
- Hugging Face
- 게시
- 중요도
- 2/5
- 화제 지수
- 17
한국어 요약
LiveMACEBench는 실시간 금융시장을 테스트베드로 삼아 LLM 에이전트의 능력을 프로세스 관점에서 평가한다. 5개 최신 LLM을 30일간 평가한 결과, 실제 성과와 능력 측정 간 큰 격차가 있음을 밝혔다.
원문 발췌 · English
Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Five frontier LLMs operate along continuous trajectories under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. We evaluate them through both realized outcomes and mechanism-specific diagnostics derived from complete decision traces. Across 30 days of live evaluation, we find a pronounced outcome-capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use. Trace-level diagnostics further expose distinct bottlenecks across capabilities, demonstrating that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability. LiveMACEBench makes this distinction measurable, turning live markets from a performance leaderboard into a diagnostic environment for agent capability