AI Daybook
AI Daybook
오늘의 지면
논문연구·논문

터미널 스타일 수학 모델링 벤치, 18개 실제 문제로 에이전트 검증

Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling

매체
Hugging Face
게시
중요도
3/5
화제 지수
9

한국어 요약

문제를 해결할 수 있는지 뿐 아니라 그 답이 물리적으로 가능한지까지 확인하는 수학 모델링 벤치마크 TB-MathModel을 제안했다. 2023~2025년 CUMCM, MCM의 실제 문제 18개를 컨테이너화한 터미널에서 실행하며, 프로그래매틱 검증과 624개 Lean4 증명을 통해 불가능한 답을 필터링한다. 상위 모델은 다중 기준 점수 64.58%에 도달했다.

원문 발췌 · English

Mathematical modeling turns an open-ended real-world problem into assumptions, a model, a solution, and a defended recommendation, and its answers matter only if they are both close to a credible reference and physically feasible; existing benchmarks either score solver fragments or grade reports with an LLM judge, leaving those numbers unchecked. We introduce TB-MathModel (the Terminal-style Mathematical Modeling Bench), the first benchmark to make complete contest-style modeling problems agentic, verifiable, and terminal at once, over 18 real CUMCM and MCM problems (2023--2025). Each task ships the verbatim statement, official attachments, and figures in a containerized terminal; the agent returns one JSON artifact, scored on the numeric core of the final question against a reproduction of the Outstanding-award paper. Two design choices distinguish it: direction-aware, one-sided scoring, which never penalises beating the anchor, and two-layer feasibility verification---a programmatic gate over six decidable invariant forms plus 624 Lean~4 obligations---with a hard gate that zeroes infeasible answers. Each task is scored against a panel of human outstanding papers rather than one, so a model must be close to many accepted answers; across seven multimodal entries (three seeds each) the strongest reach a multi-reference O-Eval of 64.58\%, while the \2.54 open model holds 59.81% at 1/100$ the cost. TB-MathModel turns the numeric core of modeling into a reproducible, machine-checkable signal for measuring long-horizon modeling competence and for collecting agent trajectories. The source code is available at https://github.com/wxj630/terminal-bench-math-modeling.

#agent#benchmark#mathematical-modeling
원문 기사 읽기
전체 보기 →