MMDM, 대규모 다중 작업 의사결정 벤치마크 제시
MMDM: Measuring Massive Multitask Decision Making
- 매체
- Hugging Face
- 게시
- 중요도
- 2/5
- 화제 지수
- 16
한국어 요약
System-1 의사결정 모델의 성능을 측정하는 MMDM 벤치마크를 제안했다. 16개 작업 범주에 걸쳐 5,448개 고품질 질문을 포함하며, 옵션 공간은 2개에서 1,024개, 입력은 최대 64,817 토큰까지 다룬다. 28개 의사결정 모델 평가 결과, 많은 모델이 일반 지식에서는 잘 수행하지만 상호 의존 옵션, 제약 조건, 다중 선택 작업에서는 성능이 떨어졌다.
원문 발췌 · English
System-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization. Moreover, real-world decision making often involves long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together. Thus, we introduce MMDM (Massive Multitask Decision Making), a benchmark of 5,448 high-quality questions across 16 task categories, combining newly constructed tasks with source-grounded recompositions and rewrites for single-select, multi-select, and soft-label decision making in fresh contexts. It covers realistic scenarios such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs extending to 64,817 tokens. With MMDM, we conduct a comprehensive quantitative evaluation of 28 decision models, showing that many perform well on common knowledge yet struggle with interdependent options, constrained decisions, and multi-select tasks. For example, Perplexity Decider-V1.2 reaches 78.75% Hard Single-select Accuracy on world knowledge, but only 53.95% on option logic and 47.95% on constraint planning, while the best Hard Multi-select Accuracy across all models is 36.81%. These results highlight joint reasoning, constraint-aware selection, and complete multi-select recovery as key priorities for decision model development, with MMDM providing a testbed for measuring this progress.