TRIAD: 멀티모달 LLM의 다중 정보 모호성 해석 벤치마크
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
- 매체
- Takara TLDR
- 게시
- 중요도
- 2/5
- 화제 지수
- —
한국어 요약
텍스트, 이미지, 오디오 채널에 분산된 모호성을 해석하는 멀티모달 LLM의 성능을 측정하는 TRIAD 벤치마크가 제안됐다. 14개 모델 평가 결과 최고 성능 모델도 인간 수준을 20점 이상 하회하며, 현재 시스템들이 오디오 신호를 충분히 활용하지 못함을 보였다.
원문 발췌 · English
Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current multimodal benchmarks mostly test perception, recognition, or reasoning over already determinate inputs, leaving unclear whether omni-modal large language models can resolve ambiguity distributed across text, vision, and audio. We introduce TRIAD, a benchmark for omni-modal ambiguity resolution. Each item is a text--image--audio triplet paired with a question and a set of answer options, and is constructed so that the full triplet determines a unique answer while removing any one modality makes the set of consistent options non-unique. TRIAD contains $327$ hand-written queries in $86$ scene groups and covers $18$ ambiguity categories across the three modalities. Evaluating $14$ omni-modal systems reveals that strong item-level performance does not translate into robust scene-level disambiguation: the best model still trails humans by over $20$ points in group accuracy. Removing answer options widens the gap further, driving model group accuracy to the single digits while humans remain robust. Leave-one-out and transcript/description counterfactuals further show that current systems often underuse audio cues when text and image are present. TRIAD reframes omni-modal evaluation around disambiguation rather than recognition, and provides a taxonomy, benchmark, and diagnostic protocol for measuring this capability.