CIRCE, 제약 구조로 LLM 작업 난이도 사전 추정
CIRCE Divines LLM Task Complexity from Constraint Structure
- 매체
- Takara TLDR
- 게시
- 중요도
- 2/5
- 화제 지수
- —
한국어 요약
제약 조건의 구조를 그래프로 변환해 LLM이 문제를 풀기 전에 해결 난이도를 예측하는 CIRCE 기법을 제안했다. 7개 추론 벤치마크에서 GNN 인코더를 활용해 예측 정확도를 개선했고, 기존 기준 대비 AUROC를 0.691에서 0.727로 높였다.
원문 발췌 · English
Estimating whether a problem is too complex for a large language model (LLM) before the model attempts it can guide model routing, how to allocate test-time compute, and suggest the need for new tools to aid the agent. While existing estimators rely on the problem text, LLM-annotated task difficulty labels, or signals obtained after a model attempts to solve the problem, such as its hidden states, none explicitly leverage the constraint structure inherent in the problem: the variables to assign and how the constraints over them interact. We introduce CIRCE, which extracts a problem's structure by using a cheap LLM to construct a Constraint Intermediate Representation that deterministically compiles into the constraint graph. Across seven reasoning benchmarks (including SAT, logic, and math) and four LLMs, even a randomly initialized GNN encoder improves predictions of LLM response correctness, reducing negative log-likelihood from 0.465 to 0.423 and increasing AUROC from 0.691 to 0.727 relative to the strongest baseline for each metric.