RealCompanion: 장기 대화 기반 인간 이해 벤치마크
RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 20
한국어 요약
AI 동반자와 수개월간 대화한 실제 관계 10개(27,218개 메시지, 최대 120일)를 포함한 벤치마크 RealCompanion을 공개했다. 연구 결과 과거 정보는 대부분 필요하지 않으며, 메모리 필요 여부를 판단하는 탐지기는 실제 메시지에서 작동하지 않고, 동일한 페르소나 재구성에서 비용 차이가 31배에 달한다.
원문 발췌 · English
A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the past bears on the message in front of it. Testing this requires a real person's record, and such records are private, so benchmarks generate the person and the questions and settle in advance what matters. We release \bench, ten real relationships with an AI companion: 27,218 messages over up to 120 days, released as the conversation and four files derived from it, a profile, a persona, a chat ground truth and a question set, each citing the messages it rests on. Every chat label carries the reasoning trace that produced it, checked stage by stage against the conversation. Three findings follow. First, the past is rarely needed and far away. Pooled measures mislead: a recency window finds the required message for 95.9\% of probes and 2.2\% of those that need memory, and at the natural rate 96\% of the gain from supplying recorded evidence comes from messages that need none. Second, no detector we tried can tell when memory is needed on real messages, authored questions over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. Third, three agent systems reconstruct the persona with the same F1 at a 31-fold difference in cost.