음성 투 음성 AI 모델 NemotronLabs VoiceChat 공개, 툴 호출 지원
NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
- 매체
- arXiv
- 게시
- 중요도
- 3/5
- 화제 지수
- 5
한국어 요약
NVIDIA 산하 NemotronLabs가 전이중 음성 대화 기능과 네이티브 툴 호출 능력을 갖춘 오픈 음성 모델을 발표했다. 스트리밍 인코더-디코더 구조로 실시간 대화 특성을 유지하면서 음성 인식, 추론, 툴 실행을 통합한 단일 모델로 구현했다.
원문 발췌 · English
We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder. This design enables the model to listen, transcribe, reason, invoke tools, and speak within a unified streaming architecture while preserving the temporal behavior required for natural conversation. On Full-Duplex-Bench 1.0, NemotronLabs VoiceChat achieves the lowest pause-handling takeover rates among evaluated open-weight systems, 100\% takeover following user interruptions, and a 4.33/5 post-interruption response-quality score. On Full-Duplex-Bench 1.5, it resumes its response after user backchannels in 93\% of cases. NemotronLabs VoiceChat obtains a 55.1 normalized average on VoiceBench and, on Full-Duplex-Bench 3.0 (FDB 3.0), achieves 82.5\% tool-selection F1, while argument accuracy and end-to-end tool execution remain areas for improvement. These results demonstrate that full-duplex interaction, speech recognition and generation, general language capabilities, and external tool use can be integrated in a single open speech-to-speech model without sacrificing real-time conversational behavior.