Last Secure Code Benchmark, AI 코딩 에이전트 보안성 평가
Last Secure Code Benchmark: Is the Code You Vibe Secure?
- 매체
- Hugging Face
- 게시
- 중요도
- 3/5
- 화제 지수
- 13
한국어 요약
150개 실제 취약점 기반의 Last Secure Code Benchmark는 450개 과제로 7개 프로그래밍 언어와 CWE Top 25 중 21개를 포함한다. Claude Fable 5도 기능과 보안을 모두 충족하는 코드는 25.1%에 불과했다.
원문 발췌 · English
AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.