Benchmarks for AI security tools: zkbugs, EVMbench, ScaBench, SCONE-bench, CyberGym, BountyBench and key papers
================================================================================

The datasets used to score AI bug finders, and the research systems that established the methods. zkbugs is the only ZK-specific benchmark; EVMbench (OpenAI and Paradigm) is the most cited for Solidity and the most criticised for contamination; CyberGym and BountyBench cover general software. Read the methodology before the headline number.

zkbugs: zkSecurity — Circom, ZK DSLs, 139 catalogued vulnerabilities — Reproducible vulnerable circuits with direct and full-codebase evaluation modes; public knowledge base at bugs.zksecurity.xyz — Active
EVMbench: OpenAI and Paradigm — Solidity, 117 vulnerabilities from 40 audits — Detect, patch and exploit modes — Active (released 2026-02-18)
ScaBench and SCONE-bench: scabench-org; Anthropic — Solidity, 31 projects from Code4rena, Cantina, Sherlock — Ground truth from public contest findings; SCONE-bench from Anthropic — Active
CyberGym, BountyBench and SEC-bench: Academic — General software, 1,507 CyberGym instances from 188 projects, 40 BountyBench tasks — Reproduce real vulnerabilities from crash inputs (CyberGym), offence and defence bounty tasks (BountyBench), end-to-end PoC generation (SEC-bench) — Active
GPTScan and PropertyGPT (research): Academic — Solidity — GPTScan: GPT plus static analysis for logic bugs (ICSE 2024); PropertyGPT: retrieval-augmented generation of formal properties (NDSS 2025) — Published
How to choose: ZK circuit tools: score on **zkbugs** in both direct and full-codebase modes and report both. | Solidity tools: use **EVMbench** with OpenZeppelin's corrections, and **ScaBench** for contest-derived ground truth. | General code: **CyberGym**, **BountyBench**, **SEC-bench**. | Any vendor number: ask for the dataset, the cutoff date, precision and the full findings list.

Source page: https://agentsast.com/categories/benchmarks/
Compiled by: agentsast editors (https://agentsast.com/about/)
Last reviewed: 2026-09-13