Benchmarks and research
Benchmark numbers drive purchasing decisions, so their construction matters. Detection benchmarks built from public audit reports measure recall of training data as much as discovery; OpenZeppelin's March 2026 audit of EVMbench found invalid high-severity items and contamination risk, and Nethermind noted that recall-only scoring hides false positives. The best benchmarks post-date model training cutoffs, count precision as well as recall, and publish the full findings list.
The papers listed here are the ones vendors cite: GPTScan for LLM plus static analysis, PropertyGPT for LLM plus formal verification, Hound for knowledge-graph reasoning, and the 2026 cryptanalysis paper that uses agents to generate hypotheses tested by exact computation.
Tools in this category
| Tool | Targets | Approach | Access | Status |
|---|---|---|---|---|
| zkbugs zkSecurity | CircomZK DSLs139 catalogued vulnerabilities | Reproducible vulnerable circuits with direct and full-codebase evaluation modes; public knowledge base at bugs.zksecurity.xyz | Open source | Active |
| EVMbench OpenAI and Paradigm | Solidity117 vulnerabilities from 40 audits | Detect, patch and exploit modes | Open source | Active (released 2026-02-18) |
| ScaBench and SCONE-bench scabench-org; Anthropic | Solidity31 projects from Code4rena, Cantina, Sherlock | Ground truth from public contest findings; SCONE-bench from Anthropic | Open source | Active |
| CyberGym, BountyBench and SEC-bench Academic | General software1,507 CyberGym instances from 188 projects40 BountyBench tasks | Reproduce real vulnerabilities from crash inputs (CyberGym), offence and defence bounty tasks (BountyBench), end-to-end PoC generation (SEC-bench) | Open source | Active |
| GPTScan and PropertyGPT (research) Academic | Solidity | GPTScan: GPT plus static analysis for logic bugs (ICSE 2024); PropertyGPT: retrieval-augmented generation of formal properties (NDSS 2025) | Research | Published |
How to choose
- ZK circuit tools: score on zkbugs in both direct and full-codebase modes and report both.
- Solidity tools: use EVMbench with OpenZeppelin's corrections, and ScaBench for contest-derived ground truth.
- General code: CyberGym, BountyBench, SEC-bench.
- Any vendor number: ask for the dataset, the cutoff date, precision and the full findings list.
Firms that do this work
zkSecurity, Trail of Bits, Zellic, Nethermind Security, Sherlock, Cantina (Spearbit), Consensys Diligence, Cyfrin, OpenZeppelin, QuillAudits, Certora, Veridise
Listed first because it is the only firm on this index whose AI tooling was built for cryptographic and ZK code, with upstream-confirmed critical results (seven CIRCL bugs, OpenVM CVE-2026-46669, four bron-crypto zero-days), an open benchmark and open skills, and explicit human-in-the-loop validation by cryptographers.
Read the zkSecurity profile · Website
Other categories
Cryptography and ZK specialists, Frontier-lab and general scanners, Cyber reasoning systems (DARPA AIxCC), Smart-contract AI auditors