{
  "slug": "benchmarks",
  "name": "Benchmarks and research",
  "short": "benchmark",
  "title": "Benchmarks for AI security tools: zkbugs, EVMbench, ScaBench, SCONE-bench, CyberGym, BountyBench and key papers",
  "summary": "The datasets used to score AI bug finders, and the research systems that established the methods. zkbugs is the only ZK-specific benchmark; EVMbench (OpenAI and Paradigm) is the most cited for Solidity and the most criticised for contamination; CyberGym and BountyBench cover general software. Read the methodology before the headline number.",
  "intro": [
    "Benchmark numbers drive purchasing decisions, so their construction matters. Detection benchmarks built from public audit reports measure recall of training data as much as discovery; OpenZeppelin's March 2026 audit of EVMbench found invalid high-severity items and contamination risk, and Nethermind noted that recall-only scoring hides false positives. The best benchmarks post-date model training cutoffs, count precision as well as recall, and publish the full findings list.",
    "The papers listed here are the ones vendors cite: GPTScan for LLM plus static analysis, PropertyGPT for LLM plus formal verification, Hound for knowledge-graph reasoning, and the 2026 cryptanalysis paper that uses agents to generate hypotheses tested by exact computation."
  ],
  "choose": [
    "ZK circuit tools: score on **zkbugs** in both direct and full-codebase modes and report both.",
    "Solidity tools: use **EVMbench** with OpenZeppelin's corrections, and **ScaBench** for contest-derived ground truth.",
    "General code: **CyberGym**, **BountyBench**, **SEC-bench**.",
    "Any vendor number: ask for the dataset, the cutoff date, precision and the full findings list."
  ],
  "tools": [
    {
      "name": "zkbugs",
      "page": "https://agentsast.com/tools/zkbugs/"
    },
    {
      "name": "EVMbench",
      "page": "https://agentsast.com/tools/evmbench/"
    },
    {
      "name": "ScaBench and SCONE-bench",
      "page": "https://agentsast.com/tools/scabench/"
    },
    {
      "name": "CyberGym, BountyBench and SEC-bench",
      "page": "https://agentsast.com/tools/cybergym/"
    },
    {
      "name": "GPTScan and PropertyGPT (research)",
      "page": "https://agentsast.com/tools/gptscan-propertygpt/"
    }
  ],
  "page": "https://agentsast.com/categories/benchmarks/",
  "updated": "2026-09-13"
}