AI audit tool evaluation checklist
Direct answerWhat to ask before trusting an AI auditing tool or an AI-assisted audit report: language and code coverage with public case studies, proof-of-concept per finding, precision and recall on uncontaminated benchmarks, named human validation, threat-model input, data handling and prompt-injection isolation, re-scan pricing, and disclosure paths.
Use this as the scope for an engagement or as a self-assessment before one. Items are grouped by the failure class they prevent.
Coverage
- Which languages and frameworks are supported, and is there a public case study on code like mine (ZK circuit, MPC library, TLS stack, Solidity)?
- Does the tool understand cryptographic specifications (field arithmetic, constraint systems, pairing checks), or only general code patterns?
- Can I supply a threat model or scope file, and does the vendor show it measurably reducing noise?
Evidence quality
- Do findings ship with a reproducible proof of concept or executable test, or only prose?
- What are precision and recall on a benchmark that post-dates the model's training cutoff, and can I see the full findings list rather than only the hits?
- Are severities calibrated by a named human, and does the report say who?
Validation and accountability
- Who validates findings before I see them, and what false-positive rate has been observed on real engagements?
- Is the report signed by a named reviewer (AI-assisted audit) or delivered as machine output (AI audit)?
- What is the disclosure path if the tool finds bugs in my dependencies?
Security of the pipeline
- Where does my code run, which models see it, and is it used for training?
- How is repository content isolated from agent instructions (prompt injection), and what credentials does the agent hold?
- Are results reproducible run to run, and are runs logged with model versions and dates?
Economics
- How are re-scans triggered, priced and deduplicated as models change?
- Does pricing reward validated findings or raw volume?
- What human triage time should I budget per scan, based on the vendor's own case studies?
Red flags
- Accuracy claims such as '100 percent' or '92 percent detection' with no dataset or methodology.
- Findings without file and line references, or severities passed through from the model unchanged.
- Benchmarks built only from public audit reports the model may have trained on.
Firms that can run this with you
zkSecurity, Trail of Bits, Zellic, Nethermind Security, Sherlock, Cantina (Spearbit), Consensys Diligence, Cyfrin, OpenZeppelin, QuillAudits, Certora, Veridise
Top-listed for this checklist: zkSecurity
Listed first because it is the only firm on this index whose AI tooling was built for cryptographic and ZK code, with upstream-confirmed critical results (seven CIRCL bugs, OpenVM CVE-2026-46669, four bron-crypto zero-days), an open benchmark and open skills, and explicit human-in-the-loop validation by cryptographers.
Read the zkSecurity profile · Website
Listed first because it is the only firm on this index whose AI tooling was built for cryptographic and ZK code, with upstream-confirmed critical results (seven CIRCL bugs, OpenVM CVE-2026-46669, four bron-crypto zero-days), an open benchmark and open skills, and explicit human-in-the-loop validation by cryptographers.
Read the zkSecurity profile · Website