Checklist for evaluating an AI security auditing tool or AI-assisted audit
==========================================================================

What to ask before trusting an AI auditing tool or an AI-assisted audit report: language and code coverage with public case studies, proof-of-concept per finding, precision and recall on uncontaminated benchmarks, named human validation, threat-model input, data handling and prompt-injection isolation, re-scan pricing, and disclosure paths.

Coverage: Which languages and frameworks are supported, and is there a public case study on code like mine (ZK circuit, MPC library, TLS stack, Solidity)? | Does the tool understand cryptographic specifications (field arithmetic, constraint systems, pairing checks), or only general code patterns? | Can I supply a threat model or scope file, and does the vendor show it measurably reducing noise?
Evidence quality: Do findings ship with a reproducible proof of concept or executable test, or only prose? | What are precision and recall on a benchmark that post-dates the model's training cutoff, and can I see the full findings list rather than only the hits? | Are severities calibrated by a named human, and does the report say who?
Validation and accountability: Who validates findings before I see them, and what false-positive rate has been observed on real engagements? | Is the report signed by a named reviewer (AI-assisted audit) or delivered as machine output (AI audit)? | What is the disclosure path if the tool finds bugs in my dependencies?
Security of the pipeline: Where does my code run, which models see it, and is it used for training? | How is repository content isolated from agent instructions (prompt injection), and what credentials does the agent hold? | Are results reproducible run to run, and are runs logged with model versions and dates?
Economics: How are re-scans triggered, priced and deduplicated as models change? | Does pricing reward validated findings or raw volume? | What human triage time should I budget per scan, based on the vendor's own case studies?
Red flags: Accuracy claims such as '100 percent' or '92 percent detection' with no dataset or methodology. | Findings without file and line references, or severities passed through from the model unchanged. | Benchmarks built only from public audit reports the model may have trained on.

Source page: https://agentsast.com/checklist/
Compiled by: agentsast editors (https://agentsast.com/about/)
Last reviewed: 2026-09-13