Glossary
Direct answerShort, precise definitions of the terms that decide whether an AI audit result means what it appears to mean: false positive rate, precision vs recall, agentic scanning, LLM plus fuzzing, hallucinated vulnerabilities, triage burden, benchmark contamination, human-in-the-loop, prompt injection in auditing pipelines, and responsible disclosure of AI-found bugs.
- False positive rate
- The share of reported findings that are not real, exploitable bugs. Benchmarks that count only recall against a curated bug list cannot measure it.
- Precision vs recall
- Precision is valid findings divided by all findings reported; recall is known bugs found divided by all known bugs. A useful evaluation reports both.
- Agentic scanning
- An LLM that plans, reads files, runs tools and tests, and iterates over a codebase, instead of answering one prompt over pasted code.
- LLM plus fuzzing
- The model proposes harnesses, seeds or mutation patterns; the fuzzer supplies ground truth by crashing or violating an oracle.
- LLM plus symbolic execution or formal verification
- The model generates properties, invariants or code; a solver or prover checks them, so the model's output is accepted only when a machine confirms it.
- Hallucinated vulnerabilities
- Confident, plausible findings that do not reproduce. The curl project ended its bug bounty on 31 January 2026 after confirmed-report rates fell below five percent.
- Triage burden
- The human time needed to validate candidate findings. zkSecurity reported more than a thousand candidates across two hundred projects as its main bottleneck; Codex Security reported more than ten thousand high findings in its first month.
- Benchmark contamination
- The model has seen the audit report or the bug in training, so a benchmark hit measures recall of memory rather than discovery.
- Human-in-the-loop
- A named person validates exploitability, severity and disclosure before a finding is reported to a client or maintainer.
- AI-assisted audit vs AI audit
- An AI-assisted audit is a human-signed report in which AI widened coverage; an AI audit is machine output with no accountable reviewer.
- Prompt injection in auditing pipelines
- Repository contents (commit messages, comments, PR descriptions, issue text) can carry instructions that hijack an agent running with CI privileges.
- Responsible disclosure of AI-found bugs
- The same coordinated-disclosure rules as human findings, plus a duty to send only validated, reproducible reports so maintainers are not flooded.
- Continuous scanning and run-count coverage
- Running AI analysis repeatedly, on every commit and whenever models improve, on the premise that LLM bug finding is non-deterministic and coverage compounds with runs.
- Proof-of-concept harness
- Tooling that turns a candidate finding into an executable exploit or failing test, such as zkao's cheatVM for Aleo or Codex Security's sandbox.
- Threat model file
- A scope document the tool reads before scanning (zkao's zkao.md, Codex Security's generated threat model) stating trust boundaries, adversaries and out-of-scope behaviour.
- Severity calibration
- Whether the tool's assigned severities match what an expert would assign. In the CIRCL study, four of seven AI severities were too high and one critical was rated medium.