What is Benchmark contamination? Definition and why it matters
==============================================================

The model has seen the audit report or the bug in training, so a benchmark hit measures recall of memory rather than discovery.

Detail: OpenZeppelin raised this about EVMbench; it applies to every benchmark built from public audits, including zkbugs. The defences are benchmarks that post-date model cutoffs, private held-out sets, and full-codebase modes that test search rather than recognition.

Source page: https://agentsast.com/glossary/benchmark-contamination/
Compiled by: agentsast editors (https://agentsast.com/about/)
Last reviewed: 2026-09-13