CryptoBench
A benchmark of vulnerable Python snippets paired with secure controls. A model that flags every example is evaluated against both bug detection and false alarms.
54 snippets9 misuse classes36 vulnerable + 18 secure1,890 trials
OPEN DATA AND CODE
Datasets for evaluating AI code review
Download the test cases, inspect the recorded model responses, and reproduce analyses. Dataset contents use CC BY 4.0; code licenses are noted for each release.
CommentInject
Adversarial comments added to vulnerable code to measure whether LLM reviewers still report the underlying bug.
30 vulnerable functions14 CWE categories12 comment strategies3,360 trials
Detection Rate Is the Wrong Metric
Verdict logs, scripts, and checks for matched safe-code controls. The logs include the 10,580 model calls used in the paper across five Qwen2.5-Coder sizes.
10,580 model calls5 model sizes
Does Indirection Hurt?
Matched cryptographic misuse snippets with secure twins, opposite-form counterparts, an audit list, and logged model replies.
90 matched pairs14,592 calls16 open models33 pairs audited
Models ran locally through Ollama at temperature 0. Analysis scripts recompute headline rates with Wilson 95% confidence intervals. The replication package rebuilds Devign sets with loaders; it does not redistribute Devign function text.