Adversarial code comments dataset for LLM reviewers
30 vulnerable Python functions, 12 comment strategies, and 3,360 recorded trials from four studies.
RESEARCH NOTES
Experiments, data, and methods
Short notes from my experiments on LLM-based code security review. Posts link to their preprints and datasets so readers can inspect the evidence and reuse the data.
LLM weak random number generator detection test
Three local LLMs review four Python examples containing the same weak random number generator bug.
CryptoBench: 54 snippets to test LLM crypto misuse detection
Nine misuse classes, 1,890 recorded trials, and a paired secure control set.
Detection rate alone ranks the worst LLM code reviewers first
Why secure twins and balanced accuracy change the model ranking.
A hex literal made LLM code reviewers raise false alarms
A secure 3072-bit key written as 0xC00 still caused false alarms in model review.
Browse the original blog archive at all research notes or subscribe through the RSS feed.