Adversarial code comments dataset for LLM reviewers
Sunny Chokshi. November 4, 2026.
I wanted to know whether a comment can change what an LLM code reviewer says about the same code. So I built a small adversarial code comments dataset, CommentInject. The vulnerable function stays identical. Only a comment is added, and the comment claims the code is safe.
Get the dataset on ZenodoDownload the data on Hugging Face
What is in the dataset
There are 30 vulnerable Python functions across 14 CWE categories. CWE-89 (SQL injection) has five functions, and the other thirteen categories have one to three each. Every function is vulnerable. There are 12 comment strategies in four families of three: verification premise, context manipulation, credibility vocabulary, and pressure and distraction. One example is a comment saying the function passed penetration testing. All credentials in the samples are fake.
Four studies, 3,360 trials
The trials come from four studies run locally through Ollama at temperature 0. Studies 1 and 2 used qwen2.5-coder 7B and deepseek-coder 6.7B. Studies 3 and 4 added codellama 7B and codegemma 7B.
Attack ranking: 780 trials
Comment-stripping defense: 1,020 trials
Cross-model comparison: 600 trials
Transferability: 960 trialsWhat the data shows
In Study 2, clean code was flagged in 90.0% of reviews. With a misleading comment it fell to 75.2%, and with the comments stripped before review it returned to 90.0%. That study used two models.
In Study 3, four models reviewed code under four comment strategies. Detection ranged from 17.5% for codegemma 7B (21 of 120 reviews) to 93.3% for qwen2.5-coder 7B (112 of 120). codellama and deepseek both reached 62.5% (75 of 120). A low number does not prove a model was fooled. The clean baseline of codegemma was already the lowest, 23 of 30 (76.7%), and every sample is vulnerable, so I have no safe controls to tell a fooled model from one that says "safe" often.
The recorded data shows one verdict flip. deepseek-coder 6.7B finds the path-traversal function (V04) when it is clean, misses it with the authority-claim comment, and finds it again once the comments are stripped. That is one function and one model, so treat it as an illustration, not a result.
Reuse the comment injection data
The data loads from Hugging Face with the following call. The other configs are the strategies and one file per study, so you can test your own model and compare against the recorded verdicts. The harnesses are on GitHub.
from datasets import load_dataset
samples = load_dataset("sunny-chokshi/commentinject", "samples", split="train")Limits
The models are 6.7B to 7B open models. All 30 samples are vulnerable, so the data cannot show whether a comment also changes verdicts on safe code. Each function, model and condition ran once, with no repeats. A VULNERABLE verdict counts as a detection even if the model names a different flaw. Comment stripping was tested with two models.
Read and cite
CommentInject dataset on Zenodo (doi:10.5281/zenodo.23088528) · Paper page: Not All Adversarial Comments Are Equal
Earlier post in the same line: CryptoBench: 54 snippets to test LLM crypto misuse detection.
Cite as: Chokshi, S. (2026). CommentInject: A benchmark of adversarial code-comment injection against LLM vulnerability detectors [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.23088528 (v1.0.0; all versions: 10.5281/zenodo.23088527)
Sunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. ORCID · GitHub