Does Indirection Hurt? Paired Evidence on LLM Review of Indirectly Written Cryptographic API Misuse
Sunny Chokshi. Preprint, Zenodo, October 2026. doi:10.5281/zenodo.23351769.
Read the preprint on Zenodo · Download the PDF · Data and code · GitHub
LLM code reviewers are trustworthy only if their verdicts follow what the code does. I test this for cryptographic API misuse with 90 pairs of Python snippets across nine misuse classes, each pairing a vulnerable snippet with a secure twin. For 80 pairs I wrote a plain form, with the misuse at the call site under its familiar name, and an indirect form that reaches the same choice through an alias, constant, default, or configuration value. Across 16 open models, the secure twins show that two models flag nearly every snippet and four more flag over 70% of secure code. On the 80 pairs, the indirect form lowered the balanced accuracy of the seven strongest models by 4.5 points (pair-cluster 95% CI 1.3 to 7.6), and by 5.3 points (CI 2.1 to 8.5) on 47 pairs that survive a post hoc stimulus audit. After the audit the effect is mainly extra flags on correct code: false positives rose 9.1 points (CI 4.2 to 14.6), mostly in three models. In exploratory splits, most of the net false-alarm rise sits in pairs where the safe value has left its usual spelling or place at the call site, for example a 3072-bit key size written as 0xC00. Detection of real misuse fell 3.6 points over all 80 pairs (CI 1.0 to 6.5) and 1.5 points (CI -1.6 to 4.8) on the audited pairs.
Preprint, author version. Available under CC BY 4.0 via Zenodo. A revised version of record may appear in future IEEE conference proceedings.