Heldt et al.: exploit, patch, then test a different exploit

Tobias Heldt, Matt Turk, Christoph Landolt and Mario Fritz, “Does AI Help Cyber Attackers or Defenders? Evidence from Nonpublic Vulnerabilities and Subsequent Attacks,” preprint posted 5 October 2026. Question hub. The team evaluated agent systems (model plus software/tool harness) on exploit and patch tasks in five hand-picked nonpublic software environments, including privately disclosed flaws, and separately on 209 disclosed memory-safety vulnerabilities and cryptographic challenges. Deterministic task graders, developed by the researchers, score exploit success and patched security/functionality; model-assisted weakness classifications are not ground truth. Attack and repair roles have different scoring criteria, and the samples are expertise-selected.

Repair scores beat attack scores in two nonpublic environments; attack beat repair in three. On the same 209 disclosed flaws, GPT-5.5 with OpenCode attacked 201 and repaired 178; Opus 4.7 with Claude Code attacked 167 and repaired 120; GPT-5.3-Codex with OpenCode attacked 161 and repaired 175, but the repair-favouring 6.7-percentage-point difference has a repository-cluster 95% interval of −1.8 to 15.1, so its sign is uncertain. Different model/harness pairings also confound a pure ‘model’ comparison. An intervention with source-location hints improved repair scores in selected tasks; structured versus generic review yielded eight ties and one improvement among nine matched pairs, so does not establish a general review benefit.

Residual risk: after a tested first exploit was stopped, another tested exploit succeeded in 92 of 524 intervals without intervening code change. Intervals can reflect intermediate software states and repeated sessions, so that is not a rate of production-patched vulnerabilities. In 117 defender-first rounds where code was changed, attackers won 115 under the full task definition, but an ‘attacker win’ can also mean the patch did not build or broke legitimate functions; only five selected rounds had detailed patch-status review. One reviewed attacker circumvented a patch shown to stop the first exploit. Agent claims that security/functionality checks passed often disagreed with external tests. The private code and raw agent logs are withheld pending disclosure, while anonymized outcome tables and scripts allow table recomputation, not independent vulnerability verification.

Implication: measured attack/repair differences change direction with environment and harness. A government’s finding tally alone does not prove defensibility, nor do offensive exploit benchmarks establish helpless defenders. A realistic race must also measure human review and time to a robust deployed patch; this experiment measures autonomous task performance, not matched human-team productivity or real malicious campaign reach. Hold the long original for weekend if the short UK operator case study receives interest.