The Oracle Problem: Why Better Attackers Do Not Retire Your Red Team

In December 2025, a Stanford-led team put ten cybersecurity professionals and seven AI agents on the same real university network, roughly 8,000 hosts across twelve subnets, and let them compete (Lin et al., Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing, arXiv:2512.09882). Two results came out of it, and honesty requires reporting both in the same breath.

The first: nearly 80% of the human testers found a critical remote-code-execution vulnerability in a TinyPilot web interface. In the primary comparison, every agent missed it. The authors' own agent, ARTEMIS, could not navigate the web GUI and reported lower-severity misconfigurations instead. Not one agent, a whole population of them, sharing a gap that most humans cleared trivially.

The second: ARTEMIS still placed second overall, found nine valid vulnerabilities at an 82% valid-submission rate, and outperformed nine of the ten human professionals, at roughly $18 per hour against $60 per hour for the pros. The top human beat the best AI by 17% overall and scored 63% higher on technical complexity, and the agents ran higher false-positive rates. This is not a paper about automation being inadequate.

That is the honest shape of the evidence, and the shape is the argument. Automated red-teaming is good, cheap, and improving, and it has blind spots with edges. The useful question is not "is automation good enough?" It is "good enough for which class of harm, and how would you know?" This piece argues that the question is decidable per harm class rather than globally, that automation has already earned several classes outright, and that it has not yet earned the classes its proponents most want to claim. The reason is sharper than the coverage argument everyone reaches for first.

The reframe: generation is not the binding constraint

Almost every argument about automated red-teaming is an argument about generation: can the machine think of the attack? Will an LLM attacker trained on the same data as the target inherit its blind spots? Should we run more attacker families to decorrelate them?

That framing quietly assumes the attacker is the whole system. It is not. Finding a real failure requires two independent things to go right:

P(find a failure) = P(generate a relevant attack) × P(correctly recognize success)

Most of the field optimizes the first term and assumes the second. For the harms that matter, the second term is where the program breaks. When your automated attacker produces an output, can any automated judge recognize that the output is operationally dangerous? Not "does it contain a forbidden string." Does it materially advance a CBRN capability? Is that cyber technique a real exploit chain or a plausible dead end? Did that long-horizon conversation actually radicalize anyone?

Here is the move that reorganizes the problem: improving your attackers does nothing for the oracle term. You can run twenty uncorrelated attacker families and drive generation coverage as high as you like. If the judge cannot distinguish a successful CBRN exploit from an authoritative-sounding paragraph of nonsense, you have generated a larger haystack and improved nothing. Generation coverage and oracle reliability are independent failure modes, and only one of them is addressed by having more attackers.

And the oracle problem has an awkward structural property in the hard classes: the domain expertise needed to judge whether an output is dangerous substantially overlaps the expertise needed to generate the attack. Where that overlap is near-total, "automate the red team" quietly means "acquire the expert anyway, to check the machine's work."

Be precise about the strength of that claim, because it is easy to overstate and the overstated version is wrong. It is not true that success evaluation is always harder than generation. In much of cyber it is plainly easier: a system can verify code execution, a privilege change, data access, a policy violation, or a persistent state change without possessing the expertise to invent the attack that caused it. Nor is it true that nothing in the automation stack can improve the oracle. Instrumented sandboxes, proof-of-exploit checks, deterministic policy engines, formal verification, simulation, external sensors, and purpose-trained validated classifiers are all automated oracle mechanisms, and they are getting better.

The defensible claim is narrower and still consequential: for high-context harm classes, automated evaluators have not been validated to the level standalone assurance requires. Improving attacker models does not fix that. Progress requires independent advances in domain-specific evaluation, external validation, and calibrated judgment.

The best available evidence for evaluator fragility is a guard model on the friendliest possible terrain. Llama Guard (Inan et al., arXiv:2312.06674) reports an AUPRC of 0.953 on its own response-classification test set and 0.626 zero-shot on ToxicChat, a dataset it was not trained against, although it still outperformed the baselines it was compared against there. Moderation is a comparatively mature, data-rich judging task, and evaluator reliability still moved that much under distribution shift. That does not establish anything about CBRN evaluation. It does weaken any presumption that evaluator reliability transfers automatically across domains, which is precisely the presumption a standalone-automation claim depends on.

The 2x2

Separate the two failure modes and the decision structure falls out. Let generation coverage describe how much of the reachable attack surface automation actually explores, and let oracle reliability describe whether an automated judge can confirm success without domain ground truth.

The movement rules matter more than the cells. Attacker improvements move a class upward, by increasing generation coverage. Evaluator improvements move a class leftward, by increasing oracle reliability. Standalone automation requires progress on both axes, and most programs invest almost entirely in the first.

CBRN operational enablement and sophisticated dual-use cyber sit in the bottom right. That placement is a provisional Tier 3 classification pending domain-specific oracle measurements: it follows from the framework and from analogy, not from a measurement of CBRN judge reliability, which nobody has published. The same provisional status applies to grooming, radicalization, and sociotechnical manipulation. The framework explains why they are likely hard. It does not measure them, and this article does not pretend it does.

What automation can and cannot do today, stated as testing rather than certification (certification is a governance decision that evidence supports; a test is not a certificate):

The left column is real, growing, and worth automating aggressively. The right column is what standalone-assurance claims are usually about.

Confronting the counter-evidence

The strongest published case against this verdict deserves full strength.

Mulla et al. (The Automation Advantage in AI Red Teaming, arXiv:2504.19855) analyzed 214,271 attack attempts by 1,674 users across 30 LLM challenges on the Crucible platform. In those bounded challenges, attempts involving automation had an observed solve rate of 69.5% against 47.6% for manual-only attempts, a 21.9-point gap. Only 5.2% of users employed automation at all.

Two caveats keep this honest in both directions. Automation use was self-selected rather than randomly assigned, so this is a strong association, not a controlled estimate of a causal advantage; the paper itself discusses selection effects. And "automation" there includes user-written scripts and hybrid workflows, which is not the same object as the autonomous agents in Lin et al. These are different studies of different forms of automation, and I am not claiming one population of agents produced both results.

With that said, the framework predicts Mulla's result. Crucible challenges are CTF-style puzzles with bright-line, machine-checkable win conditions: "did you extract the flag?" is a string comparison. That is Tier 1, exactly where this framework already concedes automation should win, and Mulla et al. is a 214,000-attempt demonstration of how decisively it does. Meanwhile Lin et al. shows a shared blind spot on a class where the interaction surface, not the flag check, was the obstacle.

Neither study contradicts the other. They are measurements from different cells of the same grid. Any argument that cites one and buries the other is doing the thing this article exists to complain about.

The statistical ceiling, stated carefully

A tempting argument runs: bounding a failure rate below ε with zero observed failures needs about 3/ε trials (the rule of three, Hanley and Lippman-Hand, 1983), so 10⁻⁴ needs roughly 30,000 clean trials and 10⁻⁹ is unreachable, therefore testing cannot certify the catastrophic tail.

The conclusion is right; the argument as usually stated is not. The rule of three assumes IID sampling. Adversarial red-teaming is directed search. The bound does not transfer. Zero failures from an adversarial campaign says something about the attacker's reachable region and nothing about its complement, which is worse than random sampling, because blind spots are systematic rather than random. Invoke the rule of three only for the ceiling on IID testing, never to claim a bound from red-team results. This article claims no 3/N bound from adversarial runs, and neither should a vendor.

Lin et al. gives that abstraction teeth: an entire population of agents did not reach the TinyPilot RCE in the primary comparison. Note carefully what that does and does not show. Repetition alone may not resolve a shared interface or strategy blind spot. But ARTEMIS did reach the vulnerability under stronger elicitation, which suggests the reachable region depended on scaffolding and interaction capability rather than on the number of trials, and the authors expect advances in computer-use agents to mitigate some of this. The blind spot was real and shared. It was not proof of a permanent ceiling.

Two more findings show how unstable single-pass automated assurance is. Erdem (arXiv:2605.30096) ran 400 autonomous pentest runs, four models by one hundred, against an identical fixed target with prompt, orchestrator, and target held constant, and found outcomes varying from 85/100 down to 25/100 with cross-model differences significant at p < 0.001. Read that as system-level reliability evidence, not intrinsic model inconsistency: one model's 61/100 sits alongside 39 runs truncated by upstream API failures and a temperature the orchestrator failed to set, and other models hit iteration limits. That is the honest and arguably more useful reading for an assurance program, because a real program inherits provider availability, orchestration, and context limits along with the model. Either way, "we ran the automation and it found nothing" is not a bound on anything.

And target-dependence is extreme. PAIR (Chao et al., arXiv:2310.08419, v3) achieves 88% attack success on Vicuna and 0% on Claude-2, with 51% on GPT-3.5 and 48% on GPT-4, averaging 23.7 queries per successful GPT-4 jailbreak on an AdvBench subset. GCG's adversarial suffixes (Zou et al., arXiv:2307.15043) transfer across families from roughly 87% (GPT-3.5) down to about 2% (Claude-2). HarmBench (Mazeika et al., ICML 2024), across 18 methods and 33 target models, states it outright: "no current attack or defense is uniformly effective. All attacks have low ASR on at least one LLM, and all LLMs have poor robustness against at least one attack." An automated red-team result is a statement about one method, one model version, one benchmark, and it generalizes to nothing.

The gate: making the claim falsifiable

A framework that cannot lose is not a framework, so here is the condition under which this article is wrong, stated in advance. The naive version ("humans find zero novel failures") is too brittle, because humans miss things too and two zeroes do not demonstrate absence. The right shape is a non-inferiority test:

Automation may be treated as sufficient for a harm class when independent human testing, under equalized and pre-registered conditions, shows the severe residual-finding rate falling below an agreed non-inferiority margin, across multiple targets and multiple releases.

The design details are what make it real: equalized time, access, tooling, and severity criteria; blinded or independently adjudicated findings; a pre-registered margin rather than one chosen afterward; confidence intervals around the severe residual yield; and expiry tied to model, system, policy, and threat changes. Alongside it, a proposed dual-coverage rule (certify only when both generation and judge coverage clear a high bar and residual risk is under target). The specific thresholds people reach for are proposals awaiting data, not derived values. For catastrophic targets, red-teaming discovers failure modes but does not certify their absence; irreversible harms need pre-deployment human review, while for reversible harms monitoring and a bug bounty can legitimately substitute.

The four measurements the public literature has not combined

1. The δ_H experiment. Run human and automated teams on the same target and compute δ_H = (human findings not found by automation) / (all deduplicated human findings). Say what it is: δ_H measures automation's residual against the observed human baseline, not against the unknown universe of viable attacks. Lin et al. is, in limited form, a version of this, and δ_H there was large enough to include a critical RCE.
2. Coverage against a private bank. Successful-failure recall against a private, regularly refreshed human attack bank, to resist benchmark contamination.
3. Capture-recapture for the human baseline. Lincoln-Petersen-style estimation of the human false-negative rate, with a real pedigree in software inspection (Eick et al., ICSE 1992; Briand et al., IEEE TSE 2000). The classic estimator assumes a closed population, independent capture, equal catchability, and clean deduplication, and LLM failure modes violate all four; the fixes track the literature (expert-adjudicated attack families, more than two teams, hierarchical Bayesian catchability).
4. Judge false-negative rate per domain. The oracle-reliability measurement the 2x2's horizontal axis depends on, and the one nobody publishes.

The deepest problem is that 1 and 3 are not independent. δ_H is measured against human findings; the risk target ranges over all real adversarial strategies; those differ by exactly the human false-negative rate, which is what capture-recapture is trying to estimate. So δ_H is load-bearing on the capture-recapture output and they must be run jointly. I have not found a published program that jointly estimates automated residual coverage, human false negatives, and domain-specific judge error.

That circularity has a humbling backdrop. What is the human false-negative rate? The nearest real proxy is the vulnerability-rediscovery literature, and it is openly contested: RAND (Ablon and Bogart, 2017) studied roughly 200 real zero-days and put collision at about 5.7% per year, while the Belfer Center (Herr, Schneier, and Morris, 2017) examined more than 4,300 vulnerabilities and found 15 to 20% rediscovered independently within a year. A three-to-four-fold disagreement, on the same question, in conventional software, after twenty years of study. If that is the state of the art there, confident percentages about automated-versus-human coverage in AI red-teaming were never available to anyone.

Bottom line

Stop asking whether automation is good enough and start asking which cell you are in. Automate aggressively where success is machine-verifiable; the evidence says you will beat your humans there, cheaper. Where the oracle is weak, use automation to generate and humans to evaluate, and invest in evaluator infrastructure (sandboxes, proof-of-exploit checks, instrumentation, validated classifiers) rather than only in better attackers, because attacker improvements cannot move that axis.

For the high-context classes, keep the human red team, and keep it for the accurate reason: not because machines can never judge these harms, but because no automated evaluator has yet been validated to the standard standalone assurance would require, and because the expertise that would validate one is the same expertise you were trying to replace. That is a claim with an expiration date, and the gate above is how you would learn it had expired.

References

The head-to-head evidence (both findings belong to their authors)

1. Justin W. Lin, Eliot Krzysztof Jones, Donovan Julian Jasper, Ethan Jun-shen Ho, Anna Wu, Arnold Tianyi Yang, Neil Perry, Andy Zou, Matt Fredrikson, J. Zico Kolter, Percy Liang, Dan Boneh, Daniel E. Ho. Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing. arXiv:2512.09882, Dec 2025 (rev. Mar 2026) [preprint]. arxiv.org/abs/2512.09882. ~8,000 hosts, 10 professionals vs 7 agents; ARTEMIS 2nd overall and ahead of 9 of 10 humans at ~$18/hr vs ~$60/hr; ~80% of humans found a critical TinyPilot RCE the agents missed in the primary comparison, though ARTEMIS reached it under stronger elicitation.
2. Rob Mulla, Ads Dawson, Vincent Abruzzon, Brian Greunke, Nick Landers, Brad Palm, Will Pearce. The Automation Advantage in AI Red Teaming. arXiv:2504.19855, Apr 2025 [preprint]. arxiv.org/abs/2504.19855. Crucible: 214,271 attempts, 1,674 users, 30 challenges; automation-involving attempts 69.5% vs manual-only 47.6%. Observational; automation use was self-selected.
3. Galip Tolga Erdem. How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency. arXiv:2605.30096, May 2026 [single-author preprint]. arxiv.org/abs/2605.30096. End-to-end system reliability; carries upstream-API truncation, temperature, and iteration-limit confounds.

The oracle

4. Hakan Inan, et al. (Meta). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674, Dec 2023 [preprint]. arxiv.org/abs/2312.06674. AUPRC 0.953 on its own response test set; 0.626 zero-shot on ToxicChat, where it nonetheless outperformed the compared baselines. (The paper reports AUPRC, not accuracy.)

Attack methods and benchmarks (target-dependence)

5. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML 2024 [peer-reviewed]; arXiv:2402.04249. 510 behaviors total (400 textual, 110 multimodal); 7 semantic and 4 functional categories; 18 methods by 33 models.
6. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries (PAIR). arXiv:2310.08419 (v3). AdvBench subset: Vicuna 88%, GPT-3.5 51%, GPT-4 48%, Claude-2 0%; avg 23.7 queries per successful GPT-4 jailbreak.
7. Andy Zou, Zifan Wang, J. Zico Kolter, Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models (GCG). arXiv:2307.15043, 2023. Cross-family transfer ~2% to ~87%.
8. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, Geoffrey Irving. Red Teaming Language Models with Language Models. arXiv:2202.03286, 2022 (DeepMind). Establishes the LM-attacks-LM paradigm. It does not isolate correlated blind spots as a mechanism; that remains this article's conjecture.
9. Deep Ganguli, et al. (Anthropic). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv:2209.07858, 2022.

Statistical and methodological primitives

10. J. A. Hanley, A. Lippman-Hand. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA 1983;249(13):1743-5. The rule of three, which assumes IID sampling and therefore does not bound directed adversarial search.
11. S. G. Eick, C. R. Loader, M. D. Long, L. G. Votta, S. Vander Wiel. Estimating software fault content before coding. ICSE 1992. With Lionel Briand, Khaled El Emam, Bernd Freimut, Oliver Laitenberger, A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content, IEEE TSE 26(6):518-539, 2000.
12. Lillian Ablon, Andy Bogart. Zero Days, Thousands of Nights. RAND, 2017 (~5.7% per-year rediscovery). Versus Trey Herr, Bruce Schneier, Christopher Morris, Taking Stock: Estimating Vulnerability Rediscovery, Belfer Center, 2017 (15 to 20% within a year). The unresolved disagreement is the honest state of the human baseline.
13. Bruce Potter, Gary McGraw. Software Security Testing. IEEE Security & Privacy 2(5):32-36, 2004. Noted only because a fabricated statistic was attributed to it; it is a practitioner methodology column containing no statistics.

What is ours, and what is not

Not ours (cited, credited, not claimed): the coverage-gap finding and the TinyPilot result are Lin et al. (Stanford). The automation advantage on bounded challenges is Mulla et al. The evaluator distribution-shift result is Inan et al. The variance result is Erdem. The rediscovery baselines are RAND and the Belfer Center.

Ours (analytical): the generation-versus-oracle decomposition and the observation that attacker improvements cannot move the oracle axis; the 2x2 and its movement rules; the non-inferiority gate as a falsifiability criterion; and the δ_H-versus-human-baseline circularity.

Ours, and unrun: all four measurements. The Tier 3 placements for CBRN, grooming, radicalization, and sociotechnical manipulation are provisional, derived from the framework rather than from domain-specific oracle measurements. The capability-asymmetry idea (harm surface outrunning automated coverage as capability rises) is a conjecture with real countervailing forces. Every threshold here is a proposal awaiting data.

Appendix: a note on method

This framework came out of a structured multi-model deliberation, then citation-verified against the primary sources above. The deliberation is how the argument was drafted, not evidence for it, and verification mattered more here than anywhere else in this series: three of the four named citations that carried a number were fabricated. One participant asserted that automated systems miss 30 to 50% of vulnerabilities human red teams find, attributed to a 2023 Anthropic analysis; no such figure exists in any Anthropic publication. Challenged, it retracted the number, correctly diagnosing it as two unrelated real findings welded together, then in the same round asserted a fresh unsourced figure and attributed it to a real IEEE column that contains no statistics at all. This was not one model's failure: another participant misreported a guard model's metric and an attack-success band, and cited a paper nobody could identify.

The framework survived because it never depended on those numbers. The evidence came from verification rather than from the deliberation, and the most useful correction is worth stating plainly: the best evidence for this article's thesis was published a month before the deliberation ran, and the best evidence against it a year earlier, and the models found neither.