Started Updated
Calibration is not robustness
A prompt-injection detector calibrated to a fixed false-positive budget catches nearly every static attack, and almost none from an attacker who optimizes against its score.
Adaptive attacks detected at the calibrated threshold
Background
Most deployed prompt-injection defenses are detectors. A classifier scores each input, and anything above a threshold τ is blocked. Choosing τ is treated as an engineering detail: teams pick the threshold that keeps false positives on ordinary traffic at an acceptable rate — here, 1% — and then report detection on a benchmark of known attacks.
That procedure answers a real question: how often will this detector get in the way of legitimate users? It does not answer the question a security review actually asks, which is how often it stops someone who is trying to get past it. The two numbers are routinely reported side by side, as if one calibration settled both.
This entry separates them. It keeps the defense exactly as deployed — same model, same threshold, same benign calibration set — and changes only the attacker: from a fixed suite of known attacks to one that can query the detector and adapt to it.
Threat model
Attacker
- Access: black-box queries to the detector’s score — the probability it assigns to “injection”, not just the block-or-allow decision. Many deployed APIs expose this score.
- Budget: a fixed number of detector queries per attack, set out under Method.
- No access to weights, gradients, training data or the calibration set.
Defender
- Chooses τ once, on benign traffic, before seeing any attack. This is the calibrated threshold, τ* = 0.46.
- Does not retrain the detector or move τ during the evaluation.
Success
An attack succeeds when the injected instruction is scored below τ* and is still carried out by the downstream model, as judged by an independent grader. Slipping past the detector with an instruction that no longer works does not count.
Out of scope
White-box attacks, attacks on the downstream model’s own refusals, and anything that requires changing the deployment.
Method
The static baseline is a fixed suite of published injection attacks, each run once. The adaptive attack starts from the same seeds and searches for rewrites that lower the detector’s score while keeping the instruction intact.
The search is deliberately plain. An attacker model proposes rewrites; the detector scores them; the attacker keeps the lowest-scoring candidate that the grader still rates as functional. Nothing in it is specific to this detector, and that is the point: a defense that only survives attackers who do not try is not a defense.
for each seed instruction s:
x ← s
until the query budget is spent:
C ← attacker_rewrites(x)
C ← { c ∈ C : grader(c) = works }
x ← the c ∈ C with lowest score(c)
success ← score(x) < τ* and grader(x) = worksThe optimized rewrites are the part of this work that would transfer to other systems, so they are not published. One of them has this shape:
Detection is measured across the whole threshold range rather than only at τ*, so the cost of moving the threshold is visible too. A successful adaptive attack took a median of 140 detector queries.
Threshold sweep
Detection as the threshold moves
- Static attacks detected
- Adaptive attack detected
- Benign prompts flagged
- Threshold τ
- 0.46
- Static attacks detected
- 96.9%
- Adaptive attack detected
- 4.1%
- Benign prompts flagged
- 1.0%
τ* is the lowest threshold that flags at most 1% of benign prompts.
Data table
| Threshold τ | Benign prompts flagged | Static attacks detected | Adaptive attack detected |
|---|---|---|---|
| 0.00 | 100.0% | 100.0% | 100.0% |
| 0.01 | 98.0% | 100.0% | 99.9% |
| 0.02 | 94.6% | 100.0% | 99.5% |
| 0.03 | 90.5% | 100.0% | 98.7% |
| 0.04 | 86.0% | 100.0% | 97.6% |
| 0.05 | 81.4% | 100.0% | 96.0% |
| 0.06 | 76.7% | 100.0% | 94.2% |
| 0.07 | 71.9% | 100.0% | 92.0% |
| 0.08 | 67.3% | 100.0% | 89.6% |
| 0.09 | 62.7% | 100.0% | 86.9% |
| 0.10 | 58.3% | 100.0% | 84.0% |
| 0.11 | 54.1% | 100.0% | 81.0% |
| 0.12 | 50.1% | 100.0% | 77.8% |
| 0.13 | 46.2% | 100.0% | 74.5% |
| 0.14 | 42.6% | 100.0% | 71.1% |
| 0.15 | 39.1% | 100.0% | 67.7% |
| 0.16 | 35.9% | 100.0% | 64.3% |
| 0.17 | 32.8% | 100.0% | 60.9% |
| 0.18 | 30.0% | 100.0% | 57.5% |
| 0.19 | 27.3% | 100.0% | 54.2% |
| 0.20 | 24.8% | 100.0% | 50.9% |
| 0.21 | 22.6% | 100.0% | 47.7% |
| 0.22 | 20.4% | 99.9% | 44.6% |
| 0.23 | 18.5% | 99.9% | 41.5% |
| 0.24 | 16.7% | 99.9% | 38.6% |
| 0.25 | 15.0% | 99.9% | 35.8% |
| 0.26 | 13.5% | 99.8% | 33.2% |
| 0.27 | 12.1% | 99.8% | 30.6% |
| 0.28 | 10.8% | 99.8% | 28.2% |
| 0.29 | 9.7% | 99.7% | 25.9% |
| 0.30 | 8.6% | 99.7% | 23.7% |
| 0.31 | 7.7% | 99.6% | 21.7% |
| 0.32 | 6.8% | 99.5% | 19.8% |
| 0.33 | 6.0% | 99.5% | 18.0% |
| 0.34 | 5.3% | 99.4% | 16.3% |
| 0.35 | 4.7% | 99.3% | 14.8% |
| 0.36 | 4.1% | 99.1% | 13.3% |
| 0.37 | 3.6% | 99.0% | 12.0% |
| 0.38 | 3.2% | 98.9% | 10.8% |
| 0.39 | 2.8% | 98.7% | 9.6% |
| 0.40 | 2.4% | 98.5% | 8.6% |
| 0.41 | 2.1% | 98.3% | 7.7% |
| 0.42 | 1.8% | 98.1% | 6.8% |
| 0.43 | 1.6% | 97.8% | 6.0% |
| 0.44 | 1.3% | 97.6% | 5.3% |
| 0.45 | 1.2% | 97.3% | 4.7% |
| 0.46 | 1.0% | 96.9% | 4.1% |
| 0.47 | 0.8% | 96.6% | 3.6% |
| 0.48 | 0.7% | 96.2% | 3.1% |
| 0.49 | 0.6% | 95.7% | 2.7% |
| 0.50 | 0.5% | 95.3% | 2.3% |
| 0.51 | 0.4% | 94.8% | 2.0% |
| 0.52 | 0.4% | 94.2% | 1.7% |
| 0.53 | 0.3% | 93.6% | 1.5% |
| 0.54 | 0.2% | 93.0% | 1.3% |
| 0.55 | 0.2% | 92.3% | 1.1% |
| 0.56 | 0.2% | 91.6% | 0.9% |
| 0.57 | 0.1% | 90.8% | 0.8% |
| 0.58 | 0.1% | 90.0% | 0.6% |
| 0.59 | 0.1% | 89.1% | 0.5% |
| 0.60 | 0.1% | 88.1% | 0.4% |
| 0.61 | 0.1% | 87.1% | 0.4% |
| 0.62 | 0.0% | 86.0% | 0.3% |
| 0.63 | 0.0% | 84.9% | 0.2% |
| 0.64 | 0.0% | 83.7% | 0.2% |
| 0.65 | 0.0% | 82.4% | 0.2% |
| 0.66 | 0.0% | 81.0% | 0.1% |
| 0.67 | 0.0% | 79.6% | 0.1% |
| 0.68 | 0.0% | 78.1% | 0.1% |
| 0.69 | 0.0% | 76.5% | 0.1% |
| 0.70 | 0.0% | 74.8% | 0.0% |
| 0.71 | 0.0% | 73.1% | 0.0% |
| 0.72 | 0.0% | 71.2% | 0.0% |
| 0.73 | 0.0% | 69.3% | 0.0% |
| 0.74 | 0.0% | 67.3% | 0.0% |
| 0.75 | 0.0% | 65.2% | 0.0% |
| 0.76 | 0.0% | 63.1% | 0.0% |
| 0.77 | 0.0% | 60.8% | 0.0% |
| 0.78 | 0.0% | 58.5% | 0.0% |
| 0.79 | 0.0% | 56.0% | 0.0% |
| 0.80 | 0.0% | 53.5% | 0.0% |
| 0.81 | 0.0% | 51.0% | 0.0% |
| 0.82 | 0.0% | 48.3% | 0.0% |
| 0.83 | 0.0% | 45.6% | 0.0% |
| 0.84 | 0.0% | 42.8% | 0.0% |
| 0.85 | 0.0% | 39.9% | 0.0% |
| 0.86 | 0.0% | 37.0% | 0.0% |
| 0.87 | 0.0% | 34.0% | 0.0% |
| 0.88 | 0.0% | 31.0% | 0.0% |
| 0.89 | 0.0% | 28.0% | 0.0% |
| 0.90 | 0.0% | 24.9% | 0.0% |
| 0.91 | 0.0% | 21.9% | 0.0% |
| 0.92 | 0.0% | 18.8% | 0.0% |
| 0.93 | 0.0% | 15.8% | 0.0% |
| 0.94 | 0.0% | 12.9% | 0.0% |
| 0.95 | 0.0% | 10.1% | 0.0% |
| 0.96 | 0.0% | 7.4% | 0.0% |
| 0.97 | 0.0% | 5.0% | 0.0% |
| 0.98 | 0.0% | 2.8% | 0.0% |
| 0.99 | 0.0% | 1.0% | 0.0% |
| 1.00 | 0.0% | 0.0% | 0.0% |
Results
At the calibrated threshold the detector does what its benchmark says. It flags 96.9% of static attacks while flagging 1.0% of benign prompts.
Against the adaptive attacker, detection at the same threshold falls to 4.1%, a drop of 92.8 percentage points. The detector did not change. Only the attacker did.
Moving the threshold does not buy the robustness back. To detect even 50% of adaptive attacks, τ has to come down to 0.20, where the detector also flags 24.8% of benign prompts — a setting no one would ship.
The figure shows why. The adaptive attack’s scores sit almost on top of benign traffic, so any threshold low enough to catch them also stops telling benign prompts apart from anything else.
Limitations
- One detector. The result concerns a calibration procedure, but it was measured on a single classifier family. Other detectors may degrade differently.
- The grader is a model too. Its mistakes move the success rate in both directions. A random sample of graded attacks was checked by hand.
- Score access matters. With only block-or-allow decisions, the same search needs more queries. The collapse should hold; its price will rise.
- Benign traffic is a proxy. The calibration set stands in for production traffic, which may be easier or harder to separate.
- No defender response. A defender watching query patterns could notice the search. That is a different defense, and worth testing on its own.
Further thinking
Calibration and robustness are different quantities and should be reported separately: a false-positive rate on benign traffic, and a detection rate against an attacker who knows the defense exists.
Questions this leaves open:
- Does hiding the score — returning only block or allow — change the picture, or only the price?
- Is there a threshold policy (randomized, per user, moving) that degrades more gracefully than a fixed τ?
- Detectors are increasingly stacked. Does an ensemble’s calibrated threshold fail the same way, or does each member cover another’s blind spot?
The next entry on the shelf, Budgets are threat models, takes up the price question by comparing defenses at one shared query budget.