Started Updated
Budgets are threat models
Re-run under one shared query budget, the defense with the best published robustness number ranks last of those compared.
Attack success on the best-reported defense at the shared budget
Background
Defense papers report robustness as an attack success rate: the fraction of attacks that get through. That number belongs to a particular attack, run with a particular budget — and the budget usually sits in the experimental details, where it varies a great deal. Among the 5 defenses compared here, the original evaluations used anywhere from 10 to 1,000 queries per attack.
A success rate without its budget is not a property of the defense. It is a property of the defense together with an attacker who stopped at some point. Comparing such numbers across papers compares attackers as much as defenses.
Threat model
The budget is part of the threat model, so it is fixed first and held equal across every defense.
- Access: queries to the defended system’s decision — block or allow — as in the original evaluations.
- Budget: the same number of queries per attack against every defense, swept from 10 to 3,000. The headline comparison is at 1,000.
- Attack: one implementation for all defenses, tuned on none of them.
- Success: the defended system carries out the injected instruction, judged by the same grader throughout.
Method
Each defense sits behind the same interface and is attacked with the same search, which spends its budget on rewrites of a fixed seed set. Success is recorded at every budget checkpoint, so a single run yields the whole curve rather than one point.
The original papers’ numbers are not re-derived. They are taken as reported, together with the budget each paper used, and plotted beside curves measured under one protocol.
Defenses are anonymized while the work is in progress. Each is named by mechanism, because the mechanism — not the product — is what the comparison is about.
Equal-budget comparison
Attack success, compared at one budget
Each defense at the budget its own paper used. Lowest attack success ranks first.
Every defense at 10 queries. Lowest attack success ranks first.
Every defense at 30 queries. Lowest attack success ranks first.
Every defense at 100 queries. Lowest attack success ranks first.
Every defense at 300 queries. Lowest attack success ranks first.
Every defense at 1,000 queries, the shared budget. Lowest attack success ranks first.
Every defense at 3,000 queries. Lowest attack success ranks first.
Attack success by budget, per defense
-
input classifier
-
perplexity filter
-
paraphrase preprocessing
-
output monitor
-
self-check prompt
- Measured, one attack for all
- As reported in the original paper
Data tables
| Defense | Mechanism | Queries in original evaluation | Attack success as reported |
|---|---|---|---|
| A | input classifier | 10 | 2% |
| B | perplexity filter | 50 | 6% |
| C | paraphrase preprocessing | 200 | 11% |
| D | output monitor | 1,000 | 27% |
| E | self-check prompt | 20 | 8% |
| Defense | Query budget | Attack success |
|---|---|---|
| A | 10 | 4% |
| A | 30 | 17% |
| A | 100 | 48% |
| A | 300 | 77% |
| A | 1,000 | 91% |
| A | 3,000 | 96% |
| B | 10 | 3% |
| B | 30 | 9% |
| B | 100 | 26% |
| B | 300 | 47% |
| B | 1,000 | 66% |
| B | 3,000 | 79% |
| C | 10 | 2% |
| C | 30 | 5% |
| C | 100 | 14% |
| C | 300 | 31% |
| C | 1,000 | 52% |
| C | 3,000 | 71% |
| D | 10 | 1% |
| D | 30 | 3% |
| D | 100 | 7% |
| D | 300 | 15% |
| D | 1,000 | 29% |
| D | 3,000 | 46% |
| E | 10 | 6% |
| E | 30 | 18% |
| E | 100 | 39% |
| E | 300 | 61% |
| E | 1,000 | 79% |
| E | 3,000 | 88% |
Results
As reported, defense A looks strongest: 2% attack success at the 10 queries its paper used. At the shared budget of 1,000 queries, the same attack gets through 91% of the time — last among the defenses compared.
Defense D, the weakest on paper, is the most robust at the shared budget, at 29%.
None of this says defense A is bad. It says that its number answered a narrower question: how it holds up against an attacker who stops after 10 queries.
Limitations
- One attack. A different search might order the defenses differently. The claim is about comparing at an equal budget, not that this ranking is final.
- Decision-only access was used to match the original papers. Score access would likely move every curve to smaller budgets.
- Reported points are not reproductions. Some original evaluations used different seed sets; their numbers are plotted as published.
- In progress. Some defenses are still being re-run with a larger seed set, and their curves may move.
Further thinking
If budgets are threat models, papers should report curves, not points — or at least the budget at which attack success crosses a fixed level.
The same attack also transferred to a closed-weight production deployment. That result is pending clearance while disclosure with the vendor is in progress, and will appear here once it is cleared.
Open questions:
- Is there a robustness score that does not depend on where the attacker stops?
- How should two defenses be ranked when their curves cross?