RESULTS

Scored arms

3 arms have a scored run on s2: one candidate and 2 negative controls. The leakage gate ran before the candidate was reported, and the scoring pipeline was checked against published System One board scorecards before either.

Precision, recall, F1 and block FPR

Every cell below is counted from rows over the 3,817 scorable cases. Each arm appears at its shipped argmax decision and at its in-sample oracle threshold.

Block false-positive rate at each arm's shipped operating point

Share of the 3,381 benign cases each arm blocks. For a guardrail this is the cost side of the decision, and it separates these arms more sharply than F1 does.

Cohort candidateMLM negative controlTrivial baseline, no model

Table view (every plotted value)

Arm	Block FPR	False blocks	Correct allows	Recall
deberta-v3-prompt-injection-v2	0.9467613132209406	3,201	180	0.9816513761467890
control-modernbert-large	0.6036675539781130	2,041	1,340	0.5940366972477065
control-modernbert-base	0.5740905057675244	1,941	1,440	0.7775229357798165
block every case	1.0	3,381	0	1.0

The candidate's block false-positive rate of 0.9467613132209406 means 3,201 of the 3,381 benign cases are blocked. Both controls sit lower, at 0.5740905057675244 and 0.6036675539781130, while blocking every case is 1.0 by construction.

F1 orders these arms differently from FPR and from AUC. Shipped F1 is 0.24963181148748159 for control-modernbert-base, 0.21057810578105782 for the candidate and 0.18932748538011696 for control-modernbert-large; length-controlled AUC is 0.5178408464714641, 0.82393827900586 and 0.49254286838283756 for the same three.

Scorer equivalence

The cohort harness imports the reference driver's own build_state and build_questions, so the prompts it sends are byte-exact and each row's context_sha256, context_bytes and truncated fields are byte-identical to the settled board rows. Before any cohort arm is reported, the pipeline re-derives 3 published System One board scorecards from their settled prediction bodies and matches the published figures.

The reference arms belong to the System One programme and their scores are published on the System One Space, not restated here. What the gate establishes for this Space is that the scoring code reproduces known answers before it is trusted on a new arm.

The arithmetic

The 3 cohort scoring scripts import their arithmetic verbatim from benchmarks/system_one/reproduce/07-analysis/rescoring/remine.py, 22,586 bytes at sha256 97df17a30891446d94a9df923ec343a6c7bd29adea3435b0303e4604440d5a6f. That module is in the repository and git-tracked, and the copy the scoring run imported hashes to the same digest, checked on 2026-09-23. The digest is re-hashed at build time. Request construction is pinned the same way, through benchmarks/scripts/benchmark_run_system_one.py.

The leakage gate

Both ModernBERT arms are bare ModernBertForMaskedLM backbones with no trained head, scored by a zero-shot masked-token yes/no readout. The prediction stated before measurement was that both should land at chance. The null band for 436 positives and 3,381 negatives is [0.471205496131191, 0.528794503868809].

Two gates are named apart here. The parity gate is inside score_cohort.py and is the scorer equivalence above. The leakage gate is this programme's name for the pre-registered prediction about the controls plus the surface-cue diagnostic that tests it; the scoring code carries no gate under that name.

Where the base control's excess comes from

The excess is the length cue. Four measurements.

• It scores below a pure counting variable with no semantic content: 0.6622467295653802 against 0.7692264380822134. Spearman against natural prompt length is 0.597867523607295.
• Inside prompt-length quintiles both controls fall to chance: 0.5178408464714641 and 0.49254286838283756.
• The two controls disagree at Pearson r = 0.1435084459826658 and sit on opposite sides of chance. A corpus-level label leak would be visible to both in the same direction.
• Both readouts are near-degenerate: 53 and 46 distinct values across 3,817 cases, each with a median of exactly 0.5.

How wide the two MLM controls' readouts are

Both controls are untrained ModernBertForMaskedLM backbones with no trained head, read by a zero-shot masked-token yes/no probe. Each has a median of exactly 0.5 and a few dozen distinct values across 3,817 cases.

5th to 95th percentile width, base5th to 95th percentile width, largeInterquartile width

Table view (every plotted value)

Control	Distinct values	Min	p05	Median	p95	Max	IQR width	Spearman against prompt length
control-modernbert-base	53	0.268941	0.407333	0.500	0.637031	0.718594	0.0933858743	0.5978675236
control-modernbert-large	46	0.233706	0.422505	0.500	0.577495	0.637031	0.0779475254	0.2035874960

deberta-v3-prompt-injection-v2

protectai/deberta-v3-base-prompt-injection-v2 at revision 90c9989b1a342275dd0d1a95aad283c04e075671, 184,423,682 parameters, licence apache-2.0. 30,310 rows, 0 errors, prediction sha256 927baa5ccc8bbf9b7885381ed50781264f848ab1c666e053166f9aab1721eeb4.

Calibration

The arm emits block on 27,460 of 30,310 rows and 3,629 of 3,817 cases, 95.1%. At argmax its recall is 0.9816513761467890, its precision 0.1179388261228989 and its block FPR 0.9467613132209406. Its oracle threshold sits at 0.9987551566979116 over 864 distinct achievable thresholds. At 186.7 rows/min CPU-only under int8 the next step is threshold calibration on held-out data.

deberta-v3-prompt-injection-v2 at two operating points

The arm emits block on 27,460 of 30,310 rows and 3,629 of 3,817 scored cases (95.1%). Its oracle threshold sits at 0.9987551566979116.

Argmax at 0.5, the shipped pointOracle thresholdBlock FPR at argmaxBlock FPR at the oracle threshold

Table view (every plotted value)

Operating point	Block-only F1	Precision	Recall	Block FPR	tp	fp	fn	tn
argmax at 0.5	0.210578105781	0.1179388261	0.9816513761	0.9467613132	428	3,201	8	180
oracle threshold 0.9987551566979116	0.480000000000	0.6020761246	0.3990825688	0.0340136054	174	115	262	3,266

The 512-token window

The cap is min(token_budget, max_position_embeddings − 2), which is 510. 17,202 of 30,310 rows exceeded it (56.75%); within the scored set 16,975 of 28,018 rows (60.59%) and 1,996 of 3,817 cases (52.29%) carry at least one truncated event. 58 cases have every event truncated. A row counted shrunk had its event state cut repeatedly to 82% of its length until it fit, so it lost context the model never saw.

Truncation does not track the arm's errors. AUC inside the truncated stratum is 0.8017085164830626 against 0.7771006013326832 untruncated, and F1 at the oracle threshold is 0.4799 against 0.4806. Recall and FPR do differ across the strata, and positive prevalence differs 5.4× between them. The mechanism is consistent with build_ids: the shrink trims event history while preserving the decision-relevant current tool call.

The per-row truncated boolean is a different thing: a corpus-level request-build flag, identical at 2,086 rows in all scored arms including the 16k-context controls.

DeBERTa's 512-token window, and whether it tracks the errors

The cap is min(token_budget, max_position_embeddings - 2) = 512 - 2 = 510, from score_arm.py:183. A row counted shrunk lost context the model never saw. AUC inside the truncated stratum is 0.8017085165 against 0.7771006013 untruncated.

AUC within the stratumF1 at the oracle thresholdPositive prevalence in the stratum

Table view (every plotted value)

Extent of truncation	Count	Share
Rows over 510 tokens in the full pass	17,202 of 30,310	56.75%
Rows over 510 tokens inside the scored set	16,975 of 28,018	60.59%
Cases with at least one truncated event	1,996 of 3,817	52.29%
Cases with every event truncated	58	1.52%

What it is measured against

Against the trivial floor it clears 0.20503 by 0.005546363481504557. Against the cohort it has the highest length-controlled AUC of the 3 scored arms, and the highest block FPR of the three.

Comparison with the System One board's cascade guardrail arms is out of scope for this Space. Those arms are measured under a different protocol and are published on the System One Space.

Two-class arms

All 3 scored arms emit one scalar per event. Three consequences are recorded as facts about those arms.

• Definitions A and B coincide, because both reduce to the maximum over events for a single monotone scalar. Their AUC carries no definition label.
• P(block) + P(confirm) and P(block) − P(confirm) are inapplicable: with no confirm class they do not exist, so neither is computed and neither gets a zero substituted for it. The second is also the variable that inverted below chance out of sample.
• Any-intervention F1 equals block-only F1 by construction, and is reported as an identity.