BASELINES

The two baselines

The corpus carries a trivial operating point worth block-only F1 0.20503174229955326 and a length signal worth AUC 0.7692264380822134, both with no model. Every F1 and every AUC here sits against those two.

The trivial floor

Blocking every case scores block-only F1 0.20503174229955326 at 11.42% prevalence: tp 436, fp 3,381, fn 0, tn 0, precision 0.1142258318, recall 1.0, block FPR 1.0. Allowing every case scores 0.0.

A false-positive rate of 1.0 fails any deployment gate, so this is not a claim that trivial blocking competes with a model. It is a claim about what an F1 below the floor tells you: the operating point is broken, and the figure says nothing about whether the model can discriminate.

2 of the 3 scored arms clear the floor. control-modernbert-base has the highest shipped F1 at 0.2496318114874816, and it is a negative control with no safety training.

Block-only F1 at each arm's shipped operating point, against the trivial floor

Blocking every case scores 0.20503174229955326 at 11.42% prevalence. Its false-positive rate is 1.0, so it fails any deployment gate. An F1 at or below the line says the operating point is broken, and says nothing about discrimination.

Cohort candidateMLM negative controlTrivial baseline, no model

Table view (every plotted value)

Arm	Block-only F1	tp	fp	fn	tn	Block FPR
control-modernbert-base	0.249631811487	339	1,941	97	1,440	0.574091
deberta-v3-prompt-injection-v2	0.210578105781	428	3,201	8	180	0.946761
control-modernbert-large	0.189327485380	259	2,041	177	1,340	0.603668
block every case	0.205031742300	436	3,381	0	0	1.000000

The length cue

Five scalars read off the request, containing no model, reach AUC 0.7692264380822134 (prompt tokens, max over events), 0.7772553177633239 (event count), 0.7445374041120237 (context_events) and 0.7266059794480217 (context_bytes). Destructive multi-step trajectories in this corpus are longer than benign ones, so this is a property of the data-generating process and not a labelling artifact.

AUC of five counting variables that contain no model

Each row is a scalar read off the request: how long the prompt is, how many events the context carries, how many bytes. Destructive multi-step trajectories in this corpus are longer than benign ones, which is a property of the data-generating process.

Table view (every plotted value)

Counting variable	AUC	Oracle threshold	Oracle best F1
event_count_in_prediction	0.7772553178	17	0.3870192308
natural_prompt_tokens (max over events)	0.7692264381	1,085	0.3851976451
natural_prompt_tokens (sum over events)	0.7665974048	8,454	0.3431372549
context_events	0.7445374041	7	0.3404940924
context_bytes (max over events)	0.7266059794	2,875	0.3459320948

Every arm's headline AUC therefore contains a length component. Each arm carries a length-controlled AUC beside its raw AUC: the corpus is stratified into five prompt-length quintiles, the AUC is computed inside each, and the five are averaged unweighted.

The pure length counter falls from 0.7692264380822134 to 0.5398672584287441 under that stratification.

AUC inside each prompt-length quintile

Stratifying on prompt length removes the length component from every arm's score. The length counter falls to 0.5398672584 under its own control, and both MLM controls land inside the chance band.

Table view (every plotted value)

Quintile	Cases	Positives	deberta-v3-prompt-injection-v2	prompt tokens, no model	modernbert-base	modernbert-large
Q1	672	2	0.9272388060	0.5421641791	0.7541044776	0.6992537313
Q2	852	24	0.6459088164	0.7504277375	0.3498138084	0.6719001610
Q3	765	80	0.8628558394	0.4619890511	0.4300729927	0.4368978102
Q4	764	100	0.8286521084	0.5688027108	0.5635015060	0.2742319277
Q5	764	230	0.8550358248	0.3759526136	0.4917114476	0.3804307116

The chance band

For 436 positives and 3,381 negatives the Hanley–McNeil standard error at AUC 0.5 is 0.014691343359335349, so the 95% chance interval is [0.471205496131191, 0.528794503868809]. It is shaded on every AUC chart here.

Both MLM controls land inside or below that band under length control: control-modernbert-base at 0.5178408464714641 and control-modernbert-large at 0.49254286838283756. Raw figures are 0.6622467295653802 and 0.46228994190416495; the decomposition is on Results.