OVERVIEW

Small and local models on destructive tool-call classification

22 arms are scored on one question: given an agent's trajectory so far and the tool call it is about to make, does that call have to be blocked. The corpus is s2 at 3,817 scorable cases, 436 of them positive. 3 arms have a score so far.

Scope

This Space carries small and local models measured on destructive tool-call classification. Cascade guardrail models on the DefenseClaw parity grid, and the Jev-family models, are published on the System One Space and appear nowhere here.

A model can be measured under both protocols. The numbers across the two Spaces are not interchangeable. The corpora differ and the label sets differ, so a figure from one set beside a figure from the other is a comparison of two measurements.

Measurements in hand

3 of 22

arms with a scored run

1 candidate and 2 negative controls

13 of 22

arms with a measured rows/min on CPU

14 arms were converted or quantized

0.82393827900586

highest length-controlled AUC

deberta-v3-prompt-injection-v2

0.9467613132209406

that arm's block false-positive rate

3,201 of 3,381 benign cases blocked

Complete

The corpus and its grade split.
The two baselines: the trivial floor and the length cue.
The leakage gate over both MLM controls.
Scorer equivalence against published System One board scorecards.
Precision, recall, F1 and block FPR for every scored arm, counted from rows.
The first scored candidate including its 512-token window and its calibration.
Laptop footprint for 13 arms.
The roster, with licence, origin and gating group for all 22 arms.

Awaiting the sweep

Accuracy for the 12 general decoders and the 5 purpose-built safety classifiers. 11 arms are running and 8 are queued.
Accuracy for Llama-Prompt-Guard-2-86M and Llama-Prompt-Guard-2-22M.
A held-out counterpart for any cohort arm. No cohort figure here is a transfer result.
Threshold calibration on held-out data for deberta-v3-prompt-injection-v2.
A rows/min figure for the third quantized encoder.

The scored arms

Precision, recall, F1 and block FPR at each arm's shipped operating point

All four are counted from rows at the arm's argmax decision over 3,817 scorable cases.

RecallPrecisionBlock-only F1Block FPR

Table view (every plotted value)

Arm	Precision	Recall	Block-only F1	Block FPR
control-modernbert-base	0.1486842105	0.7775229358	0.249631811487	0.5740905058
deberta-v3-prompt-injection-v2	0.1179388261	0.9816513761	0.210578105781	0.9467613132
control-modernbert-large	0.1126086957	0.5940366972	0.189327485380	0.6036675540

The dashed line is the F1 of blocking every case. Every value plotted here is read at build time from cohort-scoring/cohort-scores.json. The table view lists them all.

deberta-v3-prompt-injection-v2 blocks 3,201 of the 3,381 benign cases, a block false-positive rate of 0.9467613132209406, at recall 0.9816513761467890 and precision 0.1179388261228989. For a guardrail that false-positive rate is the governing figure.

On shipped F1 it scores 0.21057810578105782, and control-modernbert-base scores 0.24963181148748159. The untrained control ranks above the trained candidate on that measure. 2 of the 3 scored arms clear the trivial floor of 0.20503.

The ordering reverses on discrimination. Length-controlled AUC is 0.82393827900586 for the candidate and 0.5178408464714641 for that control, which is inside the chance band. Shipped F1 here is a statement about where a threshold sits.

Raw AUC and length-controlled AUC, for every arm with a score

Length-controlled AUC is the unweighted mean of the five within-quintile AUCs. The grey band is the 95% chance interval for 436 positives and 3,381 negatives under Hanley and McNeil, [0.471205496131191, 0.528794503868809].

Raw AUC over all 3,817 scorable casesLength-controlled, mean within prompt-length quintile

Table view (every plotted value)

Arm or variable	Raw AUC	Length-controlled AUC
deberta-v3-prompt-injection-v2	0.8390509973	0.8239382790
prompt tokens, no model	0.7692264381	0.5398672584
control-modernbert-base	0.6622467296	0.5178408465
control-modernbert-large	0.4622899419	0.4925428684

Definitions A and B coincide for every row here: each emits one monotone scalar, so both reduce to the maximum over events. Every value plotted here is read at build time from cohort-scoring/final-comparisons.json, leakage-diagnostic.json. The table view lists them all.

Disclosures

Gating

8 of 22 arms need a licence-accepted HuggingFace token, across 3 separate acceptance groups. Detail.

Encoder backbones

All 3 trained encoder classifiers are DebertaV2ForSequenceClassification: three checkpoints inside one backbone family. The only independent encoder backbone in the cohort is ModernBERT, and both ModernBERT arms are negative controls. Detail.

Llama Guard 3 1B's taxonomy

Its shipped template hardcodes 13 categories and none covers code execution or system damage. A custom category built from the I3 policy was used through the template's documented hook, with llamaguard_default_taxonomy_covers_task: false in the run metadata. Detail.

Qwen3

Qwen3 at 0.6B, 1.7B and 4B was dropped on a non-China provenance constraint, after 12.72 GiB of weights had been downloaded and before any GPU time was spent. With Gemma 3 and Llama 3.2 gated, the ungated non-China general field holds nothing that is both best-in-class for its size and permissively licensed. Detail.

Run integrity

The cohort runner writes neither complete nor prediction_sha256. A settlement step retrofits both and re-verifies, with two residual limits. Detail.

Aggregates only

No per-case detail is published. Every figure is an aggregate over the corpus or a stratum of it. Detail.

Corpus s2, cell C7/I3/Q2, --instruction-format structured, cases_sha256 39f2c1df2369952a0525cc4c5575f4bdb590fb3ca8c1bc6805cf4f376c1adbf7. Scripts and artifacts are listed on Reproduce.