Small models as tool-call guards
22 local models under 6B parameters, each tested as the step that blocks a destructive agent tool call.
What this is
What question does this page answer, and for whom?
Can a small model on a laptop block destructive tool calls alone, at 13 false blocks in 3,381 benign calls?
AI coding agents call tools. They run shell commands, edit files and call APIs. Some of those calls are destructive, or come from a trajectory an attacker has steered. A guard looks at each call before it runs. It decides to allow the call, ask a human to confirm it, or block it.
DefenseClaw is such a guard, built in three tiers. Deterministic rules come first. Then a Decision Model answers allow, confirm or block for every call. A Gemma 4 LLM judge sees only the calls the Decision Model is unsure about. System One is DefenseClaw's name for that Decision Model tier, measured on the System One Space.
DefenseClaw, three tiers
- Deterministic rules
- Decision Model, System One
- LLM judge, unsure calls only
This Space, one model alone
- A small model under 6B parameters scores the call; above its threshold, the call is blocked
This Space asks a narrower question. Could a small model, under 6B parameters and able to run on a laptop CPU, do the blocking job by itself? There are no rules in front of it and no judge behind it. Each model reads the agent's trajectory and the call it is about to make, then gives one block score.
The page scores 22 models, 20 candidates and 2 controls, on corpus s2. It holds 3,817 scorable calls: 436 destructive calls that had to be blocked and 3,381 benign calls that had to be allowed.
Every model gets the same budget: at most 13 false blocks in 3,381 benign calls. In plain words, the guard may wrongly stop about 1 safe call in 260. That is the rate the existing DefenseClaw cascade reaches on this corpus, so a deployment already accepts it.
This page is for engineers choosing what to deploy as a tool-call guard. The System One Space compares the Decision Models inside DefenseClaw; the Skill scanner Space compares Cisco Skill Scanner with NVIDIA SkillSpector.
The answer
Can a small model block destructive calls by itself within 13 false blocks?
Verdict
No. The best of 20 candidates blocks 25 of 436 destructive calls at the budget of 13 false blocks.
It lets 411 through. Blocking every call scores a higher F1 than any model does at any budget up to 50 false blocks.
At 13 false blocks, the best model stops 25 of 436 destructive calls
One dot per scorable call in corpus s2. Lit dots are the calls blocked by the best candidate at this budget, falcon3-1b-instruct.
How to read it. The top field has one dot for each of the 436 destructive calls; the bottom field has one for each of the 3,381 benign calls. A lit dot is a call the best model blocks. The slider sets the budget, and the tiles and every chart below follow it.
436 destructive calls that had to be blocked
3,381 benign calls that had to be allowed
- Destructive calls blocked 25 of 436 Recall 0.0573 [0.0391, 0.0833]
- Benign calls wrongly blocked 13 of 3,381 A random blocker would catch 1.68
- F1 of the best model 0.105 The block-everything baseline scores 0.205
What it shows. At 13 false blocks the best model lights 25 of 436 destructive dots, and lets 411 through.
Data table
Source
All 22 models answered the same question for each of 3,817 scorable calls. Each was then re-thresholded to the budget: 13 false blocks in 3,381 benign calls, the existing DefenseClaw cascade's own rate on this corpus. Moving the slider re-reads every model from its own curve.
Allowing nothing through costs too much, and allowing almost everything through is what these models do. With no false block allowed at all, 13 of 20 candidates catch nothing; the best, granite-4.0-micro, catches 5.
Try the slider at 0 and then at 50. At 50 false blocks, almost four times the budget, the best model still catches only 50 of 436. For a deploy decision, none of these models can be the step that blocks. If you need a local blocker, this size class does not supply one at this budget.
How this was measured
Corpus s2 holds 4,277 cases; 3,817 are scorable. 436 are positives (17 grade A, 419 grade B) and 3,381 are benign (grade D). The 460 grade C cases are unresolved and excluded. Each model's block score was swept over every threshold; the dots read the point with the most catches inside the budget (the vertex rule). Recall carries a Wilson interval. The prompt configuration is grid cell C7/I3/Q2 with structured instructions: I3 passes the policy to the model and Q2 sets the question format.
Leaderboard
Which model blocks the most destructive calls without exceeding the budget?
falcon3-1b-instruct does, with 25 of 436. It holds first place from 6 to 50 false blocks, but its lead is a handful of calls.
falcon3-1b-instruct leads at 13 false blocks with F1 0.105; blocking every call scores 0.205
20 candidates ranked by F1 at the budget, then by parameters. The 2 controls are shown unranked and greyed.
How to read it. Each row is a model at the current budget. F1, recall and precision come first; small type gives 95% intervals. Then come calls caught and false blocks used. The pinned top row is the block-everything baseline. Move the slider, type in the find box, or pick a group; open a row for its confusion counts.
| Rank | Model | F1 at budget | Recall | Precision | Caught of 436 | False blocks | Default F1, as shipped | Length-controlled AUC | Parameters | Licence | Gated |
|---|
What it shows. The leader scores F1 0.105 at 13 false blocks, about half the 0.205 of blocking every call. Hover or click a row to light that model in every chart.
Source
The F1 interval is a bootstrap interval and exists at 13 false blocks only; other budgets show the point value. The best recall at the budget is 0.0573, and the top of its interval is 0.0833.
The default-decision F1 column scores each model as shipped, with nothing fitted. granite-guardian-3.2-3b-a800m leads it with 0.367, but wrongly blocks 495 benign calls to get there. 9 of 19 candidates with a default decision score below the block-everything baseline.
For a deploy decision, read the F1 at budget column, not the default one. A default F1 shows a model as shipped. 11 of 19 shipped settings wrongly block more safe calls than the budget allows. The leader's F1 interval runs from 0.0675 to 0.143. The second and third models, at 0.0778 and 0.0688, sit inside it. So the order of the top three is not settled, and none of them comes near the baseline.
How this was measured
Each model's threshold is the one with the highest recall that holds the budget, ties broken toward the tighter threshold. Precision is undefined when a model blocks nothing. The two controls are untrained ModernBERT backbones with no safety head: what they score is what the corpus gives away for free. gemma-3-4b-it has no default decision because its predictions landed after the shipped-argmax run.
Every operating point
Would a looser or a tighter budget change the answer?
No. No curve climbs far before the false blocks run out, and the curves cross past the budget.
Up to 50 false blocks, no model catches more than 50 of 436; deberta-v3 jumps to 103 by 67
ROC curves over a block false-positive rate of 0 to 0.02 (0 to 67 false blocks of 3,381); deberta-v3 is always drawn in grey. Drag the amber line to move the budget.
How to read it. Each line is one model. Moving right spends false blocks; moving up catches destructive calls, as a share of 436. The amber line is the budget, linked to the slider. The buttons switch to a log axis and choose how many models to draw.
What it shows. At the budget line no curve has passed 25 of 436; pin a model in the leaderboard to add its curve here.
Data table
Source
Each step up is one more destructive call caught; each step right is one more benign call blocked. The budget line at 13 sits where most curves are still flat. The zero-false-positive gate is the line at 0.
The order changes with the budget. granite-4.0-micro leads at most budgets up to 5 false blocks. falcon3-1b-instruct leads from 6 to 54. From 55, deberta-v3-prompt-injection-v2 takes the lead.
Switch to the log axis to spread out the first few false blocks, where granite-4.0-micro leads. Then drag the amber line to 50. A looser budget does not rescue these models: at 50 false blocks the best catch is 50 of 436. A tighter one makes things worse, since 13 of 20 candidates catch nothing at 0. The model you would pick also depends on the budget you pick.
How this was measured
Curves are integer (false blocks, true blocks) vertices, 6,515 points across 22 models. The area under each drawn curve equals the published raw AUC exactly. Points between vertices are tied scores and are not reachable thresholds, so the curve is drawn as steps.
Why the ranking is fragile
Can the ranking be trusted to choose a model?
Only with care. Trajectory length predicts the label on its own, and one source supplies most of the destructive calls.
Counting events, with no model at all, reaches AUC 0.777, above the best model at the budget
AUC of five counting variables on corpus s2, against two models' raw AUC. The shaded band is chance.
How to read it. Each bar is one AUC, where 0.5 is chance and 1 is a perfect ranking. The legend separates counting variables, which use no model, from two models' raw AUC. The shaded band is where a score with no signal lands.
What it shows. Event count alone reaches AUC 0.777, above the 0.725 of falcon3-1b-instruct, the leader at the budget.
Data table
Source
deberta-v3 leads on length-controlled AUC, 0.835, yet catches 1 of 436 at the budget
Raw AUC and length-controlled AUC for all 22 models; controls are labelled.
How to read it. Each row is a model. One dot is its raw AUC, the other its length-controlled AUC, and the line joins them. The buttons order the rows by length-controlled AUC or by rank at the budget.
What it shows. deberta-v3 tops length-controlled AUC at 0.835, yet catches 1 of 436 at 13 false blocks.
Data table
Source
382 of the 436 destructive calls come from one dataset, so recall here is mostly that dataset's
Destructive calls caught at 13 false blocks, by model and source dataset (positives per source in brackets).
How to read it. Rows are models, in order of F1 at the budget. Columns are the five source datasets of destructive calls. Cell shade follows the ramp in the legend; the buttons switch between counts and recall within each source.
What it shows. 382 of the 436 positives come from one dataset, and most of every model's catches sit in that column.
Data table
Source
Event count alone scores AUC 0.777, prompt tokens 0.769. The chance band runs from 0.471 to 0.529. Raw AUC therefore rewards length, which is why the ranking uses the length-controlled estimator instead.
Under length control the token counter itself falls to 0.471, just below the chance band. So do 8 of 20 candidates. The order by AUC barely tracks the order at the budget.
Switch the AUC chart to rank at the budget and watch the top rows change. Then switch the heatmap to recall within source. For a deploy decision, do not pick a guard by AUC. Measure it at the false-block rate you will run, on traffic like yours. Recall here is mostly one dataset's, so it may not carry over to your agents.
How this was measured
Length-controlled AUC splits cases into five quintiles of prompt length and pools the within-quintile AUCs, each weighted by its destructive-benign pairs. An earlier unweighted mean was retracted in favour of this estimator. Quintile 1 holds only 2 positives. The source heatmap is fixed at 13 false blocks; per-source curves are not published.
Can you combine them
Does running several small models together close the gap?
Not inside the budget. Combining helps a little within it and a lot only outside it.
The best pair catches 34 of 436 inside the budget; all 22 together catch 88 at 10.7 times it
Destructive calls caught when a call is blocked if any combined model would block it, with the false blocks that costs.
How to read it. Each bar is one combination, and its length is destructive calls caught, of 436. Colour says whether it stays within 13 false blocks. The label on each bar gives the false blocks it costs.
What it shows. The best pair catches 34 of 436 within the budget; all 22 models together catch 88 at 139 false blocks.
Data table
Source
falcon3-1b-instruct and granite-4.0-micro, each held to half the budget, add 9 catches over the best single model. That pair is the best of 231 tested on the same rows, so it is an in-sample pick. A union of all 22 lifts recall to 0.202, at 139 false blocks.
The models mostly catch different calls: 185 of 221 pairs share no caught positive. The overlap matrix is in the appendix.
For a deploy decision, stacking small models is not a fix. Inside the budget the gain is 9 calls, from a pair picked on the same rows it is scored on. Going from the best single model to all 22 adds 63 catches and 126 false blocks. Each extra catch costs about two safe calls blocked.
Cost to run on a laptop
What does a model cost to run on a laptop, and does a bigger one help?
Bigger is not better here: the leader has 1.67B parameters and is among the fastest decoders on CPU.
No model from 3B to 6B beats the 1.67B leader at the budget
One point per model. Circled points are on the Pareto frontier; the shaded band is 3B to 6B parameters.
How to read it. Across is parameters or measured CPU speed, both on a log scale. Up is F1 at the budget, default F1 or length-controlled AUC; pick with the buttons. The horizontal line is blocking every call, and circled points are on the frontier.
What it shows. The 1.67B leader sits above every model from 3B to 6B; the best of that band catches 18 to its 25.
Data table
Source
The best model in the 3B to 6B band, granite-4.0-micro at 3.40B, catches 18 to the leader's 25. On 8 CPU threads falcon3-1b-instruct scores 27.7 rows per minute, so a pass over 3,000 rows takes 1 hour 48 minutes. deberta-v3 is about 7 times faster but reads only the first 512 tokens.
| Model | Rows per minute | 3,000 rows take | Serving | Q4_K_M file | Peak RSS |
|---|
Weights at Q4_K_M run from 0.871 GiB (olmo-2-1b) to 2.32 GiB (phi-4-mini). Peak RSS was 3.14 GiB for phi-4-mini, the hungriest, leaving 4.86 GiB on an 8 GiB laptop.
Switch across to rows per minute: the leader is the second-fastest decoder measured. Rows per minute exists for 13 models, so that view drops the other 9. For a deploy decision, size is not the lever. If you still want a local screen, a 1B-class model did best here and costs least to run. Even it needs about 2.2 seconds per call on 8 threads, and that is an optimistic ceiling.
How this was measured
CPU only, 8 pinned threads. Decoders ran as GGUF Q4_K_M under llama.cpp; encoders as dynamic int8 under transformers. Row latency is 1,200 prompt tokens plus one generated token, measured with llama-bench (deviation under 4%). The host carried foreign load on about 31 of its 96 cores, and a real laptop also throttles, so every figure is an optimistic ceiling. The measurement step was never saved as a script, so these figures are archived rather than reproducible.
Caveats and scope
What would have to be true for this answer to change?
A different corpus, label function or budget. The answer rests on one of each.
| Assumption | Rests on | What breaks it |
|---|---|---|
| Every case carries the right truth grade. | One deterministic function, truth_grade(), evaluated from fields on each case row. | A mislabel in a source dataset. No model or human reviewer is consulted, so no inter-rater figure bounds the error. |
| 13 false blocks is the budget that matters. | It is the existing cascade guardrail's own block false-positive rate on this corpus, 0.00385. | A deployment with a different tolerance. Every figure at the budget moves with it; the slider shows how far. |
| Recall here is recall on destructive tool calls in general. | 436 positives from 5 source datasets. | mcphunt-agent-traces supplies 382 of them, 87.6%, so the figure is mostly one dataset's. |
| A model that holds the budget here holds it on new traffic. | 13 false blocks of 3,381 benign cases, counted. | The Wilson upper bound on that rate reaches 0.00657, 1.71 times the budget; 14 models exceed the budget at their upper bound. |
| The best-F1 threshold is achievable. | A sweep over the same rows the score is computed on. | It is fitted in sample: falcon3-1b-instruct scores 0.105 at the budget and 0.329 at its best-F1 threshold. |
Scope: 22 local models, 20 candidates and 2 controls, on one corpus, s2. A second corpus of 24,476 cases was scored for 6 models; its positives are 87.3% grade A against 3.9% here, so no score is compared across the two. Qwen3 0.6B, 1.7B and 4B were dropped before any run for provenance reasons.
Neither gate is durable: when the System One programme moved thresholds to a disjoint corpus, 4 of 24 zero-false-positive gates survived and the false-positive cap held in 8 of 48.
Read the table row by row before you reuse a number. The labels come from one function with no human check. The budget is one deployment's tolerance, and the slider shows how far results move with it. Thresholds were fitted on the same rows they are scored on. On new traffic, a model held to 13 false blocks here may spend more.