ROSTER
The 22 arms
12 general decoders, 5 purpose-built safety classifiers, 3 trained encoder classifiers, 2 MLM negative controls. Repository, pinned revision, parameter count, licence, origin and readout family come from the harness's arm registry; on-disk snapshot sizes come from the weight download manifests. 20 of the 22 are candidates and 2 are negative controls. 3 are scored, 11 running and 8 queued, over 85.5 GiB of downloaded weights.
| Repository | Pinned revision | Class | Parameters | Licence | Origin | Gating group | Readout | Status |
|---|---|---|---|---|---|---|---|---|
answerdotai/ModernBERT-large | 45bb4654a4d5 | MLM negative control | 395,881,664 | apache-2.0 | USA/France (Answer.AI/LightOn) | ungated | mlm_control | scored |
answerdotai/ModernBERT-base | 8949b909ec90 | MLM negative control | 149,655,232 | apache-2.0 | USA/France (Answer.AI/LightOn) | ungated | mlm_control | scored |
meta-llama/Llama-Prompt-Guard-2-86M | a8ded8e697ce | Trained encoder classifier | 278,810,882 | other | USA (Meta) | promptguard2 | seqcls | queued |
protectai/deberta-v3-base-prompt-injection-v2 | 90c9989b1a34 | Trained encoder classifier | 184,423,682 | apache-2.0 | USA (ProtectAI) | ungated | seqcls | scored |
meta-llama/Llama-Prompt-Guard-2-22M | 11614a155199 | Trained encoder classifier | 70,830,722 | other | USA (Meta) | promptguard2 | seqcls | queued |
google/gemma-3-4b-it | 093f9f388b31 | General decoder | 4,300,079,472 | gemma | USA (Google) | gemma | letter3 | queued |
microsoft/Phi-4-mini-instruct | cfbefacb9925 | General decoder | 3,836,021,760 | mit | USA (Microsoft) | ungated | letter3 | running |
ibm-granite/granite-4.0-micro | 56111ae135df | General decoder | 3,402,836,480 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
tiiuae/Falcon3-3B-Instruct | 411bb94318f9 | General decoder | 3,227,655,168 | other (Falcon LLM licence) | UAE (TII) | ungated | letter3 | running |
meta-llama/Llama-3.2-3B-Instruct | 0cb88a4f764b | General decoder | 3,212,749,824 | llama3.2 | USA (Meta) | llama3.2 | letter3 | queued |
HuggingFaceTB/SmolLM3-3B | a07cc9a04f16 | General decoder | 3,075,098,624 | apache-2.0 | France/USA (HuggingFace) | ungated | letter3 | running |
HuggingFaceTB/SmolLM2-1.7B-Instruct | 31b70e2e869a | General decoder | 1,711,376,384 | apache-2.0 | France/USA (HuggingFace) | ungated | letter3 | running |
tiiuae/Falcon3-1B-Instruct | 28ba2251970a | General decoder | 1,669,408,768 | other (Falcon LLM licence) | UAE (TII) | ungated | letter3 | running |
ibm-granite/granite-4.0-1b | 6a7381ba1f54 | General decoder | 1,631,750,144 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
allenai/OLMo-2-0425-1B-Instruct | 48d788eca847 | General decoder | 1,484,916,736 | apache-2.0 | USA (Ai2) | ungated | letter3 | running |
meta-llama/Llama-3.2-1B-Instruct | 9213176726f5 | General decoder | 1,235,814,400 | llama3.2 | USA (Meta) | llama3.2 | letter3 | queued |
google/gemma-3-1b-it | dcc83ea841ab | General decoder | 999,885,952 | gemma | USA (Google) | gemma | letter3 | queued |
mistralai/Shieldstral-1.0-3B | 003ec7e2b0ba | Purpose-built safety classifier | 3,849,090,048 | apache-2.0 | France (Mistral) | ungated | shieldstral | running |
ibm-granite/granite-guardian-3.2-3b-a800m | 3de033d89b49 | Purpose-built safety classifier | 3,298,793,472 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
google/shieldgemma-2b | d1dffc9c8c92 | Purpose-built safety classifier | 2,614,341,888 | gemma | USA (Google) | gemma | shieldgemma | queued |
ibm-granite/granite-guardian-3.1-2b | 81145486e85c | Purpose-built safety classifier | 2,533,531,648 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
meta-llama/Llama-Guard-3-1B | acf7aafa60f0 | Purpose-built safety classifier | 1,498,482,688 | llama3.2 | USA (Meta) | llama3.2 | llamaguard | queued |
Parameter count across the 22 arms
8 of 22 need a licence-accepted token to fetch, across 3 separate acceptance groups. The snapshot column is the size the download wrote to disk.
Table view (every plotted value)
| Arm | Class | Parameters | Snapshot bytes | Licence | Origin | Gating | Readout | Status |
|---|---|---|---|---|---|---|---|---|
| gemma-3-4b-it | General decoder | 4,300,079,472 | 8,639,634,218 | gemma | USA (Google) | gemma | letter3 | queued |
| shieldstral-1.0-3b | Purpose-built safety classifier | 3,849,090,048 | 7,731,632,731 | apache-2.0 | France (Mistral) | ungated | shieldstral | running |
| phi-4-mini-instruct | General decoder | 3,836,021,760 | 7,694,059,344 | mit | USA (Microsoft) | ungated | letter3 | running |
| granite-4.0-micro | General decoder | 3,402,836,480 | 6,815,498,811 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
| granite-guardian-3.2-3b-a800m | Purpose-built safety classifier | 3,298,793,472 | 6,602,467,360 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
| falcon3-3b-instruct | General decoder | 3,227,655,168 | 6,465,509,473 | other | UAE (TII) | ungated | letter3 | running |
| llama-3.2-3b-instruct | General decoder | 3,212,749,824 | 6,434,752,520 | llama3.2 | USA (Meta) | llama3.2 | letter3 | queued |
| smollm3-3b | General decoder | 3,075,098,624 | 6,167,868,975 | apache-2.0 | France/USA (HuggingFace) | ungated | letter3 | running |
| shieldgemma-2b | Purpose-built safety classifier | 2,614,341,888 | 5,250,578,613 | gemma | USA (Google) | gemma | shieldgemma | queued |
| granite-guardian-3.1-2b | Purpose-built safety classifier | 2,533,531,648 | 5,071,957,207 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
| smollm2-1.7b-instruct | General decoder | 1,711,376,384 | 3,426,390,374 | apache-2.0 | France/USA (HuggingFace) | ungated | letter3 | running |
| falcon3-1b-instruct | General decoder | 1,669,408,768 | 3,348,994,667 | other | UAE (TII) | ungated | letter3 | running |
| granite-4.0-1b | General decoder | 1,631,750,144 | 3,273,295,937 | apache-2.0 | USA (IBM) | ungated | letter3 | running |
| llama-guard-3-1b | Purpose-built safety classifier | 1,498,482,688 | 3,006,170,568 | llama3.2 | USA (Meta) | llama3.2 | llamaguard | queued |
| olmo-2-1b-instruct | General decoder | 1,484,916,736 | 2,979,535,483 | apache-2.0 | USA (Ai2) | ungated | letter3 | running |
| llama-3.2-1b-instruct | General decoder | 1,235,814,400 | 2,480,847,368 | llama3.2 | USA (Meta) | llama3.2 | letter3 | queued |
| gemma-3-1b-it | General decoder | 999,885,952 | 2,039,072,537 | gemma | USA (Google) | gemma | letter3 | queued |
| control-modernbert-large | MLM negative control | 395,881,664 | 1,585,715,090 | apache-2.0 | USA/France (Answer.AI/LightOn) | ungated | mlm_control | scored |
| prompt-guard-2-86m | Trained encoder classifier | 278,810,882 | 1,131,677,308 | other | USA (Meta) | promptguard2 | seqcls | queued |
| deberta-v3-prompt-injection-v2 | Trained encoder classifier | 184,423,682 | 748,866,413 | apache-2.0 | USA (ProtectAI) | ungated | seqcls | scored |
| control-modernbert-base | MLM negative control | 149,655,232 | 600,805,271 | apache-2.0 | USA/France (Answer.AI/LightOn) | ungated | mlm_control | scored |
| prompt-guard-2-22m | Trained encoder classifier | 70,830,722 | 292,053,522 | other | USA (Meta) | promptguard2 | seqcls | queued |
harness/arms.py and harness/weights_manifest{,2,3,4}.json. The table view lists them all.Gating
8 of 22 arms need a licence-accepted HuggingFace token,
across 3 separate acceptance groups: the Gemma family, Llama 3.2, and a
third group covering Llama Prompt Guard 2 at licence other. All three were
accepted on 2026-09-23. A 403 is the signal that the token is valid and that repo's
group is unaccepted; a 401 means the token itself is not. 14 arms fetch
with no acceptance.
Anyone repeating the cohort has to accept three separate licences before 8 of the arms will fetch. Community re-uploads of the gated weights were refused: a mirror launders the provenance the licence column records.
Arms per HuggingFace acceptance group
A 403 rather than a 401 is the signal that the token is valid and that repo's group is unaccepted. Community re-uploads of the gated weights were refused, because a mirror launders the provenance the licence column exists to record.
Table view (every plotted value)
| Group | Arms | Acceptance | Members |
|---|---|---|---|
| ungated | 14 | no acceptance needed | granite-guardian-3.1-2b, granite-guardian-3.2-3b-a800m, granite-4.0-1b, granite-4.0-micro, phi-4-mini-instruct, smollm2-1.7b-instruct, smollm3-3b, olmo-2-1b-instruct, falcon3-1b-instruct, falcon3-3b-instruct, shieldstral-1.0-3b, deberta-v3-prompt-injection-v2, control-modernbert-base, control-modernbert-large |
| gemma | 3 | Google Gemma terms | shieldgemma-2b, gemma-3-4b-it, gemma-3-1b-it |
| llama3.2 | 3 | Meta Llama 3.2 community licence | llama-guard-3-1b, llama-3.2-3b-instruct, llama-3.2-1b-instruct |
| prompt guard 2 (licence other) | 2 | a third acceptance group covering Llama Prompt Guard 2 at licence `other` | prompt-guard-2-86m, prompt-guard-2-22m |
pinned/roster.json. The table view lists them all.Encoder backbones
The encoder arm of the cohort is 3 trained classifiers against
2 untrained MLM controls. Both Prompt Guard 2 sizes are
DebertaV2ForSequenceClassification with default LABEL_0 / LABEL_1 heads, so all
three trained encoders share the DeBERTa-v2 backbone family. That is three checkpoints inside
one family.
An earlier framing held that Prompt Guard 2 supplied an encoder backbone independent of DeBERTa. That was wrong and is withdrawn. The only independent encoder backbone in the cohort is ModernBERT, and both ModernBERT arms are controls. A trained-encoder result from this cohort can be shown to be non-checkpoint-specific inside the DeBERTa-v2 family; it cannot be separated from a DeBERTa-family result.
The evidence is in the download record. The weight manifest for both Prompt Guard 2 sizes
records architectures as DebertaV2ForSequenceClassification and
max_position_embeddings as 512, which is the same backbone family
and the same window as the DeBERTa candidate.
Prompt Guard 2's repo names understate its size. The headline 22M is 70,830,722 parameters and the headline 86M is 278,810,882, because the published figures exclude embeddings.
Backbone family across the five encoder arms
Prompt Guard 2 is DebertaV2ForSequenceClassification at both sizes, so all three trained encoders share the DeBERTa-v2 backbone family. That is three checkpoints inside one family. The only independent encoder backbone in the cohort is ModernBERT, and both ModernBERT arms are controls.
Table view (every plotted value)
| Arm | Backbone | Class | Parameters | How the backbone is known | Status |
|---|---|---|---|---|---|
| deberta-v3-prompt-injection-v2 | DebertaV2 | Trained encoder classifier | 184,423,682 | the repo is a DeBERTa-v3 base checkpoint with a trained 2-class head; the download that recorded it predates the architecture field in the manifest | scored |
| control-modernbert-base | ModernBERT | MLM negative control | 149,655,232 | ModernBertForMaskedLM, recorded in the scorecard's readout as mlm_control | scored |
| control-modernbert-large | ModernBERT | MLM negative control | 395,881,664 | ModernBertForMaskedLM, recorded in the scorecard's readout as mlm_control | scored |
| prompt-guard-2-86m | DebertaV2 | Trained encoder classifier | 278,810,882 | DebertaV2ForSequenceClassification | queued |
| prompt-guard-2-22m | DebertaV2 | Trained encoder classifier | 70,830,722 | DebertaV2ForSequenceClassification | queued |
pinned/roster.json. The table view lists them all.Taxonomy coverage in the safety classifiers
Three of the 5 purpose-built safety classifiers carry a policy surface. One has a mismatch with this task.
| Arm | Policy mechanism | Covers destructive tool calls |
|---|---|---|
meta-llama/Llama-Guard-3-1B | the shipped template hardcodes 13 categories, S1 to S13 | no; llamaguard_default_taxonomy_covers_task: false is recorded in the run metadata |
google/shieldgemma-2b | the chat template takes a guideline argument | yes, once the I3 policy is passed as the guideline |
mistralai/Shieldstral-1.0-3B | policy argument | yes, once the I3 policy is passed as the policy |
Llama Guard 3 1B's default categories
- S1 Violent Crimes
- S2 Non-Violent Crimes
- S3 Sex Crimes
- S4 Child Exploitation
- S5 Defamation
- S6 Specialized Advice
- S7 Privacy
- S8 Intellectual Property
- S9 Indiscriminate Weapons
- S10 Hate
- S11 Self-Harm
- S12 Sexual Content
- S13 Elections
The 8B model in the same family carries an S14 Code Interpreter Abuse category. The 1B's default list does not. S2 Non-Violent Crimes is the nearest fit and mapping onto it would have been a manufactured mapping, so a single custom category built from the I3 policy plus the Q2 block criterion, carried through the shipped template's documented `categories` hook and recorded verbatim in the run metadata. Readout is P(unsafe) against P(safe) over first-token ids [20451, 39257] against [6220, 19193].
ShieldGemma's chat template takes a guideline argument, so it is policy-adaptable
and closer to Shieldstral on this axis than expected. The I3 policy and the Q2 block criterion
were passed as the guideline and recorded verbatim, reading P(Yes) against P(No). Shieldstral
takes a policy argument and was treated the same way. Llama Guard 3 1B is the only arm in the
cohort with the taxonomy mismatch.
Qwen3
Qwen3 at 0.6B, 1.7B and 4B was dropped on a non-China provenance constraint. The download record for those 3 repositories totals 13,659,595,858 bytes, 12.72 GiB, and no GPU time was spent. They were the strongest ungated general models at their sizes.
With Gemma 3 and Llama 3.2 both gated, the ungated non-China general field in this cohort contains nothing that is simultaneously best-in-class for its size and permissively licensed.
Licence column
Both Falcon3 arms are licence other, the Falcon LLM licence, and neither is
Apache-2.0. Prompt Guard 2 is licence other at both sizes. The table records each
arm's licence as its publisher declares it.
Source: benchmarks/EXPERIMENTS.md on branch
feat/system-one-benchmarks, staged into
pinned/roster.json and cross-checked by
conform_to_space_contract.py. See Reproduce.