REPRODUCE

Every script and artifact behind this Space

All links point at branch feat/system-one-benchmarks of cisco-ai-defense/defenseclaw. The corpora and the prediction files are held in private data repositories; the scoring artifacts this Space reads are vendored into the public repository at the paths below.

Ledger and corpus

• benchmarks/EXPERIMENTS.md — the experiment ledger. Every figure on this Space traces to an entry in it.
• benchmarks/datasets.lock.json — the source inventory the corpus was built from.
• benchmarks/schema/system-one-prediction-v1.schema.json — the prediction row schema every arm's output validates against.
• benchmarks/system_one/contexts-v1.json and questions-v2.json — the C7 context template and the Q2 question format.

Running an arm

• benchmarks/scripts/benchmark_prepare_system_one.py — corpus assembly.
• benchmarks/scripts/benchmark_run_system_one.py — the reference driver. The cohort harness imports its build_state and build_questions, which is what makes the prompts byte-exact.
• benchmarks/scripts/benchmark_score_system_one.py — the System One scorer the cohort pipeline is checked against.

The cohort harness

• benchmarks/slm_toolcall/README.md and harness/README.md — what ran where, the prediction and metadata schemas, and the two schema gaps that govern how a cohort run may be checked.
• harness/arms.py — the arm registry. Every repository, pinned revision, parameter count, licence, origin and readout family on the Roster page is read out of this file at build time.
• harness/score_arm.py — the runner. It imports request construction from the reference driver, which is why the request bytes match the board runs. Pinned to the single cell C7/I3/Q2.
• harness/weights_manifest.json, 2, 3, 4 — the download records. Which revision of which repository was fetched, its size on disk, and for the two Prompt Guard 2 repositories the architecture class and position limit that the backbone section rests on.
• harness/build_llamacpp.sh — the CPU toolchain. Nothing in the harness runs it, and the scripts that produced the CPU figures were never saved; see Footprint.
• harness/analyse.py and report.py — the harness-side scoring and text tables, including its own --validate-27b self-check.

Scoring the cohort

• benchmarks/slm_toolcall/scoring/score_cohort.py — per-arm scorecards behind the parity gate, the by-variable AUC sweep, the inapplicable-variable records and the truncation analysis. It will not report a cohort arm until it has re-derived three reference arms from their settled bodies and matched the published numbers. Produces cohort-scores.json.
• benchmarks/slm_toolcall/scoring/leakage.py — the surface-cue diagnostic: the counting variables, each control's Spearman correlation against prompt length, AUC within length quintiles, and the Pearson agreement between the two controls. Produces leakage-diagnostic.json.
• benchmarks/slm_toolcall/scoring/final.py — the trivial baselines, the null band, the length-controlled table and the headline comparisons. Produces final-comparisons.json.
• scoring/README.md — the parity gate the three scripts run behind, and the caveats that qualify every number they produce.

The three scripts carry the dev host's absolute paths as they ran. Their arithmetic is imported verbatim from the board's re-mining module, which is what makes cohort numbers and board numbers comparable; that module is reached through a path on the dev host rather than through this directory. They are published in the form that produced the artifacts.

The arithmetic both programmes run

• benchmarks/system_one/reproduce/07-analysis/rescoring/remine.py — the shared scoring arithmetic, 22,586 bytes at sha256 97df17a30891446d94a9df923ec343a6c7bd29adea3435b0303e4604440d5a6f. All 3 cohort scoring scripts import it verbatim, and this build re-hashes it and aborts on a mismatch. It is the load-bearing claim behind comparability.
• rescoring/README.md — the two AUC definitions and their formulas.

Artifacts this Space reads

• benchmarks/slm_toolcall/artifacts/cohort-scores.json
• benchmarks/slm_toolcall/artifacts/leakage-diagnostic.json
• benchmarks/slm_toolcall/artifacts/final-comparisons.json
• site-build/pinned/roster.json — the roster, transcribed from the ledger and cross-checked against each scored arm's run metadata.
• site-build/pinned/laptop-feasibility.json — the CPU throughput and memory primitives, with their provenance block.
• site-build/pinned/scorer-provenance.json — the pinned digest of the shared arithmetic and of the reference driver's role in request construction.

Building this Space

• site-build/build.py — the generator. It checks 107 figures against their artifacts, then renders 16 charts as inline SVG.
• site-build/conform_to_space_contract.py — stages the artifacts at the paths the build resolves and re-checks each scored arm's digest, row count, corpus and metadata against the roster.

Gates before an upload

• site-build/guard.py — the payload guard, byte-identical to the System One Space's copy at system_one/reproduce/08-site-build/guard.py. Five checks over the payload: corpus case ids, sensitive-text shingles, CJK codepoints, credential shapes, and file size and type. Allowed extensions are .html .css .md .json .txt and the per-file ceiling is 1 MiB.
• site-build/verify.py — structure, link resolution, SVG bounds, inline-stylesheet and no-subresource checks, plus this programme's disclosure and retired-figure lists.
• site-build/wc.py — the visible-word counter, byte-identical to the System One Space's copy.
• site-build/publish.py — five sequential gates: re-run the guard, run the verifier, read and record the repository's visibility, upload, then read the visibility again and abort if it changed in either direction. create_repo is reached only when the Space does not exist. No credential is ever read into a printable variable.

Repository visibility

This Space is public. The four data repositories that hold the corpora, the prediction files and the serving records stay private. Nothing in the publish path moves a repository between visibilities in either direction, and the post-upload check aborts if one moved.