METHODOLOGY
How every number on this Space was produced
One corpus, one grid cell, one scorer. These rules govern which figures may be quoted and which may be compared.
Corpus
| Property | Value |
|---|---|
| Corpus | s2, the core parity grid |
| Cases | 4,277 |
| Scorable cases | 3,817 |
| Positives (grade A + grade B) | 436 = 17 + 419 |
| Benign (grade D) | 3,381 |
| Excluded, grade C, diagnostic only | 460 |
| Prevalence | 11.42% |
| Rows per prediction file | 30,310 |
| Grid cell | C7 / I3 / Q2,
--instruction-format structured |
cases_sha256 | 39f2c1df2369952a0525cc4c5575f4bdb590fb3ca8c1bc6805cf4f376c1adbf7 |
Grade C is excluded from scoring. The grade records an unresolved or partial adjudication, so a scored decision on one of those cases would have no settled truth value to score against. The 460 grade-C cases are kept for diagnostics and are not counted in any numerator or denominator here. Every F1, AUC, FPR and confusion matrix on this Space is over the 3,817 scorable cases.
Positive prevalence of 11.42% is what sets the trivial floor at 0.20503174229955326, so the floor is a property of this corpus rather than of any arm. See Baselines.
How the corpus splits
4,277 cases. 3,817 are scorable once the 460 grade-C cases are excluded: 436 positives (A 17 plus B 419) against 3,381 benign. Positive prevalence is 11.42%, which is what sets the trivial floor.
Table view (every plotted value)
| Grade | Cases | Role in scoring | What the grade records |
|---|---|---|---|
| A | 17 | scorable positive | confirmed destructive, highest severity; scorable positive |
| B | 419 | scorable positive | destructive; scorable positive |
| D | 3,381 | scorable negative | benign; scorable negative |
| C | 460 | excluded | diagnostic only; excluded from scoring because the grade records an unresolved or partial adjudication, so a scored decision on it would not have a settled truth value |
cohort-scoring/cohort-scores.json. The table view lists them all.Scorer equivalence
The cohort harness imports the reference driver's own build_state and
build_questions, so prompts are byte-exact. Before any cohort arm is reported the
pipeline re-derives 3 published System One board scorecards from their settled
bodies: 2 confusion matrices reproduce cell for cell, the largest
absolute block-only F1 difference is 1.30e-09, and the ranking AUC on the one
reference that publishes it reproduces with an absolute difference of 0.0.
The full table is on Results.
Those reference arms are out of scope for this Space and their scores are published on the
System One
Space. The arithmetic is one vendored module, benchmarks/system_one/reproduce/07-analysis/rescoring/remine.py at sha256
97df17a30891446d94a9df923ec343a6c7bd29adea3435b0303e4604440d5a6f, imported verbatim by all 3 cohort scoring
scripts and re-hashed at build time.
In-sample ceilings
A best F1 found by sweeping thresholds on the same rows it is scored on is an oracle upper bound. Every figure of that kind here is labelled, and no arm's oracle ceiling is quoted as a result. No cohort arm has a held-out counterpart yet, so no cohort figure here is a transfer result.
The ranking variables
P(block) − P(confirm) is never ranked on. Measured on the System One
programme it inverted below chance on a disjoint held-out corpus, and it was the argmax
variable for five of eleven re-mineable arms there, so the exclusion is a rule rather than a
case-by-case judgement. P(block), risk and
P(block) + P(confirm) all transferred. The figures behind that result belong to
the other programme and are published on the
System One
Space.
Two AUC definitions exist and are not interchangeable. Definition A takes the maximum block probability and the maximum confirm probability over a case's events, then subtracts. Definition B takes the maximum over events of the per-event difference. Every AUC on this Space is labelled with its definition, and no figure is compared across definitions. For an arm that emits a single monotone scalar the two coincide, because both reduce to the maximum over events; those AUCs carry no definition label and the coincidence is recorded on the arm.
Two-class arms have genuinely inapplicable variables. An arm with no confirm
class has no P(confirm), so P(block) + P(confirm) and
P(block) − P(confirm) are not computed and are not zero-filled. Its
any-intervention F1 equals its block-only F1 by construction and is reported as an identity.
Length control
Every arm carries a length-controlled AUC beside its raw AUC. The corpus is stratified into five prompt-length quintiles by natural prompt tokens, the AUC is computed inside each, and the five are averaged unweighted. The quintile boundaries, case counts and positive counts are in the table view of every chart that uses them, and the null band is on Baselines.
Gate transfer
Measured on the System One programme, 4 of 24 zero-false-positive gate transfers to a disjoint corpus survived and 3 of those were degenerate; 12 of 24 violated the source corpus's Wilson bound; FPR caps held in 8 of 48 transfers. Neither a zero-FP gate nor an FPR cap is presented here as durable. Cross-validation within the source corpus called one gate perfect at 5 of 5 folds and the disjoint corpus showed that threshold leaking 66 false positives, so cross-validation does not see corpus shift.
When a run counts
A run is valid only if its metadata says complete: true and the on-disk sha256 of
the prediction body equals its recorded prediction_sha256. Runs are not spliced, and
a multi-shard merge counts only when every row was produced under identical serving conditions.
The cohort runner harness/score_arm.py writes neither field and never hashes the
body. harness/settle.py retrofits both: it requires
30,310 rows with no torn final line, writes complete: true and the
digest, then re-reads the metadata and re-hashes the body and compares, so the pass is measured.
It exits non-zero if any settled arm's digest disagrees with its metadata.
Two limits remain, and they make this weaker than the reference driver's inline guarantee.
- The digest is taken at settlement time over the body's complete-line prefix, so it certifies unchanged-since-settlement and not as-the-runner-emitted-it.
- Settlement requires 30,310 untorn rows, so an unsettled metadata file means the run is incomplete rather than merely unchecked.
Of the 3 scored arms, 0 carry the settled fields in the metadata the scorer read and 3 do not. For an arm without them, the digest printed on Results is the scorer's, computed over the bytes on disk at scoring time; the conform script re-hashes the bodies and compares against it, which establishes that the bytes scored are the bytes on disk. Each scored arm covers all 3,817 scorable cases with 0 errors and 0 cases missing.
What is published here
Aggregates only. No per-case row detail appears on this Space. Every figure is a count, a rate or a distribution statistic over the corpus or a stratum of it. The payload guard indexes every corpus case id on disk and fails the build if one appears in a file.
mcptox is local-evaluation-only: aggregate numbers only, never rows. The
augur_unsafe_tool_input_eval source is aggregate-only on the same terms. Neither
contributes a row to this Space.
The four data repositories that hold the corpora and the prediction files stay private. This Space is public and carries no corpus rows, no provider rationales, no credentials and no unredacted identifiers.
Build gates
107 figures are recomputed from their primitives and checked against their artifacts before any file is written, over 12 artifacts and 16 charts. The payload guard is byte-identical to the System One Space's copy. The scripts are listed on Reproduce.
Ledger: benchmarks/EXPERIMENTS.md. Corpus:
cases_sha256 39f2c1df2369952a0525cc4c5575f4bdb590fb3ca8c1bc6805cf4f376c1adbf7.