D17-MF: multi-fact reasoning probe
D17-MF is an experimental, default-off, preregistered probe of Jev's (jev-1.13.0) multi-fact reasoning (#2850).
D17-MF: multi-fact reasoning probe
D17-MF is an experimental, default-off, preregistered probe of Jev's (`jev-1.13.0`) multi-fact reasoning (#2850). It follows up on the D17 calibrated live run, where every study error fell in the multi-fact slice.
No live collection is included. Collection runs through the shared held-out collector after operator approval.
Why: the D17 finding and the no-spend diagnosis
On the D17 test split, multi-fact accuracy was 0.7833 for the champion (single call) and 0.7567 for the challenger (3-call mean). The calibration split measured differently depending on the basis:
- Out-of-fold (the D09 qualification metrics, from the fit report): 0.95
champion, 0.92 challenger.
- In-sample (full-split mapping, used in the diagnosis below): 0.96 and
0.93.
The diagnosis re-read the existing 7,200 D17 observations without new calls; aggregates are in `.aiwg/research/d17-multifact-diagnosis-2026-10-02.json`. The whole degradation is one failure type: rows whose missing link is written with the pool's contrastive verb.
| Grammar (split) | Contrastive verb | Raw accuracy, missing link | Calibrated champion, in-sample | Disabled relay | Yes rows |
|---|---|---|---|---|---|
| tuning | `bypasses` | 0.67 (n=9) | 0.89 | 1.00 | 1.00 |
| calibration | `blocks` | 0.36 (n=25) | 0.84 | 1.00 | 1.00 |
| test | `halts` | 0.00 (n=73) | 0.16 | 1.00 | 0.97 |
- On the test grammar, Jev answers "established" for every `B halts C` link.
- The D17 member calibrator was fitted on the calibration grammar. It learned to
discount those moderate `blocks` scores, which produced the optimistic calibration-split figures.
- Model selection (BIC over every subset of the recorded factors) keeps exactly
grammar × failure type.
Hypotheses (preregistered)
| Id | Hypothesis |
|---|---|
| H1 | Wording: missing-link raw accuracy depends on the wording pool at fixed hops and encoding. |
| H2 | Depth: raw accuracy falls from one relay hop to three, at fixed pool, outcome and failure position (first or last). |
| H3 | Missing-link polarity: a missing link is read as established (permissive linking), while a disabled relay is handled. |
| H3a | Encoding: the failure is specific to the contrastive verb rather than to missing links in general; pure absence is the baseline. |
| H4 | Layout: at one hop and the last link, the D17-faithful rendering differs from the equalized one. |
Metrics
- Primary reasoning metrics are independent of any calibrator:
- the raw native answer (P(yes) > 0.5) per cell, with a Wilson 95% interval;
- the AUROC of the raw P(yes) between yes rows and missing-link rows, per
pool × hops (pooled and per encoding), with a Hanley–McNeil 95% interval. The Hanley–McNeil interval collapses to zero width at an AUROC of 0 or 1, so it is report-only: no decision rule uses it.
- Calibrated metrics (accuracy, mean confidence, decile ECE through the
registered D17 member calibrator) serve only the calibration-under-shift rule. That calibrator was fitted on the pool-b grammar, so calibrated answers would confound pool contrasts with where it sets its threshold. Both are reported.
Decision rules
The rules are frozen in `D17MF_PROTOCOL.decisionRules` (`src/decision/ensemble-study/multifact-study.ts`).
- H1 uses Holm across the whole family of pool-pair contrasts
(family-wise 0.05, two-sided pooled two-proportion z). It needs
| difference | ≥ 0.10 at every hop level of an encoding. |
- H2 restricts both arms to first- and last-position failures, and needs a
Newcombe 95% upper bound below −0.05 in two of three pools.
- H3 needs relay-off minus missing-link accuracy to have a lower bound above
0.10 in every pool, with "yes" answers on more than half of that encoding's rows.
- H3a decides between "contrastive-specific", "general" and "mixed". The
pure-absence baseline is reported against interior-position contrastive rows.
- H4 is Holm-adjusted across the three pools and compares last-link failures
only. The faithful control always breaks the relay-to-destination link; the unmatched contrast is reported too. H4 bundles four differences: 1. rule wording (D17's sufficient one-relay rule vs the exhaustive link-only rule); 2. line order (fixed vs uniformly shuffled); 3. decoy lines (none vs label-independent decoys); 4. entity names (role-prefixed vs neutral).
- Replication of D17 needs the pool-c faithful `halts` cell to have a raw
accuracy upper bound of at most 0.50.
- Coverage gate: every test cell must reach 95% observed rows. Otherwise the
report is marked incomplete and every verdict is "insufficient".
- Accuracy by failure position (first, interior, last) is reported for every
cell.
- Decision: always HOLD. This is a provider diagnostic, not a promotion.
Design
`src/decision/ensemble-study/multifact.ts` registers the generator `d17-multifact/v1`, through the outer registry `src/decision/heldout/study-generators.ts`. The D17 and D29 generator pins are unchanged, and the D17-MF digest covers the generator and the hash module it imports.
Factors
| Factor | Levels |
|---|---|
| Wording pool | The three disjoint D17 grammars |
| Relay hops | 1 and 3 |
| Outcome | yes; relay-off; missing link written as the contrastive verb, as an explicit negation (`does not link`), or as a non-connective verb; or not stated at all (pure absence, 3 hops, interior link) |
| Layout | equalized; D17-faithful control |
Pools
| Pool | Link verb | Contrastive | Non-connective | Enabled / disabled |
|---|---|---|---|---|
| pool-a | `connects` | `bypasses` | `lacquers` | `enabled` / `blocked` |
| pool-b | `routes` | `blocks` | `paints` | `active` / `paused` |
| pool-c | `links` | `halts` | `cooks` | `powered` / `dormant` |
The non-connective verbs come from unrelated semantic fields. Each matches its link verb's length and shares no letter position with the contrastive verb except the final "s". They are semantically odd between abstract entities, so an H3a "contrastive-specific" result could partly reflect how implausible the verb is rather than its contrastive meaning alone.
Rule. Every equalized item states an exhaustive, link-only rule:
An entity can reach a destination exactly when a sequence of stated facts of
the form "X `<link-verb>` Y" leads from it to that destination and every
entity in between is `<enabled>`; no other relation counts.
A contrastive, negated or non-connective verb therefore cannot make missing-link gold contestable.
Cells
| Cells | Rows each |
|---|---|
| Equalized missing-link (key): 3 pools × 2 hops × 3 written encodings, plus pure absence × 3 pools at 3 hops (21 cells) | 220 |
| Equalized yes: 3 pools × 2 hops | 110 |
| Equalized relay-off: 3 pools × 2 hops | 40 |
| D17-faithful control: 3 pools × {yes, contrastive}, one hop | 30 |
| Development rows (`tuning` split) | 2 per cell, 78 in total |
Sizing and power
- 5,700 probe rows plus 78 development rows: 5,778 in total.
- Key-cell Wilson half-width is 0.066 at p = 0.5, and at most 0.05 below 0.17 or
above 0.83.
- A Newcombe contrast of two key cells detects about 0.13 at 80% power.
- Yes rows were raised to 110 because they are the smaller class of every
shortcut-audit group, which bounds the audit's noise.
Limitations forced by the row bound. The shared credential scan bounds one corpus at about 5,880 rows (100,000 nodes at 17 per row). As a result:
- hops are the endpoints 1 and 3;
- pure absence exists only at three hops, where an interior link can be left
unstated without changing how often the query entities are mentioned;
- no verb × framing cross-cell fits. Within a pool the link verb and the
framing sentence co-vary, so H1 attributes a pool difference to the pool wording as a whole, not to the verb alone.
Failure positions are balanced within cells:
- links 0..hops for written encodings;
- interior links for absence;
- relays 0..hops−1 for relay-off.
Dispatch. Development rows come first, then the test rows in a seeded Fisher–Yates permutation. Its digest is preregistered as `analysis.dispatchOrderDigest`, so collection interleaves cells instead of running them in blocks.
Gold. Gold comes from latent-world reachability, and a separate text oracle re-derives every label.
Shortcut control (D29 audit principle)
In every equalized item:
- each cue class (link, enabled, contrastive, negation, non-connective,
disabled) appears a label-independent number of times;
- off-chain decoy lines among four decoy entities fill each class up to its
target, using distinct ordered pairs, so decoy facts never contradict each other or create paths;
- all fact lines are shuffled uniformly, with the rule last;
- names are five characters with distinct two-letter prefixes;
- the distinct-entity count is fixed per hop level.
Preparation runs the shortcut audit and refuses if it fails. Within each pool × hops group of equalized test rows, four cross-validated learners try to predict gold:
- a best single scalar threshold;
- a best token-count threshold;
- a depth-2 tree;
- an L2 logistic regression.
The features are counts, lengths, entity mentions, the mean, minimum and maximum normalized line position of each cue class, the positions and order of the first lines mentioning the query subject and destination, and the classes of the first and last fact lines.
Audit results
| Run | Maximum balanced accuracy |
|---|---|
| Public demo seed `d17mf-public-demo` | 0.552 |
| 64 sweep seeds `d17mf-sweep-0` … `d17mf-sweep-63` | at most 0.5994 (median about 0.567, minimum 0.526); all pass the 0.60 limit |
| 4 random private-format seeds | 3 pass; 1 refused at preparation |
| Positive control: the earlier chain-first line order | > 0.60, fails (regression-tested) |
| D17-faithful control (audited separately) | 1.0: D17's rendering is cue-solvable |
The margin under the limit is thin, so preparation occasionally refuses a fresh seed (1 of the 68 seeds tried so far). The operator then draws another fresh seed. The refusal depends only on the generated text, never on Jev's answers, so redrawing does not select on outcomes; the number of refused seeds is recorded in the run notes.
Calibrated evaluation
The probe is calibrated in collector `artifact` mode. The approval binds the registered D17 calibration set digest, and the scorer applies the D17 member (single-call) isotonic calibrator to the one fresh call per row.
Why not a new staged calibration? A calibrator fitted on the shifted pools would absorb exactly the miscalibration this probe measures.
The registered calibrator is re-derived from the D17 seal (re-fit, out-of-fold metrics, both reviews; the registered files must match).
The D17 preregistration pins the digests of D17's whole source tree (`src/`, `schemas/`, `tools/decision/` and the package files). The sealed study therefore reproduces only at the sealed commit, never at a later commit that contains D17-MF. The re-derivation runs D17's preparation from `d17SourceRoot`, a separate checkout of that commit. The CLI refuses the checkout unless its HEAD is exactly the sealed approval's `sourceCommit`, and unless the pinned paths have no tracked, untracked or ignored changes. The exact-reproduction check itself is unchanged.
The member artifact is qualified at three points:
1. at bundle time; 2. at collection end; 3. at the scoring clock.
The artifact expires 2026-11-01T10:02:34.263Z (its `effectiveAt`, 2026-10-02T10:02:34.263Z, plus the profile's 30 days), so collection and scoring must finish before then. The CLI checks that the registered artifact's expiry equals the bound pinned in source. The probe reuses the D17 DecisionDefinition byte-identically (asserted in tests).
Dispatch window. The collector enforces the expiry itself, keyed on the `d17-multifact/v1` generator, so a hand-built bundle cannot skip it:
- a session starts only if its start plus the preregistered session limit is no
later than the expiry;
- before every call it checks that the call's timeout also ends before the
expiry, and stops with `calibration-expired` otherwise.
`--dry-run-ledger` reports `sessionWindowOpen` and is not ready when the window has closed.
Scoring clock. Attempts carry no wall-clock time, and `qualification.json` is not covered by the evidence digest, so neither is used. The collector writes each session's start into `frozen.json` as `collectionStartedAt`. The trusted evidence digest covers that file, and the scorer checks the digests of the run and of every prior run. Collection end is bounded by the latest session start plus the preregistered session limit, which the collector never dispatches past. `--score` requires `evaluatedAt` to be:
- at least that bound across the run lineage;
- at most now.
The member calibrator is qualified at the bound and at `evaluatedAt`. A run recorded without a session start cannot be scored.
Collection and scoring run at the exact approved source commit:
- the collector checks the commit for every dispatch;
- the scorer regenerates the study from source and refuses changed
preregistration pins.
Seeds and budget
Seeds.
- Public and review seeds can regenerate gold from source. The rule is enforced
at the collector boundary (`validateHeldoutBundle`, keyed on the `d17-multifact/v1` generator) as well as by `--dev-review-material`, `--bundle` and `--dry-run-ledger`.
- It refuses the committed seeds (`d17mf-offline`, `d17mf-public-demo`,
`d17mf-smoke`, `d17mf-7f3a9c2e1b` and the review seeds) and the prefixes `review-` and `d17mf-offline|public|smoke|sweep|demo`.
- A private seed must also contain at least 24 lowercase hex characters of
fresh randomness, and a seed is at most 32 characters, for example `d17mf-$(openssl rand -hex 13)` (26 hex characters, 32 in total). Memorable and near-variant seeds are therefore refused.
- The bundle must use artifact-mode calibration.
- Seed reuse is refused at the collector:
- a corpus that already has observations under another preregistration
(`changed-preregistration`);
- a seed already frozen in the ledger under a different corpus, for example
after a generator revision (`seed-reused`).
`--dry-run-ledger` checks the first case before any spend.
Budget.
- This study's budget is 16,000 calls / 30M tokens / USD 5.
- It shares the D17 study cap of USD 8 with the D17 runs. The approval
attests all prior D17 spend as the study floor (USD 0.892652 so far) and the program total as the portfolio floor (USD 2.192652 so far).
- The worst case is 11,556 calls, 19.96M reserved tokens and USD 1.706
reserved, at the USD 0.10/1M reservation floor.
- On a fresh ledger the allowance is 12,800 calls, 24M tokens and USD 4.0. That
leaves 1,244 calls, 4.0M tokens and USD 2.29 of headroom.
- Expected spend is about USD 0.87 reserved, roughly USD 0.36 at the published
USD 0.042/1M.
Operator procedure
nice -n 19 node tools/decision/d17-multifact.mjs --dry-run FRESH_PRIVATE_SEED
nice -n 19 node tools/decision/d17-multifact.mjs --prepare FRESH_PRIVATE_SEED PREPARED_DIR
nice -n 19 node tools/decision/d17-multifact.mjs --dev-review-material PREPARED_DIR DEV_REVIEW_MATERIAL.json
# The material file is private: it must be under the artifact root, it is written mode 600, and it contains gold. Each item has `blind.payload`, then
# `unblind { goldLabel, textOracleLabel, variant/cell, factors, latentWorld }`.
# Review blind first, then unblind. Record the results in reviewTemplate.json; anchor DEV_REVIEW_DIGEST.
# Complete approval-template.json:
# - calibration { mode: 'artifact', calibrationArtifactDigest: REGISTERED_D17_SET_DIGEST }
# - the attested floors
# - an approvalReference that cites DEV_REVIEW_DIGEST
# Anchor its digest.
# D17_CONTEXT.json: d17CalibrationRun (the sealed D17 c03 run), d17CalibrationDir (registered set),
# d17CalibrationReviewFile, trustedD17CalibrationReviewDigest, d17DevelopmentReviewFile, trustedD17DevelopmentReviewDigest,
# d17SourceRoot (a clean checkout of the sealed commit, f7056a79fa3c1f857ab91aad0eaa91c37e593bda).
nice -n 19 node tools/decision/d17-multifact.mjs --bundle PREPARED_DIR APPROVAL.json DEV_REVIEW.json DEV_REVIEW_DIGEST BUNDLE.json D17_CONTEXT.json
nice -n 19 node tools/decision/d17-multifact.mjs --dry-run-ledger BUNDLE.json APPROVAL_DIGEST ARTIFACT_ROOT D17_CONTEXT.json
nice -n 19 node tools/decision/heldout-study.mjs --dry-run BUNDLE.json APPROVAL_DIGEST ARTIFACT_ROOT
AIWG_DECISION_HELDOUT_LIVE=1 nice -n 19 node tools/decision/heldout-study.mjs --collect-approved BUNDLE.json APPROVAL_DIGEST ARTIFACT_ROOT
# Resume each 30-minute checkpoint with a new run ID until complete.
nice -n 19 node tools/decision/d17-multifact.mjs --score SCORE_CONFIG.json SCORE.jsonThe score config names the run, gold and integrity files and the D17 calibration context keys, together with their trusted digests and `evaluatedAt`.
Remaining
- The live run, after independent review and operator go-ahead.
- The operator-delegated development review.
- The results summary.