Shared held-out study collector

Prompt-first procedure: Describe the outcome you want in your agent conversation. The agent should select and load the appropriate AIWG assets, explain material changes, request any needed approval, and report verification evidence. Exact commands and flags appear only in the CLI reference.

The experimental collector is default-off. It collects synthetic observations for source-controlled D17 and D29 study modules; it never promotes a model, changes a workflow, approves an SDLC gate or claims a study has passed. The checked-in example is a two-subject plumbing demonstration, not held-out data. No live collection was performed for this implementation (#2611, #2622, #2778).

Source-only commands

These commands use the existing `tsx` development dependency, without a build:

node tools/decision/heldout-study.mjs
node the import option tsx the input-type option=module -e \
  "import {prepare} from './agentic/code/addons/decision-engine/examples/heldout-collector-offline.mjs'; console.log(JSON.stringify(await prepare('demo')))"

The first command prints usage. The second generates only fictional example inputs and separate local gold. To freeze a reviewed study module, first resolve the agent-owned artifacts operation, then use a new directory immediately below that canonical artifact root:

node tools/decision/heldout-study.mjs the prepare option STUDY.mjs SEED OUTPUT_DIR
node tools/decision/heldout-study.mjs the dry-run option BUNDLE.json APPROVAL_DIGEST ARTIFACT_ROOT

Prepare stores corpus, preregistration and gold separately with exclusive file creation. Its incomplete approval template is deliberately invalid for collection. Assemble `BUNDLE.json` from the corpus, preregistration and an actual operator approval. `APPROVAL_DIGEST` is `heldoutDigest(bundle.approval)`, a canonical JSON digest, not a hash of file formatting. Keep this trusted digest in the external approval record. The bundle and all study analysis/gold files belong in the protected artifact store, outside the clean source checkout.

the collect-approved option has the same arguments as the dry-run option and additionally requires `AIWG_DECISION_HELDOUT_LIVE=1`. It is an operator execution surface; this documentation is not approval to use it. Library collection additionally requires `enabled: true`. Without that flag, the library returns `disabled` before reading inputs or invoking any hook. Injected transports are labelled `injected-transport` throughout evidence and never presented as live origin. An offline seam must contain a callable transport and host credential/disposal methods. Incomplete seams fail before preflight, artifact writes or credential lookup. The collector always supplies an explicit adapter transport; only the fully gated live path can call production fetch, after approval and resolver pin checks. The live environment and TLS gates are checked again at dispatch.

Approval and accounting

The closed v1 corpus, preregistration, approval, baseline, spend-event, frozen-input, calibration-phase, attempt, event and summary schemas are under `schemas/decision/Heldout*.v1.schema.json`; the current spend-head schema is `HeldoutSpendHead.v2.schema.json`. Definitions and receipts also pass the existing decision validators. Unknown fields, null price rates/fees, malformed calibration bindings, changed approval pins, duplicate payloads, cross-split families and non-synthetic provenance fail before dispatch.

Approval binds the corpus and preregistration, requested/served `jev-1.13.0`, all execution definition/ruleset/binding/projection pins, adapter version, calibration mode/phase and its required digests, resolver file digest, clean exact source commit, CI evidence reference, reviewer, titan workspace, provider terms reference and run ceilings. CI and provider terms references are operator attestations, not remotely verified results. The approval's closed `calibration` block replaces the former bare `calibrationDigest`; old approvals must be refreshed. No fixture digest or hash of a plan substitutes for a genuine D09 artifact.

Three calibration modes

Approval `calibration`	Preregistration `calibration`	Permitted collection
`{mode: 'uncalibrated-diagnostic'}`	`{scope: 'uncalibrated-diagnostic', allowedModes: ['uncalibrated-diagnostic']}`	All declared rows; diagnostic scoring only
`{mode: 'staged', phase: 'calibration'}`	`{scope: 'calibrated', allowedModes: ['staged'], calibrationPhaseSplits: ['tuning', 'calibration']}`	Only the declared calibration-phase splits; never test
`{mode: 'staged', phase: 'test', calibrationArtifactDigest, calibrationPhaseRecordDigest, priorApprovalDigest}`	The same immutable staged preregistration	Only test rows, after verification of the sealed calibration phase
`{mode: 'artifact', calibrationArtifactDigest}`	`{scope: 'calibrated', allowedModes: ['artifact']}`	All declared rows using an independent pre-existing artifact binding

A calibrated preregistration can allow both `staged` and `artifact`. Declaring `staged` requires a nonempty, unique `calibrationPhaseSplits` drawn only from `tuning` and `calibration`; either or both can be declared. Diagnostic scope allows only diagnostic mode. Unknown fields, null artifact digests, missing phase pins and unregistered modes fail closed. Every required digest must have the `sha256:` form. Artifact bindings establish identity only: the collector does not load, validate, qualify or register a D09 artifact. Even a digest matching a known fixture receives no validity or compatibility claim.

Dry-run reports `calibration`, `rowsInScope`, phase-only `maximumAttempts`, `reservedTokens` and `reservedUsdMicros`, plus the remaining global `allowance`. Worst-case estimates include the preregistered terminal retries. Test-phase dry-run also requires the matching seal. A feasibility estimate is not a collection approval or a D09 qualification.

Staged runbook

1. Freeze the complete corpus, split memberships and preregistration before either phase. Approve `{mode: 'staged', phase: 'calibration'}` with real source, pricing and operator attestations. No calibration artifact is needed to collect this phase. 2. Collect only the preregistered tuning/calibration splits. A session checkpoint is not a phase boundary: resume with a new approved run ID and the same corpus, preregistration, budget and original spend floors. 3. When every in-scope row is terminal, the collector exclusively writes and fsyncs `runs/RUN_ID/calibration-phase.json`. Successful requests, deterministic local rows and terminal measurement failures count as terminal. Unvisited rows skipped by a slice tolerance, checkpoints, stopping failures and unresolved reservations do not seal the phase. A seal proves completion of collection, not adequate samples or acceptable quality. 4. Fit the mapping, then separately qualify and register the D09 artifact through the existing calibration registry and qualification harness. A fitted mapping alone is not D09 qualification. Representative data, sample/slice sufficiency, compatibility, integrity, review and qualification evidence still have to satisfy D09. 5. Anchor the genuine artifact digest, `heldoutDigest(calibration-phase.json)` and the digest of the calibration approval that owns that seal in protected operator records. `summary.calibrationPhaseRecordDigest` supplies the seal digest; it is null for an incomplete calibration phase. 6. Approve the test phase with those three pins in `calibration`. The prior approval must be a calibration-phase approval for the same corpus and preregistration. The test approval keeps the shared budget and initial spend floors. No automatic approval transition or artifact fitting occurs. 7. Collect test rows. Before planning or dispatching them, the collector rereads the seal, its frozen approvals and complete journal/attempt lineage. It records `frozen.testPhaseAccessAt`, strictly later than `sealedAt`; equal, earlier or missing times refuse access. This is a host-clock observation, not an independent timestamp authority. 8. Score with the test approval and trusted evidence/integrity pins. The scorer receives the exact `approvedCalibration` binding and must compare its artifact digest to the trusted D09 artifact it actually uses, then apply existing D09 validation/compatibility and native study gates.

The seal is closed/versioned and binds all calibration-phase rows, approval, corpus, preregistration and the complete prior-run journal/attempt lineage. A changed seal, changed lineage, missing prior approval or changed study pins refuses test access and scoring. Protected storage and separately anchored approval digests remain necessary; local hashes are not signatures.

The offline example additionally exports `prepareStaged(seed)`. It constructs three fictional rows, never a D09 artifact or a live approval:

node the import option tsx the input-type option=module -e \
  "import {prepareStaged} from './agentic/code/addons/decision-engine/examples/heldout-collector-offline.mjs'; console.log(JSON.stringify(await prepareStaged('demo')))"

Reservations and shared spend

Every attempt, including retries and failures, fsyncs its full token/USD reservation before credential resolution or dispatch. The collector computes the UTF-8 byte length of the D10-projected, serialized request and adds the preregistered `providerOverheadTokens` allowance (default 512). This is the input-token reservation. `perRequestTokenBound` must cover it plus `outputAndHiddenTokenAllowance`; otherwise collection refuses the payload. The transport checks the actual body's byte length and digest against the planned request before sending it. Each input reservation costs:

ceil((serialized UTF-8 bytes + providerOverheadTokens)
     * max(attested input USD/MTok, 0.10))
+ ceil(attested per-request USD * 1,000,000)  // total in microdollars

The Jev adapter requires an attested output price of exactly zero. Any positive or unknown output price is refused, even with `outputTokenBound`, because the wire request has only `state`, `model` and `questions`: Jev has no remote generation limit. `outputTokenBound` is only a post-response halt signal; it cannot authorize paid output or increase a reservation. Token reservations include the preregistered output allowance. Null rates, fees, overhead and explicit null output bounds are invalid. Successful small usage never refunds a reservation. Unknown usage retains its reservation; observed overruns increase accounted spend and stop collection. Provider cost stays null when absent. Accounted dollars are conservative charges, never net savings.

D17 admission has a USD 8 study ceiling; D29 has USD 6. The portfolio ceiling is USD 48. Before each dispatch, accounted run spend plus the next worst-case reservation must fit the remaining allowance. Every mode fixes its 80% thresholds against the first approved study budget and original baseline remainders, then subtracts accumulated charges. Both phases and every resume share that allowance; switching mode, phase, corpus or run ID never applies 80% anew to a remainder. The collector rejects an approval whose budget cannot cover even one of its planned calls (including a USD 0.0005 approval with a larger reservation). Calls and reserved tokens also stop at 80% of the first approval, cumulatively across all modes, phases and resumes.

The collector records each operator floor once, with its approval digest and the counter's genesis digest, in `research/qualification/heldout/baselines/{D17,D29,portfolio}.json` beneath the canonical artifact root. The initial floors must include spending outside this collector; subsequent approvals repeat the original floors. Each study baseline also pins the first approval's closed USD/call/token `budget`; the portfolio baseline has `budget: null` because it has no separate study allowance. Admission and dry-run compare every subsequent approval to the stored study budget, independently of corpus identity or surviving run directories. Any changed budget is refused before credential access; existing thresholds are never rewritten by an approval.

A separate `research/qualification/heldout/spend-counter.jsonl` records every reservation and settlement across both studies. Each append is hash-chained and fsynced; settlement only adds observed excess and never subtracts a reservation. The counter stores cumulative collector charges and reserved token counts, and derives USD, call and token totals per study. For each scope, accounted spend is the maximum of its baseline, its baseline plus counter charges, and its baseline plus scanned run charges. Deleting a run directory, or its events and summary together, therefore cannot restore allowance. The counter also preserves call/token allowance and unresolved/stopped-attempt status after such deletion. Lost run receipts cannot be reconstructed from this compact counter.

`spend-head.json` independently anchors the latest sequence/digest and the complete set of baseline content digests. Its v2 record is replaced atomically and its directory fsynced after every counter append and first approval for a study, including an approval that dispatches no calls. Baselines are synced before publishing their head pins. Replacing, adding or deleting a baseline fails verification even after every run directory is deleted; a matching new approval cannot authorize changed baseline contents. Both dry-run and collection verify these pins before credential access.

A surviving baseline with a missing, truncated or broken counter, a mismatched head, or an interrupted head update refuses further collection with `spend-counter-operator-repair-required`. This includes truncation to a valid chain prefix after deleting all run directories. Preserve the counter, head, baselines and remaining run evidence; an operator must reconcile them against protected backups and provider billing. Never delete these files to restart. There is no automatic repair, migration or baseline reset. Old baselines without a genesis pin or the required budget field also require reconciliation; missing, null or malformed study budgets fail closed. Legacy v1 heads without baseline pins also require operator reconciliation; they are never automatically upgraded from unverified baseline files. Dry-run uses the same accounting without creating baseline/counter records.

Under the attested free-output tariff, the only residual per-call cost overrun is provider-side input tokens exceeding serialized bytes plus the overhead allowance. Any reported excess halts the run immediately and is charged to the counter, including the stopping call. This allowance is preregistered, not a provider-enforced limit. A tariff violation or unreported charge remains outside that assumption and requires billing reconciliation. Free output may exceed a halt threshold without increasing its attested charge. Concurrent spending by unrelated runners remains outside this ledger.

The counter is append-only in collector code; filesystem permissions and protected backups remain essential. Hashes and a local head do not prevent a coordinated rollback of the counter, head, baselines and all run evidence. A crash between durable writes fails closed and can require operator repair. The current counter reader caps the file at 32 MB and rereads its chain before each append; larger histories require reviewed archival/repair tooling, not silent truncation. Full-study throughput is not qualified by these offline tests.

A filesystem lock serializes arms, processes and studies using this collector. It does not serialize unrelated legacy runners. Dispatches are at least one second apart. Sessions are at most 30 minutes, with the same 80% stop rule. Before starting a subject, the collector checks whether every remaining arm and allowed retry can fit its worst-case time. A new approved run ID resumes completed requests without recollecting them; its evidence pins prior run journals. Frozen corpus/preregistration changes cannot reuse prior observations.

The preregistered failure policy allows zero or one retry. Only terminal invalid-output, overloaded or 5xx service-error responses qualify. Such errors may have unknown usage/identity; they remain failed observations charged at the full bound, never compatible successful results. Exhausting retries creates a measurement failure. Once failures exceed the registered tolerance (at most 5%) within a split/slice, that slice receives no further calls. Missing rows remain explicit. A completed traversal can therefore still have missing data.

Unknown execution, timeouts, cancellation, auth/privacy failures, served-model change, successful responses missing identity/usage, and usage/price overruns stop the run. Completed receipts remain durable. Unknown execution retains the global lock; restart refuses unresolved reservations or prior stopped attempts. Operator reconciliation is required. There is deliberately no automatic stale-lock removal or retry of uncertain remote execution.

Projection, credentials and evidence

The runtime reuses D10 projection and the existing Jev adapter/evaluator. Only `input.payload` reaches model-visible state, explicitly marked untrusted/public; gold, local outcomes, IDs, slices and other control fields stay local. Known credential syntax is rejected before collection, and resolved credential bytes are scanned against captured output, observations, receipts and traces. The collector suppresses stream/console output during each attempt, discards raw exception text, uses existing redaction, and disposes the resolver after the run. Trusted study modules must not launch child workers; this collector has no child-worker execution API. The existing resolver's helper output stays piped inside the resolver and its errors are fixed categories.

Live resolution uses only `tools/decision/jev-credential-resolver.mjs`, pinned by its byte digest. It enforces verified TLS and a scoped AppRole logical reference, calls revoke-self after the read, and keeps key bytes in memory. The reused resolver attempts revocation on all token-bearing paths; its revocation request is best-effort, as documented in that module. No new resolver or alternative credential lookup is introduced here.

Each hash-chained journal records reservations, terminal dispositions, request and bounded-response byte digests, served identity, nullable usage, receipt pins and trace pins. The adapter observer scans bounded response bytes before answer validation, so an invalid answer cannot hide model drift, usage overrun or credential material. Raw provider bodies are never retained. Only safe receipts/traces are stored. Reread verification rejects changed frozen files, broken chains, truncated completed journals, missing traces, mismatched receipt hashes and altered reservations. The external trusted evidence digest binds both the journal and frozen inputs, including prior-run references. File hashes detect changes against that pin; they are not signatures or a substitute for protected storage.

D11 execution uses `recorded` mode, re-reading the journal before recording a pass/fail evidence artifact and source-linked evidence manifest. This proves ledger replay integrity, not live origin, model quality, privacy approval or release readiness. No G3/G5/G6 flags are fabricated.

Study modules and remaining inputs

`HeldoutStudyModule.prepare(seed)` returns a corpus, preregistration and separate gold. CLI preparation checks the study module's scorer byte digest, requested seed and gold digest. Collection independently re-derives `generatorDigest` from the registered generator implementation bytes and requires the corpus seed to match the common seed component of every row seed. Every row additionally carries `provenance.generatorId`, `seed` and `outputDigest`. The collector re-runs the source-controlled generator and compares the complete row (excluding provenance) and its digest. Rehashed copied input, unknown generators and missing provenance fail before dispatch. Corpus data cannot register generators or choose executable module paths. World seed components and corpus seeds must match `[a-z0-9][a-z0-9-]{0,31}`; row seeds append a one-to-five-digit index and one of `example`, `single`, or `local`. The additive `heldout-lamp-splits/v1` generator appends an explicit `:tuning`, `:calibration` or `:test` split; its full output is also reproduced. Whitespace, instruction punctuation, non-ASCII text and overlong components are rejected. The dispatched world name is `w-` followed by 16 hex digits derived from the seed digest, so even permitted seed text stays out of model-visible state. A bounded identifier still needs synthetic provenance review; this constraint does not prove that an identifier has no external meaning. The registry includes the fictional `heldout-lamp/v1` and `heldout-lamp-splits/v1` examples, the frozen D29 `d29-synthetic/v1` through `d29-synthetic/v7` generators, and the source-controlled `d17-entailment/v1` synthetic generator. The current D29 study uses the `d29-synthetic/v7` registry entry, with 2,000 subjects, ten slices, disjoint train/test wording pools, matched benign annotations and identifier distributions, explicit screening-control attempts and corrected attribute-cardinality conventions. A source-pinned train-only passage baseline v3 is the primary comparator; passage v2 remains the same-wording solvability-ceiling diagnostic, with passage v1 and the original baseline secondary. Public development seeds and corpus digests through `d29-study-v7` are excluded from paid collection. The `single` and `local` row seeds encode each version’s fixed split layout; malformed layouts and out-of-range indices fail regeneration. This proves reproducibility, not held-out quality or correctness of the gold. These experimental v1 contracts are tightened in place: earlier unproven rows must be regenerated, and approvals/attempt token reservations must be refreshed. The calibration block, frozen access time, summary seal pin, spend-counter reserved tokens and baseline budget are required by the tightened v1 schemas. Existing ledgers need operator reconciliation; there is no silent conversion or spend reset. Generator changes require fresh corpus pins; existing ledgers are not migrated automatically across these experimental contract changes.

`scoreHeldoutStudy` rechecks trusted approval/evidence/gold/scorer pins and validates digest-bound upstream eval-integrity before calling the local scorer. It rechecks each journal's phase membership, and test-phase scoring re-verifies the seal and recorded access time. The scorer receives only the approved phase's `corpus.rows` and attempts; `preregistration.corpusDigest` still identifies the full frozen corpus. Gold is caller-supplied local data, not a model-visible input or a mechanism for isolating trusted host code.

`HeldoutStudyModule.score` receives `approvedCalibration: HeldoutCalibration` (the exact approved mode/phase and digests) and `calibrated: false | null`. Diagnostic mode and pre-fit calibration-phase collection pass `false`; their scorers must return `calibrated: false`; `d09Qualified` and `calibratedGate` must also be false when present. Conflicting or missing calibration declarations and a structured `decision: 'PROMOTE'` are refused. Artifact/test mode passes `null` (unknown), because the collector has not validated D09. The wrapper reports the binding, `calibrationArtifactValidation: 'not-performed'`, `d09Qualified: false` and `calibratedGate: false`. These are collector claims; native study diagnostics require their own trusted artifact validation. A scorer is trusted host code; arbitrary diagnostic prose is not validated as a qualification claim. The library wrapper always returns HOLD or preserves ROLLBACK, including when artifact/test diagnostics contain a proposed PROMOTE. Diagnostic and pre-fit calibration scorers cannot return that structured decision. A scorer callback is trusted host code, not a model-supplied executable. Its actual module digest must be independently checked by the caller, as CLI preparation does.

A row can have multiple named arm requests (D17 single champion plus independent ensemble members), or zero requests with an observed local outcome (D29 failed hard prerequisites). Model output cannot select IDs, ranks or protected status. Study modules must implement their own native observation mappings and gold oracle; the collector does not fabricate distributions, calibration or reviews. The experimental D17 module supplies its synthetic generator, source-only preparation, native mapping and statistical report scaffolding. Use its preparation command to retain all study-specific artifacts.

The following remain open, with no live acceptance claim:

Study work	Exact missing input
D17 measured ensemble report, AC7/AC14	Actual Jev observations for both phases of the staged D09 study, the operator-reviewed and D09-qualified calibration set, externally anchored integrity, approved extra-cost tradeoff and blind review; staged preparation, calibration-only fitting, review-gated registration, calibrated scoring and the native record now exist in the D17 module
D29 measured screening report, AC8/AC9/AC13	Study module implements the corpus/oracle, individual question mapping, calibration recipe and the absolute GateBinding. Actual observations, compatible D09 artifact, operator gold/reviewer audit and protected integrity/access records remain missing.
Staged D09/test handoff	Sealed real calibration observations, genuinely qualified/registered D09 artifact, protected artifact/seal/approval anchors and human test-phase approval
Any live collection	Priced approval with real evidence references, clean source/CI attestation, canonical root, actual prior spend, resolver pin, synthetic privacy approval and provider terms record
Promotion or production rollout	All native and external thresholds evaluated against complete data, compatible calibration, protected eval-integrity evidence and separate operator approval

The collector preregistration pins a separate study analysis digest. It does not implement D17/D29 quality thresholds or economics. Their existing report builders and additional study-specific gates remain the scorer's responsibility. No PROMOTE can be obtained from this collector. Incomplete native endpoints must stay missing, with a complete-case appendix and failure-as-error analysis prepared by the eventual study module; the example supplies neither analysis.