TV-12 bounded live collection (#2681)

Prompt-first procedure: Describe the outcome you want in your agent conversation. The agent should select and load the appropriate AIWG assets, explain material changes, request any needed approval, and report verification evidence. Exact commands and flags appear only in the CLI reference.

This tooling collects context estimator comparisons. Collection alone does not qualify a profile or approve a margin. Separate reviewer-gated steps, described below, record the margin and run the enforce canary. Nothing here enables production enforcement or completes #2681. All tests use fake transport and are guard/recording tests, not live qualification evidence.

The entry point is `tools/decision/context-live-qualification.mjs`. This is a source-checkout-only tool, not an installed package CLI command. With no arguments it reports usage and makes no provider calls. `the prepare option PROFILE.json CORPUS.json` builds a reproducible synthetic corpus without resolving credentials. The output must be beneath the canonical root returned by the agent-owned artifacts operation; the destination must not already exist. A current build is generated before the runtime is imported, avoiding stale compiled code during collection.

The profile must declare the `canonical-json-utf8-ceil` estimator version `1.0.0`, 64,000 aggregate and 32,000 state-plus-longest-question limits, an explicit `safetyMarginBps`, and `requestEnvelopeTokens`. The generator targets raw and effective boundaries at offsets -1, 0 and +1 using actual compiled Jev questions and D10 trust-partitioned state. It includes Unicode/nested content, 255-option Choice, 10-level Score, many short questions, and a dominant question. It pads only synthetic content. The estimator is not a provider tokenizer; its estimate is what the study measures. Every dispatched request also retains its wire digest and serialized byte count, never its body.

Prepare and freeze approval separately before collection. The `ContextLiveApproval` interface lists required fields:

• Explicit approval, reviewer, staging workspace, unique run ID, exact clean source

commit, and exact-head CI reference.

• Pinned served model, API revision `v1`, host-declared region, approved secret

service logical reference, and SHA-256 digest of the trusted credential resolver.

• Request, total-token, USD and wall-clock ceilings, plus an independently approved

per-request total-token and USD upper bound and its approval reference.

• Corpus digest and digest of `contextLivePreregistration(corpus, marginRule)`.

The generated `context-live-preregistration/v2` also freezes the provider-failure policy and minimum measured case count. Old preregistration digests are refused. Freeze these together with the margin rule before new provider usage.

The prerequisite CI reference and human approval are supplied attestations; the runner cannot independently authenticate the reviewer or inspect a CI service. Approval is not inferred from file existence or from a successful offline test. The Jev adapter currently exposes region only as a host-declared attribute, not a transport-enforced region selection. The approver must establish staging/region suitability before collection.

A trusted host module exports `async resolveCredential(logicalReference)` returning credential bytes from the approved secret service. Its contents and digest require host review. It must have no import-time side effects. The CLI accepts no transport module or arbitrary provider endpoint. Collection invokes the real built-in Jev adapter against its fixed v1 endpoint. The programmatic offline transport seam is always marked `synthetic`; full orchestration tests use manifest mode `offline`. It cannot produce live evidence or receive a live-host admission token. The CLI exposes no offline transport seam.

AIWG_DECISION_TV12_LIVE=1 node tools/decision/context-live-qualification.mjs the collect-approved option \
  /approved/approval.json sha256:APPROVAL_FILE_DIGEST /approved/corpus.json \
  /canonical/artifact/root /approved/credential-resolver.mjs \
  sha256:RESOLVER_DIGEST

Live-mode preflight and the #2681 spend cap

the collect-approved option and the canary-approved option are refused before the build, the resolver import or any credential read when:

• `AIWG_DECISION_TV12_LIVE` is not `1`;
• `NODE_TLS_REJECT_UNAUTHORIZED` is set to anything other than `1`;
• the SHA-256 of the approval file or the resolver differs from the digest given on the

command line.

`ARTIFACT_ROOT` must resolve to the canonical `artifact_root` itself, not a subdirectory. Every run directory lives in the fixed ledger `research/qualification/2681/runs/` beneath it. Every approval and canary plan is capped at USD 2.00, the #2681 limit. The cap also covers all runs together. Before the first reservation, a run takes the ledger lock and sums the earlier collection and canary spend:

• a run with a summary counts its charged reservation;
• a run without one (crashed or interrupted) counts its full approved budget;
• an unrecognized or unreadable entry refuses the run.

The run's USD ceiling is the smaller of its approval and USD 2.00 minus that prior spend, and the 80% stop applies to it. When nothing remains, the run is refused. A run writes its approval first, so an interrupted run is always charged its full budget. A stale `.tv12-spend.lock` is never removed automatically. The approval check also requires the collection profile's margin to reach the preregistered maximum before any spend. The host `dispose()` zeroes the key when a run ends.

Trusted Jev credential resolver

`tools/decision/jev-credential-resolver.mjs` is the reviewed host module for this runner. Its logical reference is `openbao-approle.NAME.typesafe-jev`, which names the scoped OpenBao reader AppRole. The reference goes into the Jev binding's `credentialRef`, so it must match `^[a-z][a-z0-9.-]+$`. The approval check enforces this. On first use it runs the host token helper (`bash $AIWG_OPENBAO_TOKEN_HELPER approle NAME`), reads the key field `token` from the KV v2 locator in `AIWG_JEV_OPENBAO_SECRET_PATH` at the HTTPS `BAO_ADDR`, and revokes the client token. The locator is host configuration because it is private; it is not in source or approval artifacts. Secret-service requests set `rejectUnauthorized: true` explicitly, and `BAO_SKIP_VERIFY` has no effect. A `NODE_TLS_REJECT_UNAUTHORIZED` value other than `1` is refused. An internal CA needs `BAO_CACERT`, `NODE_OPTIONS=the use-system-ca option` or `NODE_EXTRA_CA_CERTS`. If the helper output fails validation, any token-shaped word in it is still revoked. The key is held in memory for the process, each call returns a copy that the adapter zeroes, and the exported `dispose()` zeroes the key itself. Errors carry a fixed category, never helper, secret-service or provider text. The file digest pinned in the approval covers the code only. The AppRole's own scope, not this module, is what limits which secret can be read.

Dry run

the dry-run option takes the same arguments as the collect-approved option and needs no live gate. It validates the approval and corpus. It checks the pinned approval and resolver digests, the complete generated corpus, the clean exact source commit, the canonical artifact root, and the remaining #2681 spend. For every admissible partition it runs the same request checks as collection: adapter destination and answer-shape capabilities, the ruleset and binding schemas (including the credential reference), and the artifact pins. Oversized cases are recorded as rejected, and the run moves on, exactly as in collection. It then reports the partitions collection would send. For each case it lists the partitions and their estimated input tokens, then the initial requests, `maximumRequestsWithRetries` (twice the initial requests), the worst-case reserved token and USD bounds including every possible retry, and `requestCapacityAtStop`: the most requests the 80% stop admits under the capped USD ceiling. It never imports the resolver, resolves a credential, calls a provider or writes a file. It exits nonzero unless every precondition holds and the plan fits before the stop.

After collection: margin record and enforce canary

These steps are implemented and tested offline only. None has run against Jev.

1. Margin record. `the record-qualification option APPROVAL CORPUS RUN_DIR REVIEW OUTPUT` reads the retained `*-comparison.json` rows and rebinds each one to its frozen corpus partition by plan digest, profile digest, estimate and usage digest. Synthetic, missing, surplus or edited rows are refused. This downstream recorder still requires complete partition coverage; a collection with measurement failures can report a candidate margin but cannot create an enforcement qualification. The selected margin is the preregistered rule's output: the worst undercount plus `extraReserveBps`. If that exceeds `maximumMarginBps`, no record is produced. The reviewer's `context-margin-review/v1` must approve that exact margin, the digest of these records, and a new profile identity. The collection profile's margin must be at least the preregistered maximum, so every collected request stays a single request under the selected margin. The output record holds the qualified profile, its digest, and a `ContextQualification` accepted by `assertContextQualified`. 2. Stored-record check. `the verify-qualification option RECORD` runs the enforcement gate on the stored file, then confirms that the same record is rejected when only the profile version changes. 3. Enforce canary. `the canary-approved option CANARY_APPROVAL APPROVAL_SHA256 CORPUS RECORD ARTIFACT_ROOT RESOLVER RESOLVER_SHA256` needs a `context-canary-approval/v1` that embeds the canary plan frozen before collection (its digest must match) and names the record digest, served model and region. Each planned case first runs through `evaluateDecisionRuleset` with native batching and `rollout: enforce` using the recorded qualification. It then runs again rolled back to `observe-only`, which evaluates every question singly. Before each phase starts, every dispatch that phase needs is reserved against the approved bound. The run stops at 80% of any ceiling, on the first failed check, or on an evaluation error. A case passes when:

• every dispatched partition is within the effective limits;
• every reported input is within the documented aggregate limit and the approved bound;
• every per-call reported output is known and within the approved per-call output

ceiling (`perRequestBound.outputTokens`, a sub-ceiling of `totalTokens`);

• an oversized case produces a context rejection with no dispatch;
• the rollback sends no native batch.

Jev exposes no request-level output cap, so the output ceiling cannot be enforced before dispatch: it is enforced after dispatch by failing the canary on unknown or over-bound reported output, which stops further dispatch. A canary plan without the ceiling, or with a ceiling above the total bound, is rejected before any credential use or dispatch. Nothing here has run against Jev.

Per-case rows and the summary are metadata only.

The evaluator preflights the worst-case completion result envelope before dispatch (#2797): every evaluation clones the full context plan, and usage is recorded cumulatively, so the probe assumes every evaluation carries every usage entry with the binding's full retry attempts and answer-domain distributions. An invocation with context planning and 24 aliases (`many-short`) rejects with `invalid-input` before receipts, credentials or transport instead of producing a result document that exceeds the default entry limits (`property-count`) after dispatch. The canary carries `many-short` as a rejected-before-dispatch case with zero dispatches in both phases; each phase checks the envelope with its own preflight call and the canary verifies the exact rejection reason per phase. D11 manifest linking from #2599 and #2604 remains pending.

Missing inputs, source drift (including untracked files), changed corpus or preregistration digests, changed resolver digest, and reused run directories fail closed. Keep approval and corpus files outside the clean source checkout. The live runner verifies the corpus equals the complete deterministic generated corpus, not an arbitrary subset labelled synthetic. It snapshots the approved synthetic corpus, approval and preregistration into private mode-0600 files before dispatch. Each D11 case binds those inputs by digest. The corpus is the approved private study input, not a provider body export; comparison and D11 evidence contain only metadata. The existing D11 writer retains that evidence. The generic D11 runner never invokes executors in `live` mode. Collection therefore runs every case itself, sequentially, and then records the outcomes. It uses D11 mode `recorded` for the provider path and `offline` for the synthetic seam. Provider provenance is in each comparison record (request ID, served model, usage digest), not in the D11 mode label.

Collection boundary

Ordinary `observe-only` deliberately disables native batching. This runner therefore composes the existing D10 projection, actual question compiler, D06 planner, D02 admission controller and Jev adapter in a qualification-only path. It does not add a rollout mode or weaken the ordinary default/enforce gates. Every planned partition is independently replanned as exactly one complete request. Multiple questions use `evaluateMany`; a singleton uses `evaluate`. Both receive the same projected state, compiled questions, pinned model and policy metadata used in planning. Definition, ruleset and binding schemas and artifact pins are validated before admission.

Oversized state or questions never reach credentials or transport. An aggregate group that needs partitioning is not dispatched whole. Each partition has its own wire identity, estimate, authoritative shared input usage and comparison. Shared usage is counted once, never copied into an accounting sum per answer. The admission lease is released once per request; conservative token and USD reservations are never refunded. This intentionally overcounts uncertain or failed requests.

Before each dispatch the runner reserves one request and the approved token/USD upper bound. It will not cross 80% of any ceiling. An abort signal enforces 80% of the wall-clock budget, including credential resolution. There are no fallbacks. The v2 preregistration allows at most one retry per request for a terminal non-success `invalid-output`, `service-error` or `overloaded` observation with HTTP 200 or 5xx (including 529). This covers r3's HTTP 200 with null model and usage. Every attempt reserves and charges the full approved bound before dispatch. Each failed attempt is persisted with case/partition, attempt, reason, HTTP status and safe request ID; unavailable metadata stays null. A second failure marks the case as a measurement failure, fails its D11 check, and continues to subsequent partitions and cases. A successful retry retains its first failure evidence. This policy was amended after r3, before collecting r4 scores.

Uncertain execution, timeouts, cancellation, authentication/credential anomalies, other provider failures (including the first limit-related 4xx and rate limiting), changed served model, missing request ID/model/authoritative input-output usage on success, source drift, and usage above the approved bound still stop subsequent collection. Safety and usage checks precede the retry decision. Source is checked before each case and credential resolution; final source drift leaves a stopped summary and preserves completed comparisons. A bound is a reviewed assumption, not a provider-enforced output-token cap: an unexpected provider overrun can only be detected after that request and then stops the run.

Provider cost remains `null` when absent. The approved USD bound is retained as a reservation, not invented observed cost. Earlier partition evidence is persisted before dispatching the next partition. Errors retain generic stop reasons rather than provider, secret-service or request bodies. Evidence records include request IDs after the adapter's bounded safe-ID validation, served model, estimator, plan and profile digests, usage digest, estimated/actual input tokens, absolute token error and undercount basis points. D11 evidence remains HOLD until the separate reviewer-approved profile/margin, qualification validation, enforce canary and rollback requirements are completed. The summary reports `collectionSuccess` separately from qualification. The v2 summary lists `measured`, `failed`, `rejected` and `unmeasured` case IDs separately from `collected` request comparisons. A case is measured only when all its partitions succeeded. Every partition of a failed or incomplete case is excluded from margin derivation, though completed comparison files remain retained. A candidate margin requires no immediate stop and at least 12 measured cases (out of 13 admissible in the frozen 17-case corpus). Fewer cases yield a null candidate, never a zero undercount assumption. This permits one missing case for an exploratory candidate only; it makes no coverage claim for that input class. The rule remains worst measured undercount plus the frozen additional reserve; `marginValid` also requires the candidate to be at most the approved maximum. Nothing automatically promotes. Failed/skipped D11 cases or unmatched partition counts keep `collectionSuccess` false and the CLI exit nonzero, even when collection continues and produces a candidate margin. Legitimate rejected-boundary cases remain successful zero-dispatch checks. No live run has been performed by these tests.

r4 preparation and remaining evidence

The r3 collection stopped on `choice-255` after reserving USD 0.1008. The adapter's 255-option limit admits that case; see the local capability evidence. The corpus and estimator are unchanged. The preregistration changes, so r4 needs an approval pinned to the new clean HEAD and preregistration digest. When running from a worktree, set `AIWG_ARTIFACTS_PATH` to the existing canonical artifact store used by r3 before the artifact-root check and dry-run. Using an empty worktree-local ledger would omit prior spend. The dry-run automatically reads the existing ledger; never insert a manual zero or copy only selected runs.

Offline tests establish the failure policy, reservation, continuation and margin checks. Fresh r4 Jev responses, exact-HEAD CI sign-off, operator approval, a human margin review, complete coverage for the downstream recorder, enforce-canary and rollback evidence remain pending. The prepared approval is an operator sign-off candidate; the offline dry-run does not authenticate human approval or CI status.