Multimodal preprocessing lineage

Status: experimental, default-off. Jev remains text-only.

`PreprocessedEvidence.v1` records how text decision input was derived from a non-text source such as a scanned document, audio clip, video segment, or image. It is a provenance contract for recorded extraction outputs, not an OCR, ASR, captioning, vision, or document-processing service.

The TypeScript entry point is `resolvePreprocessedEvidence()` from `aiwg/decision`. Host code must opt in explicitly:

1. Load a closed `PreprocessedEvidence` manifest. 2. Resolve it for the destination that will receive the derived text. That is the D10 projection policy's provider and origin, for example `{ "provider": "jev", "origin": "https://api.typesafe.ai" }`. For an adapter that declares `egress: { mode: "none" }` and runs without a projection policy, it is `{ "provider": <adapter id>, "origin": PREPROCESSING_LOCAL_ORIGIN }` (`local://no-egress`). 3. If the resolution is `ready`, put exactly `resolution.state.text` in one decision input field. 4. Pass `resolution.receiptEvidence` as `DecisionEvaluationRequest.preprocessingLineage`, and host-controlled `DecisionEvaluationRequest.preprocessingVerification`. All fields are required unless marked optional:

• `manifests`: the current host-stored manifest records;
• `minQualityScore` (finite, 0-1), `maxAgeMs` (finite, at least 0) and an

optional `now` clock;

• `lifecycle`: D10 lifecycle state (`subject`, `now`, `tombstones`, `holds`);
• `inputBindings`: which input pointer carries which manifests' text, for

example `[{ "pointer": "/message", "manifestIds": ["invoice-scan-314"] }]`;

• `nonLineagePointers` (optional): every other string field in the input,

declared as ordinary non-lineage text.

The evaluator does not read raw media or derived text from the lineage evidence. When a non-empty `preprocessingLineage` is supplied, the `RulesetResult` is written as `decision.aiwg.io/v1alpha2` and contains only artifact pins, source/output digests, the authorized destination, selected segment IDs, locator/text/preprocessor-identity digests, closed quality source/flags, quality profile/label digests, retention/residency digests, counts, duration, policy outcomes and the evaluator's `dispatchGate` verdict. An empty lineage (no references and no traces, as returned by `resolvePreprocessedEvidence([])`) is treated exactly as absent, unless the host also supplied `preprocessingVerification` (see the evaluator gate).

Resolver checks

The resolver validates the manifest against the closed JSON schema (formats included), so it never accepts what the schema rejects. It also checks the media type for the extraction kind, source (when bytes are supplied) and output digests, ordered step continuity, unique step and segment identities, selected segment references, transformation links and locator policy before releasing text.

Each segment carries an `outputRange`: a UTF-8 byte range inside the digest-verified `output.value`. The segment `text` must equal that slice byte-for-byte, ranges must follow ordinal order without overlap, and the resolver releases the slices cut from the verified output, in source order. A segment whose text is not in the verified output is rejected even when its own `textDigest` matches.

Unsupported transformations, digest mismatches, real chain cycles, missing segments, wrong media types, missing media-specific locators, duplicate segment IDs/ordinals, and remote-url locators (for the source and for `output.reference`) fail closed. An identity step, whose input digest equals its output digest, must be declared `noOp: true`. A `noOp` step that changes content, or a `noOp` extraction step, is rejected. No-op step IDs are recorded in the lineage trace as `noOpStepIds`.

Human correction is never a preprocessing step kind. It enters only as an append-only `human-correction` transformation event with reviewer, rationale and a link to the previous output (ID, version, digest). Each correction takes the current output as input, and the final `output.version` must be newer than every corrected version. The original extraction steps stay unchanged. The correction events are part of the preprocessing identity digest, so a stored reference to the pre-correction output becomes stale.

Low-quality, truncated, incomplete, stale, policy-limited, untrusted (`policy.trust: "untrusted"`), derived-egress-denied, tombstoned or legal-held evidence resolves with status `review` and typed reasons. Preprocessor quality is stored as preprocessor evidence only; it is not Jev confidence, calibrated risk, or decision correctness. Free-text quality profile/label and retention/residency values stay out of receipts and traces and are represented as digests. There is no option to retain derived text or raw media in receipts.

Evaluator gate

`evaluateDecisionRuleset()` gates a non-empty lineage after request validation and before receipt acquisition, credential resolution and adapter dispatch:

• Refused (`error` / `data-boundary-denied`, no evaluations): the lineage is

malformed; a reference's recorded destination differs from the D10 projection provider/origin of any target that could be dispatched (including fallback targets); derived egress is not authorized for that destination; the destination cannot be bound (`unprojected-local` opt-out or a projection resolver that fails); a lineage reference is not covered by a host input binding, a binding names a manifest outside the lineage, or a bound pointer is outside every evaluation's input pointer (`input-unbound`); the value at a bound pointer is not exactly the text recomputed from the current verified manifests' selected output slices (`input-mismatch`); or any other string field in the decision input is neither bound nor listed in `nonLineagePointers` (`input-undeclared`). The binding checks run only when every referenced manifest is available and current; otherwise the result is review.

• Review (`review` / `insufficient-information`, no evaluations, no

outcome): no `preprocessingVerification` was supplied, its thresholds are missing, not finite or out of range, or its input bindings are malformed (`unverified`); its lifecycle state is missing (`lifecycle-unavailable`); the current manifest is unavailable (deleted or missing); a stored reference is stale against the current manifest (`checkPreprocessedEvidenceReference`); or re-resolving the current manifest under the host's thresholds and lifecycle state yields any review reason (quality, flags, trust, age, egress, tombstone or hold). Stored trace status and reasons are never trusted to allow dispatch; they can only add review reasons, as can a non-`ready` status, blocking reference flags, or missing or misaligned traces.

• Allowed: the request continues on the normal projection and dispatch

path.

The verdict is recorded as `preprocessingLineage.dispatchGate`. The gate and the recorded lineage share one immutable snapshot of the host-supplied `preprocessingLineage` and `preprocessingVerification` taken before the result is constructed, so caller-owned mutation during evaluation cannot change the verdict or the recorded lineage. A result cache is refused for requests that carry lineage, because a cache hit would bypass the gate. Input bindings come only from the host's `preprocessingVerification`. The lineage carries no pointer or text digest, and the evaluator never uses a stored digest. Checks run on the decision input the evaluator projects and dispatches; D10 projection can drop or redact a field but cannot substitute it. Every string value and every object key is a text-bearing position. Numeric, boolean, and null values are outside text-leaf binding: they carry no text, collect no value pointers, and are never compared against lineage text, so a scalar value can never satisfy a binding. Their object keys remain text-bearing member positions, so each scalar field's pointer must still be covered by a binding or a `nonLineagePointers` entry (in practice a declaration, since a bound scalar never equals the verified text). A bound pointer covers exactly its own string. A `nonLineagePointers` entry covers its whole subtree, keys included. The root pointer (`''`) is rejected as malformed host verification (`unverified` review, nothing dispatched): it would otherwise mark every text-bearing position as covered. Any other text-bearing position is refused as `input-undeclared`. For example, `{ "message": "<verified lineage text>", "attempt": 3, "dryRun": false, "note": null }` binds only `/message` to the lineage manifests while `/attempt`, `/dryRun`, and `/note` are declared in `nonLineagePointers`: the `3`, `false`, and `null` values are not text and are never text-compared, whereas an extra string value at any uncovered pointer is refused as `input-undeclared`.

When the host supplies `preprocessingVerification` but the lineage is empty or absent, the evaluator does not fall back to the text-native path. It returns a v1alpha2 `review` result with `dispatchGate: { outcome: "review", reasons: ["lineage-missing"] }` and dispatches nothing. Requests with neither a lineage nor `preprocessingVerification` are unchanged.

Offline fixtures

The checked-in fixture set `test/fixtures/decision/preprocessing/multimodal-lineage-v1.json` covers:

• scanned-document OCR;
• audio transcription;
• video captions with time/frame locators;
• image description with a bounding-box locator.

`agentic/code/addons/decision-engine/examples/preprocessing-lineage-offline.mjs` is the packaged runnable offline example. It resolves a recorded OCR fixture, sends text-only state to a local no-egress decision adapter, and produces a normalized decision result with body-free lineage references. The unit test `test/unit/decision/preprocessed-evidence.test.ts` runs the same example through the source runtime.

Text-native requests without a manifest are compared byte-for-byte against `test/fixtures/decision/preprocessing/text-native-golden-v1.json`, which was generated from origin/main's runtime with `generate-text-native-golden.ts`.

Lifecycle

Preprocessing lineage uses the common D10 surface `preprocessing-lineage`. `preprocessingLifecycleReferences(manifest)` returns the opaque lifecycle references for the manifest, its source asset, every preprocessing step and transformation event, and its output. Hosts register them as the owning subject's `links()`, so `eraseDecisionSubject()` tombstones them with the rest of the subject and `restoreDecisionSubjectBackup()` refuses to restore them. Identical sources or transformations map to the same opaque IDs, so one tombstone invalidates every dependent manifest.

Passing that lifecycle state (`subject`, `now`, `tombstones`, `holds`) to the resolver or to `preprocessingVerification.lifecycle` makes tombstoned records resolve to `review` with `lifecycle-unavailable` and no released text. A legal hold on the subject's `preprocessing-lineage` surface resolves to `review` with `legal-hold`. The evaluator routes either to review before dispatch. A deleted or unavailable manifest is never replaced by stale derived text for automatic action.

Production rollout still needs deployment store erasure wiring, backup expiry configuration, privacy approval, and live reviewer/process evidence.

Pending before production

No live Jev calls, real held-out media data, human-reviewer workflow, production retention backend, or privacy approval is claimed by this implementation. Those inputs remain required before enabling this feature beyond recorded offline fixtures.