Asynchronous decision jobs: offline experimental lifecycle
The AIWG-owned DecisionJob.v1 contract validates item identities, scope, immutable pins, budgets, retention deadline, guarded transitions, and exact…
Asynchronous decision jobs: offline experimental lifecycle
The AIWG-owned `DecisionJob.v1` contract validates item identities, scope, immutable pins, budgets, retention deadline, guarded transitions, and exact item-state summary counts. Jev has no documented asynchronous job endpoint. A job is not a provider-native bulk batch; completion does not authorize any downstream action.
`FileJobStore` persists immutable, mode-0600 revisions using exclusive hard-link publication and directory sync; `MemoryJobStore` supports offline fixtures. Both provide scoped acquisition and compare-and-swap. `DecisionJobRuntime` provides offline submit, read-only ordered pagination/poll, cancellation, eligible-failure requeue without dispatch, expiry reconciliation, tombstoning and scoped legal-hold toggles that reject deletion while held. `OfflineJobWorker` fences an individual fixture attempt durably before invoking an explicitly supplied test executor; a thrown, aborted or invalid fixture result becomes execution-unknown. Its callback is not an admitted backend provider integration. Restart reconciliation never replays a fenced attempt; finalization retries only a bounded compare-and-swap after concurrent sibling completion, never the executor. Different item subjects are presented separately. The caller must derive the full scope from trusted authentication, not model input; the runtime compares the authenticated scope before submit or transition. Other scopes receive no job record. A duplicate job ID with unchanged identity returns the existing snapshot; equivalent duplicate undispatched item IDs are coalesced at acquisition, while mismatched item pins or changed job identity are rejected before any backend call. Tombstones prevent ID reuse. An expired running item with a dispatched attempt becomes `execution-unknown`, never cleanly canceled. Retry of unknown items is forbidden by the contract. Out-of-order completions preserve original item order. All offline operations are disabled from provider-facing CLI/API surfaces.
An explicit host-only `admittedJobItemExecutor` bridge composes the existing synchronous evaluator with a durable D03 receipt store and enabled scheduler policy. It rejects unbound principal/workspace, missing cost limits, changed ruleset/definition/binding/input digests and incomplete receipts before exposing an item reference. Same-subject native batches remain within one item invocation. This bridge is not exposed through CLI/API. The worker requires an explicit durable token/cost reservation for admitted evaluator callbacks, sums reservations across all sibling item attempts under the job CAS, and refuses dispatch that exceeds the pinned job budget; the bridge also rejects plans whose host estimates exceed that fence. Reservations are conservative and not refunds. `OfflineJobScheduler` bounds one offline queue and round-robins authenticated principal/project lanes, with deterministic result slots and no retry amplification; it does not coordinate separate processes. For same-host local-filesystem deployments, `FileJobQuotaStore` wraps every journal writer in an ownership-checked directory lock and atomically checks per-project and per-principal queued/running/retained item, serialized byte, pinned token/cost, attempt-call and job ceilings before acquisition/transition. Its fail-closed lock is not automatically reclaimed after a process crash; `recoverStaleJobQuotaLock` requires an explicit host authorization decision and verified ESRCH for the recorded PID (including a second check before removal). Live, reused, invalid or unverifiable owners remain locked rather than risking unsafe theft. `FileJobPollLimiter` adds a bounded, private local-filesystem polling ledger under the same lock; a host can attach it to `DecisionJobGateway` before handle parsing so even forged-handle enumeration consumes quota. It enforces per-project and per-principal windows across same-host instances. Neither component is a distributed-filesystem coordinator or a replacement for provider admission. Bypassing the wrapper with direct `FileJobStore` writes bypasses aggregate quotas. `accountDecisionJob` reads the scoped D03 and D04 receipts to report known token/cost totals, unknown usage, labeled estimates and budget exceedance; it ignores per-answer allocations in native batches and rejects cross-item shared-state ownership. This report is read-only and not a transactional pre-dispatch quota reservation; a host must not enable jobs just because these pieces exist.
`createOfflineDecisionJobService` wires a local transactional quota journal, AES-256-GCM protected input/result store, bounded polling ledger, scheduler, worker, runtime, metadata-only telemetry and object gateway into one disabled-by-default host assembly. It requires a complete D10 lifecycle policy plus independent deletion/export authorities, separate injected handle/payload keys, and host-authenticated actors and admitted item executors. Admission rejects jobs outliving D10 retention; export is denied unless D10 permits sanitized export and the host authorizer approves it. Persisted jobs require D10 erase and backup-expiry policy. The payload store pins input values to the item subject digest and only accepts terminal results with validated receipt evidence and the item result digest. It authenticates encrypted data and signed quota metadata, enforces per-item and combined per-principal/project retained-byte ceilings under the same local lock as the journal, and purges contents only after a D10 tombstone and authorized job deletion; an independent tombstone still denies restored backups if a local marker is lost. Payload values do not appear in job revisions or telemetry. `DecisionJobGateway` is a host-only object-authorization adapter. It encrypts randomized opaque handles with an injected 256-bit host key and binds them to the authenticated tenant/project/workspace/principal plus job fingerprint, creation time and expiry. It rejects cross-scope, forged, expired and deleted handles without looking up another scope. Optional host-supplied list/export capabilities bind encrypted pagination cursors to the authenticated scope, reject stale pages and require explicit export authorization. `createDecisionJobHttpHandler` is an opt-in, unmounted HTTP boundary with a required host authenticator, queue admission decision, D10 delete controller, result-access authorizer and bounded request body. It derives scope only from the host authenticator, never from submitted JSON; exposes scoped opaque handles, list cursors, ordered partial results, guarded input upload and validated terminal result download, cancellation, retry and sanitized export. It does not create a listener or install a CLI command. The host must obtain actor and keys from trusted authentication/secret storage, supply admitted evaluator work outside the HTTP request and never emit handles to telemetry.
Do not mount this experimental HTTP handler for production users yet. There is no deployed authenticator or job-level provider scheduler, distributed quota/fairness authority, cursor limits, retry policy integration, deployed retained-result payload accounting, D10 deployment-backed erasure/backup/key-destruction evidence (local hold, purge and independent tombstone fixtures do not prove an external backup is purged), deployed D14 OpenTelemetry collector qualification (golden partial/retry/cancel/expire/restart/finalization fixtures now exist), or live deployment crash/load qualification (real killed-child crash fixtures at the submit, queue, dispatch, cancellation and finalization writes and repeated journal-soak fixtures pass offline). The journal retains digest-only historical metadata after tombstoning until the host invokes `createOfflineDecisionJobService(...).eraseJob` through D10's independently persisted lifecycle tombstone/eraser flow; the same operation purges encrypted input/result files. Purge removes revision files and leaves a local no-reuse marker; an independent `externallyDeleted` lifecycle authority is required to block backup resurrection if a restored directory loses its marker. Raw inputs/results are never stored in revisions. Do not store PII or secrets in IDs, scope fields, errors, or fingerprints. `sweepExpired` lets the host reconcile pinned deadlines across all local journals after a restart; it never replays an attempt. Retention expiry changes lifecycle state but does not automatically invoke the D10 eraser; held records remain until an authorized release/deletion. File stores require a local filesystem supporting atomic hard links; they do not provide a distributed CAS. No claim of exactly-once provider billing or execution is made.
Acceptance evidence (offline)
| #2610 criteria | Code and repeatable evidence |
|---|---|
| 1–2 | `job-contract.ts`, `job-store.ts`; contract and store tests exercise legal states, immutable pins, duplicate-equivalent item coalescing and mismatch rejection. |
| 3–8 | `job-runtime.ts`, `job-worker.ts`; mixed outcomes, ordered pages, cancel/unknown, retry lineage and exact summaries in `job-contract`, `job-store`, `job-worker` tests. |
| 9–10 | `job-evaluate.ts`, `job-accounting.ts`; same-subject request bridge, durable D03 receipt ownership and D04 shared-batch accounting in `job-evaluate` tests. |
| 11 | Real killed-child crashes immediately before (with a torn unpublished temporary revision) and immediately after the durable validation/submit, queue-transition, cancel-requested and finalization writes (`job-crash-boundaries`), plus the killed-child dispatch fence (`job-crash-process`); a fresh store and runtime reconcile each to exact summaries, retried submit/queue/cancel requests are idempotent or lose the CAS, and no executor call is repeated. Persisted D03 receipt/result materialization across service restart (`job-evaluate`), restart reconciliation and concurrent-CAS finalization (`job-worker`, `job-store`). |
| 12, 17 | Encrypted payload, scoped export/deletion, D10 hold/expiry/tombstone and independent backup-restore denial (`job-payload`, `job-lifecycle`, `job-service`); erasure also removes abandoned job-specific temporary files. |
| 13 | `fixtures/job-traces.v1.json` golden operations/status/counts for partial, retry, cancellation, expiry, restart and final reconciliation; metadata-only D14 spans. |
| 14–15, 18 | AIWG-owned job disclaimer, scoped AES-GCM handles/cursors and opt-in HTTP tests for authenticated operations, denied scope substitution and bounded metadata dimensions. |
| 16 | Per-principal/project shared-local-FS quotas, persistent polling windows, real two-process admission race, killed-owner lock recovery, capacity denial and repeated two-lane journal soak (`job-quota`, `job-poll-limiter`, `job-scheduler`). |
Resolving execution-unknown (opt-in, D16)
`execution-unknown` stays terminal by default. A host may opt in to one gated way out (#2722), backed by the effect ledger:
1. Construct `OfflineJobWorker(runtime, ledgerJobEffectRecorder({ ledger, receipts, jobs }))` with a `job`-subsystem ledger for the job's tenant and project. When an admitted item returns a validated D03 receipt, the worker records a `decision.receipt` effect (target `decision:job/<jobId>/<itemId>`, context the job scope and `attemptId`, payload digest the receipt digest) and verifies it before it writes the job record. A crash between the two leaves a signed, verified digest behind. 2. After a restart, call `runtime.reconcile(actor, jobId, { resolveUnknown: ledgerExecutionUnknownResolver({ ledger, receipts, jobs }) })`, or `runtime.resolveUnknown(actor, jobId, resolver)` for items already `execution-unknown`. The resolver re-runs the `decision.receipt` verifier with the recorded digest as `expected.digest` and resolves only on `present` / `digest-match` when the D03 receipt still has that digest. `state-match`, `absent`, `unknown`, a missing ledger record or a failing resolver leave the item `execution-unknown`. 3. A resolved item becomes `succeeded` or `review` with its result digest, and its latest attempt becomes `succeeded` with the receipt digest and `resolution: { method: 'effect-ledger', effectId, reason: 'digest-match', receiptDigest }`. This is the only contract transition out of `execution-unknown`; `assertJobTransition` rejects it unless the resolution's receipt digest equals the attempt's, nothing else in the item changes, and the job state stays the same. The job state is not rewritten: a reconciled `failed` job stays `failed` with the resolved item inside it.
The resolver only reads: it never dispatches, replays or re-evaluates. Without `resolveUnknown`, `reconcile` is exactly the default reconciliation. Evidence: `test/conformance/decision-v1/job-effect-resolver.test.ts` (resolver off, digest-match promotion without re-dispatch, mismatched/absent/unknown evidence, rejected `state-match` resolutions, and the contract gate).
Deployed host authentication, provider credentials, distributed multi-host capacity, collector traces and backup/key-destruction qualification are deployment evidence not established by these offline gates; they are tracked as live qualification in #2688. The HTTP handler remains unmounted until its host supplies those authorities.
Different item subjects must never be merged into a shared-state request; eligible same-subject question batching belongs inside a single item and its existing admission and receipt flow. Before enabling jobs, mount the opt-in handler behind a real authentication and secret-key service, implement the host-only admission/queue and D10 deletion callbacks, bind admitted provider execution to the protected input and durable D03/D04 receipt materialization, and qualify isolation, recovery, saturation and accounting under deployed infrastructure. Until then this remains an offline foundation, not a production asynchronous jobs feature.