Issue triage shadow pilot

The issue triage pilot is an experimental, default-off D25 pack for Jev-backed classification and duplicate reranking. It is implemented only for offline and shadow evaluation. It does not file, edit, label, assign, comment on, close, or merge tracker issues, and it cannot bypass issue-planner approval.

Implemented offline

• `DecisionIssueTriagePilotPack.v1` defines closed taxonomies for issue type,

project area, urgency, state completeness, clarification need, and duplicate outcomes. The duplicate outcome set is the deterministic shortlist plus explicit `none`.

• `validateIssueTriageBatchSubject()` rejects native batches that mix issue IDs.

Batching is only valid for independent questions about one canonical issue subject.

• `deterministicIssueDuplicateCandidates()` records every duplicate candidate

with generator identity, rank, score, source digest, and query digest. A model response naming an unlisted issue ID, a candidate created after the replayed issue, or a rank that differs from the deterministic list fails validation.

• Candidates are point-in-time inputs: each carries `revisions` and only the

latest revision observed at or before the triaged issue's `createdAt` is scored or shown. A candidate with no such revision is excluded. Current-state fields (`title`, `body`, `labels`, `state`, and so on) on a candidate are refused, so a later edit such as "closed as duplicate of ..." cannot leak. Candidate labels and state are never model-visible. The triaged issue's own title and body are taken as supplied; replay corpora must provide them as created.

• `projectIssueTriageModelState()` routes the model-visible replay state through

the D10 projection boundary, allowlists metadata, and omits final labels, final duplicate decisions, resolution, and close data. Title, body, author, every allowlisted metadata string, and candidate titles pass through the shared `redactText()` helper from `src/governance/redaction.ts` (PEM private keys including `ENCRYPTED`, RSA, EC, DSA, OpenSSH and PGP headers, Stripe `sk_live_`/`sk_test_`/`rk_live_` keys, `Authorization`/`Bearer` values, `ghp_`/`gho_`/`github_pat_`, `sk-`, `xox?-`, `AKIA` keys, JSON/YAML/`key=value` password, secret and token fields, and encoded secrets) plus issue-triage patterns for JWTs and `AKIA`/`ASIA` AWS keys. The assignment rule is deliberately broad: ordinary prose such as "token accounting" is also redacted. An allowlisted metadata value that is not a string, number, boolean or null is refused.

• `validateIssueTriageModelResponse()` requires provider usage receipts.

Caller `usage` totals, Jev and fallback call counts, and the cache flag must reconcile with them. Compatibility is resolved through `CalibrationRegistry` when a registry request is supplied; a registry result other than `allow`, or alias drift, disables accepted scoring. With `acceptance.calibration: required`, a response without a registry pin is never accepted.

• `runIssueTriageShadow()` returns only a shadow artifact with

`actionAuthorization: not-authorized` and `trackerMutations: 0`. The runtime takes no tracker client and imports no tracker module; a test walks its import graph to keep it that way.

• `DecisionIssueTriageEvaluationManifest.v1` preregisters minimum support,

insufficient-slice behavior, promotion thresholds, confidence interval method and level, sample rules, the digests of the frozen tuning/calibration/test splits, and a positive benefit requirement before holdout access. It also pins the calibration evidence a report may score against: the canonical digests of the calibration registry content (artifacts, relations, alias history) and the compatibility policy, plus the rule every per-sample compatibility request must satisfy (requested alias, runId bound to the sample ID, calibration artifact, not-before instant). An all-null pin allows no registry: a report that supplies one is rejected.

• `buildIssueTriageEvaluationReport()` scores only the preregistered test split,

and only with the pilot pack the manifest pins: the pack's id, version and canonical digest must equal `manifest.pilotPack`, and the report records that pin. It scores only with the calibration evidence the manifest pins, verified before scoring: a swapped registry, policy or per-sample request is rejected, and a supplied registry with no pin (or a pin with no supplied registry) is rejected. Baseline responses are checked against the same taxonomy and closed response fields as cascade responses. It never trusts a caller's acceptance verdict or compatibility pin: it re-validates each cascade response and recomputes acceptance with the same rule as the shadow runtime, resolving compatibility through the supplied `CalibrationRegistry` requests. It refuses samples whose sample, baseline and cascade issue IDs differ, whose issue type or area label is outside the taxonomy, whose slice IDs were not preregistered, or whose cascade duplicate answer does not match the sample's deterministic lineage (ID and rank) before computing top-k and nDCG. It reports class counts, macro and per-class precision/recall/F1, urgency ordinal error (from `evaluateOrdinalHeldout`), completeness and clarification metrics, duplicate recall/nDCG/top-k and ranking concordance (from `evaluateRankingHeldout`), false-duplicate rate, `none` recall, per-slice metrics, a selective-risk and coverage summary (from `evaluateBinaryHeldout`), review load, override rate, latency, receipt-derived tokens and cost, retry/fallback rate, cache hits, Jev and fallback call counts, and baseline-versus-cascade economics.

Promotion gates

Every gate reads the preregistered manifest. Any finding yields `HOLD`.

Gate	Rule
Sample support	Total samples at least `minimumTotalSamples`. Each slice needs `max(minimumSupport, minimumPerSliceSamples)` samples or it is `insufficient`. An insufficient `suppress-small-n` slice reports neither its count nor its metrics; an insufficient `report-supported` slice reports both.
Confidence interval	Only `wilson` at an integer-basis-point level strictly between 50% and 99.99% is supported. `bootstrap` and `exact` are recorded as `confidence-interval-unsupported`. Each one-sided bound below comes from the two-sided interval, so it has one-sided error (1 - level) / 2.
Quality non-inferiority	Per-issue paired outcomes (baseline correct vs cascade correct) go to `pairedBinaryDifferenceInterval` (Newcombe method 10), then `pairedNonInferiority`. The margin is cascade minus baseline and must be `<= 0`; for example, `-0.02` allows the cascade to be two points worse. A positive margin would be a superiority test and is rejected when the manifest is validated.
False-auto rate	Wilson upper bound of the error rate among accepted samples must not exceed `maximumFalseAutoRate`. No accepted samples is `false-auto-insufficient`.
Accepted coverage	Wilson lower bound of coverage must be at least `minimumAcceptedCoverage`.
False duplicates	Wilson upper bound over `none` cases must not exceed `maximumFalseDuplicateRate`. An evaluation set with no `none` cases cannot promote.
Duplicate support and recall	At least `minimumDuplicateSamples` labeled duplicate cases, and the Wilson lower bound of duplicate recall must be at least `minimumDuplicateRecall`.
Usage	Every arm's totals, call counts and cache flag must reconcile with its provider receipts, or the report records `usage-unreconciled` and token and cost comparisons become unknown.
Benefit	Computed only on the preregistered metric (`reviewer-time` or `total-task-token-cost`), from reconciled receipts that include fallback calls and cache reads. An unknown value is `benefit-insufficient`; tokens never stand in for an unknown cost.
Calibration	With `acceptance.calibration: required`, a sample is accepted only if its registry request resolves to `allow` for its alias and served model; a report built without registry requests for every sample is `calibration-required-unverified`. The registry, policy and per-sample requests must match the manifest's calibration pin before scoring.
Integrity	The upstream eval-integrity `PROMOTE`/`HOLD`/`ROLLBACK` can be preserved or tightened, never upgraded.

`validateIssueTriageEvaluationReport(report, { inputs, trustedManifestDigest })` accepts a report only when the inputs' manifest matches the separately trusted digest, the inputs' pack matches the manifest's pack pin, and rebuilding the report from those inputs gives canonically identical JSON. An edited, spread-copied or relaxed-manifest report is rejected.

• Alias/model or uncertainty-profile incompatibility disables accepted shadow

scoring and records a defer/drift event while preserving the raw suggestion.

• `applyIssueTriagePilot(false, previous, shadow)` returns the previous workflow

object unchanged and does not call the shadow runner. When enabled it returns the previous result even if the shadow runner or the artifact recorder throws or rejects, synchronously or asynchronously. A failed success record is retried once as a failure artifact; a failed failure record is dropped.

The schemas are registered in `schemas/catalog/domains/decision.json` with experimental stability and are exported through `aiwg/decision`.

Pending inputs

These criteria are scaffolded but not claimed complete in this offline change:

• Live Jev calls against approved synthetic issues.
• A real time-sliced historical corpus, immutable manifest, leakage review,

labeling guide, annotator/adjudication record, and held-out labels.

• Human reviewer judgments for correctness/usefulness and reviewer-time deltas.
• Production tracker wiring and rollout approval for any later advisory display.
• Representative token, latency, cost, override, and reviewer-time measurements

from the final holdout.

Until those inputs exist, no production or advisory promotion claim is made. The offline report generator can return `PROMOTE` only for a complete, preregistered evaluation report that satisfies every threshold; any live or advisory mode still requires a separate approved decision using the manifest and final holdout report.