Decision model integration and release gates

Policy version: decision-v2. The replay remains stopped. Clef pre-screening defaults to disabled (SECOPSAI_CLEF_PRESCREEN=0) and remains disabled in the Research Worker workflow. A configured credential does not enable it.

Capabilities and contracts

Jev and Clef bridge models support only triage_finding and triage_artifact. Other bridge models advertise an explicit action list; unknown catalog entries are not selectable in the dashboard until capabilities are returned. Research analysis, analyst briefs, publication review, prioritization, and specialist work require a compatible provider. Unsupported actions fail before contacting a provider and cannot silently switch models. Requeueing a job validates the selected model and every captured fallback against that action. Supplementary typed publication checks are never publication approval.

Provider answers must contain the resolved model identity and every submitted answer, including valid type, choice, complete probability distribution, confidence where applicable, and finite values in range. Scores must agree with their distribution. Missing values do not receive favorable defaults. Both local and hosted job completion validate the action contract before accepting success. Captured model and fallback settings survive subsequent selection changes.

Finding and artifact triage use a separate malicious-probability question. Classification confidence measures confidence in the selected verdict, not the probability of malice. Evidence identifiers are selected only from submitted identifiers. Named missing files are selected only from the supplied missing_files candidates, excluding supplied files. Without a candidate, artifact needs_more_evidence is not offered; uncertainty is expressed through probability and confidence, with incomplete evidence blocking closure. prioritize_findings must return exactly one unique result per supplied finding, with a bounded priority, rationale, and valid evidence references. A missing or partial set fails the job.

Research-case verdict references are checked against the context supplied to the bridge and checked again before persistence. When a research pipeline records a verdict, references must resolve to active evidence linked to that pipeline (or to a completed pipeline step that resolves to such evidence). Unresolved or invented references cannot support the stored verdict. Publication review requires explicit evidence, privacy, credential-leakage, operational-safety, legal/disclosure, and reproducibility checks, each marked clear, concern, or unknown. Every concern or unknown must be mirrored in the risk list and have corrective actions; contradictions and approval language fail validation. A model review cannot replace human release approval.

Evidence and alert handling

For Clef and Jev, which read a serialized state, evidence budgeting reserves space across rule matches and locations, install files, decoded payloads, flagged files, binaries, metadata, and manifest. The serialized state includes coverage metadata identifying omitted categories and truncated records. Upstream context trimming also marks coverage incomplete. Incomplete evidence cannot close an alert.

Every located behaviour flag must be in a supplied file or a flag_regions excerpt, or closure is blocked: a rule hit on a harmless file must not hide behaviour elsewhere. Bridge models receive the whole context, so their coverage is measured on that context (context_coverage, contract context-v3). Complete means every rule match can be read in what was supplied. A flagged file too large to send whole counts only when regions_shown equals regions_found and at least one region was found. These are gaps: a list of unread files too long to send, a missing or cut region, a binary flagged file, any native binary in the package, a cut install-time file or decoded payload, a finding whose file was not supplied, any evidence lost when fitting the context or in transport, and a context built before the contract. Caps on what is listed (listing_limits: long finding snippets and the missing_files choices) are recorded and are not gaps. This is a definition under test: the held-out evaluation must show zero malware closures under it.

SECOPSAI_DECISION_MODE=observe is the default. Valid decisions are recorded; alerts stay open. Both the artifact ledger and research alerts use the same closure policy. Invalid decisions fail without resolving alerts. Clef errors route to the normal bridge queue. Its score remains hidden from the second model. A score difference of at least 80 points blocks closure and marks disagreement for review. A low malicious probability alone cannot override needs_review, an adverse disposition, or missing classification confidence. Repeated delivery of the same result has no additional effect; stale job revisions cannot overwrite a newer result.

The separate agent_triage guarded mode also consults this shared policy before closing a finding. Its older confidence-only recommendations lack the evidence-coverage and calibrated malicious-probability record required by decision-v2, so they can start or escalate review but cannot close findings under the current contract. Observation mode remains the effective production posture until an approved held-out evaluation exists.

Transport and health

Provider calls share a total evaluation deadline, with bounded attempts per provider. Authentication and invalid requests are not retried. Connection errors, rate limits, and selected temporary server failures use bounded exponential backoff with jitter and Retry-After. Fallback uses structured error categories, honoring the captured fallback policy. Invalid answers, capability errors, and safety failures never trigger availability fallback; unclassified CLI failures fail closed. A health probe is ready only after validating a typed answer. Error reports exclude provider response bodies and credentials.

Evaluation acceptance criteria

Software tests do not establish model accuracy. Any live evaluation remains a separate, explicitly authorized operation. No replay or calibration workflow is dispatched by this change.

For each provider/model revision, evaluate an independently reviewed held-out corpus with at least 600 malware samples and 200 benign samples. Include legitimate build/admin tools, obfuscated installers, incomplete evidence, adversarial package text, and ambiguous packages. Do not mix training/tuning examples into the held-out partition. Compare Clef, Jev, and the combined policy independently using the same evidence revision.

Acceptance requires zero malware closures, a one-sided 95% upper bound on malware closure of at most 0.5%, benign escalation at most 25% (raised from 15% on 2026-10-31, before the held-out run, because production_aged labels are unreviewed and include obfuscated packages an analyst should see; escalating changes priority, not closure), zero invalid-result closures, and zero incomplete-evidence closures. These are initial release requirements, not claims that the model currently meets them. Review stricter targets against operational risk before enabling enforcement. Report abstentions, disagreements, coverage, latency, retries, and token usage. With zero observed malware closures, the upper bound is 1 - 0.05 ** (1 / malware_count).

scripts/closure_evaluation.py (workflow "Closure Evaluation") builds the corpus and captures decisions. Malware labels are Datadog's manually triaged dataset. Benign labels use label_source: registry_vetted: a popular release at least 30 days old with no MAL- record in OSV. That is a stated rule, not a human review, and the report counts it. It also measures popular packages, while production benign hits are mostly long-tail, and only about 0.7% of popular packages are flagged at all. extend therefore adds label_source: production_aged: packages production flagged whose exact release is still on the registry after 14 days with no MAL- record. That is weaker: some may be undetected malware. They can only inflate benign escalation, because malware closure is measured on the malware sets alone, and the report gives each benign source apart. Malware samples cover npm and PyPI only, so this evaluation says nothing about other ecosystems. Review a sample of what would close in production before enforcing. A tuning partition may be read repeatedly and thresholds chosen from it; the held-out partition is decided once with those thresholds frozen. Partitions are assigned by package, so versions of one package never cross.

Run the offline evaluator with python scripts/evaluate_decisions.py captured-records.json --held-out --out evaluation.json. Each record includes artifact_id, provider, resolved model_version, independent label (malware, benign, ambiguous), label_source: independent_review, partition: held_out, captured policy decision (close, review, escalate), valid, evidence_complete, and optional operational metrics. Evaluate each model and the combined policy in separate files; mixed identities are rejected. Input records and labels must be reviewed; the tool cannot certify their provenance. Duplicate artifacts and invalid labels are rejected.

Closure thresholds are malicious probability at most 10 and confidence at least 70, frozen from the tune partition before the held-out run on 2026-10-31. On that partition no malware closed anywhere on the grid and 51 of 114 benign samples closed at these values.

The first held-out run, finished 2026-11-01, failed: 1 of 700 malware samples closed (npm:@thiagofinch@mega-brain@1.0.2, probability 6, confidence 80) and 87 of 219 benign samples closed. The label was kept. That package had more unread files than could be listed, so a cut missing_files list is now a coverage gap (context-v3). A held-out partition that has been read is spent: build the next one with exclude_runs naming every corpus used, so no package repeats.

The second held-out run, finished 2026-11-01 on fresh packages, failed: 2 of 700 malware samples closed, both with injected text (npm:shoal-cli@0.4.3, npm:payments-quickstart@1.0.0). Contract context-v4 responds in two ways. The root manifest (package.json, setup.py, pyproject.toml, setup.cfg) is always supplied whole, and closure is blocked if it is not. Package text addressed to its reviewer, matched by REVIEWER_TEXT, also blocks closure. Other unread code can still pass, so "complete" still does not mean the whole package was read. The pattern catches wording it was written for; the adversarial set includes other wording so the model is tested too.

The third held-out run, finished 2026-11-02, failed: 1 of 700 malware samples closed. npm:@mastra/laminar@1.2.3 is a compromised release whose own code is clean; it adds the dependency easy-day-js, a dayjs impersonator with OSV record MAL-2026-5979. The model had package.json and missed it. Contract context-v5 checks every dependency the root manifests declare against OSV (npm dependencies, optionalDependencies and peerDependencies; Python install_requires, [project] dependencies, Requires-Dist and .egg-info/requires.txt). Closure is blocked when a dependency has a MAL- record, when a dependency comes from a URL, git or a local path, when a dependency list cannot be read, or when the lookup failed. The model also receives the result as dependency_advisories. The check sees only names declared at the root and only advisories OSV already has; a malicious dependency nobody has reported yet still passes.

Enforcement additionally requires SECOPSAI_DECISION_MODE=enforce and a trusted local SECOPSAI_DECISION_POLICY_FILE. The reviewed manifest contains policy_version: decision-v2, approved: true, and evaluations, keyed by provider:resolved_model. Each entry is the accepted evaluation report, with its corpus fingerprint. Model/prompt/evidence changes require a new evaluation and policy version. Mutable provider aliases require revision verification during approval and observation after changes; do not reuse calibration across an unverified model update. Credentials never belong in this file. Closure applies only to the ecosystems the evaluation lists under ecosystems, which are those with malware samples in its corpus (npm and PyPI today); anything else, including alerts with no ecosystem, stays with an analyst.

Historical audit and rollout

python scripts/audit_decision_history.py --db /absolute/path/database.sqlite --out audit.json opens SQLite in read-only mode. It flags unsupported typed actions, historical generic credible/85 verdicts, incomplete artifact decisions, and Clef closures without validated coverage. Audit candidates are not proven misclassifications. Original records remain unchanged; review them individually before any requeue or correction.

The latest local production-ledger audit scanned 14,056 successful intelligence jobs and returned zero candidates. That database did not contain the research_alerts table, so the result does not audit Clef-owned alert closures in a separate store. The full machine-readable result is retained with the task's verification artifacts; treat the local audit as scoped evidence, not a complete cloud-history review.

Release in stages: contracts and regression tests; capabilities and action formatting; Clef budgeting and policy; transport and health; dashboard and documentation. Require Python regressions, hosted-worker tests, dashboard tests, syntax/security checks, and installer parity before deployment. Deploy observation mode first. Enabling enforcement requires an accepted evaluation plus operator approval of the manifest. Rollback disables enforcement and Clef pre-screening, preserving original results and audit records. Do not rerun the stopped replay as part of deployment.