Warden plan

21 - issue #366 provenance repair (the #338 nudge carries real inputs)

tier: destructive requires: [needs-postgres]

21 - issue #366 provenance repair (the #338 nudge carries real inputs)

What it proves

Issue #366 reports that the #338 provenance nudge, live since #361, moved client-domain tagging not at all (23 → 23 tagged signals, ~32%). Both nudge inputs are broken, and the design anchors on the wrong noun:

Metadata fault — sourceTitle is the raw upload header. The entry row is pre-created before extraction and titled from the raw payload; worker.ts passes that pre-create title in as entryTitle, and persist.ts stamps provenance.sourceTitle from it — while the same persist call writes the entry row's own title from input.primaryTopic (the clean topic signal-extract just produced). One call, two sources, diverged. Every signal's nudge therefore reads [Channel: …][Uploaded by: …][Context: RAW TRANSCRIPT …] instead of a title that often literally names the client.
Reliability fault — sourceEntities is empty and the failure is silent. entity-extract runs in Promise.allSettled alongside signal-extract and is fail-open: on rejection runExtraction logs a classify/failed event and hands persist ?? new Map(), so persist stamps nothing — indistinguishable from "this entry names no people". Live pipeline_events show the rejections are structured-output schema validation: candidates: Required, matched.13.inferredName: Required, matched.28.sourceSpan: Required — i.e. the model's payload does not satisfy peopleExtractionSchema, whose buckets and per-entry inferredName / sourceSpan are all required while people-extraction.yaml only spells those fields out for the candidates bucket and never demands an empty bucket be emitted. Person rows still land (insert side-effects commit before the parse throws), which is why the failure is invisible.
Design fault — people are the wrong anchor. sourceEntities is built only from input.peopleMap/personNames, and entity-extract only ever creates entity_type='person'. Even working perfectly it can carry the call's attendees, never the client org the domain is keyed on. The repair ships the third input #338's own "open questions" named — matched domain-net terms (sourceTerms), derived from the workspace's persisted knowledge_domains.name set matched against the entry, never LLM free text.
Soft weight stays soft. #338's guardrail — nudge, never verdict — is unchanged and must extend to the new input: zero content affinity plus a matched term must still not cross the threshold, or this becomes folder-filing by source, the thing #338 explicitly rejected.

This plan is the acceptance floor for that repair. It runs without a live model on purpose: the acceptance path reuses the existing mocked-LLM dbtest harness (signal-provenance-acceptance.dbtest.test.ts), whose mock derives its verdict from the rendered prompt's content section and deliberately ignores the provenance block. That makes every predicate here about what the machinery writes and feeds, which is exactly what #366 says is broken — and leaves the live-model judgment call explicitly manual (see "Stays manual").

Baseline expectation: on an unfixed tree every load-bearing step below is RED. PR #369 is a claim-only draft (0 additions at the time of writing), so a red run of this plan against today's canary is the honest before picture, not a harness fault.

Prerequisites

greenlight-pg :5433 with the vector extension. .warden/run.sh provisions the env file from .warden/env/ci-env.template.sh + ~/.config/robin/warden-secrets.sh.
PROJECT_ROOT = a tree carrying the #366 repair (the #369 branch, or canary once merged). No live server, no OPENROUTER key, no source auth.sh — this plan has no HTTP surface.
pnpm, psql, grep on PATH.

Step 1: reset to a clean pgvector schema (shared DB floor)

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

if psql "$DATABASE_URL" -q -X -c "DROP SCHEMA IF EXISTS public CASCADE; DROP SCHEMA IF EXISTS drizzle CASCADE; CREATE SCHEMA public; CREATE EXTENSION IF NOT EXISTS vector;" >/dev/null 2>&1; then
  warden_pass "reset robin_ci to a clean public schema with the vector extension"
else
  warden_fail "could not reset robin_ci schema — is greenlight-pg (:5433) up with pgvector?"
fi

Step 2: the repair's seams exist (static, corroborating — fails fast and loud on a stale tree)

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"

PST=packages/agent/src/stages/persist.ts

# PR-S1: sourceTitle is stamped from the CLEAN topic, and the raw
# pre-create entryTitle is no longer its source. Both halves asserted —
# adding primaryTopic while leaving the old read in place is the drift
# this catches.
if grep -Eq "sourceTitle: *[^,}]*primaryTopic" "$PST" \
   && ! grep -Eq "sourceTitle: *input\.entryTitle" "$PST"; then
  warden_pass "persist stamps provenance.sourceTitle from the clean primaryTopic, not the raw pre-create entryTitle (#366 fault 3)"
else
  warden_fail "persist still stamps sourceTitle from input.entryTitle (or no longer from primaryTopic) — the raw '[Channel: …]' header is back in the nudge"
fi

# PR-S2: the third nudge input exists in the write path.
if grep -q "sourceTerms" "$PST"; then
  warden_pass "persist stamps provenance.sourceTerms — the matched domain-net terms input (#338 open question, #366 fault 1)"
else
  warden_fail "persist writes no sourceTerms — the nudge still has no anchor that can name the client account"
fi

# PR-S3: fail-open is no longer silent. persist already RECEIVES
# entityExtractionStatus ('completed' | 'failed') from runExtraction and
# today drops it on the floor; the repair must land it on a surface a
# reader can distinguish from "this entry names nobody". Either admissible
# surface counts (a provenance marker, or an entity-extract failure event
# rendered by /admin/diagnose) — Step 4 is the load-bearing proof.
if grep -q "entityExtractionStatus" "$PST" \
   && grep -Eq "entityExtractionStatus[^)]*(provenance|sourceEntitiesUnavailable|entityExtractFailed)" "$PST"; then
  warden_pass "persist consumes entityExtractionStatus and marks the row when entity-extract failed (fail-open made loud)"
elif grep -q "substage: 'entity-extract'" packages/agent/src/stages/index.ts \
     && grep -q "entity-extract" server/src/modules/admin/routes/admin.ts; then
  warden_pass "entity-extract failure is surfaced on the /admin/diagnose pipeline_events path (alternative admissible surface)"
else
  warden_fail "an entity-extract failure is still indistinguishable from 'no entities' on every surface — the silent-empty defect (#366 fault 2) is unrepaired"
fi

# PR-S4: the schema-vs-prompt mismatch behind the live validation errors.
# The repair may loosen the schema (buckets defaulted, per-entry fields
# tolerant) and/or tighten the prompt; the live errors name all three
# fields, so both halves are checked.
SCH=packages/shared/src/prompts/specs/people-extraction.schema.ts
YML=packages/shared/src/prompts/specs/people-extraction.yaml
if grep -Eq "default\(\[\]\)|optional\(\)" "$SCH" \
   && grep -Eqi "empty|even if|both buckets" "$YML"; then
  warden_pass "people-extraction schema tolerates the omissions the live model actually makes, and the prompt demands both buckets explicitly"
else
  warden_fail "people-extraction schema/prompt still disagree — 'candidates: Required' / 'matched.N.inferredName: Required' / 'matched.N.sourceSpan: Required' will keep rejecting real payloads"
fi

Step 3 (LOAD-BEARING): ingest a raw transcript WITH an upload preamble — every signal's sourceTitle is clean, entities carry attendees, terms carry the matched domain

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

# The #366 reproduction, as an extension of the existing #338 acceptance
# fixture: an entry pre-created with the reporter's own header shape
# ("[Channel: Claude]\n[Uploaded by: …]\n[Context: RAW TRANSCRIPT, PART 1 …
# <client> monitoring check-in …]"), a clean primaryTopic produced by
# signal-extract in the same job, three attendees resolved by
# entity-extract, and a workspace domain whose name the entry names.
#
# The four assertion titles below are pinned by name: a renamed or deleted
# assertion fails this step even if the file still exits 0. That is the
# drift guard remediation/02 asked for.
if pnpm --filter @robin/server exec vitest run \
     --reporter=verbose \
     src/queue/signal-provenance-acceptance.dbtest.test.ts \
     >/tmp/warden-366-acceptance.log 2>&1; then
  warden_pass "the #338 acceptance fixture (extended with the #366 upload-preamble entry) passes against a live DB"
else
  tail -40 /tmp/warden-366-acceptance.log
  warden_fail "the extended acceptance fixture FAILED — see /tmp/warden-366-acceptance.log"
fi

# PR-1 (load-bearing, NEGATIVE): the literal upload preamble appears in NO
# signal's sourceTitle. This is the assertion #366 is actually about.
if grep -qi "no signal.s sourceTitle contains the upload preamble" /tmp/warden-366-acceptance.log; then
  warden_pass "PR-1: every extracted signal's sourceTitle is free of the '[Channel:' upload preamble"
else
  warden_fail "PR-1 missing or renamed: the acceptance fixture no longer asserts that '[Channel:' appears in NO sourceTitle"
fi

# PR-2 (load-bearing, POSITIVE): the value is the clean topic-derived title,
# not merely 'not the header' (a blank or truncated title would satisfy the
# negative alone).
if grep -qi "sourceTitle is the clean primaryTopic-derived entry title" /tmp/warden-366-acceptance.log; then
  warden_pass "PR-2: every signal's sourceTitle equals the clean topic-derived entry title"
else
  warden_fail "PR-2 missing or renamed: nothing pins sourceTitle to the clean primaryTopic title"
fi

# PR-3 (load-bearing): attendees present when entity-extract succeeds.
if grep -qi "sourceEntities carries the attendee canonical names" /tmp/warden-366-acceptance.log; then
  warden_pass "PR-3: sourceEntities carries the resolved attendee canonical names on a successful entity-extract"
else
  warden_fail "PR-3 missing or renamed: sourceEntities is unasserted on the success path"
fi

# PR-4 (load-bearing): the new input carries the matched domain name.
if grep -qi "sourceTerms carries the matched domain name" /tmp/warden-366-acceptance.log; then
  warden_pass "PR-4: sourceTerms carries the workspace domain name the entry matched"
else
  warden_fail "PR-4 missing or renamed: sourceTerms is unasserted — the client anchor is not proven to reach the block"
fi

# PR-5 (corroborating, DB-wide sweep): no signal row left in robin_ci by any
# test above carries a preamble-shaped sourceTitle. Trivially true on an
# empty table, hence corroborating — its value is catching a fixture that
# writes rows through a path Step 3's in-test assertions don't cover.
BAD=$(psql "$DATABASE_URL" -tA -X -c \
  "SELECT count(*) FROM signals WHERE provenance->>'sourceTitle' LIKE '%[Channel:%' OR provenance->>'sourceTitle' LIKE '%[Uploaded by:%';" 2>/dev/null || echo ERR)
if [ "$BAD" = "0" ]; then
  warden_pass "PR-5: zero signal rows in robin_ci carry an upload-preamble sourceTitle"
elif [ "$BAD" = "ERR" ]; then
  warden_fail "PR-5: could not query signals.provenance — schema missing or DB unreachable"
else
  warden_fail "PR-5: $BAD signal rows still carry an upload-preamble sourceTitle after the acceptance run"
fi

Step 4 (LOAD-BEARING): an entity-extract failure is distinguishable from "no entities"

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

# Two ingests through the same path: one where entity-extract REJECTS
# (the live schema-validation failure), one where it RESOLVES with zero
# people. Silent-empty is the defect — the two must not produce identical
# reader-visible state. Whichever surface the repair chose (provenance
# marker or diagnose-rendered pipeline_events), the test asserts the
# distinction, and warden pins the assertion by name.
if pnpm --filter @robin/server exec vitest run \
     --reporter=verbose \
     src/queue/worker-entity-extract-failure.dbtest.test.ts \
     >/tmp/warden-366-failopen.log 2>&1; then
  warden_pass "entity-extract fail-open visibility dbtest passes against a live DB"
else
  tail -40 /tmp/warden-366-failopen.log
  warden_fail "entity-extract fail-open visibility dbtest FAILED (or does not exist) — see /tmp/warden-366-failopen.log"
fi

# PR-6 (load-bearing): the failure is visible at all.
if grep -qi "entity-extract failure is visible to a reader" /tmp/warden-366-failopen.log; then
  warden_pass "PR-6: a rejected entity-extract leaves a reader-visible failure record, not just empty maps"
else
  warden_fail "PR-6 missing or renamed: nothing asserts an entity-extract failure is visible after the fact"
fi

# PR-7 (load-bearing, the actual #366 defect): visible AND distinguishable.
if grep -qi "distinguishable from an entry that names no people" /tmp/warden-366-failopen.log; then
  warden_pass "PR-7: failure state is distinguishable from the legitimate zero-entities state (silent-empty pinned absent)"
else
  warden_fail "PR-7 missing or renamed: 'entity-extract failed' and 'entry names nobody' may still be indistinguishable"
fi

# PR-8 (corroborating): the failure does not abort capture. Fail-open is
# the intended behavior; #366 asks for it to be LOUD, not removed.
if grep -qi "signal extraction still completes when entity-extract fails" /tmp/warden-366-failopen.log; then
  warden_pass "PR-8: making fail-open loud did not make it fail-closed — signals still persist"
else
  warden_fail "PR-8 missing or renamed: nothing guards against the repair turning fail-open into a pipeline abort"
fi

Step 5 (LOAD-BEARING): backfill enriches a pre-existing signal with sourceTerms, and a second run is a no-op

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

# The reporter's ~23 already-tagged signals predate this repair; the whole
# point of a re-derivable block is that they get the new input without
# re-extraction. Note the idempotency predicate has to CHANGE: today the
# worker matches on `provenance->>'sourceTitle' IS NULL`, which skips every
# row that already has a title and therefore can never add sourceTerms to
# it. Widening the WHERE without breaking "second run enriches zero rows"
# is the substance of this step.
if pnpm --filter @robin/server exec vitest run \
     --reporter=verbose \
     src/queue/signal-provenance-backfill-worker.dbtest.test.ts \
     >/tmp/warden-366-backfill.log 2>&1; then
  warden_pass "signal-provenance backfill dbtest passes against a live DB"
else
  tail -40 /tmp/warden-366-backfill.log
  warden_fail "signal-provenance backfill dbtest FAILED — see /tmp/warden-366-backfill.log"
fi

# PR-9 (load-bearing): a signal that already has a sourceTitle but no
# sourceTerms gets enriched — the reach-the-existing-corpus case.
if grep -qi "enriches a pre-existing signal with sourceTerms" /tmp/warden-366-backfill.log; then
  warden_pass "PR-9: backfill adds sourceTerms to signals that predate the repair, including ones already carrying a sourceTitle"
else
  warden_fail "PR-9 missing or renamed: backfill is unproven against the existing corpus (the NULL-sourceTitle WHERE would skip it)"
fi

# PR-10 (load-bearing): idempotency, stated as enriched-count zero rather
# than "no error".
if grep -qi "second backfill run enriches zero rows" /tmp/warden-366-backfill.log; then
  warden_pass "PR-10: a second backfill run is a no-op (enriched = 0), not a rewrite"
else
  warden_fail "PR-10 missing or renamed: backfill idempotency after the WHERE-clause widening is unproven"
fi

# PR-11 (corroborating, static): the widened predicate is actually in the
# SQL, so PR-9 cannot pass on a fixture that only exercises NULL rows.
if grep -Eq "sourceTerms" server/src/queue/signal-provenance-backfill-worker.ts; then
  warden_pass "PR-11: the backfill worker's own SQL references sourceTerms"
else
  warden_fail "PR-11: the backfill worker never mentions sourceTerms — PR-9 is passing on something else"
fi

Step 6: nudge-never-verdict extends to sourceTerms (corroborating — the live-model check stays manual)

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"

SPEC=packages/shared/src/prompts/specs/domain-classification.yaml

# PR-12: the rule text still forbids provenance-alone assignment, and the
# new input is inside the rule's scope rather than sitting outside it.
if grep -q "must NOT be assigned to it" "$SPEC" \
   && grep -q "sourceTerms" "$SPEC"; then
  warden_pass "PR-12: the domain-classification spec renders sourceTerms and keeps the nudge-never-verdict rule"
else
  warden_fail "PR-12: sourceTerms is not in the classifier spec, or the nudge-never-verdict rule was weakened — a matched term could become a verdict (the folder-filing failure #338 rejected)"
fi

# PR-13: the prompt-render unit gate covers the new field, so a future
# template edit that drops it reds the fast pre-push gate, not just here.
if grep -q "sourceTerms" packages/shared/src/__tests__/prompts/domain-classification.test.ts; then
  warden_pass "PR-13: the hermetic prompt-render test asserts the sourceTerms rendering"
else
  warden_fail "PR-13: no hermetic test covers sourceTerms rendering — the CI guardrail has a hole"
fi

# PR-14: the mocked-LLM acceptance fixture carries the zero-affinity
# negative for the NEW input specifically: a signal with no content
# affinity, whose entry matched a domain term, gains no domain.
if grep -qi "zero-affinity signal with a matching sourceTerm gains no domain" /tmp/warden-366-acceptance.log; then
  warden_pass "PR-14: zero content affinity + a matched sourceTerm does not cross the assignment threshold"
else
  warden_fail "PR-14 missing or renamed: the guardrail is asserted for entities/title but not for the new terms input"
fi

Shape (note for the next author)

Deliberate scope calls:

No live model. Every predicate above is about what the pipeline writes and what the prompt carries — which is precisely the #366 complaint (both inputs unusable). The plan's mock is the acceptance fixture's existing content-derived stand-in, so nothing here can be gamed by the model happening to be generous on a given day.
PR-14 is corroborating, and that is a real gap. It proves the guardrail against a mock that ignores provenance by construction — i.e. it proves the fixture is honest, not that a real model resists the term. The live version of that check already exists as the zero-affinity-control fixture in server/eval/classification/domain-classification-provenance.eval.ts, but evalite is not run by pre-merge.yml — it is a manual pnpm --filter @robin/server eval with an OpenRouter key. Cited here as corroborating, not counted as a gate.
Assertion titles are load-bearing strings. Steps 3-5 grep vitest's verbose output for named assertions instead of trusting exit 0. A renamed or deleted assertion reds the plan — the same drift class that produced remediation/02.
Static greps are pre-flight, not proof. Step 2 exists so a stale or wrong tree fails in seconds with a message that says which fault is unrepaired, before the DB-backed steps burn minutes.

Stays manual / not asserted here:

Whether the repaired nudge moves the reporter's number. #366's headline is 23 → 23 tagged signals (~32%). Nothing in this plan re-runs the reporter's two live calls or measures a tagging rate; a mocked classifier cannot. Re-ingesting the same corpus post-fix and reading the tagged-signal count is a human step (and the only one that closes the issue's actual claim).
Fault 1 is mitigated, not solved. Organizations are still never captured as entity_type='organization'; sourceTerms is a substitute anchor derived from persisted domain names. An account that has no knowledge domain, or whose domain name never appears in the entry text, still gets no anchor. The issue's "capture organisations as first-class entities" branch is out of scope for this plan.
The [Context: …] blob as a term source. The repair reads terms from the entry, so a client named only inside the upload preamble may still be matched — or, if the repair strips the preamble before matching, may be lost. Whichever way it lands, this plan does not pin it; flag it in review.
Rendering. No app-side provenance chip exists (app/src/lib/ provenance.ts covers sourceClient, the capture channel). Nothing here is agent-browser-verifiable.

Complementary unit tests (hermetic — the fast pre-push gate, not this plan)

Per fault, the tests that belong in the pre-push gate rather than a DB-backed warden plan:

A. Schema-vs-prompt (packages/shared/src/__tests__/prompts/ people-extraction.test.ts, packages/agent/src/__tests__/ entity-extract.test.ts)

A payload omitting candidates entirely parses, yielding []; same for an omitted matched. (Reproduces candidates: Required.)
A matched entry with no inferredName parses and canonicalises to the mention surface form. (Reproduces matched.13.inferredName: Required.)
A matched entry with no sourceSpan parses; the mention edge is written with an empty/absent span rather than the stage rejecting. (Reproduces matched.28.sourceSpan: Required.)
Negative — the loosening is not blanket: matched as a non-array, or an entry missing mention, still rejects.
Prompt render: the template instructs both buckets on every payload including when empty, and names the required fields for matched entries (today rule 3 spells them out for candidates only).
normalisePeopleExtraction still splits the legacy flat people array correctly after the loosening.

B. Clean-title derivation (packages/agent/src/__tests__/ persist-provenance.test.ts)

Every persisted signal's provenance.sourceTitle equals input.primaryTopic.
Given entryTitle = the reporter's literal preamble ([Channel: Claude]\n[Uploaded by: …]\n[Context: RAW TRANSCRIPT …]), no signal's sourceTitle contains [Channel:, [Uploaded by:, or [Context:.
One source of truth: the entry row's title and the signals' sourceTitle come from the same value in the same call.
Empty/whitespace primaryTopic → the key is absent, not '' (preserves the existing "keys absent, not empty" provenance contract).
input.entryTitle becoming unused does not change any other persist output (guards a drive-by signature change).

C. Fail-open visibility (packages/agent/src/__tests__/ extraction-failopen.test.ts)

entity-extract rejects → persist receives entityExtractionStatus: 'failed' and the persisted provenance carries the failure marker; emitEvent is called with substage: 'entity-extract', status: 'failed', and the original error message preserved verbatim.
entity-extract resolves with zero people → status 'completed', no failure marker, sourceEntities key absent.
The two persisted rows from (1) and (2) differ on a readable field — this is the assertion that pins the silent-empty defect absent.
Signal-extract's result is unaffected by an entity-extract rejection: signals persist, link jobs enqueue (fail-open preserved).
Signal-extract rejecting still throws (the asymmetry is intentional).

D. sourceTerms derivation + guardrail (packages/agent/src/__tests__/ persist-provenance.test.ts, packages/shared/src/__tests__/prompts/ domain-classification.test.ts, acceptance fixture)

Derivation matches the workspace's persisted knowledge_domains.name values against the entry, case-insensitively, on word boundaries, and emits the canonical domain name — never the matched surface form, never LLM text.
No match → key absent; several matches → order-stable dedup (a retried job stamps an identical block, matching the existing sourceEntities contract).
Substring guard: domain Acme does not match Acmeology / acme-corp-competitor inside the entry body.
Terms are scoped to the entry's own workspace — a same-named domain in another workspace never leaks in.
Prompt loader renders sourceTerms inside [SOURCE PROVENANCE] only when non-empty, and the nudge-never-verdict rule text covers terms explicitly (not only title/entities).
Acceptance-fixture extension: an 11th signal with zero content affinity, in an entry whose sourceTerms names the domain, ends with matchedCount: 0 and no domain_signals row.
Parity: the backfill's SQL-derived sourceTerms for an entry equals the extraction-time derivation for the same entry — one derivation rule, two call sites, asserted equal rather than written twice.