Warden plan

18 - issue #337 entity/client domain inheritance for entity-scoped signals

← eval suite index

tier: destructive requires: [needs-postgres]

18 - issue #337 entity/client domain inheritance for entity-scoped signals

What it proves

Issue #337: signals extracted from an entity-scoped entry (a client call) mostly refer to the client in the first person and never name it, so content-only domain classification left them undomained — 47 of 61 signals in the reporter's live workspace. The shipped fix (#338 design, PRs #339/ #341/#343, merged into canary) is a three-part machinery, not a threshold tweak:

  1. Entry-level provenance on every signal. persist.ts stamps provenance.sourceTitle (the entry's title) and provenance.sourceEntities (canonical names behind the entry's people mentions) on every child signal at extraction time — derived, never LLM-authored text.
  2. Nudge, never verdict, in the classifier prompt. domain-classification spec v2 renders an optional [SOURCE PROVENANCE] block and an explicit rule: provenance may lift a signal that already has some content fit, but zero content affinity + provenance alone must NOT cross the assignment threshold. Both the worker link-job path and the domain-reclassify path read the block fresh off the signal row (not carried in job payloads), so retries and reclassification see the same nudge.
  3. Explicit-attach domain inheritance, independent of content scoring. Citing a signal into a domain-scoped wiki (SIGNAL_CITED_BY_WIKI) stamps the wiki's own domain membership(s) onto the signal as domain_signals rows with source='wiki-inheritance' — idempotent, never clobbering an existing membership, and immune to the reclassify worker's stale-delete (scoped to source='classifier'). This is the path that actually recovers the bulk of "zero content affinity" signals the issue describes: they were never going to cross a content threshold, nudge or not, and the design explicitly rejects making them.
  4. Nightly backfill. A pure-SQL scheduled worker derives the provenance block onto signals that predate this feature (idempotent on provenance->>'sourceTitle' IS NULL), so the fix reaches the reporter's existing 61-signal workspace without re-extraction.

This plan's fast static/dbtest steps pin all four pieces as a regression floor. What it explicitly does NOT prove — see "Stays manual" below — is whether a real (non-mocked) LLM, given the nudge, now actually classifies the ambiguous first-person fragments from the issue's own examples ("we usually try to use...", "the client's point of view...") into the client domain. That is a live-model judgment call the shipped acceptance dbtest does not exercise (it uses an honest-but-mechanical regex stand-in, by design — see Notes to the requesting agent).

Prerequisites

Step 1: reset to a clean pgvector schema (shared DB floor)

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

if psql "$DATABASE_URL" -q -X -c "DROP SCHEMA IF EXISTS public CASCADE; DROP SCHEMA IF EXISTS drizzle CASCADE; CREATE SCHEMA public; CREATE EXTENSION IF NOT EXISTS vector;" >/dev/null 2>&1; then
  warden_pass "reset robin_ci to a clean public schema with the vector extension"
else
  warden_fail "could not reset robin_ci schema — is greenlight-pg (:5433) up with pgvector?"
fi

Step 2: the provenance/nudge/inheritance machinery exists (static)

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"

# EDI-1: signals.provenance carries the entry-level block (type-only widening,
# no migration — jsonb column already existed).
SC=server/src/db/schema.ts
if grep -q "provenance: jsonb('provenance')" "$SC"; then
  warden_pass "signals.provenance column present in schema.ts"
else
  warden_fail "signals.provenance column missing from schema.ts — the provenance write has no target"
fi

# EDI-2: persist.ts stamps sourceTitle/sourceEntities on every child signal.
PST=packages/agent/src/stages/persist.ts
if grep -q "sourceTitle" "$PST" && grep -q "sourceEntities" "$PST"; then
  warden_pass "persist.ts stamps provenance.sourceTitle/sourceEntities at extraction time (#338)"
else
  warden_fail "persist.ts no longer stamps entry-level provenance — capture-time regression (#338)"
fi

# EDI-3: the domain-classification prompt is v2 with the SOURCE PROVENANCE
# nudge-never-verdict rule (rule 7) and renders sourceTitle/sourceEntities.
SPEC=packages/shared/src/prompts/specs/domain-classification.yaml
if grep -q "^version: 2" "$SPEC" \
   && grep -qi "SOURCE PROVENANCE" "$SPEC" \
   && grep -q "must NOT be assigned to it" "$SPEC" \
   && grep -q "sourceTitle" "$SPEC" && grep -q "sourceEntities" "$SPEC"; then
  warden_pass "domain-classification spec is v2 with the SOURCE PROVENANCE nudge-never-verdict rule"
else
  warden_fail "domain-classification spec regressed — v2/SOURCE PROVENANCE/nudge-never-verdict rule missing"
fi

# EDI-4: both classification call sites (capture-time link job, and
# reclassify) thread sourceProvenance through, read fresh off the row.
WK=server/src/queue/worker.ts
RW=server/src/queue/domain-reclassify-worker.ts
if grep -q "sourceProvenance" "$WK" && grep -q "sourceProvenance" "$RW"; then
  warden_pass "both the link-job worker and the domain-reclassify worker thread sourceProvenance into classification"
else
  warden_fail "a classification call site stopped passing sourceProvenance — reclassify or capture-time path regressed"
fi

# EDI-5: wiki-attach domain inheritance exists and is wired into every attach
# path (MCP handlers, HTTP wikis routes, regen citation processing).
IWD=server/src/core/citations/inherit-wiki-domains.ts
if [ -f "$IWD" ] \
   && grep -q "inheritWikiDomains" server/src/mcp/handlers.ts \
   && grep -q "inheritWikiDomains" server/src/modules/wikis/routes.ts \
   && grep -q "inheritWikiDomains" server/src/core/citations/process-citations.ts; then
  warden_pass "inheritWikiDomains exists and is wired into MCP, HTTP, and regen citation attach paths"
else
  warden_fail "wiki-attach domain inheritance is missing or dropped from an attach path (MCP/HTTP/regen)"
fi

# EDI-6: the reclassify worker's stale-delete is scoped to source='classifier'
# so inherited memberships survive reclassification.
if grep -q "source.*classifier" "$RW"; then
  warden_pass "domain-reclassify-worker scopes its stale-delete to source='classifier' (inherited rows survive)"
else
  warden_fail "domain-reclassify-worker's stale-delete no longer scopes by source — risks deleting wiki-inherited domain_signals rows"
fi

# EDI-7: the nightly backfill worker exists, is idempotent on the NULL
# sourceTitle predicate, and is registered on the scheduler (with its kill
# switch).
BF=server/src/queue/signal-provenance-backfill-worker.ts
SCH=server/src/queue/scheduler.ts
if [ -f "$BF" ] && grep -q "sourceTitle.*IS NULL\|IS NULL" "$BF" \
   && grep -q "ENABLE_SIGNAL_PROVENANCE_BACKFILL" "$SCH" \
   && grep -q "signal-provenance-backfill" "$SCH"; then
  warden_pass "signal-provenance-backfill worker exists, idempotent on NULL sourceTitle, registered on the scheduler cron"
else
  warden_fail "signal-provenance-backfill worker missing, non-idempotent, or dropped from the scheduler"
fi

Step 3: the #338 acceptance fixture — client-naming signals classify, generic ones don't, provenance-alone never crosses the threshold

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

# Proves, against a real schema (LLM mocked, but honestly derived from the
# RENDERED prompt — see the acceptance test's own doc comment): a client
# check-in entry's ~10 extracted signals all carry the backfilled provenance
# block; the classifier prompt visibly carries [SOURCE PROVENANCE] +
# the client's name for every one of them; the 3 signals that name the
# client in their own content land in the client domain; the 7 generic ones
# do not, provenance notwithstanding (the nudge-never-verdict guardrail,
# exercised end to end rather than at the prompt-string level alone).
if pnpm --filter @robin/server exec vitest run \
     src/queue/signal-provenance-acceptance.dbtest.test.ts \
     >/tmp/warden-337-acceptance.log 2>&1; then
  warden_pass "#338 acceptance fixture passes: client-naming signals classify, generic ones don't, against a live DB"
else
  tail -30 /tmp/warden-337-acceptance.log
  warden_fail "#338 acceptance fixture FAILED — see /tmp/warden-337-acceptance.log"
fi

Step 4: backfill worker + wiki-attach inheritance, against a live DB

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

# Proves: the backfill worker derives sourceTitle from the entry and
# sourceEntities from live (non-orphaned) SIGNAL_MENTIONS_PERSON edges, is
# idempotent (re-run leaves already-enriched rows untouched), and skips rows
# with no entry.
if pnpm --filter @robin/server exec vitest run \
     src/queue/signal-provenance-backfill-worker.dbtest.test.ts \
     >/tmp/warden-337-backfill.log 2>&1; then
  warden_pass "signal-provenance-backfill worker dbtest passes against a live DB"
else
  tail -25 /tmp/warden-337-backfill.log
  warden_fail "signal-provenance-backfill worker dbtest FAILED — see /tmp/warden-337-backfill.log"
fi

# Proves: attaching a signal to a domain-scoped wiki stamps the wiki's own
# domain(s) onto the signal as domain_signals(source='wiki-inheritance'),
# idempotently, never clobbering an existing membership, skipping
# soft-deleted domains — and that reclassification does not delete it.
if pnpm --filter @robin/server exec vitest run \
     src/core/citations/inherit-wiki-domains.dbtest.test.ts \
     src/queue/domain-reclassify-worker.dbtest.test.ts \
     >/tmp/warden-337-inherit.log 2>&1; then
  warden_pass "wiki-attach domain inheritance dbtest passes, and survives domain reclassification"
else
  tail -25 /tmp/warden-337-inherit.log
  warden_fail "wiki-attach domain inheritance or its reclassify-survival guarantee FAILED — see /tmp/warden-337-inherit.log"
fi

Step 5: negative case — a signal with zero content affinity, cited into a client wiki, versus one that is never cited

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

PST=packages/agent/src/stages/persist.ts
# EDI-N1: sourceEntities is built from resolved canonical names, never raw
# mention surface forms or LLM free text — re-check the specific derivation
# comment/guard, since a silent regression here would let unverified LLM
# text leak into the classifier's auxiliary context.
if grep -q "never raw mention surface forms\|canonical" "$PST"; then
  warden_pass "sourceEntities derives from resolved canonical names, not raw LLM mention text"
else
  warden_fail "cannot confirm sourceEntities still derives from canonical names only — re-check persist.ts by hand"
fi

# EDI-N2: a signal genuinely about the client but never cited into any wiki,
# and with zero content affinity, does NOT silently gain a domain membership
# purely from being backfilled with provenance. This is the acceptance
# fixture's own negative assertion (7 generic signals, 0 of which land in
# the domain) — re-stated here as its own named check so a future change
# that weakens rule 7 fails a predicate whose name says exactly what broke.
if grep -A2 "must NOT be assigned to it" packages/shared/src/prompts/specs/domain-classification.yaml \
     | grep -q "purely because"; then
  warden_pass "the nudge-never-verdict rule's exact wording (zero-affinity + provenance-naming ≠ assignment) is intact"
else
  warden_fail "the nudge-never-verdict rule's wording changed or weakened — re-review domain-classification.yaml rule 7"
fi

Shape (note for the next author)

This plan is the dedicated #337/#338 regression floor for entity-domain inheritance. It deliberately does not re-run the general classification prompt-rendering unit tests (packages/shared/src/__tests__/prompts/ domain-classification.test.ts, packages/agent/src/__tests__/ domain-classify.test.ts) or the entity-extraction unit tests — those are hermetic and belong to the fast pre-push gate, not a DB-backed warden plan; Step 2's static greps are this plan's cheap substitute for "did the wiring survive," and Steps 3-4 are the live-DB floor.

Stays manual / not asserted here — this is the residual gap: