Warden plan

25 - CR-18: Domain classifier quality (Signal→Domain macro-F1)

tier: needs-postgres requires: []

25 - CR-18: Domain classifier quality (Signal→Domain macro-F1)

Acceptance sentence. The live Signal→Domain classifier machinery is present and producing assignments, and — once the two-annotator gold-label fixture is provisioned — its per-Domain precision/recall clear the register's bars (macro-F1 ≥ 0.85, no Domain recall < 0.75).

Criterion implemented (verbatim from the register)

Register: ~/.studio/master.withrobin.ai/project/rfc.one/criteria-register.md (status there: PENDING OWNER ACCEPTANCE; accepted by the 2026-08-31 proxy screen).

### CR-18 — Domain classifier quality - Statement: Signal-to-Domain classification is accurate enough to serve as the fast path's hard scope filter. - Metric: Per-Domain precision and recall of Signal-to-Domain assignment; macro-F1. - Dataset/inputs: 500 Signals per workspace with Domain labels from two annotators. Signals, Domains, and the classifier exist in the deployed system. - Threshold: Macro-F1 ≥0.85 and no Domain with recall below 0.75. - Drop condition: Any Domain below the recall bar has its description and classification logic revised and Signals reclassified before §10.1/§10.4 run on that workspace (redesign, not drop). - Source: Anchor §10.7. - Tier: runnable — the Domain classifier is live (verified in code at the 2026-08-31 proxy screen: capture-time domain-classify stage, domain_signals rows with source: 'classifier', per-domain classifierPrompt); only annotation is missing.

What it proves

The classifier under test is real: the capture-time domain-classify stage, the source: 'classifier' writes on domain_signals, and the per-domain classifierPrompt column all exist and the restored corpus carries classifier assignments (step 1). Step 2 is the criterion's verdict: it scores those assignments against a two-annotator adjudicated gold set and gates on the register's macro-F1/recall bars.

Why the verdict is PENDING today (missing fixture, not missing machinery): the criterion's dataset — 500 Signals per workspace labeled by two annotators — does not exist anywhere yet. Nothing in the repo, the warden fixtures, or the seed corpus carries human Domain labels; domain_signals.source='manual' rows would be single-curator actions, not the register's two-annotator protocol. So step 2 carries a skip guard: it reports the verdict as pending (skip, not fail) until the label fixture lands, and activates with no plan edit the moment it does. The scoring harness itself is fully implemented and runs today against any conforming fixture.

Gold-label fixture contract (what unblocks step 2)

Path: $WARDEN_CR18_LABELS, default ~/.config/robin/warden-cr18-domain-labels.jsonl (machine-local like the secrets file — annotation data stays outside every git repository). One JSON object per line:

{"signal_id":"<signals.lookup_key>","a":["domain-slug"],"b":["domain-slug"],"gold":["domain-slug"]}

a / b — the two annotators' independent Domain-slug sets for the signal.
gold — the adjudicated set (the register's shared protocol: κ reported, re-adjudication below 0.4 happens offline, before this file is written).
Verdict asserts bind only when every workspace with labeled signals carries ≥ 500 of them in robin_ci (the register's dataset spec is per workspace, not pooled — the plan groups the labeled signals by workspace_id and gates on the minimum); below that the plan computes and records everything but emits the verdict as a skip marked underpowered.

Scoring definition (so two engineers cannot disagree): prediction = the signal's domain_signals rows with source='classifier' mapped to knowledge_domains.slug. Domain universe = union of gold and predicted slugs over labeled signals. Per domain: tp/fp/fn over (signal, domain) assignment pairs; precision = tp/(tp+fp) (1 when unassigned), recall = tp/(tp+fn) (undefined when the domain has no gold instance — such domains score F1 = 0 as pure false-positive emitters and are excluded from the min-recall clause); macro-F1 = unweighted mean F1 over the universe.

Prerequisites

None of these exist on a fresh box — all four are operator setup, and run.sh refuses to run any plan at all (exit 3) until the first is done:

~/.config/robin/warden-secrets.sh — .warden/run.sh provisions the env file from .warden/env/ci-env.template.sh plus this machine-local secrets file, fatally, before any plan runs. First-time setup (from SEQUENCE.md): mkdir -p ~/.config/robin && cp .warden/env/secrets.local.example.sh ~/.config/robin/warden-secrets.sh && chmod 600 ~/.config/robin/warden-secrets.sh, then fill in the values.
The greenlight-pg container on :5433 — not auto-started; recreate commands are in warden.config.sh.
psql and jq resolving in the shell that invokes run.sh — psql is not on PATH in a plain profile shell on the dev box; invoke from the project's nix devshell (nix develop) or install a Postgres client.
The corpus restored into robin_ci in the same invocation: bash .warden/run.sh --destructive 16-corpus-evals 25-cr-18. A default all-plans run does NOT satisfy this — 18-issue-337-entity-domain-inheritance sorts between 16 and this plan and resets robin_ci (DROP SCHEMA public CASCADE), so the fixture 16 restored is gone before this plan runs. (This plan itself is read-only; it never resets robin_ci.)
Optional (activates the verdict): the gold-label fixture above.

Step 1: classifier machinery is live (code + data)

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

# The capture-time classify stage and the reclassify worker both exist and
# write source='classifier' rows — the component CR-18 scores.
STAGE=packages/agent/src/stages/domain-classify.ts
if [ -f "$STAGE" ] && grep -q "classifierPrompt" "$STAGE"; then
  warden_pass "CR-18 — capture-time domain-classify stage present" "$STAGE reads per-domain classifierPrompt"
else
  warden_fail "CR-18 — capture-time domain-classify stage missing or renamed" "expected $STAGE with classifierPrompt input"
fi

if grep -q "source: 'classifier'" server/src/queue/worker.ts \
   && grep -q "source: 'classifier'" server/src/queue/domain-reclassify-worker.ts; then
  warden_pass "CR-18 — classifier provenance writes present" "worker.ts + domain-reclassify-worker.ts stamp domain_signals.source='classifier'"
else
  warden_fail "CR-18 — classifier provenance write missing" "a queue worker no longer stamps source='classifier'; CR-18's prediction set would be unidentifiable"
fi

if grep -q "classifier_prompt" server/src/db/schema.ts \
   && grep -qE "'classifier' \| 'manual'" server/src/db/schema.ts; then
  warden_pass "CR-18 — schema carries classifier_prompt + domain_signals.source vocabulary"
else
  warden_fail "CR-18 — schema drift" "classifier_prompt column or the 'classifier'|'manual' source vocabulary left server/src/db/schema.ts"
fi

# Empty output means psql itself failed (command missing, greenlight-pg down,
# or schema absent) — that is an infra outage, not an empty corpus.
CLS_ROWS=$(psql "$DATABASE_URL" -tAc "select count(*) from domain_signals where source = 'classifier';") || CLS_ROWS=
MAN_ROWS=$(psql "$DATABASE_URL" -tAc "select count(*) from domain_signals where source = 'manual';") || MAN_ROWS=
if [ -z "${CLS_ROWS:-}" ]; then
  warden_skip "CR-18 — corpus carries live classifier assignments" "robin_ci unreachable (psql failed — greenlight-pg down, psql not on PATH, or schema absent); corpus state unknown, see raw log"
else
  warden_observe cr18_classifier_assignments "$CLS_ROWS" "domain_signals rows with source='classifier'"
  warden_observe cr18_manual_assignments "${MAN_ROWS:-0}" "domain_signals rows with source='manual' (single-curator, NOT CR-18 gold)"
  if [ "$CLS_ROWS" -gt 0 ]; then
    warden_pass "CR-18 — corpus carries live classifier assignments" "$CLS_ROWS classifier-sourced domain_signals rows in robin_ci"
  else
    warden_skip "CR-18 — corpus carries live classifier assignments" "0 classifier-sourced rows in robin_ci — restore the corpus in the same invocation: bash .warden/run.sh --destructive 16-corpus-evals 25-cr-18"
  fi
fi

Step 2: verdict — macro-F1 and per-Domain recall against the gold fixture

set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"

LABELS="${WARDEN_CR18_LABELS:-$HOME/.config/robin/warden-cr18-domain-labels.jsonl}"

if [ ! -s "$LABELS" ]; then
  warden_skip "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "pending: two-annotator gold labels not provisioned at $LABELS (fixture contract in plan body)"
  warden_skip "CR-18 — no Domain recall < 0.75" "pending: same missing gold-label fixture"
else
  # Ids feed a SQL literal — refuse anything outside the nanoid alphabet.
  if jq -r '.signal_id' "$LABELS" | grep -qvE '^[A-Za-z0-9_-]+$'; then
    warden_fail "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "label fixture carries a malformed signal_id — refusing to interpolate into SQL"
    warden_skip "CR-18 — no Domain recall < 0.75" "blocked by malformed fixture"
  else
    N=$(jq -s 'length' "$LABELS")
    IDLIST=$(jq -r '.signal_id' "$LABELS" | paste -sd,)

    # Empty output = psql itself failed (infra outage), distinct from a
    # reachable DB that simply does not hold the corpus. Neither is drift.
    DB_PRESENT=$(psql "$DATABASE_URL" -tAc "select count(*) from signals where lookup_key = any(string_to_array('$IDLIST', ','));") || DB_PRESENT=
    if [ -z "${DB_PRESENT:-}" ]; then
      warden_skip "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "robin_ci unreachable (psql failed — greenlight-pg down, psql not on PATH, or schema absent) — verdict not measurable this run"
      warden_skip "CR-18 — no Domain recall < 0.75" "same: robin_ci unreachable"
    elif [ "$DB_PRESENT" -eq 0 ] || [ "${CLS_ROWS:-0}" -eq 0 ]; then
      warden_skip "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "corpus not restored in this invocation ($DB_PRESENT/$N labeled signals present, ${CLS_ROWS:-0} classifier rows) — run: bash .warden/run.sh --destructive 16-corpus-evals 25-cr-18; a default all-plans run resets robin_ci between 16 and this plan"
      warden_skip "CR-18 — no Domain recall < 0.75" "same: corpus not restored in this invocation"
    else
    MISSING=$((N - DB_PRESENT))
    warden_observe cr18_labeled_signals "$N" "rows in gold fixture; $MISSING not present in robin_ci"

    # The register's dataset spec is 500 Signals PER WORKSPACE, not pooled —
    # bind on the thinnest labeled workspace.
    WS_STATS=$(psql "$DATABASE_URL" -tAc "
      select count(distinct workspace_id) || ' ' || coalesce(min(n), 0) from (
        select workspace_id, count(*) as n
        from signals
        where lookup_key = any(string_to_array('$IDLIST', ','))
        group by workspace_id
      ) t;") || WS_STATS='0 0'
    WS_COUNT=${WS_STATS%% *}
    WS_MIN=${WS_STATS##* }
    warden_observe cr18_min_labels_per_workspace "${WS_MIN:-0}" "labeled signals span ${WS_COUNT:-0} workspace(s); register spec: 500 per workspace"

    PRED=$(psql "$DATABASE_URL" -tAc "
      select coalesce(json_object_agg(sid, doms), '{}'::json) from (
        select ds.signal_id as sid, json_agg(kd.slug) as doms
        from domain_signals ds
        join knowledge_domains kd on kd.id = ds.domain_id
        where ds.source = 'classifier'
          and ds.signal_id = any(string_to_array('$IDLIST', ','))
        group by ds.signal_id
      ) t;")

    JQ_PROG=$(mktemp)
    trap 'rm -f "$JQ_PROG"' EXIT
    cat > "$JQ_PROG" <<'JQEOF'
def norm: (. // []) | map(tostring) | unique;
($goldrows | map({sig: .signal_id, g: (.gold|norm), a: (.a|norm), b: (.b|norm)})) as $rows
| ($rows|length) as $n
| ($rows | map(select(.a == .b)) | length) as $agree
| ([$rows[].g[]] + [$rows[] | ($pred[.sig] // [])[]] | unique) as $domains
| ($domains | map(. as $d
    | ([$rows[] | select((.g|index($d)) != null and (($pred[.sig] // [])|index($d)) != null)] | length) as $tp
    | ([$rows[] | select((.g|index($d)) != null and (($pred[.sig] // [])|index($d)) == null)] | length) as $fn
    | ([$rows[] | select((.g|index($d)) == null and (($pred[.sig] // [])|index($d)) != null)] | length) as $fp
    | {domain: $d, tp: $tp, fp: $fp, fn: $fn,
       precision: (if ($tp+$fp)==0 then 1 else (($tp*1.0)/($tp+$fp)) end),
       recall: (if ($tp+$fn)==0 then null else (($tp*1.0)/($tp+$fn)) end)}
    | . + {f1: (if .recall == null then 0
                elif (.precision+.recall)==0 then 0
                else (2*.precision*.recall)/(.precision+.recall) end)}
  )) as $per
| {n: $n,
   agreement: (if $n>0 then (($agree*1.0)/$n) else 0 end),
   domains: ($domains|length),
   macro_f1: (if ($per|length)>0 then (($per|map(.f1)|add) / ($per|length)) else 0 end),
   min_recall: ([$per[].recall | select(. != null)] | min),
   worst_domain: ([$per[] | select(.recall != null)] | sort_by(.recall) | first | .domain),
   per_domain: $per}
JQEOF
    [ -n "${PRED:-}" ] || PRED='{}'
    METRICS=$(jq -n --slurpfile goldrows "$LABELS" --argjson pred "$PRED" -f "$JQ_PROG")

    MACRO_F1=$(printf '%s' "$METRICS" | jq -r '.macro_f1')
    MIN_RECALL=$(printf '%s' "$METRICS" | jq -r '.min_recall // empty')
    WORST=$(printf '%s' "$METRICS" | jq -r '.worst_domain // "n/a"')
    AGREEMENT=$(printf '%s' "$METRICS" | jq -r '.agreement')
    NDOM=$(printf '%s' "$METRICS" | jq -r '.domains')
    printf '%s\n' "$METRICS" > "$WARDEN_DIR/runs/cr18-metrics-$WARDEN_RUN_ID.json"

    warden_observe cr18_macro_f1 "$MACRO_F1" "over $NDOM domains, $N labeled signals"
    warden_observe cr18_min_domain_recall "${MIN_RECALL:-n/a}" "worst domain: $WORST"
    warden_observe cr18_annotator_agreement "$AGREEMENT" "exact-set a==b rate (kappa reported offline per the shared protocol)"

    if [ "$MISSING" -gt $((N / 10)) ]; then
      warden_fail "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "$MISSING/$N labeled signals absent from robin_ci — label fixture drifted from the corpus; re-restore or re-annotate"
      warden_skip "CR-18 — no Domain recall < 0.75" "blocked by fixture/corpus drift"
    elif [ "${WS_MIN:-0}" -lt 500 ]; then
      warden_skip "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "underpowered: thinnest labeled workspace carries ${WS_MIN:-0} < 500 signals (register spec: 500 per workspace, ${WS_COUNT:-0} workspace(s) labeled, N=$N pooled); measured macro-F1=$MACRO_F1, advisory only"
      warden_skip "CR-18 — no Domain recall < 0.75" "underpowered: same per-workspace shortfall; measured min recall=${MIN_RECALL:-n/a} ($WORST)"
    else
      if [ "$(awk -v v="$MACRO_F1" 'BEGIN{print (v >= 0.85)}')" = "1" ]; then
        warden_pass "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "macro-F1=$MACRO_F1 over $NDOM domains, N=$N"
      else
        warden_fail "CR-18 — macro-F1 >= 0.85 (Signal→Domain classifier)" "macro-F1=$MACRO_F1 < 0.85 — see runs/cr18-metrics-$WARDEN_RUN_ID.json"
      fi
      if [ -n "${MIN_RECALL:-}" ] && [ "$(awk -v v="$MIN_RECALL" 'BEGIN{print (v >= 0.75)}')" = "1" ]; then
        warden_pass "CR-18 — no Domain recall < 0.75" "min recall=$MIN_RECALL ($WORST)"
      else
        warden_fail "CR-18 — no Domain recall < 0.75" "domain '$WORST' recall=${MIN_RECALL:-n/a} — register drop condition: revise that domain's description/classification logic and reclassify before §10.1/§10.4 run on this workspace"
      fi
    fi
    fi
  fi
fi

Shape (note for the next author)

Read-only against robin_ci — never resets it. Ride it in the same targeted invocation as 16-corpus-evals (bash .warden/run.sh --destructive 16-corpus-evals 25-cr-18): in a default all-plans run, 18-issue-337 sorts between the two and resets robin_ci, so the verdict degrades to a corpus-not-restored skip. The gold fixture is machine-local by design (same class as warden-secrets.sh); the moment it exists with ≥ 500 labels in every labeled workspace the two verdict asserts bind with no plan edit. Per-domain tp/fp/fn land in runs/cr18-metrics-<runid>.json so a red min-recall assert names the domain to redesign (the register's redesign-not-drop disposition). Do not substitute source='manual' rows for gold: they are single-curator actions outside the two-annotator protocol, and the reclassify worker treats them as immovable — using them would score the classifier against its own exemptions.