Warden plan
tier: needs-model requires: [needs-postgres]
Acceptance sentence. ALCE citation recall and precision of the deployed pipeline's wiki bodies against their cited signals are measured, recorded, and persisted as the pre-registered floor the artifact engine's renders (CR-24) must not come in below.
Register: ~/.studio/master.withrobin.ai/project/rfc.one/criteria-register.md (status there: PENDING OWNER ACCEPTANCE; accepted by the 2026-08-31 proxy screen).
### CR-25 — ALCE citation-fidelity floor (current pipeline) - Statement: ALCE citation recall and precision of current wiki bodies against their cited signals, pre-registered as the citation-fidelity floor the engine's renders must not come in below. - Metric: ALCE citation recall and citation precision, NLI-computed with the 20% human check. - Dataset/inputs: Wiki bodies and their cited signals as stored in the deployed system (
wikis.content;SIGNAL_CITED_BY_WIKIedges plus per-section citation declarations — verified in code at the 2026-08-31 proxy screen); an NLI model. - Threshold: None on the measurement itself — it establishes the floor; the engine's renders (CR-24) must clear it, and the render-time ALCE gate (§7 stage iv) uses it as the pre-registered floor. - Drop condition: n/a (floor measurement). - Source: Companion §11.3 (runnable-today floor form), §11.7. - Tier: runnable — the deployed system stores wiki bodies and cited signals; no new architecture. Pre-registered to run first (§11.7).
The floor is real and recorded: wiki bodies with per-section citation_declarations exist in the corpus, and an NLI model judges each declared section against its cited signals. This is a measurement, not a bar — the plan gates only on the measurement completing over the register's dataset: every declaration in the corpus, every citation. A capped run (the smoke-budget defaults below) records the same observations and judgments but emits the verdict as an advisory skip and persists no floor — the register's dataset is "wiki bodies and their cited signals as stored in the deployed system", and a deterministic subsample of it is not that dataset. Only a full-coverage run lands the numbers in warden state (cr25_alce_recall_floor, cr25_alce_precision_floor) where the future CR-24 plan reads its non-inferiority floor.
Statement unit and judgment rules (pinned so the 20% human check audits the same thing the model judged):
sectionAnchor (or the anchor with a trailing -N dedupe suffix stripped — same resolution as 16-corpus-evals), up to the next heading of any depth.runs/cr25-alce-judgments-<runid>.jsonl (wiki, anchor, kind, verdict, premise/hypothesis sha256) — the sampling frame for the register's 20% human check, which is out-of-band by definition (a human cannot run inside this plan).None of these exist on a fresh box — all are operator setup, and run.sh refuses to run any plan at all (exit 3) until the first is done:
~/.config/robin/warden-secrets.sh — .warden/run.sh provisions the env file from .warden/env/ci-env.template.sh plus this machine-local secrets file, fatally, before any plan runs. First-time setup (from SEQUENCE.md): mkdir -p ~/.config/robin && cp .warden/env/secrets.local.example.sh ~/.config/robin/warden-secrets.sh && chmod 600 ~/.config/robin/warden-secrets.sh, then fill in the values.greenlight-pg container on :5433 — not auto-started; recreate commands are in warden.config.sh.psql, jq, curl, sha256sum resolving in the shell that invokes run.sh — psql is not on PATH in a plain profile shell on the dev box; invoke from the project's nix devshell (nix develop) or install a Postgres client.robin_ci in the same invocation: bash .warden/run.sh --destructive 16-corpus-evals 26-cr-25. A default all-plans run does NOT satisfy this — 18-issue-337-entity-domain-inheritance sorts between 16 and this plan and resets robin_ci (DROP SCHEMA public CASCADE), so the fixture 16 restored is gone before this plan runs. (This plan itself is read-only against robin_ci.)OPENROUTER_API_KEY in ~/.config/robin/warden-secrets.sh — the NLI judge is a real model call (a stubbed/CI-fake key makes this plan skip as pending, same convention as 06-socrates). Model override: WARDEN_NLI_MODEL (default anthropic/claude-haiku-4.5); sample size override: WARDEN_CR25_MAX_DECLS (default 12 declarations, citations capped at 6 per declaration — ≤ ~160 small-model calls worst case). The defaults are a smoke budget: the floor only binds and persists when WARDEN_CR25_MAX_DECLS is at least the corpus's declaration count (step 1 reports it) and no declaration cites more than 6 signals; a capped run is advisory.set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
NLI_MODEL="${WARDEN_NLI_MODEL:-anthropic/claude-haiku-4.5}"
CR25_DATA=0
CR25_LIVE=0
# Empty output means psql itself failed (command missing, greenlight-pg down,
# or schema absent) — an infra outage, not a corpus with 0 declarations. The
# headline verdict assertion is emitted in every branch so a pending floor is
# always visible in the run summary.
DECL_COUNT=$(psql "$DATABASE_URL" -tAc "
select count(*)
from wikis w, jsonb_array_elements(w.citation_declarations) d
where w.deleted_at is null and jsonb_array_length(d->'signalIds') > 0;") || DECL_COUNT=
if [ -z "${DECL_COUNT:-}" ]; then
warden_skip "CR-25 — corpus carries wiki bodies with citation declarations" "robin_ci unreachable (psql failed — greenlight-pg down, psql not on PATH, or schema absent); corpus state unknown, see raw log"
warden_skip "CR-25 — ALCE citation-fidelity floor measured and recorded" "pending: robin_ci unreachable — floor not measurable this run"
elif [ "$DECL_COUNT" -gt 0 ]; then
warden_pass "CR-25 — corpus carries wiki bodies with citation declarations" "$DECL_COUNT declarations with >=1 cited signalId"
CR25_DATA=1
else
warden_skip "CR-25 — corpus carries wiki bodies with citation declarations" "0 declarations in robin_ci — restore the corpus in the same invocation: bash .warden/run.sh --destructive 16-corpus-evals 26-cr-25 (a default all-plans run resets robin_ci between 16 and this plan)"
warden_skip "CR-25 — ALCE citation-fidelity floor measured and recorded" "pending: corpus not restored in this invocation — floor not measurable this run"
fi
if [ "$CR25_DATA" = "1" ]; then
PROBE_OUT=$(mktemp)
PROBE_CODE=$(curl -sS --max-time 60 -o "$PROBE_OUT" -w '%{http_code}' \
https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer ${OPENROUTER_API_KEY:?}" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg m "$NLI_MODEL" '{model:$m, temperature:0, max_tokens:2, messages:[{role:"user", content:"Reply with the single word OK."}]}')" ) || PROBE_CODE=000
if [ "$PROBE_CODE" = "200" ] && jq -e '.choices[0].message.content' "$PROBE_OUT" >/dev/null 2>&1; then
CR25_LIVE=1
else
warden_skip "CR-25 — ALCE citation-fidelity floor measured and recorded" "pending: no live OpenRouter key / NLI model on this box (probe HTTP $PROBE_CODE, model $NLI_MODEL)"
fi
rm -f "$PROBE_OUT"
fiset -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
if [ "${CR25_DATA:-0}" = "1" ] && [ "${CR25_LIVE:-0}" = "1" ]; then
MAXD="${WARDEN_CR25_MAX_DECLS:-12}"
JUDGMENTS="$WARDEN_DIR/runs/cr25-alce-judgments-$WARDEN_RUN_ID.jsonl"
SAMPLE=$(psql "$DATABASE_URL" -tAc "
select coalesce(jsonb_agg(r), '[]'::jsonb) from (
select w.lookup_key as wiki, w.content as content,
d->>'sectionAnchor' as anchor,
jsonb_array_length(d->'signalIds') as ncited,
(select coalesce(jsonb_agg(s.content order by s.lookup_key), '[]'::jsonb)
from jsonb_array_elements_text(d->'signalIds') as cited(sig)
join signals s on s.lookup_key = cited.sig
where s.deleted_at is null) as sources
from wikis w, jsonb_array_elements(w.citation_declarations) d
where w.deleted_at is null and jsonb_array_length(d->'signalIds') > 0
order by w.lookup_key, d->>'sectionAnchor'
limit $MAXD
) r;")
[ -n "${SAMPLE:-}" ] || SAMPLE='[]'
# One statement = the section under the anchored heading, up to the next
# heading of any depth. Same slug rule as 16-corpus-evals.
extract_section() { # $1=content-file $2=anchor
awk -v anchor="$2" '
function slugify(s) { s=tolower(s); gsub(/[^a-z0-9]+/,"-",s); gsub(/^-+/,"",s); gsub(/-+$/,"",s); return s }
/^#+[ \t]/ {
h=$0; sub(/^#+[ \t]+/,"",h); sub(/[ \t]+$/,"",h); slug=slugify(h)
base=anchor; sub(/-[0-9]+$/,"",base)
if (insec) exit
if (slug==anchor || slug==base) { insec=1 }
next
}
insec { print }
' "$1"
}
NLI_CALLS=0
NLI_ERRORS=0
# Sets NLI_VERDICT (SUPPORTED | NOT_SUPPORTED | ERROR) in the caller's shell
# — deliberately NOT $(...)-style, so the call counters survive.
nli() { # $1=premise $2=hypothesis
local body resp verdict
NLI_CALLS=$((NLI_CALLS + 1))
body=$(jq -n --arg p "$1" --arg h "$2" --arg m "$NLI_MODEL" '{
model: $m, temperature: 0, max_tokens: 8,
messages: [
{role: "system", content: "You are a strict NLI judge. Answer with exactly one word: SUPPORTED if the premise fully supports every factual claim in the hypothesis, otherwise NOT_SUPPORTED."},
{role: "user", content: ("Premise:\n" + $p + "\n\nHypothesis:\n" + $h)}
]}')
if ! resp=$(curl -sS --max-time 120 https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d "$body"); then
NLI_ERRORS=$((NLI_ERRORS + 1)); NLI_VERDICT=ERROR; return
fi
verdict=$(printf '%s' "$resp" | jq -r '.choices[0].message.content // empty' | tr -d '[:space:]' | tr '[:lower:]' '[:upper:]')
case "$verdict" in
SUPPORTED) NLI_VERDICT=SUPPORTED ;;
NOT_SUPPORTED|NOTSUPPORTED) NLI_VERDICT=NOT_SUPPORTED ;;
*) NLI_ERRORS=$((NLI_ERRORS + 1)); NLI_VERDICT=ERROR ;;
esac
}
record() { # $1=wiki $2=anchor $3=kind $4=index $5=verdict $6=premise $7=hypothesis
jq -cn --arg wiki "$1" --arg anchor "$2" --arg kind "$3" --arg idx "$4" \
--arg verdict "$5" --arg model "$NLI_MODEL" \
--arg psha "$(printf '%s' "$6" | sha256sum | cut -d' ' -f1)" \
--arg hsha "$(printf '%s' "$7" | sha256sum | cut -d' ' -f1)" \
'{wiki:$wiki, anchor:$anchor, kind:$kind, index:$idx, verdict:$verdict, model:$model, premise_sha256:$psha, hypothesis_sha256:$hsha}' \
>> "$JUDGMENTS"
}
DECL_TOTAL=0; DECL_JUDGED=0; DECL_SUPPORTED=0; EXTRACT_MISS=0
CIT_TOTAL=0; CIT_IRRELEVANT=0; CIT_TRUNC=0
CONTENT_FILE=$(mktemp)
trap 'rm -f "$CONTENT_FILE"' EXIT
while IFS= read -r ROW; do
DECL_TOTAL=$((DECL_TOTAL + 1))
WIKI=$(printf '%s' "$ROW" | jq -r '.wiki')
ANCHOR=$(printf '%s' "$ROW" | jq -r '.anchor')
printf '%s' "$ROW" | jq -r '.content' > "$CONTENT_FILE"
SECTION=$(extract_section "$CONTENT_FILE" "$ANCHOR")
NSRC=$(printf '%s' "$ROW" | jq -r '.sources | length')
if [ -z "${SECTION//[[:space:]]/}" ] || [ "$NSRC" = "0" ]; then
EXTRACT_MISS=$((EXTRACT_MISS + 1))
continue
fi
ALL_SOURCES=$(printf '%s' "$ROW" | jq -r '.sources[:6] | to_entries | map("[" + ((.key+1)|tostring) + "] " + .value) | join("\n\n")')
NUSE=$(printf '%s' "$ROW" | jq -r '.sources[:6] | length')
if [ "$NSRC" -gt 6 ]; then CIT_TRUNC=$((CIT_TRUNC + 1)); fi
nli "$ALL_SOURCES" "$SECTION"; RECALL_V="$NLI_VERDICT"
record "$WIKI" "$ANCHOR" recall all "$RECALL_V" "$ALL_SOURCES" "$SECTION"
if [ "$RECALL_V" = "ERROR" ]; then continue; fi
DECL_JUDGED=$((DECL_JUDGED + 1))
if [ "$RECALL_V" = "SUPPORTED" ]; then DECL_SUPPORTED=$((DECL_SUPPORTED + 1)); fi
i=0
while [ "$i" -lt "$NUSE" ]; do
SRC=$(printf '%s' "$ROW" | jq -r --argjson i "$i" '.sources[$i]')
CIT_TOTAL=$((CIT_TOTAL + 1))
nli "$SRC" "$SECTION"; ALONE_V="$NLI_VERDICT"
record "$WIKI" "$ANCHOR" precision_alone "$i" "$ALONE_V" "$SRC" "$SECTION"
if [ "$ALONE_V" = "NOT_SUPPORTED" ] && [ "$NUSE" -gt 1 ]; then
REST=$(printf '%s' "$ROW" | jq -r --argjson i "$i" '.sources[:6] | del(.[$i]) | join("\n\n")')
nli "$REST" "$SECTION"; REST_V="$NLI_VERDICT"
record "$WIKI" "$ANCHOR" precision_rest "$i" "$REST_V" "$REST" "$SECTION"
if [ "$REST_V" = "SUPPORTED" ]; then CIT_IRRELEVANT=$((CIT_IRRELEVANT + 1)); fi
fi
i=$((i + 1))
done
done < <(printf '%s' "$SAMPLE" | jq -c '.[]')
warden_observe cr25_sampled_declarations "$DECL_TOTAL" "of ${DECL_COUNT:-?} in corpus (cap $MAXD); $EXTRACT_MISS unresolved (anchor/sources), $DECL_JUDGED judged, $CIT_TRUNC citation-truncated at 6"
warden_observe cr25_nli_calls "$NLI_CALLS" "$NLI_ERRORS errors; model $NLI_MODEL, prompt cr25-nli-v1"
if [ "$DECL_TOTAL" -gt 0 ] && [ "$EXTRACT_MISS" -le $((DECL_TOTAL / 5)) ]; then
warden_pass "CR-25 — sampled declarations resolve to sections and cited signals" "$((DECL_TOTAL - EXTRACT_MISS))/$DECL_TOTAL resolved"
else
warden_fail "CR-25 — sampled declarations resolve to sections and cited signals" "$EXTRACT_MISS/$DECL_TOTAL unresolved — anchor drift or dangling signalIds; 16-corpus-evals owns that integrity gate"
fi
if [ "$DECL_JUDGED" -lt 5 ]; then
warden_skip "CR-25 — ALCE citation-fidelity floor measured and recorded" "only $DECL_JUDGED declarations judged (<5) — corpus too thin for a floor; restore the full seed"
elif [ "$NLI_CALLS" -gt 0 ] && [ "$NLI_ERRORS" -le $((NLI_CALLS / 5)) ]; then
RECALL=$(awk -v s="$DECL_SUPPORTED" -v n="$DECL_JUDGED" 'BEGIN{printf "%.4f", s/n}')
PRECISION=$(awk -v x="$CIT_IRRELEVANT" -v n="$CIT_TOTAL" 'BEGIN{ if (n==0) printf "%.4f", 1; else printf "%.4f", 1 - x/n }')
warden_observe cr25_alce_citation_recall "$RECALL" "$DECL_SUPPORTED/$DECL_JUDGED sections supported by their full citation set"
warden_observe cr25_alce_citation_precision "$PRECISION" "$CIT_IRRELEVANT/$CIT_TOTAL citations irrelevant (ALCE rule)"
warden_observe cr25_human_check_frame "$JUDGMENTS" "sample >=20% of these judgments for the register's human check (out-of-band)"
if [ "$DECL_TOTAL" -lt "${DECL_COUNT:-0}" ] || [ "$CIT_TRUNC" -gt 0 ]; then
# A deterministic prefix of the corpus is not the register's dataset —
# a truncated sample records advisory numbers but cannot become the
# binding CR-24 floor, so nothing is persisted to warden state.
warden_skip "CR-25 — ALCE citation-fidelity floor measured and recorded" "advisory only: caps truncated the register's dataset ($DECL_TOTAL/${DECL_COUNT:-?} declarations sampled, $CIT_TRUNC declaration(s) citation-truncated at 6) — floor NOT persisted; set WARDEN_CR25_MAX_DECLS >= ${DECL_COUNT:-?} and re-run to bind"
else
warden_save_state cr25_alce_recall_floor "$RECALL"
warden_save_state cr25_alce_precision_floor "$PRECISION"
warden_save_state cr25_alce_floor_provenance "run=$WARDEN_RUN_ID model=$NLI_MODEL n_decl=$DECL_JUDGED n_cit=$CIT_TOTAL"
warden_pass "CR-25 — ALCE citation-fidelity floor measured and recorded" "recall=$RECALL precision=$PRECISION over $DECL_JUDGED declarations / $CIT_TOTAL citations (full corpus: ${DECL_COUNT:-?} declarations) — the CR-24 non-inferiority floor, no threshold by design"
fi
else
warden_fail "CR-25 — ALCE citation-fidelity floor measured and recorded" "$NLI_ERRORS/$NLI_CALLS NLI calls failed (>20%) — floor not trustworthy; check OpenRouter availability and re-run"
fi
fiRead-only against robin_ci; ride it in the same targeted invocation as 16-corpus-evals (bash .warden/run.sh --destructive 16-corpus-evals 26-cr-25) so the corpus is the fixture, not a local demo — in a default all-plans run 18-issue-337 resets robin_ci between the two. The measurement lands in three places on purpose: observations (trend line across runs, every run), warden state (cr25_alce_* — the future CR-24 plan compares engine renders against these exact keys; written only by a full-coverage run, never by a capped one), and the per-judgment JSONL (the 20% human-check frame; hashes bind each judgment to the exact premise/hypothesis text reconstructable from the DB). The NLI prompt is versioned cr25-nli-v1 in the observations — change the prompt, bump the version, and expect the floor to move: compare floors only within a prompt version and (per the companion §11.3) hold the judge model fixed when CR-24 eventually compares renders against this floor.