Warden plan
tier: needs-model requires: [needs-postgres]
Acceptance sentence. The live extraction pipeline's completeness is measured on real Entries: pooled proposition recall ≥ 0.8 and a positional uniformity bound within 0.10 — once the two-annotator proposition fixture is provisioned — with the precision guard held pending until span-writing lands (the register's own caveat; each run reports the persist path's span-writing and the corpus's span-bearing rows as two separate facts).
Register: ~/.studio/master.withrobin.ai/project/rfc.one/criteria-register.md (status there: PENDING OWNER ACCEPTANCE; accepted by the 2026-08-31 proxy screen).
### CR-20 — Entry→Signal extraction completeness - Statement: Extraction recovers the atomic content of an Entry uniformly across position (the eval the anchor's §10 misses; bounds the confounder under every anchor experiment). - Metric: Primary: pooled proposition recall (proposition recovered iff ≥1 extracted Signal entails it, NLI-judged with 20% human check), cluster-bootstrap intervals over Entries. Guard (exempt from Holm per the protocol supplement, tested in the harm direction): extraction precision — fraction of extracted Signals entailed by their recorded source span (hallucination guard). Sole secondary (Holm is the identity): positional uniformity — one-sided 95% bound on first-minus-fourth-quartile recall difference, cluster-bootstrapped over Entries. Also: Signals per Entry per band, for the ledger. - Dataset/inputs: 30 Entries per pilot workspace (90 pooled) sampled from real captures across confident/triage size bands, ≥half dense (≥1,000 words); two annotators decompose to atomic propositions at Chen et al. (2024) granularity, κ reported; an NLI model. - Threshold: Pooled proposition recall ≥0.8; extraction precision ≥0.95 (guard); positional uniformity bound within 0.10. - Drop condition: None — Entry is a primitive and extraction is not droppable; this eval bounds it. Redesign: recall <0.8 with uniform position → revisit prompt and refusal thresholds; uniformity bound beyond 0.10 → per-section multi-pass extraction, extra cost charged to the ledger. - Source: Companion §11.1 (contract stated in §3.1; the historical positional-discard failure, ~45% of dense notes, motivates the uniformity clause). - Tier: runnable — entries, signals, and capture reports (confident/triage bands) are live; needs annotators and an NLI model, no new architecture. Pre-registered to run first (§11.7). Caveat, re-verified in code 2026-08-31:
signals.provenance.spanis declared (server/src/db/schema.ts:584) but the extraction persist path writes no span (packages/agent/src/stages/persist.ts), so the precision guard's input ("recorded source span") is unobtainable until span-writing lands — which §3.1's extraction contract already demands of the extractor — or entailment is respecified against the full Entry payload. The recall and uniformity clauses run today.
The component under test is real (step 1): entries with capture reports (extracted/dropped/band on raw_sources.source_metadata), signals with extraction provenance, and the register's span caveat checked live on both of its halves — does persist.ts write a span, and do any rows carry one — so the precision guard activates by data, not by plan edit, the moment span-bearing rows exist, and a code-side landing over a pre-span corpus is named as such instead of being read as the caveat standing. Step 2 probes for a live NLI model (same convention as 26-cr-25). Step 3 is the criterion's verdict: pooled proposition recall, the cluster-bootstrapped positional-uniformity bound, and (spans permitting) the precision guard, scored against the two-annotator proposition fixture below.
Why the verdict is PENDING today (missing fixture, not missing machinery): the criterion's dataset — 90 pooled Entries decomposed to atomic propositions by two annotators at Chen et al. (2024) granularity — does not exist anywhere yet. No repo fixture, warden fixture, or seed-corpus artifact carries adjudicated proposition decompositions. Step 3 therefore carries a skip guard: it reports the verdict as pending (skip, not fail) until the fixture lands, and the fully implemented scoring harness activates with no plan edit the moment it does.
Path: $WARDEN_CR20_PROPS, default ~/.config/robin/warden-cr20-propositions.jsonl (machine-local like the secrets file — annotation data stays outside every git repository). One JSON object per adjudicated proposition:
{"entry_id":"<entries.lookup_key>","quartile":1,"proposition":"<atomic proposition text>"}quartile — 1..4, which quarter of the Entry's text the proposition's source material sits in (annotator-recorded at decomposition time; the uniformity clause compares quartile 1 against quartile 4).25-cr-18's gold labels).Scoring definition (so two engineers cannot disagree):
provenance.span (read as 0-based [start,end) character offsets; if span-writing lands 1-based the guard's numbers will say so loudly), hypothesis = the Signal's content; precision = SUPPORTED fraction, bar ≥0.95, tested in the harm direction, exempt from Holm per the protocol supplement.runs/cr20-judgments-<runid>.jsonl — the sampling frame for the register's 20% human check (out-of-band by definition).None of these exist on a fresh box — all are operator setup, and run.sh refuses to run any plan at all (exit 3) until the first is done:
~/.config/robin/warden-secrets.sh — .warden/run.sh provisions the env file from .warden/env/ci-env.template.sh plus this machine-local secrets file, fatally, before any plan runs. First-time setup (from SEQUENCE.md): mkdir -p ~/.config/robin && cp .warden/env/secrets.local.example.sh ~/.config/robin/warden-secrets.sh && chmod 600 ~/.config/robin/warden-secrets.sh, then fill in the values.greenlight-pg container on :5433 — not auto-started; recreate commands are in warden.config.sh.psql, jq, curl, awk, sha256sum resolving in the shell that invokes run.sh — psql is not on PATH in a plain profile shell on the dev box; invoke from the project's nix devshell (nix develop) or install a Postgres client.robin_ci in the same invocation: bash .warden/run.sh --destructive 16-corpus-evals 27-cr-20. A default all-plans run does NOT satisfy this — 18-issue-337-entity-domain-inheritance sorts between 16 and this plan and resets robin_ci (DROP SCHEMA public CASCADE), so the fixture 16 restored is gone before this plan runs. (This plan itself is read-only against robin_ci.)OPENROUTER_API_KEY in ~/.config/robin/warden-secrets.sh — the NLI judge is a real model call. Model override: WARDEN_NLI_MODEL (default anthropic/claude-haiku-4.5). Budget caps: WARDEN_CR20_MAX_ENTRIES (default 5), WARDEN_CR20_MAX_PROPS per entry (default 10); signals per Entry capped at 8. Defaults are a smoke budget — the real 90-entry run raises the caps explicitly.set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
# Capture report with confident/triage bands — CR-20's sampling frame.
if grep -qF "band: 'confident' | 'triage' | 'over-ceiling'" server/src/db/schema.ts \
&& grep -qF "band: 'confident' | 'triage' | 'over-ceiling'" packages/agent/src/stages/index.ts; then
warden_pass "CR-20 — capture report with confident/triage bands present" "schema + stages/index.ts both carry the band vocabulary"
else
warden_fail "CR-20 — capture report with confident/triage bands present" "band vocabulary left schema.ts or stages/index.ts — CR-20's size-band sampling frame is gone"
fi
# Extraction provenance declared, span included (the guard's schema-side half).
if grep -qF "span?: { start: number; end: number }" server/src/db/schema.ts; then
warden_pass "CR-20 — signals.provenance declares span (schema side)"
else
warden_fail "CR-20 — signals.provenance declares span (schema side)" "span left the provenance type in server/src/db/schema.ts — the register caveat's terms changed; re-verify CR-20's guard spec"
fi
# The register caveat, checked live on both halves: does the persist path
# write a span (code), and do any rows carry one (data)? The precision guard
# in step 3 keys on the row count; the code side is asserted here so a
# span-writing persist path over a pre-span corpus is named, not misread as
# the caveat standing. Empty SPAN_ROWS means psql itself failed — that is an
# infra outage, and nothing about the caveat has been verified this run.
SPAN_ROWS=$(psql "$DATABASE_URL" -tAc "select count(*) from signals where provenance ? 'span';") || SPAN_ROWS=
SPAN_IN_CODE=0
grep -qE "^\s*span:" packages/agent/src/stages/persist.ts && SPAN_IN_CODE=1
warden_observe cr20_span_bearing_signals "${SPAN_ROWS:-unknown}" "signals with provenance.span in robin_ci (persist.ts writes span: $SPAN_IN_CODE)"
if [ -z "${SPAN_ROWS:-}" ]; then
warden_skip "CR-20 — precision-guard input status" "robin_ci unreachable (psql failed — greenlight-pg down, psql not on PATH, or schema absent) — span-bearing row count unknown; persist.ts writes span: $SPAN_IN_CODE"
elif [ "$SPAN_IN_CODE" -eq 0 ] && [ "$SPAN_ROWS" -eq 0 ]; then
warden_pass "CR-20 — precision-guard input status" "register caveat stands: span declared in schema, persist.ts writes none, 0 span-bearing rows — guard pending; recall and uniformity run today"
elif [ "$SPAN_IN_CODE" -eq 1 ] && [ "$SPAN_ROWS" -eq 0 ]; then
warden_pass "CR-20 — precision-guard input status" "persist.ts now writes span but robin_ci carries 0 span-bearing rows (corpus predates the change) — guard stays pending until a span-bearing corpus is restored"
elif [ "$SPAN_IN_CODE" -eq 0 ]; then
warden_pass "CR-20 — precision-guard input status" "$SPAN_ROWS span-bearing rows but persist.ts no longer writes span — guard runs in step 3 against the existing rows; the caveat's terms have changed, re-verify CR-20's guard spec"
else
warden_pass "CR-20 — precision-guard input status" "span-writing landed: persist.ts writes span and $SPAN_ROWS rows carry one — the precision guard activates in step 3"
fi
# Corpus probe: entries with signals, dense entries, band distribution.
# Empty EWS = psql failed (infra), not an empty corpus.
EWS=$(psql "$DATABASE_URL" -tAc "select count(distinct entry_id) from signals where entry_id is not null and deleted_at is null;") || EWS=
DENSE=$(psql "$DATABASE_URL" -tAc "select count(*) from raw_sources where deleted_at is null and array_length(regexp_split_to_array(content, '\s+'), 1) >= 1000;") || DENSE=
BANDS=$(psql "$DATABASE_URL" -tAc "select coalesce(string_agg(band || '=' || n, ', '), 'none') from (select source_metadata->'capture'->>'band' as band, count(*) as n from raw_sources where source_metadata ? 'capture' group by 1 order by 1) b;") || BANDS=
if [ -z "${EWS:-}" ]; then
warden_skip "CR-20 — corpus carries entries with extracted signals" "robin_ci unreachable (psql failed); corpus state unknown, see raw log"
else
warden_observe cr20_entries_with_signals "$EWS"
warden_observe cr20_dense_entries "${DENSE:-0}" ">=1000 words"
warden_observe cr20_capture_band_distribution "${BANDS:-none}"
if [ "$EWS" -gt 0 ]; then
warden_pass "CR-20 — corpus carries entries with extracted signals" "$EWS entries with >=1 signal; ${DENSE:-0} dense"
else
warden_skip "CR-20 — corpus carries entries with extracted signals" "0 entries with signals in robin_ci — restore the corpus in the same invocation: bash .warden/run.sh --destructive 16-corpus-evals 27-cr-20"
fi
fiset -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
NLI_MODEL="${WARDEN_NLI_MODEL:-anthropic/claude-haiku-4.5}"
CR20_LIVE=0
PROBE_OUT=$(mktemp)
PROBE_CODE=$(curl -sS --max-time 60 -o "$PROBE_OUT" -w '%{http_code}' \
https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer ${OPENROUTER_API_KEY:?}" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg m "$NLI_MODEL" '{model:$m, temperature:0, max_tokens:2, messages:[{role:"user", content:"Reply with the single word OK."}]}')" ) || PROBE_CODE=000
if [ "$PROBE_CODE" = "200" ] && jq -e '.choices[0].message.content' "$PROBE_OUT" >/dev/null 2>&1; then
CR20_LIVE=1
fi
rm -f "$PROBE_OUT"
warden_observe cr20_nli_probe "$PROBE_CODE" "model $NLI_MODEL, live=$CR20_LIVE"set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
NLI_MODEL="${WARDEN_NLI_MODEL:-anthropic/claude-haiku-4.5}"
PROPS="${WARDEN_CR20_PROPS:-$HOME/.config/robin/warden-cr20-propositions.jsonl}"
JUDGMENTS="$WARDEN_DIR/runs/cr20-judgments-$WARDEN_RUN_ID.jsonl"
NLI_CALLS=0
NLI_ERRORS=0
# Sets NLI_VERDICT (SUPPORTED | NOT_SUPPORTED | ERROR) in the caller's shell —
# deliberately NOT $(...)-style, so the call counters survive.
nli() { # $1=premise $2=hypothesis
local body resp verdict
NLI_CALLS=$((NLI_CALLS + 1))
body=$(jq -n --arg p "$1" --arg h "$2" --arg m "$NLI_MODEL" '{
model: $m, temperature: 0, max_tokens: 8,
messages: [
{role: "system", content: "You are a strict NLI judge. Answer with exactly one word: SUPPORTED if the premise fully supports every factual claim in the hypothesis, otherwise NOT_SUPPORTED."},
{role: "user", content: ("Premise:\n" + $p + "\n\nHypothesis:\n" + $h)}
]}')
if ! resp=$(curl -sS --max-time 120 https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d "$body"); then
NLI_ERRORS=$((NLI_ERRORS + 1)); NLI_VERDICT=ERROR; return
fi
verdict=$(printf '%s' "$resp" | jq -r '.choices[0].message.content // empty' | tr -d '[:space:]' | tr '[:lower:]' '[:upper:]')
case "$verdict" in
SUPPORTED) NLI_VERDICT=SUPPORTED ;;
NOT_SUPPORTED|NOTSUPPORTED) NLI_VERDICT=NOT_SUPPORTED ;;
*) NLI_ERRORS=$((NLI_ERRORS + 1)); NLI_VERDICT=ERROR ;;
esac
}
record() { # $1=entry $2=quartile $3=kind $4=verdict $5=premise $6=hypothesis
jq -cn --arg entry "$1" --arg quartile "$2" --arg kind "$3" --arg verdict "$4" \
--arg model "$NLI_MODEL" \
--arg psha "$(printf '%s' "$5" | sha256sum | cut -d' ' -f1)" \
--arg hsha "$(printf '%s' "$6" | sha256sum | cut -d' ' -f1)" \
'{entry:$entry, quartile:$quartile, kind:$kind, verdict:$verdict, model:$model, premise_sha256:$psha, hypothesis_sha256:$hsha}' \
>> "$JUDGMENTS"
}
# --- Precision guard: independent of the fixture; keyed on span-bearing rows ---
if [ -z "${SPAN_ROWS:-}" ]; then
warden_skip "CR-20 — extraction precision >= 0.95 (hallucination guard)" "robin_ci unreachable in step 1 (psql failed) — span status unknown; guard not measurable this run"
elif [ "$SPAN_ROWS" -eq 0 ]; then
warden_skip "CR-20 — extraction precision >= 0.95 (hallucination guard)" "pending: 0 span-bearing signals (persist.ts writes span: ${SPAN_IN_CODE:-0}, see step 1) — guard activates when span-bearing rows exist or the guard is respecified against the full Entry payload"
elif [ "${CR20_LIVE:-0}" != "1" ]; then
warden_skip "CR-20 — extraction precision >= 0.95 (hallucination guard)" "pending: no live NLI model on this box"
else
GSAMPLE=$(psql "$DATABASE_URL" -tAc "
select coalesce(jsonb_agg(r), '[]'::jsonb) from (
select s.lookup_key as sig, s.content as hyp,
substring(e.content from ((s.provenance->'span'->>'start')::int + 1)
for ((s.provenance->'span'->>'end')::int - (s.provenance->'span'->>'start')::int)) as premise
from signals s
join raw_sources e on e.lookup_key = s.entry_id
where s.provenance ? 'span' and s.deleted_at is null
order by s.lookup_key
limit 20
) r;")
[ -n "${GSAMPLE:-}" ] || GSAMPLE='[]'
G_JUDGED=0; G_SUPPORTED=0
while IFS= read -r GROW; do
PREMISE=$(printf '%s' "$GROW" | jq -r '.premise // empty')
HYP=$(printf '%s' "$GROW" | jq -r '.hyp')
[ -n "${PREMISE//[[:space:]]/}" ] || continue
nli "$PREMISE" "$HYP"
record "$(printf '%s' "$GROW" | jq -r '.sig')" "-" precision_span "$NLI_VERDICT" "$PREMISE" "$HYP"
[ "$NLI_VERDICT" = "ERROR" ] && continue
G_JUDGED=$((G_JUDGED + 1))
[ "$NLI_VERDICT" = "SUPPORTED" ] && G_SUPPORTED=$((G_SUPPORTED + 1))
done < <(printf '%s' "$GSAMPLE" | jq -c '.[]')
if [ "$G_JUDGED" -lt 20 ]; then
warden_skip "CR-20 — extraction precision >= 0.95 (hallucination guard)" "underpowered: only $G_JUDGED span-bearing signals judged (<20); supported=$G_SUPPORTED, advisory only"
else
GP=$(awk -v s="$G_SUPPORTED" -v n="$G_JUDGED" 'BEGIN{printf "%.4f", s/n}')
warden_observe cr20_extraction_precision "$GP" "$G_SUPPORTED/$G_JUDGED span-entailed signals"
if [ "$(awk -v v="$GP" 'BEGIN{print (v >= 0.95)}')" = "1" ]; then
warden_pass "CR-20 — extraction precision >= 0.95 (hallucination guard)" "precision=$GP over $G_JUDGED span-bearing signals"
else
warden_fail "CR-20 — extraction precision >= 0.95 (hallucination guard)" "precision=$GP < 0.95 — signals not entailed by their recorded spans (harm direction; guard exempt from Holm)"
fi
fi
fi
# --- Recall + uniformity: gated on the proposition fixture ---
if [ ! -s "$PROPS" ]; then
warden_skip "CR-20 — pooled proposition recall >= 0.8" "pending: two-annotator proposition fixture not provisioned at $PROPS (fixture contract in plan body)"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "pending: same missing proposition fixture"
elif [ -z "${EWS:-}" ]; then
warden_skip "CR-20 — pooled proposition recall >= 0.8" "robin_ci unreachable in step 1 (psql failed) — verdict not measurable this run"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "same: robin_ci unreachable"
elif [ "$EWS" -eq 0 ]; then
warden_skip "CR-20 — pooled proposition recall >= 0.8" "corpus not restored in this invocation (0 entries with signals) — run: bash .warden/run.sh --destructive 16-corpus-evals 27-cr-20; a default all-plans run resets robin_ci between 16 and this plan"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "same: corpus not restored in this invocation"
elif [ "${CR20_LIVE:-0}" != "1" ]; then
warden_skip "CR-20 — pooled proposition recall >= 0.8" "pending: no live NLI model on this box (probe failed, model $NLI_MODEL)"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "pending: no live NLI model on this box"
elif jq -r '.entry_id' "$PROPS" | grep -qvE '^[A-Za-z0-9_-]+$'; then
# Ids feed a SQL literal — refuse anything outside the nanoid alphabet.
warden_fail "CR-20 — pooled proposition recall >= 0.8" "proposition fixture carries a malformed entry_id — refusing to interpolate into SQL"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "blocked by malformed fixture"
elif jq -r '.quartile' "$PROPS" | grep -qvE '^[1-4]$'; then
warden_fail "CR-20 — pooled proposition recall >= 0.8" "proposition fixture carries a quartile outside 1..4 — fixture contract violated"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "blocked by malformed fixture"
else
MAXE="${WARDEN_CR20_MAX_ENTRIES:-5}"
MAXP="${WARDEN_CR20_MAX_PROPS:-10}"
N_PROPS=$(jq -s 'length' "$PROPS")
N_ENTRIES=$(jq -r '.entry_id' "$PROPS" | sort -u | wc -l)
IDLIST=$(jq -r '.entry_id' "$PROPS" | sort -u | paste -sd,)
# Empty output = psql failed (infra); 0 = a reachable DB holding none of
# the fixture's entries (corpus not restored). Neither is fixture drift.
DB_PRESENT=$(psql "$DATABASE_URL" -tAc "select count(*) from raw_sources where lookup_key = any(string_to_array('$IDLIST', ','));") || DB_PRESENT=
if [ -z "${DB_PRESENT:-}" ]; then
warden_skip "CR-20 — pooled proposition recall >= 0.8" "robin_ci became unreachable mid-plan (psql failed) — verdict not measurable this run"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "same: robin_ci unreachable"
elif [ "$DB_PRESENT" -eq 0 ]; then
warden_skip "CR-20 — pooled proposition recall >= 0.8" "none of the fixture's $N_ENTRIES entries present in robin_ci — corpus not restored in this invocation; run: bash .warden/run.sh --destructive 16-corpus-evals 27-cr-20"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "same: corpus not restored in this invocation"
else
MISSING=$((N_ENTRIES - DB_PRESENT))
DENSE_FIX=$(psql "$DATABASE_URL" -tAc "select count(*) from raw_sources where lookup_key = any(string_to_array('$IDLIST', ',')) and array_length(regexp_split_to_array(content, '\s+'), 1) >= 1000;")
warden_observe cr20_fixture_entries "$N_ENTRIES" "$N_PROPS propositions; $MISSING entries absent from robin_ci; $DENSE_FIX dense"
if [ "$MISSING" -gt $((N_ENTRIES / 10)) ]; then
warden_fail "CR-20 — pooled proposition recall >= 0.8" "$MISSING/$N_ENTRIES fixture entries absent from robin_ci — fixture drifted from the corpus; re-restore or re-annotate"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "blocked by fixture/corpus drift"
else
ENTRY_TABLE=$(mktemp)
HITS=0; JUDGED=0
JE=0
while IFS= read -r GROUP; do
[ "$JE" -ge "$MAXE" ] && break
EID=$(printf '%s' "$GROUP" | jq -r '.[0].entry_id')
SIGS=$(psql "$DATABASE_URL" -tAc "
select coalesce(jsonb_agg(s.content order by s.lookup_key), '[]'::jsonb)
from (select content, lookup_key from signals
where entry_id = '$EID' and deleted_at is null
order by lookup_key limit 8) s;")
[ -n "${SIGS:-}" ] || SIGS='[]'
NSIG=$(printf '%s' "$SIGS" | jq 'length')
ALLSIG=$(printf '%s' "$SIGS" | jq -r 'to_entries | map("[" + ((.key+1)|tostring) + "] " + .value) | join("\n\n")')
NP=$(printf '%s' "$GROUP" | jq 'length')
[ "$NP" -gt "$MAXP" ] && NP="$MAXP"
q1h=0; q1t=0; q4h=0; q4t=0
i=0
while [ "$i" -lt "$NP" ]; do
PROP=$(printf '%s' "$GROUP" | jq -r --argjson i "$i" '.[$i].proposition')
Q=$(printf '%s' "$GROUP" | jq -r --argjson i "$i" '.[$i].quartile')
i=$((i + 1))
VERDICT=MISS
if [ "$NSIG" -eq 0 ]; then
record "$EID" "$Q" recall_no_signals MISS "" "$PROP"
else
nli "$ALLSIG" "$PROP"; SCREEN="$NLI_VERDICT"
record "$EID" "$Q" recall_screen "$SCREEN" "$ALLSIG" "$PROP"
if [ "$SCREEN" = "ERROR" ]; then VERDICT=ERROR
elif [ "$SCREEN" = "SUPPORTED" ]; then
# The metric's own loop: >=1 single Signal must entail it.
j=0
while [ "$j" -lt "$NSIG" ]; do
SRC=$(printf '%s' "$SIGS" | jq -r --argjson j "$j" '.[$j]')
nli "$SRC" "$PROP"
record "$EID" "$Q" recall_signal "$NLI_VERDICT" "$SRC" "$PROP"
if [ "$NLI_VERDICT" = "SUPPORTED" ]; then VERDICT=HIT; break; fi
j=$((j + 1))
done
fi
fi
[ "$VERDICT" = "ERROR" ] && continue
JUDGED=$((JUDGED + 1))
H=0; [ "$VERDICT" = "HIT" ] && { HITS=$((HITS + 1)); H=1; }
if [ "$Q" = "1" ]; then q1h=$((q1h + H)); q1t=$((q1t + 1)); fi
if [ "$Q" = "4" ]; then q4h=$((q4h + H)); q4t=$((q4t + 1)); fi
done
echo "$EID $q1h $q1t $q4h $q4t" >> "$ENTRY_TABLE"
JE=$((JE + 1))
done < <(jq -sc 'group_by(.entry_id) | .[]' "$PROPS")
warden_observe cr20_nli_calls "$NLI_CALLS" "$NLI_ERRORS errors; model $NLI_MODEL, prompt cr20-nli-v1"
cp "$ENTRY_TABLE" "$WARDEN_DIR/runs/cr20-entry-quartiles-$WARDEN_RUN_ID.txt"
if [ "$JUDGED" -eq 0 ] || { [ "$NLI_CALLS" -gt 0 ] && [ "$NLI_ERRORS" -gt $((NLI_CALLS / 5)) ]; }; then
warden_fail "CR-20 — pooled proposition recall >= 0.8" "$NLI_ERRORS/$NLI_CALLS NLI calls failed or nothing judged — measurement not trustworthy; re-run"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "blocked by untrustworthy NLI pass"
else
RECALL=$(awk -v h="$HITS" -v n="$JUDGED" 'BEGIN{printf "%.4f", h/n}')
warden_observe cr20_pooled_proposition_recall "$RECALL" "$HITS/$JUDGED propositions over $JE entries"
BOOT=$(awk 'BEGIN{srand(4242)}
{n++; a[n]=$2; b[n]=$3; c[n]=$4; d[n]=$5}
END{
if (n == 0) { print "NA 0"; exit }
B=1000; v=0
for (t=1; t<=B; t++) {
h1=0; t1=0; h4=0; t4=0
for (i=1; i<=n; i++) { j=int(rand()*n)+1; h1+=a[j]; t1+=b[j]; h4+=c[j]; t4+=d[j] }
if (t1>0 && t4>0) { v++; x[v]=h1/t1 - h4/t4 }
}
if (v < 500) { print "NA", v; exit }
for (i=2; i<=v; i++) { k=x[i]; j=i-1; while (j>0 && x[j]>k) { x[j+1]=x[j]; j-- } x[j+1]=k }
idx=int(0.95*v); if (idx<1) idx=1
printf "%.4f %d\n", x[idx], v
}' "$ENTRY_TABLE")
UBOUND=$(printf '%s' "$BOOT" | awk '{print $1}')
VALID=$(printf '%s' "$BOOT" | awk '{print $2}')
warden_observe cr20_uniformity_upper_bound "$UBOUND" "one-sided 95% bound on Q1-Q4 recall diff; $VALID/1000 valid resamples, seed 4242"
warden_observe cr20_human_check_frame "$JUDGMENTS" "sample >=20% of these judgments for the register's human check (out-of-band)"
if [ "$JE" -lt 90 ] || [ "$DENSE_FIX" -lt $((N_ENTRIES / 2)) ]; then
warden_skip "CR-20 — pooled proposition recall >= 0.8" "underpowered: $JE entries judged (<90 pooled) or dense fraction $DENSE_FIX/$N_ENTRIES < half; measured recall=$RECALL, advisory only — raise WARDEN_CR20_MAX_ENTRIES and complete the fixture"
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "underpowered: same; measured bound=${UBOUND:-NA}, advisory only"
else
if [ "$(awk -v v="$RECALL" 'BEGIN{print (v >= 0.8)}')" = "1" ]; then
warden_pass "CR-20 — pooled proposition recall >= 0.8" "recall=$RECALL over $JUDGED propositions / $JE entries"
else
warden_fail "CR-20 — pooled proposition recall >= 0.8" "recall=$RECALL < 0.8 — register redesign path: revisit prompt and refusal thresholds (uniform position) or per-section multi-pass (positional)"
fi
if [ "$UBOUND" = "NA" ]; then
warden_skip "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "quartile coverage too thin: only $VALID/1000 resamples had Q1 and Q4 mass — annotate more Q1/Q4 propositions"
elif [ "$(awk -v v="$UBOUND" 'BEGIN{print (v <= 0.10)}')" = "1" ]; then
warden_pass "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "bound=$UBOUND (cluster bootstrap over $JE entries)"
else
warden_fail "CR-20 — positional uniformity bound within 0.10 (Q1 minus Q4)" "bound=$UBOUND > 0.10 — the positional-discard failure mode; register redesign: per-section multi-pass extraction, extra cost charged to the ledger"
fi
fi
fi
rm -f "$ENTRY_TABLE"
fi
fi
fiRead-only against robin_ci; ride it in the same targeted invocation as 16-corpus-evals (bash .warden/run.sh --destructive 16-corpus-evals 27-cr-20) — in a default all-plans run 18-issue-337 resets robin_ci between the two and the verdicts degrade to corpus-not-restored skips. Three verdicts, three independent gates: recall and uniformity unlock on the proposition fixture (machine-local, same class as warden-secrets.sh); the precision guard unlocks on span-bearing rows appearing in the DB — step 1 checks the caveat's two halves separately (does persist.ts write a span; do any rows carry one), so when span-writing lands the status assert names exactly which half moved and the guard arms the moment span-bearing rows exist. The concatenation screen is a cost optimization only — the per-Signal loop is the metric; keep it that way or the recall definition silently changes. The bootstrap seed (4242) is fixed so two runs on the same fixture produce the same bound; the per-entry quartile table lands in runs/cr20-entry-quartiles-<runid>.txt for offline re-analysis. NLI prompt is versioned cr20-nli-v1 in the observations — change the prompt, bump the version, and compare recalls only within a version. Span offsets are read as 0-based [start,end); if span-writing lands with different semantics the guard's premise extraction is the one line to fix.