Warden plan
tier: destructive requires: [needs-model, needs-server, needs-postgres]
The Socrates sprint's DANCE surface holds against the canary code (579ef85) + a live database + a real browser: a user can Ask a domain-scoped question and get a cited answer (the loop calls search then answers with a citation); the New/DRAW and Edit flows author and refine wikis through the create_wiki / propose_edit producers; an attributed pending suggestion is accepted from /inbox — mutating wikis.content and writing an edits audit row — while a non-guardian is refused (D18/D31); and the domain Graph renders signal nodes + signal↔signal relationship edges (D33 W7).
The core constraint. There is no live OpenRouter key in-sandbox, so any real model turn 401s. The Socrates loop chooses its model at packages/agent/src/socrates/loop.ts:142 — model: input.model ?? openrouter(config.models.wikiGeneration) — so a caller-supplied LanguageModel short-circuits the network provider. runAgent- Turn (server/src/agent/run.ts:238) and handleProposeEdit (server/src/agent/propose-edit.ts) both expose this model? seam for tests. The harness owns a deterministic stub — MockLanguageModelV3 from ai/test, driven by .warden/fixtures/socrates-probe.mts — that makes a Socrates turn emit a FIXED search→cited-answer sequence with zero network. The stub lives in the harness, never in shipped product code (the rules forbid editing server/, app/, packages/).
What is browser-driven vs in-process, and WHY. The three DANCE surfaces (/socrates, /inbox, /socrates/graph) are opened in a REAL headless browser (agent-browser over the Playwright chromium at ~/.cache/ms-playwright) to prove each page boots against the running app and renders its shell. But the streamed cited answer cannot be driven end-to-end through the browser here: the shipped run-start path POST /agent/runs → startAgentRun (server/src/modules/agent/routes.ts:56, run.ts:345) constructs the run WITHOUT the model? seam, then prepareTurn calls loadOpenRouterConfig() (run.ts:204) which either throws NoOpenRouterKeyError (no key) or, with the CI sk-or-ci-fake key, makes a real OpenRouter call that 401s. There is no env switch in shipped code to inject a stub into the live server, and adding one would edit product code. So the ASK turn's behavior is proven in-process through the exact same runAgentTurn entrypoint with the deterministic model (Step 3), the browser proves the /socrates surface renders (Step 3b), and the fully-browser-driven live streamed answer is recorded as a gated manual / unverifiable-here check (Step 3c) with the precise blocker. This mirrors 05-collapse, which likewise proved the create invariant two ways (source seam + in-process replay) rather than through a flaky live HTTP call.
Edit/accept/graph need no model on the read/accept side. propose_edit's PRODUCER is asserted at the source seam + the deterministic edit path; the /inbox accept (POST /inbox/suggestions/:id/accept) and the guardian denial and the graph read (GET /domains/:id/graph) are model-free server paths, so those are driven over live HTTP with real DB assertions against a seeded pending suggestion — no stub needed there.
greenlight-pg :5433 with the vector extension, greenlight-redis :6380). See .warden/HANDOFF.md.PROJECT_ROOT points at the tree under test (the integrated canary tip at /home/me/apps/master.withrobin.ai). Warden derives PROJECT_ROOT from WARDEN_DIR/.., so invoke the suite through the tree's .warden symlink (→ ~/.studio/master.withrobin.ai/warden): bash /home/me/apps/master.withrobin.ai/.warden/run.shtsx, curl, jq, psql, and npx agent-browser (Playwright chromium) on PATH. The env file is provisioned automatically by .warden/run.sh from the tracked .warden/env/ci-env.template.sh plus the machine-local secrets file ~/.config/robin/warden-secrets.sh (mode 600, outside every git repository; override with WARDEN_SECRETS_FILE); first-time setup: mkdir -p ~/.config/robin && cp .warden/env/secrets.local.example.sh ~/.config/robin/warden-secrets.sh && chmod 600 ~/.config/robin/warden-secrets.sh.This plan RESETS the robin_ci database (both public and drizzle schemas) and lets the server replay migrations from the Socrates baseline — the CI database is disposable test infra. It boots its own server on :3100 and app on :3101 and tears both down on exit.
set -uo pipefail
source "$WARDEN_LIB/assert.sh"
source "$WARDEN_LIB/wait.sh"
source "$WARDEN_LIB/db.sh"
source "$WARDEN_LIB/browser.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
# The fixtures live under the studio-symlinked .warden (outside the worktree),
# so Node ESM can't resolve their bare imports (`ai`, `nanoid`) — resolution
# walks up from the FILE's dir, not cwd. warden_fixture copies a fixture into
# the worktree `server/` dir (a run-scoped temp name) so its imports resolve
# from server/node_modules, runs it with the server's tsx, then removes it.
warden_fixture() {
local src="$WARDEN_DIR/fixtures/$1"; shift
local tmp="server/.warden-fixture-${WARDEN_RUN_ID}-$(basename "$src")"
cp "$src" "$PROJECT_ROOT/$tmp"
( cd "$PROJECT_ROOT/server" && exec node_modules/.bin/tsx "$(basename "$tmp")" "$@" )
local rc=$?
rm -f "$PROJECT_ROOT/$tmp"
return $rc
}
export -f warden_fixture
# browser.sh's wb_count_gt0 uses `agent-browser query`, which is
# not a command in agent-browser 0.26.x (the shipped lib predates this CLI). We
# do NOT edit the shared lib (other plans depend on it) — instead this plan-local
# helper uses the real `get count <selector>` surface: true when >=1 match.
wb_count_gt0() {
local n
n="$(npx agent-browser get count "$1" 2>/dev/null | tr -dc '0-9')"
[ "${n:-0}" -ge 1 ]
}
export -f wb_count_gt0
# The Next.js app proxies its API calls to `NEXT_PUBLIC_ROBIN_API`
# (app/next.config.ts, default http://localhost:3000). It is env-driven, so we
# point it at OUR server. Boot on :3100 (NOT :3000 — a long-lived dev server may
# squat :3000; its stale schema would answer /health while our reset DB stays
# empty, silently starving the seed). Our server owns the freshly-migrated DB;
# the app on :3101 proxies to it via NEXT_PUBLIC_ROBIN_API.
export PORT=3100 SERVER_PUBLIC_URL=http://localhost:3100 WIKI_ORIGIN=http://localhost:3101
SERVER_BASE="http://localhost:3100"
APP_BASE="http://localhost:3101"
# Deterministic schema (same rationale as 05-collapse): drop public + drizzle,
# restore the vector extension, let the server's idempotent runMigrations()
# replay the chain on boot. robin_ci is disposable.
psql "$DATABASE_URL" -q -X -c "DROP SCHEMA IF EXISTS public CASCADE; DROP SCHEMA IF EXISTS drizzle CASCADE; CREATE SCHEMA public; CREATE EXTENSION IF NOT EXISTS vector;" >/dev/null 2>&1
SERVER_LOG="$WARDEN_DIR/runs/socrates-server-${WARDEN_RUN_ID}.log"
APP_LOG="$WARDEN_DIR/runs/socrates-app-${WARDEN_RUN_ID}.log"
export WARDEN_SERVER_LOG="$SERVER_LOG"
( cd server && exec node_modules/.bin/tsx src/index.ts ) > "$SERVER_LOG" 2>&1 &
SERVER_PID=$!
trap 'kill "$SERVER_PID" "${APP_PID:-}" 2>/dev/null; wait 2>/dev/null' EXIT
if warden_wait_http "$SERVER_BASE/health" 120 200; then
warden_pass "Socrates server booted + healthy on :3100 (migrations replayed)"
else
warden_halt "Socrates server did not become healthy on :3100 within 120s"
fi
# The Next.js app is a prototype UI wired to the server. Boot it against the
# server origin so the browser can open the real /socrates, /inbox and graph
# pages. A dev boot is acceptable (this suite proves the surface renders, not
# production build perf).
export NEXT_PUBLIC_ROBIN_API="$SERVER_BASE"
( cd app && exec node_modules/.bin/next dev -p 3101 ) > "$APP_LOG" 2>&1 &
APP_PID=$!
if warden_wait_http "$APP_BASE" 120 200; then
warden_pass "Socrates web app booted on :3101 (wired to the :3100 API)"
else
warden_fail "Socrates app did not become reachable on :3101 within 120s (app boot; see $APP_LOG)"
fi
set -uo pipefail
source "$WARDEN_LIB/assert.sh"
source "$WARDEN_LIB/db.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
# The server's startup provisions the INITIAL_USERNAME user + org + root
# workspace (better-auth). Seed a knowledge domain, one signal bound to it, and
# one wiki via the app's own seed entrypoint so the row shapes match the schema
# exactly (ids/keys/vector columns). The seed script is harness-owned and prints
# `SEED <key>=<value>` lines this block turns into env for later steps.
SEED_OUT="$(warden_fixture socrates-seed.mts 2>>"$WARDEN_SERVER_LOG")"
sget() { echo "$SEED_OUT" | grep -m1 "^SEED $1=" | cut -d= -f2-; }
export PROBE_USER_ID="$(sget user_id)"
export PROBE_DOMAIN_ID="$(sget domain_id)"
export PROBE_WIKI_SLUG="$(sget wiki_slug)"
export PROBE_SIGNAL_TEXT="$(sget signal_text)"
{
echo "PROBE_USER_ID=$PROBE_USER_ID"
echo "PROBE_DOMAIN_ID=$PROBE_DOMAIN_ID"
echo "PROBE_WIKI_SLUG=$PROBE_WIKI_SLUG"
} > "$WARDEN_DIR/runs/socrates-seed-${WARDEN_RUN_ID}.env"
[ -n "$PROBE_USER_ID" ] && [ -n "$PROBE_DOMAIN_ID" ] && [ -n "$PROBE_WIKI_SLUG" ] \
&& warden_pass "seeded a guardian user + domain + bound signal + wiki (ids captured)" \
|| warden_fail "seed did not produce all of user/domain/wiki ids (user=$PROBE_USER_ID domain=$PROBE_DOMAIN_ID wiki=$PROBE_WIKI_SLUG)"
# The DB carries exactly the seeded knowledge rows the DANCE flows will touch.
[ "$(warden_psql_count knowledge_domains "id = '$PROBE_DOMAIN_ID'")" = "1" ] \
&& warden_pass "the seeded knowledge domain row exists" \
|| warden_fail "seeded domain row missing"
[ "$(warden_psql_count wikis "slug = '$PROBE_WIKI_SLUG'")" = "1" ] \
&& warden_pass "the seeded wiki row exists (Edit target)" \
|| warden_fail "seeded wiki row missing"
set -uo pipefail
source "$WARDEN_LIB/assert.sh"
source "$WARDEN_LIB/browser.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
# shellcheck disable=SC1091
source "$WARDEN_DIR/runs/socrates-seed-${WARDEN_RUN_ID}.env"
export PROBE_USER_ID PROBE_DOMAIN_ID PROBE_SIGNAL_TEXT="idempotent ingest keeps signals stable"
APP_BASE="http://localhost:3101"
# 3a. Behavior — drive the REAL `runAgentTurn` entrypoint (the same code
# `POST /agent/runs` backgrounds) with the harness's deterministic model. The
# turn must complete, run >1 step (search then answer), emit answer text AND a
# citation marker. This is the ASK contract, network-free.
OUT="$(warden_fixture socrates-probe.mts 2>>"$WARDEN_SERVER_LOG")"
get() { echo "$OUT" | grep -m1 "^PROBE $1=" | cut -d= -f2-; }
[ "$(get ask_completed)" = "true" ] \
&& warden_pass "a domain-scoped Socrates turn completes over runAgentTurn (deterministic model)" \
|| warden_fail "ask turn did not complete (status/fatal — see server log): $(get fatal)"
[ "$(get ask_has_answer_text)" = "true" ] \
&& warden_pass "the turn streamed answer text into the final step (the rail's cited answer)" \
|| warden_fail "ask turn produced no answer text"
{ [ "$(get ask_steps)" != "1" ] && [ "$(get ask_has_citation)" = "true" ]; } \
&& warden_pass "the turn called search then answered WITH a citation chip marker [n] (DANCE Ask)" \
|| warden_fail "expected multi-step search→cited answer (steps=$(get ask_steps) cite=$(get ask_has_citation))"
# 3b. Surface — the real /socrates route is served and mounts an interactive
# document in a real browser. The DANCE ask rail lives BEHIND the better-auth
# session (the shell redirects an unauthenticated visitor to login), and the
# browser can't hold a session for the directly-seeded user (same limitation as
# 5c/6a). So this proves the route boots + mounts an input-bearing document
# (login OR the authed rail), and the authed-rail-specific affordance is gated.
warden_browser_open "$APP_BASE/socrates" "body" 15000
{ wb_count_gt0 "textarea, input, [contenteditable='true']"; } \
&& warden_pass "/socrates is served and mounts an interactive document in a real browser" \
|| warden_fail "/socrates served no interactive document (app boot / route — see app log)"
# 3c. Live streamed answer THROUGH the browser (POST /agent/runs → WS): GATED.
warden_skip "browser-driven live streamed answer via POST /agent/runs" \
"unverifiable-here: startAgentRun (run.ts:345) builds the run WITHOUT the model? seam, so prepareTurn→loadOpenRouterConfig (run.ts:204) makes a real OpenRouter call that 401s on the sk-or-ci-fake key. Needs EITHER a live OPENROUTER_API_KEY, or a shipped env switch to inject a stub model into the live server (forbidden: product-code edit). Behavior is proven network-free in 3a via the same runAgentTurn entrypoint."
set -uo pipefail
source "$WARDEN_LIB/assert.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# 4a. NEW/DRAW — the create_wiki producer wires the New flow (a wiki is created
# and its body proposed). Assert the canonical tool + its persistence seam.
# The D20 agent gate whitelist is the single source of the tools the agent may
# call; both DANCE producers must be listed there.
TOOLS=server/src/mcp/tool-permissions.ts
{ grep -qE "create_wiki" "$TOOLS" && grep -qE "propose_edit" "$TOOLS"; } \
&& warden_pass "the agent tool whitelist exposes create_wiki (New/DRAW) and propose_edit (Edit) producers" \
|| warden_fail "create_wiki / propose_edit not both present in the agent tool whitelist"
# 4b. EDIT — propose_edit NEVER mutates wikis.content directly; it persists a
# PENDING wiki_suggestions row (D18 accept-time application). Source seam.
PE=server/src/agent/propose-edit.ts
{ grep -qE "wikiSuggestions" "$PE" && grep -qiE "pending" "$PE" && grep -qE "model\?: LanguageModel" "$PE"; } \
&& warden_pass "propose_edit persists a pending wiki_suggestions row + honours the model? test seam (D18)" \
|| warden_fail "propose_edit no longer persists a pending suggestion via the documented seam"
# 4c. A pending suggestion attributed to the guardian is seeded for the accept
# flow (the deterministic rewrite body — no live model needed to prove accept).
# The seed helper below inserts one bound to the seeded wiki and prints its id.
# shellcheck disable=SC1091
source "$WARDEN_DIR/runs/socrates-seed-${WARDEN_RUN_ID}.env"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
SUG_OUT="$(PROBE_WIKI_SLUG="$PROBE_WIKI_SLUG" PROBE_USER_ID="$PROBE_USER_ID" warden_fixture socrates-suggestion.mts 2>>"$WARDEN_SERVER_LOG")"
SUG_ID="$(echo "$SUG_OUT" | grep -m1 '^SUG id=' | cut -d= -f2-)"
echo "SUG_ID=$SUG_ID" >> "$WARDEN_DIR/runs/socrates-seed-${WARDEN_RUN_ID}.env"
{ [ -n "$SUG_ID" ] && [ "$(warden_psql_count wiki_suggestions "id = '$SUG_ID' AND status = 'pending'")" = "1" ]; } \
&& warden_pass "an attributed PENDING suggestion is persisted against the seeded wiki (Plate suggestion source)" \
|| warden_fail "pending suggestion not persisted (id=$SUG_ID)"
set -uo pipefail
source "$WARDEN_LIB/assert.sh"
source "$WARDEN_LIB/db.sh"
source "$WARDEN_LIB/browser.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
# shellcheck disable=SC1091
source "$WARDEN_DIR/runs/socrates-seed-${WARDEN_RUN_ID}.env"
# The plan's OWN server from step 1 (:3100) — NOT :3000, which may be a
# long-lived dev server on a different tree/database.
SERVER_BASE="http://localhost:3100"
APP_BASE="http://localhost:3101"
# 5a. Surface — the /inbox page boots and renders the editor/guardian inbox.
warden_browser_open "$APP_BASE/inbox" "body" 15000
{ wb_count_gt0 "main, [role='list'], [data-inbox], article, form, input"; } \
&& warden_pass "/inbox is served and mounts the inbox document in a real browser (authed content gated by session)" \
|| warden_fail "/inbox served no document (app boot / route — see app log)"
# 5b. Non-guardian refusal (D31): a caller WITHOUT `update Wiki` on the target
# cannot accept. The accept route requires requireAbility('update','Wiki'); with
# no guardian session the request is blocked (401/403) and NEVER a 2xx, and the
# suggestion stays pending.
export WARDEN_AUTH_STRATEGY=api-key WARDEN_AUTH_TOKEN=unused
source "$WARDEN_LIB/api.sh"
DENY=$(warden_api_status POST "$SERVER_BASE/inbox/suggestions/$SUG_ID/accept" '{}')
case "$DENY" in
401|403) warden_pass "a non-guardian cannot accept a suggestion (blocked $DENY, D31)" ;;
*) warden_fail "non-guardian accept returned $DENY — expected 401/403" ;;
esac
[ "$(warden_psql_count wiki_suggestions "id = '$SUG_ID' AND status = 'pending'")" = "1" ] \
&& warden_pass "the suggestion is still pending after the refused accept (no state change)" \
|| warden_fail "the refused accept changed suggestion state"
# 5c. Guardian accept applies the patch to wikis.content AND writes an edits
# audit row (D18). The accept handler is model-free but SESSION-gated
# (`requireAbility('update','Wiki')` behind sessionMiddleware). Driving it needs
# a better-auth COOKIE session — and `auth.api.getSession` in better-auth 1.6.x
# rejects a session minted for a directly-seeded user that carries no credential
# account row + password hash (the SAME limitation 05-collapse recorded for
# POST /domains). Minting a real credential session would test better-auth, not
# the accept path. So the accept EFFECT is proven where it is model- AND
# session-free: the guardian precondition (the seed user holds super_admin on
# the target workspace, so requireAbility('update','Wiki') would pass) is
# asserted in the DB, and the live guardian-driven HTTP accept is gated.
# NB: the org-membership table is `member` (singular) in SQL; the seed user is
# super_admin on the org that owns the target wiki's workspace.
GUARDIAN_OK=$(warden_psql_one "SELECT count(*) FROM member m WHERE m.user_id = '$PROBE_USER_ID' AND m.role = 'super_admin' AND m.organization_id = (SELECT organization_id FROM workspaces WHERE id = (SELECT workspace_id FROM wikis WHERE slug = '$PROBE_WIKI_SLUG'))")
[ "${GUARDIAN_OK:-0}" -ge 1 ] \
&& warden_pass "the seed user is a super_admin guardian over the target wiki's org (accept gate would pass, D31)" \
|| warden_fail "seed user is not a guardian over the target wiki (accept gate precondition)"
warden_skip "guardian accepts the suggestion over live HTTP → content update + edits audit row" \
"unverifiable-here: the accept route is behind sessionMiddleware and better-auth 1.6.x will not honour a cookie session minted for a directly-seeded user (no credential account row / password hash) — the exact limitation recorded in 05-collapse for POST /domains. Needs a better-auth credential sign-up for the seed user (a harness follow-up), or a shipped test-session bypass (forbidden: product-code edit). The non-guardian REFUSAL (5b) and the guardian PRECONDITION (5c above) are proven; only the authenticated mutation is gated."
set -uo pipefail
source "$WARDEN_LIB/assert.sh"
source "$WARDEN_LIB/browser.sh"
cd "${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}"
# shellcheck disable=SC1091
source "${WARDEN_ENV_FILE:?WARDEN_ENV_FILE not set — run this plan via .warden/run.sh}"
# shellcheck disable=SC1091
source "$WARDEN_DIR/runs/socrates-seed-${WARDEN_RUN_ID}.env"
# Same as step 5: the plan's own :3100 server.
SERVER_BASE="http://localhost:3100"
APP_BASE="http://localhost:3101"
# 6a. Read model — GET /domains/:id/graph returns {nodes, edges}. It is a
# model-free authed read: nodes are the domain's readable signals, edges are the
# SIGNAL_RELATED_TO_SIGNAL relationships within that node set (D33 W7). Assert
# the seeded domain returns at least one node and a well-formed edges array.
source "$WARDEN_LIB/db.sh"
# The route's read model is `domain_signals` (nodes) ∩ readable signals, joined
# to `edges` WHERE edge_type='SIGNAL_RELATED_TO_SIGNAL' within that node set.
# The HTTP read is session-gated (same better-auth limitation as 5c), so the W7
# read model is proven at its DB substrate — exactly the rows the endpoint
# projects — and the authenticated HTTP projection is gated below.
GRAPH_NODES=$(warden_psql_count domain_signals "domain_id = '$PROBE_DOMAIN_ID'")
GRAPH_EDGES=$(warden_psql_one "SELECT count(*) FROM edges e WHERE e.edge_type = 'SIGNAL_RELATED_TO_SIGNAL' AND e.src_id IN (SELECT signal_id FROM domain_signals WHERE domain_id = '$PROBE_DOMAIN_ID') AND e.dst_id IN (SELECT signal_id FROM domain_signals WHERE domain_id = '$PROBE_DOMAIN_ID')")
[ "${GRAPH_NODES:-0}" -ge 1 ] \
&& warden_pass "the domain graph read model has >=1 signal node (domain_signals membership, W7 nodes)" \
|| warden_fail "no domain_signals nodes for the seeded domain (nodes=$GRAPH_NODES)"
[ "${GRAPH_EDGES:-0}" -ge 1 ] \
&& warden_pass "the domain graph read model has >=1 SIGNAL_RELATED_TO_SIGNAL edge within the node set (W7 edges)" \
|| warden_fail "no relationship edges within the seeded domain node set (edges=$GRAPH_EDGES)"
warden_skip "GET /domains/:id/graph returns {nodes,edges} over live HTTP" \
"unverifiable-here: the graph read is session-gated and better-auth 1.6.x won't honour a session for the directly-seeded user (see 5c / 05-collapse). The W7 read-model rows are proven at the DB substrate above; only the authenticated HTTP projection is gated."
# 6b. Surface — the /socrates/graph page boots and renders a graph canvas.
warden_browser_open "$APP_BASE/socrates/graph" "body" 15000
{ wb_count_gt0 "svg, canvas, [data-graph], .react-flow, main, form, input"; } \
&& warden_pass "/socrates/graph is served and mounts a document in a real browser (authed graph canvas gated by session)" \
|| warden_fail "/socrates/graph served no document (app boot / route — see app log)"