feat(buzz-db): durable workflow agent delivery state machine (Node C)

[loganj]

🤖 Opened by Mongo (co-author; Logan Johnson is git author + DCO signer).

Node C — dormant DB workflow-delivery state machine

Node C of the shared workflow-delivery DAG: the complete dormant DB delivery transition system for workflow messages addressed to managed agents, built on Node B's canonical WorkflowDeliveryBinding vocabulary. Dormant by contract — no producer, runtime, API, or ACP reachability.

Stacked on B #6818 (darryl/delivery-protocol-vocabulary @ 46aae72db5d29be3545b5a9025568d7441117773), which descends from corrected A 3f18adb23610df29b61118362431f8f044becca7. #2737 / #6697 / #6818 unchanged.

What it adds

Migration 0035 + schema.sql create workflow_agent_deliveries:

• One canonical binding per (community, run, step, target).
• pending → claimed → finished | failed status enum.
• Monotonic lease_generation fence.
• Decomposed WorkflowDeliveryCause identity columns guarded by a CHECK that makes an ambiguous/absent cause unrepresentable.
• Cascading FKs to workflows / workflow_runs / events, plus the community write fence.

buzz-db module persists B's WorkflowDeliveryBinding verbatim (no duplicate tuple spelling) and owns only the lifecycle:

• lock_workflow_agent_delivery_identity + commit_workflow_agent_deliveries — the only insert path; atomic per step, idempotent across producer retries via ON CONFLICT DO NOTHING.
• claim_workflow_agent_delivery — FOR UPDATE SKIP LOCKED, scoped to authenticated target + community; an optional expected binding must match every supplied field (workflow/run/step/definition/message + full decomposed cause), so a forged/stale wake is a miss — never an alternate authority path. The winning row bumps its fence token.
• renew / finish — fenced by lease_generation; a reaped or superseded holder matches zero rows and fails closed. finish is terminal-once and reconciles uncertain completion to a single terminal idempotently.
• reap_expired_workflow_agent_deliveries — fleet-wide, filtered through community_write_allowed exactly like the scheduler prune scan, so a quiescing/fenced/deleted tenant is skipped before its write fence can abort healthy tenants in the same statement.

deletion.rs registers workflow_agent_deliveries in both EXPECTED_SCOPED_TABLES and PURGE_SCOPED_TABLES (FK-safe ordering) — any tenant-scoped + fenced table must live in both lists.

Validation (exact head 6ec79d5882e46a75f56e9d49b3562c58631bf3e8, byte-identical per-file blob shas)

• cargo fmt --check ✅
• cargo clippy -p buzz-db --all-targets -- -D warnings ✅
• unit: 111 passed / 0 failed
• workflow #[ignore] invariant matrix (fresh Postgres, single-thread): 19 passed / 0 failed — create idempotency, community/target claim isolation, mismatched-binding rejection (incl. cause- and workflow-only disagreement), lease fencing, reap + prior-holder fencing, terminal-once/idempotent reconciliation, fleet-wide lifecycle fence, run-deletion cascade, commit rollback.
• deletion-catalog drift test abort_preserves_audit_and_allows_fresh_request on a fresh db.migrate() DB: 1 passed / 0 failed (proves the catalog fix).

Note on deletion #[ignore] tests: deletion::postgres_tests and workflow::tests share one buzz DB via a fixed URL and each call db.migrate(), so co-running the deletion suite collides on an already-schema'd DB (channel_type already exists). They require per-test DB isolation; C's drift fix is proven by the single-test run above. CI's filtered --ignored matrix does not run either suite.

Co-authored-by: Mongo

---

🤖 Review fix (Mongo) — lease-boundary fencing, head a4423214

Larry's review found three places where the delivery lease boundary was not actually fenced under realistic races. All three were on the authority-decision path — a stale or foreign caller could obtain, resurrect, or be credited with authority it did not hold. Fixed at every re-resolved boundary:

1. Reconciliation dropped the generation fence. finish correctly fenced its terminal UPDATE by lease_generation, but the uncertain-completion fallback SELECT read (community_id, id) only. gen-1 expires → reaper bumps → gen-2 claims and finishes → gen-1 finish: the UPDATE missed, but the unfenced SELECT read gen-2's terminal and returned AlreadyTerminal, laundering gen-2's outcome back to gen-1. The reconciliation read is now fenced by the caller's generation (finish never bumps it, so a genuine idempotent retry still matches); a terminal at a different generation is LeaseLost.
2. renew/finish never enforced lease_until >= NOW(). Both gated only on status='claimed' + generation. Between the deadline and the reaper winning, an expired holder could renew (resurrect authority) or finish (settle out-of-lease work). Both mutations now also require an unexpired DB-clock lease — a lease no longer depends on the janitor running first to become invalid.
3. claim never checked the expected binding's own community/target. The row lookup scoped by the request (community, target) arguments and bound every expected-binding field except the binding's own community and target, so a foreign expected binding could authorize a row selected under a different request scope. claim now rejects an expected binding whose community or target disagrees with the request scope before any SQL.

New fresh-Postgres invariant tests, each proven load-bearing (fails on unfixed source, passes on the fix): the exact stale-holder-vs-newer-holder finish interleaving (same + conflicting outcomes → LeaseLost, not AlreadyTerminal); expired renew/finish before reaping (→ LeaseLost); and community-only / target-only expected-binding scope mismatch (→ claim miss).

Exact-head Blox validation (ws mongo-c-lease-fix-20260825, tree byte-identical to a4423214, blob e2eecfbb): cargo fmt --check ✅ · cargo clippy -p buzz-db --all-targets -- -D warnings ✅ · unit 111 / 0 ✅ · delivery #[ignore] fresh-PG matrix (single-thread) 22 / 0 ✅. Still dormant by contract: no producer, runtime, API, or ACP reachability.

Co-authored-by: Mongo