docs: document Omnigent host integration

README.md

@@ -155,6 +155,77 @@ Tracing setup is skipped gracefully when `APP_OWNER` is not set (e.g., local dev
 
 ---
 
+## Omnigent Host Integration
+
+CoDA can register itself as a persistent **[Omnigent](https://github.com/omnigent-ai/omnigent) agent host** — an always-on target the Omnigent server can drive coding-agent sessions into. Those sessions run *inside this container* and use the same filesystem as browser terminals. They authenticate to Databricks as the CoDA app service principal, not as the interactive browser user, so their Unity Catalog authority may differ. A deployed CoDA app becomes both an interactive terminal **and** a headless host that survives restarts and redeploys.
+
+**Off by default.** With `OMNIGENTS_SERVER_URL` unset, none of this runs and CoDA behaves exactly as before. This is opt-in, environment-specific wiring — the committed `app.yaml` keeps it commented out.
+
+### Turning it on
+
+Set three variables in your deployed `app.yaml` (see `app.yaml.lakemeter` for a ready-to-copy overlay template):
+
+```yaml
+# app.yaml
+env:
+  # The Omnigent server this app registers against on boot.
+  - name: OMNIGENTS_SERVER_URL
+    value: "https://<your-omnigent-app>.<region>.databricksapps.com"
+  # UC Volume holding the omnigent host wheels (app SP needs READ_VOLUME).
+  - name: OMNIGENTS_WHEEL_SPEC
+    value: "/Volumes/<catalog>/<schema>/artifacts/wheels"
+  # Optional: force-reinstall the host CLI on boot while rolling out a new wheel.
+  - name: OMNIGENTS_FORCE_REINSTALL
+    value: "1"
+```
+
+Before deploying, grant the CoDA app service principal `CAN_USE` on the
+Omnigent server app plus `USE_CATALOG`, `USE_SCHEMA`, `READ_VOLUME`, and
+`WRITE_VOLUME` on the wheel-volume path. The repository's grant target applies
+the complete prerequisite set:
+
+```bash
+make grant-omnigent-host PROFILE=<profile> APP_NAME=<coda-app>
+```
+
+On boot, `initialize_app()` calls `start_host()`, which — only when `OMNIGENTS_SERVER_URL` is set — installs the `omnigents host` CLI from the wheel volume and launches it as a supervised background process that dials the server over an outbound WSS tunnel.
+
+### Two credentials, two jobs
+
+The non-obvious part of this design is that the host uses **two separate credentials** (see `omnigents_host.py`):
+
+```
+┌─────────────────────── CoDA container ───────────────────────┐
+│                                                              │
+│  omnigents host  ──WSS tunnel──►  Omnigent server            │
+│      │              (auth: app-SP OAuth token)               │
+│      │                                                       │
+│      └── spawns runner ──►  AI Gateway                       │
+│                             (auth: CoDA's ANTHROPIC_* creds) │
+└──────────────────────────────────────────────────────────────┘
+```
+
+* **Host tunnel and runners** authenticate to the server through short-lived app-SP OAuth tokens. CoDA captures the SP credentials before stripping them from the environment, keeps the client secret only in Flask process memory, and exposes fresh tokens through a loopback-only broker. The on-disk `[omnigents-host]` profile contains only the workspace host; spawned Omnigent runners receive a refresh command, not a static bearer or client secret.
+* **Harness LLM** — the runner the host spawns authenticates to AI Gateway via CoDA's already-injected `ANTHROPIC_*` env. No new LLM credential is minted.
+
+### Runtime controls
+
+Beyond boot registration, the host can be driven at runtime:
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/omnigents-status` | GET | Host-integration state (FR-9 observability) |
+| `/api/omnigent-host/status` | GET | Current runtime host state |
+| `/api/omnigent-host/connect` | POST | Start a host tunnel for a supplied `server_url` |
+| `/api/omnigent-host/disconnect` | POST | Stop the active host tunnel |
+| `/api/omnigent-host/share` | POST | Share the SP-owned host with a connecting user |
+
+### Related
+
+`ENABLE_SP_APIKEYHELPER=true` enables the same loopback-broker boundary for agent gateway calls: helpers fetch short-lived app-SP OAuth tokens without persisting the SP client secret or a static token in terminal-visible configuration.
+
+---
+
 ## Quick Start
 
 ### Deploy to Databricks Apps
@@ -321,6 +392,10 @@ Open [http://localhost:8000](http://localhost:8000) — type `claude`, `codex`,
 | `MLFLOW_TRACING_ENABLED` | No | Set to `"true"` to enable MLflow tracing for Claude, Codex, and Gemini in one switch (default `"false"`) |
 | `CLAUDE_CODE_OTEL_ENABLED` | No | Set to `"true"` to enable Claude Code OTEL export to Unity Catalog (default `"false"`) |
 | `CLAUDE_CODE_OTEL_CATALOG_SCHEMA` | No | Target `<catalog>.<schema>` for `claude_otel_spans`, `claude_otel_logs`, and `claude_otel_metrics` |
+| `OMNIGENTS_SERVER_URL` | No | Omnigent server to register against on boot. Unset = host integration off (default). See [Omnigent Host Integration](#omnigent-host-integration) |
+| `OMNIGENTS_WHEEL_SPEC` | No | UC Volume path holding the `omnigents host` wheels (app SP needs `READ_VOLUME`). Required when `OMNIGENTS_SERVER_URL` is set |
+| `OMNIGENTS_FORCE_REINSTALL` | No | Set `"1"` to reinstall the host CLI on boot (for rolling out a new wheel); otherwise `uv tool install` no-ops on an existing binary |
+| `ENABLE_SP_APIKEYHELPER` | No | Set `"true"` to broker short-lived app-SP OAuth tokens over loopback without persisting the client secret in terminal-visible configuration |
 | `DEEPWIKI_MCP_URL` | No | Override or disable the DeepWiki MCP server (set to `""` to remove) |
 | `EXA_MCP_URL` | No | Override or disable the Exa MCP server (set to `""` to remove) |
 | `TEAM_MEMORY_MCP_URL` | No | Optional shared-org-memory MCP server URL |

docs/agent-instructions.md

@@ -77,12 +77,14 @@ Key runtime facts an agent should know:
   `setup_*.py`); agent CLIs install in parallel. `RUNNING` app status only
   means gunicorn bound the port — check `/api/setup-status` or boot logs before
   claiming setup-dependent features work.
-- **Auth (layered):** with `ENABLE_SP_APIKEYHELPER=true`, the app writes its own
-  SP OAuth profile at boot and all agents install without a paste. A short-lived
+- **Auth (layered):** with `ENABLE_SP_APIKEYHELPER=true`, the app keeps its SP
+  client secret in-process and brokers short-lived OAuth tokens over loopback;
+  the on-disk `omnigents-host` profile contains only the workspace host. Agents
+  install without a paste. A short-lived
   PAT pasted on first terminal session is the **fallback** and **auto-rotates
   every 10 min** (`pat_rotator.py`). The rotator rewrites `~/.databrickscfg` on
   every rotation — a known failure mode was clobbering co-owned profiles (e.g.
-  `omnigents-host` OAuth); fixed in `b5b11a6`, but any CLI call that must use
+  `omnigents-host`); fixed in `b5b11a6`, but any CLI call that must use
   the file profile should go through `databrickscfg_only_env()` (see `utils.py`).
 - Databricks CLI: in the **container**, test with `databricks current-user me`.
   On a **local Mac**, use `databricks auth describe --profile <profile>`.
@@ -153,7 +155,7 @@ of the live `coda` app can wedge platform boot.
 
 ## 3. Databricks auth notes (gotchas)
 
-- **Layered auth in-container:** SP OAuth at boot (`ENABLE_SP_APIKEYHELPER`) is
+- **Layered auth in-container:** brokered SP OAuth at boot (`ENABLE_SP_APIKEYHELPER`) is
   tried first; pasted PAT is the fallback. These coexist by design — not
   "PAT or CLIENT_ID/SECRET, pick one."
 - Ambient app-SP env vars can shadow a `~/.databrickscfg` profile. The sync uses

docs/auth-and-identity.md

@@ -4,27 +4,27 @@ Reference for who does what on a running CoDA host (e.g. `coda-01`..`coda-08`),
 how each agent authenticates its model calls, and the known zero-PAT gap for
 OpenCode / Hermes. Written 2026-07-11.
 
-## TL;DR — two identities, don't confuse them
+## TL;DR — two identities, don't confuse them
 
 A CoDA container carries **two separate Databricks identities**:
 
 | Identity | What it is | What it's used for |
 |---|---|---|
-| **The user** (e.g. `david.okeeffe1@coles.com.au`) via the PAT in `~/.databrickscfg [DEFAULT]` | A **user** personal access token (`dapi…`), kept fresh by the PAT rotator | **Everything the terminal / CLI does**: `databricks` commands, `git`, `gh`, workspace/UC ops, file writes. Also the token the content-filter proxy injects for OpenCode / Hermes / Codex model calls. |
-| **The app service principal** (e.g. `app-4n8qml coda-02`, an OAuth client_id) | The Databricks App's own SP, no PAT | **Claude & Pi model inference** (via the `apiKeyHelper` / `!command` that mints an **SP-OAuth** token from the `omnigents-host` profile), plus the Omnigent host registration / tunnel. |
+| **The user** (e.g. `user@example.com`) via the PAT in `~/.databrickscfg [DEFAULT]` | A **user** personal access token (`dapi…`), kept fresh by the PAT rotator | **Everything the terminal / CLI does**: `databricks` commands, `git`, `gh`, workspace/UC ops, and file writes. |
+| **The app service principal** (e.g. `app-4n8qml coda-02`, an OAuth client_id) | The Databricks App's own SP, no PAT | **Agent model inference**, Omnigent host registration, and spawned-runner callbacks via short-lived OAuth tokens from the loopback broker. |
 
 ### So: when Claude runs a command on this box, who is it?
 
-**You.** Any shell command Claude Code runs — `git commit`, `databricks …`, file
-edits — authenticates with the **injected user PAT** in `~/.databrickscfg`, so it
+**You.** Any shell command Claude Code runs — `git commit`, `databricks …`, file
+edits — authenticates with the **injected user PAT** in `~/.databrickscfg`, so it
 acts as **your user identity**, not the app SP. That's why commits/pushes show as
 you and CLI calls have your grants.
 
 Only the **model inference** for Claude/Pi uses the app SP (zero-PAT, via the
 token helper). The shell/tools do not.
 
 > Consequence: on a **shared team CoDA**, everyone on that host shares the one
-> injected PAT — so terminal actions all run as whoever injected it (the host
+> injected PAT — so terminal actions all run as whoever injected it (the host
 > owner), regardless of which teammate is driving. Segregation is at the Omnigent
 > host-share layer, not per-user identity inside the box.
 
@@ -34,44 +34,24 @@ token helper). The shell/tools do not.
 |---|---|---|
 | **Claude** | `apiKeyHelper` in `~/.claude/settings.json` (shared `token_helper.py`) | yes |
 | **Pi** | `!command` apiKey in `~/.pi/agent/models.json` (same `token_helper.py`) | yes |
-| **OpenCode** | `baseURL` → local **content-filter proxy** (`127.0.0.1:4000`), which injects a fresh token per request | ⚠️ only after a PAT is injected once |
-| **Hermes** | routes via the **content-filter proxy** too (same fresh-token injection) | ⚠️ only after a PAT is injected once |
-| **Codex** | content-filter proxy | ⚠️ same as above |
+| **OpenCode** | `baseURL` → local **content-filter proxy** (`127.0.0.1:4000`), which injects a fresh token per request | yes |
+| **Hermes** | routes via the **content-filter proxy** too (same fresh-token injection) | yes |
+| **Codex** | content-filter proxy | yes |
 
-### Why Claude/Pi are zero-PAT but OpenCode/Hermes aren't
+### Secret boundary
 
-- **Claude & Pi** resolve their bearer through `token_helper.py`, which mints an
-  **SP-OAuth** token directly from the `omnigents-host` profile (falling back to
-  a PAT only if that profile is absent). No user PAT required.
-- **OpenCode / Hermes / Codex** route through `content_filter_proxy.py`, whose
-  `_get_fresh_token()` reads the current token from **`~/.databrickscfg`**
-  (`content_filter_proxy.py:54`, injected at `:569-573`). The PAT rotator keeps
-  that file fresh — **but only after a PAT has been bootstrapped**. On the pure
-  SP-OAuth host path there is no PAT, so `~/.databrickscfg` has no token to read
-  until you inject one in the UI. That's the one-time PAT injection.
-- Once injected, the proxy keeps OpenCode/Hermes fresh across PAT rotation with
-  no further injection (that's the dynamic-refresh mechanism — it's not static).
-
-## Known gap / future fix (not done — intentional)
-
-To make **OpenCode / Hermes** zero-PAT like Claude/Pi, teach
-`content_filter_proxy._get_fresh_token()` to **fall back to minting an SP-OAuth
-token** from the `omnigents-host` profile (the same source `token_helper.py`
-uses) when no PAT is present in `~/.databrickscfg`. Small, well-scoped change:
-
-- `content_filter_proxy.py` — add an SP-OAuth mint (via `databricks.sdk` `Config(profile="omnigents-host").authenticate()`) as the fallback in `_get_fresh_token()`, cached with a short TTL like the current path.
-- No change needed to `setup_opencode.py` / `setup_hermes.py` — they already
-  route through the proxy; only the proxy's token source needs the fallback.
-
-Decision (2026-07-11): **left as-is.** "Claude/Pi work with no PAT; Hermes/OpenCode
-work after a one-time PAT injection" is acceptable for the workshop.
+- The Flask process alone retains the app-SP client secret.
+- A loopback-only broker mints short-lived OAuth tokens on demand.
+- The `[omnigents-host]` profile stores only `host`; it has no client ID,
+  client secret, or static token.
+- Agent helpers, the content-filter proxy, the host tunnel, and spawned
+  Omnigent runners obtain fresh tokens without exposing the client secret to a
+  browser terminal.
 
 ## Key files
 
-- `token_helper.py` — shared SP-OAuth/PAT resolver for Claude (`apiKeyHelper`) and Pi (`!command`).
-- `setup_claude.py` / `setup_pi.py` — wire the helper (default-on; opt out via `DISABLE_SP_APIKEYHELPER`).
-- `content_filter_proxy.py` — local proxy for OpenCode/Hermes/Codex; `_get_fresh_token()` (`:54`) + header injection (`:569-573`).
-- `setup_opencode.py` / `setup_hermes.py` — point the agent at the proxy (`127.0.0.1:4000`).
-- `pat_rotator.py` — mints/rotates the user PAT, writes `~/.databrickscfg`, fans out via `cli_auth.py`.
-- `cli_auth.py` — on rotation, refreshes static tokens in each agent's config (skips the `!command` / helper-owned ones).
-- `omnigents_host.py` — host path: `_ensure_{claude,pi,opencode}_settings()` re-run setup with a minted SP bearer on host-connect.
+- `sp_token_broker.py` — loopback-only app-SP token broker.
+- `token_helper.py` — shared broker/SP-OAuth/PAT resolver for agent helpers.
+- `content_filter_proxy.py` — local proxy for OpenCode/Hermes/Codex.
+- `omnigents_host.py` — host supervision and spawned-runner refresh wiring.
+- `pat_rotator.py` — optional user-PAT fallback and rotation.