feat(kubernetes): isolate workloads behind a dedicated supervisor

.agents/skills/helm-dev-environment/SKILL.md

@@ -70,9 +70,10 @@ mise run helm:skaffold:run
 ```
 
 The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images
-and deploys the OpenShell Helm chart. The Kubernetes compute driver places
-`openshell-sandbox` in the workload Pod and `openshell-supervisor` in a separate
-Pod, then creates their authenticated channel and NetworkPolicy fence. The
+and deploys the OpenShell Helm chart. The Kubernetes driver creates a
+capability-free workload Pod and a directly managed capability-free supervisor
+Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell
+workload Pod. The
 `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`)
 generates mTLS secrets on first install. The default Skaffold values export
 gateway and Kubernetes-driver traces to the collector service installed by
@@ -238,7 +239,7 @@ SPIFFE JWT-SVIDs for dynamic provider token grants:
 `openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
 annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
 OpenShell mounts the SPIFFE CSI Workload API socket at
-`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token
+`/spiffe-workload-api/spire-agent.sock` only into supervisor Pods for provider token
 grants. Supervisor-to-gateway authentication remains on the Kubernetes
 ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected
 Kubernetes compute driver validates the projected token before the gateway

.github/workflows/branch-e2e.yml

@@ -406,24 +406,18 @@ jobs:
         include:
           - agent_sandbox_api: v1beta1
             agent_sandbox_version: v0.5.0
-            topology: combined
-            extra_helm_values: ""
+            extra_helm_values: deploy/helm/openshell/ci/values-sandbox-runtime.yaml
           - agent_sandbox_api: v1alpha1
             agent_sandbox_version: v0.4.6
-            topology: combined
-            extra_helm_values: ""
-          - agent_sandbox_api: v1beta1
-            agent_sandbox_version: v0.5.0
-            topology: sidecar
-            extra_helm_values: deploy/helm/openshell/ci/values-sidecar.yaml
+            extra_helm_values: deploy/helm/openshell/ci/values-sandbox-runtime.yaml
     permissions:
       actions: read
       contents: read
       packages: read
     uses: ./.github/workflows/e2e-kubernetes-test.yml
     with:
       image-tag: ${{ github.sha }}
-      job-name: Kubernetes E2E (Rust smoke, ${{ matrix.topology }}, Agent Sandbox ${{ matrix.agent_sandbox_api }})
+      job-name: Kubernetes E2E (Rust smoke, Agent Sandbox ${{ matrix.agent_sandbox_api }})
       agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
       extra-helm-values: ${{ matrix.extra_helm_values }}
       conformance-artifact-prefix: openshell-conformance
@@ -440,6 +434,7 @@ jobs:
       image-tag: ${{ github.sha }}
       job-name: Kubernetes E2E (workspace managed mode)
       e2e-task: e2e:kubernetes:workspace-managed
+      extra-helm-values: deploy/helm/openshell/ci/values-sandbox-runtime.yaml
       conformance-artifact-prefix: openshell-conformance
 
   kubernetes-external-driver-e2e:
@@ -456,7 +451,8 @@ jobs:
       e2e-task: e2e:kubernetes:external-driver
       gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
       external-driver-binary: openshell-driver-kubernetes
-      cluster-images: supervisor
+      cluster-images: sandbox supervisor
+      extra-helm-values: deploy/helm/openshell/ci/values-sandbox-runtime.yaml
       conformance-artifact-prefix: openshell-conformance
 
   kubernetes-workspace-operator-e2e:
@@ -471,6 +467,7 @@ jobs:
       image-tag: ${{ github.sha }}
       job-name: Kubernetes E2E (workspace operator mode)
       e2e-task: e2e:kubernetes:workspace-operator
+      extra-helm-values: deploy/helm/openshell/ci/values-sandbox-runtime.yaml
       conformance-artifact-prefix: openshell-conformance
 
   kubernetes-ha-e2e:

README.md

@@ -48,7 +48,8 @@ uv add openshell
 Deploy the OpenShell gateway into a Kubernetes cluster from the OCI chart published to GHCR:
 
 ```bash
-helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart
+helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
+  --set supervisor.sandboxRuntime.networkPolicyEnforced=true
 ```
 
 See [`deploy/helm/openshell/README.md`](deploy/helm/openshell/README.md) for available versions, dev tag conventions, and configuration.

architecture/compute-runtimes.md

@@ -258,7 +258,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
 |---|---|---|---|
 | Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. |
 | Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. |
-| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses empty-egress NetworkPolicy, paired-only supervisor ingress, and a separate capability-free supervisor Deployment over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. |
+| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses one namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. |
 | VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
 | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
 
@@ -334,7 +334,7 @@ Drivers deliver the two binaries to separate trust domains:
 |---|---|
 | Docker | A digest-pinned daemon-local volume supplies `openshell-sandbox`; the companion image runs `openshell-supervisor`. |
 | Podman | Existing driver behavior; not converted by this stack. |
-| Kubernetes | A non-root init container stages `openshell-sandbox` into a memory volume; the separate Deployment image runs `openshell-supervisor`. |
+| Kubernetes | A non-root init container stages `openshell-sandbox` into a memory volume; a directly managed Pod runs `openshell-supervisor`. |
 | VM | `openshell-sandbox` is embedded in the guest rootfs; a separately digest-checked native `openshell-supervisor` runs on the host. |
 | Extension | Defined by the out-of-tree driver. |
 
@@ -365,9 +365,9 @@ and explicit `/sandbox` values select `/sandbox`; other paths must already
 exist without symlink or reserved-mount collisions and must be usable by the
 resolved identity. Kubernetes and VM use `/sandbox`.
 
-The Kubernetes driver creates the empty-egress workload fence before a
-suspended Sandbox CR, then provisions split immutable bootstrap Secrets, the
-private runtime Service, and the supervisor Deployment. A
+The Kubernetes driver creates the namespace-wide empty-egress workload fence
+before a suspended Sandbox CR, then provisions split immutable bootstrap
+Secrets, the private runtime Service, and a gated supervisor Pod. A
 non-root init container stages `openshell-sandbox` and one-use bootstrap files
 into memory volumes. The workload Pod never mounts supervisor or gateway
 credentials. The driver removes its scheduling gate only after the companions

architecture/sandbox.md

@@ -396,7 +396,7 @@ CA material, and the other sandbox half of the authenticated channel.
 
 For Kubernetes, the operator configures a Secret name and key rather than a
 gateway-host file path. Kubernetes projects that Secret only into the separate
-supervisor Deployment. The sandbox Pod never mounts corporate-proxy credentials
+supervisor Pod. The sandbox Pod never mounts corporate-proxy credentials
 or the interception CA private key.
 
 The Basic header travels over the plain-TCP connection to the `http://` proxy,

crates/openshell-driver-kubernetes/Cargo.toml

@@ -16,6 +16,8 @@ path = "src/main.rs"
 
 [dependencies]
 openshell-core = { path = "../openshell-core", default-features = false }
+openshell-isolation-interface = { path = "../openshell-isolation-interface" }
+openshell-sandbox-backend = { path = "../openshell-sandbox-backend" }
 openshell-otel = { path = "../openshell-otel" }
 openshell-policy = { path = "../openshell-policy" }
 
@@ -37,6 +39,8 @@ tracing-subscriber = { workspace = true }
 thiserror = { workspace = true }
 miette = { workspace = true }
 notify = "8"
+rand = { workspace = true }
+rcgen = { workspace = true }
 
 [dev-dependencies]
 openshell-otel-test-support = { path = "../openshell-otel-test-support" }

crates/openshell-driver-kubernetes/README.md

@@ -45,9 +45,52 @@ not a tenant isolation boundary.
 ## Runtime Model
 
 The gateway stores platform state and delegates sandbox workload creation to
-this driver. Kubernetes owns scheduling and pod lifecycle. The
-`openshell-sandbox` supervisor inside each workload owns agent isolation,
-credential injection, policy polling, logs, and the gateway relay.
+this driver. Kubernetes owns scheduling and pod lifecycle. The workload Pod
+stages the statically linked musl `openshell-sandbox` binary from
+`sandbox_runtime_image`, while a directly managed Pod runs the dynamically
+linked glibc `openshell-supervisor` from `supervisor_image`.
+
+The sandbox owns the agent process, applies Landlock and child seccomp filters,
+identifies the binary behind each network syscall, and relays mediated streams
+to the supervisor. The supervisor authenticates to the gateway with a JWT,
+loads policy and provider state, performs destination and L7 authorization, and
+opens upstream connections. The workload receives no gateway credential,
+provider identity socket, or corporate-proxy credential.
+
+Both Pods run as the namespace-resolved non-root UID/GID with
+`allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, and the runtime
+default seccomp profile. The sandbox installs a nested seccomp user-notification
+filter without requesting a capability in the Pod spec. Startup fails closed
+when the runtime blocks the required seccomp or Landlock operations.
+
+The supervisor Pod has a direct, non-controller owner reference to the Sandbox
+resource. This links its garbage-collection lifecycle to the sandbox without
+competing with the Agent Sandbox controller for workload-Pod ownership.
+
+The driver creates one namespace-wide `NetworkPolicy` before it releases any
+workload Pod. It selects every OpenShell workload, denies all workload egress,
+and permits OpenShell supervisor Pods to reach the sandbox TLS port. The
+authenticated Sandbox Protocol binds each connection to the exact sandbox and
+supervisor Pod identities. Supervisors have normal egress for gateway, DNS,
+and policy-approved upstream connections unless an operator policy restricts
+them. Set
+`sandbox_runtime.network_policy_enforced = true` only after verifying that the cluster
+CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces.
+
+Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
+container stages the sandbox bootstrap into memory, and the sandbox removes it
+before starting untrusted code. The other Secret is mounted only by the
+supervisor. The TLS channel binds the namespace, Sandbox CR, workload Pod,
+supervisor Pod, and shared network-policy identities. Stop deletes the workload
+and supervisor Pods. Start rotates both Secrets and creates a new supervisor
+Pod before releasing a new workload Pod. The shared network fence remains for
+the lifetime of the namespace.
+
+Kubernetes policies are additive, and the API does not attest that the CNI
+enforces them. Keep sandbox namespaces administrative: untrusted principals
+must not create permissive policies, create Pods, read bootstrap Secrets, or
+spoof the OpenShell role labels. Exact supervisor-to-sandbox authorization is
+still enforced by TLS, JWT claims, session generation, and recorded Pod UIDs.
 
 ## Sandbox Resource
 
@@ -95,73 +138,31 @@ mount attaches an existing PVC under `/sandbox`, which skips the default PVC.
 
 ## Credentials, TLS, and Relay
 
-The driver injects gateway callback configuration, sandbox identity, TLS client
-material, and the supervisor SSH socket path into the workload. The callback
-endpoint is required because the sandbox namespace does not identify the
-Gateway Service; Helm renders it from the release topology, while standalone
-and raw TOML configurations must set it explicitly. Driver-owned values must
-override image-provided environment variables.
-
-Sandbox pods run as `service_account_name` and keep
-`automountServiceAccountToken: false`. The only Kubernetes token exposed to the
-supervisor is an explicit, audience-bound projected token mounted at
-`/var/run/secrets/openshell/token` for the one-shot `IssueSandboxToken`
-bootstrap exchange. The Kubernetes driver authenticates that token through the
-compute-driver protocol using its own `service_account_name` and workspace-mode
-namespace policy; the gateway receives only the verified sandbox ID.
-
-The gateway uses the supervisor relay for connect, exec, and file sync. Sandbox
-pods do not need direct external ingress for SSH.
-
-The driver forwards the canonical main-process specification to the process
-supervisor and sets pod `restartPolicy: Never`. Main-process environment
-overrides stay local to that child; the sidecar bootstrap retains the unmodified
-provider environment used by later exec, editor, and SFTP sessions.
+Both Pods set `automountServiceAccountToken: false`. The supervisor receives an
+explicit audience-bound projected token for the one-shot `IssueSandboxToken`
+exchange. The driver verifies that token and the gateway returns the
+sandbox-scoped JWT used by the supervisor session. The sandbox Pod receives
+neither token.
+
+The gateway uses the supervisor relay for connect, exec, logs, and file sync.
+Sandbox Pods do not need direct external ingress for SSH.
+
+The driver sends the canonical main-process specification only to the
+supervisor. The supervisor passes admitted launch state over the protected
+channel. Provider environment updates apply to future exec sessions.
 
 ## Container Security Context
 
-The default `combined` supervisor topology grants the sandbox agent container
-the Linux capabilities the supervisor needs for namespace setup and process,
-filesystem, and network policy enforcement.
-
-The `sidecar` supervisor topology moves pod-level network setup into a root init
-container. In the default process/binary-aware mode, the long-lived network
-sidecar runs as UID 0 with `allowPrivilegeEscalation: false`, drops default
-Linux capabilities, and adds only `SYS_PTRACE` plus `DAC_READ_SEARCH` for
-cross-UID workload `/proc` inspection. The agent container also runs as the
-resolved sandbox UID/GID with `allowPrivilegeEscalation: false` and
-`capabilities.drop: ["ALL"]`.
-Set `sidecar.process_binary_aware_network_policy = false` to run the network
-sidecar as the configured non-root `sidecar.proxy_uid`, omit the extra `/proc`
-inspection capabilities, and enforce endpoint/L7 network policy without
-matching `policy.binaries`.
-In this mode OpenShell preserves gateway session and SSH behavior, but the
-process supervisor does not perform root-to-sandbox privilege dropping or
-supervisor identity mount isolation. It still applies Landlock filesystem policy
-and child seccomp filters where the kernel/runtime supports them. Network
-endpoint and L7 policy remain enforced by the network sidecar, and
-sidecar pods use a shared process namespace so the network sidecar can resolve
-process/binary identity through `/proc/<entrypoint-pid>`.
-
-Sidecar mode keeps gateway credentials in the network sidecar. The agent
-container does not mount the projected service-account token used for sandbox
-token bootstrap, does not mount the sandbox client TLS secret, and does not get
-gateway callback environment variables. The process supervisor receives policy
-and provider environment state from the sidecar over a local control socket in
-the shared sidecar state volume. The sidecar accepts only the pre-workload
-process-supervisor connection, authenticates its UID/GID/PID with peer
-credentials, and removes the listener afterward. SSH relays use a Linux
-abstract socket whose peer PID must match that authenticated supervisor. Both
-supervisors exit if the control connection closes, coupling their container
-restart lifecycle before a new authoritative client can be established.
-
-The driver uses the shared AppArmor model through `app_armor_profile`.
-Supported values are `Unconfined`, `RuntimeDefault`, and
-`Localhost/<profile-name>`; an empty or unset value omits
-`securityContext.appArmorProfile`. Docker and Podman translate the same values
-to OCI security options. Helm deployments default sandbox agent containers to
-`Unconfined` because runtime/default AppArmor profiles can block the
-supervisor's network namespace mount setup on AppArmor-enabled nodes.
+The sandbox, trusted bootstrap init container, and supervisor request no added
+Linux capability. They run as the same numeric non-root identity, disable
+privilege escalation, drop all capabilities, and inherit `RuntimeDefault`
+seccomp. The sandbox and agent must use the same complete UID, GID, and
+supplementary-group identity because the capability-free sandbox cannot change
+credentials after launch and must inspect its same-identity descendants.
+
+The workload Pod does not share host network, PID, IPC, or process namespaces.
+The driver uses a scheduling gate to inspect the admitted Pod and bind its UID
+into the bootstrap claims before kubelet starts it.
 
 ## GPU Support