diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index 15198a891f..faee323e4e 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -69,28 +69,15 @@ mise run helm:skaffold:dev mise run helm:skaffold:run ``` -**Supervisor sidecar topology** (build once and leave running): -```bash -mise run helm:skaffold:run:sidecar -``` - -**Supervisor sidecar topology with TLS/mTLS enabled** (build once and leave running): -```bash -mise run helm:skaffold:run:sidecar-mtls -``` - -Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm -chart. The sidecar profile renders an `openshell-network-init` init container for -nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying. -Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and -`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID, which -must be at least `1000` and distinct from the workload UID. The -sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores -`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install -Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first -install. The default Skaffold values export gateway and Kubernetes-driver traces to -the collector service installed by `helm:k3s:create`. Envoy Gateway opt-in; see the -Optional Add-ons section below. +The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images +and deploys the OpenShell Helm chart. The Kubernetes driver creates a +capability-free workload Pod and a directly managed capability-free supervisor +Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell +workload Pod. The +`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`) +generates mTLS secrets on first install. The default Skaffold values export +gateway and Kubernetes-driver traces to the collector service installed by +`helm:k3s:create`. Envoy Gateway is opt-in; see the Optional Add-ons section. The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or the unified local forwarding task: @@ -102,9 +89,9 @@ mise run helm:k3s:forward The task forwards OTLP/gRPC to `http://127.0.0.1:4317` and the trace UI to `http://127.0.0.1:18888`. When Skaffold has deployed a Kubernetes gateway, it also forwards the gateway to `http://127.0.0.1:8090`; otherwise it continues -with the collector ports only. A successful plaintext `helm:skaffold:run` or -`helm:skaffold:run:sidecar` registers the gateway under the worktree-specific -k3d cluster name and selects it as the active gateway. Keep the forwarding +with the collector ports only. A successful plaintext `helm:skaffold:run` +registers the gateway under the worktree-specific k3d +cluster name and selects it as the active gateway. Keep the forwarding task running while using those endpoints. ### Viewing local traces @@ -134,8 +121,7 @@ create the Secret named `openshell-ha-pg` with a `uri` key, then run ### TLS behaviour `ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run -plaintext by default. To test sidecar topology with TLS enabled, use -`mise run helm:skaffold:run:sidecar-mtls`. +plaintext by default. Override `server.disableTls=false` to exercise TLS/mTLS. | Mode | `server.disableTls` | Gateway scheme | |------|---------------------|----------------| @@ -188,12 +174,6 @@ openshell sandbox list --gateway-endpoint https://localhost:8090 mise run helm:skaffold:delete ``` -For a sidecar-profile deployment: - -```bash -mise run helm:skaffold:delete:sidecar -``` - ### Delete the cluster entirely ```bash @@ -288,7 +268,7 @@ SPIFFE JWT-SVIDs for dynamic provider token grants: `openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod annotations to `spiffe://openshell.local/openshell/sandbox/`. OpenShell mounts the SPIFFE CSI Workload API socket at -`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token +`/spiffe-workload-api/spire-agent.sock` only into supervisor Pods for provider token grants. Supervisor-to-gateway authentication remains on the Kubernetes ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected Kubernetes compute driver validates the projected token before the gateway @@ -349,7 +329,6 @@ for dependencies still declared in `Chart.yaml`. | `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay | | `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) | | `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay | -| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev | | `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay | | `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev | | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) | diff --git a/.agents/skills/sbom/SKILL.md b/.agents/skills/sbom/SKILL.md index e5c3e48c20..0848363c82 100644 --- a/.agents/skills/sbom/SKILL.md +++ b/.agents/skills/sbom/SKILL.md @@ -15,7 +15,7 @@ The OpenShell SBOM tooling produces source-tree CycloneDX JSON SBOMs using Syft, SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored). -Pushed gateway and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates. +Pushed gateway, sandbox, and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates. ## Prerequisites diff --git a/.agents/skills/test-release-canary/SKILL.md b/.agents/skills/test-release-canary/SKILL.md index c710f289f4..66310ed4c2 100644 --- a/.agents/skills/test-release-canary/SKILL.md +++ b/.agents/skills/test-release-canary/SKILL.md @@ -13,18 +13,20 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti | Job | Runner | Verifies | |---|---|---| -| `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. | -| `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. | -| `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. | -| `ubuntu-snap` | `ubuntu-latest` | Downloads the Snap artifact from Release Dev, installs it with `--dangerous`, connects the required interfaces, and waits up to 30 seconds for the recovered local gateway. | -| `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. | +| `macos` | `macos-latest-xlarge` | Installs the dev Homebrew artifacts, reaches the VM gateway, and creates, executes in, and deletes a sandbox. | +| `ubuntu` | `ubuntu-latest` | Installs the dev Debian package, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. | +| `fedora` | `fedora:latest` container | Installs the dev RPM packages, reaches the Podman gateway, and creates, executes in, and deletes a sandbox. | +| `ubuntu-snap` | `ubuntu-latest` | Installs the Release Dev Snap, connects its interfaces, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. | +| `kubernetes` | `ubuntu-latest` + kind | Installs the dev Helm chart, reaches the in-cluster gateway, and creates, executes in, and deletes a sandbox using the published runtime images. | All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject `OPENSHELL_TELEMETRY_ENABLED=false` through the service environment, and the Kubernetes job installs with `server.telemetryEnabled=false`, so smoke traffic does not contribute to product usage metrics. -`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images. +The workflow sets `OPENSHELL_VERSION=dev`, so every `install.sh` job consumes the +rolling dev release produced by the triggering workflow. Kubernetes pins the +matching `0.0.0-dev` chart and `:dev` images. The host-package jobs exercise fresh installs, not upgrades from a persisted schema-v1 gateway config. Validate Homebrew and RPM exact-default migration with @@ -83,7 +85,7 @@ gh run view --log-failed ## Iterating on the canary itself -When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published artifacts* (`0.0.0-dev` chart, `:dev` images, latest tagged install.sh assets). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts. +When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published dev artifacts* (`0.0.0-dev` chart, `:dev` images, and the `dev` GitHub release). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts. Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag. @@ -92,7 +94,7 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${h `Release Dev` publishes two chart versions for every dev build (see `.github/actions/release-helm-oci/action.yml:89-102`): - `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push. -- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.` — immutable, `appVersion` set to the same SHA so it pulls `ghcr.io/nvidia/openshell/gateway:` and `:supervisor:`. +- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images. To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose `chart_version` / `image_tag` inputs. @@ -134,7 +136,8 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i | Symptom | Likely cause | Where to look | |---|---|---| -| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | +| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Dev release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | +| Sandbox create or exec fails | Published sandbox and supervisor artifacts are missing, incompatible, or cannot establish the protected runtime channel. | Gateway logs plus Docker, Podman, VM, Snap, or Kubernetes runtime diagnostics for the job. | | `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. | | `ubuntu-snap` fails after interface connection | The gateway did not recover after Docker became available, or did not become reachable within the 30-second bound. | Failure diagnostics dump Snap service/connection/change state, gateway and snapd journals, Snap logs, and port 17670 listeners. | | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. | diff --git a/.github/actions/build-rust-binary/action.yml b/.github/actions/build-rust-binary/action.yml index 74041e9fc5..53bcfce2d1 100644 --- a/.github/actions/build-rust-binary/action.yml +++ b/.github/actions/build-rust-binary/action.yml @@ -29,6 +29,10 @@ inputs: description: Additional flags passed to cargo build required: false default: "" + interpreter: + description: Expected ELF program interpreter, "none" for static, or "skip" + required: false + default: skip runs: using: composite @@ -71,11 +75,24 @@ runs: env: INPUTS_BINARY: ${{ inputs.binary }} INPUTS_TRIPLE: ${{ inputs.triple }} + INPUTS_INTERPRETER: ${{ inputs.interpreter }} run: | + binary="target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY}" # Confirm the binary runs and reports the expected name. - target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY} --version | grep -q "^${INPUTS_BINARY} " + "$binary" --version | grep -q "^${INPUTS_BINARY} " # Confirm Syft can decode the embedded cargo-auditable metadata. - SYFT_CHECK_FOR_APP_UPDATE=false syft file:target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY} -o cyclonedx-json | grep 'pkg:cargo/' > /dev/null + SYFT_CHECK_FOR_APP_UPDATE=false syft "file:${binary}" -o cyclonedx-json | grep 'pkg:cargo/' > /dev/null + if [ "$INPUTS_INTERPRETER" != skip ]; then + actual_interpreter="$(readelf -l "$binary" | sed -n 's/.*Requesting program interpreter: \(.*\)]/\1/p')" + expected_interpreter="$INPUTS_INTERPRETER" + if [ "$expected_interpreter" = none ]; then + expected_interpreter="" + fi + if [ "$actual_interpreter" != "$expected_interpreter" ]; then + echo "error: expected interpreter '${expected_interpreter:-}', got '${actual_interpreter:-}'" >&2 + exit 1 + fi + fi - name: Upload ${{ inputs.binary }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 diff --git a/.github/actions/check-job-results/action.yml b/.github/actions/check-job-results/action.yml index bd6beb9456..5d2aa445bb 100644 --- a/.github/actions/check-job-results/action.yml +++ b/.github/actions/check-job-results/action.yml @@ -8,7 +8,6 @@ inputs: results: description: JSON-encoded GitHub Actions needs context required: true - runs: using: composite steps: diff --git a/.github/actions/setup-e2e-kind/action.yml b/.github/actions/setup-e2e-kind/action.yml index b9fa8375ba..8013c8569f 100644 --- a/.github/actions/setup-e2e-kind/action.yml +++ b/.github/actions/setup-e2e-kind/action.yml @@ -14,7 +14,7 @@ inputs: images: description: Space-separated OpenShell image components to load required: false - default: gateway supervisor + default: gateway sandbox supervisor registry: description: Container registry and namespace required: false @@ -71,7 +71,7 @@ runs: set -euo pipefail for component in $IMAGE_COMPONENTS; do case "$component" in - gateway | supervisor) ;; + gateway | sandbox | supervisor) ;; *) echo "ERROR: unsupported OpenShell image component: $component" >&2; exit 1 ;; esac image="${REGISTRY}/${component}:${IMAGE_TAG}" diff --git a/.github/workflows/branch-checks.yml b/.github/workflows/branch-checks.yml index 7b0c77c8fd..a65ea745d9 100644 --- a/.github/workflows/branch-checks.yml +++ b/.github/workflows/branch-checks.yml @@ -179,8 +179,8 @@ jobs: tasks/scripts/verify-telemetry-compiled-out.sh present target/debug/openshell-gateway cargo build -p openshell-gateway --bin openshell-gateway --no-default-features --features defaults-without-telemetry tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-gateway - cargo build -p openshell-sandbox --bin openshell-sandbox --no-default-features --features defaults-without-telemetry - tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-sandbox + cargo build -p openshell-supervisor --bin openshell-supervisor --no-default-features --features defaults-without-telemetry + tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-supervisor - name: Verify selective gateway compute-driver builds run: | @@ -197,12 +197,12 @@ jobs: - name: Verify system CA roots build mode compiles and excludes bundled Mozilla roots run: | - cargo check -p openshell-sandbox --all-targets --no-default-features --features system-ca-roots - if cargo tree -p openshell-sandbox -i webpki-roots --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-roots; then + cargo check -p openshell-supervisor --all-targets --no-default-features --features system-ca-roots + if cargo tree -p openshell-supervisor -i webpki-roots --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-roots; then echo "ERROR: webpki-roots found in system CA roots build" >&2 exit 1 fi - if cargo tree -p openshell-sandbox -i webpki-root-certs --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-root-certs; then + if cargo tree -p openshell-supervisor -i webpki-root-certs --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-root-certs; then echo "ERROR: webpki-root-certs found in system CA roots build" >&2 exit 1 fi diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 20183eecb4..59d7e9174a 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -68,6 +68,7 @@ jobs: else run_any_e2e=false fi + { echo "run_core_e2e=$run_core_e2e" echo "run_gpu_e2e=$run_gpu_e2e" @@ -127,11 +128,11 @@ jobs: cargo-version: ${{ needs.version.outputs.cargo }} secrets: inherit - build-sandbox: + build-runtimes: needs: version permissions: contents: read - uses: ./.github/workflows/build-sandbox-binaries.yml + uses: ./.github/workflows/build-runtime-binaries.yml with: cargo-version: ${{ needs.version.outputs.cargo }} secrets: inherit @@ -211,7 +212,7 @@ jobs: secrets: inherit build-vm-driver: - needs: [pr_metadata, version, build-sandbox] + needs: [pr_metadata, version, build-runtimes] if: needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: contents: read @@ -237,21 +238,34 @@ jobs: target-suffix: unknown-linux-gnu secrets: inherit - build-supervisor-image: - needs: [pr_metadata, build-sandbox] + build-sandbox-image: + needs: [pr_metadata, build-runtimes] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true' permissions: contents: read packages: write uses: ./.github/workflows/docker-build.yml with: - component: supervisor + component: sandbox binary: openshell-sandbox target-suffix: unknown-linux-musl secrets: inherit + build-supervisor-image: + needs: [pr_metadata, build-runtimes] + if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true' + permissions: + contents: read + packages: write + uses: ./.github/workflows/docker-build.yml + with: + component: supervisor + binary: openshell-supervisor + target-suffix: unknown-linux-gnu + secrets: inherit + docker-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -264,7 +278,7 @@ jobs: conformance-artifact-prefix: openshell-conformance podman-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -287,7 +301,7 @@ jobs: conformance-artifact-prefix: openshell-conformance docker-external-driver-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-docker, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-docker, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -304,7 +318,7 @@ jobs: [{"suite":"external-driver","cmd":"mise run --no-deps --skip-deps e2e:docker:external-driver","apt_packages":"openssh-client","python_proto":false,"mcp":false}] podman-external-driver-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-podman, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-podman, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -334,7 +348,7 @@ jobs: conformance-artifact-prefix: openshell-conformance gpu-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true' permissions: actions: read @@ -346,7 +360,7 @@ jobs: conformance-artifact-prefix: openshell-conformance kubernetes-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' strategy: fail-fast: false @@ -354,16 +368,8 @@ jobs: include: - agent_sandbox_api: v1beta1 agent_sandbox_version: v0.5.0 - topology: combined - extra_helm_values: "" - agent_sandbox_api: v1alpha1 agent_sandbox_version: v0.4.6 - topology: combined - extra_helm_values: "" - - agent_sandbox_api: v1beta1 - agent_sandbox_version: v0.5.0 - topology: sidecar - extra_helm_values: deploy/helm/openshell/ci/values-sidecar.yaml permissions: actions: read contents: read @@ -371,13 +377,12 @@ jobs: uses: ./.github/workflows/e2e-kubernetes-test.yml with: image-tag: ${{ github.sha }} - job-name: Kubernetes E2E (Rust smoke, ${{ matrix.topology }}, Agent Sandbox ${{ matrix.agent_sandbox_api }}) + job-name: Kubernetes E2E (Rust smoke, Agent Sandbox ${{ matrix.agent_sandbox_api }}) agent-sandbox-version: ${{ matrix.agent_sandbox_version }} - extra-helm-values: ${{ matrix.extra_helm_values }} conformance-artifact-prefix: openshell-conformance kubernetes-workspace-managed-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -391,7 +396,7 @@ jobs: conformance-artifact-prefix: openshell-conformance kubernetes-external-driver-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-kubernetes, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-kubernetes, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -404,11 +409,11 @@ jobs: e2e-task: e2e:kubernetes:external-driver gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu external-driver-binary: openshell-driver-kubernetes - cluster-images: supervisor + cluster-images: sandbox supervisor conformance-artifact-prefix: openshell-conformance kubernetes-workspace-operator-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -422,7 +427,7 @@ jobs: conformance-artifact-prefix: openshell-conformance kubernetes-ha-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true' permissions: actions: read @@ -437,7 +442,7 @@ jobs: conformance-artifact-prefix: openshell-conformance kubernetes-credential-drivers-e2e: - needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-sandbox-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true' permissions: actions: read diff --git a/.github/workflows/build-binaries.yml b/.github/workflows/build-binaries.yml index 4d95b525a8..5c51f1dedf 100644 --- a/.github/workflows/build-binaries.yml +++ b/.github/workflows/build-binaries.yml @@ -33,6 +33,11 @@ on: required: false type: string default: "" + interpreter: + description: Expected ELF program interpreter, "none" for static, or "skip" + required: false + type: string + default: skip checkout-ref: required: false type: string @@ -67,3 +72,4 @@ jobs: image-tag: ${{ inputs['image-tag'] }} artifact-name: ${{ inputs['artifact-name'] }} extra-cargo-flags: ${{ inputs['extra-cargo-flags'] }} + interpreter: ${{ inputs.interpreter }} diff --git a/.github/workflows/build-sandbox-binaries.yml b/.github/workflows/build-runtime-binaries.yml similarity index 56% rename from .github/workflows/build-sandbox-binaries.yml rename to .github/workflows/build-runtime-binaries.yml index 006d396cf0..a9fd011c69 100644 --- a/.github/workflows/build-sandbox-binaries.yml +++ b/.github/workflows/build-runtime-binaries.yml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -name: Build Sandbox Binaries +name: Build Runtime Binaries on: workflow_call: @@ -31,15 +31,32 @@ jobs: include: - triple: x86_64-unknown-linux-musl runner: linux-amd64-cpu8 + package: openshell-sandbox + binary: openshell-sandbox + interpreter: none + - triple: x86_64-unknown-linux-gnu + runner: linux-amd64-cpu8 + package: openshell-supervisor + binary: openshell-supervisor + interpreter: /lib64/ld-linux-x86-64.so.2 - triple: aarch64-unknown-linux-musl runner: linux-arm64-cpu8 + package: openshell-sandbox + binary: openshell-sandbox + interpreter: none + - triple: aarch64-unknown-linux-gnu + runner: linux-arm64-cpu8 + package: openshell-supervisor + binary: openshell-supervisor + interpreter: /lib/ld-linux-aarch64.so.1 uses: ./.github/workflows/build-binaries.yml with: - package: openshell-sandbox - binary: openshell-sandbox + package: ${{ matrix.package }} + binary: ${{ matrix.binary }} triple: ${{ matrix.triple }} runner: ${{ matrix.runner }} cargo-version: ${{ inputs.cargo-version }} image-tag: ${{ inputs.image-tag }} + interpreter: ${{ matrix.interpreter }} checkout-ref: ${{ inputs.checkout-ref }} secrets: inherit diff --git a/.github/workflows/build-vm-driver.yml b/.github/workflows/build-vm-driver.yml index 1e303b8424..2760efa977 100644 --- a/.github/workflows/build-vm-driver.yml +++ b/.github/workflows/build-vm-driver.yml @@ -25,8 +25,48 @@ permissions: contents: read jobs: + host-supervisor-macos: + name: native host supervisor (aarch64-apple-darwin) + permissions: + contents: read + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-supervisor + binary: openshell-supervisor + triple: aarch64-apple-darwin + runner: macos-15-xlarge + cargo-version: ${{ inputs.cargo-version }} + image-tag: ${{ inputs.image-tag }} + checkout-ref: ${{ inputs.checkout-ref }} + secrets: inherit + + guest-init: + name: guest init (${{ matrix.triple }}) + strategy: + matrix: + include: + - arch: x86_64 + triple: x86_64-unknown-linux-musl + runner: linux-amd64-cpu8 + - arch: aarch64 + triple: aarch64-unknown-linux-musl + runner: linux-arm64-cpu8 + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-driver-vm + binary: openshell-vm-init + triple: ${{ matrix.triple }} + runner: ${{ matrix.runner }} + cargo-version: ${{ inputs.cargo-version }} + image-tag: ${{ inputs.image-tag }} + extra-cargo-flags: --no-default-features + interpreter: none + checkout-ref: ${{ inputs.checkout-ref }} + secrets: inherit + build: name: openshell-driver-vm (${{ matrix.triple }}) + needs: [guest-init, host-supervisor-macos] strategy: matrix: include: @@ -59,15 +99,41 @@ jobs: name: openshell-sandbox-${{ matrix.arch }}-unknown-linux-musl path: sandbox + - name: Download openshell-supervisor + if: endsWith(matrix.triple, '-unknown-linux-gnu') + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-supervisor-${{ matrix.arch }}-unknown-linux-gnu + path: supervisor + + - name: Download native macOS host supervisor + if: endsWith(matrix.triple, '-apple-darwin') + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-supervisor-aarch64-apple-darwin + path: host-supervisor + + - name: Download VM guest init + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-vm-init-${{ matrix.arch }}-unknown-linux-musl + path: vm-init + - name: Build VM runtime run: nix build .#vm-runtime - name: Assemble compressed VM runtime run: | compressed_dir="${RUNNER_TEMP}/vm-runtime-compressed" + host_supervisor="supervisor/openshell-supervisor" + if [[ "${{ matrix.triple }}" == *-apple-darwin ]]; then + host_supervisor="host-supervisor/openshell-supervisor" + fi install -d "$compressed_dir" cp result/compressed/*.zst "$compressed_dir/" zstd -19 -T1 sandbox/openshell-sandbox -o "$compressed_dir/openshell-sandbox.zst" + zstd -19 -T1 "$host_supervisor" -o "$compressed_dir/openshell-supervisor.zst" + zstd -19 -T1 vm-init/openshell-vm-init -o "$compressed_dir/openshell-vm-init.zst" - name: Build openshell-driver-vm uses: ./.github/actions/build-rust-binary diff --git a/.github/workflows/e2e-docker-test.yml b/.github/workflows/e2e-docker-test.yml index 0b2338e8a6..24eb4a54f7 100644 --- a/.github/workflows/e2e-docker-test.yml +++ b/.github/workflows/e2e-docker-test.yml @@ -62,6 +62,7 @@ jobs: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} IMAGE_TAG: ${{ inputs.image-tag }} OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell + OPENSHELL_SANDBOX_RUNTIME_IMAGE: ${{ format('ghcr.io/nvidia/openshell/sandbox:{0}', inputs.image-tag) }} OPENSHELL_SUPERVISOR_IMAGE: ${{ format('ghcr.io/nvidia/openshell/supervisor:{0}', inputs.image-tag) }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/.github/workflows/e2e-kubernetes-test.yml b/.github/workflows/e2e-kubernetes-test.yml index 52ee27bc7f..7b7622320c 100644 --- a/.github/workflows/e2e-kubernetes-test.yml +++ b/.github/workflows/e2e-kubernetes-test.yml @@ -76,7 +76,7 @@ on: description: "Space-separated published images to preload into kind" required: false type: string - default: "gateway supervisor" + default: "gateway sandbox supervisor" permissions: actions: read diff --git a/.github/workflows/e2e-label-help.yml b/.github/workflows/e2e-label-help.yml index e5158dca3e..b412cdc6f2 100644 --- a/.github/workflows/e2e-label-help.yml +++ b/.github/workflows/e2e-label-help.yml @@ -42,7 +42,7 @@ jobs: case "$LABEL_NAME" in test:e2e) suite_summary="the standard E2E suite" - build_summary="gateway and supervisor images" + build_summary="gateway, sandbox, and supervisor images" status_summary="The matching required CI gate status on this PR will flip green automatically once the run finishes." ;; test:e2e-gpu) @@ -52,7 +52,7 @@ jobs: ;; test:e2e-kubernetes) suite_summary="Kubernetes HA and credential-driver E2E" - build_summary="gateway and supervisor images" + build_summary="gateway, sandbox, and supervisor images" status_summary="This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status." ;; *) echo "Unrecognized label $LABEL_NAME"; exit 1 ;; diff --git a/.github/workflows/e2e-podman-test.yml b/.github/workflows/e2e-podman-test.yml index 3ec5294c41..c285e63a3e 100644 --- a/.github/workflows/e2e-podman-test.yml +++ b/.github/workflows/e2e-podman-test.yml @@ -49,6 +49,7 @@ jobs: IMAGE_TAG: ${{ inputs.image-tag }} MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell + OPENSHELL_SANDBOX_RUNTIME_IMAGE: ${{ format('ghcr.io/nvidia/openshell/sandbox:{0}', inputs.image-tag) }} OPENSHELL_SUPERVISOR_IMAGE: ${{ format('ghcr.io/nvidia/openshell/supervisor:{0}', inputs.image-tag) }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/.github/workflows/package-release-binaries.yml b/.github/workflows/package-release-binaries.yml index f1900f3c35..2e05bc1876 100644 --- a/.github/workflows/package-release-binaries.yml +++ b/.github/workflows/package-release-binaries.yml @@ -39,9 +39,15 @@ jobs: package: gateway-binary-macos - artifact: openshell-sandbox-x86_64-unknown-linux-musl binary: openshell-sandbox - package: supervisor-binary-linux-amd64 + package: sandbox-binary-linux-amd64 - artifact: openshell-sandbox-aarch64-unknown-linux-musl binary: openshell-sandbox + package: sandbox-binary-linux-arm64 + - artifact: openshell-supervisor-x86_64-unknown-linux-gnu + binary: openshell-supervisor + package: supervisor-binary-linux-amd64 + - artifact: openshell-supervisor-aarch64-unknown-linux-gnu + binary: openshell-supervisor package: supervisor-binary-linux-arm64 - artifact: openshell-driver-vm-x86_64-unknown-linux-gnu binary: openshell-driver-vm diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index 98470584de..d7ba5d9e38 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -20,6 +20,7 @@ defaults: shell: bash env: + OPENSHELL_VERSION: dev OPENSHELL_TELEMETRY_ENABLED: "false" jobs: @@ -36,8 +37,13 @@ jobs: - name: Install and check status run: | + set -euo pipefail curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh openshell status + sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" + openshell sandbox create --name "$sandbox" --detach + openshell sandbox exec --name "$sandbox" --no-tty -- true + openshell sandbox delete "$sandbox" ubuntu: name: Ubuntu Docker @@ -59,8 +65,13 @@ jobs: - name: Install and check status run: | + set -euo pipefail curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh openshell status + sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" + openshell sandbox create --name "$sandbox" --detach + openshell sandbox exec --name "$sandbox" --no-tty -- true + openshell sandbox delete "$sandbox" fedora: name: Fedora RPM @@ -140,7 +151,9 @@ jobs: HOME=/root \ XDG_RUNTIME_DIR=/run/user/0 \ DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \ + OPENSHELL_VERSION="$OPENSHELL_VERSION" \ OPENSHELL_TELEMETRY_ENABLED="$OPENSHELL_TELEMETRY_ENABLED" \ + CANARY_SANDBOX="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \ INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh" \ bash -s <<'EOF' set -euo pipefail @@ -150,6 +163,9 @@ jobs: podman info curl -LsSf "${INSTALL_SH_URL}" | sh openshell status + openshell sandbox create --name "$CANARY_SANDBOX" --detach + openshell sandbox exec --name "$CANARY_SANDBOX" --no-tty -- true + openshell sandbox delete "$CANARY_SANDBOX" EOF - name: Stop Fedora systemd container @@ -216,6 +232,14 @@ jobs: echo "Gateway did not become ready within 30 seconds" >&2 exit 1 + - name: Create and exercise a sandbox + run: | + set -euo pipefail + sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" + openshell sandbox create --name "$sandbox" --detach + openshell sandbox exec --name "$sandbox" --no-tty -- true + openshell sandbox delete "$sandbox" + - name: Collect Snap diagnostics if: failure() run: | @@ -309,6 +333,10 @@ jobs: set -euo pipefail openshell gateway add http://127.0.0.1:8080 --local --name "$KIND_GATEWAY_NAME" openshell status + sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" + openshell sandbox create --name "$sandbox" --detach + openshell sandbox exec --name "$sandbox" --no-tty -- true + openshell sandbox delete "$sandbox" - name: Diagnostics on failure if: failure() diff --git a/.github/workflows/release-dev.yml b/.github/workflows/release-dev.yml index fb364b4e13..bc40bb6df1 100644 --- a/.github/workflows/release-dev.yml +++ b/.github/workflows/release-dev.yml @@ -99,18 +99,18 @@ jobs: image-tag: dev secrets: inherit - build-sandbox: + build-runtimes: needs: compute-versions permissions: contents: read - uses: ./.github/workflows/build-sandbox-binaries.yml + uses: ./.github/workflows/build-runtime-binaries.yml with: cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} image-tag: dev secrets: inherit build-vm-driver: - needs: [compute-versions, build-sandbox] + needs: [compute-versions, build-runtimes] permissions: contents: read uses: ./.github/workflows/build-vm-driver.yml @@ -120,7 +120,7 @@ jobs: secrets: inherit package-binaries: - needs: [build-cli, build-gateway, build-sandbox, build-vm-driver] + needs: [build-cli, build-gateway, build-runtimes, build-vm-driver] permissions: actions: read contents: read @@ -138,20 +138,32 @@ jobs: target-suffix: unknown-linux-gnu secrets: inherit - build-supervisor-image: - needs: build-sandbox + build-sandbox-image: + needs: build-runtimes permissions: contents: read packages: write uses: ./.github/workflows/docker-build.yml with: - component: supervisor + component: sandbox binary: openshell-sandbox target-suffix: unknown-linux-musl secrets: inherit + build-supervisor-image: + needs: build-runtimes + permissions: + contents: read + packages: write + uses: ./.github/workflows/docker-build.yml + with: + component: supervisor + binary: openshell-supervisor + target-suffix: unknown-linux-gnu + secrets: inherit + docker-e2e: - needs: [build-cli, build-conformance, build-gateway, build-supervisor-image] + needs: [build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image] permissions: actions: read contents: read @@ -163,7 +175,7 @@ jobs: conformance-artifact-prefix: openshell-conformance podman-e2e: - needs: [build-cli, build-conformance, build-gateway, build-supervisor-image] + needs: [build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image] permissions: actions: read contents: read @@ -185,7 +197,7 @@ jobs: tag-ghcr-dev: name: Tag GHCR Images as Dev - needs: [build-gateway-image, build-supervisor-image, release-dev] + needs: [build-gateway-image, build-sandbox-image, build-supervisor-image, release-dev] runs-on: linux-amd64-cpu8 timeout-minutes: 10 steps: @@ -196,7 +208,7 @@ jobs: run: | set -euo pipefail REGISTRY="ghcr.io/nvidia/openshell" - for component in gateway supervisor; do + for component in gateway sandbox supervisor; do echo "Tagging ${REGISTRY}/${component}:${{ github.sha }} as dev..." docker buildx imagetools create \ --prefer-index=false \ @@ -253,7 +265,7 @@ jobs: build-snap: name: Build Snap - needs: [compute-versions, build-cli, build-gateway, build-sandbox] + needs: [compute-versions, build-cli, build-gateway, build-runtimes] uses: ./.github/workflows/snap-package.yml with: checkout-ref: ${{ github.sha }} @@ -281,6 +293,20 @@ jobs: fail-fast: false matrix: include: + - name: ubuntu-20.04-binaries-amd64 + runner: linux-amd64-cpu8 + image: ubuntu:20.04 + kind: binary + artifact_arch: amd64 + target: x86_64-unknown-linux-gnu + sandbox_target: x86_64-unknown-linux-musl + - name: ubuntu-20.04-binaries-arm64 + runner: linux-arm64-cpu8 + image: ubuntu:20.04 + kind: binary + artifact_arch: arm64 + target: aarch64-unknown-linux-gnu + sandbox_target: aarch64-unknown-linux-musl - name: ubuntu-22.04-deb-amd64 runner: linux-amd64-cpu8 image: ubuntu:22.04 @@ -313,6 +339,30 @@ jobs: container: image: ${{ matrix.image }} steps: + - name: Download sandbox binary artifact + if: matrix.kind == 'binary' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: sandbox-binary-linux-${{ matrix.artifact_arch }} + path: runtime-input/ + + - name: Download supervisor binary artifact + if: matrix.kind == 'binary' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: supervisor-binary-linux-${{ matrix.artifact_arch }} + path: runtime-input/ + + - name: Smoke runtime binary artifacts + if: matrix.kind == 'binary' + run: | + set -euo pipefail + mkdir -p runtime-bin + tar -xzf "runtime-input/openshell-sandbox-${{ matrix.sandbox_target }}.tar.gz" -C runtime-bin + tar -xzf "runtime-input/openshell-supervisor-${{ matrix.target }}.tar.gz" -C runtime-bin + runtime-bin/openshell-sandbox --version + LD_BIND_NOW=1 runtime-bin/openshell-supervisor --version + - name: Download Debian package artifact if: matrix.kind == 'deb' uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 @@ -403,6 +453,13 @@ jobs: path: release/ merge-multiple: true + - name: Download sandbox binary artifacts + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + pattern: sandbox-binary-* + path: release/ + merge-multiple: true + - name: Download VM driver artifacts uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -479,7 +536,7 @@ jobs: run: | set -euo pipefail cd release - sha256sum \ + sha256sum -- \ openshell-x86_64-unknown-linux-musl.tar.gz \ openshell-aarch64-unknown-linux-musl.tar.gz \ openshell-aarch64-apple-darwin.tar.gz \ @@ -490,15 +547,19 @@ jobs: openshell-*.rpm \ *.whl > openshell-checksums-sha256.txt cat openshell-checksums-sha256.txt - sha256sum \ + sha256sum -- \ openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \ openshell-gateway-aarch64-unknown-linux-gnu.tar.gz \ openshell-gateway-aarch64-apple-darwin.tar.gz > openshell-gateway-checksums-sha256.txt cat openshell-gateway-checksums-sha256.txt - sha256sum \ + sha256sum -- \ openshell-sandbox-x86_64-unknown-linux-musl.tar.gz \ openshell-sandbox-aarch64-unknown-linux-musl.tar.gz > openshell-sandbox-checksums-sha256.txt cat openshell-sandbox-checksums-sha256.txt + sha256sum -- \ + openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz \ + openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz > openshell-supervisor-checksums-sha256.txt + cat openshell-supervisor-checksums-sha256.txt - name: Generate Homebrew formula run: | @@ -604,6 +665,8 @@ jobs: release/openshell-gateway-aarch64-apple-darwin.tar.gz release/openshell-sandbox-x86_64-unknown-linux-musl.tar.gz release/openshell-sandbox-aarch64-unknown-linux-musl.tar.gz + release/openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz + release/openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz release/openshell-driver-vm-aarch64-apple-darwin.tar.gz @@ -612,6 +675,7 @@ jobs: release/openshell-checksums-sha256.txt release/openshell-gateway-checksums-sha256.txt release/openshell-sandbox-checksums-sha256.txt + release/openshell-supervisor-checksums-sha256.txt release-helm: name: Release Helm Chart (OCI, dev) diff --git a/.github/workflows/release-tag.yml b/.github/workflows/release-tag.yml index 962d846407..dd41b53942 100644 --- a/.github/workflows/release-tag.yml +++ b/.github/workflows/release-tag.yml @@ -129,11 +129,11 @@ jobs: checkout-ref: ${{ inputs.tag || github.ref }} secrets: inherit - build-sandbox: + build-runtimes: needs: compute-versions permissions: contents: read - uses: ./.github/workflows/build-sandbox-binaries.yml + uses: ./.github/workflows/build-runtime-binaries.yml with: cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} image-tag: ${{ needs.compute-versions.outputs.semver }} @@ -141,7 +141,7 @@ jobs: secrets: inherit build-vm-driver: - needs: [compute-versions, build-sandbox] + needs: [compute-versions, build-runtimes] permissions: contents: read uses: ./.github/workflows/build-vm-driver.yml @@ -152,7 +152,7 @@ jobs: secrets: inherit package-binaries: - needs: [build-cli, build-gateway, build-sandbox, build-vm-driver] + needs: [build-cli, build-gateway, build-runtimes, build-vm-driver] permissions: actions: read contents: read @@ -172,22 +172,36 @@ jobs: checkout-ref: ${{ inputs.tag || github.ref }} secrets: inherit - build-supervisor-image: - needs: [compute-versions, build-sandbox] + build-sandbox-image: + needs: [compute-versions, build-runtimes] permissions: contents: read packages: write uses: ./.github/workflows/docker-build.yml with: - component: supervisor + component: sandbox binary: openshell-sandbox target-suffix: unknown-linux-musl image-tag: ${{ needs.compute-versions.outputs.source_sha }} checkout-ref: ${{ inputs.tag || github.ref }} secrets: inherit + build-supervisor-image: + needs: [compute-versions, build-runtimes] + permissions: + contents: read + packages: write + uses: ./.github/workflows/docker-build.yml + with: + component: supervisor + binary: openshell-supervisor + target-suffix: unknown-linux-gnu + image-tag: ${{ needs.compute-versions.outputs.source_sha }} + checkout-ref: ${{ inputs.tag || github.ref }} + secrets: inherit + docker-e2e: - needs: [compute-versions, build-cli, build-conformance, build-gateway, build-supervisor-image] + needs: [compute-versions, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image] permissions: actions: read contents: read @@ -200,7 +214,7 @@ jobs: conformance-artifact-prefix: openshell-conformance podman-e2e: - needs: [compute-versions, build-cli, build-conformance, build-gateway, build-supervisor-image] + needs: [compute-versions, build-cli, build-conformance, build-gateway, build-sandbox-image, build-supervisor-image] permissions: actions: read contents: read @@ -224,7 +238,7 @@ jobs: tag-ghcr-release: name: Tag GHCR Images - needs: [compute-versions, build-gateway-image, build-supervisor-image, release] + needs: [compute-versions, build-gateway-image, build-sandbox-image, build-supervisor-image, release] runs-on: linux-amd64-cpu8 timeout-minutes: 10 steps: @@ -239,7 +253,7 @@ jobs: REGISTRY="ghcr.io/nvidia/openshell" VERSION="${{ needs.compute-versions.outputs.semver }}" SOURCE_TAG="${{ needs.compute-versions.outputs.source_sha }}" - for component in gateway supervisor; do + for component in gateway sandbox supervisor; do echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as ${VERSION}..." docker buildx imagetools create \ --prefer-index=false \ @@ -303,7 +317,7 @@ jobs: build-snap: name: Build Snap - needs: [compute-versions, build-cli, build-gateway, build-sandbox] + needs: [compute-versions, build-cli, build-gateway, build-runtimes] uses: ./.github/workflows/snap-package.yml with: checkout-ref: ${{ inputs.tag || github.ref }} @@ -339,6 +353,7 @@ jobs: artifact_arch: amd64 rpm_arch: x86_64 target: x86_64-unknown-linux-gnu + sandbox_target: x86_64-unknown-linux-musl - name: ubuntu-20.04-binaries-arm64 runner: linux-arm64-cpu8 image: ubuntu:20.04 @@ -346,6 +361,7 @@ jobs: artifact_arch: arm64 rpm_arch: aarch64 target: aarch64-unknown-linux-gnu + sandbox_target: aarch64-unknown-linux-musl - name: ubuntu-22.04-deb runner: linux-amd64-cpu8 image: ubuntu:22.04 @@ -396,6 +412,20 @@ jobs: name: driver-vm-linux-${{ matrix.artifact_arch }} path: smoke-input/ + - name: Download sandbox binary artifact + if: matrix.kind == 'binary' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: sandbox-binary-linux-${{ matrix.artifact_arch }} + path: smoke-input/ + + - name: Download supervisor binary artifact + if: matrix.kind == 'binary' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: supervisor-binary-linux-${{ matrix.artifact_arch }} + path: smoke-input/ + - name: Smoke binary artifacts if: matrix.kind == 'binary' run: | @@ -404,8 +434,12 @@ jobs: mkdir -p smoke-bin tar -xzf smoke-input/openshell-gateway-${{ matrix.target }}.tar.gz -C smoke-bin tar -xzf smoke-input/openshell-driver-vm-${{ matrix.target }}.tar.gz -C smoke-bin + tar -xzf smoke-input/openshell-sandbox-${{ matrix.sandbox_target }}.tar.gz -C smoke-bin + tar -xzf smoke-input/openshell-supervisor-${{ matrix.target }}.tar.gz -C smoke-bin LD_BIND_NOW=1 smoke-bin/openshell-gateway --version smoke-bin/openshell-driver-vm --version + smoke-bin/openshell-sandbox --version + LD_BIND_NOW=1 smoke-bin/openshell-supervisor --version - name: Download Debian package artifact if: matrix.kind == 'deb' @@ -499,6 +533,13 @@ jobs: path: release/ merge-multiple: true + - name: Download sandbox binary artifacts + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + pattern: sandbox-binary-* + path: release/ + merge-multiple: true + - name: Download VM driver artifacts uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -550,7 +591,7 @@ jobs: run: | set -euo pipefail cd release - sha256sum \ + sha256sum -- \ openshell-x86_64-unknown-linux-musl.tar.gz \ openshell-aarch64-unknown-linux-musl.tar.gz \ openshell-aarch64-apple-darwin.tar.gz \ @@ -561,15 +602,19 @@ jobs: openshell-*.rpm \ *.whl > openshell-checksums-sha256.txt cat openshell-checksums-sha256.txt - sha256sum \ + sha256sum -- \ openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \ openshell-gateway-aarch64-unknown-linux-gnu.tar.gz \ openshell-gateway-aarch64-apple-darwin.tar.gz > openshell-gateway-checksums-sha256.txt cat openshell-gateway-checksums-sha256.txt - sha256sum \ + sha256sum -- \ openshell-sandbox-x86_64-unknown-linux-musl.tar.gz \ openshell-sandbox-aarch64-unknown-linux-musl.tar.gz > openshell-sandbox-checksums-sha256.txt cat openshell-sandbox-checksums-sha256.txt + sha256sum -- \ + openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz \ + openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz > openshell-supervisor-checksums-sha256.txt + cat openshell-supervisor-checksums-sha256.txt - name: Generate Homebrew formula if: needs.compute-versions.outputs.is_prerelease != 'true' @@ -642,6 +687,8 @@ jobs: release/openshell-gateway-aarch64-apple-darwin.tar.gz release/openshell-sandbox-x86_64-unknown-linux-musl.tar.gz release/openshell-sandbox-aarch64-unknown-linux-musl.tar.gz + release/openshell-supervisor-x86_64-unknown-linux-gnu.tar.gz + release/openshell-supervisor-aarch64-unknown-linux-gnu.tar.gz release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz release/openshell-driver-vm-aarch64-apple-darwin.tar.gz @@ -650,6 +697,7 @@ jobs: release/openshell-checksums-sha256.txt release/openshell-gateway-checksums-sha256.txt release/openshell-sandbox-checksums-sha256.txt + release/openshell-supervisor-checksums-sha256.txt - name: Upload pre-release artifacts if: needs.compute-versions.outputs.is_prerelease == 'true' @@ -706,6 +754,8 @@ jobs: run: | { echo "@nvidia:registry=https://npm.pkg.github.com" + # Keep the token reference literal for npm to expand at publish time. + # shellcheck disable=SC2016 echo '//npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}' } > .npmrc diff --git a/.github/workflows/release-vm-kernel.yml b/.github/workflows/release-vm-kernel.yml index 76f00cb784..8a31c3adbe 100644 --- a/.github/workflows/release-vm-kernel.yml +++ b/.github/workflows/release-vm-kernel.yml @@ -1,6 +1,6 @@ name: Release VM Kernel -# Build custom libkrunfw (kernel firmware) + libkrun (VMM) + gvproxy for all +# Build custom libkrunfw (kernel firmware) + libkrun (VMM) for all # supported openshell-driver-vm platforms. Artifacts are uploaded to the # rolling "vm-runtime" GitHub Release and consumed by normal dev/tag release # workflows when building the openshell-driver-vm binary. @@ -194,23 +194,28 @@ jobs: release/vm-runtime-darwin-aarch64.tar.zst - name: Ensure vm-runtime tag exists + env: + RELEASE_TAG: vm-runtime run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - git tag -fa vm-runtime -m "VM Runtime Development Build" "${GITHUB_SHA}" - git push --force origin vm-runtime + git tag -fa "$RELEASE_TAG" -m "VM Runtime Development Build" "${GITHUB_SHA}" + git push --force origin "$RELEASE_TAG" - name: Prune stale runtime assets from vm-runtime release uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + env: + RELEASE_TAG: vm-runtime with: script: | const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/'); + const tag = process.env.RELEASE_TAG; let release; try { - release = await github.rest.repos.getReleaseByTag({ owner, repo, tag: 'vm-runtime' }); + release = await github.rest.repos.getReleaseByTag({ owner, repo, tag }); } catch (err) { if (err.status === 404) { - core.info('No existing vm-runtime release; will create fresh.'); + core.info(`No existing ${tag} release; will create fresh.`); return; } throw err; @@ -237,7 +242,7 @@ jobs: ### Kernel Runtime Artifacts - Pre-built kernel runtime (libkrunfw + libkrun + gvproxy + umoci) for embedding + Pre-built kernel runtime (libkrunfw + libkrun + umoci) for embedding into the `openshell-driver-vm` binary. These are rebuilt on demand when the kernel config or pinned dependency versions change. diff --git a/AGENTS.md b/AGENTS.md index 954ffe1147..b5710a5386 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,7 +38,11 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-conformance/` | CLI conformance library | Reusable driver-agnostic scenarios and command runner | | `crates/openshell-conformance-cli/` | Conformance CLI | Distributable `list` and `run` entrypoint for gateway conformance | | `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary | -| `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing | +| `crates/openshell-sandbox/` | Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O | +| `crates/openshell-supervisor/` | Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking | +| `crates/openshell-binary-identity/` | Binary identity | Shared trusted procfs executable identity resolution for isolation backends | +| `crates/openshell-isolation-interface/` | Isolation backend interface | RFC 0012 `IsolationBackend` trait and types; the supervisor-facing runtime contract | +| `crates/openshell-sandbox-backend/` | OpenShell sandbox backend | `OpenShellRuntimeBackend` and the authenticated OpenShell Sandbox Protocol shared with `openshell-sandbox` | | `crates/openshell-policy/` | Policy engine | Filesystem, network, and process constraints | | `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage | | `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary | diff --git a/CI.md b/CI.md index 692794e8ee..365cc2fa21 100644 --- a/CI.md +++ b/CI.md @@ -373,9 +373,9 @@ These workflows run after merge to publish dev/tagged artifacts and verify them. | File | Role | |---|---| -| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway/supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | +| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | | `.github/workflows/release-tag.yml` | Publishes a tagged stable release. Its automatic tag trigger excludes `-pre.*`; manual dispatch remains maintainer-controlled. | -| `.github/workflows/release-canary.yml` | Smoke-tests published artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Triggers automatically when `Release Dev` succeeds, and via `workflow_dispatch` on any branch (`gh workflow run release-canary.yml --ref `). The `kubernetes` job pins to `0.0.0-dev` artifacts; the other jobs install the latest tagged release via `install.sh`. See the `test-release-canary` skill for the manual-dispatch playbook and local kind reproduction. | +| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Each job reaches its gateway and creates, exercises, and deletes a sandbox. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref `). See the `test-release-canary` skill for the playbook and local kind reproduction. | ## Required status contexts diff --git a/Cargo.lock b/Cargo.lock index 4584ff07ce..2853942325 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3935,6 +3935,14 @@ version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" +[[package]] +name = "openshell-binary-identity" +version = "0.0.0" +dependencies = [ + "openshell-isolation-interface", + "sha2 0.10.9", +] + [[package]] name = "openshell-bootstrap" version = "0.0.0" @@ -4036,6 +4044,7 @@ dependencies = [ "chrono", "glob", "ipnet", + "jsonwebtoken", "miette", "nix 0.29.0", "openshell-extension-core", @@ -4059,6 +4068,8 @@ dependencies = [ "tonic-prost-build", "tracing", "url", + "uuid", + "zeroize", ] [[package]] @@ -4088,15 +4099,21 @@ dependencies = [ "clap", "futures", "http 1.4.0", + "libc", "miette", "openshell-core", + "openshell-isolation-interface", "openshell-otel", "openshell-otel-test-support", + "openshell-sandbox-backend", "opentelemetry", "opentelemetry_sdk", "prost-types", + "rand 0.9.4", + "rustix 1.1.4", "serde", "serde_json", + "sha2 0.10.9", "tar", "temp-env", "tempfile", @@ -4126,13 +4143,17 @@ dependencies = [ "miette", "notify", "openshell-core", + "openshell-isolation-interface", "openshell-otel", "openshell-otel-test-support", "openshell-policy", + "openshell-sandbox-backend", "opentelemetry", "opentelemetry_sdk", "prost", "prost-types", + "rand 0.9.4", + "rcgen", "serde", "serde_json", "temp-env", @@ -4202,14 +4223,17 @@ dependencies = [ "miette", "nix 0.29.0", "openshell-core", + "openshell-isolation-interface", "openshell-otel", "openshell-otel-test-support", + "openshell-sandbox-backend", "opentelemetry", "opentelemetry_sdk", "prost-types", "rustix 1.1.4", "serde", "serde_json", + "tar", "temp-env", "thiserror 2.0.18", "tokio", @@ -4220,6 +4244,7 @@ dependencies = [ "tracing-opentelemetry", "tracing-subscriber", "url", + "uuid", ] [[package]] @@ -4248,6 +4273,7 @@ dependencies = [ name = "openshell-driver-vm" version = "0.0.0" dependencies = [ + "base64", "bollard", "clap", "flate2", @@ -4260,15 +4286,18 @@ dependencies = [ "oci-client", "openshell-core", "openshell-driver-podman", + "openshell-isolation-interface", "openshell-otel", "openshell-otel-test-support", "openshell-policy", + "openshell-sandbox-backend", "openshell-vfio", "opentelemetry", "opentelemetry_sdk", "polling", "prost", "prost-types", + "rand 0.9.4", "rustix 1.1.4", "serde", "serde_json", @@ -4350,6 +4379,18 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "openshell-isolation-interface" +version = "0.0.0" +dependencies = [ + "async-trait", + "libc", + "openshell-core", + "rustix 1.1.4", + "serde", + "tokio", +] + [[package]] name = "openshell-ocsf" version = "0.0.0" @@ -4438,31 +4479,71 @@ dependencies = [ name = "openshell-sandbox" version = "0.0.0" dependencies = [ + "anyhow", + "async-trait", + "base64", + "bytes", + "capctl", "clap", - "futures", + "hex", + "ipnet", + "landlock", + "libc", "miette", "nix 0.29.0", + "openshell-binary-identity", "openshell-core", - "openshell-extension-core", + "openshell-isolation-interface", "openshell-ocsf", "openshell-policy", - "openshell-supervisor-middleware", - "openshell-supervisor-middleware-builtins", - "openshell-supervisor-network", - "openshell-supervisor-process", - "prost", - "prost-types", + "openshell-sandbox-backend", + "rand 0.10.2", + "rcgen", + "rustix 1.1.4", "rustls", + "rustls-pemfile", + "seccompiler", "serde", "serde_json", - "temp-env", + "sha2 0.10.9", + "socket2", "tempfile", "tokio", - "tokio-tungstenite 0.26.2", + "tokio-rustls", + "tokio-stream", "tonic", "tracing", - "tracing-appender", "tracing-subscriber", +] + +[[package]] +name = "openshell-sandbox-backend" +version = "0.0.0" +dependencies = [ + "async-trait", + "hyper-util", + "libc", + "openshell-core", + "openshell-isolation-interface", + "prost", + "protoc-bin-vendored", + "rcgen", + "rustix 1.1.4", + "rustls", + "rustls-pemfile", + "serde", + "serde_json", + "sha2 0.10.9", + "socket2", + "thiserror 2.0.18", + "tokio", + "tokio-rustls", + "tokio-stream", + "tonic", + "tonic-prost", + "tonic-prost-build", + "tower", + "tracing", "uuid", ] @@ -4593,6 +4674,41 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "openshell-supervisor" +version = "0.0.0" +dependencies = [ + "clap", + "futures", + "miette", + "nix 0.29.0", + "openshell-core", + "openshell-extension-core", + "openshell-isolation-interface", + "openshell-ocsf", + "openshell-policy", + "openshell-sandbox-backend", + "openshell-supervisor-middleware", + "openshell-supervisor-middleware-builtins", + "openshell-supervisor-network", + "openshell-supervisor-process", + "prost", + "prost-types", + "rustix 1.1.4", + "rustls", + "serde", + "serde_json", + "temp-env", + "tempfile", + "tokio", + "tokio-tungstenite 0.26.2", + "tonic", + "tracing", + "tracing-appender", + "tracing-subscriber", + "uuid", +] + [[package]] name = "openshell-supervisor-middleware" version = "0.0.0" @@ -4630,6 +4746,7 @@ name = "openshell-supervisor-network" version = "0.0.0" dependencies = [ "apollo-parser", + "async-trait", "aws-credential-types", "aws-sigv4", "aws-smithy-runtime-api", @@ -4645,7 +4762,9 @@ dependencies = [ "libc", "miette", "noyalib", + "openshell-binary-identity", "openshell-core", + "openshell-isolation-interface", "openshell-ocsf", "openshell-policy", "openshell-supervisor-middleware", @@ -4682,25 +4801,20 @@ name = "openshell-supervisor-process" version = "0.0.0" dependencies = [ "anyhow", + "async-trait", "base64", "bytes", - "capctl", "hex", - "ipnet", - "landlock", "libc", "miette", "nix 0.29.0", "openshell-core", + "openshell-isolation-interface", "openshell-ocsf", - "openshell-policy", "rand 0.10.2", "russh", - "rustix 1.1.4", - "seccompiler", "serde_json", "sha2 0.10.9", - "socket2", "tempfile", "tokio", "tokio-stream", @@ -5725,6 +5839,7 @@ dependencies = [ "pem", "rustls-pki-types", "time", + "x509-parser", "yasna", ] @@ -7175,7 +7290,6 @@ version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96374855068f47402c3121c6eed88d29cb1de8f3ab27090e273e420bdabcf050" dependencies = [ - "futures", "parking_lot", ] @@ -8746,6 +8860,7 @@ dependencies = [ "lazy_static", "nom", "oid-registry", + "ring", "rusticata-macros", "thiserror 1.0.69", "time", diff --git a/Cargo.toml b/Cargo.toml index 9fbcc9a8c8..888db191b6 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -33,12 +33,13 @@ hyper-util = { version = "0.1", features = ["tokio", "server-auto"] } http = "1.2" http-body = "1.0" http-body-util = "0.1" +h2 = "0.4" # TLS tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "aws_lc_rs"] } rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "aws_lc_rs"] } rustls-pemfile = "2" -rcgen = { version = "0.13", default-features = false, features = ["crypto", "pem", "aws_lc_rs"] } +rcgen = { version = "0.13", default-features = false, features = ["crypto", "pem", "aws_lc_rs", "x509-parser"] } webpki-roots = "1" rustls-native-certs = "0.8" @@ -106,6 +107,7 @@ base64 = "0.22" sha2 = "0.10" rand = "0.9" jsonwebtoken = { version = "10", features = ["aws_lc_rs"] } +zeroize = { version = "1", features = ["derive"] } getrandom = "0.3" aws-lc-rs = "1.16" spiffe = { version = "0.15", default-features = false, features = ["workload-api-jwt", "jwt-verify-rust-crypto", "tracing"] } diff --git a/README.md b/README.md index 70e6e5c09f..3237c169f9 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,8 @@ uv add openshell Deploy the OpenShell gateway into a Kubernetes cluster from the OCI chart published to GHCR: ```bash -helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true ``` See [`deploy/helm/openshell/README.md`](deploy/helm/openshell/README.md) for available versions, dev tag conventions, and configuration. diff --git a/architecture/build.md b/architecture/build.md index a5dcfe9241..0ac80c859e 100644 --- a/architecture/build.md +++ b/architecture/build.md @@ -16,7 +16,8 @@ OpenShell builds these main artifacts: | Python SDK wheel | `python/openshell` | | TypeScript SDK package | `sdk/typescript` | | Gateway container image | `deploy/docker/Dockerfile.gateway` | -| Supervisor container image | `deploy/docker/Dockerfile.supervisor` | +| Sandbox runtime binary and container image | `crates/openshell-sandbox` and `deploy/docker/Dockerfile.sandbox` | +| Supervisor binary and container image | `crates/openshell-supervisor` and `deploy/docker/Dockerfile.supervisor` | | Helm chart | `deploy/helm/openshell` | | VM driver/runtime assets | `crates/openshell-driver-vm` | | Published docs site | `docs/` rendered by Fern config in `fern/` | @@ -25,21 +26,11 @@ Sandbox community images are built outside this repository. ## Build Features -Rust builds require Rust 1.94 or newer. TLS and certificate generation use -AWS-LC, including the CLI and standalone examples. Native and cross-build -environments must provide the C toolchain required by aws-lc-sys; the Nix -development shells provide static AWS-LC libraries. - -SQLx uses AWS-LC with native certificate roots. The server enables -`sqlx-core/rustls-native-certs` directly because SQLx's facade does not expose -that root selection independently of the crypto provider. Credential storage -continues to use the same AES-256-GCM envelope format across backend changes. - Anonymous telemetry emission is gated behind a default-on `telemetry` Cargo feature. It is defined in `openshell-core` (where the emission code, HTTP client, and endpoint live) and forwarded by the binary crates that emit or -collect telemetry: `openshell-gateway`, `openshell-sandbox` -(supervisor), and `openshell-driver-vm`. Every crate depends on +collect telemetry: `openshell-gateway`, `openshell-sandbox`, +`openshell-supervisor`, and `openshell-driver-vm`. Every crate depends on `openshell-core` with `default-features = false`, so the binary crate's feature is the single switch that enables `openshell-core/telemetry` for its build graph. In-process drivers (`docker`, `kubernetes`, `podman`) inherit the @@ -57,8 +48,7 @@ Cargo cannot subtract a single default feature, so each of the three binary crates also defines a `defaults-without-telemetry` alias listing every default except `telemetry`. Telemetry-free builds use `--no-default-features --features defaults-without-telemetry` and stay correct -as the default set grows, instead of dropping unrelated defaults the way a bare -`--no-default-features` does on `openshell-sandbox`. The alias is a keep-list, +as the default set grows. The alias is a keep-list, not a switch: enabling it on top of the defaults would otherwise yield a telemetry-on binary that reads as telemetry-free, so each crate root carries a `compile_error!` for the `telemetry` + `defaults-without-telemetry` combination. @@ -73,7 +63,7 @@ roots through `webpki-roots` plus locally-installed CAs from the system bundle. Building without `bundled-ca-roots` switches to the platform trust store via `rustls-native-certs` and excludes bundled Mozilla root crates such as `webpki-roots` and `webpki-root-certs` from the dependency graph. The -`system-ca-roots` feature alias on `openshell-sandbox` includes all other +`system-ca-roots` feature alias on `openshell-supervisor` includes all other defaults (currently `telemetry`) except `bundled-ca-roots`, so Linux distribution builds (e.g. RPM) can use `--no-default-features --features system-ca-roots` without manually re-adding @@ -96,38 +86,19 @@ The gateway bundles z3 into the release binary so Linux packages, standalone tarballs, and gateway images do not depend on distro-specific z3 shared-library SONAMEs. -The supervisor is the one binary whose libc is selectable, because it is the one -binary executed inside a userland OpenShell does not control. `SUPERVISOR_LIBC` -chooses between `musl` (default) and `glibc-static`. Both produce a fully static -binary; the choice does not change the runtime layout or the supervisor image base. -Static linkage is a hard requirement rather than a preference, so both variants -are verified by `tasks/scripts/verify-static-binary.sh`, which fails the build on -any `PT_INTERP` or `DT_NEEDED` entry. - -The two variants differ only in build-time constraints: - -| | `musl` (default) | `glibc-static` | -|---|---|---| -| Cross-compiles | yes, via `cargo zigbuild` | no — must build natively per architecture | -| Host requirement | zig + cargo-zigbuild | glibc static libraries (`glibc-static` on Fedora/RHEL, `libc6-dev` on Debian/Ubuntu) | -| libc license | MIT | LGPL-2.1-or-later, statically linked | - -`cargo zigbuild` cannot produce the `glibc-static` variant: `zig cc` accepts -`-static` for `*-linux-gnu` targets and emits a dynamically linked binary -anyway. The staging script therefore refuses to cross-compile that variant -instead of silently degrading linkage. - -Selecting `glibc-static` statically links LGPL glibc into a redistributed -binary, which carries relinking obligations that musl (MIT) does not. Treat the -default as the shipping configuration unless that has been reviewed. +The workload-side `openshell-sandbox` binary is statically linked with musl so +drivers can stage it into an arbitrary agent image without depending on that +image's libc. The supervisor is also statically linked: release images use the +default musl build, while distribution-specific builds may select the +`glibc-static` variant. ## Container Builds The Docker image pipeline is a two-step flow: build the Rust binary natively for the target architecture, then assemble the container image from the -prebuilt binary. The gateway image is built from `deploy/docker/Dockerfile.gateway` -and the supervisor image from `deploy/docker/Dockerfile.supervisor`. Neither -Dockerfile compiles Rust — both copy a staged binary out of +prebuilt binary. The gateway, sandbox, and supervisor images use distinct +Dockerfiles under `deploy/docker/`. None of the Dockerfiles compile Rust; they +copy staged binaries out of `deploy/docker/.build/prebuilt-binaries//` into the final image. Local binary staging is driven by `tasks/scripts/stage-prebuilt-binaries.sh`. Because @@ -157,28 +128,6 @@ in platform-specific Nix development shells through reusable workflows and the shared `build-rust-binary` action. The image build downloads each binary artifact into the staging directory before running Buildx. -The Nix flake exposes one development shell with target-specific toolchains. -The shared `mkToolchain` function in `nix/toolchain/default.nix` assembles native -libraries and Cargo environment settings. Linux and Darwin modules select the -compiler and sysroot and generate the compiler wrapper for their platform. -The `nix/toolchain/glibc-2.28/` directory contains the pinned glibc build, -GCC environment, and sysroot assembly used by GNU Linux targets. -The glibc build reuses a pinned historical Nixpkgs recipe with current build -tools; its headers, shared libraries, and static archives are assembled into -the sysroot from separate outputs. -Each toolchain supplies its compiler driver, assembler, archiver, native -libraries, and Cargo environment through derivation passthru. The shell omits -an implicit host C compiler; Cargo builds select the appropriate tools with -`--target`. GNU targets use a glibc 2.28 sysroot and static GCC runtimes, while -musl targets produce static executables. - -On macOS, the shell also provides a native Darwin toolchain with static Z3 -and AWS-LC. Its Clang driver uses the pinned, unprocessed Apple SDK so system -library stubs, including libiconv and libc++, retain their Apple install names. -System libraries and frameworks remain dynamically linked. The deployment -target matches the Nix host platform's minimum macOS version. The Rust toolchain -does not propagate Nix's replacement system libraries into the link environment. - Gateway and supervisor binaries staged into branch E2E, Release Dev, and Release Tag images are compiled through `cargo auditable` (pinned in `mise.toml`), which embeds a `.dep-v0` section describing the Rust dependencies actually compiled @@ -189,15 +138,14 @@ is a different artifact from the source SBOM produced by `syft dir:.` in `tasks/sbom.toml`, which describes the checkout, and from the image SBOM attestation below, which describes a published image. -The shared binary build action uses the default Nix shell and compiles release -artifacts with `cargo auditable build --target `. Verification and upload -read binaries from `target//release/`. +The shared binary build action compiles release artifacts with `cargo auditable`. Branch E2E, Release Dev, and Release Tag image jobs stage those same artifacts instead of rebuilding binaries in Docker. Each binary build scans its output with Syft and requires at least one decoded Cargo package before uploading the -artifact. The action checks each binary's `--version` output and leaves its -linkage as produced by the Nix toolchain, without post-link rewriting or -platform-specific linkage checks. The CI image gains the pinned `cargo-auditable` +artifact. Darwin builds replace Nix's `libiconv` load command with the macOS +system install name, ad-hoc sign the modified binary, and fail if `otool -L` +reports any remaining `/nix/store` dependency. Runtime and Syft verification +run after that normalization. The CI image gains the pinned `cargo-auditable` tool through `mise install --locked` but ships no auditable OpenShell binary of its own. @@ -219,8 +167,6 @@ Runtime layout: gateway binaries must not reference `GLIBC_*` symbols newer than `GLIBC_2.28`; release workflows verify this before publishing artifacts. The gateway bundles z3, so the image does not need a distro-provided z3 runtime. - The base is pinned to a multi-architecture digest; distro security updates - require refreshing that digest and rebuilding the gateway image. - **VM driver**: host GNU-linked binary installed at `/usr/libexec/openshell/openshell-driver-vm` in Linux packages and published as a release artifact. Linux GNU VM driver binaries must not reference @@ -231,16 +177,12 @@ Runtime layout: cache action runs. An explicitly configured VM runtime bundle is required to contain every non-empty embedding input; the driver build fails before packaging when an input is absent or empty. -- **Supervisor**: Alpine base with `nftables`; base packages are upgraded before - installing firewall tools to pick up distro security fixes. Static binary at - `/openshell-sandbox` (musl by default; see `SUPERVISOR_LIBC` above). Static - linkage keeps the binary usable when the image is mounted/extracted into - sandbox environments (Docker extraction, Podman image volumes, Kubernetes - init-container copy-self), whose libc and glibc version are not known at build - time, while `nftables` supports Kubernetes supervisor sidecar egress - enforcement. The VM driver bundles its own supervisor build - (`tasks/scripts/vm/build-supervisor-bundle.sh`) and does not read - `SUPERVISOR_LIBC`. +- **Sandbox**: Alpine-based `openshell/sandbox` image containing the static + musl `/openshell-sandbox` binary and its static VM guest-init helper. + Drivers stage this binary into the workload trust domain. +- **Supervisor**: Debian-based `openshell/supervisor` image containing only the + dynamically linked GNU `/openshell-supervisor` binary. GNU supervisor builds + must not reference `GLIBC_*` symbols newer than `GLIBC_2.28`. Gateway image builds bake the corresponding supervisor image tag into the gateway binary so Docker sandboxes do not depend on `:latest` by default. @@ -370,32 +312,12 @@ Triggers differ by workflow: `.github/workflows/workflow-security.yml` runs on `.github/workflows/codeql.yml` runs nightly on the default branch (`main`) via `schedule`, with `workflow_dispatch` kept for manual diagnostics; and `.github/workflows/codex-security.yml` runs on pushed `v*.*.*-pre.*` tags, and is -also callable through `workflow_call` and `workflow_dispatch`. CodeQL has no -automatic `pull_request`, `merge_group`, or push trigger, so it reports +also callable through `workflow_call` and `workflow_dispatch`. CodeQL does +not run on `pull_request`, `merge_group`, or pushes to `main`, so it reports repository-level Code Scanning state on the default branch instead of per-PR results, and its four-language matrix stays off the per-change critical path. Codex Security is release-scoped rather than change-scoped, so it never runs on -a pull request or merge group automatically. - -`.github/workflows/security-scan.yml` is a manual and reusable parent for Codex -Security, CodeQL, Trivy, Cargo Deny, and Actionlint/Zizmor. Each child runs -independently against the supplied pre-release tag; Codex retains its cumulative -stable-to-candidate range. The parent forwards OCI references to Trivy and -publishes SARIF, including Codex results on manual parent runs. Codex publishes -against the candidate commit on `main`; the other SARIF producers use the -candidate tag and commit. Cargo Deny retains its self-hosted runner and CI -container. Dependency Review and Trivy Changes remain separate comparison -workflows. Existing standalone triggers remain active. - -The parent enforces HIGH/CRITICAL findings for Codex, CodeQL, Trivy, and Zizmor. -Its `allow-high-critical` input disables that threshold while keeping execution -and publication failures fatal. Each scanner evaluates its existing report -after publication. Actionlint stays informational; Cargo Deny runs its native -advisory check, which is independent of the severity override. Standalone -finding policies are unchanged. Child concurrency groups distinguish the -scanner as well as the caller, so sibling workflows cannot cancel one another. -Codex retains its repository-wide release qualification group. -See [CI.md](../CI.md#run-the-security-scans-together) for invocation examples. +a pull request or merge group. - **Actionlint and Zizmor** analyze the workflow definitions themselves. Repository configuration lives in `.github/actionlint.yml` (self-hosted runner @@ -418,8 +340,8 @@ See [CI.md](../CI.md#run-the-security-scans-together) for invocation examples. integration targets the cfg override does not reach. Examples remain in scope, and E2E test code stays excluded because `e2e/` is not an analyzed path. Only Go requires a build; the other languages use build mode `none`. Analysis runs - on the nightly schedule, by manual dispatch, or through `workflow_call`. - Results are uploaded to Code Scanning and always retained as workflow artifacts. + on the nightly schedule or by manual dispatch. Results are uploaded to Code + Scanning and always retained as workflow artifacts. - **Codex Security** qualifies release candidates rather than individual changes. The job installs a pinned `@openai/codex-security` release into the runner temp directory before the repository is checked out and invokes it by @@ -485,9 +407,7 @@ See [CI.md](../CI.md#run-the-security-scans-together) for invocation examples. `CODEX_SECURITY_STATE_DIR`, where every shell command the agent ran is recorded. No command at all is the signal that the sandbox failed to start. -Findings do not fail standalone informational runs; reusable callers can enable -HIGH/CRITICAL enforcement with `fail-on-findings`. Scanner and build failures -always fail. A scanner that +Findings never fail these checks; scanner and build failures do. A scanner that cannot run, a CodeQL analyzer that does not complete, an unexpected Dependency Graph API error, and a Codex Security range, scan, or export failure are all errors, which keeps an informational check from silently degrading into a no-op. @@ -504,47 +424,19 @@ tags and gating stable promotion on qualification results are part of [RFC 0014](../rfc/0014-release-stability/release-qualification.md) and are not implemented yet. -## Artifact Scanning - -Two entry points share `tasks/scripts/trivy-scan.sh`: a standalone analysis -workflow and a pull-request change gate. Nix supplies Trivy, Helm and `yq`. - -The standalone workflow scans deployment configuration and supplied OCI -references independently of release publication. Detailed JSON reports feed the -differential gate; the summary and published SARIF consolidate configuration -findings across profiles while preserving resource identity and affected profiles. -Images and packaged charts retain separate identities based on their full -references. Publication batches respect GitHub's limit of 20 SARIF runs. - -The PR/merge-group gate scans base and candidate with the same scanner and rejects -new `HIGH` or `CRITICAL` configuration findings. Its stable -`OpenShell / Trivy Changes` status succeeds when nothing relevant changed. -For PRs, the baseline is the first parent of the exact merge commit being tested, -not the event's potentially older base SHA or the current branch tip. Change -detection and both scans use that same immutable pair, including on reruns. -Merge groups and manual runs retain their explicit baseline and always scan. -Image CVEs need the standalone scan. The reporting and gate invariants are: - -- A structurally invalid Trivy report is an error, not an empty finding set. -- Scanner failures prevent publication of incomplete analyses; findings alone - do not prevent publishing complete reports. -- Findings compare per profile against the same baseline profile, by semantic - identity and count rather than line number; a profile absent from the baseline - falls back to that identity's maximum across all profiles. -- The candidate's ignore file is validated, but the baseline's policy applies to - both scans, so an exemption takes effect only after merge. - -See [CI.md](../CI.md#artifact-scanning) for profiles, report paths, severity -settings, exceptions, and the contributor and maintainer workflows. +See `CI.md` for the contributor workflow, labels, and maintainer merge-queue workflow. ## Docs Site -Published docs live in `docs/`, and Fern site configuration lives in `fern/`. See [fern/README.md](../fern/README.md) for the source layout, local development commands, version model, and publishing workflows. +Published docs live in `docs/`. Navigation lives in `docs/index.yml`. Fern site +configuration, components, theme assets, and publish settings live in `fern/`. -## Validation Expectations +Use `mise run docs` for strict validation and `mise run docs:serve` for local +preview. PR previews are produced by `.github/workflows/branch-docs.yml` when +Fern credentials are available. Production docs publish from the release tag +workflow. -Rust CI rejects stale or modified Cargo lockfiles and runs Clippy for the -workspace, E2E crate, and standalone examples. +## Validation Expectations - Run `mise run pre-commit` before committing. - Run `mise run test` after code changes. diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 3018e6447d..a2bb25c312 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -1,10 +1,18 @@ # Compute Runtimes Compute runtimes create, stop, start, delete, and watch sandbox workloads for the -gateway. Supervisor-controlled runtimes start a workload that runs the -`openshell-sandbox` supervisor, which enforces the sandbox contract locally. -Driver-controlled runtimes apply the canonical sandbox policy while -provisioning and report workload readiness directly. +gateway. A supported runtime provisions `openshell-sandbox` inside the workload, +`openshell-supervisor` outside it, a protected channel between them, and an +independent outer network fence. Drivers do not implement policy evaluation. + +Podman provisions a paired workload and supervisor container using its native +libpod API. The workload uses `network=none`; the external supervisor alone joins +the configured network. A per-sandbox named volume carries their mutually +authenticated gRPC Unix socket, with supervisor credentials kept in its separate +filesystem. Both containers run as the resolved non-root identity with all +capabilities dropped. They share only a user namespace for volume ownership, +not PID, mount, or network namespaces. Podman owns paired lifecycle and health; +the common protocol owns process, identity, TCP, DNS, and forwarding semantics. ## Driver Contract @@ -12,11 +20,15 @@ Each runtime receives a sandbox spec and canonical policy from the gateway and is responsible for: - Selecting the sandbox image. -- For supervisor-controlled runtimes, injecting sandbox identity and gateway - callback configuration, supplying callback credentials, and providing the - supervisor binary or image. -- For runtimes without the standard supervisor, validating and applying the - canonical policy before launching the workload. +- Resolving an immutable non-root sandbox identity before workload creation. +- Supplying separate sandbox and supervisor bootstrap material. +- Delivering `openshell-sandbox` to the workload and `openshell-supervisor` only + to the external supervisor placement. +- Provisioning protected control and boundary configs plus a private Unix socket, + TLS-authenticated TCP, or vsock transport when the supervisor is separated. + Runtime-specific code supplies immutable resource claims and transport + coordinates; the shared boundary protocol supplies lifecycle, exec, signaling, + forwarding, and binary identity semantics. - Forwarding the exact canonical main-process argv and TTY mode without shell reconstruction. The sandbox-level environment and policy workspace apply to the main process. @@ -127,32 +139,19 @@ defines the available implementation set, while the runtime consumes a generic registry. Adding or removing a compiled driver therefore changes registration rather than the server's selection flow. Alternate gateway binaries can install their own `ComputeDriverFactory` registrations and hand the completed registry -to `run_cli_with_compute_drivers`. Factories expose the same side-effect-free -configuration validation to package preflight and runtime startup, receive only -the selected `[openshell.drivers.]` table, and return either an in-process -driver or a gateway-managed remote endpoint when built. Existing out-of-tree -factories that implement only runtime construction remain source-compatible and -can start normally, but package preflight fails closed until they explicitly -advertise and implement side-effect-free configuration validation. If a selected config file -omits the selector, preflight validates each configured table that belongs to an -auto-detectable registration. It does not run runtime detection probes because -those probes can connect local sockets or launch bounded discovery commands. -Preflight never builds a driver or connects to its transport. The server constructs the common runtime -adapter and snapshots `GetCapabilities` for either result. A configured UDS -endpoint still takes precedence over a compiled registration with the same name. - -The `openshell-gateway` composition crate exposes one feature per first-party -registration: `compute-driver-kubernetes`, `compute-driver-docker`, -`compute-driver-podman`, `compute-driver-vm`, and `compute-driver-mxc`. Builds -can enable any subset. MXC links only on Windows; the other four features -install rejection stubs on Windows and link their drivers on other platforms. -The default `in-tree-compute-drivers` feature remains an alias for all five, -preserving each platform's default registrations. -`openshell-server` has no compute driver dependencies or backend-name dispatch. -Protocol-only gateway builds disable the default features and link no -compute-driver crates. E2E lanes compose that gateway with Docker, Podman, -Kubernetes, and VM driver executables over the public UDS gRPC contract so an -in-tree driver cannot silently depend on a server-only API. +to `run_cli_with_compute_drivers`; factories receive merged driver config and +return either an in-process driver or a gateway-managed remote endpoint. The +server constructs the common runtime adapter and snapshots `GetCapabilities` +for either result. A configured UDS endpoint still takes precedence over a +compiled registration with the same name. + +The `openshell-gateway` composition crate groups first-party registrations +behind the `in-tree-compute-drivers` feature. `openshell-server` has no compute +driver dependencies or backend-name dispatch. Protocol-only gateway builds +disable the composition feature and link no compute-driver crates. E2E lanes +compose that gateway with Docker, Podman, Kubernetes, and VM driver executables +over the public UDS gRPC contract so an in-tree driver cannot silently depend +on a server-only API. ## Stop and Start Lifecycle @@ -177,6 +176,12 @@ same resource. The gateway requires a fresh supervisor session before a starting sandbox returns to `Ready`; stale driver snapshots and supervisor sessions cannot promote a `Stopped` row. +Runtime credentials are generation-scoped and memory-only after launch. A +supervisor or Sandbox Runtime process replacement does not resume a running +generation. Planned upgrades stop the sandbox first; the following start mints +a fresh session, TLS identity, and credential pair. An unexpected replacement +leaves the old workload on the normal fail-closed disconnect path. + A driver stop operation does not complete while its backend still reports an in-progress stop. This prevents an immediate start from racing the previous run's delayed exit event and regressing the new run to `Error`. @@ -260,11 +265,11 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`. | Runtime | Best fit | Sandbox boundary | Notes | |---|---|---|---| -| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. | -| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. | -| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. | -| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. | -| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_driver = ""` entry with `[openshell.drivers.].socket_path`, or at launch time by pairing `--compute-driver ` with `--compute-driver-socket=`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. | +| Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. | +| Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. | +| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses one namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. | +| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. | +| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = [""]` entry with `[openshell.drivers.].socket_path`, or at launch time by pairing `--drivers ` with `--compute-driver-socket=`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. | Per-sandbox CPU and memory values currently enter the driver layer through template resource limits. Docker and Podman apply them as runtime limits. @@ -291,42 +296,17 @@ operator override because they place gateway-host filesystem state inside the sandbox and can negate OpenShell workspace isolation and filesystem-policy controls. Driver-owned supervisor, token, and TLS bind mounts stay reserved. -Network features follow the existing driver/substrate split. Compute drivers -advertise only the runtime mechanics they can guarantee: namespace and -capability ownership, DNS/TCP capture installation, and coupled -restart ordering. The shared supervisor remains the sole owner of DNS -eligibility, synthetic mappings, process authorization, destination filtering, -pinned dialing, relay behavior, and OCSF decisions. Docker and Podman advertise -`policy-dns-transparent-tcp`; other runtimes reject explicit TCP policy until -they implement and validate the same complete contract. The capability marker -is driver-owned supervisor input and is removed from workload environments. - -Kubernetes, Docker, and Podman share one AppArmor configuration model: -`RuntimeDefault`, `Unconfined`, or `Localhost/`. Each driver translates -that model to its native API and rejects an explicitly requested confined -profile when its backend reports AppArmor unavailable. Docker keeps its -historical explicit `Unconfined` default. Podman sends no override when the -field is omitted, preserving the runtime-selected profile; development paths -that require the supervisor's namespace mount setup opt into `Unconfined` -explicitly. The Helm chart independently uses `Unconfined` for Kubernetes. - -Corporate proxy settings are driver-owned supervisor inputs. Docker, Podman, -and VM propagate `https_proxy`, `no_proxy`, an optional root-only auth file, -and the explicit cleartext-Basic-auth acknowledgement without allowing -workload environment to override them. Podman and VM can also project an -operator CA bundle for an HTTPS or TLS-intercepting proxy. The VM driver validates and -stages its credential and CA bundle under fixed guest paths, then forwards -those paths through the protected supervisor argument file rather than the -guest environment. Local containers project provider SPIFFE -through a dedicated host UNIX-socket parent mount. A VM cannot safely expose -that host socket: it accepts only a separately operated, concrete TCP listener -when `provider_spiffe_allow_guest_tcp = true` explicitly acknowledges guest -access. Host-only sockets are never implicitly forwarded to VM guests. +Network features follow the driver/substrate split. Drivers own only the outer +fence and protected channel. The sandbox owns seccomp notification, local DNS, +socket virtualization, process observation, and binary identity. The supervisor +owns DNS eligibility, policy authorization, destination filtering, upstream +dials, relay behavior, credential rewriting, and OCSF decisions. No supported +path requires nftables, a workload network namespace, proxy environment +variables, added capabilities, or an unconfined AppArmor profile. The Kubernetes deployment packaging has two ownership boundaries. The gateway chart owns the gateway workload, configuration, Services, PKI, and -cluster-scoped gateway resources. It can retain the legacy combined behavior, -or omit workspace resources. The workspace chart is installed into a +cluster-scoped gateway resources. The workspace chart is installed into a pre-provisioned sandbox namespace and owns only the sandbox ServiceAccount, namespaced RBAC, and sandbox ingress NetworkPolicy. Its RoleBinding names the gateway ServiceAccount and namespace explicitly, so the two releases have @@ -339,14 +319,10 @@ can request a specific number of GPUs or the driver-specific default behaviour. For all in-tree drivers, this is equivalent to selecting a single GPU. VM runtime state paths are derived only from driver-validated sandbox IDs -matching `[A-Za-z0-9._-]{1,128}`. Each writable overlay records its effective -sandbox UID/GID so later rootfs cache changes cannot rewrite persisted file -ownership. Unmarked pre-migration overlays recover identity from concrete -overlay or prepared-rootfs state, an explicit operator override, or the current -image account. The driver never assumes `10001:10001`; it preserves that legacy -identity only when persisted state reports it. The gateway-owned VM driver -socket uses a private `run/` directory plus Unix peer UID/PID checks. Standalone unauthenticated TCP -mode is disabled unless explicitly enabled for local development. +matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a +private `run/` directory plus Unix peer UID/PID checks. Standalone +unauthenticated TCP mode is disabled unless explicitly enabled for local +development. Runtime-specific implementation notes belong in the driver crate README: @@ -355,113 +331,57 @@ Runtime-specific implementation notes belong in the driver crate README: - `crates/openshell-driver-kubernetes/README.md` - `crates/openshell-driver-vm/README.md` -The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun -executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves -that identity when it execs the supervisor after mounting and network setup. +The VM guest bootstrap runs once as root to prepare mounts, loopback, and the +safe port-53 sysctl. It then drops to the resolved identity with empty +capability sets and executes `openshell-sandbox` as guest PID 1. ## Supervisor Delivery -The supervisor must be available inside each sandbox workload: +Drivers deliver the two binaries to separate trust domains: | Runtime | Delivery model | |---|---| -| Docker | Bind-mounted local supervisor binary, or a binary extracted from the configured supervisor image. | -| Podman | Read-only OCI image volume by default; host-cached bind mount when `userns` is configured. | -| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. | -| VM | Embedded in the guest rootfs bundle. | +| Docker | A digest-pinned daemon-local volume supplies `openshell-sandbox`; the companion image runs `openshell-supervisor`. | +| Podman | Existing driver behavior; not converted by this stack. | +| Kubernetes | A non-root init container stages `openshell-sandbox` into a memory volume; a directly managed Pod runs `openshell-supervisor`. | +| VM | `openshell-sandbox` is embedded in the guest rootfs; a separately digest-checked native `openshell-supervisor` runs on the host. | | Extension | Defined by the out-of-tree driver. | -Driver-controlled environment variables must override sandbox image or template -values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS -paths, and command metadata. +Driver-controlled sandbox bootstrap must override image or template values for +sandbox identity, command metadata, resolver configuration, and public trust +paths. Gateway endpoints, callback credentials, policy, and private TLS material +belong only to the supervisor placement. ## Process Identity -The gateway preserves whether each policy process field was omitted. The active -driver then supplies one authoritative identity input to the supervisor: - -- Docker and Podman inspect the final sandbox image, pin container creation to - its immutable image ID, and pass its raw OCI `Config.User`. Docker also - resolves the workspace from OCI `Config.WorkingDir` during that inspection. -- Kubernetes passes its platform-resolved numeric UID/GID, including OpenShift - SCC-derived values. -- VM keeps its existing guest identity behavior. - -Explicit numeric workload identities may use any Linux UID/GID from `1` -through `u32::MAX - 1`. UID/GID `0` remains prohibited as root, and -`u32::MAX` remains prohibited because Linux APIs and POSIX ACLs use it as an -invalid identity sentinel. Infrastructure identities use separate validation: -the Kubernetes network proxy UID remains at least `1000` and must not match the -workload UID because its traffic bypasses the pod egress fence. - -For Docker and Podman, policy values take precedence independently. An omitted -`run_as_user` or `run_as_group` falls back to the corresponding identity from -the image. The supervisor resolves names from the image's `/etc/passwd` and -`/etc/group` before readiness, preserves declared name or numeric components, -and uses the same privilege-drop path for direct and SSH children. When a -declaration omits the group, the supervisor fills it with the user's numeric -primary GID. It does not rewrite the account files. - -Docker uses an absolute OCI working directory as the workspace. An -empty, root (`/`), or explicit `/sandbox` declaration uses `/sandbox`, which -OpenShell creates and owns as a compatibility workspace. Any other workdir must already -exist in the immutable image without symlink components. The completed -identity, including supplementary groups, must already be able to traverse -every parent and write and enter the workdir; OpenShell does not change that -directory's ownership or mode. A one-shot validator drops to that identity and -uses kernel effective-access checks so POSIX ACL and LSM decisions are honored. -Path checks reserve the standard OCI runtime namespaces under `/proc`, `/sys`, -and `/dev`, while separate collision checks are derived from actual OpenShell -control paths. -Docker performs the check in the final container before workload launch and -rejects image `VOLUME` declarations that would mask the workdir ancestry. The -resolved workspace is the child cwd and `HOME`; when -`filesystem.include_workdir` is enabled, it becomes the automatic writable -policy path. Podman, Kubernetes/OpenShift, and VM retain their existing -`/sandbox` workspace behavior. - -Sandbox creation fails before the workload becomes ready when a required image -identity is absent, malformed, unknown, ambiguous, or resolves to UID/GID 0. -The supervisor itself remains root so it can establish isolation before -starting unprivileged children. - -Kubernetes can run the supervisor in the default combined topology or in a -sidecar topology. Combined mode keeps network and process supervision in the -agent container. Sidecar mode runs network enforcement, the proxy, and gateway -session in a dedicated sidecar, while the agent container runs only the -process-supervision leaf and launches the user workload after the sidecar -serves bootstrap state over a local control socket. The network sidecar owns -gateway credentials and sends policy plus workload-facing provider environment -state to the process leaf over that socket. It also streams provider -environment updates after settings polls so future process sessions see -updated provider env without giving the process leaf gateway access. The -pre-workload process supervisor is the only accepted control client: the -network sidecar verifies its UID, GID, and PID with peer credentials, removes -the listener after accepting it, and ignores workload-supplied relay targets. -SSH relays use a Linux abstract socket and verify its peer PID against that -authenticated process-supervisor connection, so workload filesystem access -cannot replace the relay endpoint. Either supervisor exits when this control -connection closes. This couples their restart lifecycle and prevents a workload -that survives an isolated network-sidecar restart from becoming the next -authoritative control client. In sidecar mode, an init container performs the -privileged pod-network nftables setup with -`NET_ADMIN`. The default binary-aware network sidecar runs as UID 0 without -`NET_ADMIN` and adds `SYS_PTRACE` plus `DAC_READ_SEARCH` so it can resolve -cross-UID workload process/binary identity through shared `/proc`. Operators -can set the sidecar `process_binary_aware_network_policy` flag false to run the -sidecar as the configured non-root proxy UID, omit both inspection capabilities, -and downgrade network policy to endpoint/L7 matching without `policy.binaries`. -The init path applies nftables as individual commands so optional conntrack and -log expressions can fail without rolling back the required table, chain, and -reject rules. -The agent container runs as the resolved sandbox UID/GID with no added Linux -capabilities. Sidecar mode preserves gateway session and SSH behavior, but -treats the process leaf as network-only: Landlock filesystem policy and child -seccomp still apply where supported, while process privilege dropping and -supervisor identity mount isolation do not run because the agent container is -already unprivileged. Sidecar pods use a shared process namespace so the -network sidecar can resolve workload process and binary identity through -`/proc/`. +The gateway preserves whether each policy process field was omitted and passes +the admitted selectors to the driver. The driver resolves one exact UID, GID, +and supplementary-group set before creating the immutable workload: + +- Docker pins the image ID, resolves policy selectors against the image's + `/etc/passwd` and `/etc/group`, and validates its OCI working directory. +- Kubernetes uses platform-resolved numeric values, including OpenShift + namespace ranges. +- VM uses the configured numeric guest identity. + +UID/GID zero and `u32::MAX` are invalid. The sandbox and every child start with +the resolved identity and zero capability masks; neither process performs an +in-workload UID transition. Identity-changing policy updates require sandbox +recreation, while other policy updates remain live. + +Docker uses an absolute OCI working directory as the workspace. Empty, root, +and explicit `/sandbox` values select `/sandbox`; other paths must already +exist without symlink or reserved-mount collisions and must be usable by the +resolved identity. Kubernetes and VM use `/sandbox`. + +The Kubernetes driver creates the namespace-wide empty-egress workload fence +before a suspended Sandbox CR, then provisions split immutable bootstrap +Secrets, the private runtime Service, and a gated supervisor Pod. A +non-root init container stages `openshell-sandbox` and one-use bootstrap files +into memory volumes. The workload Pod never mounts supervisor or gateway +credentials. The driver removes its scheduling gate only after the companions +exist; measured confirmation and supervisor-session registration gate public +readiness. ## Images diff --git a/architecture/gateway.md b/architecture/gateway.md index 3881618db1..0000a4f961 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -343,7 +343,7 @@ Compute-driver, credential-driver, gateway-interceptor, and supervisor-middleware services are compiled contracts for internal extension boundaries, not public gateway RPCs. The current public inventory has 74 methods, 278 messages, and 12 enums -(`8ac68c71d93e6a5e56406b8df1882ee40c6066270969e03eb99803f0e6396fc1`). +(`c6b4e1ae4d09e5d7de3831601953aa36901e51a24913a86f00559c53511fedde`). The removed `NetworkBinary.harness` field remains reserved by number and name, so protobuf implementations cannot reuse its wire slot or source identifier. The durable-policy compatibility decoder reads the former boolean before Prost @@ -902,7 +902,7 @@ system entry instead of pretending to delete package-manager owned state. - Compute runtimes own the mechanics of starting workloads and injecting callback configuration. Local Docker, Podman, and VM callback endpoints can be derived from their fixed host aliases. Kubernetes requires an explicit - endpoint from deployment topology; Helm renders it from the gateway Service + endpoint from driver placement; Helm renders it from the gateway Service name and namespace rather than inferring it from sandbox placement. - Docker-backed local gateways use Docker's `host-gateway` callback alias on macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback diff --git a/architecture/sandbox.md b/architecture/sandbox.md index 5b6245c890..af24d7aff7 100644 --- a/architecture/sandbox.md +++ b/architecture/sandbox.md @@ -1,39 +1,91 @@ # Sandbox -A sandbox is the runtime boundary where agent code executes. It is created by a -compute runtime and managed inside the workload by `openshell-sandbox`, the -sandbox supervisor. +A sandbox is the runtime boundary where agent code executes. A compute driver +creates it and connects two dedicated components: `openshell-sandbox` inside +the workload boundary and `openshell-supervisor` outside it. ## Runtime Model -Each sandbox workload has two trust levels: +Each sandbox has three trust levels: -| Process | Role | +| Component | Role | |---|---| -| Supervisor | Starts as root inside the workload, prepares isolation, runs the proxy, fetches config, injects credentials, serves the relay socket, and launches child processes. | -| Agent child | Runs as an unprivileged user with filesystem, process, and network restrictions applied. | - -The supervisor keeps enough privilege to manage the sandbox, but the agent child -loses that privilege before user code runs. On Linux, child setup clears the -capability bounding set during privilege drop so later execs cannot regain -container-granted capabilities. This is fail-closed: the supervisor retains -`CAP_SETPCAP` solely to perform the clear, and spawning the workload or SSH shell -aborts unless the bounding set ends up empty. A `setpcap` `EPERM` is tolerated -only when the set is already empty; any other outcome fails the spawn. +| Supervisor | Owns gateway credentials, admitted policy, L7 proxying, SSH, and gateway relays. It never executes inside the agent workload. | +| Sandbox | Runs as the same non-root identity as the agent, installs the workload seccomp listener, applies the Landlock baseline, owns child processes, and mediates the protected supervisor channel. | +| Agent child | Inherits the sandbox network listener and runs with zero capabilities, `no_new_privs`, Landlock, and the final syscall filter. | + +The runtime grants neither trusted component nor agent child any Linux +capability inside the workload. Drivers resolve one exact non-root UID, GID, +and supplementary-group set before launch. The sandbox and all of its children +use that immutable identity, so no in-workload privilege transition is needed. +The supervisor uses its own driver-defined identity and has no workload-creation +or backend-admin authority. + +The compute driver provisions separate protected configurations and one +mutually authenticated gRPC connection over a private Unix socket, Kubernetes +TCP Service, or VM vsock channel. Independent bidirectional `Exchange` RPCs +carry lifecycle, exec, TCP, and forwarding traffic, while one persistent +bidirectional `Mediate` RPC carries multiplexed DNS traffic. General application +UDP is unsupported; UDP DNS remains mediated by the supervisor. +The sandbox probes HTTP/2 connection liveness every five seconds and closes +connections that miss a ten-second acknowledgement deadline. Closing a +connection freezes the owned workload process tree and cancels its stream +bridges before releasing the exclusive DNS mediation lease. The supervisor has +30 seconds to reconnect, replay attach, and reconfirm the boundary. Every +supervisor process generates an ephemeral instance ID, and the sandbox pins the +first ID it accepts for its process lifetime. The same process can therefore +recover a dropped transport, but a replacement supervisor cannot reuse launch +credentials to claim the existing runtime generation. Confirmation resumes the +workload; expiration terminates it. A credential replacement does not displace +the active connection until the new connection is confirmed. Idle healthy +connections remain usable. +Unauthenticated TLS handshakes have a separate bounded asynchronous pool and +five-second deadline, never consuming authenticated control slots or threads. +The socket broker reserves the TCP control-listener port against workload +connections, including loopback aliases. Unix control listeners reject workload +descendants using kernel peer credentials and process ancestry, while ordinary +workload loopback and Unix services remain available. +NetworkPolicy is an outer reachability fence, not a confidentiality boundary. +Each sandbox generation receives a fresh CA and distinct server/client leaves; +both endpoints bind the same workload identity and immutable driver resource +claims. Driver crates do not appear in generic process, network, SSH, or +session code. + +The supervisor exposes readiness only after the sandbox is confirmed and the +gateway access plane is registered. Driver-owned channel directories limit +reachability, while mutual authentication and channel epochs prevent endpoint +replacement from granting authority. ## Startup Flow -1. The compute runtime starts the workload with sandbox identity, callback - endpoint, TLS or secret material, image metadata, and initial command. -2. The supervisor loads policy and runtime settings from local files or the - gateway, depending on mode. -3. It prepares filesystem access, process restrictions, network namespace - routing, trust stores, and provider credential resolution. -4. It launches the persisted canonical main-process argv and retains its PTY - or pipes in the main-session multiplexer. -5. It starts the policy proxy and local SSH server. -6. It opens a supervisor session back to the gateway for connect, exec, file - sync, config polling, and log push. +1. The driver resolves the immutable workload identity, installs the outer + network fence, and starts `openshell-sandbox` with one-use bootstrap state. +2. The sandbox consumes and unlinks bootstrap material, proves the admitted + runtime posture, and listens on the protected driver channel. It does not + run untrusted code yet. +3. `openshell-supervisor` loads policy and runtime settings from the gateway, + attaches to the sandbox, and verifies the driver's generation and evidence. +4. The sandbox installs its seccomp notification broker and Landlock baseline, + then reports measured confirmation. The supervisor must accept that evidence +before it sends the launch permit. +5. The sandbox starts the canonical process through its single workload + launcher. The supervisor starts SSH and registers its gateway session. +6. Exec, signaling, PTY, DNS, TCP, and loopback-forwarding operations cross the + authenticated channel for the lifetime of the sandbox generation. + +When the admitted main process exits, its status and retained terminal output +remain available. The confirmed sandbox and supervisor-owned access plane continue +to serve policy-authorized exec and loopback forwarding until explicit stop or +delete tears down the boundary and terminates any remaining workload processes. + +Completed exec output handles can be reclaimed, but execution request IDs remain +reserved for the boundary generation. The sandbox accepts at most 4,096 exec +attempts per generation, then rejects new attempts rather than forgetting replay +protection. A disconnected attachment does not authorize another execution. +While an exec handle is retained, independent waits return its stable exit or +signal status, whether or not an output attachment is open or the main process +has exited. Waiting never holds the exec registry lock, so other operations can +still signal or attach to the process. ## Isolation Layers @@ -42,10 +94,10 @@ OpenShell uses overlapping controls rather than a single sandbox primitive: | Layer | Purpose | |---|---| | Filesystem policy | Landlock restricts the paths the agent can read or write. | -| Process policy | The child process runs as a non-root user with reduced privileges. | -| Seccomp | Blocks dangerous syscalls, including raw socket paths that bypass the proxy. | -| Network namespace | Forces ordinary agent egress through the local CONNECT proxy. | -| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and endpoint-bound credential injection. | +| Process policy | Sandbox and children run as one immutable non-root identity with zero capabilities. | +| Seccomp notification | Virtualizes supported INET sockets and sends DNS/TCP decisions to the supervisor without nftables or proxy environment variables. | +| Driver outer fence | Docker `network_mode=none`, a NIC-less VM, or Kubernetes NetworkPolicy prevents any missed or unsupported kernel path from escaping. | +| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. | The supervisor may enrich baseline filesystem allowances for runtime-required paths, such as proxy support files or GPU device paths when a GPU is present. @@ -64,17 +116,88 @@ invokes `wxc-exec`, rather than running a chain with missing implementations. Remove the middleware entries or use a compute driver whose sandbox supervisor receives the gateway middleware registry. -## Network and Provider Access +The mandatory self-protection baseline is separate from optional workload +filesystem policy. It requires Landlock ABI v3, including pathname truncation +protection. Rules cover individually opened root children except `/.openshell`; +the sandbox opens entries relative to a pinned root descriptor without following +symlinks. An image-provided alias cannot grant access to the protected subtree. +The reserved `/.openshell` root must itself be a real directory if present; +a symlink or non-directory aborts preparation so private child mounts cannot +redirect into an allowed subtree. + +## Network and Inference See [Sandbox Limits](sandbox-limits.md) for the current numeric safety ceilings, their ownership, terminal behavior, and known gaps. -All ordinary agent egress is routed through the sandbox proxy. The proxy -identifies the calling binary, checks trust-on-first-use binary identity, rejects -unsafe internal destinations, and evaluates the active policy. On Linux, it -maps an accepted proxy connection back to the workload socket by matching the -complete local-to-remote TCP tuple before resolving every process that owns the -socket inode. +### Standalone network proxy + +`openshell-supervisor --role=network-proxy` runs the policy proxy without an +Isolation Backend or `openshell-sandbox`. It accepts explicit HTTP proxy and +CONNECT requests on a loopback listener and applies the same local Rego rules, +YAML policy data, destination checks, and L7 enforcement used by supervised +sandboxes: + +```shell +openshell-supervisor \ + --role=network-proxy \ + --listen=127.0.0.1:3128 \ + --tls-dir=/tmp/openshell-proxy-tls \ + --policy-rules=/path/to/sandbox-policy.rego \ + --policy-data=/path/to/sandbox-policy.yaml +``` + +The standalone listener cannot observe which process opened a connection, so +this role evaluates endpoint and protocol rules without binary identity. It +does not launch a workload, attach a Sandbox Runtime, fetch gateway policy, +inject provider credentials, or provide exec and lifecycle operations. The +listener is loopback-only. TLS interception writes its generated public CA and +combined trust bundle to `--tls-dir`; when omitted, the supervisor uses a +process-specific directory under the system temporary directory. + +The sandbox installs one seccomp user-notification listener on a dedicated +launcher thread. Every canonical and exec process inherits that listener. It +virtualizes supported INET sockets before they enter the agent FD table, copies +bounded syscall inputs from the notifying task, resolves the calling binary, +and blocks external `connect` until the supervisor returns a policy decision +and relay stream. Connected data stays on ordinary kernel sockets, so the +notification path is limited to socket setup and pointer-bearing operations. +Blocking listener accepts retain native workload socket flags. A broker-owned +watchdog interrupts an accept when its seccomp notification is cancelled or the +broker stops, including when readiness disappears before the accept syscall. +The sandbox reserves `SIGUSR2` with a non-restarting no-op handler for these +broker threads; startup rejects a conflicting handler. This signal disposition +is process-global kernel state, while registrations and cancellation state are +owned by the broker. Workload exec resets the caught handler to its default. +This sandbox runtime requires Linux 6.2 or newer for Landlock ABI v3 and treats +`SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` as mandatory so cancelled +notifications cannot race task-memory writes. + +DNS uses an exact sandbox-local resolver at `127.0.0.53:53`. The driver sets the +nameserver and permits an unprivileged bind to port 53. UDP and TCP DNS requests +are forwarded through the supervisor, which applies hostname-based DNS policy. +DNS sender identity is explicitly unavailable: native writes can come from an +inheriting process or after exec, and neither the connecting binary nor a later +descriptor-owner snapshot proves who sent an already queued query. Consumers +must not use this unavailable identity to grant binary-specific access. TCP +connection authorization still uses decision-time binary identity. + +The sandbox retains only bounded DNS socket-admission records, consumes TCP +records on accept, and reclaims closed UDP records when capacity is reached. +The kernel delivers replies from the configured nameserver address, including +for strict musl and c-ares resolvers. No proxy environment variable, nftables +rule, or workload network namespace setup is part of enforcement. The supervisor +retries failed DNS accepts with backoff, preserving service across a channel +reconnect. + +External TCP opens wait at most 30 seconds for a supervisor decision, then fail +with `ETIMEDOUT` and release their worker quota. An approval is tied to the +original socket identity; replacing the descriptor during policy evaluation +cannot transfer that approval to another socket. + +The outer fence remains mandatory. If notification handling misses a syscall, +loses the supervisor, exceeds a bound, or encounters an unsupported socket +type, the request fails and the driver-owned fence still blocks direct egress. CONNECT and absolute-form forward HTTP are explicit-proxy adapters over the same egress pipeline. Each adapter normalizes its request into an egress intent, and @@ -100,29 +223,6 @@ captured before the bypass fence, mapped back to its workload process, authorize through the same egress pipeline, and dialed only through the pinned addresses. Omitted protocol endpoints retain explicit-proxy behavior. -The DNS store is in-memory and sandbox-local. A combined-supervisor restart also -restarts its workload; before execution, the supervisor advances a persisted -boot epoch and installs only that epoch's synthetic capture ranges. An address -cached from the preceding epoch therefore falls through to the bypass fence -instead of inheriting a new mapping. Policy reload, expiry, wrong ports, direct real-IP access, missing -mappings, or pool exhaustion fail closed. Resolver injection, DNS listeners, -capture rules, and the transparent listener are all ready before workload -execution. A runtime that cannot provide the complete contract rejects a policy -containing explicit TCP endpoints rather than partially activating it. Because -that substrate is startup infrastructure, a sandbox created without explicit -TCP endpoints rejects a hot reload that introduces one and keeps its complete -previous policy active; recreating the sandbox installs the substrate before -the workload starts. A sandbox that started with the substrate may continue to -remove and re-add TCP endpoints through ordinary atomic policy reloads. -Workload DNS targets port 53, while nftables redirects eligible IPv4 DNS traffic -to an unprivileged supervisor listener. The filter admits DNS and transparent -TCP only when the kernel records the traffic as DNATed to the corresponding -supervisor listener, so direct dials to either unprivileged listener port remain -fenced. `SO_ORIGINAL_DST`, synthetic mapping lookup, endpoint correlation, and -generation-pinned authorization form the transparent TCP security boundary. -Docker and Podman do not currently advertise usable IPv6 egress for this -substrate, so AAAA queries return NOERROR/NODATA and IPv6 DNS remains fenced. - Provider credential placeholders are resolved through the live provider state for each HTTP request, after destination and L7 policy admission. A static credential resolves only when the request host, port, and path match an endpoint @@ -135,12 +235,12 @@ partially active or last-known-good static set. Invalid metadata preserves the supplied dynamic snapshot, while a fetch failure preserves the currently active dynamic snapshot. -In the Kubernetes sidecar topology, the provider environment revision remains +Across the protected sandbox/supervisor channel, the provider environment revision remains an opaque content fingerprint and has no numeric ordering semantics. The network supervisor assigns a separate, connection-local monotonic generation to each distinct environment it publishes. The process supervisor applies only newer generations, which accepts descending fingerprint values while rejecting -duplicate or delayed sidecar messages. +duplicate or delayed supervisor messages. Gateway-managed refresh credentials use an opaque workload handle derived from the sandbox, provider identity, credential key, refresh authorization epoch, @@ -171,7 +271,7 @@ support `params` matchers; generic JSON-RPC rules match only the method. JSON-RPC responses and server-to-client MCP messages on response or SSE streams are relayed but are not currently parsed for policy enforcement. -Every `protocol: mcp` endpoint carries a canonical, nonempty `mcp.versions` allowlist drawn from OpenShell's exact revision registry: `2025-03-26`, `2025-06-18`, and `2025-11-25`. A policy author may omit the entire `mcp` object when using the other endpoint defaults, or omit `mcp.versions` while setting another MCP option. Both forms resolve immediately to the exact allowlist `["2025-11-25"]`; omission never means latest or all known revisions. Defaulting applies only when the corresponding YAML key is absent: `mcp: null`, `versions: null`, and an explicit `versions: []` are invalid. At protobuf ingress, an empty repeated field means omission and uses the same default because protobuf repeated fields do not preserve presence. Normalization stores and serializes the materialized allowlist in semantic order, so adding a supported revision to the registry never widens a previously normalized policy. An explicit nonempty allowlist remains available as an advanced compatibility or downgrade control. The registry is a closed set rather than a date range, so duplicate or padded values, unknown dates, and moving aliases such as `draft` or `latest` are rejected. The sessionless `2026-07-28` revision is not accepted until OpenShell supports its distinct per-request contract. A version names a core protocol revision only; there is no policy syntax for layering a separately named SEP onto it. For every MCP HTTP request except a valid standalone `initialize`, the supervisor reads exactly one `MCP-Protocol-Version` header and requires that revision to appear in the endpoint allowlist. If the header is absent, the MCP transport specification defines `2025-03-26` as the compatibility fallback; OpenShell permits that fallback only when the allowlist contains it. Duplicate, empty, or unsupported header values receive `400 Bad Request`, while a supported revision outside the allowlist receives `403 Forbidden`. The supervisor repeats this check after middleware changes the request and before any upstream write. This check does not store session state or infer a version from a previous connection request. The registry also owns immutable batch-shape metadata: `2025-03-26` permits nonempty same-side top-level JSON-RPC batches, which OpenShell's planned enforcement caps at 64 members, while `2025-06-18` and `2025-11-25` prohibit top-level arrays. The current request parser does not yet apply these version-specific batch rules. +Every `protocol: mcp` endpoint carries a canonical, nonempty `mcp.versions` allowlist drawn from OpenShell's exact revision registry: `2025-03-26`, `2025-06-18`, and `2025-11-25`. A policy author may omit the entire `mcp` object when using the other endpoint defaults, or omit `mcp.versions` while setting another MCP option. Both forms resolve immediately to the exact allowlist `["2025-11-25"]`; omission never means latest or all known revisions. Defaulting applies only when the corresponding YAML key is absent: `mcp: null`, `versions: null`, and an explicit `versions: []` are invalid. At protobuf ingress, an empty repeated field means omission and uses the same default because protobuf repeated fields do not preserve presence. Normalization stores and serializes the materialized allowlist in semantic order, so adding a supported revision to the registry never widens a previously normalized policy. An explicit nonempty allowlist remains available as an advanced compatibility or downgrade control. The registry is a closed set rather than a date range, so duplicate or padded values, unknown dates, and moving aliases such as `draft` or `latest` are rejected. The sessionless `2026-07-28` revision is not accepted until OpenShell supports its distinct per-request runtime contract. A version names a core protocol revision only; there is no policy syntax for layering a separately named SEP onto it. The registry owns immutable batch-shape metadata: `2025-03-26` permits nonempty same-side top-level JSON-RPC batches, which OpenShell's planned enforcement caps at 64 members, while `2025-06-18` and `2025-11-25` prohibit top-level arrays. These are declared profile facts, not current forwarding claims. The allowlist does not yet select request parsing or forwarding behavior. Later response-aware runtime state must observe the successful server response, require the selected revision to be in the allowlist, and apply that one exact profile without a union or fallback; OpenShell must not bind the client proposal in `initialize` as though it were the server-selected revision. For admitted HTTP requests, the proxy can run an ordered supervisor middleware chain after L7 policy evaluation and before credential injection. Destination @@ -230,11 +330,17 @@ security logs. See [Supervisor Middleware](../docs/extensibility/supervisor-middleware.mdx) for configuration and protocol details. -Inference providers use the same egress path as other external services. An -attached provider profile contributes endpoint and binary policy. The proxy -then resolves the provider's credential placeholder only when both policy and -the profile's endpoint binding authorize the native request. Model selection, -request shape, headers, streaming, and timeouts remain client concerns. +`https://inference.local` is special. It bypasses OPA network policy and is +handled by the inference interception path: + +1. The proxy terminates the local TLS connection with the sandbox CA. +2. It detects known OpenAI, Anthropic, and compatible inference request shapes. +3. It strips caller-supplied credentials and disallowed headers. +4. It forwards through `openshell-router` using the route bundle fetched from + the gateway. + +External inference endpoints that do not use `inference.local` are treated like +ordinary network traffic and must be allowed by policy. In proxy-required networks, the supervisor chains upstream TLS tunnels through a corporate forward proxy with HTTP CONNECT instead of connecting directly, @@ -271,8 +377,9 @@ last resort for proxies whose ACLs filter on hostnames and reject IP CONNECT targets — with it, the proxy resolves the name itself and its ACLs become the effective egress control for proxied TLS. (Resolving through the proxy's own DNS view, e.g. DoH tunneled via CONNECT, is a possible future -enhancement and out of scope.) The workload child's proxy variables are -unaffected — they are always rewritten to point at the local policy proxy. +enhancement and out of scope.) Workload proxy variables are removed from the +protected launch environment; transparent socket mediation does not depend on +them. Template environment is treated like user-provided sandbox environment. It can shape the workload child, but it cannot override driver-controlled identity, @@ -291,10 +398,9 @@ sandbox-create time through validators shared with the supervisor (`openshell_core::driver_utils::parse_upstream_proxy_url` and `parse_upstream_proxy_credential`). -An optional operator CA bundle (`--upstream-proxy-ca-bundle`, a PEM path the -driver bind-mounts read-only into the sandbox) extends the trust boundary for -corporate proxies. A CA certificate is not secret, so unlike the auth file it -travels as a plain read-only bind mount rather than a driver secret. It is +An optional operator CA bundle (`--upstream-proxy-ca-bundle`, a supervisor-only +PEM path) extends the trust boundary for corporate proxies. A CA certificate is +not secret, but the supervisor is still its only configuration authority. It is trusted in two places: the TLS handshake with an `https://` proxy, and — because a TLS-intercepting proxy (mitmproxy, squid `ssl-bump`) re-signs tunneled server certificates with the same CA — the sandbox combined trust @@ -309,48 +415,28 @@ plain HTTP) and is fail-closed: an unreadable or certificate-free file is fatal. Proxy credentials are never embedded in the URL: an inline `user:pass@` is rejected because it would be stored in `gateway.toml` and exposed in container metadata. Operators supply credentials via `proxy_auth_file`; the driver -stages them as a root-only secret mounted at a fixed path and passes only +stages them as a supervisor-only secret mounted at a fixed path and passes only that path on the supervisor's command line. The supervisor reads the file and builds the `Proxy-Authorization: Basic` header; a credential that is empty, contains control characters, or is not in `user:pass` form is fatal on both sides. -The VM driver has no argv seam of its own: its guest init script runs as PID 1 -and execs a fixed supervisor command line, and the libkrun and QEMU launch -backends both reach the supervisor through that script. Driver-owned -supervisor arguments therefore travel in a per-sandbox file the driver writes -into the overlay upperdir at a fixed guest path, one argument per line, which -the guest reads verbatim (no word splitting or globbing) and appends to every -supervisor exec. The file is written on **every** launch, including an empty -file when there is nothing to pass: the upperdir copy always shadows the -read-only image layer, so a sandbox image can neither supply its own -supervisor arguments by baking a file at that path nor disable the operator's -by omitting one. This mirrors the driver-authored `init.d` manifest, which -solves the same trust problem for guest init drop-ins. - -A microVM has no bind mounts or container secrets, so the VM driver stages the -credential and the CA bundle into the per-sandbox overlay disk instead — the -credential root-only, the CA world-readable, both at fixed `/opt/openshell` -paths and both removed with the sandbox state directory. The consequence, -which differs from the Podman secret model, is that the credential is at rest -inside that overlay image on the gateway host; the per-sandbox gateway JWT -already travels the same path. Proxy reachability differs by VM backend. libkrun-backed -sandboxes egress through gvproxy, so a proxy on the gateway host's loopback is -reachable through the host alias `host.openshell.internal`, which gvproxy NATs -to the host's `127.0.0.1`. QEMU/TAP sandboxes (GPU) have no equivalent: that -alias resolves to the TAP host address, and the driver's nftables `input` -chain accepts only the gateway port from the guest, so no gateway-host proxy -is reachable. The driver rejects a gateway-host proxy URL on the QEMU path at -launch rather than producing CONNECT timeouts. The guest's gateway callback is -unaffected in both backends and never traverses the proxy. - -For Kubernetes sandboxes, the operator configures a Secret name and key rather -than a gateway-host file path. Kubernetes projects that Secret only into the -container that runs network supervision. Proxy credential Secrets require the -sidecar topology, which gives them a separate container boundary from the -workload. Combined topology is rejected because Kubernetes `fsGroup` volume -permission handling can make a shared credential mount readable by the sandbox -group. +The VM driver starts `openshell-supervisor` on the host and +`openshell-sandbox` as capability-free guest PID 1. Corporate proxy arguments, +credentials, private CA keys, policy, and gateway credentials stay host-side. +Both libkrun and QEMU guests are NIC-less; intercepted workload connections +cross the authenticated vsock channel. A gateway-host proxy is addressed as +`host.openshell.internal`, which the host supervisor normalizes to `127.0.0.1`. + +The Docker driver runs `openshell-supervisor` in a separate companion container. +Its private named volume contains supervisor bootstrap and channel material. +The workload container receives only `openshell-sandbox`, public interception +CA material, and the other sandbox half of the authenticated channel. + +For Kubernetes, the operator configures a Secret name and key rather than a +gateway-host file path. Kubernetes projects that Secret only into the separate +supervisor Pod. The sandbox Pod never mounts corporate-proxy credentials +or the interception CA private key. The Basic header travels over the plain-TCP connection to the `http://` proxy, so it is readable on the network path between sandbox host and proxy. @@ -369,16 +455,14 @@ agent process and SSH child processes. Driver-controlled environment variables override template values so sandbox images cannot spoof identity, callback, or relay settings. -Supervisor bootstrap identity is not inherited by agent child processes. When -provider token grants mount a SPIFFE Workload API socket, the socket path must -live under a dedicated directory. Children also enter a private mount namespace -where that socket directory is hidden before privilege drop. +Supervisor bootstrap identity and provider workload-identity sockets never +enter the sandbox workload. The authenticated channel carries only the +policy-authorized provider environment intended for child launch and public +trust material intended for TLS clients. -Credential placeholders in proxied HTTP requests can be resolved by the proxy -when policy allows the target endpoint. For GCP providers, a loopback metadata -server inside the network namespace serves placeholders to SDKs that bypass the -proxy (e.g. Go's `cloud.google.com/go/compute/metadata`). Secrets must not be -logged in OCSF or plain tracing output. The supervisor uses revision-scoped +Credential placeholders in mediated HTTP requests can be resolved by the proxy +when policy allows the target endpoint. Secrets must not be logged in OCSF or +plain tracing output. The supervisor uses revision-scoped placeholders for unmanaged rotating credentials and identity-stable opaque handles for gateway-managed refresh credentials. Provider environment keys beginning with `v_` or `s<64 lowercase hex characters>_` are reserved @@ -469,8 +553,6 @@ sandbox workload directly. The relay supports: Sandbox logs are emitted locally and can also be pushed back to the gateway. Security-relevant sandbox behavior uses OCSF structured events; internal diagnostics use ordinary tracing. -The OCSF device describes the sandbox environment, with type ID Other and type -label `Sandbox`; its operating system is a separate attribute. ## Policy Proposals @@ -524,22 +606,29 @@ refreshes and cannot permanently lose the initial acknowledgement. Only sandbox-scoped revisions (`PolicySource::Sandbox`, version greater than zero) are acknowledged. Global policies and local-file development policies do not use the sandbox revision API and produce no acknowledgement. When explicit -local Rego and data files are configured, the supervisor continues polling the -gateway for settings and provider refreshes but never replaces the local OPA -engine with a gateway policy revision. +local Rego and data files are provisioned into the supervisor, it continues +polling the gateway for settings and provider refreshes but never replaces the +local OPA engine with a gateway policy revision. Workload image files and +environment variables do not configure the separately isolated supervisor. ## Failure Behavior - If gateway config polling fails, the sandbox keeps its last-known-good policy. - If a live policy or middleware-registry update is invalid, the supervisor - rejects the combined update and keeps the current runtime pair. + rejects the update and keeps the current runtime pair. - If an operator-run middleware call fails, the selected config's `on_error` behavior decides whether to deny the request or continue without that stage. - Existing raw byte streams are connection scoped. Dynamic policy changes apply to new connections or the next parsed HTTP request where the proxy can safely re-evaluate. -- If the supervisor relay drops, the sandbox can keep running, but connect and - exec operations fail until the supervisor registers again. +- If the supervisor relay drops, the sandbox stops the canonical agent and exec + process groups, rejects new runtime operations, and closes mediated streams. + A replacement supervisor has 30 seconds to authenticate, replay the identical + attach, and reconfirm the boundary. Successful confirmation resumes the + process tree; otherwise the sandbox sends `SIGTERM`, waits the normal stop + grace period, sends `SIGKILL` to survivors, and makes the session terminal. + Explicit supervisor shutdown uses the same terminal transition and requires + an acknowledgement before treating the boundary as stopped. - If the canonical main process exits, the supervisor durably reports the normalized result immediately. A foreground create declares a one-shot main attachment, so the supervisor accepts it even after a fast process exits, @@ -552,3 +641,23 @@ engine with a gateway policy revision. `Error/MainProcessFailed`. Infrastructure failures also use `Error`, with a distinct condition reason and no fabricated canonical-process result. Runtime restart policies must not replace the canonical process. + +## Shared Boundary Primitives + +`openshell-isolation-interface` owns the common boundary protocol and Linux +mechanisms. Drivers provide the protected transport and immutable resource +identity; they do not implement their own process or network protocol. All +remote traffic uses one mutually authenticated gRPC connection. Independent +streams carry process control, exec output, and TCP bytes; a persistent +`Mediate` stream carries DNS queries and supervisor-produced answers. There is +no alternate raw-TLS application protocol or general UDP framing. + +The shared process-signal mediator resolves each positive target PID or TID to +its thread-group leader, excludes the sandbox leader, retains a pidfd, and sends +the signal through that descriptor. It never continues the original numeric-PID +syscall after inspection. This prevents TID aliases or PID reuse from turning an +agent signal into a signal to the sandbox. Ordinary mediated `kill` reports the +broker as its sender, not the original calling agent's `SI_USER` identity. +Queued signals preserve permitted application siginfo payloads; they cannot +forge kernel-generated or `SI_TKILL` codes. Programs requiring original sender +identity must account for this mediation boundary. diff --git a/architecture/security-policy.md b/architecture/security-policy.md index 2f1fd2ea1d..2c387d11b2 100644 --- a/architecture/security-policy.md +++ b/architecture/security-policy.md @@ -172,13 +172,15 @@ policy without provenance applies neither the raw-tunnel refusal nor the WebSocket binary-frame refusal. The request-body backstop still applies, because it keys off the presence of a secret resolver rather than endpoint provenance. -Two paths load a policy without provenance. A supervisor booting from a -container-image policy is a bounded window: that policy is resynchronized to the -gateway, which then serves a stamped effective policy. An explicit local Rego and -data override is permanent, because gateway revisions are observed for settings -and providers but never replace the local policy. When that override is combined -with injected provider credentials, the supervisor emits a high-severity -detection finding at startup naming the inactive controls. +Two supervisor-local paths load a policy without provenance. A supervisor +booting from an explicitly provisioned policy file has a bounded window before +that policy is resynchronized to the gateway, which then serves a stamped +effective policy. An explicit supervisor Rego and data override is permanent, +because gateway revisions are observed for settings and providers but never +replace the local policy. Workload-image files and environment variables cannot +configure the separately isolated supervisor. When a supervisor override is +combined with injected provider credentials, the supervisor emits a +high-severity detection finding at startup naming the inactive controls. ## Live Updates diff --git a/crates/openshell-binary-identity/Cargo.toml b/crates/openshell-binary-identity/Cargo.toml new file mode 100644 index 0000000000..a8b8714be4 --- /dev/null +++ b/crates/openshell-binary-identity/Cargo.toml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-binary-identity" +description = "Trusted executable identity resolution for OpenShell isolation backends" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +sha2 = { workspace = true } + +[lints] +workspace = true diff --git a/crates/openshell-binary-identity/README.md b/crates/openshell-binary-identity/README.md new file mode 100644 index 0000000000..9f11201b7e --- /dev/null +++ b/crates/openshell-binary-identity/README.md @@ -0,0 +1,17 @@ +# Binary identity + +`openshell-binary-identity` provides shared executable-identity resolution for +RFC 0012 isolation backends. Runtime-specific observers remain in their backend: +Docker obtains an authoritative thread ID from seccomp notification, while the +co-located Linux path maps an accepted socket to its owning processes. + +Given an authoritative Linux PID and an optional trusted process-tree root, the +crate reads the executable path from procfs, hashes the live `/proc//exe` +object, and collects bounded executable ancestry and diagnostic command-line +paths. Resolution failures are returned as `ResolveError` so the caller can +deny the associated connection. + +The crate does not intercept connections, authenticate remote observers, or +evaluate policy. The isolation backend remains responsible for binding the +resolved identity to the active boundary and exact accepted connection before +constructing `MediatedConnection`. diff --git a/crates/openshell-binary-identity/src/lib.rs b/crates/openshell-binary-identity/src/lib.rs new file mode 100644 index 0000000000..eb06934eaf --- /dev/null +++ b/crates/openshell-binary-identity/src/lib.rs @@ -0,0 +1,475 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared executable-identity resolution for RFC 0012 isolation backends. +//! +//! Runtime-specific observation remains inside each isolation backend. Once an +//! observer has an authoritative PID in its procfs view, this crate +//! canonicalizes the executable path, hashes the live executable object, and +//! collects its process ancestry. Backends bind the returned identity to the +//! intercepted connection before constructing a `MediatedConnection`. + +#[cfg(target_os = "linux")] +use openshell_isolation_interface::contract::Sha256Digest; +use openshell_isolation_interface::contract::{BinaryIdentity, ResolveError}; +#[cfg(target_os = "linux")] +use std::collections::HashMap; +#[cfg(target_os = "linux")] +use std::sync::{Arc, Mutex}; + +#[cfg(target_os = "linux")] +const EXECUTABLE_DIGEST_CACHE_CAPACITY: usize = 1_024; + +/// Resolves executable identity from a Linux procfs process identifier. +/// +/// The configured scope bounds ancestry and cmdline collection to the observed +/// PID namespace or a known workload process tree. +#[derive(Clone, Debug)] +pub struct ProcfsIdentityResolver { + ancestry_scope: AncestryScope, + #[cfg(target_os = "linux")] + cache: Arc>>, +} + +#[derive(Clone, Copy, Debug)] +enum AncestryScope { + PidNamespace, + ProcessTree(u32), +} + +impl Default for ProcfsIdentityResolver { + fn default() -> Self { + Self::for_pid_namespace() + } +} + +impl ProcfsIdentityResolver { + /// Build a resolver that discovers a nested PID namespace's init process + /// and never reports host-runtime ancestors outside that namespace. + #[must_use] + pub fn for_pid_namespace() -> Self { + Self { + ancestry_scope: AncestryScope::PidNamespace, + #[cfg(target_os = "linux")] + cache: Arc::new(Mutex::new(HashMap::new())), + } + } + + /// Build a resolver bounded by the workload's trusted process-tree root. + #[must_use] + pub fn for_process_tree(ancestor_root: u32) -> Self { + Self { + ancestry_scope: AncestryScope::ProcessTree(ancestor_root), + #[cfg(target_os = "linux")] + cache: Arc::new(Mutex::new(HashMap::new())), + } + } + + /// Resolve the identity for an authoritative process ID. + pub fn resolve(&self, pid: u32) -> Result { + #[cfg(target_os = "linux")] + { + let ancestor_root = match self.ancestry_scope { + AncestryScope::PidNamespace => nested_pid_namespace_init(pid), + AncestryScope::ProcessTree(root) => Some(root), + }; + resolve_linux_process(pid, ancestor_root, &self.cache) + } + + #[cfg(not(target_os = "linux"))] + { + match self.ancestry_scope { + AncestryScope::PidNamespace => {} + AncestryScope::ProcessTree(ancestor_root) => { + let _ = ancestor_root; + } + } + let _ = pid; + Err(ResolveError::Failed( + "procfs binary identity is only available on Linux".to_string(), + )) + } + } +} + +#[cfg(target_os = "linux")] +fn resolve_linux_process( + pid: u32, + ancestor_root: Option, + cache: &Mutex>, +) -> Result { + let (snapshot, mut executable) = open_process_snapshot(pid)?; + let binary_path = snapshot.binary_path.clone(); + let executable_key = snapshot.executable_cache_key(); + let cached_digest = cached_executable_digest(cache, executable_key); + let binary_digest = cached_digest.map_or_else(|| hash_executable(pid, &mut executable), Ok)?; + let ancestor_processes = collect_ancestor_processes(&snapshot, ancestor_root); + let ancestors = ancestor_processes + .iter() + .map(|snapshot| snapshot.binary_path.clone()) + .collect::>(); + + let mut excluded_paths = ancestors.clone(); + excluded_paths.push(binary_path.clone()); + let cmdline_paths = cmdline_absolute_paths(&snapshot.cmdline) + .into_iter() + .chain( + ancestor_processes + .iter() + .flat_map(|snapshot| cmdline_absolute_paths(&snapshot.cmdline)), + ) + .filter(|path| !excluded_paths.contains(path)) + .fold(Vec::new(), |mut paths, path| { + if !paths.contains(&path) { + paths.push(path); + } + paths + }); + + validate_process_snapshot(pid, &snapshot)?; + for ancestor in &ancestor_processes { + validate_process_snapshot(ancestor.pid, ancestor)?; + } + if cached_digest.is_none() { + cache_executable_digest(cache, executable_key, binary_digest); + } + + Ok(BinaryIdentity { + binary_path, + binary_digest: Some(binary_digest), + ancestors, + cmdline_paths, + }) +} + +#[cfg(target_os = "linux")] +#[derive(Debug, PartialEq, Eq)] +struct ProcessSnapshot { + pid: u32, + parent_pid: u32, + binary_path: std::path::PathBuf, + executable_device: u64, + executable_inode: u64, + executable_size: u64, + executable_mtime: i64, + executable_mtime_nsec: i64, + executable_ctime: i64, + executable_ctime_nsec: i64, + start_time: u64, + cmdline: Vec, +} + +#[cfg(target_os = "linux")] +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +struct ExecutableCacheKey { + device: u64, + inode: u64, + size: u64, + mtime: i64, + mtime_nsec: i64, + ctime: i64, + ctime_nsec: i64, +} + +#[cfg(target_os = "linux")] +impl ProcessSnapshot { + fn executable_cache_key(&self) -> ExecutableCacheKey { + ExecutableCacheKey { + device: self.executable_device, + inode: self.executable_inode, + size: self.executable_size, + mtime: self.executable_mtime, + mtime_nsec: self.executable_mtime_nsec, + ctime: self.executable_ctime, + ctime_nsec: self.executable_ctime_nsec, + } + } +} + +#[cfg(target_os = "linux")] +fn cached_executable_digest( + cache: &Mutex>, + key: ExecutableCacheKey, +) -> Option { + cache + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .get(&key) + .copied() +} + +#[cfg(target_os = "linux")] +fn cache_executable_digest( + cache: &Mutex>, + key: ExecutableCacheKey, + digest: Sha256Digest, +) { + let mut cache = cache + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if cache.len() >= EXECUTABLE_DIGEST_CACHE_CAPACITY { + cache.clear(); + } + cache.insert(key, digest); +} + +#[cfg(target_os = "linux")] +fn open_process_snapshot(pid: u32) -> Result<(ProcessSnapshot, std::fs::File), ResolveError> { + use std::os::unix::fs::MetadataExt as _; + + let path = format!("/proc/{pid}/exe"); + let binary_path = executable_path(pid)?; + let executable = std::fs::File::open(&path) + .map_err(|error| ResolveError::Failed(format!("open {path}: {error}")))?; + let metadata = executable + .metadata() + .map_err(|error| ResolveError::Failed(format!("stat {path}: {error}")))?; + let (parent_pid, start_time) = process_stat(pid)?; + let snapshot = ProcessSnapshot { + pid, + parent_pid, + binary_path, + executable_device: metadata.dev(), + executable_inode: metadata.ino(), + executable_size: metadata.size(), + executable_mtime: metadata.mtime(), + executable_mtime_nsec: metadata.mtime_nsec(), + executable_ctime: metadata.ctime(), + executable_ctime_nsec: metadata.ctime_nsec(), + start_time, + cmdline: read_process_cmdline(pid)?, + }; + validate_process_snapshot(pid, &snapshot)?; + Ok((snapshot, executable)) +} + +#[cfg(target_os = "linux")] +fn validate_process_snapshot(pid: u32, expected: &ProcessSnapshot) -> Result<(), ResolveError> { + use std::os::unix::fs::MetadataExt as _; + + let path = format!("/proc/{pid}/exe"); + let metadata = std::fs::metadata(&path) + .map_err(|error| ResolveError::Failed(format!("stat {path}: {error}")))?; + let (parent_pid, start_time) = process_stat(pid)?; + let current = ProcessSnapshot { + pid, + parent_pid, + binary_path: executable_path(pid)?, + executable_device: metadata.dev(), + executable_inode: metadata.ino(), + executable_size: metadata.size(), + executable_mtime: metadata.mtime(), + executable_mtime_nsec: metadata.mtime_nsec(), + executable_ctime: metadata.ctime(), + executable_ctime_nsec: metadata.ctime_nsec(), + start_time, + cmdline: read_process_cmdline(pid)?, + }; + if ¤t == expected { + Ok(()) + } else { + Err(ResolveError::Failed(format!( + "process {pid} changed while its executable identity was collected" + ))) + } +} + +#[cfg(target_os = "linux")] +fn process_stat(pid: u32) -> Result<(u32, u64), ResolveError> { + let path = format!("/proc/{pid}/stat"); + let stat = std::fs::read_to_string(&path) + .map_err(|error| ResolveError::Failed(format!("read {path}: {error}")))?; + let fields = stat + .rsplit_once(") ") + .map(|(_, fields)| fields) + .ok_or_else(|| ResolveError::Failed(format!("parse {path}: missing command field")))?; + let mut fields = fields.split_whitespace(); + let _state = fields.next(); + let parent_pid = fields + .next() + .ok_or_else(|| ResolveError::Failed(format!("parse {path}: missing parent PID")))? + .parse() + .map_err(|error| ResolveError::Failed(format!("parse {path} parent PID: {error}")))?; + let start_time = fields + .nth(17) + .ok_or_else(|| ResolveError::Failed(format!("parse {path}: missing start time")))? + .parse() + .map_err(|error| ResolveError::Failed(format!("parse {path} start time: {error}")))?; + Ok((parent_pid, start_time)) +} + +#[cfg(target_os = "linux")] +fn read_process_cmdline(pid: u32) -> Result, ResolveError> { + let path = format!("/proc/{pid}/cmdline"); + std::fs::read(&path).map_err(|error| ResolveError::Failed(format!("read {path}: {error}"))) +} + +#[cfg(target_os = "linux")] +fn executable_path(pid: u32) -> Result { + use std::ffi::OsString; + use std::io::ErrorKind; + use std::os::unix::ffi::{OsStrExt as _, OsStringExt as _}; + + const DELETED_SUFFIX: &[u8] = b" (deleted)"; + + let link = format!("/proc/{pid}/exe"); + let target = std::fs::read_link(&link) + .map_err(|error| ResolveError::Failed(format!("read {link}: {error}")))?; + let target_missing = + matches!(std::fs::metadata(&target), Err(error) if error.kind() == ErrorKind::NotFound); + let bytes = target.as_os_str().as_bytes(); + + if target_missing && bytes.ends_with(DELETED_SUFFIX) { + let stripped = bytes[..bytes.len() - DELETED_SUFFIX.len()].to_vec(); + return Ok(std::path::PathBuf::from(OsString::from_vec(stripped))); + } + + Ok(target) +} + +#[cfg(target_os = "linux")] +fn hash_executable(pid: u32, executable: &mut std::fs::File) -> Result { + use sha2::{Digest as _, Sha256}; + use std::io::Read as _; + + let path = format!("/proc/{pid}/exe"); + let mut digest = Sha256::new(); + let mut buffer = [0_u8; 8 * 1024]; + loop { + let length = executable + .read(&mut buffer) + .map_err(|error| ResolveError::Failed(format!("hash {path}: {error}")))?; + if length == 0 { + break; + } + digest.update(&buffer[..length]); + } + format!("{:x}", digest.finalize()).parse() +} + +#[cfg(target_os = "linux")] +fn collect_ancestor_processes( + process: &ProcessSnapshot, + ancestor_root: Option, +) -> Vec { + const MAX_DEPTH: usize = 64; + + if ancestor_root == Some(process.pid) { + return Vec::new(); + } + + let mut ancestors = Vec::new(); + let mut parent = process.parent_pid; + for _ in 0..MAX_DEPTH { + if parent == 0 + || ancestors + .iter() + .any(|current: &ProcessSnapshot| current.pid == parent) + { + break; + } + + // PID 1 is host or guest init rather than workload ancestry unless it + // is the explicitly supplied process-tree root. + if parent == 1 && ancestor_root != Some(1) { + break; + } + + let Ok((snapshot, _executable)) = open_process_snapshot(parent) else { + break; + }; + let next_parent = snapshot.parent_pid; + ancestors.push(snapshot); + if ancestor_root == Some(parent) || parent == 1 { + break; + } + parent = next_parent; + } + ancestors +} + +#[cfg(target_os = "linux")] +fn parent_pid(pid: u32) -> Option { + std::fs::read_to_string(format!("/proc/{pid}/status")) + .ok()? + .lines() + .find_map(|line| line.strip_prefix("PPid:"))? + .trim() + .parse() + .ok() +} + +#[cfg(target_os = "linux")] +fn nested_pid_namespace_init(pid: u32) -> Option { + const MAX_DEPTH: usize = 64; + + let mut current = pid; + for _ in 0..MAX_DEPTH { + if namespace_pid(current) == Some(1) { + // Host PID 1 is outside every workload. A nested namespace init + // has a distinct host PID and is a valid workload ancestry root. + return (current != 1).then_some(current); + } + current = parent_pid(current).filter(|parent| *parent > 0 && *parent != current)?; + } + None +} + +#[cfg(target_os = "linux")] +fn namespace_pid(pid: u32) -> Option { + std::fs::read_to_string(format!("/proc/{pid}/status")) + .ok()? + .lines() + .find_map(|line| line.strip_prefix("NSpid:"))? + .split_whitespace() + .next_back()? + .parse() + .ok() +} + +#[cfg(target_os = "linux")] +fn cmdline_absolute_paths(cmdline: &[u8]) -> Vec { + cmdline + .split(|byte| *byte == 0) + .filter(|argument| argument.first() == Some(&b'/')) + .map(|argument| std::path::PathBuf::from(String::from_utf8_lossy(argument).into_owned())) + .collect() +} + +#[cfg(all(test, target_os = "linux"))] +mod tests { + use super::*; + + #[cfg(target_os = "linux")] + #[test] + fn resolver_cache_is_owned_and_only_explicit_clones_share_it() { + let first = ProcfsIdentityResolver::for_pid_namespace(); + let shared = first.clone(); + let separate = ProcfsIdentityResolver::for_pid_namespace(); + assert!(Arc::ptr_eq(&first.cache, &shared.cache)); + assert!(!Arc::ptr_eq(&first.cache, &separate.cache)); + first.resolve(std::process::id()).unwrap(); + assert!(!shared.cache.lock().unwrap().is_empty()); + assert!(separate.cache.lock().unwrap().is_empty()); + } + + #[test] + fn resolves_current_process_from_live_executable() { + let identity = ProcfsIdentityResolver::for_pid_namespace() + .resolve(std::process::id()) + .expect("resolve current process"); + + assert!(identity.binary_path.is_absolute()); + assert!(identity.binary_digest.is_some()); + } + + #[test] + fn process_tree_root_does_not_escape_into_host_ancestry() { + let pid = std::process::id(); + let identity = ProcfsIdentityResolver::for_process_tree(pid) + .resolve(pid) + .expect("resolve process-tree root"); + + assert!(identity.ancestors.is_empty()); + } +} diff --git a/crates/openshell-core/Cargo.toml b/crates/openshell-core/Cargo.toml index ff2cc638f1..d231f0c16b 100644 --- a/crates/openshell-core/Cargo.toml +++ b/crates/openshell-core/Cargo.toml @@ -26,10 +26,13 @@ serde = { workspace = true } serde_json = { workspace = true } tracing = { workspace = true } url = { workspace = true } +uuid = { workspace = true } ipnet = "2" rustls = { workspace = true } rustls-pemfile = { workspace = true } base64 = { workspace = true } +jsonwebtoken = { workspace = true, optional = true } +zeroize = { workspace = true, optional = true } chrono = { version = "0.4", default-features = false, features = ["clock", "std"], optional = true } reqwest = { workspace = true, features = ["blocking", "rustls-tls-native-roots-no-provider"], optional = true } tar = { version = "0.4", optional = true } @@ -48,6 +51,8 @@ driver-extraction = ["dep:tar", "dep:tempfile"] telemetry = ["dep:reqwest", "dep:chrono", "reqwest?/blocking"] ## OAuth2 token request helpers used by supervisor and gateway. oauth = ["dep:reqwest"] +## Strict Ed25519 JWT issuance and verification for authenticated sandbox sessions. +jwt = ["dep:jsonwebtoken", "dep:zeroize"] [build-dependencies] tonic-prost-build = { workspace = true } diff --git a/crates/openshell-core/src/config.rs b/crates/openshell-core/src/config.rs index c308ac37e0..e38631ecd7 100644 --- a/crates/openshell-core/src/config.rs +++ b/crates/openshell-core/src/config.rs @@ -86,6 +86,18 @@ impl FromStr for PolicyValidationFailureMode { /// Default OCI repository for the supervisor image (no tag). pub const DEFAULT_SUPERVISOR_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/supervisor"; +/// Default OCI repository for the sandbox runtime image (no tag). +pub const DEFAULT_SANDBOX_RUNTIME_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/sandbox"; + +/// Return the default sandbox runtime image reference with a version-pinned tag. +#[must_use] +pub fn default_sandbox_runtime_image() -> String { + format!( + "{DEFAULT_SANDBOX_RUNTIME_IMAGE_REPO}:{}", + default_supervisor_image_tag() + ) +} + /// Return the default supervisor image reference with a version-pinned tag. #[must_use] pub fn default_supervisor_image() -> String { @@ -1539,10 +1551,15 @@ mod tests { #[test] fn default_supervisor_image_is_version_pinned() { - use super::default_supervisor_image; + use super::{default_sandbox_runtime_image, default_supervisor_image}; let image = default_supervisor_image(); assert!(image.starts_with("ghcr.io/nvidia/openshell/supervisor:")); let tag = image.rsplit_once(':').unwrap().1; assert!(!tag.is_empty()); + + let sandbox_image = default_sandbox_runtime_image(); + assert!(sandbox_image.starts_with("ghcr.io/nvidia/openshell/sandbox:")); + let sandbox_tag = sandbox_image.rsplit_once(':').unwrap().1; + assert!(!sandbox_tag.is_empty()); } } diff --git a/crates/openshell-core/src/driver_utils.rs b/crates/openshell-core/src/driver_utils.rs index a0c46729c4..0ab1049533 100644 --- a/crates/openshell-core/src/driver_utils.rs +++ b/crates/openshell-core/src/driver_utils.rs @@ -7,10 +7,10 @@ use std::path::{Path, PathBuf}; use crate::proto::compute::v1::DriverSandbox; -/// Built-in sandbox network topologies used to derive a callback endpoint +/// Built-in sandbox network callback routes used to derive a callback endpoint /// when an operator does not configure a per-driver `grpc_endpoint` override. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum GatewayCallbackTopology { +pub enum GatewayCallbackRoute { /// A Docker container reaches the host through Docker's gateway alias. Docker, /// A Podman container reaches the host through Podman's gateway alias. @@ -19,41 +19,41 @@ pub enum GatewayCallbackTopology { Vm, } -/// Build the endpoint a sandbox uses to call its gateway for a known topology. +/// Build the endpoint a sandbox uses to call its gateway for a known route. /// /// The result is deliberately derived by the gateway rather than baked into /// individual driver defaults. A configured `grpc_endpoint` remains an /// operator override for remote or non-standard deployments. #[must_use] pub fn gateway_callback_endpoint( - topology: GatewayCallbackTopology, + route: GatewayCallbackRoute, gateway_port: u16, gateway_tls_enabled: bool, ) -> String { let scheme = if gateway_tls_enabled { "https" } else { "http" }; - let host = match topology { - GatewayCallbackTopology::Docker | GatewayCallbackTopology::Vm => "host.openshell.internal", - GatewayCallbackTopology::Podman => "host.containers.internal", + let host = match route { + GatewayCallbackRoute::Docker | GatewayCallbackRoute::Vm => "host.openshell.internal", + GatewayCallbackRoute::Podman => "host.containers.internal", }; format!("{scheme}://{host}:{gateway_port}") } #[cfg(test)] mod callback_endpoint_tests { - use super::{GatewayCallbackTopology, gateway_callback_endpoint}; + use super::{GatewayCallbackRoute, gateway_callback_endpoint}; #[test] - fn derives_endpoint_for_each_builtin_topology() { + fn derives_endpoint_for_each_builtin_route() { assert_eq!( - gateway_callback_endpoint(GatewayCallbackTopology::Docker, 17670, false), + gateway_callback_endpoint(GatewayCallbackRoute::Docker, 17670, false), "http://host.openshell.internal:17670" ); assert_eq!( - gateway_callback_endpoint(GatewayCallbackTopology::Podman, 17670, true), + gateway_callback_endpoint(GatewayCallbackRoute::Podman, 17670, true), "https://host.containers.internal:17670" ); assert_eq!( - gateway_callback_endpoint(GatewayCallbackTopology::Vm, 17670, true), + gateway_callback_endpoint(GatewayCallbackRoute::Vm, 17670, true), "https://host.openshell.internal:17670" ); } @@ -134,7 +134,10 @@ pub const CONDITION_STOPPED: &str = "ContainerStopped"; /// All compute drivers must launch this binary as the container entrypoint to /// start the sandboxed environment. The value must be kept in sync with the /// path used when building the `openshell-sandbox` image layer. -pub const SUPERVISOR_IMAGE_BINARY_PATH: &str = "/openshell-sandbox"; +pub const SANDBOX_RUNTIME_IMAGE_BINARY_PATH: &str = "/openshell-sandbox"; + +/// Legacy name for [`SANDBOX_RUNTIME_IMAGE_BINARY_PATH`]. +pub const SUPERVISOR_IMAGE_BINARY_PATH: &str = SANDBOX_RUNTIME_IMAGE_BINARY_PATH; /// Directory inside sandbox containers where the supervisor binary is mounted. /// diff --git a/crates/openshell-core/src/grpc_client.rs b/crates/openshell-core/src/grpc_client.rs index 38b91e8501..e72b4552cb 100644 --- a/crates/openshell-core/src/grpc_client.rs +++ b/crates/openshell-core/src/grpc_client.rs @@ -70,6 +70,9 @@ static TOKEN_INIT_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(( /// One-shot guard so the renewal loop spawns at most once per process. static REFRESH_SPAWNED: OnceLock<()> = OnceLock::new(); +#[cfg(feature = "jwt")] +static SANDBOX_BEARER_SLOT: OnceLock = OnceLock::new(); + #[derive(Clone, Debug)] enum RefreshMode { GatewayJwt(TokenSource), @@ -82,16 +85,64 @@ struct AcquiredToken { } fn install_token_slot(token: &str) -> Result { - let bearer = AsciiMetadataValue::try_from(format!("Bearer {token}")) + let bearer = validate_gateway_bearer(token)?; + Ok(install_validated_token_slot(bearer)) +} + +fn validate_gateway_bearer(token: &str) -> Result { + AsciiMetadataValue::try_from(format!("Bearer {token}")) .into_diagnostic() - .wrap_err("sandbox JWT contained characters not valid for a header value")?; + .wrap_err("sandbox JWT contained characters not valid for a header value") +} + +fn install_validated_token_slot(bearer: AsciiMetadataValue) -> TokenSlot { if let Some(existing) = TOKEN_SLOT.get() { - *existing.write().expect("token slot poisoned") = bearer; - return Ok(existing.clone()); + *existing + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = bearer; + return existing.clone(); } let slot: TokenSlot = Arc::new(RwLock::new(bearer)); let _ = TOKEN_SLOT.set(slot.clone()); - Ok(TOKEN_SLOT.get().cloned().unwrap_or(slot)) + TOKEN_SLOT.get().cloned().unwrap_or(slot) +} + +#[cfg(feature = "jwt")] +struct ValidatedSandboxRefresh { + token: crate::jwt::SecretJwt, + expires_at: i64, + credential_epoch: crate::jwt::CredentialEpoch, +} + +#[cfg(feature = "jwt")] +fn validate_sandbox_refresh( + response: &crate::proto::RefreshSandboxTokenResponse, +) -> std::result::Result { + let token = crate::jwt::SecretJwt::parse(response.sandbox_token.clone())?; + let credential_epoch = crate::jwt::CredentialEpoch::new(response.credential_epoch)?; + let expires_at = response.sandbox_expires_at_ms / 1000; + crate::jwt::SessionBearerTokenSlot::new(token.clone(), expires_at, credential_epoch)?; + Ok(ValidatedSandboxRefresh { + token, + expires_at, + credential_epoch, + }) +} + +/// Install the gateway-session credential supplied in trusted supervisor +/// launch state before any gateway client is constructed. +#[cfg(feature = "jwt")] +pub fn install_supervisor_auth_bundle( + bundle: &crate::jwt::SupervisorAuthBundle, +) -> Result { + install_token_slot(bundle.gateway_token.expose_secret())?; + let _ = TOKEN_REFRESH_MODE.set(RefreshMode::GatewayJwt(TokenSource::File)); + let slot = bundle + .sandbox_bearer_slot() + .into_diagnostic() + .wrap_err("invalid Sandbox Protocol credential")?; + let _ = SANDBOX_BEARER_SLOT.set(slot.clone()); + Ok(SANDBOX_BEARER_SLOT.get().cloned().unwrap_or(slot)) } /// gRPC interceptor that injects `authorization: Bearer ` on every @@ -360,16 +411,38 @@ async fn refresh_token_loop( .await { Ok(resp) => { - let new_token = resp.into_inner().token; - match AsciiMetadataValue::try_from(format!("Bearer {new_token}")) { - Ok(value) => { - if let Ok(mut guard) = slot.write() { - *guard = value; - info!("renewed gateway sandbox JWT in-place"); - } + let response = resp.into_inner(); + #[cfg(feature = "jwt")] + let sandbox_refresh = match validate_sandbox_refresh(&response) { + Ok(refresh) => refresh, + Err(error) => { + warn!(%error, "gateway returned an invalid Sandbox Protocol credential"); + continue; + } + }; + let gateway_bearer = match validate_gateway_bearer(&response.token) { + Ok(value) => value, + Err(error) => { + warn!(%error, "refreshed JWT contained invalid header bytes"); + continue; } - Err(e) => warn!(error = %e, "refreshed JWT contained invalid header bytes"), + }; + + *slot + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = gateway_bearer; + #[cfg(feature = "jwt")] + if let Some(sandbox_slot) = SANDBOX_BEARER_SLOT.get() + && let Err(error) = sandbox_slot.update( + sandbox_refresh.token, + sandbox_refresh.expires_at, + sandbox_refresh.credential_epoch, + ) + && error != crate::jwt::SessionJwtError::StaleCredentialEpoch + { + warn!(%error, "gateway returned an invalid Sandbox Protocol credential"); } + info!("renewed gateway and Sandbox Protocol credentials in-place"); } Err(status) => { if status.code() == tonic::Code::Unauthenticated @@ -462,11 +535,11 @@ async fn refresh_extension_credentials_with_client( .wrap_err("failed to refresh extension service credentials")? .into_inner(); - // The same refresh response renews the gateway credential. Install it - // before returning so all process-wide gateway clients stay current. This - // is a superset of what the dedicated renewal loop would do, so letting it - // land early is harmless. - install_token_slot(&response.token)?; + let gateway_bearer = validate_gateway_bearer(&response.token)?; + #[cfg(feature = "jwt")] + let sandbox_refresh = validate_sandbox_refresh(&response) + .into_diagnostic() + .wrap_err("gateway returned an invalid Sandbox Protocol credential")?; // Validate the whole response before mutating any slot, so a malformed or // partial reply cannot leave the store half-rotated. @@ -495,6 +568,32 @@ async fn refresh_extension_credentials_with_client( } let now_ms = now_ms(); + for (token, expires_at_ms) in validated.values() { + BearerTokenSlot::new(token, *expires_at_ms) + .into_diagnostic() + .wrap_err("gateway returned an invalid extension credential")?; + } + + // Commit only after the gateway, Sandbox Protocol, and extension + // credentials have all been parsed and validated. The remaining updates + // repeat those validations but cannot fail for the validated inputs. + install_validated_token_slot(gateway_bearer); + #[cfg(feature = "jwt")] + if let Some(sandbox_slot) = SANDBOX_BEARER_SLOT.get() { + match sandbox_slot.update( + sandbox_refresh.token, + sandbox_refresh.expires_at, + sandbox_refresh.credential_epoch, + ) { + Ok(()) | Err(crate::jwt::SessionJwtError::StaleCredentialEpoch) => {} + Err(error) => { + return Err(miette::miette!( + "validated Sandbox Protocol credential could not be installed: {error}" + )); + } + } + } + let mut selected = HashMap::with_capacity(validated.len()); for (name, (token, expires_at_ms)) in validated { let slot = store @@ -554,6 +653,22 @@ fn parse_jwt_exp_ms(jwt: &str) -> Option { mod auth_tests { use super::*; + #[cfg(feature = "jwt")] + #[test] + fn sandbox_refresh_validation_rejects_invalid_lifetime_before_installation() { + let response = crate::proto::RefreshSandboxTokenResponse { + sandbox_token: "sandbox-token".to_string(), + sandbox_expires_at_ms: 999, + credential_epoch: 2, + ..Default::default() + }; + + assert_eq!( + validate_sandbox_refresh(&response).err(), + Some(crate::jwt::SessionJwtError::InvalidLifetime) + ); + } + #[test] fn parse_jwt_exp_reads_unsigned_payload() { use base64::Engine as _; diff --git a/crates/openshell-core/src/jwt.rs b/crates/openshell-core/src/jwt.rs index 62af827524..90ea9a3d55 100644 --- a/crates/openshell-core/src/jwt.rs +++ b/crates/openshell-core/src/jwt.rs @@ -29,12 +29,790 @@ pub fn parse_exp_secs(token: &str) -> Option { value.get("exp")?.as_i64() } +#[cfg(feature = "jwt")] +mod session { + use std::collections::BTreeMap; + use std::fmt; + use std::sync::Arc; + use std::time::{Duration, SystemTime, UNIX_EPOCH}; + + use jsonwebtoken::{ + Algorithm, DecodingKey, EncodingKey, Header, Validation, decode, decode_header, encode, + }; + use serde::{Deserialize, Serialize}; + use uuid::Uuid; + use zeroize::Zeroizing; + + use crate::SandboxSessionId; + use crate::sandbox_generation::SandboxGenerationId; + + pub const GATEWAY_SESSION_JWT_TYPE: &str = "openshell-gateway-session+jwt"; + pub const SANDBOX_SESSION_JWT_TYPE: &str = "openshell-sandbox-session+jwt"; + pub const SANDBOX_SESSION_AUDIENCE: &str = "openshell-sandbox"; + pub const DEFAULT_SESSION_TOKEN_TTL: Duration = Duration::from_hours(1); + pub const MIN_SESSION_TOKEN_TTL: Duration = Duration::from_mins(1); + pub const MAX_SESSION_TOKEN_TTL: Duration = Duration::from_hours(1); + pub const MAX_SESSION_CLOCK_LEEWAY: Duration = Duration::from_secs(30); + + const GATEWAY_ISSUER_PREFIX: &str = "openshell-gateway:"; + const SANDBOX_SUBJECT_PREFIX: &str = "spiffe://openshell/sandbox/"; + + /// Canonical sandbox identity carried by both session-token profiles. + #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] + #[serde(transparent)] + pub struct SandboxId(String); + + impl SandboxId { + pub fn parse(value: impl Into) -> Result { + let value = value.into(); + if value.is_empty() || value.trim() != value || value.chars().any(char::is_whitespace) { + return Err(SessionJwtError::InvalidSandboxId); + } + Ok(Self(value)) + } + + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } + } + + impl fmt::Display for SandboxId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(&self.0) + } + } + + /// Monotonic order for authenticated Sandbox Protocol connections. + #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] + #[serde(transparent)] + pub struct CredentialEpoch(u64); + + impl CredentialEpoch { + pub fn new(value: u64) -> Result { + if value == 0 { + return Err(SessionJwtError::InvalidCredentialEpoch); + } + Ok(Self(value)) + } + + #[must_use] + pub const fn get(self) -> u64 { + self.0 + } + } + + /// Monotonic identity for Sandbox Protocol attachment state. + /// + /// This is a protocol coordination value, not part of JWT authorization. + #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] + #[serde(transparent)] + pub struct SessionRotation(u64); + + impl SessionRotation { + pub fn new(value: u64) -> Result { + if value == 0 { + return Err(SessionJwtError::InvalidSessionRotation); + } + Ok(Self(value)) + } + + #[must_use] + pub const fn get(self) -> u64 { + self.0 + } + + pub fn successor(self) -> Result { + self.0 + .checked_add(1) + .ok_or(SessionJwtError::SessionRotationOverflow) + .and_then(Self::new) + } + } + + /// The only component authorized by either sandbox-session token profile. + #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] + pub enum SessionComponent { + #[serde(rename = "openshell-supervisor")] + OpenShellSupervisor, + } + + /// Exact token profile. The profile chooses both the JOSE `typ` and JWT `aud`. + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + pub enum SessionTokenProfile { + Gateway, + Sandbox, + } + + impl SessionTokenProfile { + #[must_use] + pub const fn token_type(self) -> &'static str { + match self { + Self::Gateway => GATEWAY_SESSION_JWT_TYPE, + Self::Sandbox => SANDBOX_SESSION_JWT_TYPE, + } + } + + fn audience(self, issuer: &str) -> &str { + match self { + Self::Gateway => issuer, + Self::Sandbox => SANDBOX_SESSION_AUDIENCE, + } + } + } + + #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] + #[serde(deny_unknown_fields)] + struct SessionClaims { + iss: String, + sub: String, + aud: String, + iat: i64, + exp: i64, + jti: String, + sandbox_id: SandboxId, + runtime_generation: SandboxGenerationId, + auth_epoch: CredentialEpoch, + component: SessionComponent, + } + + /// Durable identity shared by every short-lived token for one sandbox runtime. + #[derive(Clone, Debug, PartialEq, Eq)] + pub struct SandboxRuntimeIdentity { + pub sandbox_id: SandboxId, + pub runtime_generation: SandboxGenerationId, + pub auth_epoch: CredentialEpoch, + } + + /// A JWT whose contents are deliberately omitted from `Debug` output and + /// zeroed when its final owner is dropped. + #[derive(Clone)] + pub struct SecretJwt(Zeroizing); + + impl SecretJwt { + pub fn parse(value: impl Into) -> Result { + let value = value.into(); + if value.is_empty() || value.chars().any(char::is_whitespace) { + return Err(SessionJwtError::InvalidTokenEncoding); + } + Ok(Self(Zeroizing::new(value))) + } + + #[must_use] + pub fn expose_secret(&self) -> &str { + self.0.as_str() + } + } + + impl fmt::Debug for SecretJwt { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("SecretJwt([REDACTED])") + } + } + + impl Serialize for SecretJwt { + fn serialize(&self, serializer: S) -> Result + where + S: serde::Serializer, + { + serializer.serialize_str(self.expose_secret()) + } + } + + impl<'de> Deserialize<'de> for SecretJwt { + fn deserialize(deserializer: D) -> Result + where + D: serde::Deserializer<'de>, + { + let value = String::deserialize(deserializer)?; + Self::parse(value).map_err(serde::de::Error::custom) + } + } + + /// Trusted launch input delivered only to `openshell-supervisor`. + #[derive(Clone, Serialize, Deserialize)] + #[serde(deny_unknown_fields)] + pub struct SupervisorAuthBundle { + /// Correlates the two ends of one Sandbox Protocol launch. This value + /// is not carried in JWTs and is not an authorization identity. + pub session_id: SandboxSessionId, + pub runtime_generation: SandboxGenerationId, + /// Orders protocol attachment replacement, independently of auth. + pub session_rotation: SessionRotation, + pub auth_epoch: CredentialEpoch, + pub gateway_token: SecretJwt, + pub gateway_expires_at: i64, + pub sandbox_token: SecretJwt, + pub sandbox_expires_at: i64, + } + + /// Gateway-created authentication input trusted by a compute driver. + /// + /// Drivers split this structure: the supervisor receives `supervisor`, + /// while the sandbox receives only the gateway identity and public keys. + #[derive(Clone, Serialize, Deserialize)] + #[serde(deny_unknown_fields)] + pub struct SandboxLaunchAuthentication { + pub supervisor: SupervisorAuthBundle, + pub gateway_id: String, + pub verification_keys: Vec, + } + + impl SandboxLaunchAuthentication { + pub fn validate(&self) -> Result<(), SessionJwtError> { + self.supervisor.validate()?; + validate_gateway_id(&self.gateway_id)?; + if self.verification_keys.is_empty() { + return Err(SessionJwtError::NoVerificationKeys); + } + let mut key_ids = std::collections::BTreeSet::new(); + for key in &self.verification_keys { + validate_key_id(key.key_id.clone())?; + if key.public_key_pem.is_empty() { + return Err(SessionJwtError::InvalidVerificationKey); + } + if !key_ids.insert(key.key_id.as_str()) { + return Err(SessionJwtError::DuplicateKeyId); + } + } + Ok(()) + } + } + + impl fmt::Debug for SandboxLaunchAuthentication { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SandboxLaunchAuthentication") + .field("supervisor", &self.supervisor) + .field("gateway_id", &self.gateway_id) + .field( + "verification_key_ids", + &self + .verification_keys + .iter() + .map(|key| key.key_id.as_str()) + .collect::>(), + ) + .finish() + } + } + + impl SupervisorAuthBundle { + pub fn validate(&self) -> Result<(), SessionJwtError> { + if self.gateway_expires_at <= 0 || self.sandbox_expires_at <= 0 { + return Err(SessionJwtError::InvalidLifetime); + } + SandboxGenerationId::parse(self.runtime_generation.to_string()) + .map_err(|_| SessionJwtError::InvalidRuntimeIdentity)?; + SessionRotation::new(self.session_rotation.get())?; + Ok(()) + } + + pub fn sandbox_bearer_slot(&self) -> Result { + SessionBearerTokenSlot::new( + self.sandbox_token.clone(), + self.sandbox_expires_at, + self.auth_epoch, + ) + } + } + + impl fmt::Debug for SupervisorAuthBundle { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SupervisorAuthBundle") + .field("session_id", &self.session_id) + .field("runtime_generation", &self.runtime_generation) + .field("session_rotation", &self.session_rotation) + .field("auth_epoch", &self.auth_epoch) + .field("gateway_token", &"[REDACTED]") + .field("gateway_expires_at", &self.gateway_expires_at) + .field("sandbox_token", &"[REDACTED]") + .field("sandbox_expires_at", &self.sandbox_expires_at) + .finish() + } + } + + #[derive(Clone, Debug)] + pub struct MintedSessionToken { + pub token: SecretJwt, + pub expires_at: i64, + pub token_id: Uuid, + } + + #[derive(Clone, Debug)] + pub struct MintedSessionTokenPair { + pub gateway: MintedSessionToken, + pub sandbox: MintedSessionToken, + pub auth_epoch: CredentialEpoch, + } + + /// Refreshable Sandbox Protocol bearer credential shared by all streams on + /// the supervisor's current HTTP/2 connection. + #[derive(Clone)] + pub struct SessionBearerTokenSlot { + inner: Arc>>, + } + + #[derive(Clone)] + struct StoredBearer { + token: SecretJwt, + expires_at: i64, + credential_epoch: CredentialEpoch, + } + + impl SessionBearerTokenSlot { + #[must_use] + pub fn empty() -> Self { + Self { + inner: Arc::new(std::sync::RwLock::new(None)), + } + } + + pub fn new( + token: SecretJwt, + expires_at: i64, + credential_epoch: CredentialEpoch, + ) -> Result { + let slot = Self::empty(); + slot.update(token, expires_at, credential_epoch)?; + Ok(slot) + } + + pub fn update( + &self, + token: SecretJwt, + expires_at: i64, + credential_epoch: CredentialEpoch, + ) -> Result<(), SessionJwtError> { + if expires_at <= 0 { + return Err(SessionJwtError::InvalidLifetime); + } + let mut stored = self + .inner + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if stored + .as_ref() + .is_some_and(|current| credential_epoch < current.credential_epoch) + { + return Err(SessionJwtError::StaleCredentialEpoch); + } + *stored = Some(StoredBearer { + token, + expires_at, + credential_epoch, + }); + Ok(()) + } + + pub fn clear(&self) { + *self + .inner + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = None; + } + + #[must_use] + pub fn expires_at(&self) -> Option { + self.inner + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .as_ref() + .map(|stored| stored.expires_at) + } + + #[must_use] + pub fn credential_epoch(&self) -> Option { + self.inner + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .as_ref() + .map(|stored| stored.credential_epoch) + } + + pub fn authorization_metadata( + &self, + ) -> Result { + let stored = self + .inner + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let stored = stored.as_ref().ok_or(SessionJwtError::TokenUnavailable)?; + if stored.expires_at <= SystemJwtClock.now_unix_seconds() { + return Err(SessionJwtError::Expired); + } + format!("Bearer {}", stored.token.expose_secret()) + .parse() + .map_err(|_| SessionJwtError::InvalidTokenEncoding) + } + } + + impl Default for SessionBearerTokenSlot { + fn default() -> Self { + Self::empty() + } + } + + impl fmt::Debug for SessionBearerTokenSlot { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SessionBearerTokenSlot") + .field("expires_at", &self.expires_at()) + .field("credential_epoch", &self.credential_epoch()) + .finish_non_exhaustive() + } + } + + pub trait JwtClock: Send + Sync { + fn now_unix_seconds(&self) -> i64; + } + + #[derive(Debug)] + pub struct SystemJwtClock; + + impl JwtClock for SystemJwtClock { + fn now_unix_seconds(&self) -> i64 { + i64::try_from( + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |duration| duration.as_secs()), + ) + .unwrap_or(i64::MAX) + } + } + + /// Gateway-side issuer shared by both token profiles. + pub struct SessionJwtIssuer { + encoding_key: EncodingKey, + key_id: String, + issuer: String, + ttl: Duration, + clock: Arc, + } + + impl fmt::Debug for SessionJwtIssuer { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SessionJwtIssuer") + .field("key_id", &self.key_id) + .field("issuer", &self.issuer) + .field("ttl", &self.ttl) + .finish_non_exhaustive() + } + } + + impl SessionJwtIssuer { + pub fn from_ed25519_pem( + signing_key_pem: &[u8], + key_id: impl Into, + gateway_id: &str, + ttl: Duration, + clock: Arc, + ) -> Result { + install_crypto_provider(); + validate_ttl(ttl)?; + let key_id = validate_key_id(key_id.into())?; + let gateway_id = validate_gateway_id(gateway_id)?; + let encoding_key = EncodingKey::from_ed_pem(signing_key_pem) + .map_err(|_| SessionJwtError::InvalidSigningKey)?; + Ok(Self { + encoding_key, + key_id, + issuer: format!("{GATEWAY_ISSUER_PREFIX}{gateway_id}"), + ttl, + clock, + }) + } + + pub fn mint_pair( + &self, + identity: &SandboxRuntimeIdentity, + ) -> Result { + Ok(MintedSessionTokenPair { + gateway: self.mint(SessionTokenProfile::Gateway, identity)?, + sandbox: self.mint(SessionTokenProfile::Sandbox, identity)?, + auth_epoch: identity.auth_epoch, + }) + } + + fn mint( + &self, + profile: SessionTokenProfile, + identity: &SandboxRuntimeIdentity, + ) -> Result { + let issued_at = self.clock.now_unix_seconds(); + let expires_at = issued_at.saturating_add( + i64::try_from(self.ttl.as_secs()).map_err(|_| SessionJwtError::InvalidLifetime)?, + ); + let token_id = Uuid::new_v4(); + let claims = SessionClaims { + iss: self.issuer.clone(), + sub: format!("{SANDBOX_SUBJECT_PREFIX}{}", identity.sandbox_id), + aud: profile.audience(&self.issuer).to_string(), + iat: issued_at, + exp: expires_at, + jti: token_id.to_string(), + sandbox_id: identity.sandbox_id.clone(), + runtime_generation: identity.runtime_generation.clone(), + auth_epoch: identity.auth_epoch, + component: SessionComponent::OpenShellSupervisor, + }; + let mut header = Header::new(Algorithm::EdDSA); + header.kid = Some(self.key_id.clone()); + header.typ = Some(profile.token_type().to_string()); + let token = encode(&header, &claims, &self.encoding_key) + .map_err(|_| SessionJwtError::SigningFailed)?; + Ok(MintedSessionToken { + token: SecretJwt::parse(token)?, + expires_at, + token_id, + }) + } + } + + /// One accepted public key from the immutable sandbox verification bundle. + #[derive(Clone, Serialize, Deserialize)] + #[serde(deny_unknown_fields)] + pub struct SessionVerificationKey { + pub key_id: String, + pub public_key_pem: Vec, + } + + /// Strict verifier used by either the gateway or the Sandbox Protocol. + pub struct SessionJwtVerifier { + keys: BTreeMap, + issuer: String, + profile: SessionTokenProfile, + clock: Arc, + } + + impl fmt::Debug for SessionJwtVerifier { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SessionJwtVerifier") + .field("key_ids", &self.keys.keys().collect::>()) + .field("issuer", &self.issuer) + .field("profile", &self.profile) + .finish_non_exhaustive() + } + } + + #[derive(Clone, Debug, PartialEq, Eq)] + pub struct AuthenticatedSandboxSession { + pub sandbox_id: SandboxId, + pub runtime_generation: SandboxGenerationId, + pub auth_epoch: CredentialEpoch, + pub token_id: Uuid, + pub issued_at: i64, + pub expires_at: i64, + } + + impl SessionJwtVerifier { + pub fn new( + gateway_id: &str, + profile: SessionTokenProfile, + keys: impl IntoIterator, + clock: Arc, + ) -> Result { + install_crypto_provider(); + let gateway_id = validate_gateway_id(gateway_id)?; + let mut parsed = BTreeMap::new(); + for key in keys { + let key_id = validate_key_id(key.key_id)?; + let decoding_key = DecodingKey::from_ed_pem(&key.public_key_pem) + .map_err(|_| SessionJwtError::InvalidVerificationKey)?; + if parsed.insert(key_id, decoding_key).is_some() { + return Err(SessionJwtError::DuplicateKeyId); + } + } + if parsed.is_empty() { + return Err(SessionJwtError::NoVerificationKeys); + } + Ok(Self { + keys: parsed, + issuer: format!("{GATEWAY_ISSUER_PREFIX}{gateway_id}"), + profile, + clock, + }) + } + + pub fn verify(&self, token: &str) -> Result { + install_crypto_provider(); + let header = decode_header(token).map_err(|_| SessionJwtError::InvalidToken)?; + if header.alg != Algorithm::EdDSA { + return Err(SessionJwtError::WrongAlgorithm); + } + if header.typ.as_deref() != Some(self.profile.token_type()) { + return Err(SessionJwtError::WrongTokenType); + } + let key_id = header.kid.ok_or(SessionJwtError::MissingKeyId)?; + let key = self + .keys + .get(&key_id) + .ok_or(SessionJwtError::UnknownKeyId)?; + let mut validation = Validation::new(Algorithm::EdDSA); + validation.algorithms = vec![Algorithm::EdDSA]; + validation.validate_exp = false; + validation.validate_aud = false; + validation.set_required_spec_claims(&["iss", "aud", "iat", "exp", "sub", "jti"]); + let claims = decode::(token, key, &validation) + .map_err(|_| SessionJwtError::InvalidToken)? + .claims; + self.validate_claims(claims) + } + + fn validate_claims( + &self, + claims: SessionClaims, + ) -> Result { + if claims.iss != self.issuer { + return Err(SessionJwtError::WrongIssuer); + } + if claims.aud != self.profile.audience(&self.issuer) { + return Err(SessionJwtError::WrongAudience); + } + if claims.sub != format!("{SANDBOX_SUBJECT_PREFIX}{}", claims.sandbox_id) { + return Err(SessionJwtError::SubjectMismatch); + } + SandboxGenerationId::parse(claims.runtime_generation.to_string()) + .map_err(|_| SessionJwtError::InvalidRuntimeIdentity)?; + let token_id = Uuid::parse_str(&claims.jti).map_err(|_| SessionJwtError::InvalidJti)?; + if claims.exp <= claims.iat { + return Err(SessionJwtError::InvalidLifetime); + } + let lifetime = claims.exp.saturating_sub(claims.iat); + if lifetime > i64::try_from(MAX_SESSION_TOKEN_TTL.as_secs()).unwrap_or(i64::MAX) { + return Err(SessionJwtError::InvalidLifetime); + } + let now = self.clock.now_unix_seconds(); + let leeway = i64::try_from(MAX_SESSION_CLOCK_LEEWAY.as_secs()).unwrap_or(30); + if claims.iat > now.saturating_add(leeway) { + return Err(SessionJwtError::IssuedInFuture); + } + if claims.exp < now.saturating_sub(leeway) { + return Err(SessionJwtError::Expired); + } + Ok(AuthenticatedSandboxSession { + sandbox_id: claims.sandbox_id, + runtime_generation: claims.runtime_generation, + auth_epoch: claims.auth_epoch, + token_id, + issued_at: claims.iat, + expires_at: claims.exp, + }) + } + } + + #[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] + pub enum SessionJwtError { + #[error("sandbox ID is invalid")] + InvalidSandboxId, + #[error("gateway ID is invalid")] + InvalidGatewayId, + #[error("credential epoch must be positive")] + InvalidCredentialEpoch, + #[error("credential epoch does not advance the active credential")] + StaleCredentialEpoch, + #[error("session rotation must be positive")] + InvalidSessionRotation, + #[error("session rotation overflow")] + SessionRotationOverflow, + #[error("sandbox runtime identity is missing")] + MissingRuntimeIdentity, + #[error("sandbox runtime identity is invalid")] + InvalidRuntimeIdentity, + #[error("key ID is invalid")] + InvalidKeyId, + #[error("verification key IDs must be unique")] + DuplicateKeyId, + #[error("at least one verification key is required")] + NoVerificationKeys, + #[error("Ed25519 signing key is invalid")] + InvalidSigningKey, + #[error("Ed25519 verification key is invalid")] + InvalidVerificationKey, + #[error("session token lifetime must be between 60 and 3600 seconds")] + InvalidLifetime, + #[error("session token profile does not match its claims")] + ProfileMismatch, + #[error("session token could not be signed")] + SigningFailed, + #[error("session token encoding is invalid")] + InvalidTokenEncoding, + #[error("session token is unavailable")] + TokenUnavailable, + #[error("session token is invalid")] + InvalidToken, + #[error("session token algorithm must be EdDSA")] + WrongAlgorithm, + #[error("session token type is invalid")] + WrongTokenType, + #[error("session token key ID is missing")] + MissingKeyId, + #[error("session token key ID is unknown")] + UnknownKeyId, + #[error("session token issuer is invalid")] + WrongIssuer, + #[error("session token audience is invalid")] + WrongAudience, + #[error("session token subject does not match its sandbox ID")] + SubjectMismatch, + #[error("session token ID is not a UUID")] + InvalidJti, + #[error("session token was issued in the future")] + IssuedInFuture, + #[error("session token has expired")] + Expired, + } + + fn install_crypto_provider() { + let _ = jsonwebtoken::crypto::aws_lc::DEFAULT_PROVIDER.install_default(); + } + + fn validate_ttl(ttl: Duration) -> Result<(), SessionJwtError> { + if !(MIN_SESSION_TOKEN_TTL..=MAX_SESSION_TOKEN_TTL).contains(&ttl) { + return Err(SessionJwtError::InvalidLifetime); + } + Ok(()) + } + + fn validate_gateway_id(gateway_id: &str) -> Result<&str, SessionJwtError> { + if gateway_id.is_empty() + || gateway_id.trim() != gateway_id + || gateway_id.chars().any(char::is_whitespace) + { + return Err(SessionJwtError::InvalidGatewayId); + } + Ok(gateway_id) + } + + fn validate_key_id(key_id: String) -> Result { + if key_id.is_empty() || key_id.trim() != key_id || key_id.chars().any(char::is_whitespace) { + return Err(SessionJwtError::InvalidKeyId); + } + Ok(key_id) + } +} + +#[cfg(feature = "jwt")] +pub use session::*; + #[cfg(test)] mod tests { use super::*; use base64::Engine as _; + use serde::Serialize; + + #[derive(Serialize)] + struct TestClaims<'a> { + #[serde(skip_serializing_if = "Option::is_none")] + exp: Option, + #[serde(skip_serializing_if = "Option::is_none")] + sub: Option<&'a str>, + } - fn jwt_with_payload(payload: &serde_json::Value) -> String { + fn jwt_with_payload(payload: &TestClaims<'_>) -> String { let b64 = |bytes: &[u8]| base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(bytes); let header = b64(br#"{"alg":"none","typ":"JWT"}"#); let body = b64(serde_json::to_vec(payload).unwrap().as_slice()); @@ -43,23 +821,214 @@ mod tests { #[test] fn reads_integer_exp() { - let token = jwt_with_payload(&serde_json::json!({ "exp": 1_900_000_000_i64 })); + let token = jwt_with_payload(&TestClaims { + exp: Some(1_900_000_000), + sub: None, + }); assert_eq!(parse_exp_secs(&token), Some(1_900_000_000)); } #[test] fn tolerates_bearer_prefix() { - let token = jwt_with_payload(&serde_json::json!({ "exp": 42 })); + let token = jwt_with_payload(&TestClaims { + exp: Some(42), + sub: None, + }); assert_eq!(parse_exp_secs(&format!("Bearer {token}")), Some(42)); } #[test] fn none_for_missing_exp_or_non_jwt() { assert_eq!( - parse_exp_secs(&jwt_with_payload(&serde_json::json!({ "sub": "x" }))), + parse_exp_secs(&jwt_with_payload(&TestClaims { + exp: None, + sub: Some("x"), + })), None ); assert_eq!(parse_exp_secs("not-a-jwt"), None); assert_eq!(parse_exp_secs(""), None); } + + #[cfg(feature = "jwt")] + mod session_tests { + use std::sync::Arc; + + use jsonwebtoken::{Algorithm, EncodingKey, Header, encode}; + use rcgen::{KeyPair, PKCS_ED25519}; + use serde::Serialize; + + use super::super::session::*; + use crate::SandboxSessionId; + use crate::sandbox_generation::SandboxGenerationId; + + #[derive(Debug)] + struct FixedClock(i64); + + impl JwtClock for FixedClock { + fn now_unix_seconds(&self) -> i64 { + self.0 + } + } + + fn fixture() -> ( + SessionJwtIssuer, + SessionJwtVerifier, + SessionJwtVerifier, + SandboxRuntimeIdentity, + ) { + let key = KeyPair::generate_for(&PKCS_ED25519).expect("generate Ed25519 key"); + let public_key_pem = key.public_key_pem().into_bytes(); + let clock: Arc = Arc::new(FixedClock(1_900_000_000)); + let issuer = SessionJwtIssuer::from_ed25519_pem( + key.serialize_pem().as_bytes(), + "current", + "test", + DEFAULT_SESSION_TOKEN_TTL, + clock.clone(), + ) + .expect("issuer"); + let key = || SessionVerificationKey { + key_id: "current".to_string(), + public_key_pem: public_key_pem.clone(), + }; + let gateway = SessionJwtVerifier::new( + "test", + SessionTokenProfile::Gateway, + [key()], + clock.clone(), + ) + .expect("gateway verifier"); + let sandbox = + SessionJwtVerifier::new("test", SessionTokenProfile::Sandbox, [key()], clock) + .expect("sandbox verifier"); + let identity = SandboxRuntimeIdentity { + sandbox_id: SandboxId::parse("sandbox-a").expect("sandbox ID"), + runtime_generation: SandboxGenerationId::parse("generation-1") + .expect("runtime generation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + }; + (issuer, gateway, sandbox, identity) + } + + #[test] + fn token_profiles_are_not_interchangeable() { + let (issuer, gateway, sandbox, identity) = fixture(); + let pair = issuer.mint_pair(&identity).expect("token pair"); + + let gateway_session = gateway + .verify(pair.gateway.token.expose_secret()) + .expect("gateway token"); + assert_eq!(gateway_session.auth_epoch, identity.auth_epoch); + + let sandbox_session = sandbox + .verify(pair.sandbox.token.expose_secret()) + .expect("sandbox token"); + assert_eq!(sandbox_session.auth_epoch, identity.auth_epoch); + + assert_eq!( + gateway.verify(pair.sandbox.token.expose_secret()), + Err(SessionJwtError::WrongTokenType) + ); + assert_eq!( + sandbox.verify(pair.gateway.token.expose_secret()), + Err(SessionJwtError::WrongTokenType) + ); + } + + #[test] + fn token_debug_is_redacted() { + let (issuer, _gateway, _sandbox, identity) = fixture(); + let pair = issuer.mint_pair(&identity).expect("token pair"); + let debug = format!("{:?}", pair.sandbox.token); + assert_eq!(debug, "SecretJwt([REDACTED])"); + assert!(!debug.contains(pair.sandbox.token.expose_secret())); + } + + #[test] + fn supervisor_auth_bundle_round_trips_without_exposing_secrets_in_debug() { + let (issuer, _gateway, _sandbox, identity) = fixture(); + let pair = issuer.mint_pair(&identity).expect("token pair"); + let bundle = SupervisorAuthBundle { + session_id: SandboxSessionId::new(), + session_rotation: SessionRotation::new(1).expect("session rotation"), + runtime_generation: identity.runtime_generation.clone(), + auth_epoch: identity.auth_epoch, + gateway_token: pair.gateway.token, + gateway_expires_at: pair.gateway.expires_at, + sandbox_token: pair.sandbox.token, + sandbox_expires_at: pair.sandbox.expires_at, + }; + + let encoded = serde_json::to_vec(&bundle).expect("serialize auth bundle"); + let decoded: SupervisorAuthBundle = + serde_json::from_slice(&encoded).expect("deserialize auth bundle"); + assert_eq!(decoded.auth_epoch, bundle.auth_epoch); + assert_eq!( + decoded.gateway_token.expose_secret(), + bundle.gateway_token.expose_secret() + ); + assert_eq!( + decoded.sandbox_token.expose_secret(), + bundle.sandbox_token.expose_secret() + ); + + let debug = format!("{bundle:?}"); + assert!(!debug.contains(bundle.gateway_token.expose_secret())); + assert!(!debug.contains(bundle.sandbox_token.expose_secret())); + assert_eq!(debug.matches("[REDACTED]").count(), 2); + } + + #[derive(Serialize)] + struct AudienceArrayClaims<'a> { + iss: &'a str, + sub: &'a str, + aud: [&'a str; 1], + iat: i64, + exp: i64, + jti: String, + sandbox_id: &'a str, + runtime_generation: &'a str, + auth_epoch: CredentialEpoch, + component: SessionComponent, + } + + #[test] + fn audience_arrays_are_rejected() { + let key = KeyPair::generate_for(&PKCS_ED25519).expect("generate Ed25519 key"); + let clock: Arc = Arc::new(FixedClock(1_900_000_000)); + let verifier = SessionJwtVerifier::new( + "test", + SessionTokenProfile::Gateway, + [SessionVerificationKey { + key_id: "current".to_string(), + public_key_pem: key.public_key_pem().into_bytes(), + }], + clock, + ) + .expect("verifier"); + let claims = AudienceArrayClaims { + iss: "openshell-gateway:test", + sub: "spiffe://openshell/sandbox/sandbox-a", + aud: ["openshell-gateway:test"], + iat: 1_900_000_000, + exp: 1_900_003_600, + jti: uuid::Uuid::new_v4().to_string(), + sandbox_id: "sandbox-a", + runtime_generation: "generation-1", + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + component: SessionComponent::OpenShellSupervisor, + }; + let mut header = Header::new(Algorithm::EdDSA); + header.kid = Some("current".to_string()); + header.typ = Some(GATEWAY_SESSION_JWT_TYPE.to_string()); + let token = encode( + &header, + &claims, + &EncodingKey::from_ed_pem(key.serialize_pem().as_bytes()).expect("encoding key"), + ) + .expect("token"); + assert_eq!(verifier.verify(&token), Err(SessionJwtError::InvalidToken)); + } + } } diff --git a/crates/openshell-core/src/lib.rs b/crates/openshell-core/src/lib.rs index 10e2008167..50d0e054a6 100644 --- a/crates/openshell-core/src/lib.rs +++ b/crates/openshell-core/src/lib.rs @@ -43,6 +43,8 @@ pub mod proto; pub mod proto_struct; pub mod provider_credentials; pub mod sandbox_env; +pub mod sandbox_generation; +pub mod sandbox_session; pub mod secrets; pub mod settings; pub mod shell; @@ -63,6 +65,7 @@ pub use error::{ComputeDriverError, Error, Result}; pub use metadata::{ GetResourceVersion, ObjectId, ObjectLabels, ObjectName, ObjectWorkspace, SetResourceVersion, }; +pub use sandbox_session::{SandboxSessionId, SandboxSessionIdError}; /// Build version string derived from git metadata. /// diff --git a/crates/openshell-core/src/proposals.rs b/crates/openshell-core/src/proposals.rs index 53586629ca..39810f0668 100644 --- a/crates/openshell-core/src/proposals.rs +++ b/crates/openshell-core/src/proposals.rs @@ -4,7 +4,7 @@ //! Shared state controlling agent-driven policy proposals. //! //! Initialised once during sandbox start from the `agent_policy_proposals_enabled` -//! setting and updated by the policy poll loop or authoritative sidecar control +//! setting and updated by the policy poll loop or authoritative supervisor //! when the setting changes. Read by the `policy.local` route handler and by //! the skills installer to gate the agent-controlled mutation surface. @@ -15,7 +15,7 @@ use std::sync::atomic::{AtomicBool, Ordering}; /// /// Clones point at the same atomic value, so the sandbox orchestrator can pass /// this into the process and network supervisors and then update it from the -/// settings poll loop or sidecar control. +/// settings poll loop or supervisor control. #[derive(Clone, Debug)] pub struct AgentProposals { enabled: Arc, diff --git a/crates/openshell-core/src/provider_credentials.rs b/crates/openshell-core/src/provider_credentials.rs index f82fd066af..4d893fa362 100644 --- a/crates/openshell-core/src/provider_credentials.rs +++ b/crates/openshell-core/src/provider_credentials.rs @@ -171,8 +171,8 @@ impl ProviderCredentialState { /// Build a static provider state from an already-prepared child /// environment snapshot. /// - /// Kubernetes sidecar topology uses this in the process-only supervisor: - /// the network sidecar owns provider credential resolvers and sends the + /// The Kubernetes sandbox runtime uses this in the sandbox process: + /// the supervisor owns provider credential resolvers and sends the /// workload-facing env map over a local control channel. The process leaf /// must inject that map into child processes without re-placeholderizing it /// or holding the gateway-side resolver material. @@ -407,12 +407,35 @@ impl ProviderCredentialState { /// here so SDKs can read them at startup. /// 3. Everything else stays as placeholders for proxy-time resolution. pub fn child_env_with_gcp_resolved(&self) -> HashMap { - use crate::google_cloud; - let inner = self .inner .read() .expect("provider credential state poisoned"); + Self::resolve_child_env_snapshot(&inner).1 + } + + /// Return the current revision and its workload-facing environment from + /// one state snapshot. + /// + /// Remote isolation boundaries use the pair as a revisioned update. The + /// revision must describe the exact environment sent across the boundary, + /// so callers must not obtain the two values through separate lock + /// acquisitions. + pub fn child_env_snapshot_with_gcp_resolved( + &self, + ) -> std::io::Result<(u64, HashMap)> { + let inner = self + .inner + .read() + .map_err(|_| std::io::Error::other("provider credential state poisoned"))?; + Ok(Self::resolve_child_env_snapshot(&inner)) + } + + fn resolve_child_env_snapshot( + inner: &ProviderCredentialStateInner, + ) -> (u64, HashMap) { + use crate::google_cloud; + let mut env = inner.current.child_env.clone(); let has_gcp_metadata = env.contains_key("GCE_METADATA_HOST") @@ -424,7 +447,7 @@ impl ProviderCredentialState { .any(|key| env.contains_key(*key) && inner.non_secret_environment_keys.contains(*key)); if !has_gcp_metadata && !has_gcp_config { - return env; + return (inner.current.revision, env); } if has_gcp_metadata { @@ -462,7 +485,44 @@ impl ProviderCredentialState { } } - env + (inner.current.revision, env) + } + + /// Compare and install a workload-facing environment snapshot. + /// + /// Provider environment revisions are opaque content identities, not + /// ordered counters. The expected revision makes retries idempotent while + /// rejecting updates based on a stale view of the boundary state. + pub fn compare_and_install_child_env_snapshot( + &self, + expected_revision: u64, + revision: u64, + mut child_env: HashMap, + ) -> std::io::Result { + let mut inner = self + .inner + .write() + .map_err(|_| std::io::Error::other("provider credential state poisoned"))?; + if revision == inner.current.revision || expected_revision != inner.current.revision { + return Ok(inner.current.revision); + } + + for key in &inner.suppressed_keys { + child_env.remove(key); + } + inner.current = Arc::new(ProviderCredentialSnapshot { + revision, + child_env, + dynamic_credentials: HashMap::new(), + }); + inner.generations.clear(); + inner.current_resolver = None; + inner.combined_resolver = None; + inner.non_secret_environment_keys.clear(); + inner.static_credential_bindings.clear(); + inner.known_static_credential_keys.clear(); + inner.static_credential_identity_epochs.clear(); + Ok(revision) } /// Return the GCP token placeholder and its remaining lifetime in seconds. @@ -2327,6 +2387,66 @@ mod tests { ); } + #[test] + fn child_env_snapshot_update_uses_opaque_revision_cas() { + let state = ProviderCredentialState::from_child_env_snapshot( + 4, + HashMap::from([("TOKEN".to_string(), "four".to_string())]), + ); + + assert_eq!( + state + .compare_and_install_child_env_snapshot( + 4, + 6, + HashMap::from([("TOKEN".to_string(), "six".to_string())]), + ) + .unwrap(), + 6 + ); + assert_eq!( + state + .compare_and_install_child_env_snapshot( + 4, + 5, + HashMap::from([("TOKEN".to_string(), "stale".to_string())]), + ) + .unwrap(), + 6 + ); + assert_eq!( + state + .compare_and_install_child_env_snapshot(6, 2, HashMap::new()) + .unwrap(), + 2, + "opaque revisions may move numerically backwards" + ); + + let (revision, env) = state.child_env_snapshot_with_gcp_resolved().unwrap(); + assert_eq!(revision, 2); + assert!(env.is_empty(), "an empty snapshot must revoke the old env"); + } + + #[test] + fn poisoned_environment_update_returns_error_without_recovering_state() { + let state = ProviderCredentialState::from_child_env_snapshot(4, HashMap::new()); + let poison = state.clone(); + assert!( + std::thread::spawn(move || { + let _guard = poison.inner.write().unwrap(); + panic!("poison state during mutation"); + }) + .join() + .is_err() + ); + assert!( + state + .compare_and_install_child_env_snapshot(4, 5, HashMap::new()) + .is_err() + ); + assert!(state.child_env_snapshot_with_gcp_resolved().is_err()); + } + #[test] fn stale_generation_falls_back_to_current_credential_after_retention_window() { let state = ProviderCredentialState::from_environment( diff --git a/crates/openshell-core/src/sandbox_env.rs b/crates/openshell-core/src/sandbox_env.rs index 2ce8e4b058..c1c91822b6 100644 --- a/crates/openshell-core/src/sandbox_env.rs +++ b/crates/openshell-core/src/sandbox_env.rs @@ -39,7 +39,8 @@ const MAIN_PROCESS_SPEC_BASE64URL_PREFIX: &str = "base64url:"; pub struct MainProcessConfig { pub version: u32, /// Canonical command. Empty means "no command supplied": the supervisor - /// resolves the default login shell against the sandbox image. A non-empty + /// asks the sandbox boundary to resolve the default login shell against + /// the agent image. A non-empty /// command is the exact program+args and is run verbatim. pub command: Vec, pub tty: bool, @@ -51,8 +52,8 @@ impl MainProcessConfig { pub const VERSION: u32 = 1; /// Default config for a sandbox created without a command. The command is - /// left empty on purpose: the supervisor picks a login shell that exists in - /// the sandbox image (bash when present, otherwise `/bin/sh`). A TTY is + /// left empty on purpose: the sandbox boundary picks a login shell that + /// exists in the agent image (bash when present, otherwise `/bin/sh`). A TTY is /// requested because the default is an interactive login shell. #[must_use] pub fn scratch() -> Self { @@ -99,7 +100,7 @@ impl MainProcessConfig { )); } // An empty command is valid: it means "no command supplied", and the - // supervisor resolves the default login shell. Only a present-but-blank + // sandbox boundary resolves the default login shell. Only a present-but-blank // program is rejected. if !config.command.is_empty() && config.command[0].is_empty() { return Err(format!( @@ -134,6 +135,13 @@ pub const TELEMETRY_ENABLED: &str = "OPENSHELL_TELEMETRY_ENABLED"; /// `"sidecar"`; the default combined supervisor path omits it. pub const SUPERVISOR_TOPOLOGY: &str = "OPENSHELL_SUPERVISOR_TOPOLOGY"; +/// The isolation backend admitted by the deployment configuration (RFC 0012). +/// +/// Delivered on a channel separate from the topology descriptor so descriptor +/// verification against the admitted backend is not self-referential. Required +/// whenever a topology descriptor is supplied. +pub const ADMITTED_ISOLATION_BACKEND: &str = "OPENSHELL_ADMITTED_ISOLATION_BACKEND"; + /// Network enforcement backend selected by the compute driver. pub const NETWORK_ENFORCEMENT_MODE: &str = "OPENSHELL_NETWORK_ENFORCEMENT_MODE"; @@ -166,6 +174,17 @@ pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME"; /// by workload child processes. pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR"; +/// Optional path to a durable PEM-encoded interception CA certificate. +/// Must be configured together with [`PROXY_CA_KEY`]. +pub const PROXY_CA_CERT: &str = "OPENSHELL_PROXY_CA_CERT"; + +/// Optional path to the private key for [`PROXY_CA_CERT`]. +/// Must be configured together with the certificate path. +pub const PROXY_CA_KEY: &str = "OPENSHELL_PROXY_CA_KEY"; + +/// Whether the control-owned SSH Unix socket is shared across trusted UIDs. +pub const SSH_SOCKET_SHARED: &str = "OPENSHELL_SSH_SOCKET_SHARED"; + /// Path to the CA certificate for mTLS communication with the gateway. pub const TLS_CA: &str = "OPENSHELL_TLS_CA"; @@ -298,7 +317,7 @@ mod tests { #[test] fn omitted_command_stays_empty_for_supervisor_resolution() { - // No command supplied → empty command; the supervisor resolves the + // No command supplied → empty command; the sandbox boundary resolves the // default login shell against the sandbox image. let empty = crate::proto::compute::v1::DriverSandboxSpec::default(); assert!( diff --git a/crates/openshell-core/src/sandbox_generation.rs b/crates/openshell-core/src/sandbox_generation.rs new file mode 100644 index 0000000000..cf5e5dedf5 --- /dev/null +++ b/crates/openshell-core/src/sandbox_generation.rs @@ -0,0 +1,95 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::fmt; + +use serde::{Deserialize, Serialize}; + +/// Identifies one requested start of a stable sandbox. +/// +/// The gateway derives this value from the durable lifecycle transition so a +/// retried driver call carries the same identity after process restart. +#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(transparent)] +pub struct SandboxGenerationId(String); + +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum SandboxGenerationIdError { + #[error("sandbox generation ID is required")] + Empty, + #[error("sandbox generation ID exceeds 64 characters")] + TooLong, + #[error("sandbox generation ID contains an unsupported character")] + InvalidCharacter, +} + +impl SandboxGenerationId { + pub const MAX_LEN: usize = 64; + + /// Construct the stable generation assigned to a gateway start + /// transition. Resource versions are monotonic within one sandbox record. + #[must_use] + pub fn from_start_resource_version(resource_version: u64) -> Self { + Self(format!("g{resource_version:016x}")) + } + + pub fn parse(value: impl Into) -> Result { + let value = value.into(); + if value.is_empty() { + return Err(SandboxGenerationIdError::Empty); + } + if value.len() > Self::MAX_LEN { + return Err(SandboxGenerationIdError::TooLong); + } + if !value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-') + { + return Err(SandboxGenerationIdError::InvalidCharacter); + } + Ok(Self(value)) + } + + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } + + #[must_use] + pub fn into_string(self) -> String { + self.0 + } +} + +impl fmt::Display for SandboxGenerationId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +#[cfg(test)] +mod tests { + use crate::sandbox_generation::{SandboxGenerationId, SandboxGenerationIdError}; + + #[test] + fn resource_version_generation_is_stable_and_valid() { + let generation = SandboxGenerationId::from_start_resource_version(42); + assert_eq!(generation.as_str(), "g000000000000002a"); + assert_eq!( + SandboxGenerationId::parse(generation.to_string()), + Ok(generation) + ); + } + + #[test] + fn rejects_values_that_are_not_safe_runtime_identifiers() { + assert_eq!( + SandboxGenerationId::parse(""), + Err(SandboxGenerationIdError::Empty) + ); + assert_eq!( + SandboxGenerationId::parse("Generation/1"), + Err(SandboxGenerationIdError::InvalidCharacter) + ); + } +} diff --git a/crates/openshell-core/src/sandbox_session.rs b/crates/openshell-core/src/sandbox_session.rs new file mode 100644 index 0000000000..4e7678c4b9 --- /dev/null +++ b/crates/openshell-core/src/sandbox_session.rs @@ -0,0 +1,119 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Strong identity for one sandbox runtime launch. + +use std::{fmt, str::FromStr}; + +use serde::{Deserialize, Deserializer, Serialize, Serializer, de}; +use thiserror::Error; +use uuid::Uuid; + +/// Identifies one create or start-from-stopped sandbox launch. +/// +/// Retries of the same durable launch reuse this value. A later launch gets a +/// new value even when the compute platform reuses the same sandbox resource. +#[derive(Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct SandboxSessionId(Uuid); + +impl SandboxSessionId { + /// Generate a fresh launch identity. + #[must_use] + pub fn new() -> Self { + Self(Uuid::new_v4()) + } + + /// Return the underlying UUID. + #[must_use] + pub const fn as_uuid(&self) -> &Uuid { + &self.0 + } +} + +impl Default for SandboxSessionId { + fn default() -> Self { + Self::new() + } +} + +impl fmt::Display for SandboxSessionId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + self.0.hyphenated().fmt(formatter) + } +} + +impl fmt::Debug for SandboxSessionId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_tuple("SandboxSessionId") + .field(&self.to_string()) + .finish() + } +} + +impl FromStr for SandboxSessionId { + type Err = SandboxSessionIdError; + + fn from_str(value: &str) -> Result { + let parsed = Uuid::parse_str(value).map_err(|_| SandboxSessionIdError)?; + if parsed.is_nil() || parsed.hyphenated().to_string() != value { + return Err(SandboxSessionIdError); + } + Ok(Self(parsed)) + } +} + +impl Serialize for SandboxSessionId { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + serializer.collect_str(self) + } +} + +impl<'de> Deserialize<'de> for SandboxSessionId { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let value = String::deserialize(deserializer)?; + value.parse().map_err(|_| { + de::Error::invalid_value( + de::Unexpected::Str(&value), + &"a non-nil canonical lowercase UUID", + ) + }) + } +} + +/// A sandbox session ID was not a non-nil canonical lowercase UUID. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)] +#[error("sandbox session ID must be a non-nil canonical lowercase UUID")] +pub struct SandboxSessionIdError; + +#[cfg(test)] +mod tests { + use crate::sandbox_session::SandboxSessionId; + + #[test] + fn round_trips_canonical_id() { + let session = SandboxSessionId::new(); + let encoded = serde_json::to_string(&session).expect("serialize session ID"); + let decoded: SandboxSessionId = + serde_json::from_str(&encoded).expect("deserialize session ID"); + assert_eq!(decoded, session); + } + + #[test] + fn rejects_nil_and_noncanonical_ids() { + for invalid in [ + "00000000-0000-0000-0000-000000000000", + "550E8400-E29B-41D4-A716-446655440000", + "550e8400e29b41d4a716446655440000", + "not-a-uuid", + ] { + assert!(invalid.parse::().is_err(), "{invalid}"); + } + } +} diff --git a/crates/openshell-core/src/shell.rs b/crates/openshell-core/src/shell.rs index 09610afe1a..d24a7bced2 100644 --- a/crates/openshell-core/src/shell.rs +++ b/crates/openshell-core/src/shell.rs @@ -3,7 +3,7 @@ //! Login-shell resolution for sandbox images. //! -//! The default sandbox command and the interactive SSH session need a shell, +//! The default sandbox command and interactive SSH sessions need a shell, //! but not every base image ships the same one. Debian-based images provide //! `bash`; minimal images such as Alpine only provide `/bin/sh` (`BusyBox` //! `ash`). Hard-coding `/bin/bash` makes sandbox startup fail on those images @@ -51,8 +51,9 @@ pub fn is_executable(path: &str) -> bool { /// Resolve a login shell that exists in the current root filesystem. /// /// Tries [`SHELL_CANDIDATES`] in order and falls back to [`POSIX_SH`]. Because -/// this inspects the filesystem, call it from the supervisor (inside the -/// sandbox), never on the gateway. +/// this inspects the filesystem, call it from the sandbox boundary or another +/// process inside the workload filesystem, never from the external supervisor +/// or gateway. /// /// `$SHELL` is intentionally not consulted: it is image/user-controlled, the /// result is later invoked with `-lc`, and an executable that is not a diff --git a/crates/openshell-driver-docker/Cargo.toml b/crates/openshell-driver-docker/Cargo.toml index 0b03fd3880..e9751f7763 100644 --- a/crates/openshell-driver-docker/Cargo.toml +++ b/crates/openshell-driver-docker/Cargo.toml @@ -16,6 +16,8 @@ path = "src/main.rs" [dependencies] openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } openshell-otel = { path = "../openshell-otel" } opentelemetry = { workspace = true } @@ -38,15 +40,19 @@ miette = { workspace = true } toml = { workspace = true } tower-http = { workspace = true } http = { workspace = true } +rand = { workspace = true } +sha2 = { workspace = true } +rustix = { workspace = true } +libc = "0.2" +tar = "0.4" +tempfile = "3" [dev-dependencies] openshell-otel-test-support = { path = "../openshell-otel-test-support" } opentelemetry = { workspace = true } opentelemetry_sdk = { workspace = true, features = ["testing"] } prost-types = { workspace = true } -tar = "0.4" -temp-env = { version = "0.3", features = ["async_closure"] } -tempfile = "3" +temp-env = "0.3" tracing-subscriber = { workspace = true } [lints] diff --git a/crates/openshell-driver-docker/README.md b/crates/openshell-driver-docker/README.md index 10e085ecf0..b82eb3f3d7 100644 --- a/crates/openshell-driver-docker/README.md +++ b/crates/openshell-driver-docker/README.md @@ -1,148 +1,115 @@ # openshell-driver-docker -Docker-backed compute driver for local OpenShell gateways. +Docker-backed compute driver for local and remote OpenShell gateways. -When the gateway configures `[openshell.gateway.otlp]`, Docker compute-driver -spans export to the same OTLP/gRPC collector with the service name -`openshell-driver-docker`. The in-process driver preserves the gateway trace -context and emits the compute-driver RPC boundary that a standalone driver -would expose. +The driver uses `bollard` to manage sandbox resources through the configured +Docker API socket. When `socket_path` is unset, it selects the first standard +local socket that responds to an API ping. An explicitly selected Docker driver +falls back to `/var/run/docker.sock` when no candidate responds. -`mise run gateway:docker` enables this export only when a local collector is -listening on `127.0.0.1:4317`. Otherwise, it omits the gateway OTLP configuration -so the development gateway does not repeatedly report export failures. - -The standalone `openshell-driver-docker` binary accepts -`OPENSHELL_OTLP_ENDPOINT`. When set, it exports Docker driver spans to that -collector, continues W3C trace context from gateway RPC metadata, and flushes -spans during graceful shutdown. - -The driver manages sandbox containers through the local Docker daemon with the -`bollard` client. It is intended for developer environments where Docker is -already available and running Kubernetes would be unnecessary. - -The driver connects to `[openshell.drivers.docker].socket_path` when configured. -Otherwise, it uses the first standard local Docker socket that responds to an -API ping, which is the same selection mechanism used by gateway auto-detection. -An explicitly selected Docker driver falls back to `/var/run/docker.sock` when -no candidate responds. +When the gateway configures `[openshell.gateway.otlp]`, the in-process driver +exports spans to the same OTLP/gRPC collector as +`openshell-driver-docker`. The standalone driver accepts +`OPENSHELL_OTLP_ENDPOINT`, continues W3C trace context from gateway RPC +metadata, and flushes spans during graceful shutdown. ## Runtime Model -The gateway runs as a host process. The Docker driver creates one container per -sandbox and starts the `openshell-sandbox` supervisor inside that container. The -supervisor then creates the nested sandbox namespace for the agent process. - -## Stop and Start - -Stop stops the managed container without removing it. Docker retains the -container writable layer, attached volumes, labels, token material, and restart -policy. Start starts that same container, so files in the resolved OCI -workspace remain available. A durably stopped sandbox is excluded from -gateway startup recovery and stays stopped across gateway restarts. Delete -continues to force-remove the container and clean up driver-owned material. -Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted -phase requires running compute without changing that persisted intent. On -startup, the gateway sends an idempotent `StartSandbox` request for the same -sandboxes, restarting their retained containers. Explicitly stopped sandboxes -remain excluded. - -Before creating the container, the driver inspects the final sandbox image and -captures its immutable image ID, raw OCI `Config.User`, and OCI -`Config.WorkingDir`. Container creation uses that image ID, preventing a -mutable tag from changing between inspection and launch. The supervisor runs as -root, resolves omitted policy identity fields from the image declaration, and -drops only agent children to the resulting identity. Named OCI components -remain names after validation; a missing group is filled with the user's -numeric primary GID. Explicit `process.run_as_user` and -`process.run_as_group` values take precedence independently. - -An absolute OCI working directory becomes the agent workspace. An empty, -root (`/`), or explicit `/sandbox` declaration uses `/sandbox`, which OpenShell -creates when necessary and owns as a compatibility workspace. Any other image -workdir must already exist without symlink components. The completed identity, -including supplementary groups, must already be able to traverse every parent -and write and enter the workdir. OpenShell does not change its ownership or -mode. - -OpenShell deliberately asks the Linux kernel to make this access decision -under the completed sandbox identity instead of reproducing permission rules -from ownership and mode bits. Mode-bit inspection alone can reject authority -granted by a POSIX ACL or overlook a denial imposed by a Linux Security Module -such as SELinux or AppArmor. OpenShell does not configure or otherwise manage -ACLs or LSM policy here; the one-shot validator only observes the kernel's -effective decision. This keeps the no-authority-expansion invariant aligned -with the access the eventual workload will receive without adding a separate, -incomplete permission model to OpenShell. - -Image `VOLUME` declarations must not cover the workdir or one of its parents -because Docker would mount the volume before the supervisor could validate the -immutable image path. -Workdirs under the standard OCI runtime namespaces `/proc`, `/sys`, and `/dev` -are rejected, as are paths that overlap concrete OpenShell control resources. -The workspace is the child cwd and `HOME`. The supervisor starts from `/`, then -reports an invalid workdir as a readiness failure. - -Docker containers join an OpenShell-managed bridge network. The driver injects -`host.openshell.internal` and `host.docker.internal` so supervisors have stable -names for reaching the gateway host. On Docker Desktop, Colima, Rancher -Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's -`host-gateway` alias. The driver requests a separate IPv4 loopback callback -listener when the primary listener does not already cover it. On native Linux -Docker, the gateway also binds the bridge gateway IP so containers can call -back to the host process. +The driver creates two containers for each sandbox: + +- `openshell-sandbox` is PID 1 in the workload container. It owns the workload + process tree, seccomp notification broker, mandatory Landlock baseline, + binary identity, exec/signal/wait/PTY operations, and loopback forwarding. +- `openshell-supervisor` runs in a separate companion container. It owns the + gateway session, policy engine, credentials, interception CA, SSH relay, L7 + inspection, DNS policy, and external upstream connections. + +Both containers are non-root, request no capabilities, and set +no-new-privileges. A shared named volume carries the authenticated Unix socket +and sandbox bootstrap material. A second supervisor-only volume carries the +supervisor JWT and gateway client credentials and is never mounted into the +workload. + +The workload uses `network_mode=none`. Its seccomp user-notification broker +mediates every supported TCP and DNS operation, attributes it to the calling +binary, and sends the request across the private channel. The supervisor +authorizes the request before it opens an upstream connection. Docker's absent +workload network is the mandatory outer fence if mediation fails or is +bypassed. Only the trusted supervisor companion joins the driver-owned bridge, +where it originates approved egress and can resolve other services on that +network. + +The driver copies trusted runtime bytes from the configured supervisor image +through the Docker archive API. No workload launch depends on a host bind +mount or a tool supplied by the workload image, so the same path works with +local, remote, and VM-backed Docker daemons. + +## Identity and Workspace + +Before creating the workload, the driver pins the image ID and reads its +passwd/group databases through a stopped metadata container. It resolves the +admitted policy identity, or the image `Config.User` fallback, into one exact +non-root UID, primary GID, and supplementary-group set. Docker launches +`openshell-sandbox` with that identity, and the sandbox uses the same identity +for every canonical and exec process. UID or GID zero and unresolved symbolic +identities are rejected. + +An absolute OCI working directory becomes the workspace. An empty, root (`/`), +or explicit `/sandbox` declaration uses `/sandbox`. Any other workdir must +already exist without symlink components. The resolved identity must be able to +traverse every parent and write and enter the workdir; OpenShell does not +change its ownership or mode. + +Image `VOLUME` declarations and user mounts must not cover the workdir, one of +its parents, or the reserved `/.openshell` runtime/channel tree. OpenShell asks +the kernel to validate access under the final identity, so POSIX ACL and host +LSM decisions remain authoritative. ## Container Contract -The driver-controlled container settings are part of the sandbox security -contract: - | Setting | Purpose | |---|---| -| `user = "0"` | The supervisor needs root inside the container to prepare namespaces, mounts, Landlock, and seccomp. | -| `network_mode = openshell` | Places the supervisor on the managed Docker bridge network. | -| `cap_add` | Grants supervisor-only capabilities required for namespace setup and process inspection. | -| `apparmor=unconfined` | Avoids Docker's default profile blocking required mount operations. | -| `restart_policy = no` | A canonical main-process exit remains terminal and is not silently restarted by Docker. | -| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. `[openshell.drivers.docker].sandbox_pids_limit` defaults to `2048`; explicit `0` is invalid. | -| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. | -| `policy-dns-transparent-tcp` capability | Declares that the combined Docker supervisor can own namespace-local DNS/TCP capture and coupled workload restart. The shared supervisor still owns DNS eligibility, mappings, authorization, pinned dialing, relaying, and OCSF decisions. The marker is stripped from the workload environment. | - -The agent child process does not retain these supervisor privileges. +| Exact non-root `user` and `group_add` | Gives sandbox and workload the same immutable UID/GID/group identity required for capability-free observation. | +| `cap_drop = ALL`, no `cap_add`, no-new-privileges | Prevents either container from acquiring Linux capabilities. | +| Docker default seccomp and AppArmor profiles | Retains runtime hardening; startup confirmation fails closed if nested seccomp notification is unavailable. | +| `network_mode = none` on the workload | Removes direct external routes. | +| Driver-owned bridge on the supervisor | Lets the trusted supervisor originate approved gateway and upstream connections and use Docker service discovery. | +| `restart_policy = no` | Keeps canonical main-process exit terminal. | +| `PidsLimit` | Applies the configured sandbox PID budget. Omit `sandbox_pids_limit` to use OpenShell's default. Explicit zero is invalid. | +| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. | +| In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. | +| CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. | + +## Stop, Start, and Delete + +Stop terminates the supervisor companion and stops the workload container +without removing it. Docker retains the workload writable layer and attached +volumes. Start stages a fresh sandbox bootstrap bundle, restarts that workload, +and creates a new supervisor companion. A durably stopped sandbox stays stopped +across gateway restarts. + +Delete force-removes both containers, the driver-owned runtime volumes, and the +host-private runtime descriptor. Missing or altered descriptor and channel resources +fail closed; the driver does not run an older combined-supervisor layout. ## Driver Config Mounts -The gateway forwards the `docker` block from `--driver-config-json` to this -driver. The driver accepts user-supplied `mounts` entries with these Docker -mount types: - -- `bind`: mounts an absolute host path when `[openshell.drivers.docker]` - has `enable_bind_mounts = true`. -- `volume`: mounts an existing Docker named volume. The driver validates that - the volume exists before provisioning and never creates or removes it. - Docker local-driver volumes created with bind options are treated as host - bind mounts and require `enable_bind_mounts = true`. -- `tmpfs`: mounts an in-memory filesystem with optional `options`, - `size_bytes`, and `mode`. - -Host bind mounts are disabled by default because they expose gateway host -paths to sandbox requests. Image mounts are not part of the Docker -driver-config schema. The driver still uses internal bind mounts for -OpenShell-owned supervisor, token, and TLS material. - -Docker `bind` mounts accept `source`, `target`, optional `read_only`, and an -optional `selinux_label` of `shared` (applies `:z`) or `private` (applies -`:Z`) for SELinux-enforcing hosts. Docker `volume` mounts may include -`subpath`. User-supplied bind and volume mounts are read-only by default; set -`read_only: false` to make them writable. Mount `source`, `target`, and -`subpath` values must not contain surrounding whitespace. Mount targets must be -absolute container paths and must not replace or contain the resolved workspace -root. Nested workspace mounts remain valid. Mounts also must not overlap the -configured SSH socket or the reserved `/opt/openshell`, `/etc/openshell`, -`/etc/openshell-tls`, `/run/openshell`, `/run/openshell-sidecar`, and network -namespace roots. - -Example named-volume usage: +The gateway forwards the `docker` block from `--driver-config-json`. Supported +mount types are: + +- `bind`: an absolute daemon-host path, allowed only when + `[openshell.drivers.docker].enable_bind_mounts = true`. +- `volume`: an existing named volume. The driver never creates or removes a + user-supplied volume. Bind-backed local volumes require + `enable_bind_mounts = true`. +- `tmpfs`: an in-memory filesystem with optional size and mode. + +Host bind mounts are disabled by default because they expose daemon-host paths +to sandbox requests. User bind and volume mounts are read-only by default. +Targets must be absolute, normalized paths and cannot overlap the workspace +root or OpenShell control paths. + +Example: ```shell docker volume create openshell-work @@ -152,70 +119,42 @@ openshell sandbox create \ -- claude ``` -## Supervisor Binary Resolution +## Runtime Image -The Docker driver bind-mounts a host-side Linux `openshell-sandbox` binary into -each sandbox container. Resolution order is: - -1. `supervisor_bin` in `[openshell.drivers.docker]`. -2. `supervisor_image` in `[openshell.drivers.docker]`, extracting - `/openshell-sandbox` from that image. -3. A sibling `openshell-sandbox` next to the running `openshell-gateway` binary. -4. A local Linux cargo target build for the Docker daemon architecture. -5. The release-matched default supervisor image, extracting `/openshell-sandbox`. - -Release and Docker-image gateway builds bake the matching supervisor image tag -into the binary at compile time. The default Docker supervisor image is not -`:latest` unless a custom build explicitly sets that tag. +`sandbox_runtime_image` contains the statically linked musl +`/openshell-sandbox` binary. The driver extracts that binary as bytes and +stages it into the stopped workload. `supervisor_image` contains the +dynamically linked glibc `/openshell-supervisor` binary that runs in the +host-networked supervisor container. Release and gateway image builds bake +matching image tags into the binary. ## Callback and TLS -`OPENSHELL_ENDPOINT` is injected from the gateway's configured gRPC endpoint. -When no endpoint is configured, the driver uses -`host.openshell.internal:` with the appropriate HTTP or HTTPS -scheme. Set `host_gateway_ip` only when the host has an explicit, locally -assigned address that containers should use for callbacks; package-managed -macOS gateways should leave it unset. - -For HTTPS endpoints, the server certificate must include the endpoint host as a -subject alternative name. Docker sandboxes also need the client TLS bundle -mounted into the container and exposed with: - -- `OPENSHELL_TLS_CA` -- `OPENSHELL_TLS_CERT` -- `OPENSHELL_TLS_KEY` - -HTTP endpoints reject TLS material because the supervisor would not use it. - -## Corporate proxy, SPIFFE, and AppArmor - -`https_proxy`, `no_proxy`, and `proxy_auth_file` in -`[openshell.drivers.docker]` are operator-owned supervisor settings. Docker -passes the proxy URL and bypass list on the supervisor command line and mounts -an optional `user:pass` auth file read-only at a root-only path. Credentials -never appear in container environment or Docker labels. An auth file used with -an `http://` proxy requires `proxy_auth_allow_insecure = true`; an `https://` -proxy protects the Basic-auth header in its TLS session. +`OPENSHELL_ENDPOINT` and gateway authentication material are injected only into +the supervisor companion. The workload never receives the sandbox JWT, gateway +client TLS key, policy authority, or interception CA private key. -Set `provider_spiffe_workload_api_socket` to an absolute host UNIX socket to -project its dedicated parent directory into the supervisor and set -`OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET` to the guest path. TCP URIs are -rejected for this projection. `app_armor_profile` uses the shared -`RuntimeDefault`, `Unconfined`, or `Localhost/` vocabulary. Docker -uses explicit `Unconfined` by default because the supervisor's namespace mount -setup is incompatible with `docker-default`; requested confined profiles fail -at startup if Docker does not report AppArmor support. +When no endpoint is configured, the driver derives +`host.openshell.internal:`. Native Linux uses the managed bridge +gateway. Docker Desktop and compatible VM-backed daemons use Docker's +`host-gateway` route. A configured HTTPS server certificate must include the +endpoint host in its subject alternative names. -## Environment Ownership +The driver pins a concrete managed-bridge address in the sandbox descriptor. +For Docker's special `host-gateway` route, it leaves the address unpinned so +the supervisor resolves the driver-injected alias in its own container. This +avoids treating the Docker VM's loopback as the desktop host. -The driver merges template environment and sandbox spec environment first, then -overwrites security-critical keys: +The supervisor owns these security-critical variables: - `OPENSHELL_ENDPOINT` - `OPENSHELL_SANDBOX_ID` - `OPENSHELL_SANDBOX` +- `OPENSHELL_SANDBOX_TOKEN_FILE` - `OPENSHELL_SSH_SOCKET_PATH` - `OPENSHELL_MAIN_PROCESS_SPEC` - TLS path variables when HTTPS is enabled -Do not allow sandbox images or templates to override these values. +Template and sandbox environment is encoded in the protected bootstrap and +exposed only to workload children. Workload input cannot override +security-critical supervisor variables. diff --git a/crates/openshell-driver-docker/src/isolation.rs b/crates/openshell-driver-docker/src/isolation.rs new file mode 100644 index 0000000000..0ece989584 --- /dev/null +++ b/crates/openshell-driver-docker/src/isolation.rs @@ -0,0 +1,162 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Docker provisioning for the shared authenticated boundary protocol. +//! +//! Docker owns only container placement, the protected socket, and immutable OCI resource +//! claims. Lifecycle, process, network, identity, and wire behavior live in +//! `openshell-isolation-interface` and `openshell-sandbox`. + +use std::collections::{BTreeMap, HashMap}; +use std::net::IpAddr; +use std::path::PathBuf; + +use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity}; +use openshell_sandbox_backend::boundary_protocol::{ + BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor, + SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport, +}; + +/// Driver-owned inputs that bind one Docker container to one boundary. +pub struct DockerBoundarySpec { + pub boundary_id: String, + pub generation: String, + pub session_id: openshell_core::SandboxSessionId, + pub session_rotation: openshell_core::jwt::SessionRotation, + pub auth_epoch: openshell_core::jwt::CredentialEpoch, + pub gateway_id: String, + pub verification_keys: Vec, + pub container_id: String, + pub image_identity: String, + pub listener_socket: PathBuf, + pub control_socket: PathBuf, + pub sandbox_tls: SandboxTlsServerConfig, + pub supervisor_tls: SandboxTlsClientConfig, + pub host_gateway_ip: Option, + pub workload_identity: ResolvedWorkloadIdentity, + pub child_env: HashMap, +} + +/// Protected container config and matching host descriptor. +pub struct DockerBoundaryProvisioning { + pub boundary_config: BoundaryConfig, + pub runtime_descriptor: SandboxRuntimeDescriptor, +} + +impl DockerBoundarySpec { + /// Produce both sides of the common protocol from the same immutable + /// Docker coordinates so attach cannot bind a different container. + #[must_use] + pub fn provision(self) -> DockerBoundaryProvisioning { + let resource_claims = BTreeMap::from([ + ("docker.container_id".to_string(), self.container_id), + ("docker.image_identity".to_string(), self.image_identity), + ]); + let driver_fence = DriverFenceEvidence::Docker { + container_id: resource_claims["docker.container_id"].clone(), + network_mode: "none".to_string(), + unexpected_networks: Vec::new(), + }; + DockerBoundaryProvisioning { + boundary_config: BoundaryConfig { + boundary_id: self.boundary_id.clone(), + generation: self.generation.clone(), + session_id: self.session_id, + session_rotation: self.session_rotation, + auth_epoch: self.auth_epoch, + gateway_id: self.gateway_id, + verification_keys: self.verification_keys, + listener: BoundaryListener::Unix { + socket_path: self.listener_socket, + tls: self.sandbox_tls, + }, + resource_claims: resource_claims.clone(), + resource_claim_files: BTreeMap::new(), + workload_identity: self.workload_identity.clone(), + driver_fence: driver_fence.clone(), + child_env: self.child_env, + }, + runtime_descriptor: SandboxRuntimeDescriptor { + boundary_id: self.boundary_id, + generation: self.generation, + session_id: self.session_id, + workload_identity: self.workload_identity, + transport: SandboxTransport::Unix { + socket_path: self.control_socket, + }, + tls: self.supervisor_tls, + host_gateway_ip: self.host_gateway_ip, + resource_claims, + driver_fence, + }, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn provisioning_binds_container_and_image_claims() { + let session_id = openshell_core::SandboxSessionId::new(); + let tls = + openshell_sandbox_backend::boundary_protocol::generate_sandbox_tls_material(session_id) + .unwrap(); + let provisioned = DockerBoundarySpec { + boundary_id: "sandbox-1".to_string(), + generation: "generation-1".to_string(), + session_id, + session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(), + auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).unwrap(), + gateway_id: "gateway-1".to_string(), + verification_keys: vec![GatewayVerificationKey { + key_id: "key-1".to_string(), + public_key_pem: "public-key".to_string(), + }], + container_id: "sha256:container".to_string(), + image_identity: "sha256:image".to_string(), + listener_socket: PathBuf::from("/run/openshell/boundary/control.sock"), + control_socket: PathBuf::from("/host/control.sock"), + sandbox_tls: SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from("/run/openshell/boundary/server.crt"), + private_key_path: PathBuf::from("/run/openshell/boundary/server.key"), + }, + supervisor_tls: SandboxTlsClientConfig { + server_name: tls.server_name, + trust_anchor_pem: tls.trust_anchor_pem, + }, + host_gateway_ip: Some(IpAddr::from([127, 0, 0, 1])), + workload_identity: ResolvedWorkloadIdentity::new( + 1000, + 1000, + Vec::new(), + "image".to_string(), + "sha256:image".to_string(), + ) + .unwrap(), + child_env: HashMap::new(), + } + .provision(); + + assert_eq!( + provisioned.boundary_config.resource_claims, + provisioned.runtime_descriptor.resource_claims + ); + assert_eq!( + provisioned.runtime_descriptor.resource_claims["docker.container_id"], + "sha256:container" + ); + assert_eq!( + provisioned.boundary_config.driver_fence, + provisioned.runtime_descriptor.driver_fence + ); + assert!( + provisioned + .runtime_descriptor + .driver_fence + .validate() + .is_ok() + ); + } +} diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index 813c07f6c7..c3a2b12d3e 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -5,31 +5,32 @@ #![allow(clippy::result_large_err)] +mod isolation; pub mod otel_tracing; use bollard::Docker; use bollard::errors::Error as BollardError; use bollard::models::{ ContainerCreateBody, ContainerState, ContainerStateStatusEnum, ContainerSummary, - ContainerSummaryStateEnum, CreateImageInfo, DeviceRequest, EndpointSettings, HostConfig, Mount, - MountTmpfsOptions, MountTypeEnum, MountVolumeOptions, NetworkCreateRequest, NetworkingConfig, - ProgressDetail, SystemInfo, + ContainerSummaryStateEnum, CreateImageInfo, DeviceRequest, HealthConfig, HealthStatusEnum, + HostConfig, Mount, MountTmpfsOptions, MountTypeEnum, MountVolumeOptions, NetworkCreateRequest, + ProgressDetail, SystemInfo, VolumeCreateRequest, }; use bollard::query_parameters::{ CreateContainerOptionsBuilder, CreateImageOptions, DownloadFromContainerOptionsBuilder, - ListContainersOptionsBuilder, RemoveContainerOptionsBuilder, StopContainerOptionsBuilder, + ListContainersOptionsBuilder, ListVolumesOptionsBuilder, LogsOptionsBuilder, + RemoveContainerOptionsBuilder, StopContainerOptionsBuilder, UploadToContainerOptionsBuilder, }; use bytes::Bytes; use futures::{Stream, StreamExt}; use openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS; use openshell_core::driver_mounts; use openshell_core::driver_utils::{ - CONDITION_EXITED, CONDITION_RUNTIME_RESTART, CONDITION_WORKSPACE_VALIDATION_FAILED, - GatewayCallbackTopology, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, - LABEL_SANDBOX_NAME, LABEL_SANDBOX_NAMESPACE, LABEL_SANDBOX_WORKSPACE, - SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED, SUPERVISOR_IMAGE_BINARY_PATH, - extract_first_tar_entry, gateway_callback_endpoint, supervisor_image_should_refresh, - temp_extract_container_name, validate_linux_elf_binary, write_cache_binary_atomic, + CONDITION_EXITED, CONDITION_RUNTIME_RESTART, GatewayCallbackRoute, LABEL_MANAGED_BY, + LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, LABEL_SANDBOX_NAMESPACE, + LABEL_SANDBOX_WORKSPACE, SANDBOX_RUNTIME_IMAGE_BINARY_PATH, extract_first_tar_entry, + gateway_callback_endpoint, supervisor_image_should_refresh, temp_extract_container_name, + validate_linux_elf_binary, }; use openshell_core::gpu::{ CdiGpuDefaultSelector, CdiGpuInventory, CdiGpuSelectionError, driver_gpu_requirements, @@ -60,14 +61,24 @@ use openshell_core::proto_struct::{ use openshell_core::{ AppArmorProfile, Error, ImagePullPolicy, Result as CoreResult, UpstreamProxyConfig, }; +use openshell_isolation_interface::contract::ResolvedWorkloadIdentity; +use openshell_sandbox_backend::boundary_protocol::{ + BoundaryConfig, GatewayVerificationKey, SandboxRuntimeDescriptor, SandboxTlsClientConfig, + SandboxTlsServerConfig, generate_sandbox_tls_material, +}; use opentelemetry::trace::TraceContextExt as _; +use sha2::{Digest as _, Sha256}; use std::collections::{HashMap, HashSet}; +use std::fmt::Write as _; use std::net::{IpAddr, Ipv4Addr, SocketAddr}; +#[cfg(unix)] use std::path::{Path, PathBuf}; use std::pin::Pin; use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; use std::time::Duration; -use tokio::sync::{Mutex, broadcast, mpsc}; +use tokio::io::AsyncWriteExt as _; +use tokio::sync::{Mutex, broadcast, mpsc, oneshot}; use tokio::task::JoinHandle; use tokio_stream::wrappers::ReceiverStream; use tonic::{Request, Response, Status}; @@ -78,17 +89,46 @@ use url::Url; const WATCH_BUFFER: usize = 128; const WATCH_POLL_INTERVAL: Duration = Duration::from_secs(2); const WATCH_POLL_MAX_BACKOFF: Duration = Duration::from_secs(30); - -const SUPERVISOR_MOUNT_PATH: &str = openshell_core::driver_utils::SUPERVISOR_CONTAINER_BINARY; -const TLS_CA_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_CA_MOUNT_PATH; -const TLS_CERT_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_CERT_MOUNT_PATH; -const TLS_KEY_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_KEY_MOUNT_PATH; -const SANDBOX_TOKEN_MOUNT_PATH: &str = openshell_core::driver_utils::SANDBOX_TOKEN_MOUNT_PATH; -const UPSTREAM_PROXY_AUTH_MOUNT_PATH: &str = - openshell_core::driver_utils::UPSTREAM_PROXY_AUTH_MOUNT_PATH; +const SUPERVISOR_READY_TIMEOUT: Duration = Duration::from_secs(90); +// The gateway closes a supervisor session as soon as it commits a sandbox to +// Stopping, just before the compute-driver StopSandbox RPC arrives. Give that +// request a bounded opportunity to mark the control exit intentional before +// publishing a fail-closed runtime error. +const SUPERVISOR_INTENTIONAL_SHUTDOWN_GRACE: Duration = Duration::from_secs(1); +const SUPERVISOR_HEALTH_INTERVAL_NS: i64 = 250_000_000; +const SUPERVISOR_HEALTH_TIMEOUT_NS: i64 = 2_000_000_000; +const SUPERVISOR_HEALTH_START_PERIOD_NS: i64 = 60_000_000_000; + +const SANDBOX_BINARY_PATH: &str = "/.openshell/runtime/openshell-sandbox"; +const SUPERVISOR_IMAGE_CONTROL_BINARY_PATH: &str = "/openshell-supervisor"; +const SUPERVISOR_HEALTH_SOCKET_PATH: &str = "/run/openshell/health.sock"; +const SUPERVISOR_UID: u32 = 65_534; +const SUPERVISOR_GID: u32 = 65_534; +const BOUNDARY_MOUNT_PATH: &str = "/.openshell/channel"; +const BOUNDARY_CONFIG_MOUNT_PATH: &str = "/.openshell/channel/sandbox/bootstrap.json"; +const BOUNDARY_SOCKET_MOUNT_PATH: &str = "/.openshell/channel/sandbox/control.sock"; +const BOUNDARY_CERTIFICATE_MOUNT_PATH: &str = "/.openshell/channel/sandbox/server.crt"; +const BOUNDARY_PRIVATE_KEY_MOUNT_PATH: &str = "/.openshell/channel/sandbox/server.key"; +const SUPERVISOR_STATE_MOUNT_PATH: &str = "/.openshell/supervisor"; +const SUPERVISOR_PROXY_AUTH_MOUNT_PATH: &str = "/.openshell/supervisor/upstream-proxy-auth"; const PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR: &str = openshell_core::driver_utils::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR; -const SUPERVISOR_PATH: &str = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"; +const DRIVER_ADMITTED_BACKEND: &str = openshell_sandbox_backend::BACKEND_NAME; +const LABEL_ISOLATION_BACKEND: &str = "openshell.ai/isolation-backend"; +const LABEL_ISOLATION_BACKEND_OPEN_SHELL: &str = openshell_sandbox_backend::BACKEND_NAME; +const LABEL_ISOLATION_ROLE: &str = "openshell.ai/isolation-role"; +const LABEL_ISOLATION_ROLE_SANDBOX: &str = "sandbox"; +const LABEL_ISOLATION_ROLE_SUPERVISOR: &str = "supervisor"; +const LABEL_ISOLATION_ROLE_STAGING: &str = "staging"; +const LABEL_ISOLATION_ROLE_IDENTITY: &str = "identity"; +const RUNTIME_DESCRIPTOR_FILE: &str = "runtime-descriptor.json"; +const MAIN_PROCESS_SPEC_FILE: &str = "main-process.json"; +const WORKSPACE_ROOT_FILE: &str = "workspace-root"; +const BOUNDARY_CONFIG_FILE: &str = "boundary-bootstrap.json"; +const BOUNDARY_CERTIFICATE_FILE: &str = "boundary-server.crt"; +const BOUNDARY_PRIVATE_KEY_FILE: &str = "boundary-server.key"; +const SUPERVISOR_AUTH_BUNDLE_FILE: &str = "supervisor-auth.json"; +const START_GENERATION_FILE: &str = "start-generation"; const HOST_OPENSHELL_INTERNAL: &str = "host.openshell.internal"; const HOST_DOCKER_INTERNAL: &str = "host.docker.internal"; const DOCKER_NETWORK_DRIVER: &str = "bridge"; @@ -136,12 +176,16 @@ pub struct DockerComputeConfig { /// Gateway gRPC endpoint the sandbox connects back to. pub grpc_endpoint: String, - /// Optional override for the Linux `openshell-sandbox` binary mounted into containers. + /// Image containing the trusted `openshell-sandbox` binary. + pub sandbox_runtime_image: Option, + + /// Optional host path to the trusted `openshell-sandbox` binary. + /// + /// This preserves the original `supervisor_bin` configuration name while + /// the split runtime transitions to the explicit sandbox image setting. pub supervisor_bin: Option, - /// Optional image used to extract the Linux `openshell-sandbox` binary. - /// Ignored when `supervisor_bin` is set. See `resolve_supervisor_bin` for - /// the full resolution order. + /// Image containing the trusted `openshell-supervisor` binary. pub supervisor_image: Option, /// Host-side CA certificate for Docker sandbox mTLS. @@ -159,13 +203,13 @@ pub struct DockerComputeConfig { /// Host gateway IP used for sandbox host aliases. pub host_gateway_ip: String, - /// Unix socket path the in-container supervisor bridges relay traffic to. + /// Unix socket path used for interactive sandbox access. pub ssh_socket_path: String, /// Container cgroup PID limit for Docker-managed sandboxes. /// - /// Omit the field to use `OpenShell`'s 2048-process sandbox limit. Explicit - /// zero is invalid. + /// Omit the field to use `OpenShell`'s default sandbox process limit. + /// Explicit zero is invalid. #[serde( default = "openshell_core::config::default_sandbox_pids_limit", skip_serializing_if = "Option::is_none" @@ -177,19 +221,15 @@ pub struct DockerComputeConfig { #[serde(default)] pub enable_bind_mounts: bool, - /// Corporate forward-proxy settings supplied to the supervisor on argv. - /// The flattened fields retain the common `https_proxy`, `no_proxy`, and - /// `proxy_auth_*` gateway TOML contract. + /// Corporate forward-proxy settings supplied to the supervisor. #[serde(flatten)] pub upstream_proxy: UpstreamProxyConfig, - /// Host UNIX socket to project into sandbox supervisors for provider - /// SPIFFE token exchange. + /// Host UNIX socket projected into the supervisor for provider identity. pub provider_spiffe_workload_api_socket: Option, - /// `AppArmor` confinement requested for sandbox containers. The explicit - /// default preserves the prior supervisor-compatible Docker behavior. - #[serde(skip_serializing_if = "Option::is_none")] + /// `AppArmor` confinement requested for the workload container. + #[serde(default, skip_serializing_if = "Option::is_none")] pub app_armor_profile: Option, } @@ -207,6 +247,7 @@ impl DockerComputeConfig { validate_sandbox_pids_limit(self.sandbox_pids_limit)?; validate_image_pull_policy(self.image_pull_policy)?; self.upstream_proxy.validate().map_err(Error::config)?; + validate_docker_proxy_auth_file(&self.upstream_proxy)?; if let Some(socket) = self.provider_spiffe_workload_api_socket.as_deref() { openshell_core::driver_utils::validate_provider_spiffe_unix_socket(socket) .map_err(Error::config)?; @@ -229,6 +270,7 @@ impl Default for DockerComputeConfig { image_pull_policy: ImagePullPolicy::default(), sandbox_label: "default".to_string(), grpc_endpoint: String::new(), + sandbox_runtime_image: None, supervisor_bin: None, supervisor_image: None, guest_tls_ca: None, @@ -241,7 +283,7 @@ impl Default for DockerComputeConfig { enable_bind_mounts: false, upstream_proxy: UpstreamProxyConfig::default(), provider_spiffe_workload_api_socket: None, - app_armor_profile: Some(AppArmorProfile::Unconfined), + app_armor_profile: None, } } } @@ -257,15 +299,17 @@ pub(crate) struct DockerGuestTlsPaths { struct DockerDriverRuntimeConfig { default_image: String, image_pull_policy: ImagePullPolicy, - sandbox_label: String, - grpc_endpoint: String, + sandbox_namespace: String, network_name: String, gateway_route: DockerGatewayRoute, gateway_callback_bind_address: Option, - ssh_socket_path: String, stop_timeout_secs: u32, log_level: String, - supervisor_bin: PathBuf, + sandbox_binary: Arc>, + supervisor_image_id: String, + supervisor_grpc_endpoint: String, + gateway_tls_server_name: Option, + ssh_socket_path: String, guest_tls: Option, daemon_version: String, gpu: DockerGpuRuntimeCapabilities, @@ -284,10 +328,7 @@ struct DockerGpuRuntimeCapabilities { #[derive(Debug, Clone, PartialEq, Eq)] enum DockerGatewayRoute { - Bridge { - bind_address: SocketAddr, - host_alias_ip: IpAddr, - }, + Bridge { bind_address: SocketAddr }, HostGateway, } @@ -299,6 +340,31 @@ pub struct DockerComputeDriver { pending: Arc>>, gpu_selector: Arc, lifecycle_event_fences: DockerLifecycleEventFences, + control_processes: Arc>>, + runtime_failures: Arc>>, +} + +struct DockerControlProcess { + shutdown: Option>, + intentional_shutdown: Arc, + task: JoinHandle<()>, +} + +#[derive(Clone)] +struct DockerRuntimeFailure { + reason: &'static str, + message: String, +} + +#[derive(Clone)] +struct DockerRuntimeFailureContext { + docker: Arc, + events: broadcast::Sender, + failures: Arc>>, + sandbox: DriverSandbox, + sandbox_namespace: String, + container_id: String, + stop_timeout_secs: u32, } /// Per-sandbox container exit timestamps that fence snapshots from an earlier run. @@ -316,6 +382,7 @@ struct DockerLifecycleEventFences { struct DockerLifecycleFenceState { previous_finished_at: HashMap, starts_in_progress: HashSet, + stops_requested: HashSet, } impl DockerLifecycleEventFences { @@ -343,6 +410,46 @@ impl DockerLifecycleEventFences { .contains(sandbox_id) } + fn request_stop(&self, sandbox_id: &str, sandbox_name: &str) { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if !sandbox_id.is_empty() { + state.stops_requested.insert(format!("id:{sandbox_id}")); + } + if !sandbox_name.is_empty() { + state.stops_requested.insert(format!("name:{sandbox_name}")); + } + } + + fn clear_stop(&self, sandbox_id: &str, sandbox_name: &str) { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if !sandbox_id.is_empty() { + state.stops_requested.remove(&format!("id:{sandbox_id}")); + } + if !sandbox_name.is_empty() { + state + .stops_requested + .remove(&format!("name:{sandbox_name}")); + } + } + + fn stop_requested(&self, sandbox_id: &str, sandbox_name: &str) -> bool { + let state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + (!sandbox_id.is_empty() && state.stops_requested.contains(&format!("id:{sandbox_id}"))) + || (!sandbox_name.is_empty() + && state + .stops_requested + .contains(&format!("name:{sandbox_name}"))) + } + fn record_previous_exit(&self, sandbox_id: &str, finished_at: Option<&str>) { let mut state = self .state @@ -369,13 +476,17 @@ impl DockerLifecycleEventFences { .cloned() } - fn remove(&self, sandbox_id: &str) { + fn remove(&self, sandbox_id: &str, sandbox_name: &str) { let mut state = self .state .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); state.previous_finished_at.remove(sandbox_id); state.starts_in_progress.remove(sandbox_id); + state.stops_requested.remove(&format!("id:{sandbox_id}")); + state + .stops_requested + .remove(&format!("name:{sandbox_name}")); } } @@ -398,6 +509,195 @@ struct DockerImageMetadata { volumes: Vec, } +#[derive(Debug, Clone, PartialEq, Eq)] +struct DockerPasswdEntry { + name: String, + uid: u32, + gid: u32, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct DockerGroupEntry { + name: String, + gid: u32, + members: Vec, +} + +fn parse_docker_passwd(bytes: &[u8]) -> Result, Status> { + let contents = std::str::from_utf8(bytes).map_err(|error| { + Status::failed_precondition(format!("image /etc/passwd is not UTF-8: {error}")) + })?; + let mut entries = Vec::new(); + for (index, line) in contents.lines().enumerate() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + let fields = line.split(':').collect::>(); + if fields.len() < 4 || fields[0].is_empty() { + return Err(Status::failed_precondition(format!( + "image /etc/passwd line {} is malformed", + index + 1 + ))); + } + let uid = fields[2].parse::().map_err(|_| { + Status::failed_precondition(format!( + "image /etc/passwd line {} has an invalid UID", + index + 1 + )) + })?; + let gid = fields[3].parse::().map_err(|_| { + Status::failed_precondition(format!( + "image /etc/passwd line {} has an invalid GID", + index + 1 + )) + })?; + entries.push(DockerPasswdEntry { + name: fields[0].to_string(), + uid, + gid, + }); + } + Ok(entries) +} + +fn parse_docker_group(bytes: &[u8]) -> Result, Status> { + let contents = std::str::from_utf8(bytes).map_err(|error| { + Status::failed_precondition(format!("image /etc/group is not UTF-8: {error}")) + })?; + let mut entries = Vec::new(); + for (index, line) in contents.lines().enumerate() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + let fields = line.split(':').collect::>(); + if fields.len() < 4 || fields[0].is_empty() { + return Err(Status::failed_precondition(format!( + "image /etc/group line {} is malformed", + index + 1 + ))); + } + let gid = fields[2].parse::().map_err(|_| { + Status::failed_precondition(format!( + "image /etc/group line {} has an invalid GID", + index + 1 + )) + })?; + entries.push(DockerGroupEntry { + name: fields[0].to_string(), + gid, + members: fields[3] + .split(',') + .filter(|member| !member.is_empty()) + .map(str::to_string) + .collect(), + }); + } + Ok(entries) +} + +fn resolve_numeric_or_named_user<'a>( + selector: &str, + passwd: &'a [DockerPasswdEntry], +) -> Result<(u32, Option<&'a DockerPasswdEntry>), Status> { + if let Ok(uid) = selector.parse::() { + return Ok((uid, passwd.iter().find(|entry| entry.uid == uid))); + } + let entry = passwd + .iter() + .find(|entry| entry.name == selector) + .ok_or_else(|| { + Status::failed_precondition(format!( + "workload user '{selector}' does not exist in the pinned image" + )) + })?; + Ok((entry.uid, Some(entry))) +} + +fn resolve_numeric_or_named_group( + selector: &str, + groups: &[DockerGroupEntry], +) -> Result { + if let Ok(gid) = selector.parse::() { + return Ok(gid); + } + groups + .iter() + .find(|entry| entry.name == selector) + .map(|entry| entry.gid) + .ok_or_else(|| { + Status::failed_precondition(format!( + "workload group '{selector}' does not exist in the pinned image" + )) + }) +} + +fn resolve_docker_identity_from_accounts( + sandbox: &DriverSandbox, + image: &DockerImageMetadata, + passwd_bytes: &[u8], + group_bytes: &[u8], +) -> Result { + let passwd = parse_docker_passwd(passwd_bytes)?; + let groups = parse_docker_group(group_bytes)?; + let request = sandbox + .spec + .as_ref() + .and_then(|spec| spec.workload_identity.as_ref()); + let requested_user = request.map_or("", |request| request.user.trim()); + let requested_group = request.map_or("", |request| request.group.trim()); + let (image_user, image_group) = image.user.split_once(':').unwrap_or((&image.user, "")); + let user_selector = if requested_user.is_empty() { + image_user.trim() + } else { + requested_user + }; + if user_selector.is_empty() { + return Err(Status::failed_precondition( + "the pinned image defaults to root; configure a non-root process.run_as_user", + )); + } + let (uid, passwd_entry) = resolve_numeric_or_named_user(user_selector, &passwd)?; + let username = passwd_entry.map(|entry| entry.name.as_str()); + let group_selector = if requested_group.is_empty() { + image_group.trim() + } else { + requested_group + }; + let gid = if group_selector.is_empty() { + passwd_entry.map(|entry| entry.gid).ok_or_else(|| { + Status::failed_precondition(format!( + "numeric workload UID {uid} has no passwd entry; configure process.run_as_group" + )) + })? + } else { + resolve_numeric_or_named_group(group_selector, &groups)? + }; + let supplementary_gids = username.map_or_else(Vec::new, |username| { + groups + .iter() + .filter(|entry| { + entry.gid != gid && entry.members.iter().any(|member| member == username) + }) + .map(|entry| entry.gid) + .collect() + }); + let source = if !requested_user.is_empty() || !requested_group.is_empty() { + "policy" + } else { + "image" + }; + ResolvedWorkloadIdentity::new( + uid, + gid, + supplementary_gids, + source.to_string(), + image.id.clone(), + ) + .map_err(|error| Status::failed_precondition(error.to_string())) +} + #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] struct DockerResourceLimits { nano_cpus: Option, @@ -543,7 +843,6 @@ impl DockerComputeDriver { gateway_log_level: &str, docker_config: &DockerComputeConfig, ) -> CoreResult { - docker_config.validate_configuration(gateway_bind_address)?; let socket_path = docker_config .socket_path .clone() @@ -576,9 +875,15 @@ impl DockerComputeDriver { cdi_supported, wsl_all_gpu_fallback_enabled, }; - validate_docker_proxy_auth_file(&docker_config.upstream_proxy)?; + validate_sandbox_pids_limit(docker_config.sandbox_pids_limit)?; + validate_image_pull_policy(docker_config.image_pull_policy)?; validate_docker_app_armor_profile(docker_config.app_armor_profile.as_ref(), &info)?; let gateway_port = gateway_bind_address.port(); + if gateway_port == 0 { + return Err(Error::config( + "docker compute driver requires a fixed non-zero gateway bind port", + )); + } let network_name = docker_network_name(docker_config); let bridge_gateway_ip = ensure_bridge_network(&docker, &network_name).await?; let host_gateway_ip = parse_optional_host_gateway_ip(&docker_config.host_gateway_ip)?; @@ -589,34 +894,84 @@ impl DockerComputeDriver { let mut docker_config = docker_config.clone(); if docker_config.grpc_endpoint.trim().is_empty() { docker_config.grpc_endpoint = gateway_callback_endpoint( - GatewayCallbackTopology::Docker, + GatewayCallbackRoute::Docker, gateway_port, docker_guest_tls_configured(&docker_config), ); } - let grpc_endpoint = docker_container_openshell_endpoint( - &docker_config.grpc_endpoint, - HOST_OPENSHELL_INTERNAL, - gateway_port, - ); - let daemon_arch = normalize_docker_arch(version.arch.as_deref().unwrap_or_default()); - let supervisor_bin = resolve_supervisor_bin(&docker, &docker_config, &daemon_arch).await?; + let host_grpc_endpoint = + docker_host_openshell_endpoint(&docker_config.grpc_endpoint, &gateway_route)?; + let original_gateway_url = Url::parse(&docker_config.grpc_endpoint).map_err(|error| { + Error::config(format!( + "invalid docker grpc_endpoint '{}': {error}", + docker_config.grpc_endpoint + )) + })?; + let host_gateway_url = Url::parse(&host_grpc_endpoint).map_err(|error| { + Error::config(format!( + "invalid normalized Docker host grpc_endpoint '{host_grpc_endpoint}': {error}" + )) + })?; + let gateway_tls_server_name = (original_gateway_url.scheme() == "https" + && original_gateway_url.host_str() != host_gateway_url.host_str()) + .then(|| { + original_gateway_url + .host_str() + .unwrap_or_default() + .to_string() + }); + let supervisor_grpc_endpoint = match &gateway_route { + DockerGatewayRoute::Bridge { .. } => host_grpc_endpoint, + DockerGatewayRoute::HostGateway => docker_config.grpc_endpoint.clone(), + }; + let supervisor_image = docker_config + .supervisor_image + .clone() + .unwrap_or_else(openshell_core::config::default_supervisor_image); + let supervisor_image_id = + ensure_runtime_image(&docker, &supervisor_image, "supervisor").await?; + let sandbox_binary = if let Some(path) = docker_config.supervisor_bin.as_deref() { + validate_linux_elf_binary(path).map_err(Error::config)?; + Arc::new(tokio::fs::read(path).await.map_err(|error| { + Error::config(format!( + "failed to read trusted sandbox binary '{}': {error}", + path.display() + )) + })?) + } else { + let sandbox_runtime_image = docker_config + .sandbox_runtime_image + .clone() + .unwrap_or_else(openshell_core::config::default_sandbox_runtime_image); + let sandbox_runtime_image_id = + ensure_runtime_image(&docker, &sandbox_runtime_image, "sandbox runtime").await?; + Arc::new( + extract_sandbox_binary_bytes(&docker, &sandbox_runtime_image_id) + .await + .map_err(|error| { + Error::config(format!( + "failed to load trusted sandbox binary from Docker image '{sandbox_runtime_image}': {error}" + )) + })?, + ) + }; let guest_tls = docker_guest_tls_paths(&docker_config)?; - let driver = Self { docker: Arc::new(docker), config: DockerDriverRuntimeConfig { default_image: docker_config.default_image.clone(), image_pull_policy: docker_config.image_pull_policy, - sandbox_label: docker_config.sandbox_label.clone(), - grpc_endpoint, + sandbox_namespace: docker_config.sandbox_label.clone(), network_name, gateway_route, gateway_callback_bind_address, - ssh_socket_path: docker_config.ssh_socket_path.clone(), stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, log_level: gateway_log_level.to_string(), - supervisor_bin, + sandbox_binary, + supervisor_image_id, + supervisor_grpc_endpoint, + gateway_tls_server_name, + ssh_socket_path: docker_config.ssh_socket_path.clone(), guest_tls, daemon_version: version.version.unwrap_or_else(|| "unknown".to_string()), gpu, @@ -635,8 +990,19 @@ impl DockerComputeDriver { gpu.wsl_all_gpu_fallback_enabled, )), lifecycle_event_fences: DockerLifecycleEventFences::default(), + control_processes: Arc::new(Mutex::new(HashMap::new())), + runtime_failures: Arc::new(Mutex::new(HashMap::new())), }; + Box::pin(driver.reconcile_runtime_resources_at_startup()) + .await + .map_err(|error| { + Error::config(format!( + "failed to reconcile Docker isolation resources: {}", + error.message() + )) + })?; + let poll_driver = driver.clone(); tokio::spawn(async move { poll_driver.poll_loop().await; @@ -731,17 +1097,16 @@ impl DockerComputeDriver { } fn validate_sandbox_auth(sandbox: &DriverSandbox) -> Result<(), Status> { - let token_present = sandbox + let authentication = sandbox .spec .as_ref() - .is_some_and(|spec| !spec.sandbox_token.trim().is_empty()); - if token_present { - return Ok(()); - } - - Err(Status::failed_precondition( - "docker sandboxes require gateway JWT auth; configure [openshell.gateway.gateway_jwt]", - )) + .filter(|spec| !spec.launch_authentication.is_empty()) + .ok_or_else(|| { + Status::failed_precondition( + "docker sandboxes require launch-scoped gateway authentication", + ) + })?; + decode_docker_launch_authentication(&authentication.launch_authentication).map(|_| ()) } fn validate_gpu_request( @@ -842,21 +1207,93 @@ impl DockerComputeDriver { .map_err(docker_gpu_selection_status) } + async fn resolve_docker_workload_identity( + &self, + sandbox: &DriverSandbox, + image: &DockerImageMetadata, + ) -> Result { + let container_name = format!("{}-identity", temp_extract_container_name()); + self.docker + .create_container( + Some( + CreateContainerOptionsBuilder::default() + .name(container_name.as_str()) + .build(), + ), + ContainerCreateBody { + image: Some(image.id.clone()), + labels: Some(docker_auxiliary_container_labels( + sandbox, + &self.config, + LABEL_ISOLATION_ROLE_IDENTITY, + )), + ..Default::default() + }, + ) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "create Docker identity resolver container: {error}" + )) + })?; + + let result = async { + let passwd = + download_path_from_container(&self.docker, &container_name, "/etc/passwd", true) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read immutable image /etc/passwd for workload identity: {error}" + )) + })?; + let group = + download_path_from_container(&self.docker, &container_name, "/etc/group", true) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read immutable image /etc/group for workload identity: {error}" + )) + })?; + resolve_docker_identity_from_accounts(sandbox, image, &passwd, &group) + } + .await; + + if let Err(error) = self + .docker + .remove_container( + &container_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await + { + warn!( + container = container_name, + %error, + "Failed to remove Docker identity resolver container" + ); + } + result + } + async fn get_sandbox_snapshot( &self, sandbox_id: &str, sandbox_name: &str, ) -> Result, Status> { + if let Some(pending) = self.pending_snapshot(sandbox_id, sandbox_name).await? { + return Ok(Some(pending)); + } let container = self .find_managed_container_summary(sandbox_id, sandbox_name) .await?; - if let Some(sandbox) = + if let Some(mut sandbox) = container.and_then(|summary| sandbox_from_container_summary(&summary)) { + self.apply_runtime_failure(&mut sandbox).await; return Ok(Some(sandbox)); } - self.pending_snapshot(sandbox_id, sandbox_name).await + Ok(None) } async fn current_snapshots(&self) -> Result, Status> { @@ -866,36 +1303,49 @@ impl DockerComputeDriver { let Some(mut sandbox) = sandbox_from_container_summary(summary) else { continue; }; - // Docker's list summary carries no exit code, so an exited - // container is reported as the generic terminal `ContainerExited`. - // Inspect it to tell a machine/daemon-restart signal kill apart - // from an ordinary application exit, mirroring the Podman driver, - // so startup recovery can revive restart victims while leaving - // crashes terminal. - if summary.state == Some(ContainerSummaryStateEnum::EXITED) - && let Some(container_id) = summary.id.as_deref() - { + if let Some(container_id) = summary.id.as_deref() { match self.docker.inspect_container(container_id, None).await { - Ok(inspected) => { + Ok(inspected) if summary.state == Some(ContainerSummaryStateEnum::EXITED) => { + // Docker's list summary carries no exit code. Inspect + // exited containers so daemon-restart kills remain + // distinguishable from terminal application exits. if let Some(state) = inspected.state.as_ref() { apply_docker_exit_classification(&mut sandbox, state); } } + Ok(inspected) if summary.state == Some(ContainerSummaryStateEnum::RUNNING) => { + if let Err(status) = validate_docker_outer_fence(&inspected) { + let context = self + .control_failure_context(sandbox.clone(), container_id.to_string()); + handle_docker_runtime_failure( + context, + "OuterFenceViolation", + status.message().to_string(), + ) + .await; + } + } + Ok(_) => {} Err(err) => { debug!( container_id, error = %err, - "Could not inspect exited Docker container to classify its exit" + "Could not inspect Docker sandbox container during reconciliation" ); } } } + self.apply_runtime_failure(&mut sandbox).await; container_sandboxes.push(sandbox); } - let mut by_id = self.pending_snapshot_map().await; - for sandbox in container_sandboxes { - by_id.insert(sandbox.id.clone(), sandbox); - } + let mut by_id = container_sandboxes + .into_iter() + .map(|sandbox| (sandbox.id.clone(), sandbox)) + .collect::>(); + // Provisioning state is authoritative until the supervisor has + // attached to both the sandbox and gateway. A running workload + // container alone is not a usable sandbox. + by_id.extend(self.pending_snapshot_map().await); let mut sandboxes = by_id.into_values().collect::>(); sandboxes.sort_by(|left, right| left.id.cmp(&right.id)); Ok(sandboxes) @@ -932,7 +1382,7 @@ impl DockerComputeDriver { ); self.publish_sandbox_snapshot(pending_sandbox_snapshot( sandbox, - &self.config.sandbox_label, + &self.config.sandbox_namespace, provisioning_condition(), false, )); @@ -944,7 +1394,7 @@ impl DockerComputeDriver { let provisioning_span = provisioning_span(&parent, sandbox, &image); let task = tokio::spawn( async move { - driver.provision_sandbox(sandbox_for_task).await; + Box::pin(driver.provision_sandbox(sandbox_for_task)).await; } .instrument(provisioning_span), ); @@ -960,9 +1410,19 @@ impl DockerComputeDriver { } async fn provision_sandbox(&self, sandbox: DriverSandbox) { - match self.provision_sandbox_inner(&sandbox).await { + match Box::pin(self.provision_sandbox_inner(&sandbox)).await { Ok(()) => { self.clear_pending_sandbox(&sandbox.id).await; + if let Err(error) = self + .publish_container_snapshot(&sandbox.id, &sandbox.name) + .await + { + warn!( + sandbox_id = %sandbox.id, + %error, + "Failed to publish Docker sandbox snapshot after provisioning" + ); + } } Err(failure) => { self.fail_pending_sandbox(&sandbox, &failure).await; @@ -996,40 +1456,83 @@ impl DockerComputeDriver { image.ref = %template.image, )) .await?; - let token_file_created = write_sandbox_token_file(sandbox, &self.config) + let workload_identity = self + .resolve_docker_workload_identity(sandbox, &image) + .await + .map_err(|status| { + DockerProvisioningFailure::new("IdentityResolutionFailed", status.message()) + })?; + prepare_docker_boundary_state_dir(sandbox, &self.config).map_err(|status| { + DockerProvisioningFailure::new("BoundaryStateCreateFailed", status.message()) + })?; + create_docker_channel_volume(&self.docker, sandbox, &self.config) .await .map_err(|status| { - DockerProvisioningFailure::new("SandboxTokenWriteFailed", status.message()) + cleanup_docker_boundary_state(sandbox, &self.config); + DockerProvisioningFailure::new("BoundaryChannelCreateFailed", status.message()) })?; + let token_file_created = match write_sandbox_token_file(sandbox, &self.config).await { + Ok(created) => created, + Err(status) => { + let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) + .await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "SandboxTokenWriteFailed", + status.message(), + )); + } + }; + if !token_file_created { + let _ = + remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "SandboxTokenWriteFailed", + "Docker control mode requires a gateway sandbox token", + )); + } let container_name = container_name_for_sandbox(sandbox); - let gpu_devices = self + let gpu_devices = match self .resolve_gpu_cdi_devices( validated.gpu_requirements, &validated.driver_config, CdiGpuDefaultSelector::next_device_ids, ) .await - .map_err(|status| { - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); - } - DockerProvisioningFailure::new("ContainerCreateFailed", status.message()) - })?; - let create_body = build_container_create_body_for_image( + { + Ok(devices) => devices, + Err(status) => { + let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) + .await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "ContainerCreateFailed", + status.message(), + )); + } + }; + let create_body = match build_container_create_body_for_image( sandbox, &self.config, &validated.driver_config, gpu_devices.as_deref(), &image, - ) - .map_err(|status| { - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); + &workload_identity, + ) { + Ok(body) => body, + Err(status) => { + let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) + .await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "ContainerCreateFailed", + status.message(), + )); } - DockerProvisioningFailure::new("ContainerCreateFailed", status.message()) - })?; - async { + }; + let create_result = async { openshell_otel::record_error_result( self.docker .create_container( @@ -1040,16 +1543,7 @@ impl DockerComputeDriver { ), create_body, ) - .await - .map_err(|err| { - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); - } - DockerProvisioningFailure::from_status( - "ContainerCreateFailed", - create_status_from_docker_error("create docker sandbox container", err), - ) - }), + .await, ) } .instrument(tracing::info_span!( @@ -1059,7 +1553,56 @@ impl DockerComputeDriver { sandbox.id = %sandbox.id, container.name = %container_name, )) - .await?; + .await; + let created = match create_result { + Ok(created) => created, + Err(error) => { + let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) + .await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::from_status( + "ContainerCreateFailed", + create_status_from_docker_error("create docker sandbox container", error), + )); + } + }; + let inspected = match self.docker.inspect_container(&created.id, None).await { + Ok(inspected) => inspected, + Err(error) => { + let _ = self + .docker + .remove_container( + &created.id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) + .await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::from_status( + "OuterFenceInspectFailed", + internal_status("inspect Docker sandbox outer fence", error), + )); + } + }; + let outer_fence_error = validate_docker_outer_fence(&inspected).err(); + drop(inspected); + if let Some(status) = outer_fence_error { + let _ = self + .docker + .remove_container( + &created.id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + let _ = + remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::from_status( + "OuterFenceRejected", + status, + )); + } self.publish_docker_progress( &sandbox.id, "Created", @@ -1067,6 +1610,35 @@ impl DockerComputeDriver { HashMap::from([("container_name".to_string(), container_name.clone())]), ); + match prepare_docker_boundary_files( + &self.docker, + sandbox, + &self.config, + &created.id, + &image, + &workload_identity, + ) + .await + { + Ok(()) => {} + Err(status) => { + let _ = self + .docker + .remove_container( + &container_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) + .await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "BoundaryConfigWriteFailed", + status.message(), + )); + } + } + let start_result = async { openshell_otel::record_error_result( self.docker.start_container(&container_name, None).await, @@ -1096,32 +1668,351 @@ impl DockerComputeDriver { "Failed to clean up Docker container after start failure" ); } - if token_file_created { - cleanup_sandbox_token_file(sandbox, &self.config); - } + cleanup_docker_boundary_state(sandbox, &self.config); + let _ = + remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config).await; return Err(DockerProvisioningFailure::from_status( "ContainerStartFailed", create_status_from_docker_error("start docker sandbox container", err), )); } - self.publish_docker_progress( - &sandbox.id, - "Started", - format!("Started Docker container \"{container_name}\""), + self.clear_runtime_failure(&sandbox.id).await; + let failure_context = self.control_failure_context(sandbox.clone(), created.id.clone()); + let control = match spawn_docker_control_process( + &self.docker, + sandbox, + &self.config, + failure_context, + ) + .await + { + Ok(control) => control, + Err(status) => { + if self + .lifecycle_event_fences + .stop_requested(&sandbox.id, &sandbox.name) + { + debug!( + sandbox_id = %sandbox.id, + "Ignoring Docker supervisor startup interruption after an explicit stop" + ); + return span_status.finish(Ok(())); + } + let _ = self + .docker + .remove_container( + &container_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) + .await; + cleanup_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::new( + "ControlSupervisorStartFailed", + status.message(), + )); + } + }; + if self + .lifecycle_event_fences + .stop_requested(&sandbox.id, &sandbox.name) + { + stop_docker_control_process(control).await; + debug!( + sandbox_id = %sandbox.id, + "Discarded Docker supervisor that became ready after an explicit stop" + ); + return span_status.finish(Ok(())); + } + self.replace_control_process(&sandbox.id, control).await; + self.publish_docker_progress( + &sandbox.id, + "Started", + format!("Started Docker container \"{container_name}\""), HashMap::from([("container_name".to_string(), container_name)]), ); - if let Err(err) = self - .publish_container_snapshot(&sandbox.id, &sandbox.name) + span_status.finish(Ok(())) + } + + async fn replace_control_process(&self, sandbox_id: &str, process: DockerControlProcess) { + let previous = self + .control_processes + .lock() + .await + .insert(sandbox_id.to_string(), process); + if let Some(previous) = previous { + stop_docker_control_process(previous).await; + } + } + + async fn clear_runtime_failure(&self, sandbox_id: &str) { + self.runtime_failures.lock().await.remove(sandbox_id); + } + + fn control_failure_context( + &self, + sandbox: DriverSandbox, + container_id: String, + ) -> DockerRuntimeFailureContext { + DockerRuntimeFailureContext { + docker: self.docker.clone(), + events: self.events.clone(), + failures: self.runtime_failures.clone(), + sandbox, + sandbox_namespace: self.config.sandbox_namespace.clone(), + container_id, + stop_timeout_secs: self.config.stop_timeout_secs, + } + } + + async fn apply_runtime_failure(&self, sandbox: &mut DriverSandbox) { + let container_is_running = sandbox.status.as_ref().is_some_and(|status| { + status.conditions.iter().any(|condition| { + condition.r#type == "Ready" + && condition.status == "True" + && condition.reason == "BackendReady" + }) + }); + if !container_is_running { + return; + } + let failure = self.runtime_failures.lock().await.get(&sandbox.id).cloned(); + if let Some(failure) = failure { + set_sandbox_ready_condition(sandbox, error_condition(failure.reason, &failure.message)); + } + } + + async fn stop_control_process(&self, sandbox_id: &str) { + let process = self.control_processes.lock().await.remove(sandbox_id); + if let Some(process) = process { + stop_docker_control_process(process).await; + } + } + + async fn remove_auxiliary_containers_for_sandbox( + &self, + sandbox_id: &str, + ) -> Result { + let filters = managed_resource_label_filters( + &self.config.sandbox_namespace, + [format!("{LABEL_SANDBOX_ID}={sandbox_id}")], + ); + let containers = self + .docker + .list_containers(Some( + ListContainersOptionsBuilder::default() + .all(true) + .filters(&filters) + .build(), + )) + .await + .map_err(|error| internal_status("list Docker auxiliary containers", error))?; + let mut removed = false; + for container in containers { + let role = container + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_ISOLATION_ROLE)) + .map(String::as_str); + if !matches!( + role, + Some( + LABEL_ISOLATION_ROLE_SUPERVISOR + | LABEL_ISOLATION_ROLE_STAGING + | LABEL_ISOLATION_ROLE_IDENTITY + ) + ) { + continue; + } + let Some(target) = summary_container_target(&container) else { + continue; + }; + self.docker + .remove_container( + &target, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await + .or_else(|error| { + if is_not_found_error(&error) || is_removal_in_progress_error(&error) { + Ok(()) + } else { + Err(error) + } + }) + .map_err(|error| internal_status("remove Docker auxiliary container", error))?; + removed = true; + } + Ok(removed) + } + + async fn reconcile_runtime_resources_at_startup(&self) -> Result<(), Status> { + let sandboxes = self.list_managed_container_summaries().await?; + let sandbox_ids = sandboxes + .iter() + .filter_map(|container| { + container + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) + .cloned() + }) + .collect::>(); + + let filters = managed_resource_label_filters(&self.config.sandbox_namespace, []); + let auxiliary = self + .docker + .list_containers(Some( + ListContainersOptionsBuilder::default() + .all(true) + .filters(&filters) + .build(), + )) + .await + .map_err(|error| internal_status("list Docker startup resources", error))?; + for container in auxiliary { + let role = container + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_ISOLATION_ROLE)) + .map(String::as_str); + if !matches!( + role, + Some( + LABEL_ISOLATION_ROLE_SUPERVISOR + | LABEL_ISOLATION_ROLE_STAGING + | LABEL_ISOLATION_ROLE_IDENTITY + ) + ) { + continue; + } + let Some(target) = summary_container_target(&container) else { + continue; + }; + self.docker + .remove_container( + &target, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await + .or_else(|error| { + if is_not_found_error(&error) { + Ok(()) + } else { + Err(error) + } + }) + .map_err(|error| internal_status("remove stale Docker auxiliary", error))?; + } + + let volume_filters = label_filters([ + format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE}"), + format!( + "{LABEL_SANDBOX_NAMESPACE}={}", + self.config.sandbox_namespace + ), + format!("{LABEL_ISOLATION_BACKEND}={LABEL_ISOLATION_BACKEND_OPEN_SHELL}"), + ]); + let volumes = self + .docker + .list_volumes(Some( + ListVolumesOptionsBuilder::default() + .filters(&volume_filters) + .build(), + )) .await + .map_err(|error| internal_status("list Docker startup volumes", error))?; + for volume in volumes.volumes.unwrap_or_default() { + let sandbox_id = volume.labels.get(LABEL_SANDBOX_ID); + if sandbox_id.is_some_and(|id| sandbox_ids.contains(id)) { + continue; + } + self.docker + .remove_volume( + &volume.name, + None::, + ) + .await + .or_else(|error| { + if is_not_found_error(&error) { + Ok(()) + } else { + Err(error) + } + }) + .map_err(|error| internal_status("remove orphan Docker channel volume", error))?; + } + + Ok(()) + } + + async fn ensure_control_process_for_container( + &self, + container: &ContainerSummary, + ) -> Result<(), Status> { + let Some(sandbox) = sandbox_from_container_summary(container) else { + return Err(Status::internal( + "managed Docker container is missing sandbox identity labels", + )); + }; + let stale = { + let mut processes = self.control_processes.lock().await; + match processes.get(&sandbox.id) { + Some(process) if !process.task.is_finished() => return Ok(()), + Some(_) => processes.remove(&sandbox.id), + None => None, + } + }; + if let Some(stale) = stale { + stop_docker_control_process(stale).await; + } + if read_docker_runtime_descriptor(&sandbox.id, &self.config) + .await? + .is_none() { - warn!( - sandbox_id = %sandbox.id, - error = %err, - "Failed to publish Docker sandbox snapshot after start" + let container_id = summary_container_target(container) + .ok_or_else(|| Status::internal("managed Docker container has no id or name"))?; + let failure_context = self.control_failure_context(sandbox.clone(), container_id); + let status = Status::failed_precondition( + "Docker sandbox runtime descriptor is missing; refusing to leave the workload running without its supervisor", ); + handle_docker_runtime_failure( + failure_context, + "ControlSupervisorExited", + status.message().to_string(), + ) + .await; + return Err(status); } - - span_status.finish(Ok(())) + let container_id = summary_container_target(container) + .ok_or_else(|| Status::internal("managed Docker container has no id or name"))?; + self.clear_runtime_failure(&sandbox.id).await; + let failure_context = self.control_failure_context(sandbox.clone(), container_id); + let process = match spawn_docker_control_process( + &self.docker, + &sandbox, + &self.config, + failure_context.clone(), + ) + .await + { + Ok(process) => process, + Err(status) => { + handle_docker_runtime_failure( + failure_context, + "ControlSupervisorExited", + format!( + "failed to start Docker control supervisor: {}", + status.message() + ), + ) + .await; + return Err(status); + } + }; + self.replace_control_process(&sandbox.id, process).await; + Ok(()) } async fn delete_sandbox_inner( @@ -1137,6 +2028,11 @@ impl DockerComputeDriver { { task.abort(); } + if let Some(record) = pending.as_ref() { + self.stop_control_process(&record.sandbox.id).await; + self.remove_auxiliary_containers_for_sandbox(&record.sandbox.id) + .await?; + } let Some(container) = self .find_managed_container_summary(sandbox_id, sandbox_name) @@ -1153,11 +2049,25 @@ impl DockerComputeDriver { .await { Ok(()) => { - cleanup_sandbox_token_file(&record.sandbox, &self.config); + self.clear_runtime_failure(&record.sandbox.id).await; + remove_docker_channel_volume_by_id( + &self.docker, + &record.sandbox.id, + &self.config, + ) + .await?; + cleanup_docker_boundary_state(&record.sandbox, &self.config); return Ok(true); } Err(err) if is_not_found_error(&err) => { - cleanup_sandbox_token_file(&record.sandbox, &self.config); + self.clear_runtime_failure(&record.sandbox.id).await; + let _ = remove_docker_channel_volume_by_id( + &self.docker, + &record.sandbox.id, + &self.config, + ) + .await; + cleanup_docker_boundary_state(&record.sandbox, &self.config); return Ok(true); } Err(err) => { @@ -1165,16 +2075,29 @@ impl DockerComputeDriver { } } } - // Container gone and no in-memory record survived (gateway - // restarted after an out-of-band `docker rm`). DeleteSandbox is - // the only thing that ever reclaims the token file, so reclaim it - // here too. - cleanup_sandbox_token_file_for_delete(sandbox_id, None, &self.config); + if !sandbox_id.is_empty() { + self.stop_control_process(sandbox_id).await; + let removed = self + .remove_auxiliary_containers_for_sandbox(sandbox_id) + .await?; + remove_docker_channel_volume_by_id(&self.docker, sandbox_id, &self.config).await?; + cleanup_docker_boundary_state_by_id(sandbox_id, &self.config); + self.clear_runtime_failure(sandbox_id).await; + return Ok(removed); + } return Ok(false); }; let Some(target) = summary_container_target(&container) else { return Ok(pending.is_some()); }; + let resolved_sandbox_id = container + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) + .map_or(sandbox_id, String::as_str); + self.stop_control_process(resolved_sandbox_id).await; + self.remove_auxiliary_containers_for_sandbox(resolved_sandbox_id) + .await?; match self .docker @@ -1185,11 +2108,21 @@ impl DockerComputeDriver { .await { Ok(()) => { - cleanup_sandbox_token_file_for_delete(sandbox_id, pending.as_ref(), &self.config); + self.clear_runtime_failure(resolved_sandbox_id).await; + remove_docker_channel_volume_by_id(&self.docker, resolved_sandbox_id, &self.config) + .await?; + cleanup_docker_boundary_state_by_id(resolved_sandbox_id, &self.config); Ok(true) } Err(err) if is_not_found_error(&err) => { - cleanup_sandbox_token_file_for_delete(sandbox_id, pending.as_ref(), &self.config); + self.clear_runtime_failure(resolved_sandbox_id).await; + let _ = remove_docker_channel_volume_by_id( + &self.docker, + resolved_sandbox_id, + &self.config, + ) + .await; + cleanup_docker_boundary_state_by_id(resolved_sandbox_id, &self.config); Ok(pending.is_some()) } Err(err) => Err(internal_status("delete docker sandbox container", err)), @@ -1205,10 +2138,16 @@ impl DockerComputeDriver { .remove_pending_sandbox(sandbox_id, sandbox_name) .await? { + self.stop_control_process(&record.sandbox.id).await; + self.remove_auxiliary_containers_for_sandbox(&record.sandbox.id) + .await?; + self.clear_runtime_failure(&record.sandbox.id).await; if let Some(task) = record.task { task.abort(); } - cleanup_sandbox_token_file(&record.sandbox, &self.config); + remove_docker_channel_volume_by_id(&self.docker, &record.sandbox.id, &self.config) + .await?; + cleanup_docker_boundary_state(&record.sandbox, &self.config); self.publish_deleted(record.sandbox.id); return Ok(()); } @@ -1217,8 +2156,16 @@ impl DockerComputeDriver { let Some(target) = summary_container_target(&container) else { return Err(Status::not_found("sandbox container has no id or name")); }; + let resolved_sandbox_id = container + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) + .map_or(sandbox_id, String::as_str); + self.stop_control_process(resolved_sandbox_id).await; + self.remove_auxiliary_containers_for_sandbox(resolved_sandbox_id) + .await?; - match self + let result = match self .docker .stop_container( &target, @@ -1234,7 +2181,11 @@ impl DockerComputeDriver { Err(err) if is_not_modified_error(&err) => Ok(()), Err(err) if is_not_found_error(&err) => Err(Status::not_found("sandbox not found")), Err(err) => Err(internal_status("stop docker sandbox container", err)), + }; + if result.is_ok() { + self.clear_runtime_failure(resolved_sandbox_id).await; } + result } /// Start a managed sandbox container that was previously stopped. Used @@ -1247,7 +2198,7 @@ impl DockerComputeDriver { /// the sandbox, and `Err(...)` for any Docker failure. #[tracing::instrument( name = "docker.start_sandbox", - skip(self), + skip_all, fields( otel.name = "docker.start_sandbox", otel.status_code = tracing::field::Empty, @@ -1259,13 +2210,25 @@ impl DockerComputeDriver { &self, sandbox_id: &str, sandbox_name: &str, + generation_id: &str, + launch_authentication: &[u8], ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); require_sandbox_identifier(sandbox_id, sandbox_name)?; + let generation = openshell_core::sandbox_generation::SandboxGenerationId::parse( + generation_id.to_string(), + ) + .map_err(|error| Status::invalid_argument(error.to_string()))?; + self.lifecycle_event_fences + .clear_stop(sandbox_id, sandbox_name); self.lifecycle_event_fences.begin_start(sandbox_id); - let result = self - .start_sandbox_with_lifecycle_fence(sandbox_id, sandbox_name) - .await; + let result = Box::pin(self.start_sandbox_with_lifecycle_fence( + sandbox_id, + sandbox_name, + &generation, + launch_authentication, + )) + .await; self.lifecycle_event_fences.finish_start(sandbox_id); span_status.finish(result) } @@ -1274,6 +2237,8 @@ impl DockerComputeDriver { &self, sandbox_id: &str, sandbox_name: &str, + generation: &openshell_core::sandbox_generation::SandboxGenerationId, + launch_authentication: &[u8], ) -> Result { let Some(container) = self .find_managed_container_summary(sandbox_id, sandbox_name) @@ -1284,20 +2249,14 @@ impl DockerComputeDriver { let Some(target) = summary_container_target(&container) else { return Ok(false); }; + let inspected = self + .docker + .inspect_container(&target, None) + .await + .map_err(|error| internal_status("inspect Docker sandbox outer fence", error))?; + validate_docker_outer_fence(&inspected)?; let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY); - if !container_state_needs_start(state) { - return Ok(true); - } - - // Fence a poll that observed this stopped run but has not published it - // yet. Use Docker's transition timestamp so a later, genuine exit from - // the restarted container remains observable. let previous_finished_at = if state == ContainerSummaryStateEnum::EXITED { - let inspected = self - .docker - .inspect_container(&target, None) - .await - .map_err(|err| internal_status("inspect docker sandbox before start", err))?; inspected .state .as_ref() @@ -1306,26 +2265,137 @@ impl DockerComputeDriver { } else { None }; + drop(inspected); + let resolved_sandbox_id = container + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) + .map_or(sandbox_id, String::as_str); + if !container_state_needs_start(state) { + adopt_or_verify_docker_start_generation(resolved_sandbox_id, &self.config, generation) + .await?; + if launch_authentication.is_empty() { + self.ensure_control_process_for_container(&container) + .await?; + return Ok(true); + } + + // A gateway restart refreshes the supervisor-facing credentials. + // The running sandbox keeps its driver-owned channel identity, + // TLS identity, and process tree. + self.stop_control_process(resolved_sandbox_id).await; + refresh_docker_supervisor_authentication( + resolved_sandbox_id, + &self.config, + launch_authentication, + ) + .await?; + self.ensure_control_process_for_container(&container) + .await?; + return Ok(true); + } + + // Fence a poll that observed this stopped run but has not published it + // yet. Use Docker's transition timestamp so a later, genuine exit from + // the restarted container remains observable. self.lifecycle_event_fences .record_previous_exit(sandbox_id, previous_finished_at.as_deref()); + write_docker_boundary_file( + &docker_boundary_state_dir_by_id(resolved_sandbox_id, &self.config)? + .join(START_GENERATION_FILE), + generation.as_str().as_bytes(), + ) + .await?; + // Rotate launch-scoped credentials before either process starts. + if !launch_authentication.is_empty() { + refresh_docker_boundary_authentication( + resolved_sandbox_id, + &self.config, + launch_authentication, + ) + .await?; + } + let Some(runtime_descriptor) = + read_docker_runtime_descriptor(resolved_sandbox_id, &self.config).await? + else { + return Err(Status::failed_precondition( + "Docker sandbox runtime descriptor is missing; refusing to start the workload without its supervisor", + )); + }; + let boundary_config = tokio::fs::read( + docker_boundary_state_dir_by_id(resolved_sandbox_id, &self.config)? + .join(BOUNDARY_CONFIG_FILE), + ) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read Docker sandbox bootstrap for restart: {error}" + )) + })?; + let boundary_directory = + docker_boundary_state_dir_by_id(resolved_sandbox_id, &self.config)?; + let boundary_certificate = + tokio::fs::read(boundary_directory.join(BOUNDARY_CERTIFICATE_FILE)) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read Docker sandbox channel certificate for restart: {error}" + )) + })?; + let boundary_private_key = + tokio::fs::read(boundary_directory.join(BOUNDARY_PRIVATE_KEY_FILE)) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read Docker sandbox channel private key for restart: {error}" + )) + })?; + let workspace_root = tokio::fs::read_to_string( + docker_boundary_state_dir_by_id(resolved_sandbox_id, &self.config)? + .join(WORKSPACE_ROOT_FILE), + ) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read Docker sandbox workspace for restart: {error}" + )) + })?; + stage_docker_sandbox_bundle( + &self.docker, + &target, + &self.config, + &runtime_descriptor.workload_identity, + &boundary_config, + DockerSandboxTls { + certificate: &boundary_certificate, + private_key: &boundary_private_key, + }, + &workspace_root, + ) + .await?; + match self.docker.start_container(&target, None).await { - Ok(()) => Ok(true), + Ok(()) => {} // Already running — race with another start path or the // restart policy. Treat as success. - Err(err) if is_not_modified_error(&err) => Ok(true), - Err(err) if is_not_found_error(&err) => Ok(false), - Err(err) => Err(internal_status("start docker sandbox container", err)), + Err(err) if is_not_modified_error(&err) => {} + Err(err) if is_not_found_error(&err) => return Ok(false), + Err(err) => return Err(internal_status("start docker sandbox container", err)), } + adopt_or_verify_docker_start_generation(resolved_sandbox_id, &self.config, generation) + .await?; + self.ensure_control_process_for_container(&container) + .await?; + Ok(true) } async fn reserve_pending_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), Status> { let mut pending = self.pending.lock().await; - if pending.values().any(|record| { - record.sandbox.id == sandbox.id - || (record.sandbox.name == sandbox.name - && record.sandbox.workspace == sandbox.workspace) - }) { + if pending + .values() + .any(|record| record.sandbox.id == sandbox.id) + { return Err(Status::already_exists("sandbox already exists")); } @@ -1334,7 +2404,7 @@ impl DockerComputeDriver { PendingSandboxRecord { sandbox: pending_sandbox_snapshot( sandbox, - &self.config.sandbox_label, + &self.config.sandbox_namespace, provisioning_condition(), false, ), @@ -1350,10 +2420,8 @@ impl DockerComputeDriver { sandbox_name: &str, ) -> Result, Status> { let pending = self.pending.lock().await; - let Some(id) = resolve_pending_id(&pending, sandbox_id, sandbox_name)? else { - return Ok(None); - }; - Ok(pending.get(&id).map(|record| record.sandbox.clone())) + let id = pending_sandbox_record_id(&pending, sandbox_id, sandbox_name)?; + Ok(id.and_then(|id| pending.get(&id).map(|record| record.sandbox.clone()))) } async fn pending_snapshot_map(&self) -> HashMap { @@ -1375,7 +2443,7 @@ impl DockerComputeDriver { sandbox_name: &str, ) -> Result, Status> { let mut pending = self.pending.lock().await; - let Some(id) = resolve_pending_id(&pending, sandbox_id, sandbox_name)? else { + let Some(id) = pending_sandbox_record_id(&pending, sandbox_id, sandbox_name)? else { return Ok(None); }; Ok(pending.remove(&id)) @@ -1386,10 +2454,10 @@ impl DockerComputeDriver { sandbox: &DriverSandbox, failure: &DockerProvisioningFailure, ) { - cleanup_sandbox_token_file(sandbox, &self.config); + cleanup_docker_boundary_state(sandbox, &self.config); let snapshot = pending_sandbox_snapshot( sandbox, - &self.config.sandbox_label, + &self.config.sandbox_namespace, error_condition(failure.reason, &failure.message), false, ); @@ -1420,11 +2488,16 @@ impl DockerComputeDriver { sandbox_id: &str, sandbox_name: &str, ) -> Result<(), Status> { + if let Some(pending) = self.pending_snapshot(sandbox_id, sandbox_name).await? { + self.publish_sandbox_snapshot(pending); + return Ok(()); + } if let Some(summary) = self .find_managed_container_summary(sandbox_id, sandbox_name) .await? - && let Some(sandbox) = sandbox_from_container_summary(&summary) + && let Some(mut sandbox) = sandbox_from_container_summary(&summary) { + self.apply_runtime_failure(&mut sandbox).await; self.publish_sandbox_snapshot(sandbox); } Ok(()) @@ -1595,7 +2668,7 @@ impl DockerComputeDriver { } async fn list_managed_container_summaries(&self) -> Result, Status> { - let filters = managed_container_label_filters(&self.config.sandbox_label, []); + let filters = managed_container_label_filters(&self.config.sandbox_namespace, []); self.docker .list_containers(Some( ListContainersOptionsBuilder::default() @@ -1620,7 +2693,7 @@ impl DockerComputeDriver { } let filters = - managed_container_label_filters(&self.config.sandbox_label, label_filter_values); + managed_container_label_filters(&self.config.sandbox_namespace, label_filter_values); let containers = self .docker .list_containers(Some( @@ -1633,14 +2706,21 @@ impl DockerComputeDriver { .map_err(|err| internal_status("find Docker sandbox container", err))?; Ok(containers.into_iter().find(|summary| { - summary.labels.as_ref().is_some_and(|labels| { - managed_container_identity_matches( - labels, - &self.config.sandbox_label, - sandbox_id, - sandbox_name, - ) - }) + let Some(labels) = summary.labels.as_ref() else { + return false; + }; + let namespace_matches = labels + .get(LABEL_SANDBOX_NAMESPACE) + .is_some_and(|value| value == &self.config.sandbox_namespace); + let id_matches = sandbox_id.is_empty() + || labels + .get(LABEL_SANDBOX_ID) + .is_some_and(|value| value == sandbox_id); + let name_matches = sandbox_name.is_empty() + || labels + .get(LABEL_SANDBOX_NAME) + .is_some_and(|value| value == sandbox_name); + namespace_matches && id_matches && name_matches })) } @@ -1686,14 +2766,14 @@ impl DockerComputeDriver { } Err(err) if is_not_found_error(&err) => { return Err(Status::failed_precondition(format!( - "docker image '{image}' is not present locally and image_pull_policy = \"never\"" + "docker image '{image}' is not present locally and image_pull_policy=Never" ))); } Err(err) => return Err(internal_status("inspect Docker image", err)), }, ImagePullPolicy::Newer => { return Err(Status::failed_precondition( - "image_pull_policy = \"newer\" is supported only by the Podman compute driver", + "docker image_pull_policy = \"newer\" is supported only by the Podman compute driver", )); } }; @@ -1764,6 +2844,36 @@ impl DockerComputeDriver { // Standalone and in-process servers both use this wrapper. Delegating to the // driver's canonical tonic implementation keeps request validation and Docker // operation spans identical across both deployment modes. +fn validate_docker_outer_fence( + inspected: &bollard::models::ContainerInspectResponse, +) -> Result<(), Status> { + let network_mode = inspected + .host_config + .as_ref() + .and_then(|config| config.network_mode.as_deref()); + if network_mode != Some("none") { + return Err(Status::failed_precondition(format!( + "Docker sandbox outer fence requires network_mode=none, got {}", + network_mode.unwrap_or("") + ))); + } + let unexpected_networks = inspected + .network_settings + .as_ref() + .and_then(|settings| settings.networks.as_ref()) + .into_iter() + .flat_map(HashMap::keys) + .filter(|network| network.as_str() != "none") + .cloned() + .collect::>(); + if !unexpected_networks.is_empty() { + return Err(Status::failed_precondition(format!( + "Docker sandbox outer fence found attached networks: {}", + unexpected_networks.join(", ") + ))); + } + Ok(()) +} #[tonic::async_trait] impl ComputeDriver for ComputeDriverService { type WatchSandboxesStream = WatchStream; @@ -2066,10 +3176,24 @@ impl ComputeDriver for DockerComputeDriver { let request = request.into_inner(); require_sandbox_identifier(&request.sandbox_id, &request.sandbox_name)?; - self.stop_sandbox_inner(&request.sandbox_id, &request.sandbox_name) - .await?; - self.publish_container_snapshot(&request.sandbox_id, &request.sandbox_name) - .await?; + self.lifecycle_event_fences + .request_stop(&request.sandbox_id, &request.sandbox_name); + if let Err(error) = self + .stop_sandbox_inner(&request.sandbox_id, &request.sandbox_name) + .await + { + self.lifecycle_event_fences + .clear_stop(&request.sandbox_id, &request.sandbox_name); + return Err(error); + } + if let Err(error) = self + .publish_container_snapshot(&request.sandbox_id, &request.sandbox_name) + .await + { + self.lifecycle_event_fences + .clear_stop(&request.sandbox_id, &request.sandbox_name); + return Err(error); + } span_status.finish(Ok(Response::new(StopSandboxResponse {}))) } @@ -2078,7 +3202,15 @@ impl ComputeDriver for DockerComputeDriver { request: Request, ) -> Result, Status> { let request = request.into_inner(); - if !Self::start_sandbox(self, &request.sandbox_id, &request.sandbox_name).await? { + if !Box::pin(Self::start_sandbox( + self, + &request.sandbox_id, + &request.sandbox_name, + &request.generation_id, + &request.launch_authentication, + )) + .await? + { return Err(Status::not_found("sandbox not found")); } self.publish_container_snapshot(&request.sandbox_id, &request.sandbox_name) @@ -2108,7 +3240,8 @@ impl ComputeDriver for DockerComputeDriver { let deleted = self .delete_sandbox_inner(&request.sandbox_id, &request.sandbox_name) .await?; - self.lifecycle_event_fences.remove(&event_sandbox_id); + self.lifecycle_event_fences + .remove(&event_sandbox_id, &request.sandbox_name); if deleted && !event_sandbox_id.is_empty() { let _ = self.events.send(WatchSandboxesEvent { payload: Some(watch_sandboxes_event::Payload::Deleted( @@ -2222,78 +3355,34 @@ fn pending_sandbox_snapshot( sandbox_fd: String::new(), conditions: vec![condition], deleting, + ..Default::default() }), workspace: sandbox.workspace.clone(), } } -/// Decides whether a managed container satisfies a lifecycle request. -/// -/// `sandbox_id` is authoritative, matching [`resolve_pending_id`]. Requiring -/// the name to agree as well would discard a correct id match whenever the -/// caller pairs it with a stale name, leaving the container and its token file -/// behind while the driver reports the sandbox as absent. -/// -/// A request with no identifier matches nothing. `require_sandbox_identifier` -/// rejects that upstream, but the label filters degenerate to "every managed -/// container in the namespace", so this does not rely on the caller to guard it. -fn managed_container_identity_matches( - labels: &HashMap, - namespace: &str, - sandbox_id: &str, - sandbox_name: &str, -) -> bool { - if labels - .get(LABEL_SANDBOX_NAMESPACE) - .is_none_or(|value| value != namespace) - { - return false; - } - if !sandbox_id.is_empty() { - return labels - .get(LABEL_SANDBOX_ID) - .is_some_and(|value| value == sandbox_id); - } - !sandbox_name.is_empty() - && labels - .get(LABEL_SANDBOX_NAME) - .is_some_and(|value| value == sandbox_name) -} - -/// Resolves a lifecycle request to at most one pending sandbox id. -/// -/// `sandbox_id` is authoritative: when the caller supplies one, the name is -/// never consulted as an alternative. The name fallback rejects ambiguity -/// instead of letting `HashMap` iteration order pick a match, because sandbox -/// names are unique per workspace and the driver request carries no workspace. -fn resolve_pending_id( - pending: &HashMap, +fn pending_sandbox_record_id( + pending: &HashMap, sandbox_id: &str, sandbox_name: &str, ) -> Result, Status> { if !sandbox_id.is_empty() { return Ok(pending - .contains_key(sandbox_id) - .then(|| sandbox_id.to_string())); - } - if sandbox_name.is_empty() { - return Ok(None); + .get(sandbox_id) + .map(|record| record.sandbox.id.clone())); } let mut matches = pending - .iter() - .filter(|(_, record)| record.sandbox.name == sandbox_name) - .map(|(id, _)| id.clone()); - - let Some(id) = matches.next() else { - return Ok(None); - }; - if matches.next().is_some() { - return Err(Status::failed_precondition( - "sandbox_name matches multiple pending sandboxes; specify sandbox_id", - )); + .values() + .filter(|record| !sandbox_name.is_empty() && record.sandbox.name == sandbox_name) + .map(|record| record.sandbox.id.clone()); + let first = matches.next(); + if first.is_some() && matches.next().is_some() { + return Err(Status::failed_precondition(format!( + "multiple pending Docker sandboxes are named '{sandbox_name}'; use sandbox_id" + ))); } - Ok(Some(id)) + Ok(first) } fn provisioning_condition() -> DriverCondition { @@ -2316,6 +3405,21 @@ fn error_condition(reason: &str, message: &str) -> DriverCondition { } } +fn set_sandbox_ready_condition(sandbox: &mut DriverSandbox, condition: DriverCondition) { + let Some(status) = sandbox.status.as_mut() else { + return; + }; + if let Some(existing) = status + .conditions + .iter_mut() + .find(|existing| existing.r#type == "Ready") + { + *existing = condition; + } else { + status.conditions.push(condition); + } +} + fn platform_event( source: &str, event_type: &str, @@ -2737,96 +3841,168 @@ fn docker_volume_is_bind_backed(volume: &bollard::models::Volume) -> bool { }) } -/// Verify the configured credential without exposing its contents. Docker -/// bind-mounts the root-owned file directly, unlike Podman which uses a native -/// secret object; this preflight makes a bad file fail before any sandbox is -/// created. -fn validate_docker_proxy_auth_file(config: &UpstreamProxyConfig) -> CoreResult<()> { - let Some(path) = config.proxy_auth_file.as_ref() else { - return Ok(()); - }; - let raw = openshell_core::driver_utils::read_upstream_proxy_credential_file( - path.to_str() - .ok_or_else(|| Error::config("proxy_auth_file must be valid UTF-8"))?, - ) - .map_err(Error::config)?; - openshell_core::driver_utils::parse_upstream_proxy_credential(&raw) - .map_err(|error| Error::config(format!("proxy_auth_file is invalid: {error}")))?; - Ok(()) +fn build_binds(_sandbox: &DriverSandbox, _config: &DockerDriverRuntimeConfig) -> Vec { + Vec::new() } -/// Build immutable operator-owned proxy arguments. Credentials never appear on -/// argv: only the fixed in-container root-only file path is supplied. -fn docker_upstream_proxy_cli_args(config: &UpstreamProxyConfig) -> Vec { - let mut args = Vec::new(); - if let Some(url) = config.https_proxy.as_ref() { - args.extend(["--upstream-proxy".to_string(), url.clone()]); - } - if let Some(no_proxy) = config.no_proxy.as_ref() { - args.extend(["--upstream-no-proxy".to_string(), no_proxy.clone()]); - } - if config.proxy_auth_file.is_some() { - args.extend([ - "--upstream-proxy-auth-file".to_string(), - UPSTREAM_PROXY_AUTH_MOUNT_PATH.to_string(), - ]); - } - if config.proxy_auth_allow_insecure == Some(true) { - args.push("--upstream-proxy-auth-allow-insecure".to_string()); - } - if config.proxy_connect_by_hostname == Some(true) { - args.push("--upstream-proxy-connect-by-hostname".to_string()); - } - args +fn docker_boundary_state_dir( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) -> Result { + docker_boundary_state_dir_by_id(&sandbox.id, config) } -fn build_binds( +fn docker_boundary_state_dir_by_id( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result { + sandbox_token_host_path_by_id(sandbox_id, config).and_then(|path| { + path.parent() + .map(Path::to_path_buf) + .ok_or_else(|| Status::internal("docker boundary state path has no parent")) + }) +} + +fn docker_channel_volume_name( sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, -) -> Result, Status> { - let mut binds = vec![format!( - "{}:{}:ro,z", - config.supervisor_bin.display(), - SUPERVISOR_MOUNT_PATH - )]; - if let Some(tls) = &config.guest_tls { - binds.push(format!("{}:{}:ro,z", tls.ca.display(), TLS_CA_MOUNT_PATH)); - binds.push(format!( - "{}:{}:ro,z", - tls.cert.display(), - TLS_CERT_MOUNT_PATH - )); - binds.push(format!("{}:{}:ro,z", tls.key.display(), TLS_KEY_MOUNT_PATH)); - } - if sandbox - .spec - .as_ref() - .is_some_and(|spec| !spec.sandbox_token.is_empty()) +) -> String { + docker_channel_volume_name_by_id(&sandbox.id, config) +} + +fn docker_channel_volume_name_by_id( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> String { + let mut hasher = Sha256::new(); + hasher.update(config.sandbox_namespace.as_bytes()); + hasher.update([0]); + hasher.update(sandbox_id.as_bytes()); + let digest = format!("{:x}", hasher.finalize()); + format!("openshell-channel-{}", &digest[..32]) +} + +fn docker_supervisor_volume_name( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) -> String { + docker_supervisor_volume_name_by_id(&sandbox.id, config) +} + +fn docker_supervisor_volume_name_by_id( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> String { + let mut hasher = Sha256::new(); + hasher.update(config.sandbox_namespace.as_bytes()); + hasher.update([0]); + hasher.update(sandbox_id.as_bytes()); + let digest = format!("{:x}", hasher.finalize()); + format!("openshell-supervisor-{}", &digest[..32]) +} + +async fn create_docker_channel_volume( + docker: &Docker, + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + create_docker_runtime_volume( + docker, + sandbox, + config, + docker_channel_volume_name(sandbox, config), + ) + .await?; + if let Err(error) = create_docker_runtime_volume( + docker, + sandbox, + config, + docker_supervisor_volume_name(sandbox, config), + ) + .await { - binds.push(format!( - "{}:{}:ro,z", - sandbox_token_host_path(sandbox, config)?.display(), - SANDBOX_TOKEN_MOUNT_PATH - )); - } - if let Some(path) = config.upstream_proxy.proxy_auth_file.as_ref() { - binds.push(format!( - "{}:{}:ro,z", - path.display(), - UPSTREAM_PROXY_AUTH_MOUNT_PATH - )); + let _ = remove_docker_volume(docker, &docker_channel_volume_name(sandbox, config)).await; + return Err(error); } - if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() { - let parent = socket.parent().ok_or_else(|| { - Status::failed_precondition("provider SPIFFE socket has no parent directory") + Ok(()) +} + +async fn create_docker_runtime_volume( + docker: &Docker, + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + name: String, +) -> Result<(), Status> { + let expected_labels = HashMap::from([ + ( + LABEL_MANAGED_BY.to_string(), + LABEL_MANAGED_BY_VALUE.to_string(), + ), + (LABEL_SANDBOX_ID.to_string(), sandbox.id.clone()), + ( + LABEL_SANDBOX_NAMESPACE.to_string(), + config.sandbox_namespace.clone(), + ), + ( + LABEL_ISOLATION_BACKEND.to_string(), + LABEL_ISOLATION_BACKEND_OPEN_SHELL.to_string(), + ), + ]); + docker + .create_volume(VolumeCreateRequest { + name: Some(name.clone()), + labels: Some(expected_labels.clone()), + ..Default::default() + }) + .await + .map_err(|error| { + Status::internal(format!("create Docker sandbox runtime volume: {error}")) })?; - binds.push(format!( - "{}:{}:ro", - parent.display(), - PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR - )); + let volume = docker.inspect_volume(&name).await.map_err(|error| { + Status::internal(format!("inspect Docker sandbox channel volume: {error}")) + })?; + if volume.driver != "local" + || !volume.options.is_empty() + || expected_labels + .iter() + .any(|(key, value)| volume.labels.get(key) != Some(value)) + { + return Err(Status::failed_precondition(format!( + "Docker sandbox runtime volume '{name}' already exists without the expected local-driver ownership labels" + ))); } - Ok(binds) + Ok(()) +} + +async fn remove_docker_channel_volume_by_id( + docker: &Docker, + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + remove_docker_volume( + docker, + &docker_channel_volume_name_by_id(sandbox_id, config), + ) + .await?; + remove_docker_volume( + docker, + &docker_supervisor_volume_name_by_id(sandbox_id, config), + ) + .await +} + +async fn remove_docker_volume(docker: &Docker, name: &str) -> Result<(), Status> { + docker + .remove_volume(name, None::) + .await + .or_else(|error| { + if is_not_found_error(&error) { + Ok(()) + } else { + Err(error) + } + }) + .map_err(|error| Status::internal(format!("remove Docker sandbox runtime volume: {error}"))) } fn sandbox_token_host_path( @@ -2842,7 +4018,7 @@ fn sandbox_token_host_path_by_id( ) -> Result { openshell_core::driver_utils::sandbox_token_path( "docker-sandbox-tokens", - Some(&config.sandbox_label), + Some(&config.sandbox_namespace), sandbox_id, ) .map_err(|err| { @@ -2873,188 +4049,1319 @@ async fn write_sandbox_token_file( } tokio::fs::write(&path, format!("{}\n", spec.sandbox_token)) .await - .map_err(|err| { - Status::internal(format!( - "write sandbox token file {} failed: {err}", - path.display() - )) - })?; - openshell_core::paths::set_file_owner_only(&path).map_err(|err| { - Status::internal(format!( - "restrict sandbox token file {} failed: {err}", - path.display() - )) - })?; - Ok(true) + .map_err(|err| { + Status::internal(format!( + "write sandbox token file {} failed: {err}", + path.display() + )) + })?; + openshell_core::paths::set_file_owner_only(&path).map_err(|err| { + Status::internal(format!( + "restrict sandbox token file {} failed: {err}", + path.display() + )) + })?; + Ok(true) +} + +fn prepare_docker_boundary_state_dir( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) -> Result { + let directory = docker_boundary_state_dir(sandbox, config)?; + openshell_core::paths::create_dir_restricted(&directory).map_err(|error| { + Status::internal(format!( + "create Docker boundary state directory {}: {error}", + directory.display() + )) + })?; + Ok(directory) +} + +async fn write_docker_boundary_file(path: &Path, contents: &[u8]) -> Result<(), Status> { + tokio::fs::write(path, contents).await.map_err(|error| { + Status::internal(format!( + "write Docker boundary file {}: {error}", + path.display() + )) + })?; + openshell_core::paths::set_file_owner_only(path).map_err(|error| { + Status::internal(format!( + "restrict Docker boundary file {}: {error}", + path.display() + )) + }) +} + +async fn adopt_or_verify_docker_start_generation( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, + requested: &openshell_core::sandbox_generation::SandboxGenerationId, +) -> Result<(), Status> { + let path = docker_boundary_state_dir_by_id(sandbox_id, config)?.join(START_GENERATION_FILE); + adopt_or_verify_docker_start_generation_path(&path, requested).await +} + +async fn adopt_or_verify_docker_start_generation_path( + path: &Path, + requested: &openshell_core::sandbox_generation::SandboxGenerationId, +) -> Result<(), Status> { + let active = match tokio::fs::read_to_string(&path).await { + Ok(active) => active, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + let mut marker = match tokio::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&path) + .await + { + Ok(marker) => marker, + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => { + let active = tokio::fs::read_to_string(&path).await.map_err(|error| { + Status::failed_precondition(format!( + "read concurrently adopted Docker sandbox generation {}: {error}", + path.display() + )) + })?; + return verify_docker_start_generation_value(&active, requested); + } + Err(error) => { + return Err(Status::failed_precondition(format!( + "adopt active Docker sandbox generation {}: {error}", + path.display() + ))); + } + }; + marker + .write_all(requested.as_str().as_bytes()) + .await + .map_err(|error| { + Status::internal(format!( + "write adopted Docker sandbox generation {}: {error}", + path.display() + )) + })?; + marker.flush().await.map_err(|error| { + Status::internal(format!( + "flush adopted Docker sandbox generation {}: {error}", + path.display() + )) + })?; + openshell_core::paths::set_file_owner_only(path).map_err(|error| { + Status::internal(format!( + "restrict adopted Docker sandbox generation {}: {error}", + path.display() + )) + })?; + return Ok(()); + } + Err(error) => { + return Err(Status::failed_precondition(format!( + "read active Docker sandbox generation {}: {error}", + path.display() + ))); + } + }; + verify_docker_start_generation_value(&active, requested) +} + +fn verify_docker_start_generation_value( + active: &str, + requested: &openshell_core::sandbox_generation::SandboxGenerationId, +) -> Result<(), Status> { + if active.trim() == requested.as_str() { + return Ok(()); + } + Err(Status::failed_precondition(format!( + "Docker sandbox is already running generation {}", + active.trim() + ))) +} + +fn append_docker_archive_directory( + archive: &mut tar::Builder>, + path: &str, + mode: u32, + uid: u32, + gid: u32, +) -> Result<(), Status> { + let mut header = tar::Header::new_gnu(); + header.set_entry_type(tar::EntryType::Directory); + header.set_mode(mode); + header.set_uid(u64::from(uid)); + header.set_gid(u64::from(gid)); + header.set_mtime(0); + header.set_size(0); + header.set_cksum(); + archive + .append_data(&mut header, path, std::io::empty()) + .map_err(|error| Status::internal(format!("build Docker sandbox archive: {error}"))) +} + +fn append_docker_archive_file( + archive: &mut tar::Builder>, + path: &str, + mode: u32, + uid: u32, + gid: u32, + contents: &[u8], +) -> Result<(), Status> { + let mut header = tar::Header::new_gnu(); + header.set_entry_type(tar::EntryType::Regular); + header.set_mode(mode); + header.set_uid(u64::from(uid)); + header.set_gid(u64::from(gid)); + header.set_mtime(0); + header.set_size(contents.len() as u64); + header.set_cksum(); + archive + .append_data(&mut header, path, contents) + .map_err(|error| Status::internal(format!("build Docker sandbox archive: {error}"))) +} + +#[derive(Clone, Copy)] +struct DockerSandboxTls<'a> { + certificate: &'a [u8], + private_key: &'a [u8], +} + +fn docker_sandbox_bundle_archive( + sandbox_binary: &[u8], + boundary_config: &[u8], + boundary_tls: DockerSandboxTls<'_>, + identity: &ResolvedWorkloadIdentity, + workspace_root: &str, +) -> Result, Status> { + let mut archive = tar::Builder::new(Vec::new()); + append_docker_archive_directory(&mut archive, ".openshell", 0o755, 0, 0)?; + append_docker_archive_directory(&mut archive, ".openshell/runtime", 0o555, 0, 0)?; + append_docker_archive_directory(&mut archive, ".openshell/channel", 0o755, 0, 0)?; + append_docker_archive_directory( + &mut archive, + ".openshell/channel/sandbox", + // The sandbox owns this directory so it can consume bootstrap files + // and create the control socket. The separate non-root supervisor + // needs execute-only traversal to that known socket path; mutual TLS + // authenticates the endpoint and the files beneath remain 0600. + 0o711, + identity.uid, + identity.gid, + )?; + append_docker_archive_file( + &mut archive, + ".openshell/runtime/openshell-sandbox", + 0o555, + 0, + 0, + sandbox_binary, + )?; + append_docker_archive_file( + &mut archive, + ".openshell/channel/sandbox/bootstrap.json", + 0o600, + identity.uid, + identity.gid, + boundary_config, + )?; + for (path, contents) in [ + ( + ".openshell/channel/sandbox/server.crt", + boundary_tls.certificate, + ), + ( + ".openshell/channel/sandbox/server.key", + boundary_tls.private_key, + ), + ] { + append_docker_archive_file( + &mut archive, + path, + 0o600, + identity.uid, + identity.gid, + contents, + )?; + } + if workspace_root == driver_mounts::DEFAULT_WORKSPACE_ROOT { + // The default workspace is driver-managed. Create it before the + // capability-free sandbox starts because that process deliberately + // has no authority to create or chown a directory beneath `/`. + append_docker_archive_directory( + &mut archive, + workspace_root.trim_start_matches('/'), + 0o700, + identity.uid, + identity.gid, + )?; + } + archive + .into_inner() + .map_err(|error| Status::internal(format!("finish Docker sandbox archive: {error}"))) +} + +async fn stage_docker_sandbox_bundle( + docker: &Docker, + container_id: &str, + config: &DockerDriverRuntimeConfig, + identity: &ResolvedWorkloadIdentity, + boundary_config: &[u8], + boundary_tls: DockerSandboxTls<'_>, + workspace_root: &str, +) -> Result<(), Status> { + let archive = docker_sandbox_bundle_archive( + config.sandbox_binary.as_slice(), + boundary_config, + boundary_tls, + identity, + workspace_root, + )?; + let options = UploadToContainerOptionsBuilder::default() + .path("/") + .copy_uidgid("true") + .build(); + docker + .upload_to_container( + container_id, + Some(options), + bollard::body_full(Bytes::from(archive)), + ) + .await + .map_err(|error| Status::internal(format!("stage Docker sandbox bundle: {error}"))) +} + +fn decode_docker_launch_authentication( + encoded: &[u8], +) -> Result { + let authentication = + serde_json::from_slice::(encoded) + .map_err(|error| { + Status::failed_precondition(format!( + "decode Docker sandbox launch authentication: {error}" + )) + })?; + authentication.validate().map_err(|error| { + Status::failed_precondition(format!( + "validate Docker sandbox launch authentication: {error}" + )) + })?; + Ok(authentication) +} + +fn gateway_verification_keys( + keys: &[openshell_core::jwt::SessionVerificationKey], +) -> Result, Status> { + keys.iter() + .map(|key| { + String::from_utf8(key.public_key_pem.clone()) + .map(|public_key_pem| GatewayVerificationKey { + key_id: key.key_id.clone(), + public_key_pem, + }) + .map_err(|error| { + Status::failed_precondition(format!( + "Docker sandbox verification key is not UTF-8 PEM: {error}" + )) + }) + }) + .collect() +} + +async fn prepare_docker_boundary_files( + docker: &Docker, + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + container_id: &str, + image: &DockerImageMetadata, + workload_identity: &ResolvedWorkloadIdentity, +) -> Result<(), Status> { + let directory = docker_boundary_state_dir(sandbox, config)?; + let workspace_root = driver_mounts::resolve_oci_workspace_root(&image.working_dir) + .map_err(Status::failed_precondition)?; + let launch_authentication = sandbox + .spec + .as_ref() + .filter(|spec| !spec.launch_authentication.is_empty()) + .ok_or_else(|| { + Status::failed_precondition("Docker sandbox launch authentication is required") + }) + .and_then(|spec| decode_docker_launch_authentication(&spec.launch_authentication))?; + let host_gateway_ip = docker_boundary_host_gateway_ip(&config.gateway_route); + let session_id = launch_authentication.supervisor.session_id; + let tls = generate_sandbox_tls_material(session_id) + .map_err(|error| Status::internal(format!("generate Docker boundary TLS: {error}")))?; + let verification_keys = gateway_verification_keys(&launch_authentication.verification_keys)?; + let provisioning = isolation::DockerBoundarySpec { + boundary_id: sandbox.id.clone(), + generation: launch_authentication + .supervisor + .runtime_generation + .to_string(), + session_id, + session_rotation: launch_authentication.supervisor.session_rotation, + auth_epoch: launch_authentication.supervisor.auth_epoch, + gateway_id: launch_authentication.gateway_id, + verification_keys, + container_id: container_id.to_string(), + image_identity: image.id.clone(), + listener_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH), + control_socket: PathBuf::from(BOUNDARY_SOCKET_MOUNT_PATH), + sandbox_tls: SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from(BOUNDARY_CERTIFICATE_MOUNT_PATH), + private_key_path: PathBuf::from(BOUNDARY_PRIVATE_KEY_MOUNT_PATH), + }, + supervisor_tls: SandboxTlsClientConfig { + server_name: tls.server_name.clone(), + trust_anchor_pem: tls.trust_anchor_pem.clone(), + }, + host_gateway_ip, + workload_identity: workload_identity.clone(), + child_env: docker_child_environment(sandbox), + } + .provision(); + let boundary_config = provisioning + .boundary_config + .encode() + .map_err(|error| Status::internal(error.to_string()))?; + write_docker_boundary_file(&directory.join(BOUNDARY_CONFIG_FILE), &boundary_config).await?; + write_docker_boundary_file( + &directory.join(BOUNDARY_CERTIFICATE_FILE), + tls.certificate_chain_pem.as_bytes(), + ) + .await?; + write_docker_boundary_file( + &directory.join(BOUNDARY_PRIVATE_KEY_FILE), + tls.private_key_pem.as_bytes(), + ) + .await?; + stage_docker_sandbox_bundle( + docker, + container_id, + config, + workload_identity, + &boundary_config, + DockerSandboxTls { + certificate: tls.certificate_chain_pem.as_bytes(), + private_key: tls.private_key_pem.as_bytes(), + }, + &workspace_root, + ) + .await?; + let descriptor = provisioning + .runtime_descriptor + .backend_descriptor() + .map_err(|error| Status::internal(error.to_string()))?; + write_docker_boundary_file( + &directory.join(RUNTIME_DESCRIPTOR_FILE), + &descriptor.payload, + ) + .await?; + let supervisor_auth = serde_json::to_vec(&launch_authentication.supervisor) + .map_err(|error| Status::internal(format!("encode Docker supervisor auth: {error}")))?; + write_docker_boundary_file( + &directory.join(SUPERVISOR_AUTH_BUNDLE_FILE), + &supervisor_auth, + ) + .await?; + let main_process_spec = openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec( + sandbox.spec.as_ref(), + ) + .map_err(|error| Status::internal(format!("encode Docker main process spec: {error}")))?; + write_docker_boundary_file( + &directory.join(MAIN_PROCESS_SPEC_FILE), + main_process_spec.as_bytes(), + ) + .await?; + write_docker_boundary_file( + &directory.join(WORKSPACE_ROOT_FILE), + workspace_root.as_bytes(), + ) + .await?; + Ok(()) +} + +async fn docker_supervisor_bundle_archive( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) -> Result, Status> { + let directory = docker_boundary_state_dir(sandbox, config)?; + let runtime_descriptor = tokio::fs::read(directory.join(RUNTIME_DESCRIPTOR_FILE)) + .await + .map_err(|error| Status::internal(format!("read Docker runtime descriptor: {error}")))?; + let auth_bundle = tokio::fs::read(directory.join(SUPERVISOR_AUTH_BUNDLE_FILE)) + .await + .map_err(|error| { + Status::failed_precondition(format!("read Docker supervisor auth bundle: {error}")) + })?; + if auth_bundle.is_empty() { + return Err(Status::failed_precondition( + "Docker supervisor requires launch authentication", + )); + } + let mut archive = tar::Builder::new(Vec::new()); + append_docker_archive_file( + &mut archive, + "runtime-descriptor.json", + 0o600, + SUPERVISOR_UID, + SUPERVISOR_GID, + &runtime_descriptor, + )?; + append_docker_archive_file( + &mut archive, + "auth.json", + 0o600, + SUPERVISOR_UID, + SUPERVISOR_GID, + &auth_bundle, + )?; + if let Some(tls) = &config.guest_tls { + append_docker_archive_directory( + &mut archive, + "tls", + 0o700, + SUPERVISOR_UID, + SUPERVISOR_GID, + )?; + for (name, path) in [ + ("ca.pem", &tls.ca), + ("cert.pem", &tls.cert), + ("key.pem", &tls.key), + ] { + let contents = tokio::fs::read(path).await.map_err(|error| { + Status::internal(format!( + "read Docker supervisor TLS file {}: {error}", + path.display() + )) + })?; + append_docker_archive_file( + &mut archive, + &format!("tls/{name}"), + 0o600, + SUPERVISOR_UID, + SUPERVISOR_GID, + &contents, + )?; + } + } + if let Some(path) = config.upstream_proxy.proxy_auth_file.as_ref() { + let contents = tokio::fs::read(path).await.map_err(|error| { + Status::internal(format!( + "read Docker upstream proxy credential {}: {error}", + path.display() + )) + })?; + append_docker_archive_file( + &mut archive, + "upstream-proxy-auth", + 0o600, + SUPERVISOR_UID, + SUPERVISOR_GID, + &contents, + )?; + } + archive + .into_inner() + .map_err(|error| Status::internal(format!("finish Docker supervisor archive: {error}"))) +} + +async fn refresh_docker_boundary_authentication( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, + encoded_authentication: &[u8], +) -> Result<(), Status> { + let authentication = decode_docker_launch_authentication(encoded_authentication)?; + let directory = docker_boundary_state_dir_by_id(sandbox_id, config)?; + let mut boundary_config = serde_json::from_slice::( + &tokio::fs::read(directory.join(BOUNDARY_CONFIG_FILE)) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read Docker sandbox bootstrap for authentication rotation: {error}" + )) + })?, + ) + .map_err(|error| { + Status::failed_precondition(format!( + "decode Docker sandbox bootstrap for authentication rotation: {error}" + )) + })?; + let Some(mut runtime_descriptor) = read_docker_runtime_descriptor(sandbox_id, config).await? + else { + return Err(Status::failed_precondition( + "Docker sandbox runtime descriptor is missing during authentication rotation", + )); + }; + let session_id = authentication.supervisor.session_id; + let tls = generate_sandbox_tls_material(session_id) + .map_err(|error| Status::internal(format!("rotate Docker boundary TLS: {error}")))?; + boundary_config.session_id = session_id; + boundary_config.generation = authentication.supervisor.runtime_generation.to_string(); + boundary_config.session_rotation = authentication.supervisor.session_rotation; + boundary_config.auth_epoch = authentication.supervisor.auth_epoch; + boundary_config.gateway_id = authentication.gateway_id; + boundary_config.verification_keys = + gateway_verification_keys(&authentication.verification_keys)?; + runtime_descriptor.session_id = session_id; + runtime_descriptor.tls = SandboxTlsClientConfig { + server_name: tls.server_name, + trust_anchor_pem: tls.trust_anchor_pem, + }; + let encoded_boundary_config = boundary_config + .encode() + .map_err(|error| Status::internal(error.to_string()))?; + let descriptor = runtime_descriptor + .backend_descriptor() + .map_err(|error| Status::internal(error.to_string()))?; + let supervisor_auth = serde_json::to_vec(&authentication.supervisor) + .map_err(|error| Status::internal(format!("encode Docker supervisor auth: {error}")))?; + write_docker_boundary_file( + &directory.join(BOUNDARY_CONFIG_FILE), + &encoded_boundary_config, + ) + .await?; + write_docker_boundary_file( + &directory.join(BOUNDARY_CERTIFICATE_FILE), + tls.certificate_chain_pem.as_bytes(), + ) + .await?; + write_docker_boundary_file( + &directory.join(BOUNDARY_PRIVATE_KEY_FILE), + tls.private_key_pem.as_bytes(), + ) + .await?; + write_docker_boundary_file( + &directory.join(RUNTIME_DESCRIPTOR_FILE), + &descriptor.payload, + ) + .await?; + write_docker_boundary_file( + &directory.join(SUPERVISOR_AUTH_BUNDLE_FILE), + &supervisor_auth, + ) + .await +} + +async fn refresh_docker_supervisor_authentication( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, + encoded_authentication: &[u8], +) -> Result<(), Status> { + let authentication = decode_docker_launch_authentication(encoded_authentication)?; + let directory = docker_boundary_state_dir_by_id(sandbox_id, config)?; + let Some(runtime_descriptor) = read_docker_runtime_descriptor(sandbox_id, config).await? else { + return Err(Status::failed_precondition( + "Docker sandbox runtime descriptor is missing during supervisor authentication rotation", + )); + }; + let descriptor = runtime_descriptor + .backend_descriptor() + .map_err(|error| Status::internal(error.to_string()))?; + let supervisor_auth = serde_json::to_vec(&authentication.supervisor) + .map_err(|error| Status::internal(format!("encode Docker supervisor auth: {error}")))?; + write_docker_boundary_file( + &directory.join(RUNTIME_DESCRIPTOR_FILE), + &descriptor.payload, + ) + .await?; + write_docker_boundary_file( + &directory.join(SUPERVISOR_AUTH_BUNDLE_FILE), + &supervisor_auth, + ) + .await +} + +async fn read_docker_runtime_descriptor( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result, Status> { + let path = docker_boundary_state_dir_by_id(sandbox_id, config)?.join(RUNTIME_DESCRIPTOR_FILE); + let bytes = match tokio::fs::read(&path).await { + Ok(bytes) => bytes, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => { + return Err(Status::internal(format!( + "read Docker runtime descriptor {}: {error}", + path.display() + ))); + } + }; + serde_json::from_slice(&bytes).map(Some).map_err(|error| { + Status::internal(format!( + "decode Docker runtime descriptor {}: {error}", + path.display() + )) + }) +} + +async fn stage_docker_supervisor_bundle( + docker: &Docker, + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + archive: Vec, +) -> Result<(), Status> { + let stager_name = format!("{}-supervisor-stage", container_name_for_sandbox(sandbox)); + let _ = docker + .remove_container( + &stager_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + let created = docker + .create_container( + Some( + CreateContainerOptionsBuilder::default() + .name(stager_name.as_str()) + .build(), + ), + ContainerCreateBody { + image: Some(config.supervisor_image_id.clone()), + entrypoint: Some(vec![SUPERVISOR_IMAGE_CONTROL_BINARY_PATH.to_string()]), + labels: Some(docker_auxiliary_container_labels( + sandbox, + config, + LABEL_ISOLATION_ROLE_STAGING, + )), + host_config: Some(HostConfig { + network_mode: Some("none".to_string()), + mounts: Some(vec![Mount { + target: Some(SUPERVISOR_STATE_MOUNT_PATH.to_string()), + source: Some(docker_supervisor_volume_name(sandbox, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(false), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }]), + cap_drop: Some(vec!["ALL".to_string()]), + security_opt: Some(vec!["no-new-privileges:true".to_string()]), + ..Default::default() + }), + ..Default::default() + }, + ) + .await + .map_err(|error| { + Status::internal(format!( + "create Docker supervisor staging container: {error}" + )) + })?; + let options = UploadToContainerOptionsBuilder::default() + .path(SUPERVISOR_STATE_MOUNT_PATH) + .copy_uidgid("true") + .build(); + let result = docker + .upload_to_container( + &created.id, + Some(options), + bollard::body_full(Bytes::from(archive)), + ) + .await + .map_err(|error| Status::internal(format!("stage Docker supervisor bundle: {error}"))); + let cleanup = docker + .remove_container( + &created.id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await + .map_err(|error| { + Status::internal(format!( + "remove Docker supervisor staging container: {error}" + )) + }); + result?; + cleanup +} + +fn docker_auxiliary_container_labels( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + role: &str, +) -> HashMap { + HashMap::from([ + ( + LABEL_MANAGED_BY.to_string(), + LABEL_MANAGED_BY_VALUE.to_string(), + ), + (LABEL_SANDBOX_ID.to_string(), sandbox.id.clone()), + (LABEL_SANDBOX_NAME.to_string(), sandbox.name.clone()), + ( + LABEL_SANDBOX_NAMESPACE.to_string(), + config.sandbox_namespace.clone(), + ), + (LABEL_ISOLATION_ROLE.to_string(), role.to_string()), + ]) +} + +async fn spawn_docker_control_process( + docker: &Docker, + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + failure_context: DockerRuntimeFailureContext, +) -> Result { + let directory = docker_boundary_state_dir(sandbox, config)?; + let main_process_spec = tokio::fs::read_to_string(directory.join(MAIN_PROCESS_SPEC_FILE)) + .await + .map_err(|error| Status::internal(format!("read Docker main process spec: {error}")))?; + let workspace_root = tokio::fs::read_to_string(directory.join(WORKSPACE_ROOT_FILE)) + .await + .map_err(|error| Status::internal(format!("read Docker workspace root: {error}")))?; + let supervisor_name = format!("{}-supervisor", container_name_for_sandbox(sandbox)); + let _ = docker + .remove_container( + &supervisor_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + let runtime_descriptor_path = format!("{SUPERVISOR_STATE_MOUNT_PATH}/runtime-descriptor.json"); + let auth_bundle_path = format!("{SUPERVISOR_STATE_MOUNT_PATH}/auth.json"); + let mut environment = vec![ + format!( + "{}={DRIVER_ADMITTED_BACKEND}", + openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND + ), + format!( + "{}={main_process_spec}", + openshell_core::sandbox_env::MAIN_PROCESS_SPEC + ), + format!( + "{}={}", + openshell_core::sandbox_env::ENDPOINT, + config.supervisor_grpc_endpoint + ), + format!("{}={}", openshell_core::sandbox_env::SANDBOX_ID, sandbox.id), + format!("{}={}", openshell_core::sandbox_env::SANDBOX, sandbox.name), + format!( + "{}={}", + openshell_core::sandbox_env::SSH_SOCKET_PATH, + config.ssh_socket_path + ), + format!( + "{}=/run/openshell/proxy-tls", + openshell_core::sandbox_env::PROXY_TLS_DIR + ), + format!( + "{}={}", + openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, + openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY + ), + format!( + "{}={}", + openshell_core::sandbox_env::LOG_LEVEL, + openshell_core::driver_utils::sandbox_log_level(sandbox, &config.log_level) + ), + format!( + "{}={}", + openshell_core::sandbox_env::TELEMETRY_ENABLED, + openshell_core::telemetry::enabled_env_value() + ), + ]; + if let Some(server_name) = config.gateway_tls_server_name.as_deref() { + environment.push(format!( + "{}={server_name}", + openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME + )); + } + if config.guest_tls.is_some() { + environment.extend([ + format!( + "{}={SUPERVISOR_STATE_MOUNT_PATH}/tls/ca.pem", + openshell_core::sandbox_env::TLS_CA + ), + format!( + "{}={SUPERVISOR_STATE_MOUNT_PATH}/tls/cert.pem", + openshell_core::sandbox_env::TLS_CERT + ), + format!( + "{}={SUPERVISOR_STATE_MOUNT_PATH}/tls/key.pem", + openshell_core::sandbox_env::TLS_KEY + ), + ]); + } + if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() { + let projected = openshell_core::driver_utils::projected_provider_spiffe_socket_path(socket) + .map_err(Status::failed_precondition)?; + environment.push(format!( + "{}={projected}", + openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET + )); + } + let supervisor_archive = docker_supervisor_bundle_archive(sandbox, config).await?; + stage_docker_supervisor_bundle(docker, sandbox, config, supervisor_archive).await?; + let labels = HashMap::from([ + (LABEL_MANAGED_BY.to_string(), "openshell".to_string()), + (LABEL_SANDBOX_ID.to_string(), sandbox.id.clone()), + (LABEL_SANDBOX_NAME.to_string(), sandbox.name.clone()), + ( + LABEL_SANDBOX_NAMESPACE.to_string(), + config.sandbox_namespace.clone(), + ), + ( + LABEL_ISOLATION_ROLE.to_string(), + LABEL_ISOLATION_ROLE_SUPERVISOR.to_string(), + ), + ]); + let mut command = vec![ + "--backend-descriptor-file".to_string(), + runtime_descriptor_path, + "--auth-bundle-file".to_string(), + auth_bundle_path, + "--workdir".to_string(), + workspace_root, + format!("--health-socket-path={SUPERVISOR_HEALTH_SOCKET_PATH}"), + ]; + command.extend(docker_upstream_proxy_cli_args(&config.upstream_proxy)); + let mut supervisor_mounts = vec![ + Mount { + target: Some(BOUNDARY_MOUNT_PATH.to_string()), + source: Some(docker_channel_volume_name(sandbox, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(true), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }, + Mount { + target: Some(SUPERVISOR_STATE_MOUNT_PATH.to_string()), + source: Some(docker_supervisor_volume_name(sandbox, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(true), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }, + ]; + if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() { + let parent = socket.parent().ok_or_else(|| { + Status::failed_precondition("provider SPIFFE socket has no parent directory") + })?; + supervisor_mounts.push(Mount { + target: Some(PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR.to_string()), + source: Some(parent.display().to_string()), + typ: Some(MountTypeEnum::BIND), + read_only: Some(true), + ..Default::default() + }); + } + let create = ContainerCreateBody { + image: Some(config.supervisor_image_id.clone()), + user: Some(format!("{SUPERVISOR_UID}:{SUPERVISOR_GID}")), + entrypoint: Some(vec![SUPERVISOR_IMAGE_CONTROL_BINARY_PATH.to_string()]), + cmd: Some(command), + env: Some(environment), + labels: Some(labels), + healthcheck: Some(HealthConfig { + test: Some(vec![ + "CMD".to_string(), + SUPERVISOR_IMAGE_CONTROL_BINARY_PATH.to_string(), + "health".to_string(), + "--socket".to_string(), + SUPERVISOR_HEALTH_SOCKET_PATH.to_string(), + ]), + interval: Some(SUPERVISOR_HEALTH_INTERVAL_NS), + timeout: Some(SUPERVISOR_HEALTH_TIMEOUT_NS), + retries: Some(3), + start_period: Some(SUPERVISOR_HEALTH_START_PERIOD_NS), + start_interval: Some(SUPERVISOR_HEALTH_INTERVAL_NS), + }), + host_config: Some(HostConfig { + // The supervisor is trusted infrastructure and originates every + // approved upstream connection. The driver-owned bridge provides + // Docker DNS and service discovery while the workload remains + // fenced by network=none. + network_mode: Some(config.network_name.clone()), + mounts: Some(supervisor_mounts), + cap_drop: Some(vec!["ALL".to_string()]), + cap_add: None, + security_opt: Some(vec!["no-new-privileges:true".to_string()]), + readonly_rootfs: Some(true), + tmpfs: Some(HashMap::from([ + ( + "/run".to_string(), + format!( + "rw,noexec,nosuid,size=64m,uid={SUPERVISOR_UID},gid={SUPERVISOR_GID},mode=0700" + ), + ), + ( + "/tmp".to_string(), + "rw,noexec,nosuid,size=64m,mode=1777".to_string(), + ), + ( + "/var/log".to_string(), + format!( + "rw,noexec,nosuid,size=64m,uid={SUPERVISOR_UID},gid={SUPERVISOR_GID},mode=0700" + ), + ), + ])), + extra_hosts: Some(vec![ + format!( + "{HOST_OPENSHELL_INTERNAL}:{}", + docker_supervisor_host_alias(&config.gateway_route) + ), + format!( + "{HOST_DOCKER_INTERNAL}:{}", + docker_supervisor_host_alias(&config.gateway_route) + ), + ]), + restart_policy: None, + ..Default::default() + }), + ..Default::default() + }; + let created = docker + .create_container( + Some( + CreateContainerOptionsBuilder::default() + .name(supervisor_name.as_str()) + .build(), + ), + create, + ) + .await + .map_err(|error| { + Status::internal(format!("create Docker supervisor container: {error}")) + })?; + if let Err(error) = docker.start_container(&created.id, None).await { + let _ = docker + .remove_container( + &created.id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + return Err(Status::internal(format!( + "start Docker supervisor container: {error}" + ))); + } + let sandbox_id = sandbox.id.clone(); + let (shutdown, mut shutdown_requested) = oneshot::channel(); + let intentional_shutdown = Arc::new(AtomicBool::new(false)); + let monitored_shutdown = intentional_shutdown.clone(); + let supervisor_id = created.id; + let monitored_supervisor_id = supervisor_id.clone(); + let monitored_docker = failure_context.docker.clone(); + let task = tokio::spawn(async move { + let wait = async { + let mut stream = monitored_docker.wait_container( + &monitored_supervisor_id, + None::, + ); + stream.next().await + }; + tokio::select! { + biased; + _ = &mut shutdown_requested => { + let _ = monitored_docker.stop_container( + &monitored_supervisor_id, + Some(StopContainerOptionsBuilder::default().t(5).build()), + ).await; + let _ = monitored_docker.remove_container( + &monitored_supervisor_id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ).await; + } + result = wait => { + if monitored_shutdown.load(Ordering::Acquire) { + let _ = monitored_docker.remove_container( + &monitored_supervisor_id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ).await; + return; + } + if matches!( + tokio::time::timeout( + SUPERVISOR_INTENTIONAL_SHUTDOWN_GRACE, + &mut shutdown_requested, + ) + .await, + Ok(Ok(())), + ) || monitored_shutdown.load(Ordering::Acquire) + { + let _ = monitored_docker.remove_container( + &monitored_supervisor_id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ).await; + return; + } + let mut message = match result { + Some(Ok(status)) => { + warn!(%sandbox_id, status = status.status_code, "Docker supervisor container exited unexpectedly"); + format!("Docker supervisor container exited with status {}", status.status_code) + } + Some(Err(error)) => { + warn!(%sandbox_id, %error, "Failed to wait for Docker supervisor container"); + format!("failed to wait for Docker supervisor container: {error}") + } + None => "Docker supervisor wait stream ended unexpectedly".to_string(), + }; + let log_tail = + docker_container_log_tail(&monitored_docker, &monitored_supervisor_id).await; + if !log_tail.is_empty() { + write!(message, "; log tail: {log_tail}").ok(); + } + let sandbox_log_tail = + docker_container_log_tail(&monitored_docker, &failure_context.container_id) + .await; + if !sandbox_log_tail.is_empty() { + write!(message, "; sandbox log tail: {sandbox_log_tail}").ok(); + } + let _ = monitored_docker.remove_container( + &monitored_supervisor_id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ).await; + // The gateway can close the supervisor session as soon as it + // commits Stopping. Re-check after collecting diagnostics so + // an overlapping driver stop cannot be published as an + // unexpected control failure. + if monitored_shutdown.load(Ordering::Acquire) { + return; + } + handle_docker_runtime_failure( + failure_context, + "ControlSupervisorExited", + message, + ) + .await; + }, + } + }); + let process = DockerControlProcess { + shutdown: Some(shutdown), + intentional_shutdown, + task, + }; + if let Err(error) = wait_for_docker_supervisor_ready(docker, &supervisor_id).await { + stop_docker_control_process(process).await; + return Err(error); + } + Ok(process) +} + +async fn wait_for_docker_supervisor_ready( + docker: &Docker, + supervisor_id: &str, +) -> Result<(), Status> { + let wait = async { + loop { + let inspected = docker + .inspect_container(supervisor_id, None) + .await + .map_err(|error| { + Status::internal(format!("inspect Docker supervisor container: {error}")) + })?; + let state = inspected.state.unwrap_or_default(); + match state.health.and_then(|health| health.status) { + Some(HealthStatusEnum::HEALTHY) => return Ok(()), + Some(HealthStatusEnum::UNHEALTHY) => { + let log_tail = docker_container_log_tail(docker, supervisor_id).await; + return Err(Status::unavailable(format!( + "Docker supervisor failed its readiness check{}", + format_log_tail(&log_tail) + ))); + } + _ if state.running == Some(false) => { + let log_tail = docker_container_log_tail(docker, supervisor_id).await; + return Err(Status::unavailable(format!( + "Docker supervisor exited before becoming ready{}", + format_log_tail(&log_tail) + ))); + } + _ => tokio::time::sleep(Duration::from_millis(100)).await, + } + } + }; + + if let Ok(result) = tokio::time::timeout(SUPERVISOR_READY_TIMEOUT, wait).await { + result + } else { + let log_tail = docker_container_log_tail(docker, supervisor_id).await; + Err(Status::deadline_exceeded(format!( + "Docker supervisor did not become ready within {} seconds{}", + SUPERVISOR_READY_TIMEOUT.as_secs(), + format_log_tail(&log_tail) + ))) + } +} + +fn format_log_tail(log_tail: &str) -> String { + if log_tail.is_empty() { + String::new() + } else { + format!("; log tail: {log_tail}") + } +} + +async fn docker_container_log_tail(docker: &Docker, container_id: &str) -> String { + const MAX_LOG_TAIL_BYTES: usize = 16 * 1024; + let options = LogsOptionsBuilder::default() + .stdout(true) + .stderr(true) + .tail("80") + .build(); + let mut stream = docker.logs(container_id, Some(options)); + let mut output = Vec::new(); + while let Some(result) = stream.next().await { + let Ok(chunk) = result else { + break; + }; + output.extend_from_slice(chunk.as_ref()); + if output.len() > MAX_LOG_TAIL_BYTES { + output.drain(..output.len() - MAX_LOG_TAIL_BYTES); + } + } + String::from_utf8_lossy(&output).trim().to_string() +} + +async fn handle_docker_runtime_failure( + context: DockerRuntimeFailureContext, + reason: &'static str, + message: String, +) { + context.failures.lock().await.insert( + context.sandbox.id.clone(), + DockerRuntimeFailure { + reason, + message: message.clone(), + }, + ); + + let mut snapshot = pending_sandbox_snapshot( + &context.sandbox, + &context.sandbox_namespace, + error_condition(reason, &message), + false, + ); + if let Some(status) = snapshot.status.as_mut() { + status.instance_id.clone_from(&context.container_id); + } + let _ = context.events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(snapshot), + }, + )), + }); + let _ = context.events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::PlatformEvent( + WatchSandboxesPlatformEvent { + sandbox_id: context.sandbox.id.clone(), + event: Some(platform_event( + "docker", + "Warning", + reason, + format!("{message}; stopping the isolated workload container"), + )), + }, + )), + }); + + match context + .docker + .stop_container( + &context.container_id, + Some( + StopContainerOptionsBuilder::default() + .t(docker_stop_timeout_secs(context.stop_timeout_secs)) + .build(), + ), + ) + .await + { + Ok(()) => info!( + sandbox_id = %context.sandbox.id, + container_id = %context.container_id, + "Stopped Docker sandbox after control supervisor failure" + ), + Err(error) if is_not_found_error(&error) || is_not_modified_error(&error) => {} + Err(error) => warn!( + sandbox_id = %context.sandbox.id, + container_id = %context.container_id, + %error, + "Failed to stop Docker sandbox after control supervisor failure" + ), + } } -fn cleanup_sandbox_token_file(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) { - cleanup_sandbox_token_file_by_id(&sandbox.id, config); +async fn stop_docker_control_process(mut process: DockerControlProcess) { + process.intentional_shutdown.store(true, Ordering::Release); + if let Some(shutdown) = process.shutdown.take() { + let _ = shutdown.send(()); + } + let _ = process.task.await; } -fn cleanup_sandbox_token_file_for_delete( - sandbox_id: &str, - pending: Option<&PendingSandboxRecord>, - config: &DockerDriverRuntimeConfig, -) { - if !sandbox_id.is_empty() { - cleanup_sandbox_token_file_by_id(sandbox_id, config); - } else if let Some(record) = pending { - cleanup_sandbox_token_file(&record.sandbox, config); - } +fn cleanup_docker_boundary_state(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) { + cleanup_docker_boundary_state_by_id(&sandbox.id, config); } -fn cleanup_sandbox_token_file_by_id(sandbox_id: &str, config: &DockerDriverRuntimeConfig) { - let Ok(path) = sandbox_token_host_path_by_id(sandbox_id, config) else { +fn cleanup_docker_boundary_state_by_id(sandbox_id: &str, config: &DockerDriverRuntimeConfig) { + let Ok(directory) = docker_boundary_state_dir_by_id(sandbox_id, config) else { return; }; - if let Err(err) = std::fs::remove_file(&path) - && err.kind() != std::io::ErrorKind::NotFound + if let Err(error) = std::fs::remove_dir_all(&directory) + && error.kind() != std::io::ErrorKind::NotFound { warn!( - sandbox_id = %sandbox_id, - path = %path.display(), - error = %err, - "Failed to remove Docker sandbox token file" + %sandbox_id, + path = %directory.display(), + %error, + "Failed to remove Docker boundary state directory" ); } - if let Some(dir) = path.parent() { - let _ = std::fs::remove_dir(dir); - } } -#[cfg(test)] -fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) -> Vec { - build_environment_for_oci_user(sandbox, config, "") +fn docker_child_environment(sandbox: &DriverSandbox) -> HashMap { + let mut environment = sandbox + .spec + .as_ref() + .and_then(|spec| spec.template.as_ref()) + .map_or_else(HashMap::new, |template| template.environment.clone()); + if let Some(spec) = sandbox.spec.as_ref() { + environment.extend(spec.environment.clone()); + } + for protected in [ + openshell_core::sandbox_env::ENDPOINT, + openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME, + openshell_core::sandbox_env::MAIN_PROCESS_SPEC, + openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, + openshell_core::sandbox_env::OCI_IMAGE_USER, + openshell_core::sandbox_env::SANDBOX, + openshell_core::sandbox_env::SANDBOX_GID, + openshell_core::sandbox_env::SANDBOX_ID, + openshell_core::sandbox_env::SANDBOX_TOKEN, + openshell_core::sandbox_env::SANDBOX_TOKEN_FILE, + openshell_core::sandbox_env::SANDBOX_UID, + openshell_core::sandbox_env::SSH_SOCKET_PATH, + openshell_core::sandbox_env::TLS_CA, + openshell_core::sandbox_env::TLS_CERT, + openshell_core::sandbox_env::TLS_KEY, + openshell_core::sandbox_env::USER_ENVIRONMENT, + ] { + environment.remove(protected); + } + environment } -fn build_environment_for_oci_user( +fn build_boundary_environment( sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, - oci_user: &str, ) -> Vec { - let mut environment = HashMap::from([ - ("HOME".to_string(), "/root".to_string()), - ("PATH".to_string(), SUPERVISOR_PATH.to_string()), - ("TERM".to_string(), "xterm".to_string()), - ( - "OPENSHELL_LOG_LEVEL".to_string(), - openshell_core::driver_utils::sandbox_log_level(sandbox, &config.log_level), + vec![ + format!( + "{}={}", + openshell_core::sandbox_env::LOG_LEVEL, + openshell_core::driver_utils::sandbox_log_level(sandbox, &config.log_level) ), - ]); - - if let Some(spec) = sandbox.spec.as_ref() { - let mut user_env = HashMap::new(); - if let Some(template) = spec.template.as_ref() { - user_env.extend(template.environment.clone()); - } - user_env.extend(spec.environment.clone()); - environment.extend(user_env.clone()); - if !user_env.is_empty() - && let Ok(json) = serde_json::to_string(&user_env) - { - environment.insert( - openshell_core::sandbox_env::USER_ENVIRONMENT.to_string(), - json, - ); - } - } - - environment.insert( - openshell_core::sandbox_env::ENDPOINT.to_string(), - config.grpc_endpoint.clone(), - ); - environment.insert( - openshell_core::sandbox_env::SANDBOX_ID.to_string(), - sandbox.id.clone(), - ); - environment.insert( - openshell_core::sandbox_env::SANDBOX.to_string(), - sandbox.name.clone(), - ); - environment.insert( - openshell_core::sandbox_env::SSH_SOCKET_PATH.to_string(), - config.ssh_socket_path.clone(), - ); - let main_process = - openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec(sandbox.spec.as_ref()) - .expect("main process config serialization cannot fail"); - environment.insert( - openshell_core::sandbox_env::MAIN_PROCESS_SPEC.to_string(), - main_process, - ); - environment.insert( - openshell_core::sandbox_env::TELEMETRY_ENABLED.to_string(), - openshell_core::telemetry::enabled_env_value().to_string(), - ); - environment.insert( - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES.to_string(), - openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY.to_string(), - ); - // The root supervisor executes namespace helpers during bootstrap; keep - // their search path driver-owned even when the template/spec set PATH. - environment.insert("PATH".to_string(), SUPERVISOR_PATH.to_string()); - if config.guest_tls.is_some() { - environment.insert( - openshell_core::sandbox_env::TLS_CA.to_string(), - TLS_CA_MOUNT_PATH.to_string(), - ); - environment.insert( - openshell_core::sandbox_env::TLS_CERT.to_string(), - TLS_CERT_MOUNT_PATH.to_string(), - ); - environment.insert( - openshell_core::sandbox_env::TLS_KEY.to_string(), - TLS_KEY_MOUNT_PATH.to_string(), - ); - } - if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() - && let Ok(path) = - openshell_core::driver_utils::projected_provider_spiffe_socket_path(socket) - { - environment.insert( - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET.to_string(), - path, - ); - } - - environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN); - environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN_FILE); - // Prevent user-supplied environment from overriding the TLS server name - // the supervisor verifies — a sandbox user who can redirect the gateway - // hostname could otherwise present a certificate for a name they control - // and intercept the sandbox JWT. - environment.remove(openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME); - environment.insert( - openshell_core::sandbox_env::OCI_IMAGE_USER.to_string(), - oci_user.to_string(), - ); - environment.insert( - openshell_core::sandbox_env::SANDBOX_UID.to_string(), - String::new(), - ); - environment.insert( - openshell_core::sandbox_env::SANDBOX_GID.to_string(), - String::new(), - ); - - // Gateway-minted sandbox JWT. Keep the raw bearer out of container - // metadata; the supervisor reads it from this driver-owned bind mount. - if let Some(spec) = sandbox.spec.as_ref() - && !spec.sandbox_token.is_empty() - { - environment.insert( - openshell_core::sandbox_env::SANDBOX_TOKEN_FILE.to_string(), - SANDBOX_TOKEN_MOUNT_PATH.to_string(), - ); - } - - let mut pairs = environment.into_iter().collect::>(); - pairs.sort_by(|left, right| left.0.cmp(&right.0)); - pairs - .into_iter() - .map(|(key, value)| format!("{key}={value}")) - .collect() + format!( + "{}={}", + openshell_core::sandbox_env::TELEMETRY_ENABLED, + openshell_core::telemetry::enabled_env_value() + ), + ] } fn docker_cdi_gpu_inventory(info: &SystemInfo) -> CdiGpuInventory { @@ -3128,6 +5435,14 @@ fn build_container_create_body_with_gpu_devices( .as_ref() .and_then(|spec| spec.template.as_ref()) .ok_or_else(|| Status::invalid_argument("sandbox.spec.template is required"))?; + let workload_identity = ResolvedWorkloadIdentity::new( + 1000, + 1000, + Vec::new(), + "test".to_string(), + template.image.clone(), + ) + .map_err(|error| Status::internal(error.to_string()))?; build_container_create_body_for_image( sandbox, config, @@ -3139,6 +5454,7 @@ fn build_container_create_body_with_gpu_devices( working_dir: String::new(), volumes: Vec::new(), }, + &workload_identity, ) } @@ -3148,6 +5464,7 @@ fn build_container_create_body_for_image( driver_config: &DockerSandboxDriverConfig, gpu_device_ids: Option<&[String]>, image: &DockerImageMetadata, + workload_identity: &ResolvedWorkloadIdentity, ) -> Result { let spec = sandbox .spec @@ -3160,7 +5477,7 @@ fn build_container_create_body_for_image( let resource_limits = docker_resource_limits(template)?; let workspace_root = driver_mounts::resolve_oci_workspace_root(&image.working_dir) .map_err(Status::failed_precondition)?; - driver_mounts::validate_workspace_control_path(&workspace_root, &config.ssh_socket_path) + driver_mounts::validate_workspace_control_path(&workspace_root, BOUNDARY_MOUNT_PATH) .map_err(Status::failed_precondition)?; for volume in &image.volumes { driver_mounts::validate_container_mount_target(volume).map_err(|error| { @@ -3173,7 +5490,7 @@ fn build_container_create_body_for_image( "image-declared volume '{volume}' masks OCI WorkingDir '{workspace_root}' before workspace validation" )) })?; - driver_mounts::validate_mount_control_path(volume, &config.ssh_socket_path) + driver_mounts::validate_mount_control_path(volume, BOUNDARY_MOUNT_PATH) .map_err(Status::failed_precondition)?; } for mount in &driver_config.mounts { @@ -3185,10 +5502,21 @@ fn build_container_create_body_for_image( }; driver_mounts::validate_workspace_mount_target(target, &workspace_root) .map_err(Status::failed_precondition)?; - driver_mounts::validate_mount_control_path(target, &config.ssh_socket_path) + driver_mounts::validate_mount_control_path(target, BOUNDARY_MOUNT_PATH) .map_err(Status::failed_precondition)?; } - let user_mounts = docker_driver_mounts(driver_config)?; + let mut user_mounts = docker_driver_mounts(driver_config)?; + user_mounts.push(Mount { + target: Some(BOUNDARY_MOUNT_PATH.to_string()), + source: Some(docker_channel_volume_name(sandbox, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(false), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }); let user_bind_strings = docker_driver_bind_strings(driver_config)?; let device_requests = gpu_device_ids.map(|device_ids| { vec![DeviceRequest { @@ -3208,30 +5536,39 @@ fn build_container_create_body_for_image( LABEL_SANDBOX_WORKSPACE.to_string(), sandbox.workspace.clone(), ); - // The list/get/find paths filter by `config.sandbox_label`, so use + // The list/get/find paths filter by `config.sandbox_namespace`, so use // the same value here. `DriverSandbox.namespace` is unset on the request // path (the gateway elides it), and using it would produce containers // that the driver itself cannot find afterwards. labels.insert( LABEL_SANDBOX_NAMESPACE.to_string(), - config.sandbox_label.clone(), + config.sandbox_namespace.clone(), + ); + labels.insert( + LABEL_ISOLATION_BACKEND.to_string(), + LABEL_ISOLATION_BACKEND_OPEN_SHELL.to_string(), + ); + labels.insert( + LABEL_ISOLATION_ROLE.to_string(), + LABEL_ISOLATION_ROLE_SANDBOX.to_string(), ); Ok(ContainerCreateBody { image: Some(image.id.clone()), - user: Some("0".to_string()), + user: Some(format!( + "{}:{}", + workload_identity.uid, workload_identity.gid + )), // The image workspace may need to be created or rejected by the // supervisor, so do not let the OCI runtime chdir there first. working_dir: Some("/".to_string()), - env: Some(build_environment_for_oci_user(sandbox, config, &image.user)), - entrypoint: Some(vec![SUPERVISOR_MOUNT_PATH.to_string()]), - // Replace the image CMD with the supervisor's resolved workspace - // argument so Docker cannot append inherited image arguments. - cmd: { - let mut args = vec!["--workdir".to_string(), workspace_root]; - args.extend(docker_upstream_proxy_cli_args(&config.upstream_proxy)); - Some(args) - }, + env: Some(build_boundary_environment(sandbox, config)), + entrypoint: Some(vec![SANDBOX_BINARY_PATH.to_string()]), + // The image cannot append inherited arguments or select either role. + cmd: Some(vec![ + "--bootstrap".to_string(), + BOUNDARY_CONFIG_MOUNT_PATH.to_string(), + ]), labels: Some(labels), host_config: Some(HostConfig { nano_cpus: resource_limits.nano_cpus, @@ -3239,7 +5576,7 @@ fn build_container_create_body_for_image( pids_limit: docker_pids_limit(config.sandbox_pids_limit)?, device_requests, binds: { - let mut binds = build_binds(sandbox, config)?; + let mut binds = build_binds(sandbox, config); binds.extend(user_bind_strings); Some(binds) }, @@ -3247,29 +5584,43 @@ fn build_container_create_body_for_image( // Canonical main-process exit is terminal. Runtime restart would // silently create a new process generation behind the gateway. restart_policy: None, - cap_add: Some(vec![ - "SYS_ADMIN".to_string(), - "NET_ADMIN".to_string(), - "SYS_PTRACE".to_string(), - "SYSLOG".to_string(), - ]), - // The default is explicitly Unconfined because the supervisor - // needs mount operations commonly denied by docker-default. - security_opt: config - .app_armor_profile - .as_ref() - .and_then(AppArmorProfile::oci_security_opt) - .map(|option| vec![option]), - network_mode: Some(config.network_name.clone()), - extra_hosts: Some(docker_extra_hosts(&config.gateway_route)), - ..Default::default() - }), - networking_config: Some(NetworkingConfig { - endpoints_config: Some(HashMap::from([( - config.network_name.clone(), - EndpointSettings::default(), + group_add: Some( + workload_identity + .supplementary_gids + .iter() + .map(u32::to_string) + .collect(), + ), + cap_drop: Some(vec!["ALL".to_string()]), + cap_add: None, + security_opt: Some({ + let mut options = vec!["no-new-privileges:true".to_string()]; + if let Some(option) = config + .app_armor_profile + .as_ref() + .and_then(AppArmorProfile::oci_security_opt) + { + options.push(option); + } + options + }), + network_mode: Some("none".to_string()), + dns: Some(vec!["127.0.0.53".to_string()]), + tmpfs: Some(HashMap::from([( + openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_DIR.to_string(), + format!( + "rw,noexec,nosuid,nodev,size=1m,uid={},gid={},mode=0755", + workload_identity.uid, workload_identity.gid + ), + )])), + sysctls: Some(HashMap::from([( + "net.ipv4.ip_unprivileged_port_start".to_string(), + "0".to_string(), )])), + extra_hosts: None, + ..Default::default() }), + networking_config: None, ..Default::default() }) } @@ -3288,16 +5639,45 @@ fn require_sandbox_identifier(sandbox_id: &str, sandbox_name: &str) -> Result<() Ok(()) } -fn docker_container_openshell_endpoint(endpoint: &str, host: &str, port: u16) -> String { - let Ok(mut url) = Url::parse(endpoint) else { - return endpoint.to_string(); +fn docker_host_openshell_endpoint( + endpoint: &str, + route: &DockerGatewayRoute, +) -> CoreResult { + let mut url = Url::parse(endpoint) + .map_err(|error| Error::config(format!("invalid docker grpc_endpoint: {error}")))?; + if !matches!( + url.host_str(), + Some(HOST_OPENSHELL_INTERNAL | HOST_DOCKER_INTERNAL) + ) { + return Ok(url.to_string()); + } + let host = match route { + DockerGatewayRoute::Bridge { bind_address, .. } => bind_address.ip(), + DockerGatewayRoute::HostGateway => IpAddr::V4(Ipv4Addr::LOCALHOST), }; + url.set_host(Some(&host.to_string())).map_err(|error| { + Error::config(format!( + "failed to map Docker gateway alias to its host listener: {error}" + )) + })?; + Ok(url.to_string()) +} - if url.set_host(Some(host)).is_ok() && url.set_port(Some(port)).is_ok() { - return url.to_string(); +fn docker_supervisor_host_alias(route: &DockerGatewayRoute) -> String { + match route { + DockerGatewayRoute::Bridge { bind_address } => bind_address.ip().to_string(), + DockerGatewayRoute::HostGateway => "host-gateway".to_string(), } +} - endpoint.to_string() +fn docker_boundary_host_gateway_ip(route: &DockerGatewayRoute) -> Option { + match route { + DockerGatewayRoute::Bridge { bind_address } => Some(bind_address.ip()), + // Docker resolves this special alias inside the supervisor container. + // Pinning it to loopback would target the daemon VM rather than the + // desktop host on Docker Desktop and compatible runtimes. + DockerGatewayRoute::HostGateway => None, + } } fn docker_network_name(config: &DockerComputeConfig) -> String { @@ -3345,7 +5725,6 @@ fn docker_gateway_route_for_host( if let Some(host_alias_ip) = host_gateway_ip { return DockerGatewayRoute::Bridge { bind_address: SocketAddr::new(host_alias_ip, port), - host_alias_ip, }; } @@ -3354,7 +5733,6 @@ fn docker_gateway_route_for_host( } else { DockerGatewayRoute::Bridge { bind_address: SocketAddr::new(bridge_gateway_ip, port), - host_alias_ip: bridge_gateway_ip, } } } @@ -3419,19 +5797,6 @@ fn uses_host_gateway_alias(info: &SystemInfo) -> bool { }) } -fn docker_extra_hosts(route: &DockerGatewayRoute) -> Vec { - match route { - DockerGatewayRoute::Bridge { host_alias_ip, .. } => vec![ - format!("{HOST_DOCKER_INTERNAL}:{host_alias_ip}"), - format!("{HOST_OPENSHELL_INTERNAL}:{host_alias_ip}"), - ], - DockerGatewayRoute::HostGateway => vec![ - format!("{HOST_DOCKER_INTERNAL}:host-gateway"), - format!("{HOST_OPENSHELL_INTERNAL}:host-gateway"), - ], - } -} - async fn ensure_bridge_network(docker: &Docker, network_name: &str) -> CoreResult { match docker.inspect_network(network_name, None).await { Ok(network) => return validate_bridge_network(network_name, &network), @@ -3544,8 +5909,45 @@ fn docker_resource_limits( }) } +fn validate_docker_proxy_auth_file(config: &UpstreamProxyConfig) -> CoreResult<()> { + let Some(path) = config.proxy_auth_file.as_ref() else { + return Ok(()); + }; + let raw = openshell_core::driver_utils::read_upstream_proxy_credential_file( + path.to_str() + .ok_or_else(|| Error::config("proxy_auth_file must be valid UTF-8"))?, + ) + .map_err(Error::config)?; + openshell_core::driver_utils::parse_upstream_proxy_credential(&raw) + .map_err(|error| Error::config(format!("proxy_auth_file is invalid: {error}")))?; + Ok(()) +} + +fn docker_upstream_proxy_cli_args(config: &UpstreamProxyConfig) -> Vec { + let mut args = Vec::new(); + if let Some(url) = config.https_proxy.as_ref() { + args.extend(["--upstream-proxy".to_string(), url.clone()]); + } + if let Some(no_proxy) = config.no_proxy.as_ref() { + args.extend(["--upstream-no-proxy".to_string(), no_proxy.clone()]); + } + if config.proxy_auth_file.is_some() { + args.extend([ + "--upstream-proxy-auth-file".to_string(), + SUPERVISOR_PROXY_AUTH_MOUNT_PATH.to_string(), + ]); + } + if config.proxy_auth_allow_insecure == Some(true) { + args.push("--upstream-proxy-auth-allow-insecure".to_string()); + } + if config.proxy_connect_by_hostname == Some(true) { + args.push("--upstream-proxy-connect-by-hostname".to_string()); + } + args +} + fn validate_sandbox_pids_limit(value: Option) -> CoreResult<()> { - if value.is_some_and(|limit| limit.get() < 0) { + if value.is_some_and(|value| value.get() <= 0) { return Err(Error::config( "docker sandbox_pids_limit must be positive when set", )); @@ -3587,7 +5989,7 @@ fn validate_docker_app_armor_profile( } fn docker_pids_limit(value: Option) -> Result, Status> { - if value.is_some_and(|limit| limit.get() < 0) { + if value.is_some_and(|value| value.get() < 0) { return Err(Status::failed_precondition( "docker sandbox_pids_limit must be positive when set", )); @@ -3718,15 +6120,14 @@ fn driver_status_from_summary( last_transition_time: String::new(), }], deleting, + ..Default::default() } } /// Refine an exited Docker sandbox's `Ready` condition from inspected state. /// -/// A workspace-validation exit is reported distinctly so users can repair the -/// OCI working directory rather than diagnose a generic crash. A signal kill -/// (exit 137/143 = SIGKILL/SIGTERM, not OOM) is the signature of a -/// machine/daemon restart terminating a running container. Reclassify it from +/// A signal kill (exit 137/143 = SIGKILL/SIGTERM, not OOM) is the signature of +/// a machine/daemon restart terminating a running container. Reclassify it from /// the generic terminal `ContainerExited` to the recoverable /// `ContainerRuntimeRestart` so gateway startup can revive it. OOM kills and /// ordinary application exits stay `ContainerExited` and terminal. @@ -3734,7 +6135,7 @@ fn apply_docker_exit_classification(sandbox: &mut DriverSandbox, state: &Contain if state.oom_killed == Some(true) { return; } - let Some(code) = state.exit_code else { + let Some(code) = state.exit_code.filter(|&code| matches!(code, 137 | 143)) else { return; }; let Some(condition) = sandbox @@ -3747,13 +6148,8 @@ fn apply_docker_exit_classification(sandbox: &mut DriverSandbox, state: &Contain if condition.reason != CONDITION_EXITED { return; } - if code == i64::from(SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED) { - condition.reason = CONDITION_WORKSPACE_VALIDATION_FAILED.to_string(); - condition.message = "OCI WorkingDir is not usable by the sandbox identity".to_string(); - } else if matches!(code, 137 | 143) { - condition.reason = CONDITION_RUNTIME_RESTART.to_string(); - condition.message = format!("Container terminated by signal (exit code {code})"); - } + condition.reason = CONDITION_RUNTIME_RESTART.to_string(); + condition.message = format!("Container terminated by signal (exit code {code})"); } fn container_ready_condition( @@ -3852,12 +6248,23 @@ fn label_filters(values: impl IntoIterator) -> HashMap, +) -> HashMap> { + let mut values = vec![format!( + "{LABEL_ISOLATION_ROLE}={LABEL_ISOLATION_ROLE_SANDBOX}" + )]; + values.extend(extra_values); + managed_resource_label_filters(sandbox_namespace, values) +} + +fn managed_resource_label_filters( + sandbox_namespace: &str, extra_values: impl IntoIterator, ) -> HashMap> { let mut values = vec![ format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE}"), - format!("{LABEL_SANDBOX_NAMESPACE}={sandbox_label}"), + format!("{LABEL_SANDBOX_NAMESPACE}={sandbox_namespace}"), ]; values.extend(extra_values); label_filters(values) @@ -3932,190 +6339,7 @@ fn sanitize_docker_name(value: &str) -> String { .to_string() } -fn normalize_docker_arch(arch: &str) -> String { - match arch { - "x86_64" => "amd64".to_string(), - "aarch64" => "arm64".to_string(), - other => other.to_ascii_lowercase(), - } -} - -#[derive(Debug, Eq, PartialEq)] -enum SupervisorBinSource { - Binary(PathBuf), - Image(String), -} - -fn resolve_supervisor_bin_source( - docker_config: &DockerComputeConfig, - current_exe: Option<&Path>, - target_candidates: &[PathBuf], -) -> CoreResult { - // Tier 1: explicit supervisor_bin in [openshell.drivers.docker]. - if let Some(path) = docker_config.supervisor_bin.clone() { - let path = canonicalize_existing_file(&path, "docker supervisor binary")?; - validate_linux_elf_binary(&path).map_err(Error::config)?; - return Ok(SupervisorBinSource::Binary(path)); - } - - // Tier 2: explicit supervisor_image in [openshell.drivers.docker]. - // A configured image should be the source of truth even when a local - // developer build is present under target/. - if let Some(image) = docker_config.supervisor_image.clone() { - return Ok(SupervisorBinSource::Image(image)); - } - - // Tier 3: sibling `openshell-sandbox` next to the running gateway - // (release artifact layout). Linux-only because the sibling must be a - // Linux ELF to bind-mount into a Linux container. - if cfg!(target_os = "linux") - && let Some(current_exe) = current_exe - && let Some(parent) = current_exe.parent() - { - let sibling = parent.join("openshell-sandbox"); - if sibling.is_file() { - let path = canonicalize_existing_file(&sibling, "docker supervisor binary")?; - if validate_linux_elf_binary(&path).is_ok() { - return Ok(SupervisorBinSource::Binary(path)); - } - } - } - - // Tier 4: local cargo target build (developer workflow). Preferred - // over the default registry image when available because it matches - // whatever the developer just built. - for candidate in target_candidates { - if candidate.is_file() { - let path = canonicalize_existing_file(candidate, "docker supervisor binary")?; - if validate_linux_elf_binary(&path).is_ok() { - return Ok(SupervisorBinSource::Binary(path)); - } - } - } - - // Tier 5: pull the release-matched default supervisor image and extract - // the binary to a host-side cache keyed by image content digest. - Ok(SupervisorBinSource::Image( - openshell_core::config::default_supervisor_image(), - )) -} - -pub(crate) async fn resolve_supervisor_bin( - docker: &Docker, - docker_config: &DockerComputeConfig, - daemon_arch: &str, -) -> CoreResult { - let current_exe = - if cfg!(target_os = "linux") - && docker_config.supervisor_bin.is_none() - && docker_config.supervisor_image.is_none() - { - Some(std::env::current_exe().map_err(|err| { - Error::config(format!("failed to resolve current executable: {err}")) - })?) - } else { - None - }; - let target_candidates = linux_supervisor_candidates(daemon_arch); - - match resolve_supervisor_bin_source(docker_config, current_exe.as_deref(), &target_candidates)? - { - SupervisorBinSource::Binary(path) => Ok(path), - SupervisorBinSource::Image(image) => { - extract_supervisor_bin_from_image(docker, &image).await - } - } -} - -fn linux_supervisor_candidates(daemon_arch: &str) -> Vec { - match daemon_arch { - "arm64" => vec![PathBuf::from( - "target/aarch64-unknown-linux-gnu/release/openshell-sandbox", - )], - "amd64" => vec![PathBuf::from( - "target/x86_64-unknown-linux-gnu/release/openshell-sandbox", - )], - _ => Vec::new(), - } -} - -/// Pull the supervisor image (if not already local), extract -/// `/openshell-sandbox` to a host cache keyed by the image's content -/// digest, and return the cache path. -/// -/// The extraction is atomic: the binary is written to a sibling temp file -/// inside the digest-keyed directory and renamed into place, so concurrent -/// gateway starts don't observe a partial file. -async fn extract_supervisor_bin_from_image(docker: &Docker, image: &str) -> CoreResult { - let refresh_attempted = if supervisor_image_should_refresh(image) { - info!(image = image, "Refreshing mutable docker supervisor image"); - match pull_supervisor_image(docker, image).await { - Ok(()) => true, - Err(err) => { - warn!( - image = image, - error = %err, - "failed to refresh mutable docker supervisor image; falling back to local image if present", - ); - true - } - } - } else { - false - }; - - // Inspect first to see if the image is already present; only pull on miss. - let inspect = match docker.inspect_image(image).await { - Ok(inspect) => inspect, - Err(err) if is_not_found_error(&err) && !refresh_attempted => { - info!(image = image, "Pulling docker supervisor image"); - pull_supervisor_image(docker, image).await?; - docker.inspect_image(image).await.map_err(|err| { - Error::config(format!( - "failed to inspect docker supervisor image '{image}' after pull: {err}", - )) - })? - } - Err(err) if is_not_found_error(&err) => { - return Err(Error::config(format!( - "docker supervisor image '{image}' is not present locally after refresh attempt", - ))); - } - Err(err) => { - return Err(Error::config(format!( - "failed to inspect docker supervisor image '{image}': {err}", - ))); - } - }; - - let digest = inspect.id.clone().ok_or_else(|| { - Error::config(format!( - "docker supervisor image '{image}' inspect response has no Id", - )) - })?; - - let cache_path = - openshell_core::driver_utils::supervisor_cache_path("docker-supervisor", &digest) - .map_err(Error::config)?; - if cache_path.is_file() { - validate_linux_elf_binary(&cache_path).map_err(Error::config)?; - return Ok(cache_path); - } - - info!( - image = image, - digest = digest, - cache_path = %cache_path.display(), - "Extracting supervisor binary from image to host cache", - ); - - let binary_bytes = extract_supervisor_binary_bytes(docker, image).await?; - write_cache_binary_atomic(&cache_path, &binary_bytes).map_err(Error::config)?; - validate_linux_elf_binary(&cache_path).map_err(Error::config)?; - Ok(cache_path) -} - -async fn pull_supervisor_image(docker: &Docker, image: &str) -> CoreResult<()> { +async fn pull_runtime_image(docker: &Docker, image: &str, role: &str) -> CoreResult<()> { let mut stream = docker.create_image( Some(CreateImageOptions { from_image: Some(image.to_string()), @@ -4127,17 +6351,66 @@ async fn pull_supervisor_image(docker: &Docker, image: &str) -> CoreResult<()> { while let Some(result) = stream.next().await { result.map_err(|err| { Error::config(format!( - "failed to pull docker supervisor image '{image}': {err}", + "failed to pull Docker {role} image '{image}': {err}", )) })?; } Ok(()) } -/// Create a short-lived container from `image`, stream out the supervisor +async fn ensure_runtime_image(docker: &Docker, image: &str, role: &str) -> CoreResult { + let local_image_present = docker.inspect_image(image).await.is_ok(); + if supervisor_image_should_refresh(image) { + info!( + image = image, + role, "Refreshing mutable Docker runtime image" + ); + if let Err(error) = pull_runtime_image(docker, image, role).await { + if !local_image_present { + return Err(error); + } + warn!( + image = image, + error = %error, + "failed to refresh mutable Docker runtime image; using the local image", + ); + } + } else if !local_image_present { + pull_runtime_image(docker, image, role).await?; + } + let inspect = docker.inspect_image(image).await.map_err(|error| { + Error::config(format!( + "failed to inspect Docker {role} image '{image}': {error}" + )) + })?; + inspect.id.filter(|id| !id.is_empty()).ok_or_else(|| { + Error::config(format!( + "Docker {role} image '{image}' has no immutable image ID" + )) + }) +} + +/// Create a short-lived container from `image`, stream out the sandbox /// binary as a tar archive, and return the untarred file bytes. The /// container is always removed, even on error paths. -async fn extract_supervisor_binary_bytes(docker: &Docker, image: &str) -> CoreResult> { +async fn extract_sandbox_binary_bytes(docker: &Docker, image: &str) -> CoreResult> { + let bytes = + extract_runtime_path_archive(docker, image, SANDBOX_RUNTIME_IMAGE_BINARY_PATH, true) + .await?; + if !bytes.starts_with(b"\x7fELF") { + return Err(Error::config(format!( + "Docker sandbox runtime image '{image}' contains an invalid sandbox binary" + ))); + } + Ok(bytes) +} + +async fn extract_runtime_path_archive( + docker: &Docker, + image: &str, + path: &str, + extract_single_file: bool, +) -> CoreResult> { let container_name = temp_extract_container_name(); docker .create_container( @@ -4148,7 +6421,7 @@ async fn extract_supervisor_binary_bytes(docker: &Docker, image: &str) -> CoreRe ), ContainerCreateBody { image: Some(image.to_string()), - entrypoint: Some(vec![SUPERVISOR_IMAGE_BINARY_PATH.to_string()]), + entrypoint: Some(vec![path.to_string()]), cmd: Some(Vec::new()), ..Default::default() }, @@ -4161,7 +6434,8 @@ async fn extract_supervisor_binary_bytes(docker: &Docker, image: &str) -> CoreRe })?; // Always tear down the extractor container, even if extraction fails. - let result = download_binary_from_container(docker, &container_name).await; + let result = + download_path_from_container(docker, &container_name, path, extract_single_file).await; if let Err(remove_err) = docker .remove_container( &container_name, @@ -4172,18 +6446,20 @@ async fn extract_supervisor_binary_bytes(docker: &Docker, image: &str) -> CoreRe warn!( container = container_name, error = %remove_err, - "Failed to remove supervisor extractor container", + "Failed to remove runtime image extractor container", ); } result } -async fn download_binary_from_container( +async fn download_path_from_container( docker: &Docker, container_name: &str, + path: &str, + extract_single_file: bool, ) -> CoreResult> { let options = DownloadFromContainerOptionsBuilder::default() - .path(SUPERVISOR_IMAGE_BINARY_PATH) + .path(path) .build(); let mut stream = docker.download_from_container(container_name, Some(options)); @@ -4197,11 +6473,15 @@ async fn download_binary_from_container( tar_bytes.extend_from_slice(&chunk); } - extract_first_tar_entry(&tar_bytes).map_err(|err| { - Error::config(format!( - "failed to extract supervisor binary from tar archive returned by '{container_name}': {err}", - )) - }) + if extract_single_file { + extract_first_tar_entry(&tar_bytes).map_err(|err| { + Error::config(format!( + "failed to extract supervisor binary from tar archive returned by '{container_name}': {err}", + )) + }) + } else { + Ok(tar_bytes) + } } fn canonicalize_existing_file(path: &Path, description: &str) -> CoreResult { @@ -4221,8 +6501,8 @@ fn canonicalize_existing_file(path: &Path, description: &str) -> CoreResult bool { docker_config.guest_tls_ca.is_some() - || docker_config.guest_tls_cert.is_some() - || docker_config.guest_tls_key.is_some() + && docker_config.guest_tls_cert.is_some() + && docker_config.guest_tls_key.is_some() } pub(crate) fn docker_guest_tls_paths( @@ -4296,6 +6576,16 @@ fn is_conflict_error(err: &BollardError) -> bool { ) } +fn is_removal_in_progress_error(err: &BollardError) -> bool { + matches!( + err, + BollardError::DockerResponseServerError { + status_code: 409, + message, + } if message.contains("removal of container") && message.contains("is already in progress") + ) +} + fn is_not_modified_error(err: &BollardError) -> bool { matches!( err, diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index 772dc69198..1f11dafb72 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -4,9 +4,12 @@ use super::*; use openshell_core::config::DEFAULT_SERVER_PORT; use openshell_core::driver_utils::{ - CONDITION_WORKSPACE_VALIDATION_FAILED, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, - LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, LABEL_SANDBOX_NAMESPACE, - SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED, supervisor_cache_path_with_base, + LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, + LABEL_SANDBOX_NAMESPACE, +}; +use openshell_core::jwt::{ + CredentialEpoch, SandboxLaunchAuthentication, SecretJwt, SessionVerificationKey, + SupervisorAuthBundle, }; use openshell_core::progress::{ PROGRESS_ACTIVE_DETAIL_KEY, PROGRESS_ACTIVE_STEP_KEY, PROGRESS_COMPLETE_LABEL_KEY, @@ -16,15 +19,36 @@ use openshell_core::progress::{ use openshell_core::proto::compute::v1::{ DriverResourceRequirements, DriverSandboxSpec, DriverSandboxTemplate, GetGatewayListenerRequirementsRequest, GpuResourceRequirements, ResourceRequirements, - gateway_listener_requirement::Selector, + WorkloadIdentityRequest, gateway_listener_requirement::Selector, }; use std::fs; use std::net::{IpAddr, Ipv4Addr, SocketAddr}; -use std::sync::{Arc, LazyLock, Mutex}; +use std::sync::Arc; use tempfile::TempDir; -const TLS_MOUNT_DIR: &str = "/etc/openshell/tls/client"; -static ENV_LOCK: LazyLock> = LazyLock::new(|| Mutex::new(())); +fn test_launch_authentication() -> Vec { + serde_json::to_vec(&SandboxLaunchAuthentication { + supervisor: SupervisorAuthBundle { + session_id: openshell_core::SandboxSessionId::new(), + runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-1", + ) + .unwrap(), + session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(), + auth_epoch: CredentialEpoch::new(1).unwrap(), + gateway_token: SecretJwt::parse("gateway.token.value").unwrap(), + gateway_expires_at: i64::MAX, + sandbox_token: SecretJwt::parse("sandbox.token.value").unwrap(), + sandbox_expires_at: i64::MAX, + }, + gateway_id: "gateway-test".to_string(), + verification_keys: vec![SessionVerificationKey { + key_id: "test-key".to_string(), + public_key_pem: b"public-key".to_vec(), + }], + }) + .unwrap() +} fn test_sandbox() -> DriverSandbox { // Mirrors the gateway-supplied request: the public `Sandbox` API no @@ -50,6 +74,8 @@ fn test_sandbox() -> DriverSandbox { command: Vec::new(), tty: false, await_main_process_attachment: false, + workload_identity: None, + launch_authentication: test_launch_authentication(), }), status: None, workspace: String::new(), @@ -97,24 +123,25 @@ fn runtime_config() -> DockerDriverRuntimeConfig { DockerDriverRuntimeConfig { default_image: "image:latest".to_string(), image_pull_policy: ImagePullPolicy::IfNotPresent, - sandbox_label: "default".to_string(), - grpc_endpoint: "https://localhost:8443".to_string(), + sandbox_namespace: "default".to_string(), network_name: DEFAULT_DOCKER_NETWORK_NAME.to_string(), gateway_route: DockerGatewayRoute::Bridge { bind_address: SocketAddr::new( IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), DEFAULT_SERVER_PORT, ), - host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), }, gateway_callback_bind_address: Some(SocketAddr::new( IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), DEFAULT_SERVER_PORT, )), - ssh_socket_path: "/run/openshell/ssh.sock".to_string(), stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, log_level: "info".to_string(), - supervisor_bin: PathBuf::from("/tmp/openshell-sandbox"), + sandbox_binary: Arc::new(b"\x7fELFtest".to_vec()), + supervisor_image_id: "sha256:supervisor-test".to_string(), + supervisor_grpc_endpoint: "https://host.openshell.internal:8443".to_string(), + gateway_tls_server_name: None, + ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), guest_tls: Some(DockerGuestTlsPaths { ca: PathBuf::from("/tmp/ca.crt"), cert: PathBuf::from("/tmp/tls.crt"), @@ -125,7 +152,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig { cdi_supported: false, wsl_all_gpu_fallback_enabled: false, }, - sandbox_pids_limit: None, + sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(), enable_bind_mounts: false, upstream_proxy: UpstreamProxyConfig::default(), provider_spiffe_workload_api_socket: None, @@ -133,151 +160,15 @@ fn runtime_config() -> DockerDriverRuntimeConfig { } } -#[test] -fn docker_config_uses_canonical_sandbox_label_name() { - let config: DockerComputeConfig = - serde_json::from_value(serde_json::json!({ "sandbox_label": "tenant-a" })).unwrap(); - assert_eq!(config.sandbox_label, "tenant-a"); - - let serialized = serde_json::to_value(config).unwrap(); - assert_eq!(serialized["sandbox_label"], "tenant-a"); - assert!(serialized.get("sandbox_namespace").is_none()); -} - -#[test] -fn docker_config_rejects_legacy_sandbox_namespace() { - let error = serde_json::from_value::(serde_json::json!({ - "sandbox_namespace": "tenant-a" - })) - .expect_err("legacy sandbox_namespace must be rejected"); - assert!(error.to_string().contains("sandbox_namespace")); -} - -#[test] -fn docker_config_keeps_explicit_unconfined_apparmor_default() { - let config: DockerComputeConfig = serde_json::from_value(serde_json::json!({})) - .expect("default Docker config should deserialize"); - assert_eq!(config.app_armor_profile, Some(AppArmorProfile::Unconfined)); - let serialized = serde_json::to_value(config).expect("config should serialize"); - assert_eq!(serialized["app_armor_profile"], "Unconfined"); -} - -#[test] -fn docker_config_defaults_to_driver_owned_pids_limit() { - let config: DockerComputeConfig = serde_json::from_value(serde_json::json!({})) - .expect("default Docker config should deserialize"); - assert_eq!( - config.sandbox_pids_limit.map(std::num::NonZeroI64::get), - Some(openshell_core::config::DEFAULT_SANDBOX_PIDS_LIMIT) - ); -} - -#[test] -fn docker_config_rejects_invalid_pids_limits() { - let zero = serde_json::from_value::(serde_json::json!({ - "sandbox_pids_limit": 0 - })) - .expect_err("zero PID limit must be rejected"); - assert!(zero.to_string().contains("invalid value: integer `0`")); - - let negative: DockerComputeConfig = serde_json::from_value(serde_json::json!({ - "sandbox_pids_limit": -1 - })) - .expect("nonzero integer deserializes before semantic validation"); - let error = validate_sandbox_pids_limit(negative.sandbox_pids_limit).unwrap_err(); - assert!(error.to_string().contains("must be positive")); -} - -#[test] -fn docker_rejects_newer_image_pull_policy() { - let error = validate_image_pull_policy(ImagePullPolicy::Newer).unwrap_err(); - assert!(error.to_string().contains("supported only by the Podman")); -} - -#[test] -fn docker_apparmor_profiles_render_and_require_daemon_capability() { - for (profile, expected) in [ - (AppArmorProfile::RuntimeDefault, None), - ( - AppArmorProfile::Unconfined, - Some(vec!["apparmor=unconfined".to_string()]), - ), - ( - AppArmorProfile::Localhost("openshell-supervisor".to_string()), - Some(vec!["apparmor=openshell-supervisor".to_string()]), - ), - ] { - let mut config = runtime_config(); - config.app_armor_profile = Some(profile.clone()); - let body = build_container_create_body(&test_sandbox(), &config).unwrap(); - assert_eq!(body.host_config.unwrap().security_opt, expected); - } - - let unavailable = SystemInfo::default(); - assert!( - validate_docker_app_armor_profile(Some(&AppArmorProfile::Unconfined), &unavailable).is_ok() - ); - for confined in [ - AppArmorProfile::RuntimeDefault, - AppArmorProfile::Localhost("openshell-supervisor".to_string()), - ] { - let error = validate_docker_app_armor_profile(Some(&confined), &unavailable) - .expect_err("confined profile requires daemon AppArmor support"); - assert!( - error - .to_string() - .contains("Docker reports it is unavailable") - ); - } - - let available = SystemInfo { - security_options: Some(vec!["name=apparmor".to_string()]), - ..Default::default() - }; - assert!( - validate_docker_app_armor_profile( - Some(&AppArmorProfile::Localhost( - "openshell-supervisor".to_string() - )), - &available - ) - .is_ok() - ); -} - -#[test] -fn docker_config_uses_shared_proxy_contract_and_explicit_apparmor_default() { - let config: DockerComputeConfig = toml::from_str( - r#" -https_proxy = "http://proxy.example:8080" -no_proxy = ".svc" -proxy_auth_file = "/run/secrets/proxy-auth" -proxy_auth_allow_insecure = true -app_armor_profile = "Localhost/openshell-supervisor" -provider_spiffe_workload_api_socket = "/run/spire/agent.sock" -"#, +fn test_workload_identity() -> ResolvedWorkloadIdentity { + ResolvedWorkloadIdentity::new( + 1234, + 1235, + vec![1236], + "test".to_string(), + "sha256:immutable".to_string(), ) - .unwrap(); - assert_eq!( - config.upstream_proxy.https_proxy.as_deref(), - Some("http://proxy.example:8080") - ); - assert_eq!( - config.app_armor_profile, - Some(AppArmorProfile::Localhost( - "openshell-supervisor".to_string() - )) - ); - assert!(config.upstream_proxy.validate().is_ok()); - assert!( - openshell_core::driver_utils::validate_provider_spiffe_unix_socket( - config - .provider_spiffe_workload_api_socket - .as_deref() - .unwrap() - ) - .is_ok() - ); + .unwrap() } fn json_struct(value: serde_json::Value) -> prost_types::Struct { @@ -312,12 +203,14 @@ fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDr ), config, events: broadcast::channel(WATCH_BUFFER).0, - pending: Arc::new(tokio::sync::Mutex::new(HashMap::new())), + pending: Arc::new(Mutex::new(HashMap::new())), gpu_selector: Arc::new(CdiGpuDefaultSelector::new( CdiGpuInventory::default(), wsl_all_gpu_fallback_enabled, )), lifecycle_event_fences: DockerLifecycleEventFences::default(), + control_processes: Arc::new(Mutex::new(HashMap::new())), + runtime_failures: Arc::new(Mutex::new(HashMap::new())), } } @@ -359,39 +252,16 @@ fn request_with_traceparent(message: T) -> Request { request } -async fn fake_docker_with_no_containers() -> (String, JoinHandle<()>) { - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let address = listener.local_addr().unwrap(); - let server = tokio::spawn(async move { - while let Ok((mut stream, _)) = listener.accept().await { - openshell_core::net::set_tcp_nodelay_best_effort(&stream); - let mut scratch = [0_u8; 4096_usize]; - let _ = stream.read(&mut scratch).await; - let _ = stream - .write_all( - b"HTTP/1.1 200 OK\r\n\ - Content-Type: application/json\r\n\ - Content-Length: 2\r\n\r\n[]", - ) - .await; - let _ = stream.flush().await; - } - }); - (format!("http://{address}"), server) -} - async fn standalone_traced_client() -> ( TestDriverClient, - tokio::sync::oneshot::Sender<()>, + oneshot::Sender<()>, JoinHandle>, ) { use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let address = listener.local_addr().unwrap(); - let (shutdown, shutdown_rx) = tokio::sync::oneshot::channel(); + let (shutdown, shutdown_rx) = oneshot::channel(); let service = ComputeDriverService::new(test_driver_with_config(runtime_config())); let server = tokio::spawn(async move { tonic::transport::Server::builder() @@ -478,6 +348,78 @@ async fn tracing_standalone_rpc_layer_propagates_context_and_records_errors() { provider.shutdown().unwrap(); } +#[tokio::test] +async fn control_failure_overrides_running_container_readiness() { + let driver = test_driver_with_config(runtime_config()); + driver.runtime_failures.lock().await.insert( + "sbx-123".to_string(), + DockerRuntimeFailure { + reason: "ControlSupervisorExited", + message: "control exited unexpectedly".to_string(), + }, + ); + let mut sandbox = pending_sandbox_snapshot( + &test_sandbox(), + "default", + DriverCondition { + r#type: "Ready".to_string(), + status: "True".to_string(), + reason: "BackendReady".to_string(), + message: "Container is running".to_string(), + last_transition_time: String::new(), + }, + false, + ); + + driver.apply_runtime_failure(&mut sandbox).await; + + let ready = sandbox + .status + .unwrap() + .conditions + .into_iter() + .find(|condition| condition.r#type == "Ready") + .expect("ready condition"); + assert_eq!(ready.status, "False"); + assert_eq!(ready.reason, "ControlSupervisorExited"); + assert!(ready.message.contains("control exited unexpectedly")); +} + +#[tokio::test] +async fn control_failure_does_not_hide_a_terminal_container_exit() { + let driver = test_driver_with_config(runtime_config()); + driver.runtime_failures.lock().await.insert( + "sbx-123".to_string(), + DockerRuntimeFailure { + reason: "ControlSupervisorExited", + message: "control exited unexpectedly".to_string(), + }, + ); + let mut sandbox = pending_sandbox_snapshot( + &test_sandbox(), + "default", + DriverCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: CONDITION_EXITED.to_string(), + message: "Container exited".to_string(), + last_transition_time: String::new(), + }, + false, + ); + + driver.apply_runtime_failure(&mut sandbox).await; + + let ready = sandbox + .status + .unwrap() + .conditions + .into_iter() + .find(|condition| condition.r#type == "Ready") + .expect("ready condition"); + assert_eq!(ready.reason, CONDITION_EXITED); +} + #[tokio::test] async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; @@ -628,6 +570,45 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { driver_provider.shutdown().unwrap(); } +#[tokio::test] +async fn start_sandbox_span_does_not_capture_launch_authentication() { + use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); + + test_driver_with_config(runtime_config()) + .start_sandbox( + "sandbox-1", + "sandbox", + "invalid-generation", + b"secret-launch-authentication", + ) + .with_subscriber(subscriber) + .await + .expect_err("invalid generation must fail before contacting Docker"); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|span| span.name == "docker.start_sandbox") + .expect("start operation span"); + assert!(span.attributes.iter().all(|attribute| { + !matches!( + attribute.key.as_str(), + "launch_authentication" | "encoded_authentication" + ) + })); + provider.shutdown().unwrap(); +} + #[tokio::test] async fn tracing_lifecycle_rpc_failures_export_docker_operation_spans() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; @@ -696,10 +677,11 @@ async fn tracing_direct_start_exports_a_docker_start_span() { let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); let driver = test_driver_with_config(runtime_config()); - DockerComputeDriver::start_sandbox(&driver, "", "") - .with_subscriber(subscriber) - .await - .expect_err("missing identifier should fail"); + Box::pin( + DockerComputeDriver::start_sandbox(&driver, "", "", "", &[]).with_subscriber(subscriber), + ) + .await + .expect_err("missing identifier should fail"); provider.force_flush().unwrap(); let spans = exporter.get_finished_spans().unwrap(); @@ -731,13 +713,15 @@ async fn tracing_image_preparation_failure_exports_nested_failed_spans() { let driver = test_driver_with_config(config); async { - driver - .provision_sandbox_inner(&test_sandbox()) - .instrument(tracing::info_span!( - "docker.provision", - otel.status_code = tracing::field::Empty - )) - .await + Box::pin( + driver + .provision_sandbox_inner(&test_sandbox()) + .instrument(tracing::info_span!( + "docker.provision", + otel.status_code = tracing::field::Empty + )), + ) + .await } .with_subscriber(subscriber) .await @@ -1015,34 +999,6 @@ async fn host_gateway_route_reports_ipv4_loopback_callback_listener() { ); } -#[test] -fn container_visible_endpoint_rewrites_loopback_hosts() { - assert_eq!( - docker_container_openshell_endpoint( - "https://localhost:8443", - HOST_OPENSHELL_INTERNAL, - DEFAULT_SERVER_PORT, - ), - "https://host.openshell.internal:17670/" - ); - assert_eq!( - docker_container_openshell_endpoint( - "http://127.0.0.1:8080", - HOST_OPENSHELL_INTERNAL, - DEFAULT_SERVER_PORT, - ), - "http://host.openshell.internal:17670/" - ); - assert_eq!( - docker_container_openshell_endpoint( - "https://gateway.internal:8443", - HOST_OPENSHELL_INTERNAL, - DEFAULT_SERVER_PORT, - ), - "https://host.openshell.internal:17670/" - ); -} - #[test] fn docker_bridge_gateway_ip_requires_ipv4_gateway() { let network = bollard::models::NetworkInspect { @@ -1107,13 +1063,21 @@ fn docker_gateway_route_uses_host_gateway_for_docker_desktop() { ), DockerGatewayRoute::HostGateway ); - assert_eq!( - docker_extra_hosts(&DockerGatewayRoute::HostGateway), - vec![ - "host.docker.internal:host-gateway".to_string(), - "host.openshell.internal:host-gateway".to_string() - ] - ); +} + +#[test] +fn vm_backed_docker_daemon_uses_daemon_local_companion_transport() { + let desktop = SystemInfo { + operating_system: Some("Docker Desktop".to_string()), + ..Default::default() + }; + let native = SystemInfo { + operating_system: Some("Ubuntu 24.04".to_string()), + ..Default::default() + }; + + assert!(uses_host_gateway_alias(&desktop)); + assert!(!uses_host_gateway_alias(&native)); } #[test] @@ -1158,13 +1122,6 @@ fn docker_gateway_route_uses_host_gateway_for_colima() { ), DockerGatewayRoute::HostGateway ); - assert_eq!( - docker_extra_hosts(&DockerGatewayRoute::HostGateway), - vec![ - "host.docker.internal:host-gateway".to_string(), - "host.openshell.internal:host-gateway".to_string() - ] - ); } #[test] @@ -1249,16 +1206,8 @@ fn docker_gateway_route_uses_bridge_gateway_for_linux_docker() { route, DockerGatewayRoute::Bridge { bind_address: "172.18.0.1:17670".parse().unwrap(), - host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), } ); - assert_eq!( - docker_extra_hosts(&route), - vec![ - "host.docker.internal:172.18.0.1".to_string(), - "host.openshell.internal:172.18.0.1".to_string() - ] - ); } #[test] @@ -1298,15 +1247,35 @@ fn docker_gateway_route_prefers_configured_host_gateway_ip() { route, DockerGatewayRoute::Bridge { bind_address: "172.20.0.4:17670".parse().unwrap(), - host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 20, 0, 4)), } ); +} + +#[test] +fn docker_supervisor_alias_matches_the_trusted_gateway_route() { assert_eq!( - docker_extra_hosts(&route), - vec![ - "host.docker.internal:172.20.0.4".to_string(), - "host.openshell.internal:172.20.0.4".to_string() - ] + docker_supervisor_host_alias(&DockerGatewayRoute::Bridge { + bind_address: "172.20.0.4:17670".parse().unwrap(), + }), + "172.20.0.4" + ); + assert_eq!( + docker_supervisor_host_alias(&DockerGatewayRoute::HostGateway), + "host-gateway" + ); +} + +#[test] +fn docker_boundary_pins_only_concrete_host_gateway_addresses() { + assert_eq!( + docker_boundary_host_gateway_ip(&DockerGatewayRoute::Bridge { + bind_address: "172.20.0.4:17670".parse().unwrap(), + }), + Some(IpAddr::V4(Ipv4Addr::new(172, 20, 0, 4))) + ); + assert_eq!( + docker_boundary_host_gateway_ip(&DockerGatewayRoute::HostGateway), + None ); } @@ -1381,10 +1350,11 @@ fn docker_resource_limits_applies_cpu_and_memory_limits() { } #[test] -fn docker_pids_limit_uses_runtime_default_when_omitted() { +fn docker_pids_limit_uses_driver_default_and_allows_runtime_inherit() { + let default = openshell_core::config::default_sandbox_pids_limit(); assert_eq!( - docker_pids_limit(std::num::NonZeroI64::new(2048)).unwrap(), - Some(2048) + docker_pids_limit(default).unwrap(), + default.map(std::num::NonZeroI64::get) ); assert_eq!(docker_pids_limit(None).unwrap(), None); assert!(docker_pids_limit(std::num::NonZeroI64::new(-1)).is_err()); @@ -1397,109 +1367,67 @@ fn docker_compute_config_disables_bind_mounts_by_default() { } #[test] -fn container_create_body_omits_pids_limit_by_default() { - let body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); - let host_config = body.host_config.expect("host config"); - assert_eq!(host_config.pids_limit, None); -} - -#[test] -fn container_create_body_emits_configured_positive_pids_limit() { - let mut config = runtime_config(); - config.sandbox_pids_limit = std::num::NonZeroI64::new(4096); - let body = build_container_create_body(&test_sandbox(), &config).unwrap(); - assert_eq!( - body.host_config.expect("host config").pids_limit, - Some(4096) - ); -} +fn repository_e2e_docker_configuration_uses_the_supported_schema() { + let source = include_str!("../../../e2e/configs/gateway/docker.toml"); + let (_, docker_table) = source + .split_once("[openshell.drivers.docker]") + .expect("Docker E2E config contains a driver table"); + let config: DockerComputeConfig = + toml::from_str(docker_table).expect("Docker E2E driver config parses"); -#[test] -fn build_environment_sets_docker_tls_paths() { - let env = build_environment(&test_sandbox(), &runtime_config()); - assert!(env.contains(&format!("OPENSHELL_TLS_CA={TLS_CA_MOUNT_PATH}"))); - assert!(env.contains(&format!("OPENSHELL_TLS_CERT={TLS_CERT_MOUNT_PATH}"))); - assert!(env.contains(&format!("OPENSHELL_TLS_KEY={TLS_KEY_MOUNT_PATH}"))); - assert!(env.contains(&"TEMPLATE_ENV=template".to_string())); - assert!(env.contains(&"SPEC_ENV=spec".to_string())); - assert!(env.contains(&format!( - "{}={}", - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, - openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY - ))); - let encoded = env - .iter() - .find_map(|entry| { - entry - .strip_prefix("OPENSHELL_MAIN_PROCESS_SPEC=") - .map(str::to_string) - }) - .expect("main-process transport"); - let main = openshell_core::sandbox_env::MainProcessConfig::decode(&encoded).unwrap(); - // An omitted command is forwarded empty; the supervisor resolves the default - // login shell against the sandbox image at startup. - assert!(main.command.is_empty()); - assert!(main.tty); + assert_eq!(config.image_pull_policy, ImagePullPolicy::IfNotPresent); + assert_eq!(config.sandbox_label, "openshell-e2e"); } #[test] -fn build_environment_keeps_network_capabilities_driver_controlled() { - let mut sandbox = test_sandbox(); - sandbox.spec.as_mut().unwrap().environment.insert( - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES.to_string(), - "spoofed".to_string(), +fn container_create_body_sets_driver_owned_pids_limit() { + let body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); + let host_config = body.host_config.expect("host config"); + assert_eq!( + host_config.pids_limit, + openshell_core::config::default_sandbox_pids_limit().map(std::num::NonZeroI64::get) ); - let env = build_environment(&sandbox, &runtime_config()); - assert!(env.contains(&format!( - "{}={}", - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, - openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY - ))); - assert!(!env.iter().any(|entry| entry.ends_with("=spoofed"))); } #[test] -fn build_environment_protects_oci_identity_metadata() { +fn docker_child_environment_strips_supervisor_control_keys() { let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); - for (key, value) in [ - (openshell_core::sandbox_env::OCI_IMAGE_USER, "spoofed"), - (openshell_core::sandbox_env::SANDBOX_UID, "9999"), - (openshell_core::sandbox_env::SANDBOX_GID, "9999"), + for key in [ + openshell_core::sandbox_env::ENDPOINT, + openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME, + openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, + openshell_core::sandbox_env::OCI_IMAGE_USER, + openshell_core::sandbox_env::SANDBOX_TOKEN, + openshell_core::sandbox_env::SANDBOX_TOKEN_FILE, ] { - spec.environment.insert(key.to_string(), value.to_string()); + spec.environment + .insert(key.to_string(), "spoofed".to_string()); } + spec.environment + .insert("PATH".to_string(), "/agent/bin".to_string()); - let env = build_environment_for_oci_user(&sandbox, &runtime_config(), "app:staff"); + let env = docker_child_environment(&sandbox); - assert!(env.contains(&format!( - "{}=app:staff", - openshell_core::sandbox_env::OCI_IMAGE_USER - ))); - assert!(env.contains(&format!("{}=", openshell_core::sandbox_env::SANDBOX_UID))); - assert!(env.contains(&format!("{}=", openshell_core::sandbox_env::SANDBOX_GID))); - assert!(!env.iter().any(|entry| entry.ends_with("=spoofed"))); - assert!(!env.iter().any(|entry| entry.ends_with("=9999"))); + assert_eq!(env.get("PATH").map(String::as_str), Some("/agent/bin")); + assert!(env.contains_key("TEMPLATE_ENV")); + assert!(env.contains_key("SPEC_ENV")); + assert!(!env.values().any(|value| value == "spoofed")); } #[test] -fn build_environment_strips_gateway_tls_server_name() { - let mut sandbox = test_sandbox(); - let spec = sandbox.spec.as_mut().unwrap(); - spec.environment.insert( - openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME.to_string(), - "evil.attacker.example.com".to_string(), - ); - - let env = build_environment(&sandbox, &runtime_config()); +fn boundary_environment_contains_only_driver_owned_values() { + let env = build_boundary_environment(&test_sandbox(), &runtime_config()); + assert_eq!(env.len(), 2); assert!( - !env.iter().any(|entry| entry.starts_with(&format!( - "{}=", - openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME - ))), - "GATEWAY_TLS_SERVER_NAME must be stripped from the supervisor environment" + env.iter() + .any(|entry| entry.starts_with("OPENSHELL_LOG_LEVEL=")) ); + assert!(env.iter().any(|entry| entry.starts_with(&format!( + "{}=", + openshell_core::sandbox_env::TELEMETRY_ENABLED + )))); } #[test] @@ -1517,49 +1445,279 @@ fn container_creation_uses_inspected_immutable_image() { &DockerSandboxDriverConfig::default(), None, &metadata, + &test_workload_identity(), ) .unwrap(); assert_eq!(body.image.as_deref(), Some("sha256:immutable")); - assert_eq!(body.user.as_deref(), Some("0")); + assert_eq!(body.user.as_deref(), Some("1234:1235")); assert_eq!(body.working_dir.as_deref(), Some("/")); + assert_eq!( + body.labels + .as_ref() + .and_then(|labels| labels.get(LABEL_ISOLATION_BACKEND)) + .map(String::as_str), + Some(LABEL_ISOLATION_BACKEND_OPEN_SHELL) + ); + assert_eq!( + body.labels + .as_ref() + .and_then(|labels| labels.get(LABEL_ISOLATION_ROLE)) + .map(String::as_str), + Some(LABEL_ISOLATION_ROLE_SANDBOX) + ); assert_eq!( body.cmd.as_deref(), - Some(&["--workdir".to_string(), "/workspace/project".to_string()][..]) + Some( + &[ + "--bootstrap".to_string(), + BOUNDARY_CONFIG_MOUNT_PATH.to_string(), + ][..] + ) ); - assert!(body.env.unwrap().contains(&format!( - "{}=1234:1235", - openshell_core::sandbox_env::OCI_IMAGE_USER - ))); + assert!(body.env.unwrap().iter().all(|entry| { + !entry.starts_with(&format!("{}=", openshell_core::sandbox_env::OCI_IMAGE_USER)) + })); + let host = body.host_config.unwrap(); + assert_eq!(host.cap_add, None); + assert_eq!(host.cap_drop, Some(vec!["ALL".to_string()])); + assert_eq!(host.group_add, Some(vec!["1236".to_string()])); + assert_eq!( + host.security_opt, + Some(vec![ + "no-new-privileges:true".to_string(), + "apparmor=unconfined".to_string(), + ]) + ); + assert_eq!(host.network_mode.as_deref(), Some("none")); + assert_eq!(host.dns, Some(vec!["127.0.0.53".to_string()])); } #[test] -fn container_creation_rejects_invalid_oci_working_dir() { - let metadata = DockerImageMetadata { - id: "sha256:immutable".to_string(), - user: "1234:1235".to_string(), - working_dir: "relative/workspace".to_string(), - volumes: Vec::new(), +fn docker_outer_fence_accepts_network_none_without_attachments() { + let inspected = bollard::models::ContainerInspectResponse { + host_config: Some(HostConfig { + network_mode: Some("none".to_string()), + ..Default::default() + }), + network_settings: Some(bollard::models::NetworkSettings { + networks: Some(HashMap::from([( + "none".to_string(), + bollard::models::EndpointSettings::default(), + )])), + ..Default::default() + }), + ..Default::default() }; - let err = build_container_create_body_for_image( - &test_sandbox(), - &runtime_config(), - &DockerSandboxDriverConfig::default(), - None, - &metadata, - ) - .unwrap_err(); - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert!(err.message().contains("must be an absolute container path")); + assert!(validate_docker_outer_fence(&inspected).is_ok()); } #[test] -fn container_creation_rejects_openshell_control_path_working_dir() { - let metadata = DockerImageMetadata { - id: "sha256:immutable".to_string(), +fn docker_outer_fence_rejects_network_mode_or_attached_network_drift() { + let bridge_mode = bollard::models::ContainerInspectResponse { + host_config: Some(HostConfig { + network_mode: Some("bridge".to_string()), + ..Default::default() + }), + ..Default::default() + }; + let attached_network = bollard::models::ContainerInspectResponse { + host_config: Some(HostConfig { + network_mode: Some("none".to_string()), + ..Default::default() + }), + network_settings: Some(bollard::models::NetworkSettings { + networks: Some(HashMap::from([( + "unexpected".to_string(), + bollard::models::EndpointSettings::default(), + )])), + ..Default::default() + }), + ..Default::default() + }; + + assert!(validate_docker_outer_fence(&bridge_mode).is_err()); + assert!(validate_docker_outer_fence(&attached_network).is_err()); +} + +#[test] +fn sandbox_bundle_prepares_only_the_driver_managed_workspace() { + let identity = test_workload_identity(); + let default_archive = docker_sandbox_bundle_archive( + b"sandbox-binary", + b"{}", + DockerSandboxTls { + certificate: b"server-cert", + private_key: b"server-key", + }, + &identity, + driver_mounts::DEFAULT_WORKSPACE_ROOT, + ) + .unwrap(); + let mut archive = tar::Archive::new(default_archive.as_slice()); + let sandbox_entry = archive + .entries() + .unwrap() + .map(Result::unwrap) + .find(|entry| entry.path().unwrap().as_ref() == Path::new("sandbox")) + .expect("managed /sandbox entry"); + assert!(sandbox_entry.header().entry_type().is_dir()); + assert_eq!(sandbox_entry.header().mode().unwrap(), 0o700); + assert_eq!( + sandbox_entry.header().uid().unwrap(), + u64::from(identity.uid) + ); + assert_eq!( + sandbox_entry.header().gid().unwrap(), + u64::from(identity.gid) + ); + + let image_archive = docker_sandbox_bundle_archive( + b"sandbox-binary", + b"{}", + DockerSandboxTls { + certificate: b"server-cert", + private_key: b"server-key", + }, + &identity, + "/workspace/project", + ) + .unwrap(); + let mut archive = tar::Archive::new(image_archive.as_slice()); + assert!( + archive + .entries() + .unwrap() + .map(Result::unwrap) + .all(|entry| { entry.path().unwrap().as_ref() != Path::new("workspace/project") }) + ); +} + +#[test] +fn sandbox_bundle_stages_private_tls_server_material() { + let identity = test_workload_identity(); + let archive = docker_sandbox_bundle_archive( + b"sandbox-binary", + b"{}", + DockerSandboxTls { + certificate: b"server-cert", + private_key: b"server-key", + }, + &identity, + driver_mounts::DEFAULT_WORKSPACE_ROOT, + ) + .unwrap(); + let mut archive = tar::Archive::new(archive.as_slice()); + let entries = archive + .entries() + .unwrap() + .map(Result::unwrap) + .filter_map(|entry| { + let path = entry.path().ok()?.into_owned(); + Some(( + path, + ( + entry.header().mode().ok()?, + entry.header().uid().ok()?, + entry.header().gid().ok()?, + ), + )) + }) + .collect::>(); + for path in [ + ".openshell/channel/sandbox/server.crt", + ".openshell/channel/sandbox/server.key", + ] { + assert_eq!( + entries.get(Path::new(path)), + Some(&(0o600, u64::from(identity.uid), u64::from(identity.gid))) + ); + } + assert_eq!( + entries.get(Path::new(".openshell/runtime/openshell-sandbox")), + Some(&(0o555, 0, 0)), + "the trusted sandbox executable must not be writable by the workload" + ); + assert_eq!( + entries.get(Path::new(".openshell/channel")), + Some(&(0o755, 0, 0)), + "the workload must not be able to replace the supervisor secret directory" + ); + assert_eq!( + entries.get(Path::new(".openshell/channel/sandbox")), + Some(&(0o711, u64::from(identity.uid), u64::from(identity.gid))), + "the supervisor must be able to traverse to the authenticated socket without reading sandbox secrets" + ); +} + +#[test] +fn docker_identity_resolution_uses_pinned_image_accounts_and_exact_groups() { + let sandbox = test_sandbox(); + let image = DockerImageMetadata { + id: "sha256:image".to_string(), + user: "agent".to_string(), + working_dir: "/sandbox".to_string(), + volumes: Vec::new(), + }; + let resolved = resolve_docker_identity_from_accounts( + &sandbox, + &image, + b"root:x:0:0:root:/root:/bin/sh\nagent:x:10001:10002::/sandbox:/bin/sh\n", + b"root:x:0:\nagent:x:10002:\nrender:x:10003:agent\n", + ) + .unwrap(); + + assert_eq!(resolved.uid, 10001); + assert_eq!(resolved.gid, 10002); + assert_eq!(resolved.supplementary_gids, vec![10003]); + assert_eq!(resolved.source, "image"); + assert_eq!(resolved.resource_digest, "sha256:image"); +} + +#[test] +fn docker_identity_resolution_honors_policy_selectors_and_rejects_root() { + let mut sandbox = test_sandbox(); + sandbox.spec.as_mut().unwrap().workload_identity = Some(WorkloadIdentityRequest { + user: "10001".to_string(), + group: "workers".to_string(), + }); + let image = DockerImageMetadata { + id: "sha256:image".to_string(), + user: String::new(), + working_dir: "/sandbox".to_string(), + volumes: Vec::new(), + }; + let resolved = resolve_docker_identity_from_accounts( + &sandbox, + &image, + b"agent:x:10001:10002::/sandbox:/bin/sh\n", + b"workers:x:10004:agent\n", + ) + .unwrap(); + assert_eq!((resolved.uid, resolved.gid), (10001, 10004)); + assert_eq!(resolved.source, "policy"); + + sandbox.spec.as_mut().unwrap().workload_identity = Some(WorkloadIdentityRequest { + user: "root".to_string(), + group: "root".to_string(), + }); + let error = resolve_docker_identity_from_accounts( + &sandbox, + &image, + b"root:x:0:0:root:/root:/bin/sh\n", + b"root:x:0:\n", + ) + .unwrap_err(); + assert!(error.message().contains("UID or GID zero")); +} + +#[test] +fn container_creation_rejects_invalid_oci_working_dir() { + let metadata = DockerImageMetadata { + id: "sha256:immutable".to_string(), user: "1234:1235".to_string(), - working_dir: "/opt/openshell/bin/project".to_string(), + working_dir: "relative/workspace".to_string(), volumes: Vec::new(), }; let err = build_container_create_body_for_image( @@ -1568,60 +1726,61 @@ fn container_creation_rejects_openshell_control_path_working_dir() { &DockerSandboxDriverConfig::default(), None, &metadata, + &test_workload_identity(), ) .unwrap_err(); assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert!(err.message().contains("OpenShell control path")); + assert!(err.message().contains("must be an absolute container path")); } #[test] -fn container_creation_rejects_image_volume_that_masks_working_dir() { - let sandbox = test_sandbox(); +fn container_creation_rejects_openshell_control_path_working_dir() { let metadata = DockerImageMetadata { id: "sha256:immutable".to_string(), user: "1234:1235".to_string(), - working_dir: "/workspace/project".to_string(), - volumes: vec!["/workspace".to_string()], + working_dir: "/opt/openshell/bin/project".to_string(), + volumes: Vec::new(), }; - - let error = build_container_create_body_for_image( - &sandbox, + let err = build_container_create_body_for_image( + &test_sandbox(), &runtime_config(), &DockerSandboxDriverConfig::default(), None, &metadata, + &test_workload_identity(), ) .unwrap_err(); - assert!( - error - .message() - .contains("masks OCI WorkingDir '/workspace/project'") - ); + assert_eq!(err.code(), tonic::Code::FailedPrecondition); + assert!(err.message().contains("OpenShell control path")); } #[test] -fn container_creation_rejects_image_volume_over_configured_ssh_socket() { +fn container_creation_rejects_image_volume_that_masks_working_dir() { + let sandbox = test_sandbox(); let metadata = DockerImageMetadata { id: "sha256:immutable".to_string(), user: "1234:1235".to_string(), - working_dir: "/workspace".to_string(), - volumes: vec!["/custom-runtime".to_string()], + working_dir: "/workspace/project".to_string(), + volumes: vec!["/workspace".to_string()], }; - let mut config = runtime_config(); - config.ssh_socket_path = "/custom-runtime/ssh.sock".to_string(); let error = build_container_create_body_for_image( - &test_sandbox(), - &config, + &sandbox, + &runtime_config(), &DockerSandboxDriverConfig::default(), None, &metadata, + &test_workload_identity(), ) .unwrap_err(); - assert!(error.message().contains("OpenShell control path")); + assert!( + error + .message() + .contains("masks OCI WorkingDir '/workspace/project'") + ); } #[test] @@ -1642,6 +1801,7 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( &root_mount, None, &metadata, + &test_workload_identity(), ) .unwrap_err(); assert!( @@ -1664,6 +1824,7 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( &ancestor_mount, None, &nested_metadata, + &test_workload_identity(), ) .unwrap_err(); assert!( @@ -1681,6 +1842,7 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( &nested_mount, None, &metadata, + &test_workload_identity(), ) .expect("nested workspace mounts remain supported"); @@ -1695,84 +1857,15 @@ fn container_creation_reserves_resolved_workspace_root_but_allows_nested_mounts( &compatibility_path_mount, None, &metadata, + &test_workload_identity(), ) .expect("/sandbox remains mountable when the inspected workspace is elsewhere"); } #[test] -fn build_environment_keeps_path_driver_controlled() { - let mut sandbox = test_sandbox(); - let spec = sandbox.spec.as_mut().unwrap(); - spec.environment - .insert("PATH".to_string(), "/malicious/spec/bin".to_string()); - spec.template - .as_mut() - .unwrap() - .environment - .insert("PATH".to_string(), "/malicious/template/bin".to_string()); - - let env = build_environment(&sandbox, &runtime_config()); - let path_entries = env - .iter() - .filter(|entry| entry.starts_with("PATH=")) - .collect::>(); - - let expected_path = format!("PATH={SUPERVISOR_PATH}"); - assert_eq!(path_entries.len(), 1); - assert_eq!(path_entries[0], &expected_path); -} - -#[test] -fn build_environment_keeps_telemetry_toggle_driver_controlled() { - let _guard = ENV_LOCK.lock().unwrap(); - temp_env::with_vars( - [( - openshell_core::sandbox_env::TELEMETRY_ENABLED, - Some("false"), - )], - || { - let mut sandbox = test_sandbox(); - sandbox.spec.as_mut().unwrap().environment.insert( - openshell_core::sandbox_env::TELEMETRY_ENABLED.to_string(), - "true".to_string(), - ); - - let env = build_environment(&sandbox, &runtime_config()); - let telemetry_entries = env - .iter() - .filter(|entry| { - entry.starts_with(&format!( - "{}=", - openshell_core::sandbox_env::TELEMETRY_ENABLED - )) - }) - .collect::>(); - - assert_eq!(telemetry_entries.len(), 1); - assert_eq!( - telemetry_entries[0], - &format!("{}=false", openshell_core::sandbox_env::TELEMETRY_ENABLED) - ); - }, - ); -} - -#[test] -fn build_binds_uses_docker_tls_directory() { - let binds = build_binds(&test_sandbox(), &runtime_config()).unwrap(); - let targets = binds - .iter() - .filter_map(|bind| bind.split(':').nth(1).map(String::from)) - .collect::>(); - assert!(targets.contains(&SUPERVISOR_MOUNT_PATH.to_string())); - assert!(targets.contains(&TLS_CA_MOUNT_PATH.to_string())); - assert!(targets.contains(&TLS_CERT_MOUNT_PATH.to_string())); - assert!(targets.contains(&TLS_KEY_MOUNT_PATH.to_string())); - assert!( - targets - .iter() - .all(|target| target.starts_with(TLS_MOUNT_DIR) || target == SUPERVISOR_MOUNT_PATH) - ); +fn build_binds_does_not_expose_host_runtime_material() { + let binds = build_binds(&test_sandbox(), &runtime_config()); + assert!(binds.is_empty()); } #[test] @@ -1805,7 +1898,7 @@ fn build_container_create_body_includes_driver_config_mounts() { .mounts .expect("driver config mounts should be set"); - assert_eq!(mounts.len(), 2); + assert_eq!(mounts.len(), 3); assert_eq!(mounts[0].typ, Some(MountTypeEnum::VOLUME)); assert_eq!(mounts[0].source.as_deref(), Some("work-nfs")); assert_eq!(mounts[0].target.as_deref(), Some("/sandbox/work")); @@ -1819,6 +1912,9 @@ fn build_container_create_body_includes_driver_config_mounts() { ); assert_eq!(mounts[1].typ, Some(MountTypeEnum::TMPFS)); assert_eq!(mounts[1].target.as_deref(), Some("/sandbox/cache")); + assert_eq!(mounts[2].typ, Some(MountTypeEnum::VOLUME)); + assert_eq!(mounts[2].target.as_deref(), Some(BOUNDARY_MOUNT_PATH)); + assert_eq!(mounts[2].read_only, Some(false)); assert_eq!( mounts[1] .tmpfs_options @@ -2244,36 +2340,6 @@ fn driver_config_rejects_reserved_mount_targets() { assert!(err.message().contains("reserved OpenShell path")); } -#[test] -fn driver_config_rejects_mount_over_configured_ssh_socket() { - let mount_config: DockerSandboxDriverConfig = serde_json::from_value(serde_json::json!({ - "mounts": [{ - "type": "tmpfs", - "target": "/custom-runtime" - }] - })) - .unwrap(); - let metadata = DockerImageMetadata { - id: "sha256:immutable".to_string(), - user: "1234:1235".to_string(), - working_dir: "/workspace".to_string(), - volumes: Vec::new(), - }; - let mut config = runtime_config(); - config.ssh_socket_path = "/custom-runtime/ssh.sock".to_string(); - - let error = build_container_create_body_for_image( - &test_sandbox(), - &config, - &mount_config, - None, - &metadata, - ) - .unwrap_err(); - - assert!(error.message().contains("OpenShell control path")); -} - #[test] fn docker_local_volume_with_bind_option_is_bind_backed() { let volume = inspected_volume( @@ -2326,72 +2392,6 @@ fn docker_nonlocal_volume_with_bind_option_is_not_bind_backed() { assert!(!docker_volume_is_bind_backed(&volume)); } -#[test] -fn build_environment_uses_token_file_without_raw_token_env() { - let mut sandbox = test_sandbox(); - let spec = sandbox.spec.as_mut().unwrap(); - spec.sandbox_token = "secret.jwt.value".to_string(); - spec.environment.insert( - openshell_core::sandbox_env::SANDBOX_TOKEN.to_string(), - "user-provided-token".to_string(), - ); - - let env = build_environment(&sandbox, &runtime_config()); - - assert!(!env.iter().any(|entry| { - entry.starts_with(&format!("{}=", openshell_core::sandbox_env::SANDBOX_TOKEN)) - })); - assert!(env.contains(&format!( - "{}={SANDBOX_TOKEN_MOUNT_PATH}", - openshell_core::sandbox_env::SANDBOX_TOKEN_FILE - ))); -} - -#[test] -fn docker_container_projects_proxy_and_spiffe_without_credential_metadata() { - let mut config = runtime_config(); - config.upstream_proxy = UpstreamProxyConfig { - https_proxy: Some("https://proxy.example:8443".to_string()), - no_proxy: Some(".svc".to_string()), - proxy_auth_file: Some(PathBuf::from("/run/secrets/proxy-auth")), - proxy_auth_allow_insecure: None, - proxy_connect_by_hostname: Some(true), - }; - config.provider_spiffe_workload_api_socket = Some(PathBuf::from("/run/spire/agent.sock")); - let body = build_container_create_body(&test_sandbox(), &config).unwrap(); - let command = body.cmd.unwrap(); - assert!( - command - .windows(2) - .any(|args| args == ["--upstream-proxy", "https://proxy.example:8443"]) - ); - assert!( - command - .windows(2) - .any(|args| args == ["--upstream-proxy-auth-file", UPSTREAM_PROXY_AUTH_MOUNT_PATH]) - ); - assert!( - command - .windows(2) - .any(|args| args == ["--upstream-no-proxy", ".svc"]) - ); - assert!(command.contains(&"--upstream-proxy-connect-by-hostname".to_string())); - let binds = body.host_config.unwrap().binds.unwrap(); - assert!( - binds - .iter() - .any(|bind| bind.contains(UPSTREAM_PROXY_AUTH_MOUNT_PATH)) - ); - assert!(binds.contains(&format!( - "/run/spire:{PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR}:ro" - ))); - assert!(binds.iter().all(|bind| !bind.contains("rbind"))); - let env = body.env.unwrap(); - assert!(env.iter().any(|entry| entry - == "OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET=/spiffe-workload-api/agent.sock")); - assert!(!env.iter().any(|entry| entry.contains("proxy-auth"))); -} - #[test] fn managed_container_label_filters_include_gateway_namespace() { let filters = @@ -2400,20 +2400,26 @@ fn managed_container_label_filters_include_gateway_namespace() { assert!(labels.contains(&format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE}"))); assert!(labels.contains(&format!("{LABEL_SANDBOX_NAMESPACE}=tenant-a"))); + assert!(labels.contains(&format!( + "{LABEL_ISOLATION_ROLE}={LABEL_ISOLATION_ROLE_SANDBOX}" + ))); assert!(labels.contains(&format!("{LABEL_SANDBOX_ID}=sbx-123"))); } #[test] -fn build_container_create_body_replaces_inherited_cmd_with_workspace_arg() { +fn build_container_create_body_replaces_inherited_cmd_with_sandbox_bootstrap() { let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); assert_eq!( create_body.entrypoint, - Some(vec![SUPERVISOR_MOUNT_PATH.to_string()]) + Some(vec![SANDBOX_BINARY_PATH.to_string()]) ); assert_eq!( create_body.cmd, - Some(vec!["--workdir".to_string(), "/sandbox".to_string()]) + Some(vec![ + "--bootstrap".to_string(), + BOUNDARY_CONFIG_MOUNT_PATH.to_string(), + ]) ); assert_eq!( create_body @@ -2429,27 +2435,14 @@ fn build_container_create_body_replaces_inherited_cmd_with_workspace_arg() { ); assert_eq!( host_config.security_opt.as_ref(), - Some(&vec!["apparmor=unconfined".to_string()]) - ); - assert_eq!( - host_config.network_mode.as_deref(), - Some(DEFAULT_DOCKER_NETWORK_NAME) - ); - assert_eq!( - host_config.extra_hosts.as_ref(), Some(&vec![ - "host.docker.internal:172.18.0.1".to_string(), - "host.openshell.internal:172.18.0.1".to_string() + "no-new-privileges:true".to_string(), + "apparmor=unconfined".to_string(), ]) ); - assert_eq!( - create_body - .networking_config - .as_ref() - .and_then(|config| config.endpoints_config.as_ref()) - .and_then(|endpoints| endpoints.get(DEFAULT_DOCKER_NETWORK_NAME)), - Some(&EndpointSettings::default()) - ); + assert_eq!(host_config.network_mode.as_deref(), Some("none")); + assert_eq!(host_config.extra_hosts, None); + assert!(create_body.networking_config.is_none()); } #[test] @@ -2624,24 +2617,22 @@ fn validate_sandbox_rejects_template_errors_before_device_config() { } #[test] -fn validate_sandbox_auth_requires_gateway_token() { +fn validate_sandbox_auth_requires_launch_authentication() { let mut sandbox = test_sandbox(); - sandbox.spec.as_mut().unwrap().sandbox_token.clear(); + sandbox.spec.as_mut().unwrap().launch_authentication.clear(); let err = DockerComputeDriver::validate_sandbox_auth(&sandbox).unwrap_err(); assert_eq!(err.code(), tonic::Code::FailedPrecondition); assert_eq!( err.message(), - "docker sandboxes require gateway JWT auth; configure [openshell.gateway.gateway_jwt]" + "docker sandboxes require launch-scoped gateway authentication" ); } #[test] -fn validate_sandbox_auth_accepts_gateway_token() { - let mut sandbox = test_sandbox(); - sandbox.spec.as_mut().unwrap().sandbox_token = "secret.jwt.value".to_string(); - +fn validate_sandbox_auth_accepts_launch_authentication() { + let sandbox = test_sandbox(); DockerComputeDriver::validate_sandbox_auth(&sandbox).unwrap(); } @@ -2898,23 +2889,31 @@ fn require_sandbox_identifier_rejects_when_id_and_name_are_empty() { } #[test] -fn build_container_create_body_uses_bridge_network() { +fn build_container_create_body_disables_docker_networking() { let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); let host_config = create_body.host_config.expect("host_config is populated"); assert_eq!( host_config.network_mode, - Some(DEFAULT_DOCKER_NETWORK_NAME.to_string()), - "sandbox should join the driver-managed bridge network" + Some("none".to_string()), + "the sandbox must not receive direct Docker networking" ); + assert_eq!(host_config.extra_hosts, None); + assert_eq!(host_config.dns, Some(vec!["127.0.0.53".to_string()])); +} + +#[test] +fn build_container_create_body_limits_writable_runtime_storage_to_supervisor_ca() { + let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); + let host_config = create_body.host_config.expect("host_config is populated"); + let tmpfs = host_config.tmpfs.expect("sandbox tmpfs is populated"); + + assert_eq!(tmpfs.len(), 1); assert_eq!( - host_config.extra_hosts, - Some(vec![ - "host.docker.internal:172.18.0.1".to_string(), - "host.openshell.internal:172.18.0.1".to_string() - ]), - "sandbox should expose stable host aliases for gateway callbacks" + tmpfs.get(openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_DIR), + Some(&"rw,noexec,nosuid,nodev,size=1m,uid=1000,gid=1000,mode=0755".to_string()) ); + assert!(!tmpfs.contains_key("/run")); } #[test] @@ -2923,10 +2922,10 @@ fn build_container_create_body_uses_runtime_namespace_label() { // runtime config, not from `DriverSandbox.namespace`. The gateway // does not populate `DriverSandbox.namespace`, so a container created // with that empty value would not match subsequent list/get/find - // queries (which filter on `config.sandbox_label`), leaking + // queries (which filter on `config.sandbox_namespace`), leaking // sandboxes that the driver itself cannot observe. let mut config = runtime_config(); - config.sandbox_label = "tenant-a".to_string(); + config.sandbox_namespace = "tenant-a".to_string(); let mut sandbox = test_sandbox(); sandbox.namespace = "ignored-by-driver".to_string(); @@ -3107,16 +3106,20 @@ fn pending_sandbox_snapshot_uses_docker_namespace_and_starting_condition() { assert_eq!(snapshot.name, "demo"); assert_eq!(snapshot.namespace, "docker-dev"); assert!(snapshot.spec.is_none()); - let pending = pending_map(&[&snapshot]); + let pending = HashMap::from([( + snapshot.id.clone(), + PendingSandboxRecord { + sandbox: snapshot.clone(), + task: None, + }, + )]); assert_eq!( - resolve_pending_id(&pending, "sbx-123", "") - .unwrap() - .as_deref(), - Some("sbx-123") + pending_sandbox_record_id(&pending, "sbx-123", "wrong-name").unwrap(), + Some("sbx-123".to_string()) ); assert_eq!( - resolve_pending_id(&pending, "", "demo").unwrap().as_deref(), - Some("sbx-123") + pending_sandbox_record_id(&pending, "", "demo").unwrap(), + Some("sbx-123".to_string()) ); let status = snapshot.status.expect("status"); @@ -3130,13 +3133,67 @@ fn pending_sandbox_snapshot_uses_docker_namespace_and_starting_condition() { } #[test] -fn validate_linux_elf_binary_rejects_non_elf_files() { - let tempdir = TempDir::new().unwrap(); - let path = tempdir.path().join("openshell-sandbox"); - fs::write(&path, b"not-elf").unwrap(); +fn pending_lookup_is_id_authoritative_and_rejects_ambiguous_names() { + let mut alpha = test_sandbox(); + alpha.id = "sbx-alpha".to_string(); + alpha.workspace = "workspace-alpha".to_string(); + let mut beta = alpha.clone(); + beta.id = "sbx-beta".to_string(); + beta.workspace = "workspace-beta".to_string(); + let pending = [alpha, beta] + .into_iter() + .map(|sandbox| { + ( + sandbox.id.clone(), + PendingSandboxRecord { + sandbox, + task: None, + }, + ) + }) + .collect(); + + assert_eq!( + pending_sandbox_record_id(&pending, "sbx-alpha", "demo").unwrap(), + Some("sbx-alpha".to_string()) + ); + assert!(pending_sandbox_record_id(&pending, "", "demo").is_err()); +} + +#[test] +fn workload_mounts_only_the_shared_channel_volume() { + let config = runtime_config(); + let sandbox = test_sandbox(); + let identity = ResolvedWorkloadIdentity::new( + 65_534, + 65_534, + Vec::new(), + "65534:65534".to_string(), + "sha256:immutable".to_string(), + ) + .unwrap(); + let body = build_container_create_body_for_image( + &sandbox, + &config, + &DockerSandboxDriverConfig::default(), + None, + &DockerImageMetadata { + id: "sha256:immutable".to_string(), + user: "65534:65534".to_string(), + working_dir: "/sandbox".to_string(), + volumes: Vec::new(), + }, + &identity, + ) + .unwrap(); + let mounts = body.host_config.unwrap().mounts.unwrap(); + let sources = mounts + .iter() + .filter_map(|mount| mount.source.as_deref()) + .collect::>(); - let err = validate_linux_elf_binary(&path).unwrap_err(); - assert!(err.contains("Linux ELF executable")); + assert!(sources.contains(&docker_channel_volume_name(&sandbox, &config).as_str())); + assert!(!sources.contains(&docker_supervisor_volume_name(&sandbox, &config).as_str())); } #[test] @@ -3154,22 +3211,6 @@ fn docker_guest_tls_paths_require_all_files_for_https() { assert!(err.to_string().contains("guest_tls_cert")); } -#[test] -fn linux_supervisor_candidates_follow_daemon_arch() { - assert_eq!( - linux_supervisor_candidates("amd64"), - vec![PathBuf::from( - "target/x86_64-unknown-linux-gnu/release/openshell-sandbox", - )] - ); - assert_eq!( - linux_supervisor_candidates("arm64"), - vec![PathBuf::from( - "target/aarch64-unknown-linux-gnu/release/openshell-sandbox", - )] - ); -} - #[test] fn container_name_preserves_id_suffix_for_long_names() { // Names up to 253 chars are permitted by the gRPC layer. The id @@ -3256,32 +3297,6 @@ fn docker_guest_tls_paths_allows_plain_http_without_tls_flags() { assert!(result.is_none()); } -#[test] -fn docker_automatic_tls_detection_is_fail_closed_for_partial_bundles() { - for mask in 0_u8..8 { - let config = DockerComputeConfig { - guest_tls_ca: (mask & 1 != 0).then(|| PathBuf::from("/tmp/ca.pem")), - guest_tls_cert: (mask & 2 != 0).then(|| PathBuf::from("/tmp/cert.pem")), - guest_tls_key: (mask & 4 != 0).then(|| PathBuf::from("/tmp/key.pem")), - ..Default::default() - }; - assert_eq!( - docker_guest_tls_configured(&config), - mask != 0, - "TLS presence mask {mask:03b}" - ); - - if mask != 0 && mask != 7 { - let mut inferred = config; - inferred.grpc_endpoint = "https://host.openshell.internal:8080".to_string(); - assert!( - docker_guest_tls_paths(&inferred).is_err(), - "partial TLS presence mask {mask:03b} must fail" - ); - } - } -} - #[test] fn default_docker_supervisor_image_uses_nvidia_ghcr_repo() { let image = openshell_core::config::default_supervisor_image(); @@ -3291,36 +3306,6 @@ fn default_docker_supervisor_image_uses_nvidia_ghcr_repo() { ); } -#[test] -fn configured_supervisor_image_takes_precedence_over_local_binaries() { - let tempdir = TempDir::new().unwrap(); - let bin_dir = tempdir.path().join("bin"); - fs::create_dir_all(&bin_dir).unwrap(); - let current_exe = bin_dir.join("openshell-gateway"); - let sibling = bin_dir.join("openshell-sandbox"); - fs::write(¤t_exe, b"gateway").unwrap(); - fs::write(&sibling, b"\x7fELFsibling").unwrap(); - - let local_build = tempdir.path().join("target/openshell-sandbox"); - fs::create_dir_all(local_build.parent().unwrap()).unwrap(); - fs::write(&local_build, b"\x7fELFlocal").unwrap(); - - let source = resolve_supervisor_bin_source( - &DockerComputeConfig { - supervisor_image: Some("example.com/openshell/supervisor:test".to_string()), - ..Default::default() - }, - Some(¤t_exe), - &[local_build], - ) - .unwrap(); - - assert_eq!( - source, - SupervisorBinSource::Image("example.com/openshell/supervisor:test".to_string()) - ); -} - #[test] fn docker_supervisor_image_tag_prefers_explicit_build_tags() { use openshell_core::config::resolve_supervisor_image_tag; @@ -3365,63 +3350,6 @@ fn docker_supervisor_image_refreshes_mutable_tags_only() { )); } -#[test] -fn supervisor_cache_path_namespaces_by_digest_under_openshell_data_dir() { - let base = PathBuf::from("/var/cache/share"); - let path = supervisor_cache_path_with_base( - &base, - "docker-supervisor", - "sha256:abc123deadbeef0123456789cafe0123456789fe", - ); - - assert_eq!( - path, - PathBuf::from( - "/var/cache/share/openshell/docker-supervisor/sha256-abc123deadbeef0123456789cafe0123456789fe/openshell-sandbox", - ), - ); -} - -#[test] -fn supervisor_cache_path_isolates_different_digests() { - let base = PathBuf::from("/data"); - let left = supervisor_cache_path_with_base(&base, "docker-supervisor", "sha256:aaaaaaaa"); - let right = supervisor_cache_path_with_base(&base, "docker-supervisor", "sha256:bbbbbbbb"); - assert_ne!( - left.parent().unwrap(), - right.parent().unwrap(), - "digest-keyed directories must differ so rollouts are isolated", - ); -} - -#[test] -fn write_cache_binary_atomic_materializes_file_with_executable_mode() { - let tempdir = TempDir::new().unwrap(); - let target = tempdir.path().join("nested").join("openshell-sandbox"); - fs::create_dir_all(target.parent().unwrap()).unwrap(); - - write_cache_binary_atomic(&target, b"\x7fELFpayload").unwrap(); - - assert!(target.is_file()); - assert_eq!(fs::read(&target).unwrap(), b"\x7fELFpayload"); - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt; - let mode = fs::metadata(&target).unwrap().permissions().mode() & 0o777; - assert_eq!(mode, 0o755, "expected 0755, got {mode:04o}"); - } -} - -#[test] -fn write_cache_binary_atomic_overwrites_existing_file() { - let tempdir = TempDir::new().unwrap(); - let target = tempdir.path().join("openshell-sandbox"); - fs::write(&target, b"stale").unwrap(); - - write_cache_binary_atomic(&target, b"\x7fELFfresh").unwrap(); - assert_eq!(fs::read(&target).unwrap(), b"\x7fELFfresh"); -} - #[test] fn temp_extract_container_names_are_unique_per_call() { let first = temp_extract_container_name(); @@ -3494,6 +3422,13 @@ fn lifecycle_fence_rejects_polled_exit_from_before_restart() { fences.finish_start("sandbox-1"); assert!(!fences.start_in_progress("sandbox-1")); + fences.request_stop("sandbox-1", "demo"); + assert!(fences.stop_requested("sandbox-1", "")); + assert!(fences.stop_requested("", "demo")); + fences.clear_stop("sandbox-1", "demo"); + assert!(!fences.stop_requested("sandbox-1", "demo")); + fences.request_stop("sandbox-1", "demo"); + fences.record_previous_exit("sandbox-1", Some("2026-08-12T16:39:13Z")); assert_eq!( fences.previous_exit("sandbox-1").as_deref(), @@ -3528,8 +3463,10 @@ fn lifecycle_fence_rejects_polled_exit_from_before_restart() { Some(&new_exit), )); - fences.remove("sandbox-1"); + fences.remove("sandbox-1", "demo"); assert!(fences.previous_exit("sandbox-1").is_none()); + assert!(!fences.stop_requested("sandbox-1", "")); + assert!(!fences.stop_requested("", "demo")); } fn exited_sandbox_with_ready_reason(reason: &str) -> DriverSandbox { @@ -3551,6 +3488,7 @@ fn exited_sandbox_with_ready_reason(reason: &str) -> DriverSandbox { last_transition_time: String::new(), }], deleting: false, + ..Default::default() }), workspace: String::new(), } @@ -3565,15 +3503,6 @@ fn ready_reason(sandbox: &DriverSandbox) -> &str { .expect("Ready condition present") } -fn ready_message(sandbox: &DriverSandbox) -> &str { - sandbox - .status - .as_ref() - .and_then(|status| status.conditions.iter().find(|c| c.r#type == "Ready")) - .map(|c| c.message.as_str()) - .expect("Ready condition present") -} - #[test] fn docker_signal_kill_reclassified_as_runtime_restart() { // 137 (128+SIGKILL) and 143 (128+SIGTERM) mark an external termination — @@ -3611,24 +3540,6 @@ fn docker_ordinary_exit_stays_terminal() { assert_eq!(ready_reason(&sandbox), CONDITION_EXITED); } -#[test] -fn docker_workspace_validation_exit_is_reported_explicitly() { - let mut sandbox = exited_sandbox_with_ready_reason(CONDITION_EXITED); - let state = ContainerState { - status: Some(ContainerStateStatusEnum::EXITED), - exit_code: Some(i64::from(SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED)), - ..Default::default() - }; - - apply_docker_exit_classification(&mut sandbox, &state); - - assert_eq!( - ready_reason(&sandbox), - CONDITION_WORKSPACE_VALIDATION_FAILED - ); - assert!(ready_message(&sandbox).contains("WorkingDir")); -} - #[test] fn docker_oom_kill_stays_terminal_despite_137() { // An OOM kill reports exit 137 but must NOT be treated as a recoverable @@ -3644,406 +3555,57 @@ fn docker_oom_kill_stays_terminal_despite_137() { assert_eq!(ready_reason(&sandbox), CONDITION_EXITED); } -/// Minimal pending-map entry. Only the identity fields matter for lookup -/// resolution, so the spec and status are left empty on purpose. -fn pending_sandbox(id: &str, name: &str, workspace: &str) -> DriverSandbox { - DriverSandbox { - id: id.to_string(), - name: name.to_string(), - namespace: String::new(), - spec: None, - status: None, - workspace: workspace.to_string(), - } -} - -fn pending_map(sandboxes: &[&DriverSandbox]) -> HashMap { - sandboxes - .iter() - .map(|sandbox| { - ( - sandbox.id.clone(), - PendingSandboxRecord { - sandbox: (*sandbox).clone(), - task: None, - }, - ) - }) - .collect() -} - -async fn driver_with_pending(sandboxes: &[&DriverSandbox]) -> DockerComputeDriver { - let driver = test_driver_with_config(runtime_config()); - for sandbox in sandboxes { - driver - .reserve_pending_sandbox(sandbox) - .await - .expect("reserving a distinct sandbox must succeed"); - } - driver -} - -fn pending_ids(pending: &HashMap) -> Vec { - let mut ids: Vec = pending.keys().cloned().collect(); - ids.sort(); - ids -} - #[test] -fn resolve_pending_id_prefers_sandbox_id_over_sandbox_name() { - // The id is authoritative. A stale or mismatched name travelling in the - // same request must not change which record is resolved. - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let pending = pending_map(&[&alpha]); - - assert_eq!( - resolve_pending_id(&pending, "sbx-alpha", "stale-name") - .unwrap() - .as_deref(), - Some("sbx-alpha") - ); -} - -#[test] -fn resolve_pending_id_ignores_the_name_when_the_id_is_not_pending() { - // Regression for the `id OR name` match. `demo` exists in two workspaces: - // the beta copy is still provisioning, the alpha copy is already running. - // Deleting the alpha copy sends alpha's id plus the shared name. Matching - // on the name alone resolved to the beta record and evicted it, aborting - // an unrelated sandbox's provisioning task. - let beta = pending_sandbox("sbx-beta", "demo", "beta"); - let pending = pending_map(&[&beta]); - - assert_eq!( - resolve_pending_id(&pending, "sbx-alpha", "demo").unwrap(), - None - ); -} - -#[test] -fn resolve_pending_id_falls_back_to_the_name_when_no_id_is_supplied() { - // Direct driver callers may omit the id; a unique name still resolves. - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let pending = pending_map(&[&alpha]); - - assert_eq!( - resolve_pending_id(&pending, "", "demo").unwrap().as_deref(), - Some("sbx-alpha") - ); -} - -#[test] -fn resolve_pending_id_rejects_an_ambiguous_name_only_lookup() { - // Two pending sandboxes share a name across workspaces and the driver - // request carries no workspace. Picking either one would make the outcome - // depend on `HashMap` iteration order, so refuse instead. - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let beta = pending_sandbox("sbx-beta", "demo", "beta"); - let pending = pending_map(&[&alpha, &beta]); - - let err = resolve_pending_id(&pending, "", "demo") - .expect_err("an ambiguous name-only lookup must be rejected"); - assert_eq!(err.code(), tonic::Code::FailedPrecondition); -} - -#[test] -fn resolve_pending_id_returns_none_without_any_identifier() { - // `require_sandbox_identifier` rejects this upstream, but the resolver - // stays total so an empty request can never match an arbitrary record. - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let pending = pending_map(&[&alpha]); - - assert_eq!(resolve_pending_id(&pending, "", "").unwrap(), None); -} - -#[tokio::test] -async fn remove_pending_sandbox_by_id_keeps_a_same_named_sandbox_in_another_workspace() { - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let beta = pending_sandbox("sbx-beta", "demo", "beta"); - let driver = driver_with_pending(&[&alpha, &beta]).await; - - let removed = driver - .remove_pending_sandbox("sbx-alpha", "demo") - .await - .expect("an id-scoped removal must succeed") - .expect("the alpha record must be removed"); - - assert_eq!(removed.sandbox.id, "sbx-alpha"); - assert_eq!( - pending_ids(&driver.pending_snapshot_map().await), - ["sbx-beta"] - ); -} - -#[tokio::test] -async fn remove_pending_sandbox_by_a_unique_name_still_removes_the_record() { - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let driver = driver_with_pending(&[&alpha]).await; - - let removed = driver - .remove_pending_sandbox("", "demo") - .await - .expect("a unique name-only removal must succeed") - .expect("the alpha record must be removed"); - - assert_eq!(removed.sandbox.id, "sbx-alpha"); - assert!(driver.pending_snapshot_map().await.is_empty()); -} - -#[tokio::test] -async fn remove_pending_sandbox_rejects_an_ambiguous_name_and_keeps_both_records() { - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let beta = pending_sandbox("sbx-beta", "demo", "beta"); - let driver = driver_with_pending(&[&alpha, &beta]).await; - - let err = driver - .remove_pending_sandbox("", "demo") - .await - .map(|record| record.map(|record| record.sandbox.id)) - .expect_err("an ambiguous name-only removal must be rejected"); - - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert_eq!( - pending_ids(&driver.pending_snapshot_map().await), - ["sbx-alpha", "sbx-beta"] - ); -} - -#[tokio::test] -async fn pending_snapshot_by_id_ignores_a_same_named_sandbox_in_another_workspace() { - // `GetSandbox` falls through to the pending map when no container exists. - // Resolving by name there leaked another workspace's snapshot. - let beta = pending_sandbox("sbx-beta", "demo", "beta"); - let driver = driver_with_pending(&[&beta]).await; +fn concurrent_container_removal_is_idempotent() { + let removing = BollardError::DockerResponseServerError { + status_code: 409, + message: "removal of container abc123 is already in progress".to_string(), + }; + let other_conflict = BollardError::DockerResponseServerError { + status_code: 409, + message: "container abc123 is running".to_string(), + }; - assert!( - driver - .pending_snapshot("sbx-alpha", "demo") - .await - .expect("an id-scoped snapshot lookup must succeed") - .is_none() - ); + assert!(is_removal_in_progress_error(&removing)); + assert!(!is_removal_in_progress_error(&other_conflict)); } #[tokio::test] -async fn pending_snapshot_rejects_an_ambiguous_name_only_lookup() { - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let beta = pending_sandbox("sbx-beta", "demo", "beta"); - let driver = driver_with_pending(&[&alpha, &beta]).await; - - let err = driver - .pending_snapshot("", "demo") - .await - .expect_err("an ambiguous name-only snapshot lookup must be rejected"); - - assert_eq!(err.code(), tonic::Code::FailedPrecondition); -} +async fn missing_start_generation_is_adopted() { + let directory = TempDir::new().expect("create temporary directory"); + let path = directory.path().join(START_GENERATION_FILE); + let generation = openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-one".to_string(), + ) + .expect("valid generation"); -#[tokio::test] -async fn reserve_pending_sandbox_allows_the_same_name_in_a_different_workspace() { - // Sandbox names are unique per workspace, so this is a legitimate create. - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let beta = pending_sandbox("sbx-beta", "demo", "beta"); - let driver = driver_with_pending(&[&alpha]).await; - - driver - .reserve_pending_sandbox(&beta) + adopt_or_verify_docker_start_generation_path(&path, &generation) .await - .expect("a same-named sandbox in another workspace must be allowed"); + .expect("adopt missing marker"); assert_eq!( - pending_ids(&driver.pending_snapshot_map().await), - ["sbx-alpha", "sbx-beta"] + fs::read_to_string(&path).expect("read adopted marker"), + generation.as_str() ); -} - -#[tokio::test] -async fn reserve_pending_sandbox_rejects_a_duplicate_name_in_the_same_workspace() { - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let duplicate = pending_sandbox("sbx-other", "demo", "alpha"); - let driver = driver_with_pending(&[&alpha]).await; - - let err = driver - .reserve_pending_sandbox(&duplicate) + adopt_or_verify_docker_start_generation_path(&path, &generation) .await - .expect_err("a duplicate name within one workspace must be rejected"); - - assert_eq!(err.code(), tonic::Code::AlreadyExists); - assert_eq!( - pending_ids(&driver.pending_snapshot_map().await), - ["sbx-alpha"] - ); + .expect("accept adopted generation"); } #[tokio::test] -async fn reserve_pending_sandbox_rejects_a_duplicate_id() { - let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); - let duplicate = pending_sandbox("sbx-alpha", "other-name", "beta"); - let driver = driver_with_pending(&[&alpha]).await; +async fn different_start_generation_is_rejected() { + let directory = TempDir::new().expect("create temporary directory"); + let path = directory.path().join(START_GENERATION_FILE); + fs::write(&path, "generation-one").expect("write active marker"); + let requested = openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-two".to_string(), + ) + .expect("valid generation"); - let err = driver - .reserve_pending_sandbox(&duplicate) + let error = adopt_or_verify_docker_start_generation_path(&path, &requested) .await - .expect_err("a duplicate sandbox id must be rejected regardless of workspace"); - - assert_eq!(err.code(), tonic::Code::AlreadyExists); - assert_eq!( - pending_ids(&driver.pending_snapshot_map().await), - ["sbx-alpha"] - ); -} - -fn managed_container_labels( - namespace: &str, - sandbox_id: &str, - sandbox_name: &str, -) -> HashMap { - HashMap::from([ - ( - LABEL_MANAGED_BY.to_string(), - LABEL_MANAGED_BY_VALUE.to_string(), - ), - (LABEL_SANDBOX_NAMESPACE.to_string(), namespace.to_string()), - (LABEL_SANDBOX_ID.to_string(), sandbox_id.to_string()), - (LABEL_SANDBOX_NAME.to_string(), sandbox_name.to_string()), - ]) -} - -#[test] -fn managed_container_identity_matches_on_id_despite_a_stale_name() { - // Requiring the name to agree with an authoritative id dropped the match - // and made the driver report a live sandbox as absent, stranding the - // container and leaking its token file. - let labels = managed_container_labels("default", "sbx-alpha", "demo"); - - assert!(managed_container_identity_matches( - &labels, - "default", - "sbx-alpha", - "stale-name" - )); -} - -#[test] -fn managed_container_identity_rejects_a_name_match_when_the_id_differs() { - // The mirror of the pending-map fix: a shared name must not stand in for - // an id that explicitly disagrees. - let labels = managed_container_labels("default", "sbx-beta", "demo"); - - assert!(!managed_container_identity_matches( - &labels, - "default", - "sbx-alpha", - "demo" - )); -} - -#[test] -fn managed_container_identity_falls_back_to_the_name_without_an_id() { - let labels = managed_container_labels("default", "sbx-alpha", "demo"); - - assert!(managed_container_identity_matches( - &labels, "default", "", "demo" - )); - assert!(!managed_container_identity_matches( - &labels, "default", "", "other" - )); -} - -#[test] -fn managed_container_identity_matches_nothing_without_an_identifier() { - // The label filters degenerate to "every managed container in the - // namespace" when neither identifier is supplied, so the predicate must - // not wave the container through. - let labels = managed_container_labels("default", "sbx-alpha", "demo"); - - assert!(!managed_container_identity_matches( - &labels, "default", "", "" - )); -} - -#[test] -fn managed_container_identity_requires_the_configured_namespace() { - let labels = managed_container_labels("other-namespace", "sbx-alpha", "demo"); - - assert!(!managed_container_identity_matches( - &labels, - "default", - "sbx-alpha", - "demo" - )); -} - -#[tokio::test] -async fn delete_sandbox_reclaims_token_file_when_container_and_pending_are_gone() { - let state_dir = tempfile::tempdir().unwrap(); - let (endpoint, server) = fake_docker_with_no_containers().await; - - temp_env::async_with_vars([("XDG_STATE_HOME", Some(state_dir.path()))], async { - let config = runtime_config(); - let mut driver = test_driver_with_config(config.clone()); - driver.docker = Arc::new( - Docker::connect_with_http(&endpoint, 5, bollard::API_DEFAULT_VERSION).unwrap(), - ); - - // Arrange the leak: token on disk, container gone, `pending` empty. - let token = openshell_core::driver_utils::sandbox_token_path( - "docker-sandbox-tokens", - Some(&config.sandbox_label), - "sandbox-1", - ) - .unwrap(); - - fs::create_dir_all(token.parent().unwrap()).unwrap(); - fs::write(&token, "jwt\n").unwrap(); - - let deleted = driver.delete_sandbox_inner("sandbox-1", "").await.unwrap(); - assert!(!deleted, "nothing was removed, must not claim a deletion"); - assert!(!token.exists(), "token file must be reclaimed"); - }) - .await; - - server.abort(); -} - -#[tokio::test] -async fn delete_sandbox_by_name_only_leaves_the_namespace_directory_alone() { - // `DeleteSandbox` accepts a name without an id. With no id there is no - // token path to derive, so the cleanup must be a no-op: deriving a path - // from an empty id yields `/sandbox.jwt`, whose parent is the - // shared namespace directory. - let state_dir = tempfile::tempdir().unwrap(); - let (endpoint, server) = fake_docker_with_no_containers().await; - - temp_env::async_with_vars([("XDG_STATE_HOME", Some(state_dir.path()))], async { - let config = runtime_config(); - let mut driver = test_driver_with_config(config.clone()); - driver.docker = Arc::new( - Docker::connect_with_http(&endpoint, 5, bollard::API_DEFAULT_VERSION).unwrap(), - ); - - let namespace_dir = openshell_core::driver_utils::sandbox_token_path( - "docker-sandbox-tokens", - Some(&config.sandbox_label), - "sandbox-1", - ) - .unwrap() - .parent() - .and_then(Path::parent) - .unwrap() - .to_path_buf(); - fs::create_dir_all(&namespace_dir).unwrap(); - - let deleted = driver.delete_sandbox_inner("", "sandbox-1").await.unwrap(); - - assert!(!deleted, "nothing was removed, must not claim a deletion"); - assert!( - namespace_dir.is_dir(), - "namespace directory must survive a name-only delete: {}", - namespace_dir.display() - ); - }) - .await; + .expect_err("reject a different generation"); - server.abort(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!(error.message().contains("generation-one")); } diff --git a/crates/openshell-driver-kubernetes/Cargo.toml b/crates/openshell-driver-kubernetes/Cargo.toml index 3a3d843f1a..5b498a24d8 100644 --- a/crates/openshell-driver-kubernetes/Cargo.toml +++ b/crates/openshell-driver-kubernetes/Cargo.toml @@ -16,6 +16,8 @@ path = "src/main.rs" [dependencies] openshell-core = { path = "../openshell-core", default-features = false } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } openshell-otel = { path = "../openshell-otel" } openshell-policy = { path = "../openshell-policy" } @@ -37,6 +39,8 @@ tracing-subscriber = { workspace = true } thiserror = { workspace = true } miette = { workspace = true } notify = "8" +rand = { workspace = true } +rcgen = { workspace = true } [dev-dependencies] openshell-otel-test-support = { path = "../openshell-otel-test-support" } diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index b64bf0c6e4..c24cd9c9dd 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -45,9 +45,52 @@ not a tenant isolation boundary. ## Runtime Model The gateway stores platform state and delegates sandbox workload creation to -this driver. Kubernetes owns scheduling and pod lifecycle. The -`openshell-sandbox` supervisor inside each workload owns agent isolation, -credential injection, policy polling, logs, and the gateway relay. +this driver. Kubernetes owns scheduling and pod lifecycle. The workload Pod +stages the statically linked musl `openshell-sandbox` binary from +`sandbox_runtime_image`, while a directly managed Pod runs the dynamically +linked glibc `openshell-supervisor` from `supervisor_image`. + +The sandbox owns the agent process, applies Landlock and child seccomp filters, +identifies the binary behind each network syscall, and relays mediated streams +to the supervisor. The supervisor authenticates to the gateway with a JWT, +loads policy and provider state, performs destination and L7 authorization, and +opens upstream connections. The workload receives no gateway credential, +provider identity socket, or corporate-proxy credential. + +Both Pods run as the namespace-resolved non-root UID/GID with +`allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, and the runtime +default seccomp profile. The sandbox installs a nested seccomp user-notification +filter without requesting a capability in the Pod spec. Startup fails closed +when the runtime blocks the required seccomp or Landlock operations. + +The supervisor Pod has a direct, non-controller owner reference to the Sandbox +resource. This links its garbage-collection lifecycle to the sandbox without +competing with the Agent Sandbox controller for workload-Pod ownership. + +The driver creates one namespace-wide `NetworkPolicy` before it releases any +workload Pod. It selects every OpenShell workload, denies all workload egress, +and permits OpenShell supervisor Pods to reach the sandbox TLS port. The +authenticated Sandbox Protocol binds each connection to the exact sandbox and +supervisor Pod identities. Supervisors have normal egress for gateway, DNS, +and policy-approved upstream connections unless an operator policy restricts +them. Set +`sandbox_runtime.network_policy_enforced = true` only after verifying that the cluster +CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces. + +Each sandbox generation uses two immutable bootstrap Secrets. A trusted init +container stages the sandbox bootstrap into memory, and the sandbox removes it +before starting untrusted code. The other Secret is mounted only by the +supervisor. The TLS channel binds the namespace, Sandbox CR, workload Pod, +supervisor Pod, and shared network-policy identities. Stop deletes the workload +and supervisor Pods. Start rotates both Secrets and creates a new supervisor +Pod before releasing a new workload Pod. The shared network fence remains for +the lifetime of the namespace. + +Kubernetes policies are additive, and the API does not attest that the CNI +enforces them. Keep sandbox namespaces administrative: untrusted principals +must not create permissive policies, create Pods, read bootstrap Secrets, or +spoof the OpenShell role labels. Exact supervisor-to-sandbox authorization is +still enforced by TLS, JWT claims, session generation, and recorded Pod UIDs. ## Sandbox Resource @@ -95,73 +138,31 @@ mount attaches an existing PVC under `/sandbox`, which skips the default PVC. ## Credentials, TLS, and Relay -The driver injects gateway callback configuration, sandbox identity, TLS client -material, and the supervisor SSH socket path into the workload. The callback -endpoint is required because the sandbox namespace does not identify the -Gateway Service; Helm renders it from the release topology, while standalone -and raw TOML configurations must set it explicitly. Driver-owned values must -override image-provided environment variables. - -Sandbox pods run as `service_account_name` and keep -`automountServiceAccountToken: false`. The only Kubernetes token exposed to the -supervisor is an explicit, audience-bound projected token mounted at -`/var/run/secrets/openshell/token` for the one-shot `IssueSandboxToken` -bootstrap exchange. The Kubernetes driver authenticates that token through the -compute-driver protocol using its own `service_account_name` and workspace-mode -namespace policy; the gateway receives only the verified sandbox ID. - -The gateway uses the supervisor relay for connect, exec, and file sync. Sandbox -pods do not need direct external ingress for SSH. - -The driver forwards the canonical main-process specification to the process -supervisor and sets pod `restartPolicy: Never`. Main-process environment -overrides stay local to that child; the sidecar bootstrap retains the unmodified -provider environment used by later exec, editor, and SFTP sessions. +Both Pods set `automountServiceAccountToken: false`. The supervisor receives an +explicit audience-bound projected token for the one-shot `IssueSandboxToken` +exchange. The driver verifies that token and the gateway returns the +sandbox-scoped JWT used by the supervisor session. The sandbox Pod receives +neither token. + +The gateway uses the supervisor relay for connect, exec, logs, and file sync. +Sandbox Pods do not need direct external ingress for SSH. + +The driver sends the canonical main-process specification only to the +supervisor. The supervisor passes admitted launch state over the protected +channel. Provider environment updates apply to future exec sessions. ## Container Security Context -The default `combined` supervisor topology grants the sandbox agent container -the Linux capabilities the supervisor needs for namespace setup and process, -filesystem, and network policy enforcement. - -The `sidecar` supervisor topology moves pod-level network setup into a root init -container. In the default process/binary-aware mode, the long-lived network -sidecar runs as UID 0 with `allowPrivilegeEscalation: false`, drops default -Linux capabilities, and adds only `SYS_PTRACE` plus `DAC_READ_SEARCH` for -cross-UID workload `/proc` inspection. The agent container also runs as the -resolved sandbox UID/GID with `allowPrivilegeEscalation: false` and -`capabilities.drop: ["ALL"]`. -Set `sidecar.process_binary_aware_network_policy = false` to run the network -sidecar as the configured non-root `sidecar.proxy_uid`, omit the extra `/proc` -inspection capabilities, and enforce endpoint/L7 network policy without -matching `policy.binaries`. -In this mode OpenShell preserves gateway session and SSH behavior, but the -process supervisor does not perform root-to-sandbox privilege dropping or -supervisor identity mount isolation. It still applies Landlock filesystem policy -and child seccomp filters where the kernel/runtime supports them. Network -endpoint and L7 policy remain enforced by the network sidecar, and -sidecar pods use a shared process namespace so the network sidecar can resolve -process/binary identity through `/proc/`. - -Sidecar mode keeps gateway credentials in the network sidecar. The agent -container does not mount the projected service-account token used for sandbox -token bootstrap, does not mount the sandbox client TLS secret, and does not get -gateway callback environment variables. The process supervisor receives policy -and provider environment state from the sidecar over a local control socket in -the shared sidecar state volume. The sidecar accepts only the pre-workload -process-supervisor connection, authenticates its UID/GID/PID with peer -credentials, and removes the listener afterward. SSH relays use a Linux -abstract socket whose peer PID must match that authenticated supervisor. Both -supervisors exit if the control connection closes, coupling their container -restart lifecycle before a new authoritative client can be established. - -The driver uses the shared AppArmor model through `app_armor_profile`. -Supported values are `Unconfined`, `RuntimeDefault`, and -`Localhost/`; an empty or unset value omits -`securityContext.appArmorProfile`. Docker and Podman translate the same values -to OCI security options. Helm deployments default sandbox agent containers to -`Unconfined` because runtime/default AppArmor profiles can block the -supervisor's network namespace mount setup on AppArmor-enabled nodes. +The sandbox, trusted bootstrap init container, and supervisor request no added +Linux capability. They run as the same numeric non-root identity, disable +privilege escalation, drop all capabilities, and inherit `RuntimeDefault` +seccomp. The sandbox and agent must use the same complete UID, GID, and +supplementary-group identity because the capability-free sandbox cannot change +credentials after launch and must inspect its same-identity descendants. + +The workload Pod does not share host network, PID, IPC, or process namespaces. +The driver uses a scheduling gate to inspect the admitted Pod and bind its UID +into the bootstrap claims before kubelet starts it. ## GPU Support diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index 01244da066..f29ba79f5b 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -1,9 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -pub use openshell_core::AppArmorProfile; pub use openshell_core::DynamicStringAllowlist as OperatorNamespaceAllowlist; -use openshell_core::{ImagePullPolicy, config}; +use openshell_core::config; use serde::{Deserialize, Deserializer, Serialize}; use std::collections::BTreeMap; #[cfg(test)] @@ -11,88 +10,89 @@ use std::collections::BTreeSet; use std::path::Path; use std::str::FromStr; -/// Default gateway identity used in managed-mode namespace naming. -pub const DEFAULT_GATEWAY_ID: &str = "openshell"; - -/// Default Kubernetes namespace for sandbox resources. -pub const DEFAULT_K8S_NAMESPACE: &str = "openshell"; - -/// Default Kubernetes `ServiceAccount` assigned to sandbox pods. -pub const DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME: &str = "default"; - -/// Default storage size for the workspace PVC. -pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi"; - -/// Default non-root UID for relaxed Kubernetes network supervisor sidecars. -pub const DEFAULT_PROXY_UID: u32 = 1337; - -/// How the supervisor binary is delivered into sandbox pods. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub enum SupervisorSideloadMethod { - /// Mount the supervisor OCI image directly as a read-only volume - /// (requires Kubernetes >= v1.33 with the `ImageVolume` feature gate, - /// or >= v1.36 where it is GA). - #[default] - ImageVolume, - /// Copy the binary via an init container and emptyDir volume. - /// Works on all Kubernetes versions. - InitContainer, +/// Image pull policies accepted by the Kubernetes API. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum KubernetesImagePullPolicy { + #[serde(alias = "Always")] + Always, + #[serde(alias = "IfNotPresent")] + IfNotPresent, + #[serde(alias = "Never")] + Never, } -impl std::fmt::Display for SupervisorSideloadMethod { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { +impl KubernetesImagePullPolicy { + /// Return the spelling required by Kubernetes Pod specs. + #[must_use] + pub const fn as_kubernetes_str(self) -> &'static str { match self { - Self::ImageVolume => f.write_str("image-volume"), - Self::InitContainer => f.write_str("init-container"), + Self::Always => "Always", + Self::IfNotPresent => "IfNotPresent", + Self::Never => "Never", } } } -impl FromStr for SupervisorSideloadMethod { +impl FromStr for KubernetesImagePullPolicy { type Err = String; - fn from_str(s: &str) -> Result { - match s { - "image-volume" => Ok(Self::ImageVolume), - "init-container" => Ok(Self::InitContainer), + fn from_str(value: &str) -> Result { + match value { + "always" | "Always" => Ok(Self::Always), + "if_not_present" | "IfNotPresent" => Ok(Self::IfNotPresent), + "never" | "Never" => Ok(Self::Never), other => Err(format!( - "unknown supervisor sideload method '{other}'; expected 'image-volume' or 'init-container'" + "invalid Kubernetes image pull policy '{other}'; expected always, if_not_present, or never" )), } } } -/// How the supervisor is arranged inside Kubernetes sandbox pods. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub enum SupervisorTopology { - /// Run networking and process supervision in the agent container. - #[default] - Combined, - /// Run network supervision in a privileged sidecar and process supervision - /// as a low-capability wrapper in the agent container. - Sidecar, +/// Default gateway identity used in managed-mode namespace naming. +pub const DEFAULT_GATEWAY_ID: &str = "openshell"; + +/// Default Kubernetes namespace for sandbox resources. +pub const DEFAULT_K8S_NAMESPACE: &str = "openshell"; + +/// Default Kubernetes `ServiceAccount` assigned to sandbox pods. +pub const DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME: &str = "default"; + +/// Default storage size for the workspace PVC. +pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi"; + +/// Driver-owned requirements for the Kubernetes sandbox runtime. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(default, deny_unknown_fields)] +pub struct KubernetesSandboxRuntimeConfig { + /// Explicit operator assertion that the cluster CNI enforces + /// `networking.k8s.io/v1` `NetworkPolicy` for the sandbox namespaces. + pub network_policy_enforced: bool, + /// TCP port exposed by the workload boundary to its paired control pod. + pub boundary_port: u16, } -impl std::fmt::Display for SupervisorTopology { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - Self::Combined => f.write_str("combined"), - Self::Sidecar => f.write_str("sidecar"), +impl Default for KubernetesSandboxRuntimeConfig { + fn default() -> Self { + Self { + network_policy_enforced: false, + boundary_port: 5500, } } } -impl FromStr for SupervisorTopology { - type Err = String; - - fn from_str(s: &str) -> Result { - match s { - "combined" => Ok(Self::Combined), - "sidecar" => Ok(Self::Sidecar), - other => Err(format!("unknown topology '{other}'")), +impl KubernetesSandboxRuntimeConfig { + pub fn validate(&self) -> Result<(), String> { + if !self.network_policy_enforced { + return Err( + "sandbox_runtime.network_policy_enforced must be true after the operator has verified CNI NetworkPolicy enforcement" + .to_string(), + ); + } + if self.boundary_port < 1024 { + return Err("sandbox_runtime.boundary_port must be at least 1024".to_string()); } + Ok(()) } } @@ -138,62 +138,6 @@ impl FromStr for WorkspaceMode { } } -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default, deny_unknown_fields)] -pub struct KubernetesSidecarConfig { - /// UID used by relaxed long-running network sidecars in `sidecar` - /// topology. The network init container installs nftables rules that - /// exempt this UID, so it must not match the sandbox workload UID. - /// Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants - /// the requested `/proc` inspection capabilities into the effective set. - pub proxy_uid: u32, - /// Require process/binary-aware network policy enforcement in sidecar - /// topology. When disabled, the network sidecar runs as `proxy_uid`, - /// drops the extra `/proc` inspection permissions, and evaluates - /// endpoint/L7 policy without matching `policy.binaries`. - pub process_binary_aware_network_policy: bool, -} - -impl Default for KubernetesSidecarConfig { - fn default() -> Self { - Self { - proxy_uid: DEFAULT_PROXY_UID, - process_binary_aware_network_policy: true, - } - } -} - -impl KubernetesSidecarConfig { - pub fn validate_proxy_uid(&self) -> Result<(), String> { - if !(openshell_policy::MIN_SANDBOX_PROXY_UID..=openshell_policy::MAX_SANDBOX_UID) - .contains(&self.proxy_uid) - { - return Err(format!( - "sidecar.proxy_uid must be in range [{}, {}]", - openshell_policy::MIN_SANDBOX_PROXY_UID, - openshell_policy::MAX_SANDBOX_UID, - )); - } - Ok(()) - } -} - -fn deserialize_optional_app_armor_profile<'de, D>( - deserializer: D, -) -> Result, D::Error> -where - D: Deserializer<'de>, -{ - let value = Option::::deserialize(deserializer)?; - match value.as_deref() { - None | Some("") => Ok(None), - Some(value) => value - .parse::() - .map(Some) - .map_err(serde::de::Error::custom), - } -} - fn deserialize_provider_spiffe_workload_api_socket_path<'de, D>( deserializer: D, ) -> Result @@ -227,33 +171,33 @@ pub struct KubernetesComputeConfig { /// operator mode. Hot-reloaded on change. Delivered via `ConfigMap` volume mount. #[serde(default, skip_serializing_if = "Option::is_none")] pub operator_namespace_file: Option, - /// Kubernetes `ServiceAccount` assigned to sandbox pods and accepted by - /// the driver's `TokenReview` bootstrap authenticator. + /// Kubernetes `ServiceAccount` assigned to the workload and supervisor Pods. Automatic + /// token mounting is disabled; only the supervisor receives an explicit + /// audience-bound projected token accepted by the bootstrap authenticator. pub service_account_name: String, pub default_image: String, - /// Pull policy for sandbox images. Omit to use Kubernetes's image default. #[serde(default, skip_serializing_if = "Option::is_none")] - pub image_pull_policy: Option, + pub image_pull_policy: Option, /// Kubernetes `imagePullSecrets` names attached to sandbox pods. pub image_pull_secrets: Vec, /// Managed-mode SSH ingress isolation. When enabled, the driver creates a /// `NetworkPolicy` in each managed workspace namespace that permits TCP 2222 /// only from gateway pods matching this peer. pub managed_ssh_ingress: ManagedSshIngressConfig, - /// Image that provides the `openshell-sandbox` supervisor binary. - /// Mounted directly as an image volume, or copied via an init container, - /// depending on `supervisor_sideload_method`. + /// Image that provides the trusted `openshell-sandbox` bootstrap binary. + pub sandbox_runtime_image: String, + /// Kubernetes `imagePullPolicy` for the sandbox runtime image. + /// When omitted, Kubernetes selects its default. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub sandbox_runtime_image_pull_policy: Option, + /// Image that provides the trusted `openshell-supervisor` control binary. pub supervisor_image: String, - /// Pull policy for the supervisor image. Omit to use Kubernetes's image - /// default. + /// Kubernetes `imagePullPolicy` for the supervisor image. + /// When omitted, Kubernetes selects its default. #[serde(default, skip_serializing_if = "Option::is_none")] - pub supervisor_image_pull_policy: Option, - /// How the supervisor binary is delivered into sandbox pods. - pub supervisor_sideload_method: SupervisorSideloadMethod, - /// How the supervisor is arranged for Kubernetes sandbox pods. - pub topology: SupervisorTopology, - /// Sidecar-only settings used when `topology = "sidecar"`. - pub sidecar: KubernetesSidecarConfig, + pub supervisor_image_pull_policy: Option, + /// Cross-pod sandbox/supervisor settings. + pub sandbox_runtime: KubernetesSandboxRuntimeConfig, /// Corporate HTTP forward proxy used by the network supervisor for /// policy-approved TLS CONNECT egress. pub https_proxy: Option, @@ -278,14 +222,6 @@ pub struct KubernetesComputeConfig { pub client_tls_secret_name: String, pub host_gateway_ip: String, pub enable_user_namespaces: bool, - /// Kubernetes `AppArmor` profile requested for the sandbox agent container. - /// Empty/None omits the `appArmorProfile` field from sandbox pod specs. - #[serde( - default, - skip_serializing_if = "Option::is_none", - deserialize_with = "deserialize_optional_app_armor_profile" - )] - pub app_armor_profile: Option, pub workspace_default_storage_size: String, /// Kubernetes `StorageClass` name for the default workspace PVC. /// Empty string (default) = omit `storageClassName`, using the cluster's @@ -314,13 +250,10 @@ pub struct KubernetesComputeConfig { deserialize_with = "deserialize_provider_spiffe_workload_api_socket_path" )] pub provider_spiffe_workload_api_socket_path: String, - /// UID used for privilege-drop operations and workspace init container - /// ownership. The supervisor container always runs as UID 0 (root) to - /// create network namespaces and configure Landlock/seccomp; the - /// `sandbox_uid` is injected as the `SANDBOX_UID` environment variable so - /// the supervisor knows which UID to drop to for child processes. + /// Exact UID shared by `openshell-sandbox`, its agent children, the trusted + /// workspace/bootstrap init containers, and `openshell-supervisor`. /// When empty, the driver auto-detects from `OpenShift` SCC annotations on - /// the target namespace; if those are also absent, falls back to `1000`. + /// the target namespace; if those are also absent, falls back to `10001`. #[serde(default, skip_serializing_if = "Option::is_none")] pub sandbox_uid: Option, /// GID used alongside `sandbox_uid` for PVC init container operations. @@ -347,7 +280,7 @@ pub const MAX_SA_TOKEN_TTL_SECS: i64 = 86_400; /// Default sandbox UID used when neither config nor `OpenShift` SCC annotations /// provide a resolved value. -pub(crate) const DEFAULT_SANDBOX_UID: u32 = 1000; +pub(crate) const DEFAULT_SANDBOX_UID: u32 = 10001; /// The annotation key for the `OpenShift` `ServiceAccount` UID range. /// Format: `/` (e.g. `1000000000/10000`). @@ -367,16 +300,18 @@ impl Default for KubernetesComputeConfig { operator_namespace_file: None, service_account_name: DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME.to_string(), default_image: openshell_core::image::default_sandbox_image(), - // Omit the field so Kubernetes applies its own default (Always for - // `latest`, IfNotPresent otherwise). + // Default empty so the gateway omits `imagePullPolicy` from pod + // specs and Kubernetes applies its own default (Always for `latest`, + // IfNotPresent otherwise). `DEFAULT_IMAGE_PULL_POLICY` ("missing") + // is Podman vocabulary and is not a valid Kubernetes value. image_pull_policy: None, image_pull_secrets: Vec::new(), managed_ssh_ingress: ManagedSshIngressConfig::default(), + sandbox_runtime_image: config::default_sandbox_runtime_image(), + sandbox_runtime_image_pull_policy: None, supervisor_image: config::default_supervisor_image(), supervisor_image_pull_policy: None, - supervisor_sideload_method: SupervisorSideloadMethod::default(), - topology: SupervisorTopology::default(), - sidecar: KubernetesSidecarConfig::default(), + sandbox_runtime: KubernetesSandboxRuntimeConfig::default(), https_proxy: None, no_proxy: None, proxy_auth_secret_name: None, @@ -388,7 +323,6 @@ impl Default for KubernetesComputeConfig { client_tls_secret_name: String::new(), host_gateway_ip: String::new(), enable_user_namespaces: false, - app_armor_profile: None, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(), workspace_storage_class: String::new(), default_runtime_class_name: String::new(), @@ -407,7 +341,6 @@ impl KubernetesComputeConfig { self.validate_provider_spiffe_workload_api_socket_path()?; self.validate_sandbox_identity_config()?; self.validate_proxy_uid()?; - self.validate_image_pull_policies()?; self.validate_upstream_proxy_config() } @@ -439,57 +372,20 @@ impl KubernetesComputeConfig { } pub fn validate_proxy_uid(&self) -> Result<(), String> { - self.sidecar.validate_proxy_uid() - } - - /// Reject pull policies Kubernetes cannot express before creating pods. - pub fn validate_image_pull_policies(&self) -> Result<(), String> { - for (field, policy) in [ - ("image_pull_policy", self.image_pull_policy), - ( - "supervisor_image_pull_policy", - self.supervisor_image_pull_policy, - ), - ] { - if policy == Some(ImagePullPolicy::Newer) { - return Err(format!( - "{field} = \"newer\" is supported only by the Podman compute driver" - )); - } - } - Ok(()) - } - - /// Translate a shared policy to Kubernetes's API vocabulary. - pub fn image_pull_policy_value(policy: ImagePullPolicy) -> Result<&'static str, String> { - match policy { - ImagePullPolicy::Always => Ok("Always"), - ImagePullPolicy::IfNotPresent => Ok("IfNotPresent"), - ImagePullPolicy::Never => Ok("Never"), - ImagePullPolicy::Newer => Err( - "image pull policy 'newer' is supported only by the Podman compute driver" - .to_string(), - ), - } + self.sandbox_runtime.validate() } /// Validate the operator-owned corporate upstream proxy configuration. pub fn validate_upstream_proxy_config(&self) -> Result<(), String> { use openshell_core::driver_utils::{UpstreamProxyUrlError, parse_upstream_proxy_url}; - let proxy_addr = self - .https_proxy - .as_deref() - .map(|url| { - parse_upstream_proxy_url(url).map_err(|err| match err { - UpstreamProxyUrlError::Empty => { - "https_proxy must not be empty when set".to_string() - } - UpstreamProxyUrlError::InlineCredentials => "https_proxy must not embed credentials in the URL; supply them through proxy_auth_secret_name and proxy_auth_secret_key".to_string(), - err => format!("https_proxy {err}"), - }) - }) - .transpose()?; + if let Some(url) = &self.https_proxy { + parse_upstream_proxy_url(url).map_err(|err| match err { + UpstreamProxyUrlError::Empty => "https_proxy must not be empty when set".to_string(), + UpstreamProxyUrlError::InlineCredentials => "https_proxy must not embed credentials in the URL; supply them through proxy_auth_secret_name and proxy_auth_secret_key".to_string(), + err => format!("https_proxy {err}"), + })?; + } if let Some(list) = self.no_proxy.as_deref() { if list.trim().is_empty() { @@ -548,17 +444,9 @@ impl KubernetesComputeConfig { .to_string(), ); } - if proxy_addr.as_ref().is_some_and(|proxy| !proxy.secure) - && self.proxy_auth_allow_insecure != Some(true) - { + if self.proxy_auth_allow_insecure != Some(true) { return Err("proxy credentials use cleartext Basic auth over the connection to the http:// proxy; set proxy_auth_allow_insecure = true to accept that exposure, or remove the credential Secret".to_string()); } - if self.topology == SupervisorTopology::Combined { - return Err( - "proxy credential Secrets require topology = \"sidecar\"; combined topology shares the credential mount with the workload and fsGroup can make it readable by the sandbox user" - .to_string(), - ); - } } _ => { return Err( @@ -583,7 +471,7 @@ impl KubernetesComputeConfig { /// 2. `OpenShift` SCC namespace annotations (`sa.scc.uid-range`, /// `sa.scc.supplemental-groups`) — passed in as the optional /// `namespace_annotations` map - /// 3. Fallback defaults: UID=`1000`, GID=UID + /// 3. Fallback defaults: UID=`10001`, GID=UID pub fn resolve_sandbox_uid( &self, namespace_annotations: Option<&BTreeMap>, @@ -870,102 +758,76 @@ mod tests { use std::collections::BTreeMap as HashMap; #[test] - fn default_workspace_storage_size_is_2gi() { - let cfg = KubernetesComputeConfig::default(); - assert_eq!( - cfg.workspace_default_storage_size, - DEFAULT_WORKSPACE_STORAGE_SIZE - ); + fn image_pull_policy_accepts_config_and_kubernetes_spellings() { + for (value, expected) in [ + ("always", KubernetesImagePullPolicy::Always), + ("Always", KubernetesImagePullPolicy::Always), + ("if_not_present", KubernetesImagePullPolicy::IfNotPresent), + ("IfNotPresent", KubernetesImagePullPolicy::IfNotPresent), + ("never", KubernetesImagePullPolicy::Never), + ("Never", KubernetesImagePullPolicy::Never), + ] { + assert_eq!(value.parse(), Ok(expected)); + } + assert!("newer".parse::().is_err()); + assert!("sometimes".parse::().is_err()); } #[test] - fn default_workspace_storage_class_is_empty() { - let cfg = KubernetesComputeConfig::default(); - assert!(cfg.workspace_storage_class.is_empty()); + fn image_pull_policy_fields_reject_unsupported_values() { + for field in [ + "image_pull_policy", + "sandbox_runtime_image_pull_policy", + "supervisor_image_pull_policy", + ] { + let input = format!("{field} = \"sometimes\""); + assert!(toml::from_str::(&input).is_err()); + } } #[test] - fn default_topology_is_combined() { - let cfg = KubernetesComputeConfig::default(); - assert_eq!(cfg.topology, SupervisorTopology::Combined); - assert_eq!(cfg.topology.to_string(), "combined"); + fn published_kubernetes_example_is_valid_toml() { + let docs = include_str!("../../../docs/reference/gateway-config.mdx"); + let section = docs + .split_once("### Kubernetes") + .expect("Kubernetes documentation section") + .1; + let example = section + .split_once("```toml") + .expect("Kubernetes TOML fence") + .1 + .split_once("```") + .expect("closed Kubernetes TOML fence") + .0; + toml::from_str::(example).expect("valid Kubernetes gateway TOML example"); } #[test] - fn default_proxy_uid_is_dedicated_non_root_uid() { + fn default_workspace_storage_size_is_2gi() { let cfg = KubernetesComputeConfig::default(); - assert_eq!(cfg.sidecar.proxy_uid, DEFAULT_PROXY_UID); + assert_eq!( + cfg.workspace_default_storage_size, + DEFAULT_WORKSPACE_STORAGE_SIZE + ); } #[test] - fn default_sidecar_requires_process_binary_aware_network_policy() { + fn default_workspace_storage_class_is_empty() { let cfg = KubernetesComputeConfig::default(); - assert!(cfg.sidecar.process_binary_aware_network_policy); - } - - #[test] - fn image_pull_policy_uses_shared_canonical_values() { - let cfg: KubernetesComputeConfig = serde_json::from_value(serde_json::json!({ - "image_pull_policy": "if_not_present", - "supervisor_image_pull_policy": "never" - })) - .unwrap(); - assert_eq!(cfg.image_pull_policy, Some(ImagePullPolicy::IfNotPresent)); - assert_eq!( - cfg.supervisor_image_pull_policy, - Some(ImagePullPolicy::Never) - ); - - for (policy, expected) in [ - (ImagePullPolicy::Always, "Always"), - (ImagePullPolicy::IfNotPresent, "IfNotPresent"), - (ImagePullPolicy::Never, "Never"), - ] { - assert_eq!( - KubernetesComputeConfig::image_pull_policy_value(policy).unwrap(), - expected - ); - } + assert!(cfg.workspace_storage_class.is_empty()); } #[test] - fn image_pull_policy_rejects_newer_for_sandbox_and_supervisor_images() { + fn sandbox_runtime_requires_network_policy_enforcement_acknowledgement() { + let mut cfg = KubernetesComputeConfig::default(); assert!( - KubernetesComputeConfig::image_pull_policy_value(ImagePullPolicy::Newer) + cfg.validate_proxy_uid() .unwrap_err() - .contains("supported only by the Podman") + .contains("network_policy_enforced") ); - for (sandbox, supervisor) in [ - (Some(ImagePullPolicy::Newer), None), - (None, Some(ImagePullPolicy::Newer)), - ] { - let cfg = KubernetesComputeConfig { - image_pull_policy: sandbox, - supervisor_image_pull_policy: supervisor, - ..KubernetesComputeConfig::default() - }; - let error = cfg.validate_image_pull_policies().unwrap_err(); - assert!(error.contains("supported only by the Podman")); - } - } - - #[test] - fn serde_override_topology_sidecar() { - let json = serde_json::json!({ - "topology": "sidecar" - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.topology, SupervisorTopology::Sidecar); - } - - #[test] - fn serde_override_topology_combined() { - let json = serde_json::json!({ - "topology": "combined" - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.topology, SupervisorTopology::Combined); + cfg.sandbox_runtime.network_policy_enforced = true; + cfg.validate_proxy_uid().unwrap(); } #[test] @@ -979,64 +841,6 @@ mod tests { assert!(err.to_string().contains("unknown field")); } - #[test] - fn serde_override_sidecar_process_binary_aware_network_policy_nested() { - let json = serde_json::json!({ - "sidecar": { - "process_binary_aware_network_policy": false - } - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert!(!cfg.sidecar.process_binary_aware_network_policy); - } - - #[test] - fn serde_override_sidecar_proxy_uid_nested() { - let json = serde_json::json!({ - "sidecar": { - "proxy_uid": 2000 - } - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.sidecar.proxy_uid, 2000); - cfg.validate_proxy_uid().unwrap(); - } - - #[test] - fn validate_proxy_uid_rejects_privileged_uid() { - let cfg = KubernetesComputeConfig { - sidecar: KubernetesSidecarConfig { - proxy_uid: 999, - ..KubernetesSidecarConfig::default() - }, - ..KubernetesComputeConfig::default() - }; - let err = cfg.validate_proxy_uid().unwrap_err(); - assert!(err.contains("proxy_uid")); - } - - #[test] - fn serde_rejects_invalid_topology() { - let json = serde_json::json!({ - "topology": "unsupported" - }); - let err = serde_json::from_value::(json).unwrap_err(); - assert!(err.to_string().contains("unknown variant")); - } - - #[test] - fn serde_rejects_removed_topology_alias_field() { - let mut json = serde_json::Map::new(); - json.insert( - ["supervisor", "topology"].join("_"), - serde_json::json!("sidecar"), - ); - let err = - serde_json::from_value::(serde_json::Value::Object(json)) - .unwrap_err(); - assert!(err.to_string().contains("unknown field")); - } - #[test] fn serde_rejects_removed_flat_sidecar_fields() { for json in [ @@ -1108,53 +912,6 @@ mod tests { assert!(cfg.default_runtime_class_name.is_empty()); } - #[test] - fn default_app_armor_profile_is_none() { - let cfg = KubernetesComputeConfig::default(); - assert!(cfg.app_armor_profile.is_none()); - } - - #[test] - fn serde_override_app_armor_profile_unconfined() { - let json = serde_json::json!({ - "app_armor_profile": "Unconfined" - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.app_armor_profile, Some(AppArmorProfile::Unconfined)); - } - - #[test] - fn serde_override_app_armor_profile_runtime_default() { - let json = serde_json::json!({ - "app_armor_profile": "RuntimeDefault" - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.app_armor_profile, Some(AppArmorProfile::RuntimeDefault)); - } - - #[test] - fn serde_override_app_armor_profile_localhost() { - let json = serde_json::json!({ - "app_armor_profile": "Localhost/openshell-supervisor" - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!( - cfg.app_armor_profile, - Some(AppArmorProfile::Localhost( - "openshell-supervisor".to_string() - )) - ); - } - - #[test] - fn serde_empty_app_armor_profile_disables_field() { - let json = serde_json::json!({ - "app_armor_profile": "" - }); - let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); - assert_eq!(cfg.app_armor_profile, None); - } - #[test] fn serde_accepts_absolute_provider_spiffe_socket_path() { let json = serde_json::json!({ @@ -1184,15 +941,6 @@ mod tests { } } - #[test] - fn serde_rejects_invalid_app_armor_profile() { - let json = serde_json::json!({ - "app_armor_profile": "runtime/default" - }); - let err = serde_json::from_value::(json).unwrap_err(); - assert!(err.to_string().contains("unknown AppArmor profile")); - } - #[test] fn serde_override_image_pull_secrets() { let json = serde_json::json!({ @@ -1399,7 +1147,6 @@ mod tests { #[test] fn upstream_proxy_config_accepts_secret_credentials_with_acknowledgement() { let cfg = KubernetesComputeConfig { - topology: SupervisorTopology::Sidecar, https_proxy: Some("http://proxy.corp.example:8080".to_string()), proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()), proxy_auth_secret_key: Some("credentials".to_string()), @@ -1410,22 +1157,9 @@ mod tests { } #[test] - fn upstream_proxy_config_accepts_tls_protected_secret_credentials() { - let cfg = KubernetesComputeConfig { - topology: SupervisorTopology::Sidecar, - https_proxy: Some("https://proxy.corp.example:8443".to_string()), - proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()), - proxy_auth_secret_key: Some("credentials".to_string()), - ..KubernetesComputeConfig::default() - }; - assert!(cfg.validate_upstream_proxy_config().is_ok()); - } - - #[test] - fn toml_deserializes_sidecar_upstream_proxy_settings() { + fn toml_deserializes_upstream_proxy_settings() { let cfg: KubernetesComputeConfig = toml::from_str( r#" - topology = "sidecar" https_proxy = "http://proxy.corp.example:8080" no_proxy = ".svc.cluster.local,10.96.0.0/12" proxy_auth_secret_name = "corporate-proxy-auth" @@ -1547,7 +1281,6 @@ mod tests { "bad key".to_string(), // whitespace is outside the allowed charset ] { let cfg = KubernetesComputeConfig { - topology: SupervisorTopology::Sidecar, https_proxy: Some("http://proxy.corp.example:8080".to_string()), proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()), proxy_auth_secret_key: Some(key.clone()), @@ -1565,7 +1298,6 @@ mod tests { #[test] fn upstream_proxy_config_accepts_max_length_secret_key() { let cfg = KubernetesComputeConfig { - topology: SupervisorTopology::Sidecar, https_proxy: Some("http://proxy.corp.example:8080".to_string()), proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()), proxy_auth_secret_key: Some("a".repeat(253)), @@ -1575,20 +1307,6 @@ mod tests { assert!(cfg.validate_upstream_proxy_config().is_ok()); } - #[test] - fn upstream_proxy_config_rejects_credentials_in_combined_topology() { - let cfg = KubernetesComputeConfig { - topology: SupervisorTopology::Combined, - https_proxy: Some("http://proxy.corp.example:8080".to_string()), - proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()), - proxy_auth_secret_key: Some("credentials".to_string()), - proxy_auth_allow_insecure: Some(true), - ..KubernetesComputeConfig::default() - }; - let err = cfg.validate_upstream_proxy_config().unwrap_err(); - assert!(err.contains("topology = \"sidecar\""), "{err}"); - } - #[test] fn upstream_proxy_config_allows_explicit_false_acknowledgement_without_credentials() { let cfg = KubernetesComputeConfig { diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index b7114c0e99..b1bee60a9a 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -3,20 +3,28 @@ //! Kubernetes compute driver. -use super::AppArmorProfile; use crate::config::{ - DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_SANDBOX_UID, - DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, OperatorNamespaceAllowlist, - SupervisorSideloadMethod, SupervisorTopology, WorkspaceMode, is_dns_1123_label, + DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_SANDBOX_UID, DEFAULT_WORKSPACE_STORAGE_SIZE, + KubernetesComputeConfig, OperatorNamespaceAllowlist, WorkspaceMode, is_dns_1123_label, managed_namespace, managed_namespace_prefix, validate_managed_namespace_name, }; +use crate::isolation::{ + BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesSandboxRuntimeBoundarySpec, +}; +use crate::sandbox_runtime::{ + BOUNDARY_CERTIFICATE_PATH, BOUNDARY_CONFIG_PATH, BOUNDARY_PRIVATE_KEY_PATH, + SANDBOX_SECRET_COMPONENT, SUPERVISOR_SECRET_COMPONENT, SandboxRuntimeNames, boundary_service, + generate_proxy_ca_material, sandbox_bootstrap_secret, + sandbox_owner_reference as sandbox_runtime_sandbox_owner_reference, + supervisor_bootstrap_secret, supervisor_pod, workload_fence, +}; use futures::{Stream, StreamExt, TryStreamExt}; use k8s_openapi::api::authentication::v1::{ TokenReview, TokenReviewSpec, TokenReviewStatus, UserInfo, }; use k8s_openapi::api::core::v1::{ Event as KubeEventObj, Namespace, Node, PersistentVolumeClaimVolumeSource, Pod, Secret, - ServiceAccount, Volume, VolumeMount, + Service, ServiceAccount, Volume, VolumeMount, }; use k8s_openapi::api::networking::v1::{ NetworkPolicy, NetworkPolicyIngressRule, NetworkPolicyPeer, NetworkPolicyPort, @@ -30,13 +38,13 @@ use kube::api::{ use kube::core::gvk::GroupVersionKind; use kube::core::{DynamicObject, ObjectMeta}; use kube::runtime::WatchStreamExt; +use kube::runtime::wait::await_condition; use kube::runtime::watcher::{self, Event}; use kube::{Client, Error as KubeError}; use openshell_core::driver_mounts; use openshell_core::driver_utils::{ LABEL_GATEWAY_ID, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, - LABEL_SANDBOX_NAME, LABEL_SANDBOX_WORKSPACE, SUPERVISOR_IMAGE_BINARY_PATH, - openshell_sandbox_label_selector, + LABEL_SANDBOX_NAME, LABEL_SANDBOX_WORKSPACE, openshell_sandbox_label_selector, }; use openshell_core::gpu::{driver_gpu_requirements, effective_driver_gpu_count}; use openshell_core::progress::{ @@ -53,8 +61,14 @@ use openshell_core::proto::compute::v1::{ WatchSandboxesSandboxEvent, watch_sandboxes_event, }; use openshell_core::proto_struct::{struct_to_json_object, value_to_json}; +use openshell_isolation_interface::contract::ResolvedWorkloadIdentity; +use openshell_sandbox_backend::boundary_protocol::{ + GatewayVerificationKey, SandboxTlsClientConfig, SandboxTlsServerConfig, + generate_sandbox_tls_material, +}; +use rand::RngCore as _; use serde::Deserialize; -use std::collections::{BTreeMap, BTreeSet, HashSet}; +use std::collections::{BTreeMap, HashSet}; use std::path::{Path, PathBuf}; use std::pin::Pin; use std::sync::Arc; @@ -68,6 +82,64 @@ pub type WatchStream = const MANAGED_SSH_NETWORK_POLICY_NAME: &str = "openshell-sandbox-ssh"; const AGENT_SANDBOX_TRACE_CONTEXT_ANNOTATION: &str = "opentelemetry.io/trace-context"; +const ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING: &str = "openshell.ai/sandbox-runtime-bootstrapping"; +const ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT: &str = + "openshell.ai/sandbox-runtime-bootstrap-started-at-ms"; +const ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION: &str = + "openshell.ai/sandbox-runtime-bootstrap-operation"; +const ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE: &str = + "openshell.ai/sandbox-runtime-bootstrap-phase"; +const ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_SESSION: &str = + "openshell.ai/sandbox-runtime-bootstrap-session"; +const ANNOTATION_SANDBOX_RUNTIME_GENERATION: &str = "openshell.ai/sandbox-runtime-generation"; +const ANNOTATION_SANDBOX_RUNTIME_READINESS: &str = "openshell.ai/sandbox-runtime-readiness"; +const ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID: &str = "openshell.ai/sandbox-runtime-workload-uid"; +const ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: &str = + "openshell.ai/sandbox-runtime-supervisor-uid"; +const ANNOTATION_SANDBOX_RUNTIME_MAIN_PROCESS_SPEC: &str = + "openshell.ai/sandbox-runtime-main-process-spec"; +const ANNOTATION_SANDBOX_RUNTIME_LOG_LEVEL: &str = "openshell.ai/sandbox-runtime-log-level"; +const ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: &str = + "openshell.ai/sandbox-runtime-network-policy-uid"; +const ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_VERSION: &str = + "openshell.ai/sandbox-runtime-network-policy-version"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum SandboxRuntimeBootstrapPhase { + Preparing, + Released, + RollingBack, +} + +impl SandboxRuntimeBootstrapPhase { + const fn as_str(self) -> &'static str { + match self { + Self::Preparing => "preparing", + Self::Released => "released", + Self::RollingBack => "rolling-back", + } + } + + fn parse(value: &str) -> Option { + match value { + "preparing" => Some(Self::Preparing), + "released" => Some(Self::Released), + "rolling-back" => Some(Self::RollingBack), + _ => None, + } + } +} + +fn boundary_service_authority( + namespace: &str, + names: &SandboxRuntimeNames, + boundary_port: u16, +) -> String { + format!( + "{}.{}.svc:{boundary_port}", + names.boundary_service, namespace + ) +} #[derive(Debug, thiserror::Error)] pub enum KubernetesDriverError { @@ -86,12 +158,56 @@ pub enum KubernetesDriverError { impl KubernetesDriverError { fn from_kube(err: KubeError) -> Self { match err { - KubeError::Api(api) if api.code == 409 => Self::AlreadyExists, + KubeError::Api(api) if api.code == 409 && api.reason == "AlreadyExists" => { + Self::AlreadyExists + } + KubeError::Api(api) if api.code == 404 => Self::NotFound, other => Self::Message(other.to_string()), } } } +fn is_kube_resource_version_conflict(error: &KubeError) -> bool { + matches!(error, KubeError::Api(api) if api.code == 409 && api.reason == "Conflict") +} + +async fn patch_dynamic_object_with_resource_version_retry( + api: &Api, + name: &str, + mut patch_for_resource_version: impl FnMut(&str) -> serde_json::Value, +) -> Result { + let operation = async { + loop { + let current = api.get(name).await?; + let resource_version = current + .metadata + .resource_version + .as_deref() + .unwrap_or_default(); + let patch = patch_for_resource_version(resource_version); + match api + .patch(name, &PatchParams::default(), &Patch::Merge(&patch)) + .await + { + Ok(updated) => return Ok(updated), + Err(error) if is_kube_resource_version_conflict(&error) => { + tokio::task::yield_now().await; + } + Err(error) => return Err(error), + } + } + }; + tokio::time::timeout(KUBE_API_TIMEOUT, operation) + .await + .map_err(|_| { + KubernetesDriverError::Message(format!( + "timed out after {}s updating Kubernetes resource {name}", + KUBE_API_TIMEOUT.as_secs() + )) + })? + .map_err(KubernetesDriverError::from_kube) +} + impl From for openshell_core::ComputeDriverError { fn from(err: KubernetesDriverError) -> Self { match err { @@ -108,6 +224,58 @@ impl From for openshell_core::ComputeDriverError { /// This prevents gRPC handlers from blocking indefinitely when the k8s /// API server is unreachable or slow. const KUBE_API_TIMEOUT: Duration = Duration::from_secs(30); +const SANDBOX_RUNTIME_RECONCILE_INTERVAL: Duration = Duration::from_secs(30); +/// Bound how long a crash-interrupted, fail-closed bootstrap may remain stranded. +const SANDBOX_RUNTIME_BOOTSTRAP_GRACE: Duration = Duration::from_mins(5); + +fn random_sandbox_runtime_token() -> String { + use std::fmt::Write as _; + + let mut bytes = [0_u8; 32]; + rand::rng().fill_bytes(&mut bytes); + let mut token = String::with_capacity(bytes.len() * 2); + for byte in bytes { + write!(token, "{byte:02x}").expect("writing to String cannot fail"); + } + token +} + +fn decode_launch_authentication( + encoded: &[u8], +) -> Result { + let authentication = + serde_json::from_slice::(encoded) + .map_err(|error| { + KubernetesDriverError::Precondition(format!( + "decode Kubernetes sandbox launch authentication: {error}" + )) + })?; + authentication.validate().map_err(|error| { + KubernetesDriverError::Precondition(format!( + "validate Kubernetes sandbox launch authentication: {error}" + )) + })?; + Ok(authentication) +} + +fn gateway_verification_keys( + keys: &[openshell_core::jwt::SessionVerificationKey], +) -> Result, KubernetesDriverError> { + keys.iter() + .map(|key| { + String::from_utf8(key.public_key_pem.clone()) + .map(|public_key_pem| GatewayVerificationKey { + key_id: key.key_id.clone(), + public_key_pem, + }) + .map_err(|error| { + KubernetesDriverError::Precondition(format!( + "Kubernetes sandbox verification key is not UTF-8 PEM: {error}" + )) + }) + }) + .collect() +} /// Kubernetes defaults pod termination to 30 seconds when the pod template /// omits `terminationGracePeriodSeconds`. @@ -289,7 +457,11 @@ const KUBERNETES_DRIVER_RESERVED_VOLUME_NAMES: &[&str] = &[ UPSTREAM_PROXY_AUTH_VOLUME_NAME, SERVICE_ACCOUNT_TOKEN_VOLUME_NAME, SPIFFE_WORKLOAD_API_VOLUME_NAME, - SUPERVISOR_VOLUME_NAME, + SANDBOX_RUNTIME_VOLUME_NAME, + SANDBOX_STATE_VOLUME_NAME, + SANDBOX_BOOTSTRAP_VOLUME_NAME, + SANDBOX_POD_IDENTITY_VOLUME_NAME, + SANDBOX_PROXY_CA_VOLUME_NAME, WORKSPACE_VOLUME_NAME, ]; @@ -447,7 +619,7 @@ const WORKSPACE_MOUNT_PATH: &str = "/sandbox"; /// Mount path for the workspace PVC in the **init** container. A temporary /// path so the init container can see the image's original `/sandbox` and /// copy it into the PVC. -const WORKSPACE_INIT_MOUNT_PATH: &str = "/workspace-pvc"; +const WORKSPACE_INIT_MOUNT_PATH: &str = "/mnt/openshell-workspace"; /// Name of the init container that seeds the workspace PVC. const WORKSPACE_INIT_CONTAINER_NAME: &str = "workspace-init"; @@ -498,7 +670,19 @@ impl KubernetesComputeDriver { shutdown_rx: tokio::sync::watch::Receiver, ) -> Result { config - .validate_configuration() + .validate_workspace_mode() + .map_err(KubernetesDriverError::Precondition)?; + config + .validate_provider_spiffe_workload_api_socket_path() + .map_err(KubernetesDriverError::Precondition)?; + config + .validate_sandbox_identity_config() + .map_err(KubernetesDriverError::Precondition)?; + config + .validate_proxy_uid() + .map_err(KubernetesDriverError::Precondition)?; + config + .validate_upstream_proxy_config() .map_err(KubernetesDriverError::Precondition)?; let base_config = match kube::Config::incluster() { Ok(c) => c, @@ -624,7 +808,7 @@ impl KubernetesComputeDriver { })?; validate_pod_uid(&pod, &identity.pod_uid)?; let sandbox_id = pod_sandbox_id(&pod)?; - let owner = sandbox_owner_reference(&pod)?; + let (owner, via_proxy_control) = Self::resolve_sandbox_owner(&pod, &sandbox_id)?; let sandboxes = self .supported_agent_sandbox_api(self.client.clone(), &identity.namespace) .await @@ -635,10 +819,35 @@ impl KubernetesComputeDriver { warn!(sandbox = %owner.name, %error, "failed to read authenticated Sandbox resource"); tonic::Status::internal("failed to read authenticated Sandbox resource") })?.ok_or_else(|| tonic::Status::permission_denied("sandbox owner not found"))?; - validate_sandbox_owner_identity(owner, &sandbox_id, &sandbox)?; + validate_sandbox_owner_identity(&owner, &sandbox_id, &sandbox)?; + require_proxy_control_authentication(via_proxy_control)?; Ok(sandbox_id) } + #[allow(clippy::result_large_err)] + fn resolve_sandbox_owner( + pod: &Pod, + sandbox_id: &str, + ) -> Result<(OwnerReference, bool), tonic::Status> { + let via_supervisor = pod + .metadata + .labels + .as_ref() + .and_then(|labels| labels.get(BOUNDARY_ROLE_LABEL)) + .is_some_and(|role| role == "supervisor"); + if via_supervisor { + validate_proxy_control_labels(pod, sandbox_id)?; + let names = SandboxRuntimeNames::new(sandbox_id); + if pod.metadata.name.as_deref() != Some(names.supervisor_pod.as_str()) { + return Err(tonic::Status::permission_denied( + "supervisor pod name does not match the sandbox runtime", + )); + } + } + let owner = sandbox_owner_reference(pod, !via_supervisor)?.clone(); + Ok((owner, via_supervisor)) + } + fn accepts_auth_namespace(&self, namespace: &str) -> bool { accepts_auth_namespace(&self.config, self.operator_allowlist.as_ref(), namespace) } @@ -1091,17 +1300,9 @@ impl KubernetesComputeDriver { } fn validate_driver_config_for_sandbox( - &self, sandbox: &Sandbox, ) -> Result { - kubernetes_driver_config_for_spec( - sandbox.spec.as_ref(), - self.config.provider_spiffe_enabled().then_some( - self.config - .provider_spiffe_workload_api_socket_path - .as_str(), - ), - ) + kubernetes_driver_config_for_spec(sandbox.spec.as_ref()) } fn agent_sandbox_api( @@ -1286,8 +1487,7 @@ impl KubernetesComputeDriver { } pub async fn validate_sandbox_create(&self, sandbox: &Sandbox) -> Result<(), tonic::Status> { - let _ = self - .validate_driver_config_for_sandbox(sandbox) + let _ = Self::validate_driver_config_for_sandbox(sandbox) .map_err(tonic::Status::invalid_argument)?; match self.config.workspace_mode { WorkspaceMode::Shared => { @@ -1328,20 +1528,23 @@ impl KubernetesComputeDriver { let selector = self.sandbox_lookup_selector(sandbox_id); let lp = ListParams::default().labels(&selector); match tokio::time::timeout(KUBE_API_TIMEOUT, agent_sandbox_api.api.list(&lp)).await { - Ok(Ok(list)) => list.items.into_iter().next().map_or_else( - || { + Ok(Ok(list)) => { + let Some(obj) = list.items.into_iter().next() else { debug!(sandbox_id = %sandbox_id, "Sandbox not found in Kubernetes"); - Ok(None) - }, - |obj| { - let ns = obj - .metadata - .namespace - .clone() - .unwrap_or_else(|| self.config.namespace.clone()); - Ok(sandbox_from_object(&ns, obj).ok().map(|(_, s)| s)) - }, - ), + return Ok(None); + }; + let ns = obj + .metadata + .namespace + .clone() + .unwrap_or_else(|| self.config.namespace.clone()); + Ok( + sandbox_from_object_with_sandbox_runtime_readiness(&self.client, &ns, obj) + .await + .ok() + .map(|(_, sandbox)| sandbox), + ) + } Ok(Err(err)) => { warn!( sandbox_id = %sandbox_id, @@ -1383,25 +1586,23 @@ impl KubernetesComputeDriver { .await { Ok(Ok(list)) => { - let mut sandboxes: Vec = list - .items - .into_iter() - .filter_map(|obj| { - let name = obj.metadata.name.clone().unwrap_or_default(); - let ns = obj - .metadata - .namespace - .clone() - .unwrap_or_else(|| self.config.namespace.clone()); - match sandbox_from_object(&ns, obj) { - Ok((_, s)) => Some(s), - Err(err) => { - warn!(object_name = %name, error = %err, "skipping unrecognized Sandbox in list"); - None - } + let mut sandboxes = Vec::new(); + for obj in list.items { + let name = obj.metadata.name.clone().unwrap_or_default(); + let ns = obj + .metadata + .namespace + .clone() + .unwrap_or_else(|| self.config.namespace.clone()); + match sandbox_from_object_with_sandbox_runtime_readiness(&self.client, &ns, obj) + .await + { + Ok((_, sandbox)) => sandboxes.push(sandbox), + Err(err) => { + warn!(object_name = %name, error = %err, "skipping unrecognized Sandbox in list"); } - }) - .collect(); + } + } sandboxes.sort_by(|left, right| { left.name .cmp(&right.name) @@ -1506,65 +1707,76 @@ impl KubernetesComputeDriver { .resolve_sandbox_identity_in_namespace(&target_namespace) .await; - let image_pull_policy = self - .config - .image_pull_policy - .map(KubernetesComputeConfig::image_pull_policy_value) - .transpose() - .map_err(KubernetesDriverError::Precondition)?; - let supervisor_image_pull_policy = self - .config - .supervisor_image_pull_policy - .map(KubernetesComputeConfig::image_pull_policy_value) - .transpose() - .map_err(KubernetesDriverError::Precondition)?; + let generation = random_sandbox_runtime_token(); + let proxy_names = SandboxRuntimeNames::for_generation(&sandbox.id, &generation); + let main_process_spec = openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec( + sandbox.spec.as_ref(), + ) + .map_err(|error| { + KubernetesDriverError::InvalidArgument(format!("encode main process spec: {error}")) + })?; + let log_level = openshell_core::driver_utils::sandbox_log_level(sandbox, "info"); let params = SandboxPodParams { default_image: &self.config.default_image, - image_pull_policy, + image_pull_policy: self.config.image_pull_policy, image_pull_secrets: &self.config.image_pull_secrets, - supervisor_image: &self.config.supervisor_image, - supervisor_image_pull_policy, - supervisor_sideload_method: self.config.supervisor_sideload_method, - topology: self.config.topology, - proxy_uid: self.config.sidecar.proxy_uid, - process_binary_aware_network_policy: self - .config - .sidecar - .process_binary_aware_network_policy, - https_proxy: self.config.https_proxy.as_deref(), - no_proxy: self.config.no_proxy.as_deref(), - proxy_auth_secret_name: self.config.proxy_auth_secret_name.as_deref(), - proxy_auth_secret_key: self.config.proxy_auth_secret_key.as_deref(), - proxy_auth_allow_insecure: self.config.proxy_auth_allow_insecure == Some(true), - proxy_connect_by_hostname: self.config.proxy_connect_by_hostname == Some(true), + sandbox_runtime_image: &self.config.sandbox_runtime_image, + sandbox_runtime_image_pull_policy: self.config.sandbox_runtime_image_pull_policy, service_account_name: &self.config.service_account_name, sandbox_id: &sandbox.id, - sandbox_name: &sandbox.name, - grpc_endpoint: &self.config.grpc_endpoint, - ssh_socket_path: self.ssh_socket_path(), - client_tls_secret_name: &self.config.client_tls_secret_name, - host_gateway_ip: &self.config.host_gateway_ip, enable_user_namespaces: self.config.enable_user_namespaces, - app_armor_profile: self.config.app_armor_profile.as_ref(), workspace_default_storage_size: &self.config.workspace_default_storage_size, workspace_storage_class: &self.config.workspace_storage_class, default_runtime_class_name: &self.config.default_runtime_class_name, - sa_token_ttl_secs: self.config.effective_sa_token_ttl_secs(), - provider_spiffe_enabled: self.config.provider_spiffe_enabled(), - provider_spiffe_workload_api_socket_path: &self - .config - .provider_spiffe_workload_api_socket_path, sandbox_uid: resolved_user_id, sandbox_gid: resolved_group_id, + boundary_port: self.config.sandbox_runtime.boundary_port, + sandbox_secret_name: &proxy_names.sandbox_secret, }; - validate_sidecar_proxy_identity(¶ms)?; - - let data = sandbox_to_k8s_spec(sandbox.spec.as_ref(), ¶ms) - .map_err(KubernetesDriverError::InvalidArgument)?; let kube_name = self.config.kube_resource_name(workspace, name); + let mut data = sandbox_to_k8s_spec(sandbox.spec.as_ref(), ¶ms) + .map_err(KubernetesDriverError::InvalidArgument)?; + self.create_sandbox_runtime_fence(&target_namespace, &proxy_names) + .await?; + // A missing bootstrap Secret keeps both pods inert as defense in + // depth, but the CR is also created suspended so the controller + // never races an unfenced workload into execution. + if agent_sandbox_api.resource.version == SANDBOX_VERSION_V1ALPHA1 { + data["spec"]["replicas"] = serde_json::json!(0); + } else { + data["spec"]["operatingMode"] = serde_json::json!("Suspended"); + } let mut obj = DynamicObject::new(&kube_name, &agent_sandbox_api.resource); let mut annotations = sandbox_annotations(sandbox); add_trace_context_annotation(&mut annotations); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING.to_string(), + "true".to_string(), + ); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT.to_string(), + openshell_core::time::now_ms().to_string(), + ); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION.to_string(), + "create".to_string(), + ); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE.to_string(), + SandboxRuntimeBootstrapPhase::Preparing.as_str().to_string(), + ); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_GENERATION.to_string(), + generation.clone(), + ); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_MAIN_PROCESS_SPEC.to_string(), + main_process_spec.clone(), + ); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_LOG_LEVEL.to_string(), + log_level.clone(), + ); for key in [ crate::config::ANNOTATION_SCC_UID_RANGE, crate::config::ANNOTATION_SCC_SUPPLEMENTAL_GROUPS, @@ -1574,27 +1786,27 @@ impl KubernetesComputeDriver { } } obj.metadata = ObjectMeta { - name: Some(kube_name), - namespace: Some(target_namespace), + name: Some(kube_name.clone()), + namespace: Some(target_namespace.clone()), labels: Some(sandbox_labels(sandbox, Some(&self.config.gateway_id))), annotations: Some(annotations), ..Default::default() }; obj.data = data; - match tokio::time::timeout( + let created = match tokio::time::timeout( KUBE_API_TIMEOUT, agent_sandbox_api.api.create(&PostParams::default(), &obj), ) .await { - Ok(Ok(_result)) => { + Ok(Ok(result)) => { info!( sandbox_id = %sandbox.id, sandbox_name = %name, "Sandbox created in Kubernetes successfully" ); - Ok(()) + result } Ok(Err(err)) => { warn!( @@ -1603,7 +1815,7 @@ impl KubernetesComputeDriver { error = %err, "Failed to create sandbox in Kubernetes" ); - Err(KubernetesDriverError::from_kube(err)) + return Err(KubernetesDriverError::from_kube(err)); } Err(_elapsed) => { warn!( @@ -1612,165 +1824,1234 @@ impl KubernetesComputeDriver { timeout_secs = KUBE_API_TIMEOUT.as_secs(), "Timed out creating sandbox in Kubernetes" ); - Err(KubernetesDriverError::Message(format!( + return Err(KubernetesDriverError::Message(format!( "timed out after {}s waiting for Kubernetes API", KUBE_API_TIMEOUT.as_secs() - ))) + ))); } + }; + if let Err(error) = self + .create_sandbox_runtime_companions( + sandbox, + &target_namespace, + &kube_name, + &agent_sandbox_api, + &created, + &proxy_names, + &generation, + resolved_user_id, + resolved_group_id, + &main_process_spec, + &log_level, + ) + .await + { + warn!(sandbox_id = %sandbox.id, %error, "sandbox-runtime provisioning failed; rolling back Sandbox CR"); + let _ = agent_sandbox_api + .api + .delete(&kube_name, &DeleteParams::default()) + .await; + return Err(error); } + Ok(()) } - #[tracing::instrument( - name = "kubernetes.stop_sandbox", - skip(self), - fields( - otel.name = "kubernetes.stop_sandbox", - otel.status_code = tracing::field::Empty, - sandbox.id = %sandbox_id, - ) - )] - pub async fn stop_sandbox(&self, sandbox_id: &str) -> Result<(), KubernetesDriverError> { - let span_status = openshell_otel::ErrorStatusGuard::current(); - let result = self.stop_sandbox_inner(sandbox_id).await; - span_status.finish(result) + async fn create_sandbox_runtime_fence( + &self, + namespace: &str, + names: &SandboxRuntimeNames, + ) -> Result<(), KubernetesDriverError> { + let fence = workload_fence(namespace, names, self.config.sandbox_runtime.boundary_port); + let policies: Api = Api::namespaced(self.client.clone(), namespace); + for (mut policy, component) in [ + (fence.workload_policy, "sandbox-workload-fence"), + (fence.supervisor_policy, "sandbox-supervisor-egress"), + ] { + let labels = policy.metadata.labels.get_or_insert_default(); + labels.insert( + LABEL_MANAGED_BY.to_string(), + LABEL_MANAGED_BY_VALUE.to_string(), + ); + labels.insert("openshell.ai/component".to_string(), component.to_string()); + create_or_validate_sandbox_runtime_fence(&policies, &policy).await?; + } + Ok(()) } - async fn stop_sandbox_inner(&self, sandbox_id: &str) -> Result<(), KubernetesDriverError> { - let (agent_sandbox_api, kube_name, pod_name, namespace, stop_timeout) = self - .patch_sandbox_operating_state(sandbox_id, false) - .await?; - let pod_api = Api::::namespaced(self.client.clone(), &namespace); - - let deadline = tokio::time::Instant::now() + stop_timeout; - let mut poll_interval = STOP_INITIAL_POLL_INTERVAL; + async fn wait_for_bootstrap_workload_pod( + &self, + pods: &Api, + pod_name: &str, + sandbox_uid: &str, + ) -> Result { + let deadline = tokio::time::Instant::now() + KUBE_API_TIMEOUT; loop { - let now = tokio::time::Instant::now(); - if now >= deadline { - return Err(KubernetesDriverError::Message(format!( - "timed out after {}s waiting for Kubernetes sandbox to stop", - stop_timeout.as_secs() - ))); - } - let request_timeout = KUBE_API_TIMEOUT.min(deadline.saturating_duration_since(now)); - let object = tokio::time::timeout( - request_timeout, - agent_sandbox_api.api.get(&kube_name), - ) - .await - .map_err(|_| { - KubernetesDriverError::Message(format!( - "timed out after {}s waiting for Kubernetes API while checking sandbox stop", - request_timeout.as_secs() - )) - })? - .map_err(KubernetesDriverError::from_kube)?; - if let Some(error) = kubernetes_sandbox_stop_failure(&object) { - return Err(KubernetesDriverError::Message(error)); - } - let pod_is_gone = kubernetes_sandbox_pod_is_gone(&pod_api, &pod_name, deadline) - .await - .map_err(KubernetesDriverError::Message)?; - let stop_is_complete = kubernetes_sandbox_stop_is_complete( - &agent_sandbox_api.resource.version, - &object, - pod_is_gone, - ); - if stop_is_complete { - return Ok(()); - } - let now = tokio::time::Instant::now(); - if now >= deadline { - return Err(KubernetesDriverError::Message(format!( - "timed out after {}s waiting for Kubernetes sandbox to stop", - stop_timeout.as_secs() - ))); + match pods.get_opt(pod_name).await { + Ok(Some(pod)) => { + let owned = pod + .metadata + .owner_references + .as_deref() + .unwrap_or_default() + .iter() + .any(|owner| { + owner.controller == Some(true) + && owner.kind == SANDBOX_KIND + && owner.uid == sandbox_uid + }); + if !owned { + return Err(KubernetesDriverError::Precondition(format!( + "workload Pod {pod_name} is not controlled by the created Sandbox UID" + ))); + } + return Ok(pod); + } + Ok(None) if tokio::time::Instant::now() < deadline => { + tokio::time::sleep(Duration::from_millis(100)).await; + } + Ok(None) => { + return Err(KubernetesDriverError::Message(format!( + "timed out waiting for gated workload Pod {pod_name}" + ))); + } + Err(error) => return Err(KubernetesDriverError::from_kube(error)), } - tokio::time::sleep(poll_interval.min(deadline.saturating_duration_since(now))).await; - poll_interval = next_stop_poll_interval(poll_interval); } } - #[tracing::instrument( - name = "kubernetes.start_sandbox", - skip(self), - fields( - otel.name = "kubernetes.start_sandbox", - otel.status_code = tracing::field::Empty, - sandbox.id = %sandbox_id, - ) - )] - pub async fn start_sandbox(&self, sandbox_id: &str) -> Result<(), KubernetesDriverError> { - let span_status = openshell_otel::ErrorStatusGuard::current(); - let result = self - .patch_sandbox_operating_state(sandbox_id, true) - .await - .map(|_| ()); - span_status.finish(result) - } - - async fn patch_sandbox_operating_state( - &self, + #[allow(clippy::too_many_arguments)] + fn validate_capability_free_workload_pod( + pod: &Pod, + sandbox_resource_uid: &str, sandbox_id: &str, - running: bool, - ) -> Result<(AgentSandboxApi, String, String, String, Duration), KubernetesDriverError> { - let lookup_api = self - .supported_sandbox_api_for_lookup(self.client.clone()) - .await - .map_err(KubernetesDriverError::Message)?; - let selector = self.sandbox_lookup_selector(sandbox_id); - let list = tokio::time::timeout( - KUBE_API_TIMEOUT, - lookup_api - .api - .list(&ListParams::default().labels(&selector)), - ) - .await - .map_err(|_| { - KubernetesDriverError::Message(format!( - "timed out after {}s waiting for Kubernetes API", - KUBE_API_TIMEOUT.as_secs() + uid: u32, + gid: u32, + sandbox_secret_name: &str, + ) -> Result<(), KubernetesDriverError> { + let fail = |message: &str| { + KubernetesDriverError::Precondition(format!( + "admitted workload Pod does not preserve capability-free isolation: {message}" )) - })? - .map_err(KubernetesDriverError::from_kube)?; - let object = list - .items - .into_iter() - .next() - .ok_or(KubernetesDriverError::NotFound)?; - let namespace = object + }; + if pod.metadata.uid.as_deref().is_none() { + return Err(fail("missing Pod UID")); + } + let owner_matches = pod .metadata - .namespace - .clone() - .unwrap_or_else(|| self.config.namespace.clone()); - let agent_sandbox_api = Self::agent_sandbox_api( - self.client.clone(), - &lookup_api.resource.version, - &namespace, - ); - let stop_timeout = kubernetes_sandbox_stop_timeout(&object); - let kube_name = object.metadata.name.ok_or_else(|| { - KubernetesDriverError::Message("sandbox resource has no name".to_string()) - })?; - let pod_name = object + .owner_references + .as_deref() + .unwrap_or_default() + .iter() + .any(|owner| { + owner.controller == Some(true) + && owner.kind == SANDBOX_KIND + && owner.uid == sandbox_resource_uid + }); + if !owner_matches { + return Err(fail("Sandbox owner UID changed")); + } + let labels = pod .metadata - .annotations + .labels .as_ref() - .and_then(|annotations| annotations.get(SANDBOX_POD_NAME_ANNOTATION)) - .cloned() - .unwrap_or_else(|| kube_name.clone()); - let resource_version = object.metadata.resource_version.unwrap_or_default(); - let desired = sandbox_operating_state_patch( - &agent_sandbox_api.resource.version, - &resource_version, - running, + .ok_or_else(|| fail("missing labels"))?; + let expected_pair = crate::sandbox_runtime::pair_label_value(sandbox_id); + if labels.get(BOUNDARY_ROLE_LABEL).map(String::as_str) != Some("workload") + || labels.get(BOUNDARY_PAIR_LABEL).map(String::as_str) != Some(expected_pair.as_str()) + { + return Err(fail("pair labels changed")); + } + let spec = pod.spec.as_ref().ok_or_else(|| fail("missing Pod spec"))?; + if spec.host_network == Some(true) + || spec.host_pid == Some(true) + || spec.host_ipc == Some(true) + || spec.share_process_namespace == Some(true) + || spec.automount_service_account_token != Some(false) + { + return Err(fail("host namespaces or ServiceAccount token enabled")); + } + let projected_service_account_token = spec + .volumes + .as_deref() + .unwrap_or_default() + .iter() + .filter_map(|volume| volume.projected.as_ref()) + .flat_map(|projected| projected.sources.as_deref().unwrap_or_default()) + .any(|source| source.service_account_token.is_some()); + if projected_service_account_token { + return Err(fail( + "workload Pod must not receive a projected ServiceAccount token", + )); + } + if spec.restart_policy.as_deref() != Some("Never") + || spec.dns_policy.as_deref() != Some("None") + || spec + .dns_config + .as_ref() + .is_none_or(|dns| dns.nameservers.as_deref() != Some(&["127.0.0.53".to_string()])) + { + return Err(fail("restart or DNS posture changed")); + } + if !spec + .scheduling_gates + .as_deref() + .unwrap_or_default() + .iter() + .any(|gate| gate.name == SANDBOX_BOOTSTRAP_SCHEDULING_GATE) + { + return Err(fail("bootstrap scheduling gate missing")); + } + let security = spec + .security_context + .as_ref() + .ok_or_else(|| fail("missing Pod security context"))?; + if security.run_as_user != Some(i64::from(uid)) + || security.run_as_group != Some(i64::from(gid)) + || security.run_as_non_root != Some(true) + || security.fs_group != Some(i64::from(gid)) + || security + .supplemental_groups + .as_deref() + .is_some_and(|groups| !groups.is_empty()) + || security + .seccomp_profile + .as_ref() + .is_none_or(|profile| profile.type_ != "RuntimeDefault") + { + return Err(fail("numeric identity, groups, or seccomp profile changed")); + } + let pod_json = serde_json::to_value(pod) + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; + if let Some(policy) = pod_json.pointer("/spec/securityContext/supplementalGroupsPolicy") + && policy != &serde_json::json!("Strict") + { + return Err(fail("supplementalGroupsPolicy is not Strict")); + } + if pod_json + .pointer("/spec/securityContext/supplementalGroupsPolicy") + .is_none() + { + tracing::warn!( + pod = pod.metadata.name.as_deref().unwrap_or(""), + "Kubernetes omitted supplementalGroupsPolicy; exact runtime groups remain enforced by boundary confirmation" + ); + } + let unprivileged_port_sysctl = pod_json + .pointer("/spec/securityContext/sysctls") + .and_then(serde_json::Value::as_array) + .is_some_and(|sysctls| { + sysctls.iter().any(|sysctl| { + sysctl.get("name").and_then(serde_json::Value::as_str) + == Some("net.ipv4.ip_unprivileged_port_start") + && sysctl.get("value").and_then(serde_json::Value::as_str) == Some("0") + }) + }); + if !unprivileged_port_sysctl { + return Err(fail("safe unprivileged-port sysctl changed")); + } + let check_container = |container: &k8s_openapi::api::core::v1::Container, + name: &str| + -> Result<(), KubernetesDriverError> { + let context = container + .security_context + .as_ref() + .ok_or_else(|| fail(&format!("{name} has no security context")))?; + let drops_all = context + .capabilities + .as_ref() + .and_then(|capabilities| capabilities.drop.as_deref()) + .is_some_and(|drops| drops.iter().any(|capability| capability == "ALL")); + let adds_none = context + .capabilities + .as_ref() + .and_then(|capabilities| capabilities.add.as_deref()) + .is_none_or(<[String]>::is_empty); + if context.run_as_user != Some(i64::from(uid)) + || context.run_as_group != Some(i64::from(gid)) + || context.run_as_non_root != Some(true) + || context.allow_privilege_escalation != Some(false) + || !drops_all + || !adds_none + { + return Err(fail(&format!("{name} security context changed"))); + } + Ok(()) + }; + let agent = spec + .containers + .iter() + .find(|container| container.name == "agent") + .ok_or_else(|| fail("agent container missing"))?; + check_container(agent, "agent")?; + let bootstrap = spec + .init_containers + .as_deref() + .unwrap_or_default() + .iter() + .find(|container| container.name == "openshell-sandbox-bootstrap") + .ok_or_else(|| fail("trusted bootstrap init container missing"))?; + check_container(bootstrap, "bootstrap init container")?; + let mounts_volume = |container: &k8s_openapi::api::core::v1::Container, + volume_name: &str| { + container + .volume_mounts + .as_deref() + .unwrap_or_default() + .iter() + .any(|mount| mount.name == volume_name) + }; + if mounts_volume(agent, SANDBOX_BOOTSTRAP_VOLUME_NAME) + || !mounts_volume(bootstrap, SANDBOX_BOOTSTRAP_VOLUME_NAME) + { + return Err(fail( + "bootstrap Secret must be mounted only by the trusted init container", + )); + } + let secret_matches = spec + .volumes + .as_deref() + .unwrap_or_default() + .iter() + .any(|volume| { + volume.name == SANDBOX_BOOTSTRAP_VOLUME_NAME + && volume + .secret + .as_ref() + .and_then(|secret| secret.secret_name.as_deref()) + == Some(sandbox_secret_name) + }); + if !secret_matches { + return Err(fail("generation-specific sandbox Secret changed")); + } + Ok(()) + } + + #[allow(clippy::too_many_arguments, clippy::similar_names)] + async fn create_sandbox_runtime_companions( + &self, + sandbox: &Sandbox, + namespace: &str, + cr_name: &str, + sandbox_api: &AgentSandboxApi, + sandbox_cr: &DynamicObject, + names: &SandboxRuntimeNames, + _generation: &str, + agent_uid: u32, + agent_gid: u32, + main_process_spec: &str, + log_level: &str, + ) -> Result<(), KubernetesDriverError> { + let cr_uid = sandbox_cr.metadata.uid.as_deref().ok_or_else(|| { + KubernetesDriverError::Message("created Sandbox CR has no UID".to_string()) + })?; + let namespace_uid = Api::::all(self.client.clone()) + .get(namespace) + .await + .map_err(KubernetesDriverError::from_kube)? + .metadata + .uid + .ok_or_else(|| { + KubernetesDriverError::Message("sandbox namespace has no UID".to_string()) + })?; + let dependent_owner = sandbox_runtime_sandbox_owner_reference( + cr_name, + cr_uid, + &sandbox_api.resource.api_version, + false, ); - tokio::time::timeout( + let services: Api = Api::namespaced(self.client.clone(), namespace); + let service = services + .create( + &PostParams::default(), + &boundary_service( + namespace, + names, + &sandbox.id, + self.config.sandbox_runtime.boundary_port, + dependent_owner.clone(), + ), + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + let service_ip: std::net::IpAddr = service + .spec + .and_then(|spec| spec.cluster_ip) + .filter(|ip| ip != "None") + .ok_or_else(|| { + KubernetesDriverError::Message("boundary Service has no ClusterIP".to_string()) + })? + .parse() + .map_err(|error| { + KubernetesDriverError::Message(format!( + "invalid boundary Service ClusterIP: {error}" + )) + })?; + + let pods: Api = Api::namespaced(self.client.clone(), namespace); + let supervisor = pods + .create( + &PostParams::default(), + &supervisor_pod( + namespace, + names, + &sandbox.id, + &sandbox.name, + &self.config.gateway_id, + &self.config.supervisor_image, + self.config.supervisor_image_pull_policy, + &self.config.service_account_name, + agent_uid, + agent_gid, + &self.config.image_pull_secrets, + &self.config.grpc_endpoint, + &self.config.client_tls_secret_name, + main_process_spec, + log_level, + self.config.effective_sa_token_ttl_secs(), + self.config.https_proxy.as_deref(), + self.config.no_proxy.as_deref(), + self.config + .proxy_auth_secret_name + .as_deref() + .zip(self.config.proxy_auth_secret_key.as_deref()), + self.config.proxy_auth_allow_insecure == Some(true), + self.config.proxy_connect_by_hostname == Some(true), + self.config.provider_spiffe_enabled().then_some( + self.config + .provider_spiffe_workload_api_socket_path + .as_str(), + ), + dependent_owner.clone(), + ) + .map_err(KubernetesDriverError::Message)?, + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + let supervisor_uid = supervisor.metadata.uid.ok_or_else(|| { + KubernetesDriverError::Message("supervisor Pod has no UID".to_string()) + })?; + + let policies: Api = Api::namespaced(self.client.clone(), namespace); + let fence = policies + .get(&names.workload_policy) + .await + .map_err(KubernetesDriverError::from_kube)?; + let fence_uid = fence.metadata.uid.ok_or_else(|| { + KubernetesDriverError::Message("workload NetworkPolicy has no UID".to_string()) + })?; + let fence_resource_version = fence.metadata.resource_version.ok_or_else(|| { + KubernetesDriverError::Message( + "workload NetworkPolicy has no resourceVersion".to_string(), + ) + })?; + + // Release the Sandbox CR only far enough for the controller to create + // the workload Pod. The Pod remains unschedulable because its template + // carries the OpenShell scheduling gate and references a Secret that + // does not exist yet. + patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { + sandbox_operating_state_patch(&sandbox_api.resource.version, version, true) + }) + .await?; + + let workload_pod = self + .wait_for_bootstrap_workload_pod(&pods, cr_name, cr_uid) + .await?; + Self::validate_capability_free_workload_pod( + &workload_pod, + cr_uid, + &sandbox.id, + agent_uid, + agent_gid, + &names.sandbox_secret, + )?; + let workload_pod_uid = + workload_pod.metadata.uid.clone().ok_or_else(|| { + KubernetesDriverError::Message("workload Pod has no UID".to_string()) + })?; + let workload_pod_name = workload_pod.metadata.name.clone().ok_or_else(|| { + KubernetesDriverError::Message("workload Pod has no name".to_string()) + })?; + patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { + serde_json::json!({ + "metadata": { + "resourceVersion": version, + "annotations": { + ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID: workload_pod_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: supervisor_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: fence_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_VERSION: fence_resource_version.clone(), + } + } + }) + }) + .await?; + + let launch_authentication = sandbox + .spec + .as_ref() + .filter(|spec| !spec.launch_authentication.is_empty()) + .ok_or_else(|| { + KubernetesDriverError::Precondition( + "Kubernetes sandbox launch authentication is required".to_string(), + ) + }) + .and_then(|spec| decode_launch_authentication(&spec.launch_authentication))?; + let mut child_env = sandbox + .spec + .as_ref() + .and_then(|spec| spec.template.as_ref()) + .map_or_else(std::collections::HashMap::new, |template| { + template.environment.clone() + }); + if let Some(spec) = sandbox.spec.as_ref() { + child_env.extend(spec.environment.clone()); + } + child_env.retain(|name, _| !name.starts_with("OPENSHELL_")); + let host_gateway_ip = self.config.host_gateway_ip.parse().ok(); + let session_id = launch_authentication.supervisor.session_id; + let tls = generate_sandbox_tls_material(session_id) + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; + let verification_keys = + gateway_verification_keys(&launch_authentication.verification_keys)?; + let proxy_ca = generate_proxy_ca_material().map_err(KubernetesDriverError::Message)?; + let workload_identity = ResolvedWorkloadIdentity::new( + agent_uid, + agent_gid, + Vec::new(), + "kubernetes-config".to_string(), + format!("sandbox:{cr_uid}"), + ) + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; + let provisioned = KubernetesSandboxRuntimeBoundarySpec { + boundary_id: sandbox.id.clone(), + generation: launch_authentication + .supervisor + .runtime_generation + .to_string(), + session_id, + session_rotation: launch_authentication.supervisor.session_rotation, + auth_epoch: launch_authentication.supervisor.auth_epoch, + gateway_id: launch_authentication.gateway_id, + verification_keys, + namespace_uid, + sandbox_resource_uid: cr_uid.to_string(), + workload_pod_uid: workload_pod_uid.clone(), + workload_pod_uid_path: PathBuf::from(SANDBOX_POD_UID_PATH), + supervisor_pod_uid: supervisor_uid.clone(), + egress_policy_uid: fence_uid.clone(), + egress_policy_resource_version: fence_resource_version.clone(), + boundary_listener: std::net::SocketAddr::new( + if service_ip.is_ipv6() { + std::net::IpAddr::V6(std::net::Ipv6Addr::UNSPECIFIED) + } else { + std::net::IpAddr::V4(std::net::Ipv4Addr::UNSPECIFIED) + }, + self.config.sandbox_runtime.boundary_port, + ), + control_authority: boundary_service_authority( + namespace, + names, + self.config.sandbox_runtime.boundary_port, + ), + control_address: std::net::SocketAddr::new( + service_ip, + self.config.sandbox_runtime.boundary_port, + ), + sandbox_tls: SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from(BOUNDARY_CERTIFICATE_PATH), + private_key_path: PathBuf::from(BOUNDARY_PRIVATE_KEY_PATH), + }, + supervisor_tls: SandboxTlsClientConfig { + server_name: tls.server_name.clone(), + trust_anchor_pem: tls.trust_anchor_pem.clone(), + }, + host_gateway_ip, + workload_identity, + child_env, + } + .provision(); + let descriptor = provisioned + .runtime_descriptor + .backend_descriptor() + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; + let sandbox_secret = sandbox_bootstrap_secret( + namespace, + names, + &sandbox.id, + provisioned + .boundary_config + .encode() + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?, + tls.certificate_chain_pem.into_bytes(), + tls.private_key_pem.into_bytes(), + OwnerReference { + api_version: "v1".to_string(), + kind: "Pod".to_string(), + name: workload_pod_name.clone(), + uid: workload_pod_uid, + controller: Some(false), + block_owner_deletion: Some(false), + }, + ); + let supervisor_secret = supervisor_bootstrap_secret( + namespace, + names, + &sandbox.id, + descriptor.payload, + serde_json::to_vec(&launch_authentication.supervisor).map_err(|error| { + KubernetesDriverError::Message(format!("encode supervisor auth bundle: {error}")) + })?, + proxy_ca.certificate_pem.into_bytes(), + proxy_ca.private_key_pem.into_bytes(), + OwnerReference { + api_version: "v1".to_string(), + kind: "Pod".to_string(), + name: names.supervisor_pod.clone(), + uid: supervisor_uid, + controller: Some(false), + block_owner_deletion: Some(false), + }, + ); + let secrets = Api::::namespaced(self.client.clone(), namespace); + secrets + .create(&PostParams::default(), &sandbox_secret) + .await + .map_err(KubernetesDriverError::from_kube)?; + secrets + .create(&PostParams::default(), &supervisor_secret) + .await + .map_err(KubernetesDriverError::from_kube)?; + + pods.patch( + &names.supervisor_pod, + &PatchParams::default(), + &Patch::Merge(&serde_json::json!({"spec": {"schedulingGates": []}})), + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + pods.patch( + &workload_pod_name, + &PatchParams::default(), + &Patch::Merge(&serde_json::json!({"spec": {"schedulingGates": []}})), + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { + sandbox_runtime_bootstrap_phase_patch(version, SandboxRuntimeBootstrapPhase::Released) + }) + .await?; + spawn_sandbox_runtime_bootstrap_completion( + pods.clone(), + sandbox_api.api.clone(), + names.supervisor_pod.clone(), + cr_name.to_string(), + Some(cr_uid.to_string()), + ); + // Return while the CR remains explicitly bootstrapping. The gateway + // can now commit the sandbox configuration required by a policy-less + // control process without deadlocking behind this driver call. Only + // boundary PID 1 is running at this point; the agent process cannot + // start until control attaches and confirms enforcement. Reconcile + // removes the marker after the supervisor Pod becomes Ready. + Ok(()) + } + + #[allow(clippy::too_many_arguments, clippy::similar_names)] + async fn install_sandbox_runtime_generation( + &self, + namespace: &str, + cr_name: &str, + sandbox_api: &AgentSandboxApi, + sandbox_id: &str, + cr_uid: &str, + names: &SandboxRuntimeNames, + _generation: &str, + supervisor_uid: &str, + agent_uid: u32, + agent_gid: u32, + child_env: std::collections::HashMap, + launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication, + ) -> Result<(), KubernetesDriverError> { + let namespace_uid = Api::::all(self.client.clone()) + .get(namespace) + .await + .map_err(KubernetesDriverError::from_kube)? + .metadata + .uid + .ok_or_else(|| { + KubernetesDriverError::Message("sandbox namespace has no UID".to_string()) + })?; + let services = Api::::namespaced(self.client.clone(), namespace); + let service_ip: std::net::IpAddr = services + .get(&names.boundary_service) + .await + .map_err(KubernetesDriverError::from_kube)? + .spec + .and_then(|spec| spec.cluster_ip) + .filter(|ip| ip != "None") + .ok_or_else(|| { + KubernetesDriverError::Message("boundary Service has no ClusterIP".to_string()) + })? + .parse() + .map_err(|error| { + KubernetesDriverError::Message(format!( + "invalid boundary Service ClusterIP: {error}" + )) + })?; + let policies = Api::::namespaced(self.client.clone(), namespace); + let fence = policies + .get(&names.workload_policy) + .await + .map_err(KubernetesDriverError::from_kube)?; + let fence_uid = fence.metadata.uid.ok_or_else(|| { + KubernetesDriverError::Message("workload NetworkPolicy has no UID".to_string()) + })?; + let fence_resource_version = fence.metadata.resource_version.ok_or_else(|| { + KubernetesDriverError::Message( + "workload NetworkPolicy has no resourceVersion".to_string(), + ) + })?; + + let pods = Api::::namespaced(self.client.clone(), namespace); + let workload_pod = self + .wait_for_bootstrap_workload_pod(&pods, cr_name, cr_uid) + .await?; + Self::validate_capability_free_workload_pod( + &workload_pod, + cr_uid, + sandbox_id, + agent_uid, + agent_gid, + &names.sandbox_secret, + )?; + let workload_pod_uid = + workload_pod.metadata.uid.clone().ok_or_else(|| { + KubernetesDriverError::Message("workload Pod has no UID".to_string()) + })?; + let workload_pod_name = workload_pod.metadata.name.clone().ok_or_else(|| { + KubernetesDriverError::Message("workload Pod has no name".to_string()) + })?; + patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { + serde_json::json!({ + "metadata": { + "resourceVersion": version, + "annotations": { + ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID: workload_pod_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: supervisor_uid, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: fence_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_VERSION: fence_resource_version.clone(), + } + } + }) + }) + .await?; + + let session_id = launch_authentication.supervisor.session_id; + let tls = generate_sandbox_tls_material(session_id) + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; + let verification_keys = + gateway_verification_keys(&launch_authentication.verification_keys)?; + let proxy_ca = generate_proxy_ca_material().map_err(KubernetesDriverError::Message)?; + let workload_identity = ResolvedWorkloadIdentity::new( + agent_uid, + agent_gid, + Vec::new(), + "kubernetes-config".to_string(), + format!("sandbox:{cr_uid}"), + ) + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; + let provisioned = KubernetesSandboxRuntimeBoundarySpec { + boundary_id: sandbox_id.to_string(), + generation: launch_authentication + .supervisor + .runtime_generation + .to_string(), + session_id, + session_rotation: launch_authentication.supervisor.session_rotation, + auth_epoch: launch_authentication.supervisor.auth_epoch, + gateway_id: launch_authentication.gateway_id.clone(), + verification_keys, + namespace_uid, + sandbox_resource_uid: cr_uid.to_string(), + workload_pod_uid: workload_pod_uid.clone(), + workload_pod_uid_path: PathBuf::from(SANDBOX_POD_UID_PATH), + supervisor_pod_uid: supervisor_uid.to_string(), + egress_policy_uid: fence_uid.clone(), + egress_policy_resource_version: fence_resource_version.clone(), + boundary_listener: std::net::SocketAddr::new( + if service_ip.is_ipv6() { + std::net::IpAddr::V6(std::net::Ipv6Addr::UNSPECIFIED) + } else { + std::net::IpAddr::V4(std::net::Ipv4Addr::UNSPECIFIED) + }, + self.config.sandbox_runtime.boundary_port, + ), + control_authority: boundary_service_authority( + namespace, + names, + self.config.sandbox_runtime.boundary_port, + ), + control_address: std::net::SocketAddr::new( + service_ip, + self.config.sandbox_runtime.boundary_port, + ), + sandbox_tls: SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from(BOUNDARY_CERTIFICATE_PATH), + private_key_path: PathBuf::from(BOUNDARY_PRIVATE_KEY_PATH), + }, + supervisor_tls: SandboxTlsClientConfig { + server_name: tls.server_name.clone(), + trust_anchor_pem: tls.trust_anchor_pem.clone(), + }, + host_gateway_ip: self.config.host_gateway_ip.parse().ok(), + workload_identity, + child_env, + } + .provision(); + let descriptor = provisioned + .runtime_descriptor + .backend_descriptor() + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; + let sandbox_secret = sandbox_bootstrap_secret( + namespace, + names, + sandbox_id, + provisioned + .boundary_config + .encode() + .map_err(|error| KubernetesDriverError::Message(error.to_string()))?, + tls.certificate_chain_pem.into_bytes(), + tls.private_key_pem.into_bytes(), + OwnerReference { + api_version: "v1".to_string(), + kind: "Pod".to_string(), + name: workload_pod_name.clone(), + uid: workload_pod_uid, + controller: Some(false), + block_owner_deletion: Some(false), + }, + ); + let supervisor_secret = supervisor_bootstrap_secret( + namespace, + names, + sandbox_id, + descriptor.payload, + serde_json::to_vec(&launch_authentication.supervisor).map_err(|error| { + KubernetesDriverError::Message(format!("encode supervisor auth bundle: {error}")) + })?, + proxy_ca.certificate_pem.into_bytes(), + proxy_ca.private_key_pem.into_bytes(), + OwnerReference { + api_version: "v1".to_string(), + kind: "Pod".to_string(), + name: names.supervisor_pod.clone(), + uid: supervisor_uid.to_string(), + controller: Some(false), + block_owner_deletion: Some(false), + }, + ); + let secrets = Api::::namespaced(self.client.clone(), namespace); + secrets + .create(&PostParams::default(), &sandbox_secret) + .await + .map_err(KubernetesDriverError::from_kube)?; + secrets + .create(&PostParams::default(), &supervisor_secret) + .await + .map_err(KubernetesDriverError::from_kube)?; + + pods.patch( + &names.supervisor_pod, + &PatchParams::default(), + &Patch::Merge(&serde_json::json!({"spec": {"schedulingGates": []}})), + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + pods.patch( + &workload_pod_name, + &PatchParams::default(), + &Patch::Merge(&serde_json::json!({"spec": {"schedulingGates": []}})), + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { + sandbox_runtime_bootstrap_phase_patch(version, SandboxRuntimeBootstrapPhase::Released) + }) + .await?; + spawn_sandbox_runtime_bootstrap_completion( + pods, + sandbox_api.api.clone(), + names.supervisor_pod.clone(), + cr_name.to_string(), + Some(cr_uid.to_string()), + ); + Ok(()) + } + + #[tracing::instrument( + name = "kubernetes.stop_sandbox", + skip(self), + fields( + otel.name = "kubernetes.stop_sandbox", + otel.status_code = tracing::field::Empty, + sandbox.id = %sandbox_id, + ) + )] + pub async fn stop_sandbox(&self, sandbox_id: &str) -> Result<(), KubernetesDriverError> { + let span_status = openshell_otel::ErrorStatusGuard::current(); + let result = Box::pin(self.stop_sandbox_inner(sandbox_id)).await; + span_status.finish(result) + } + + async fn stop_sandbox_inner(&self, sandbox_id: &str) -> Result<(), KubernetesDriverError> { + let (agent_sandbox_api, kube_name, pod_name, namespace, stop_timeout) = self + .patch_sandbox_operating_state(sandbox_id, false) + .await?; + let pod_api = Api::::namespaced(self.client.clone(), &namespace); + + let deadline = tokio::time::Instant::now() + stop_timeout; + let mut poll_interval = STOP_INITIAL_POLL_INTERVAL; + loop { + let now = tokio::time::Instant::now(); + if now >= deadline { + return Err(KubernetesDriverError::Message(format!( + "timed out after {}s waiting for Kubernetes sandbox to stop", + stop_timeout.as_secs() + ))); + } + let request_timeout = KUBE_API_TIMEOUT.min(deadline.saturating_duration_since(now)); + let object = tokio::time::timeout( + request_timeout, + agent_sandbox_api.api.get(&kube_name), + ) + .await + .map_err(|_| { + KubernetesDriverError::Message(format!( + "timed out after {}s waiting for Kubernetes API while checking sandbox stop", + request_timeout.as_secs() + )) + })? + .map_err(KubernetesDriverError::from_kube)?; + if let Some(error) = kubernetes_sandbox_stop_failure(&object) { + return Err(KubernetesDriverError::Message(error)); + } + let pod_is_gone = kubernetes_sandbox_pod_is_gone(&pod_api, &pod_name, deadline) + .await + .map_err(KubernetesDriverError::Message)?; + let stop_is_complete = kubernetes_sandbox_stop_is_complete( + &agent_sandbox_api.resource.version, + &object, + pod_is_gone, + ); + if stop_is_complete { + self.delete_sandbox_runtime_supervisor(sandbox_id, &namespace) + .await?; + patch_dynamic_object_with_resource_version_retry( + &agent_sandbox_api.api, + &kube_name, + sandbox_runtime_rollback_completion_patch, + ) + .await?; + return Ok(()); + } + let now = tokio::time::Instant::now(); + if now >= deadline { + return Err(KubernetesDriverError::Message(format!( + "timed out after {}s waiting for Kubernetes sandbox to stop", + stop_timeout.as_secs() + ))); + } + tokio::time::sleep(poll_interval.min(deadline.saturating_duration_since(now))).await; + poll_interval = next_stop_poll_interval(poll_interval); + } + } + + #[tracing::instrument( + name = "kubernetes.start_sandbox", + skip_all, + fields( + otel.name = "kubernetes.start_sandbox", + otel.status_code = tracing::field::Empty, + sandbox.id = %sandbox_id, + ) + )] + pub async fn start_sandbox( + &self, + sandbox_id: &str, + generation_id: &str, + launch_authentication: &[u8], + ) -> Result<(), KubernetesDriverError> { + let span_status = openshell_otel::ErrorStatusGuard::current(); + let result = Box::pin(self.start_sandbox_runtime_generation( + sandbox_id, + generation_id, + launch_authentication, + )) + .await; + span_status.finish(result) + } + + #[allow(clippy::similar_names)] + async fn start_sandbox_runtime_generation( + &self, + sandbox_id: &str, + encoded_generation: &str, + encoded_authentication: &[u8], + ) -> Result<(), KubernetesDriverError> { + let generation = openshell_core::sandbox_generation::SandboxGenerationId::parse( + encoded_generation.to_string(), + ) + .map_err(|error| KubernetesDriverError::InvalidArgument(error.to_string()))?; + let launch_authentication = decode_launch_authentication(encoded_authentication)?; + let lookup_api = self + .supported_sandbox_api_for_lookup(self.client.clone()) + .await + .map_err(KubernetesDriverError::Message)?; + let selector = self.sandbox_lookup_selector(sandbox_id); + let mut objects = lookup_api + .api + .list(&ListParams::default().labels(&selector)) + .await + .map_err(KubernetesDriverError::from_kube)? + .items; + let mut object = objects.pop().ok_or(KubernetesDriverError::NotFound)?; + if sandbox_runtime_bootstrap_in_progress(&object) { + let phase = sandbox_runtime_bootstrap_phase(&object); + if phase != Some(SandboxRuntimeBootstrapPhase::RollingBack) + && sandbox_runtime_bootstrap_operation(&object) != Some("restart") + { + return Err(KubernetesDriverError::Precondition( + "initial sandbox bootstrap has not completed; wait for reconciliation" + .to_string(), + )); + } + if sandbox_runtime_generation(&object) == Some(generation.as_str()) + && sandbox_runtime_bootstrap_phase(&object) + == Some(SandboxRuntimeBootstrapPhase::Released) + { + let namespace = object + .metadata + .namespace + .as_deref() + .unwrap_or(&self.config.namespace); + if sandbox_runtime_control_availability(&self.client, namespace, sandbox_id).await + == SandboxRuntimeControlAvailability::Available + && sandbox_runtime_runtime_is_ready(&object) + { + self.complete_sandbox_runtime_bootstrap(&lookup_api, &object) + .await; + return Ok(()); + } + } + + // A retry adopts the durable generation identity, but reconstructs + // its resources from a clean suspended state. This is safe even + // after gateway replacement because launch credentials stay in + // memory and are supplied again by the caller. + self.stop_sandbox_inner(sandbox_id).await?; + let mut refreshed = lookup_api + .api + .list(&ListParams::default().labels(&selector)) + .await + .map_err(KubernetesDriverError::from_kube)? + .items; + object = refreshed.pop().ok_or(KubernetesDriverError::NotFound)?; + } + let namespace = object + .metadata + .namespace + .clone() + .unwrap_or_else(|| self.config.namespace.clone()); + let cr_name = object.metadata.name.as_deref().ok_or_else(|| { + KubernetesDriverError::Message("sandbox resource has no name".to_string()) + })?; + let cr_uid = object.metadata.uid.as_deref().ok_or_else(|| { + KubernetesDriverError::Message("sandbox resource has no UID".to_string()) + })?; + let sandbox_api = Self::agent_sandbox_api( + self.client.clone(), + &lookup_api.resource.version, + &namespace, + ); + let pods = Api::::namespaced(self.client.clone(), &namespace); + if pods + .get_opt(cr_name) + .await + .map_err(KubernetesDriverError::from_kube)? + .is_some() + { + if !encoded_authentication.is_empty() { + // Gateway restart creates a fresh in-memory launch session. + // Recreate both Pods so neither side retains credentials for + // the gateway instance that was replaced. + self.stop_sandbox_inner(sandbox_id).await?; + return Box::pin(self.start_sandbox_runtime_generation( + sandbox_id, + encoded_generation, + encoded_authentication, + )) + .await; + } + return Err(KubernetesDriverError::Precondition( + "cannot rotate a running workload Pod; stop the sandbox first".to_string(), + )); + } + + let names = SandboxRuntimeNames::for_generation(sandbox_id, generation.as_str()); + self.create_sandbox_runtime_fence(&namespace, &names) + .await?; + self.delete_sandbox_runtime_supervisor(sandbox_id, &namespace) + .await?; + let main_process_spec = + required_sandbox_annotation(&object, ANNOTATION_SANDBOX_RUNTIME_MAIN_PROCESS_SPEC)?; + let log_level = required_sandbox_annotation(&object, ANNOTATION_SANDBOX_RUNTIME_LOG_LEVEL)?; + let sandbox_name = annotation_or_label(&object, LABEL_SANDBOX_NAME).ok_or_else(|| { + KubernetesDriverError::Precondition( + "Sandbox resource is missing its OpenShell sandbox name".to_string(), + ) + })?; + let (agent_uid, agent_gid, _) = + self.resolve_sandbox_identity_in_namespace(&namespace).await; + let supervisor = pods + .create( + &PostParams::default(), + &supervisor_pod( + &namespace, + &names, + sandbox_id, + &sandbox_name, + &self.config.gateway_id, + &self.config.supervisor_image, + self.config.supervisor_image_pull_policy, + &self.config.service_account_name, + agent_uid, + agent_gid, + &self.config.image_pull_secrets, + &self.config.grpc_endpoint, + &self.config.client_tls_secret_name, + &main_process_spec, + &log_level, + self.config.effective_sa_token_ttl_secs(), + self.config.https_proxy.as_deref(), + self.config.no_proxy.as_deref(), + self.config + .proxy_auth_secret_name + .as_deref() + .zip(self.config.proxy_auth_secret_key.as_deref()), + self.config.proxy_auth_allow_insecure == Some(true), + self.config.proxy_connect_by_hostname == Some(true), + self.config.provider_spiffe_enabled().then_some( + self.config + .provider_spiffe_workload_api_socket_path + .as_str(), + ), + sandbox_runtime_sandbox_owner_reference( + cr_name, + cr_uid, + &sandbox_api.resource.api_version, + false, + ), + ) + .map_err(KubernetesDriverError::Message)?, + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + let supervisor_uid = supervisor.metadata.uid.ok_or_else(|| { + KubernetesDriverError::Message("replacement supervisor Pod has no UID".to_string()) + })?; + + let mut volumes = object + .data + .pointer("/spec/podTemplate/spec/volumes") + .and_then(serde_json::Value::as_array) + .cloned() + .ok_or_else(|| { + KubernetesDriverError::Precondition( + "Sandbox pod template has no volume list".to_string(), + ) + })?; + let sandbox_secret = volumes + .iter_mut() + .find(|volume| { + volume.get("name").and_then(serde_json::Value::as_str) + == Some(SANDBOX_BOOTSTRAP_VOLUME_NAME) + }) + .and_then(|volume| volume.get_mut("secret")) + .ok_or_else(|| { + KubernetesDriverError::Precondition( + "Sandbox pod template is missing its bootstrap Secret volume".to_string(), + ) + })?; + sandbox_secret["secretName"] = serde_json::json!(names.sandbox_secret); + let restart = async { + patch_dynamic_object_with_resource_version_retry( + &sandbox_api.api, + cr_name, + |version| { + let mut running_patch = + sandbox_operating_state_patch(&sandbox_api.resource.version, version, true); + running_patch["metadata"]["annotations"] = serde_json::json!({ + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING: "true", + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT: openshell_core::time::now_ms().to_string(), + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION: "restart", + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE: SandboxRuntimeBootstrapPhase::Preparing.as_str(), + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_SESSION: launch_authentication.supervisor.session_id.to_string(), + ANNOTATION_SANDBOX_RUNTIME_GENERATION: generation.as_str(), + ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: supervisor_uid.clone(), + }); + running_patch["spec"]["podTemplate"]["spec"]["volumes"] = + serde_json::Value::Array(volumes.clone()); + running_patch + }, + ) + .await?; + + let child_env = child_environment_from_sandbox_object(&object); + self.install_sandbox_runtime_generation( + &namespace, + cr_name, + &sandbox_api, + sandbox_id, + cr_uid, + &names, + generation.as_str(), + &supervisor_uid, + agent_uid, + agent_gid, + child_env, + &launch_authentication, + ) + .await + } + .await; + + if let Err(error) = restart { + if let Err(rollback_error) = self.stop_sandbox_inner(sandbox_id).await { + return Err(KubernetesDriverError::Message(format!( + "restart failed: {error}; rollback remains pending: {rollback_error}" + ))); + } + return Err(error); + } + Ok(()) + } + + async fn patch_sandbox_operating_state( + &self, + sandbox_id: &str, + running: bool, + ) -> Result<(AgentSandboxApi, String, String, String, Duration), KubernetesDriverError> { + let lookup_api = self + .supported_sandbox_api_for_lookup(self.client.clone()) + .await + .map_err(KubernetesDriverError::Message)?; + let selector = self.sandbox_lookup_selector(sandbox_id); + let list = tokio::time::timeout( KUBE_API_TIMEOUT, - agent_sandbox_api.api.patch( - &kube_name, - &PatchParams::default(), - &Patch::Merge(&desired), - ), + lookup_api + .api + .list(&ListParams::default().labels(&selector)), ) .await .map_err(|_| { @@ -1780,6 +3061,40 @@ impl KubernetesComputeDriver { )) })? .map_err(KubernetesDriverError::from_kube)?; + let object = list + .items + .into_iter() + .next() + .ok_or(KubernetesDriverError::NotFound)?; + let namespace = object + .metadata + .namespace + .clone() + .unwrap_or_else(|| self.config.namespace.clone()); + let agent_sandbox_api = Self::agent_sandbox_api( + self.client.clone(), + &lookup_api.resource.version, + &namespace, + ); + let stop_timeout = kubernetes_sandbox_stop_timeout(&object); + let kube_name = object.metadata.name.clone().ok_or_else(|| { + KubernetesDriverError::Message("sandbox resource has no name".to_string()) + })?; + let pod_name = object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(SANDBOX_POD_NAME_ANNOTATION)) + .cloned() + .unwrap_or_else(|| kube_name.clone()); + patch_dynamic_object_with_resource_version_retry( + &agent_sandbox_api.api, + &kube_name, + |version| { + sandbox_operating_state_patch(&agent_sandbox_api.resource.version, version, running) + }, + ) + .await?; info!( sandbox_id, @@ -1796,6 +3111,84 @@ impl KubernetesComputeDriver { )) } + async fn delete_sandbox_runtime_supervisor( + &self, + sandbox_id: &str, + namespace: &str, + ) -> Result<(), KubernetesDriverError> { + let names = SandboxRuntimeNames::new(sandbox_id); + let pods = Api::::namespaced(self.client.clone(), namespace); + if let Some(pod) = pods + .get_opt(&names.supervisor_pod) + .await + .map_err(KubernetesDriverError::from_kube)? + { + pods.delete( + &names.supervisor_pod, + &DeleteParams::foreground().preconditions(Preconditions { + uid: pod.metadata.uid, + resource_version: None, + }), + ) + .await + .map_err(KubernetesDriverError::from_kube)?; + let deadline = tokio::time::Instant::now() + KUBE_API_TIMEOUT; + loop { + if pods + .get_opt(&names.supervisor_pod) + .await + .map_err(KubernetesDriverError::from_kube)? + .is_none() + { + break; + } + if tokio::time::Instant::now() >= deadline { + return Err(KubernetesDriverError::Message( + "timed out waiting for supervisor Pod deletion".to_string(), + )); + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + self.delete_sandbox_runtime_generation_secrets(sandbox_id, namespace) + .await + } + + async fn delete_sandbox_runtime_generation_secrets( + &self, + sandbox_id: &str, + namespace: &str, + ) -> Result<(), KubernetesDriverError> { + let secrets = Api::::namespaced(self.client.clone(), namespace); + for component in [SANDBOX_SECRET_COMPONENT, SUPERVISOR_SECRET_COMPONENT] { + let selector = + format!("openshell.ai/sandbox-id={sandbox_id},openshell.ai/component={component}"); + let items = secrets + .list(&ListParams::default().labels(&selector)) + .await + .map_err(KubernetesDriverError::from_kube)?; + for secret in items { + let Some(name) = secret.metadata.name else { + continue; + }; + match secrets + .delete( + &name, + &DeleteParams::default().preconditions(Preconditions { + uid: secret.metadata.uid, + resource_version: None, + }), + ) + .await + { + Ok(_) | Err(KubeError::Api(kube::core::ErrorResponse { code: 404, .. })) => {} + Err(error) => return Err(KubernetesDriverError::from_kube(error)), + } + } + } + Ok(()) + } + #[tracing::instrument( name = "kubernetes.delete_sandbox", skip(self), @@ -1823,115 +3216,657 @@ impl KubernetesComputeDriver { .await?; let selector = self.sandbox_lookup_selector(sandbox_id); let lp = ListParams::default().labels(&selector); - let (kube_name, obj_namespace, _workspace, preconditions) = match tokio::time::timeout( + let (kube_name, obj_namespace, _workspace, preconditions, pod_name) = + match tokio::time::timeout(KUBE_API_TIMEOUT, lookup_api.api.list(&lp)).await { + Ok(Ok(list)) => { + if let Some(obj) = list.items.into_iter().next() { + match obj.metadata.name.clone() { + Some(name) => { + let ns = obj + .metadata + .namespace + .clone() + .unwrap_or_else(|| self.config.namespace.clone()); + let ws = obj + .metadata + .labels + .as_ref() + .and_then(|l| l.get(LABEL_SANDBOX_WORKSPACE).cloned()) + .unwrap_or_default(); + let pc = Preconditions { + uid: obj.metadata.uid, + resource_version: obj.metadata.resource_version, + }; + let pod_name = obj + .metadata + .annotations + .as_ref() + .and_then(|annotations| { + annotations.get(SANDBOX_POD_NAME_ANNOTATION) + }) + .cloned() + .unwrap_or_else(|| name.clone()); + (name, ns, ws, pc, pod_name) + } + None => return Ok(false), + } + } else { + debug!(sandbox_id = %sandbox_id, "Sandbox not found in Kubernetes (already deleted)"); + return Ok(false); + } + } + Ok(Err(err)) => { + warn!( + sandbox_id = %sandbox_id, + error = %err, + "Failed to list sandbox for deletion from Kubernetes" + ); + return Err(err.to_string()); + } + Err(_elapsed) => { + warn!( + sandbox_id = %sandbox_id, + timeout_secs = KUBE_API_TIMEOUT.as_secs(), + "Timed out listing sandbox for deletion from Kubernetes" + ); + return Err(format!( + "timed out after {}s waiting for Kubernetes API", + KUBE_API_TIMEOUT.as_secs() + )); + } + }; + + let delete_api = self + .supported_agent_sandbox_api(self.client.clone(), &obj_namespace) + .await?; + let dp = DeleteParams::default().preconditions(preconditions); + match tokio::time::timeout(KUBE_API_TIMEOUT, delete_api.api.delete(&kube_name, &dp)).await { + Ok(Ok(_response)) => { + info!(sandbox_id = %sandbox_id, namespace = %obj_namespace, "Sandbox deleted from Kubernetes"); + { + let pod_api = Api::::namespaced(self.client.clone(), &obj_namespace); + let deadline = tokio::time::Instant::now() + + DEFAULT_POD_TERMINATION_GRACE_PERIOD + + KUBE_API_TIMEOUT; + loop { + match kubernetes_sandbox_pod_is_gone(&pod_api, &pod_name, deadline).await { + Ok(true) => break, + Ok(false) if tokio::time::Instant::now() < deadline => { + tokio::time::sleep(STOP_INITIAL_POLL_INTERVAL).await; + } + Ok(false) | Err(_) => { + warn!( + sandbox_id, + "retaining sandbox-runtime workload fence because workload Pod deletion was not confirmed" + ); + return Ok(true); + } + } + } + // API acceptance of DELETE does not mean the CR is gone. + // Keep the unowned fence while finalizers can still leave + // the controller able to reconcile a workload Pod. + if !matches!( + tokio::time::timeout(KUBE_API_TIMEOUT, delete_api.api.get(&kube_name)) + .await, + Ok(Err(KubeError::Api(kube::core::ErrorResponse { + code: 404, + .. + }))) + ) { + warn!(sandbox_id, "Sandbox CR deletion was not confirmed"); + return Ok(true); + } + } + Ok(true) + } + Ok(Err(KubeError::Api(err))) if err.code == 404 || err.code == 409 => { + debug!(sandbox_id = %sandbox_id, "Sandbox not found in Kubernetes (already deleted or replaced)"); + Ok(false) + } + Ok(Err(err)) => { + warn!( + sandbox_id = %sandbox_id, + error = %err, + "Failed to delete sandbox from Kubernetes" + ); + Err(err.to_string()) + } + Err(_elapsed) => { + warn!( + sandbox_id = %sandbox_id, + timeout_secs = KUBE_API_TIMEOUT.as_secs(), + "Timed out deleting sandbox from Kubernetes" + ); + Err(format!( + "timed out after {}s waiting for Kubernetes API", + KUBE_API_TIMEOUT.as_secs() + )) + } + } + } + + pub async fn sandbox_exists(&self, sandbox_id: &str) -> Result { + let agent_sandbox_api = self + .supported_sandbox_api_for_lookup(self.client.clone()) + .await?; + let selector = self.sandbox_lookup_selector(sandbox_id); + let lp = ListParams::default().labels(&selector); + match tokio::time::timeout(KUBE_API_TIMEOUT, agent_sandbox_api.api.list(&lp)).await { + Ok(Ok(list)) => Ok(!list.items.is_empty()), + Ok(Err(err)) => Err(err.to_string()), + Err(_elapsed) => Err(format!( + "timed out after {}s waiting for Kubernetes API", + KUBE_API_TIMEOUT.as_secs() + )), + } + } + + /// Repair lifecycle drift and reap unowned workload fences while the + /// gateway's sandbox watch is alive. Bootstrap material is immutable and + /// intentionally not read by the gateway, so this pass only repairs state + /// that can be proven from the Sandbox CR and named companion objects. + async fn reconcile_sandbox_runtime_resources(&self) { + let lookup_api = match self + .supported_sandbox_api_for_lookup(self.client.clone()) + .await + { + Ok(api) => api, + Err(error) => { + warn!(%error, "skipping sandbox-runtime reconciliation: Sandbox API unavailable"); + return; + } + }; + let list = match tokio::time::timeout( + KUBE_API_TIMEOUT, + lookup_api + .api + .list(&ListParams::default().labels(&self.openshell_sandbox_selector())), + ) + .await + { + Ok(Ok(list)) => list, + Ok(Err(error)) => { + warn!(%error, "skipping sandbox-runtime reconciliation: Sandbox list failed"); + return; + } + Err(_) => { + warn!("skipping sandbox-runtime reconciliation: Sandbox list timed out"); + return; + } + }; + + for object in list.items { + let Ok(sandbox_id) = sandbox_id_from_object(&object) else { + continue; + }; + let namespace = object + .metadata + .namespace + .as_deref() + .unwrap_or(&self.config.namespace); + let cr_name = object.metadata.name.as_deref().unwrap_or_default(); + if sandbox_runtime_bootstrap_phase(&object) + == Some(SandboxRuntimeBootstrapPhase::RollingBack) + { + self.reconcile_sandbox_runtime_rollback( + &lookup_api, + &object, + &sandbox_id, + namespace, + cr_name, + ) + .await; + continue; + } + let names = SandboxRuntimeNames::new(&sandbox_id); + let policies = Api::::namespaced(self.client.clone(), namespace); + match self.create_sandbox_runtime_fence(namespace, &names).await { + Ok(()) => {} + Err(KubernetesDriverError::Precondition(error)) => { + warn!(sandbox_id, %error, "sandbox-runtime workload fence is altered; suspending workload"); + self.suspend_sandbox_runtime_after_dependency_failure(&lookup_api, &object) + .await; + continue; + } + Err(error) => { + warn!(sandbox_id, %error, "could not verify sandbox-runtime workload fence; reconciliation will retry"); + continue; + } + } + let Ok(Ok(fence)) = + tokio::time::timeout(KUBE_API_TIMEOUT, policies.get(&names.workload_policy)).await + else { + continue; + }; + if !sandbox_runtime_namespace_fence_generation_matches(&fence, &object) { + warn!( + sandbox_id, + "namespace workload fence generation changed; suspending stale boundary" + ); + self.suspend_sandbox_runtime_after_dependency_failure(&lookup_api, &object) + .await; + continue; + } + if sandbox_runtime_bootstrap_in_progress(&object) { + if sandbox_runtime_control_availability(&self.client, namespace, &sandbox_id).await + == SandboxRuntimeControlAvailability::Available + && sandbox_runtime_runtime_is_ready(&object) + { + self.complete_sandbox_runtime_bootstrap(&lookup_api, &object) + .await; + } else { + self.reap_stale_sandbox_runtime_bootstrap(&lookup_api, &object) + .await; + } + continue; + } + let desired_running = sandbox_runtime_should_run(&object); + if desired_running { + match sandbox_runtime_workload_generation_matches( + &self.client, + namespace, + cr_name, + &object, + ) + .await + { + SandboxRuntimeControlAvailability::Available => {} + SandboxRuntimeControlAvailability::Unavailable => { + warn!( + sandbox_id, + "sandbox-runtime workload generation changed; suspending stale boundary" + ); + self.suspend_sandbox_runtime_after_dependency_failure(&lookup_api, &object) + .await; + continue; + } + SandboxRuntimeControlAvailability::Unknown => { + warn!( + sandbox_id, + "could not verify sandbox-runtime workload generation; reconciliation will retry" + ); + continue; + } + } + match sandbox_runtime_supervisor_generation_matches( + &self.client, + namespace, + &sandbox_id, + &object, + ) + .await + { + SandboxRuntimeControlAvailability::Available => {} + SandboxRuntimeControlAvailability::Unavailable => { + warn!( + sandbox_id, + "sandbox-runtime supervisor generation changed; suspending stale boundary" + ); + self.suspend_sandbox_runtime_after_dependency_failure(&lookup_api, &object) + .await; + continue; + } + SandboxRuntimeControlAvailability::Unknown => continue, + } + } + match self + .reconcile_sandbox_runtime_supervisor(&sandbox_id, namespace, desired_running) + .await + { + Ok(()) => {} + Err(KubernetesDriverError::Precondition(error)) => { + warn!(sandbox_id, %error, "sandbox-runtime companion is missing; suspending workload"); + self.suspend_sandbox_runtime_after_dependency_failure(&lookup_api, &object) + .await; + } + Err(error) => { + warn!(sandbox_id, %error, "failed to reconcile sandbox-runtime supervisor Pod"); + } + } + let availability = + sandbox_runtime_control_availability(&self.client, namespace, &sandbox_id).await; + if desired_running && availability == SandboxRuntimeControlAvailability::Unavailable { + warn!( + sandbox_id, + "sandbox-runtime supervisor is unavailable; suspending workload" + ); + self.suspend_sandbox_runtime_after_dependency_failure(&lookup_api, &object) + .await; + continue; + } + self.publish_sandbox_runtime_readiness_transition(&lookup_api, &object, availability) + .await; + } + } + + async fn reconcile_sandbox_runtime_rollback( + &self, + lookup_api: &AgentSandboxApi, + object: &DynamicObject, + sandbox_id: &str, + namespace: &str, + cr_name: &str, + ) { + let pod_name = object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(SANDBOX_POD_NAME_ANNOTATION)) + .map_or(cr_name, String::as_str); + let pod_api = Api::::namespaced(self.client.clone(), namespace); + let deadline = tokio::time::Instant::now() + KUBE_API_TIMEOUT; + match kubernetes_sandbox_pod_is_gone(&pod_api, pod_name, deadline).await { + Ok(true) => {} + Ok(false) => return, + Err(error) => { + debug!(sandbox_id, %error, "could not verify sandbox-runtime rollback; reconciliation will retry"); + return; + } + } + if let Err(error) = self + .delete_sandbox_runtime_supervisor(sandbox_id, namespace) + .await + { + warn!(sandbox_id, %error, "could not finish sandbox-runtime rollback cleanup"); + return; + } + let Some(resource_version) = object.metadata.resource_version.as_deref() else { + return; + }; + let api = + Self::agent_sandbox_api(self.client.clone(), &lookup_api.resource.version, namespace); + let patch = sandbox_runtime_rollback_completion_patch(resource_version); + match tokio::time::timeout( + KUBE_API_TIMEOUT, + api.api + .patch(cr_name, &PatchParams::default(), &Patch::Merge(&patch)), + ) + .await + { + Ok(Ok(_)) => {} + Ok(Err(error)) => { + debug!(sandbox_id, %error, "sandbox-runtime rollback completion raced; reconciliation will retry"); + } + Err(_) => { + warn!(sandbox_id, "timed out completing sandbox-runtime rollback"); + } + } + } + + async fn publish_sandbox_runtime_readiness_transition( + &self, + lookup_api: &AgentSandboxApi, + object: &DynamicObject, + availability: SandboxRuntimeControlAvailability, + ) { + let state = match availability { + SandboxRuntimeControlAvailability::Available => "ready", + SandboxRuntimeControlAvailability::Unavailable => "unavailable", + SandboxRuntimeControlAvailability::Unknown => return, + }; + if object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_READINESS)) + .is_some_and(|current| current == state) + { + return; + } + let (Some(name), Some(resource_version)) = ( + object.metadata.name.as_deref(), + object.metadata.resource_version.as_deref(), + ) else { + return; + }; + let namespace = object + .metadata + .namespace + .as_deref() + .unwrap_or(&self.config.namespace); + let api = + Self::agent_sandbox_api(self.client.clone(), &lookup_api.resource.version, namespace); + let patch = sandbox_runtime_readiness_transition_patch(resource_version, state); + match tokio::time::timeout( KUBE_API_TIMEOUT, - lookup_api.api.list(&lp), + api.api + .patch(name, &PatchParams::default(), &Patch::Merge(&patch)), ) .await { - Ok(Ok(list)) => { - if let Some(obj) = list.items.into_iter().next() { - match obj.metadata.name { - Some(name) => { - let ns = obj - .metadata - .namespace - .clone() - .unwrap_or_else(|| self.config.namespace.clone()); - let ws = obj - .metadata - .labels - .as_ref() - .and_then(|l| l.get(LABEL_SANDBOX_WORKSPACE).cloned()) - .unwrap_or_default(); - let pc = Preconditions { - uid: obj.metadata.uid, - resource_version: obj.metadata.resource_version, - }; - (name, ns, ws, pc) - } - None => return Ok(false), - } - } else { - debug!(sandbox_id = %sandbox_id, "Sandbox not found in Kubernetes (already deleted)"); - return Ok(false); - } - } - Ok(Err(err)) => { - warn!( - sandbox_id = %sandbox_id, - error = %err, - "Failed to list sandbox for deletion from Kubernetes" - ); - return Err(err.to_string()); + Ok(Ok(_)) => {} + Ok(Err(error)) => { + debug!(sandbox = name, %error, "sandbox-runtime readiness transition publication raced; reconciliation will retry"); } - Err(_elapsed) => { + Err(_) => { warn!( - sandbox_id = %sandbox_id, - timeout_secs = KUBE_API_TIMEOUT.as_secs(), - "Timed out listing sandbox for deletion from Kubernetes" + sandbox = name, + "timed out publishing sandbox-runtime readiness transition" ); - return Err(format!( - "timed out after {}s waiting for Kubernetes API", - KUBE_API_TIMEOUT.as_secs() - )); } - }; + } + } - let delete_api = self - .supported_agent_sandbox_api(self.client.clone(), &obj_namespace) - .await?; - let dp = DeleteParams::default().preconditions(preconditions); - match tokio::time::timeout(KUBE_API_TIMEOUT, delete_api.api.delete(&kube_name, &dp)).await { - Ok(Ok(_response)) => { - info!(sandbox_id = %sandbox_id, namespace = %obj_namespace, "Sandbox deleted from Kubernetes"); - Ok(true) + async fn suspend_sandbox_runtime_after_dependency_failure( + &self, + lookup_api: &AgentSandboxApi, + object: &DynamicObject, + ) { + let Some(name) = object.metadata.name.as_deref() else { + return; + }; + let namespace = object + .metadata + .namespace + .as_deref() + .unwrap_or(&self.config.namespace); + let api = + Self::agent_sandbox_api(self.client.clone(), &lookup_api.resource.version, namespace); + let patch = sandbox_runtime_rollback_patch( + &lookup_api.resource.version, + object + .metadata + .resource_version + .as_deref() + .unwrap_or_default(), + ); + match tokio::time::timeout( + KUBE_API_TIMEOUT, + api.api + .patch(name, &PatchParams::default(), &Patch::Merge(&patch)), + ) + .await + { + Ok(Ok(_)) => {} + Ok(Err(error)) => { + warn!(sandbox = name, %error, "failed to suspend sandbox-runtime after fence failure"); } - Ok(Err(KubeError::Api(err))) if err.code == 404 || err.code == 409 => { - debug!(sandbox_id = %sandbox_id, "Sandbox not found in Kubernetes (already deleted or replaced)"); - Ok(false) + Err(error) => { + warn!(sandbox = name, %error, "timed out suspending sandbox-runtime after fence failure"); } - Ok(Err(err)) => { - warn!( - sandbox_id = %sandbox_id, - error = %err, - "Failed to delete sandbox from Kubernetes" - ); - Err(err.to_string()) + } + } + + async fn reap_stale_sandbox_runtime_bootstrap( + &self, + lookup_api: &AgentSandboxApi, + object: &DynamicObject, + ) { + if !sandbox_runtime_bootstrap_is_stale( + object, + SystemTime::now(), + SANDBOX_RUNTIME_BOOTSTRAP_GRACE, + ) { + return; + } + if sandbox_runtime_bootstrap_operation(object) != Some("create") { + warn!( + sandbox = object.metadata.name.as_deref().unwrap_or(""), + "suspending stale sandbox-runtime restart bootstrap" + ); + self.suspend_sandbox_runtime_after_dependency_failure(lookup_api, object) + .await; + return; + } + let (Some(name), Some(uid), Some(resource_version)) = ( + object.metadata.name.as_deref(), + object.metadata.uid.clone(), + object.metadata.resource_version.clone(), + ) else { + return; + }; + let namespace = object + .metadata + .namespace + .as_deref() + .unwrap_or(&self.config.namespace); + let api = + Self::agent_sandbox_api(self.client.clone(), &lookup_api.resource.version, namespace); + let params = DeleteParams::default().preconditions(Preconditions { + uid: Some(uid), + resource_version: Some(resource_version), + }); + match tokio::time::timeout(KUBE_API_TIMEOUT, api.api.delete(name, ¶ms)).await { + Ok(Ok(_)) => warn!( + sandbox = name, + "rolled back stale fail-closed sandbox-runtime bootstrap" + ), + Ok(Err(KubeError::Api(error))) if error.code == 404 || error.code == 409 => {} + Ok(Err(error)) => { + warn!(sandbox = name, %error, "failed to roll back stale sandbox-runtime bootstrap"); } - Err(_elapsed) => { + Err(_) => { warn!( - sandbox_id = %sandbox_id, - timeout_secs = KUBE_API_TIMEOUT.as_secs(), - "Timed out deleting sandbox from Kubernetes" + sandbox = name, + "timed out rolling back stale sandbox-runtime bootstrap" ); - Err(format!( - "timed out after {}s waiting for Kubernetes API", - KUBE_API_TIMEOUT.as_secs() - )) } } } - pub async fn sandbox_exists(&self, sandbox_id: &str) -> Result { - let agent_sandbox_api = self - .supported_sandbox_api_for_lookup(self.client.clone()) - .await?; - let selector = self.sandbox_lookup_selector(sandbox_id); - let lp = ListParams::default().labels(&selector); - match tokio::time::timeout(KUBE_API_TIMEOUT, agent_sandbox_api.api.list(&lp)).await { - Ok(Ok(list)) => Ok(!list.items.is_empty()), - Ok(Err(err)) => Err(err.to_string()), - Err(_elapsed) => Err(format!( - "timed out after {}s waiting for Kubernetes API", - KUBE_API_TIMEOUT.as_secs() - )), + async fn complete_sandbox_runtime_bootstrap( + &self, + lookup_api: &AgentSandboxApi, + object: &DynamicObject, + ) { + let (Some(name), Some(resource_version)) = ( + object.metadata.name.as_deref(), + object.metadata.resource_version.as_deref(), + ) else { + return; + }; + let namespace = object + .metadata + .namespace + .as_deref() + .unwrap_or(&self.config.namespace); + let api = + Self::agent_sandbox_api(self.client.clone(), &lookup_api.resource.version, namespace); + let patch = sandbox_runtime_bootstrap_completion_patch(resource_version); + match tokio::time::timeout( + KUBE_API_TIMEOUT, + api.api + .patch(name, &PatchParams::default(), &Patch::Merge(&patch)), + ) + .await + { + Ok(Ok(_)) => {} + Ok(Err(error)) => { + debug!(sandbox = name, %error, "sandbox-runtime bootstrap completion raced; reconciliation will retry"); + } + Err(_) => warn!( + sandbox = name, + "timed out completing sandbox-runtime bootstrap" + ), + } + } + + async fn reconcile_sandbox_runtime_supervisor( + &self, + sandbox_id: &str, + namespace: &str, + desired_running: bool, + ) -> Result<(), KubernetesDriverError> { + if !desired_running { + return self + .delete_sandbox_runtime_supervisor(sandbox_id, namespace) + .await; + } + let names = SandboxRuntimeNames::new(sandbox_id); + let services = Api::::namespaced(self.client.clone(), namespace); + let service_exists = + tokio::time::timeout(KUBE_API_TIMEOUT, services.get_opt(&names.boundary_service)) + .await + .map_err(|_| { + KubernetesDriverError::Message( + "timed out reading sandbox-runtime boundary Service".to_string(), + ) + })? + .map_err(KubernetesDriverError::from_kube)? + .is_some(); + if !service_exists { + return Err(KubernetesDriverError::Precondition(format!( + "sandbox-runtime boundary Service {} is missing and its allocated address cannot be safely reconstructed", + names.boundary_service + ))); } + let pods = Api::::namespaced(self.client.clone(), namespace); + tokio::time::timeout( + KUBE_API_TIMEOUT, + pods.get_opt(&names.supervisor_pod), + ) + .await + .map_err(|_| { + KubernetesDriverError::Message( + "timed out reading sandbox-runtime supervisor Pod".to_string(), + ) + })? + .map_err(KubernetesDriverError::from_kube)? + .ok_or_else(|| { + KubernetesDriverError::Precondition(format!( + "sandbox-runtime supervisor Pod {} is missing; start a fresh sandbox generation", + names.supervisor_pod + )) + })?; + Ok(()) + } + + fn spawn_sandbox_runtime_periodic_reconcile( + &self, + tx: mpsc::Sender>, + ) { + let driver = self.clone(); + tokio::spawn(async move { + let mut interval = tokio::time::interval(SANDBOX_RUNTIME_RECONCILE_INTERVAL); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + interval.tick().await; + loop { + tokio::select! { + _ = interval.tick() => { + driver.reconcile_sandbox_runtime_resources().await; + if let Ok(sandboxes) = driver.list_sandboxes().await { + for sandbox in sandboxes { + if tx.send(Ok(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { sandbox: Some(sandbox) }, + )), + })).await.is_err() { + return; + } + } + } + } + () = tx.closed() => return, + } + } + }); } // Kept `async` to match the gRPC handler signature in `grpc.rs`, which awaits this method. #[allow(clippy::unused_async)] pub async fn watch_sandboxes(&self) -> Result { + self.reconcile_sandbox_runtime_resources().await; if self.config.is_multi_namespace() { self.watch_sandboxes_cluster_wide().await } else { @@ -1957,6 +3892,8 @@ impl KubernetesComputeDriver { ) .boxed(); let (tx, rx) = mpsc::channel(256); + self.spawn_sandbox_runtime_periodic_reconcile(tx.clone()); + let readiness_client = self.watch_client.clone(); tokio::spawn(async move { let mut sandbox_name_to_id = std::collections::HashMap::::new(); @@ -1966,7 +3903,7 @@ impl KubernetesComputeDriver { tokio::select! { event = sandbox_stream.next() => match event { Some(Event::Apply(obj) | Event::InitApply(obj)) => { - if let Ok((kube_name, sandbox)) = sandbox_from_object(&namespace, obj) { + if let Ok((kube_name, sandbox)) = sandbox_from_object_with_sandbox_runtime_readiness(&readiness_client, &namespace, obj).await { update_indexes(&mut sandbox_name_to_id, &mut agent_pod_to_id, &kube_name, &sandbox); let event = WatchSandboxesEvent { payload: Some(watch_sandboxes_event::Payload::Sandbox( @@ -2002,7 +3939,7 @@ impl KubernetesComputeDriver { } }, event = event_stream.next() => match event { - Some(Event::Apply(obj)) => { + Some(Event::Apply(obj) | Event::InitApply(obj)) => { if let Some((sandbox_id, event)) = map_kube_event_to_platform( &sandbox_name_to_id, &agent_pod_to_id, @@ -2018,8 +3955,8 @@ impl KubernetesComputeDriver { } } } - Some(Event::Delete(_) | Event::InitApply(_) | Event::InitDone) => {} - Some(Event::Init) => { + Some(Event::Delete(_)) => {} + Some(Event::Init | Event::InitDone) => { debug!(namespace = %namespace, "Kubernetes event watcher restarted"); } None => { @@ -2054,15 +3991,222 @@ impl KubernetesComputeDriver { Ok(cluster_wide_watch_stream( sandbox_stream, self.config.namespace.clone(), + self.watch_client.clone(), + self.clone(), )) } } -fn cluster_wide_watch_stream(mut sandbox_stream: S, default_namespace: String) -> WatchStream +fn sandbox_runtime_bootstrap_completion_patch(resource_version: &str) -> serde_json::Value { + serde_json::json!({ + "metadata": { + "resourceVersion": resource_version, + "annotations": { + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_SESSION: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_READINESS: "ready", + } + } + }) +} + +fn spawn_sandbox_runtime_bootstrap_completion( + pods: Api, + sandboxes: Api, + supervisor_pod_name: String, + sandbox_name: String, + expected_sandbox_uid: Option, +) { + tokio::spawn(async move { + let deadline = tokio::time::Instant::now() + SANDBOX_RUNTIME_BOOTSTRAP_GRACE; + let available = |pod: Option<&Pod>| { + pod.is_some_and(|pod| { + sandbox_runtime_control_availability_from_pod(pod) + == SandboxRuntimeControlAvailability::Available + }) + }; + match tokio::time::timeout_at( + deadline, + await_condition(pods, &supervisor_pod_name, available), + ) + .await + { + Ok(Ok(Some(_))) => {} + Ok(Ok(None)) | Err(_) => return, + Ok(Err(error)) => { + debug!(%error, sandbox = sandbox_name, "sandbox-runtime bootstrap availability watch failed; reconciliation will retry"); + return; + } + } + + // A replacement control can become Available before the Agent + // Sandbox controller has replaced the prior Suspended condition. + // Clearing the bootstrap marker in that window publishes a terminal + // Stopped event to the gateway. Wait for the controller's real Ready + // condition so the marker removal and readiness annotation expose one + // causally complete transition for both API versions. + let runtime_ready = |object: Option<&DynamicObject>| { + object.is_some_and(|object| { + object.metadata.uid == expected_sandbox_uid + && sandbox_runtime_bootstrap_in_progress(object) + && sandbox_runtime_runtime_is_ready(object) + }) + }; + let object = match tokio::time::timeout_at( + deadline, + await_condition(sandboxes.clone(), &sandbox_name, runtime_ready), + ) + .await + { + Ok(Ok(Some(object))) => object, + Ok(Ok(None)) | Err(_) => return, + Ok(Err(error)) => { + debug!(%error, sandbox = sandbox_name, "sandbox-runtime runtime readiness watch failed; reconciliation will retry"); + return; + } + }; + let Some(resource_version) = object.metadata.resource_version.as_deref() else { + return; + }; + let patch = sandbox_runtime_bootstrap_completion_patch(resource_version); + if let Err(error) = sandboxes + .patch( + &sandbox_name, + &PatchParams::default(), + &Patch::Merge(&patch), + ) + .await + { + debug!(%error, sandbox = sandbox_name, "sandbox-runtime bootstrap completion raced; reconciliation will retry"); + } + }); +} + +fn sandbox_runtime_runtime_is_ready(object: &DynamicObject) -> bool { + let Some(generation) = object.metadata.generation else { + return false; + }; + let Some(conditions) = object + .data + .pointer("/status/conditions") + .and_then(serde_json::Value::as_array) + else { + return false; + }; + let observes_current_generation = |condition: &serde_json::Value| { + condition + .get("observedGeneration") + .and_then(serde_json::Value::as_i64) + == Some(generation) + }; + let ready = conditions.iter().any(|condition| { + condition.get("type").and_then(serde_json::Value::as_str) == Some("Ready") + && condition.get("status").and_then(serde_json::Value::as_str) == Some("True") + && observes_current_generation(condition) + }); + let suspended = conditions.iter().any(|condition| { + condition.get("type").and_then(serde_json::Value::as_str) + == Some(SANDBOX_SUSPENDED_CONDITION) + && condition.get("status").and_then(serde_json::Value::as_str) == Some("True") + && observes_current_generation(condition) + }); + ready && !suspended +} + +fn sandbox_runtime_bootstrap_in_progress(object: &DynamicObject) -> bool { + object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING)) + .is_some_and(|value| value == "true") +} + +fn sandbox_runtime_bootstrap_operation(object: &DynamicObject) -> Option<&str> { + object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION)) + .map(String::as_str) +} + +fn sandbox_runtime_bootstrap_phase(object: &DynamicObject) -> Option { + object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE)) + .and_then(|value| SandboxRuntimeBootstrapPhase::parse(value)) +} + +fn sandbox_runtime_generation(object: &DynamicObject) -> Option<&str> { + object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_GENERATION)) + .map(String::as_str) +} + +fn sandbox_runtime_bootstrap_phase_patch( + resource_version: &str, + phase: SandboxRuntimeBootstrapPhase, +) -> serde_json::Value { + serde_json::json!({ + "metadata": { + "resourceVersion": resource_version, + "annotations": { + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE: phase.as_str(), + } + } + }) +} + +fn sandbox_runtime_bootstrap_is_stale( + object: &DynamicObject, + now: SystemTime, + minimum_age: Duration, +) -> bool { + let Some(started_at_ms) = object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT)) + .and_then(|value| value.parse::().ok()) + else { + return true; + }; + now.duration_since(SystemTime::UNIX_EPOCH + Duration::from_millis(started_at_ms)) + .is_ok_and(|age| age >= minimum_age) +} + +fn sandbox_runtime_readiness_transition_patch( + resource_version: &str, + state: &str, +) -> serde_json::Value { + serde_json::json!({ + "metadata": { + "resourceVersion": resource_version, + "annotations": { ANNOTATION_SANDBOX_RUNTIME_READINESS: state }, + } + }) +} + +fn cluster_wide_watch_stream( + mut sandbox_stream: S, + default_namespace: String, + readiness_client: Client, + driver: KubernetesComputeDriver, +) -> WatchStream where S: Stream> + Send + Unpin + 'static, { let (tx, rx) = mpsc::channel(256); + driver.spawn_sandbox_runtime_periodic_reconcile(tx.clone()); tokio::spawn(async move { loop { @@ -2071,7 +4215,7 @@ where Some(Event::Apply(obj) | Event::InitApply(obj)) => { let ns = obj.metadata.namespace.clone() .unwrap_or_else(|| default_namespace.clone()); - if let Ok((_kube_name, sandbox)) = sandbox_from_object(&ns, obj) { + if let Ok((_kube_name, sandbox)) = sandbox_from_object_with_sandbox_runtime_readiness(&readiness_client, &ns, obj).await { let event = WatchSandboxesEvent { payload: Some(watch_sandboxes_event::Payload::Sandbox( WatchSandboxesSandboxEvent { sandbox: Some(sandbox) } @@ -2413,17 +4557,62 @@ fn pod_sandbox_id(pod: &Pod) -> Result { } #[allow(clippy::result_large_err)] -fn validate_pod_uid(pod: &Pod, expected_uid: &str) -> Result<(), tonic::Status> { - if pod.metadata.uid.as_deref() == Some(expected_uid) { - return Ok(()); +fn validate_pod_uid(pod: &Pod, expected_uid: &str) -> Result<(), tonic::Status> { + if pod.metadata.uid.as_deref() == Some(expected_uid) { + return Ok(()); + } + Err(tonic::Status::permission_denied( + "sandbox credential pod UID mismatch", + )) +} + +fn require_proxy_control_authentication(via_proxy_control: bool) -> Result<(), tonic::Status> { + if via_proxy_control { + Ok(()) + } else { + Err(tonic::Status::permission_denied( + "sandbox JWT authentication must originate from the paired supervisor Pod", + )) + } +} + +#[allow(clippy::result_large_err)] +fn validate_proxy_control_labels(pod: &Pod, sandbox_id: &str) -> Result<(), tonic::Status> { + validate_proxy_control_labels_from_metadata(&pod.metadata, sandbox_id) +} + +#[allow(clippy::result_large_err)] +fn validate_proxy_control_labels_from_metadata( + metadata: &ObjectMeta, + sandbox_id: &str, +) -> Result<(), tonic::Status> { + let labels = metadata.labels.as_ref().ok_or_else(|| { + tonic::Status::permission_denied("control workload has no sandbox-runtime labels") + })?; + let expected_pair = crate::sandbox_runtime::pair_label_value(sandbox_id); + let matches = labels + .get(BOUNDARY_ROLE_LABEL) + .is_some_and(|role| role == "supervisor") + && labels + .get(BOUNDARY_PAIR_LABEL) + .is_some_and(|pair| pair == &expected_pair) + && labels + .get(LABEL_SANDBOX_ID) + .is_some_and(|actual| actual == sandbox_id); + if matches { + Ok(()) + } else { + Err(tonic::Status::permission_denied( + "control workload sandbox-runtime labels do not match the sandbox identity", + )) } - Err(tonic::Status::permission_denied( - "sandbox credential pod UID mismatch", - )) } #[allow(clippy::result_large_err)] -fn sandbox_owner_reference(pod: &Pod) -> Result<&OwnerReference, tonic::Status> { +fn sandbox_owner_reference( + pod: &Pod, + require_controller: bool, +) -> Result<&OwnerReference, tonic::Status> { let mut owners = pod .metadata .owner_references @@ -2439,9 +4628,10 @@ fn sandbox_owner_reference(pod: &Pod) -> Result<&OwnerReference, tonic::Status> }); let owner = owners .next() - .ok_or_else(|| tonic::Status::permission_denied("pod is not controlled by a Sandbox"))?; + .ok_or_else(|| tonic::Status::permission_denied("pod is not owned by a Sandbox"))?; if owners.next().is_some() - || owner.controller != Some(true) + || (require_controller && owner.controller != Some(true)) + || (!require_controller && owner.controller == Some(true)) || owner.name.is_empty() || owner.uid.is_empty() { @@ -2548,6 +4738,304 @@ fn sandbox_from_object(namespace: &str, obj: DynamicObject) -> Result<(String, S )) } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum SandboxRuntimeControlAvailability { + Available, + Unavailable, + Unknown, +} + +fn sandbox_runtime_control_availability_from_pod(pod: &Pod) -> SandboxRuntimeControlAvailability { + if pod.metadata.deletion_timestamp.is_none() + && pod + .status + .as_ref() + .and_then(|status| status.conditions.as_ref()) + .is_some_and(|conditions| { + conditions + .iter() + .any(|condition| condition.type_ == "Ready" && condition.status == "True") + }) + { + SandboxRuntimeControlAvailability::Available + } else { + SandboxRuntimeControlAvailability::Unavailable + } +} + +async fn sandbox_runtime_control_availability( + client: &Client, + namespace: &str, + sandbox_id: &str, +) -> SandboxRuntimeControlAvailability { + let names = SandboxRuntimeNames::new(sandbox_id); + let pods = Api::::namespaced(client.clone(), namespace); + let services = Api::::namespaced(client.clone(), namespace); + let policies = Api::::namespaced(client.clone(), namespace); + let supervisor = Box::pin(tokio::time::timeout( + KUBE_API_TIMEOUT, + pods.get_opt(&names.supervisor_pod), + )); + let service = Box::pin(tokio::time::timeout( + KUBE_API_TIMEOUT, + services.get_opt(&names.boundary_service), + )); + let fence = Box::pin(tokio::time::timeout( + KUBE_API_TIMEOUT, + policies.get_opt(&names.workload_policy), + )); + let supervisor_fence = Box::pin(tokio::time::timeout( + KUBE_API_TIMEOUT, + policies.get_opt(&names.supervisor_policy), + )); + let (supervisor, service, fence, supervisor_fence) = + tokio::join!(supervisor, service, fence, supervisor_fence); + let control = match supervisor { + Ok(Ok(Some(pod))) => sandbox_runtime_control_availability_from_pod(&pod), + Ok(Ok(None)) => SandboxRuntimeControlAvailability::Unavailable, + Ok(Err(error)) => { + warn!(sandbox_id, %error, "could not determine sandbox-runtime control availability"); + SandboxRuntimeControlAvailability::Unknown + } + Err(_) => { + warn!( + sandbox_id, + "timed out checking sandbox-runtime control availability" + ); + SandboxRuntimeControlAvailability::Unknown + } + }; + let service = match service { + Ok(Ok(Some(_))) => SandboxRuntimeControlAvailability::Available, + Ok(Ok(None)) => SandboxRuntimeControlAvailability::Unavailable, + Ok(Err(error)) => { + warn!(sandbox_id, %error, "could not determine sandbox-runtime boundary Service availability"); + SandboxRuntimeControlAvailability::Unknown + } + Err(_) => { + warn!( + sandbox_id, + "timed out checking sandbox-runtime boundary Service availability" + ); + SandboxRuntimeControlAvailability::Unknown + } + }; + + // Supervisor readiness alone is not sufficient: deletion of the shared + // fence would otherwise leave a live boundary with direct pod + // egress while the driver continued to publish Ready. + let fence = match fence { + Ok(Ok(Some(_))) => SandboxRuntimeControlAvailability::Available, + Ok(Ok(None)) => SandboxRuntimeControlAvailability::Unavailable, + Ok(Err(error)) => { + warn!(sandbox_id, %error, "could not determine sandbox-runtime workload fence availability"); + SandboxRuntimeControlAvailability::Unknown + } + Err(_) => { + warn!( + sandbox_id, + "timed out checking sandbox-runtime workload fence availability" + ); + SandboxRuntimeControlAvailability::Unknown + } + }; + let supervisor_fence = match supervisor_fence { + Ok(Ok(Some(_))) => SandboxRuntimeControlAvailability::Available, + Ok(Ok(None)) => SandboxRuntimeControlAvailability::Unavailable, + Ok(Err(error)) => { + warn!(sandbox_id, %error, "could not determine sandbox-runtime supervisor fence availability"); + SandboxRuntimeControlAvailability::Unknown + } + Err(_) => { + warn!( + sandbox_id, + "timed out checking sandbox-runtime supervisor fence availability" + ); + SandboxRuntimeControlAvailability::Unknown + } + }; + let dependencies = [control, service, fence, supervisor_fence]; + if dependencies.contains(&SandboxRuntimeControlAvailability::Unavailable) { + SandboxRuntimeControlAvailability::Unavailable + } else if dependencies.contains(&SandboxRuntimeControlAvailability::Unknown) { + SandboxRuntimeControlAvailability::Unknown + } else { + SandboxRuntimeControlAvailability::Available + } +} + +async fn sandbox_runtime_workload_generation_matches( + client: &Client, + namespace: &str, + pod_name: &str, + sandbox: &DynamicObject, +) -> SandboxRuntimeControlAvailability { + let Some(expected_uid) = sandbox + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID)) + else { + return SandboxRuntimeControlAvailability::Unavailable; + }; + let pods = Api::::namespaced(client.clone(), namespace); + match tokio::time::timeout(KUBE_API_TIMEOUT, pods.get_opt(pod_name)).await { + Ok(Ok(Some(pod))) if pod.metadata.uid.as_deref() == Some(expected_uid.as_str()) => { + SandboxRuntimeControlAvailability::Available + } + Ok(Ok(_)) => SandboxRuntimeControlAvailability::Unavailable, + Ok(Err(error)) => { + warn!(pod = pod_name, %error, "could not verify sandbox-runtime workload generation"); + SandboxRuntimeControlAvailability::Unknown + } + Err(_) => { + warn!( + pod = pod_name, + "timed out checking sandbox-runtime workload generation" + ); + SandboxRuntimeControlAvailability::Unknown + } + } +} + +async fn sandbox_runtime_supervisor_generation_matches( + client: &Client, + namespace: &str, + sandbox_id: &str, + sandbox: &DynamicObject, +) -> SandboxRuntimeControlAvailability { + let Some(expected_uid) = sandbox + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID)) + else { + return SandboxRuntimeControlAvailability::Unavailable; + }; + let names = SandboxRuntimeNames::new(sandbox_id); + let pods = Api::::namespaced(client.clone(), namespace); + match tokio::time::timeout(KUBE_API_TIMEOUT, pods.get_opt(&names.supervisor_pod)).await { + Ok(Ok(Some(pod))) if pod.metadata.uid.as_deref() == Some(expected_uid.as_str()) => { + SandboxRuntimeControlAvailability::Available + } + Ok(Ok(_)) => SandboxRuntimeControlAvailability::Unavailable, + Ok(Err(error)) => { + warn!(pod = names.supervisor_pod, %error, "could not verify sandbox-runtime supervisor generation"); + SandboxRuntimeControlAvailability::Unknown + } + Err(_) => { + warn!( + pod = names.supervisor_pod, + "timed out checking sandbox-runtime supervisor generation" + ); + SandboxRuntimeControlAvailability::Unknown + } + } +} + +async fn sandbox_from_object_with_sandbox_runtime_readiness( + client: &Client, + namespace: &str, + obj: DynamicObject, +) -> Result<(String, Sandbox), String> { + let bootstrapping = sandbox_runtime_bootstrap_in_progress(&obj); + let sandbox_id = sandbox_id_from_object(&obj).unwrap_or_default(); + let object_namespace = obj + .metadata + .namespace + .clone() + .unwrap_or_else(|| namespace.to_string()); + let (name, mut sandbox) = sandbox_from_object(namespace, obj.clone())?; + if bootstrapping { + mark_sandbox_runtime_bootstrapping(&mut sandbox); + } + if !sandbox_id.is_empty() { + let dependencies = Box::pin(sandbox_runtime_control_availability( + client, + &object_namespace, + &sandbox_id, + )); + let workload_generation = Box::pin(sandbox_runtime_workload_generation_matches( + client, + &object_namespace, + &name, + &obj, + )); + let supervisor_generation = Box::pin(sandbox_runtime_supervisor_generation_matches( + client, + &object_namespace, + &sandbox_id, + &obj, + )); + let (dependencies, workload_generation, supervisor_generation) = + tokio::join!(dependencies, workload_generation, supervisor_generation); + if dependencies != SandboxRuntimeControlAvailability::Available + || workload_generation != SandboxRuntimeControlAvailability::Available + || supervisor_generation != SandboxRuntimeControlAvailability::Available + { + mark_sandbox_runtime_control_unavailable(&mut sandbox); + } + } + Ok((name, sandbox)) +} + +fn mark_sandbox_runtime_bootstrapping(sandbox: &mut Sandbox) { + if let Some(status) = sandbox.status.as_mut() { + status.conditions.retain(|condition| { + condition.r#type != SANDBOX_SUSPENDED_CONDITION && condition.r#type != "Bootstrapping" + }); + status.conditions.push(SandboxCondition { + r#type: "Bootstrapping".to_string(), + status: "True".to_string(), + reason: "SandboxRuntimeGenerationStarting".to_string(), + message: "replacement sandbox-runtime generation is starting".to_string(), + last_transition_time: String::new(), + }); + } + mark_sandbox_runtime_control_unavailable(sandbox); +} + +fn mark_sandbox_runtime_control_unavailable(sandbox: &mut Sandbox) { + const REASON: &str = "DependenciesNotReady"; + const MESSAGE: &str = "sandbox-runtime enforcement dependencies are not ready"; + let Some(status) = sandbox.status.as_mut() else { + return; + }; + if let Some(ready) = status + .conditions + .iter_mut() + .find(|condition| condition.r#type == "Ready") + { + ready.status = "False".to_string(); + ready.reason = REASON.to_string(); + ready.message = MESSAGE.to_string(); + } else { + status.conditions.push(SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: REASON.to_string(), + message: MESSAGE.to_string(), + last_transition_time: String::new(), + }); + } +} + +fn sandbox_runtime_should_run(obj: &DynamicObject) -> bool { + if let Some(mode) = obj + .data + .get("spec") + .and_then(|spec| spec.get("operatingMode")) + .and_then(serde_json::Value::as_str) + { + return !mode.eq_ignore_ascii_case("Suspended"); + } + obj.data + .get("spec") + .and_then(|spec| spec.get("replicas")) + .and_then(serde_json::Value::as_i64) + .is_none_or(|replicas| replicas > 0) +} + fn update_indexes( sandbox_name_to_id: &mut std::collections::HashMap, agent_pod_to_id: &mut std::collections::HashMap, @@ -2656,726 +5144,262 @@ fn attach_kube_progress_metadata( fn pulling_image_from_kube_message(message: &str) -> Option { let image = message - .strip_prefix("Pulling image ") - .map(str::trim) - .map(|value| value.trim_matches('"'))?; - (!image.is_empty()).then(|| image.to_string()) -} - -fn pulled_image_label(message: &str) -> String { - extract_image_size(message).map_or_else( - || "Image pulled".to_string(), - |bytes| format!("Image pulled ({})", format_bytes(bytes)), - ) -} - -fn extract_image_size(message: &str) -> Option { - let size_prefix = "Image size: "; - let start = message.find(size_prefix)? + size_prefix.len(); - let rest = &message[start..]; - let end = rest.find(' ')?; - rest[..end].parse().ok() -} - -/// Path where the supervisor binary is mounted inside the agent container. -const SUPERVISOR_MOUNT_PATH: &str = openshell_core::driver_utils::SUPERVISOR_CONTAINER_DIR; - -/// Name of the volume used to side-load the supervisor binary. -const SUPERVISOR_VOLUME_NAME: &str = "openshell-supervisor-bin"; - -/// Name of the init container that installs the supervisor binary. -const SUPERVISOR_INIT_CONTAINER_NAME: &str = "openshell-supervisor-install"; - -/// Name of the init container that prepares pod-level sidecar networking. -const SUPERVISOR_NETWORK_INIT_CONTAINER_NAME: &str = "openshell-network-init"; - -/// Container name for the network-only supervisor sidecar. -const SUPERVISOR_NETWORK_SIDECAR_NAME: &str = "openshell-supervisor-network"; - -/// UID used by strict process/binary-aware sidecars so Kubernetes grants the -/// requested capability set into the effective set without privilege escalation. -const BINARY_AWARE_SIDECAR_PROXY_UID: u32 = 0; - -/// Shared volume used by the network sidecar and process-only supervisor for -/// local coordination in sidecar topology. -const SIDECAR_STATE_VOLUME_NAME: &str = "openshell-sidecar-state"; -const SIDECAR_STATE_MOUNT_PATH: &str = openshell_core::container_paths::SIDECAR_RUN_ROOT; -const SIDECAR_CONTROL_SOCKET: &str = openshell_core::container_paths::SIDECAR_CONTROL_SOCKET; -// Linux abstract socket names are scoped to the pod's shared network namespace. -// Unlike a filesystem socket in the shared state volume, the workload cannot -// unlink and replace this relay endpoint after the trusted supervisor binds it. -const SIDECAR_SSH_SOCKET_FILE: &str = "@openshell-sidecar-ssh"; - -/// Shared TLS work directory. The network sidecar writes the proxy CA bundle -/// here, while the agent container consumes it after sidecar bootstrap. -const SIDECAR_TLS_VOLUME_NAME: &str = "openshell-supervisor-tls"; -const SIDECAR_TLS_MOUNT_PATH: &str = openshell_core::container_paths::SIDECAR_TLS_DIR; -const SIDECAR_CLIENT_TLS_MOUNT_PATH: &str = openshell_core::container_paths::SIDECAR_CLIENT_TLS_DIR; - -/// Build the emptyDir volume that holds the supervisor binary. -/// -/// The init container writes the binary here; the agent container reads it. -fn supervisor_volume() -> serde_json::Value { - serde_json::json!({ - "name": SUPERVISOR_VOLUME_NAME, - "emptyDir": {} - }) -} - -/// Build the read-only volume mount for the supervisor binary in the agent container. -fn supervisor_volume_mount() -> serde_json::Value { - serde_json::json!({ - "name": SUPERVISOR_VOLUME_NAME, - "mountPath": SUPERVISOR_MOUNT_PATH, - "readOnly": true - }) -} - -/// Build an image volume that mounts the supervisor OCI image directly. -/// -/// Requires Kubernetes >= v1.33 (`ImageVolume` beta) or >= v1.36 (GA). -/// The entire image filesystem is mounted read-only, making the binary -/// available at `{SUPERVISOR_MOUNT_PATH}/openshell-sandbox`. -fn supervisor_image_volume( - supervisor_image: &str, - supervisor_image_pull_policy: Option<&str>, -) -> serde_json::Value { - let mut image_spec = serde_json::json!({ - "reference": supervisor_image, - }); - if let Some(policy) = supervisor_image_pull_policy { - image_spec["pullPolicy"] = serde_json::json!(policy); - } - serde_json::json!({ - "name": SUPERVISOR_VOLUME_NAME, - "image": image_spec - }) -} - -/// Build the init container that copies the supervisor binary into the emptyDir. -/// -/// The supervisor image contains the supervisor binary at `/openshell-sandbox`. -/// We invoke that binary with the `copy-self` subcommand so it copies itself -/// into the shared emptyDir volume, where the agent container then executes it -/// from a fixed, writable path. This pattern (binary self-copy) avoids requiring -/// `sh`/`cp` in the supervisor image and mirrors the approach used by argoexec's -/// emissary executor. -fn supervisor_init_container( - supervisor_image: &str, - supervisor_image_pull_policy: Option<&str>, -) -> serde_json::Value { - let installed_path = format!("{SUPERVISOR_MOUNT_PATH}/openshell-sandbox"); - let mut spec = serde_json::json!({ - "name": SUPERVISOR_INIT_CONTAINER_NAME, - "image": supervisor_image, - "command": [ - SUPERVISOR_IMAGE_BINARY_PATH, - "copy-self", - installed_path, - ], - "securityContext": {"runAsUser": 0}, - "volumeMounts": [{ - "name": SUPERVISOR_VOLUME_NAME, - "mountPath": SUPERVISOR_MOUNT_PATH, - "readOnly": false - }] - }); - if let Some(policy) = supervisor_image_pull_policy { - spec["imagePullPolicy"] = serde_json::json!(policy); - } - spec -} - -fn apply_supervisor_binary_source( - spec: &mut serde_json::Map, - supervisor_image: &str, - supervisor_image_pull_policy: Option<&str>, - method: SupervisorSideloadMethod, -) { - let volumes = spec - .entry("volumes") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(volumes) = volumes { - match method { - SupervisorSideloadMethod::ImageVolume => { - volumes.push(supervisor_image_volume( - supervisor_image, - supervisor_image_pull_policy, - )); - } - SupervisorSideloadMethod::InitContainer => { - volumes.push(supervisor_volume()); - } - } - } - - if method == SupervisorSideloadMethod::InitContainer { - let init_containers = spec - .entry("initContainers") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(init_containers) = init_containers { - init_containers.push(supervisor_init_container( - supervisor_image, - supervisor_image_pull_policy, - )); - } - } -} - -/// Apply supervisor side-load transforms to an already-built pod template JSON. -/// -/// Depending on the sideload method: -/// - **`ImageVolume`**: mounts the supervisor OCI image directly as a read-only -/// volume (no init container needed, requires K8s >= v1.33). -/// - **`InitContainer`**: injects an emptyDir volume and an init container that -/// copies the supervisor binary from the supervisor image into that volume. -/// -/// In both cases, the agent container gets a command override to run the -/// side-loaded binary as root so it can create network namespaces, set up the -/// proxy, and configure Landlock/seccomp. -#[allow(clippy::similar_names)] -fn apply_supervisor_sideload_with_params( - pod_template: &mut serde_json::Value, - params: &SandboxPodParams<'_>, -) { - let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { - return; - }; - - apply_supervisor_binary_source( - spec, - params.supervisor_image, - params.supervisor_image_pull_policy, - params.supervisor_sideload_method, - ); - - // Find the agent container and add volume mount + command override - let Some(containers) = spec.get_mut("containers").and_then(|v| v.as_array_mut()) else { - return; - }; - - let mut target_index = None; - for (i, c) in containers.iter().enumerate() { - if c.get("name").and_then(|v| v.as_str()) == Some("agent") { - target_index = Some(i); - break; - } - } - let index = target_index.unwrap_or(0); - - if let Some(container) = containers.get_mut(index).and_then(|v| v.as_object_mut()) { - // Override command to use the side-loaded supervisor binary - let mut command = vec![ - format!("{}/openshell-sandbox", SUPERVISOR_MOUNT_PATH), - "--workdir".to_string(), - driver_mounts::DEFAULT_WORKSPACE_ROOT.to_string(), - ]; - command.extend(upstream_proxy_cli_args(params)); - container.insert("command".to_string(), serde_json::json!(command)); - - // Force the supervisor to run as root (UID 0). Sandbox images may set - // a non-root USER directive (e.g. `USER sandbox`), but the supervisor - // needs root to create network namespaces, set up the proxy, and - // configure Landlock/seccomp. The supervisor itself drops privileges - // for child processes via the policy's `run_as_user`/`run_as_group`. - let security_context = container - .entry("securityContext") - .or_insert_with(|| serde_json::json!({})); - if let Some(sc) = security_context.as_object_mut() { - sc.insert("runAsUser".to_string(), serde_json::json!(0)); - } - - // Add volume mount - let volume_mounts = container - .entry("volumeMounts") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(volume_mounts) = volume_mounts { - volume_mounts.push(supervisor_volume_mount()); - } - - // Inject the protected resolved identity contract. Clearing the OCI - // input prevents image or user environment from selecting a - // conflicting identity path. - let env = container - .entry("env") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(env) = env { - apply_resolved_identity_env(env, params.sandbox_uid, params.sandbox_gid); - } - if has_upstream_proxy_credentials(params) { - let volume_mounts = container - .entry("volumeMounts") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(volume_mounts) = volume_mounts { - volume_mounts.push(upstream_proxy_auth_volume_mount()); - } - } - } -} - -#[cfg(test)] -#[allow(clippy::similar_names)] -fn apply_supervisor_sideload( - pod_template: &mut serde_json::Value, - supervisor_image: &str, - supervisor_image_pull_policy: Option<&str>, - method: SupervisorSideloadMethod, - sandbox_uid: u32, - sandbox_gid: u32, -) { - let params = SandboxPodParams { - supervisor_image, - supervisor_image_pull_policy, - supervisor_sideload_method: method, - sandbox_uid, - sandbox_gid, - ..SandboxPodParams::default() - }; - apply_supervisor_sideload_with_params(pod_template, ¶ms); -} - -fn upstream_proxy_cli_args(params: &SandboxPodParams<'_>) -> Vec { - let mut args = Vec::new(); - if let Some(url) = params.https_proxy { - args.extend(["--upstream-proxy".to_string(), url.to_string()]); - } - if let Some(list) = params.no_proxy { - args.extend(["--upstream-no-proxy".to_string(), list.to_string()]); - } - if has_upstream_proxy_credentials(params) { - args.extend([ - "--upstream-proxy-auth-file".to_string(), - openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH.to_string(), - ]); - } - if params.proxy_auth_allow_insecure { - args.push("--upstream-proxy-auth-allow-insecure".to_string()); - } - if params.proxy_connect_by_hostname { - args.push("--upstream-proxy-connect-by-hostname".to_string()); - } - args -} - -fn upstream_proxy_auth_volume_mount() -> serde_json::Value { - serde_json::json!({ - "name": UPSTREAM_PROXY_AUTH_VOLUME_NAME, - "mountPath": upstream_proxy_auth_volume_mount_path(), - "readOnly": true, - }) -} - -fn upstream_proxy_auth_volume_mount_path() -> &'static str { - Path::new(openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH) - .parent() - .and_then(Path::to_str) - .expect("upstream proxy auth path has a parent directory") -} - -fn upstream_proxy_auth_file_name() -> &'static str { - Path::new(openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH) - .file_name() - .and_then(|name| name.to_str()) - .expect("upstream proxy auth path has a UTF-8 file name") -} - -fn has_upstream_proxy_credentials(params: &SandboxPodParams<'_>) -> bool { - params.proxy_auth_secret_name.is_some() && params.proxy_auth_secret_key.is_some() + .strip_prefix("Pulling image ") + .map(str::trim) + .map(|value| value.trim_matches('"'))?; + (!image.is_empty()).then(|| image.to_string()) } -fn sidecar_state_volume_mount() -> serde_json::Value { - serde_json::json!({ - "name": SIDECAR_STATE_VOLUME_NAME, - "mountPath": SIDECAR_STATE_MOUNT_PATH, - }) +fn pulled_image_label(message: &str) -> String { + extract_image_size(message).map_or_else( + || "Image pulled".to_string(), + |bytes| format!("Image pulled ({})", format_bytes(bytes)), + ) } -fn sidecar_tls_volume_mount() -> serde_json::Value { - serde_json::json!({ - "name": SIDECAR_TLS_VOLUME_NAME, - "mountPath": SIDECAR_TLS_MOUNT_PATH, - }) +fn extract_image_size(message: &str) -> Option { + let size_prefix = "Image size: "; + let start = message.find(size_prefix)? + size_prefix.len(); + let rest = &message[start..]; + let end = rest.find(' ')?; + rest[..end].parse().ok() } -fn copy_log_level_env( - env: &mut Vec, - template_environment: &std::collections::HashMap, - spec_environment: &std::collections::HashMap, -) { - if let Some(value) = spec_environment - .get(openshell_core::sandbox_env::LOG_LEVEL) - .or_else(|| template_environment.get(openshell_core::sandbox_env::LOG_LEVEL)) - { - upsert_env(env, openshell_core::sandbox_env::LOG_LEVEL, value); - } -} +const SANDBOX_RUNTIME_VOLUME_NAME: &str = "openshell-runtime"; +const SANDBOX_STATE_VOLUME_NAME: &str = "openshell-runtime-state"; +const SANDBOX_BOOTSTRAP_VOLUME_NAME: &str = "openshell-sandbox-bootstrap"; +const SANDBOX_POD_IDENTITY_VOLUME_NAME: &str = "openshell-pod-identity"; +const SANDBOX_RUNTIME_MOUNT_PATH: &str = "/.openshell/runtime"; +const SANDBOX_STATE_MOUNT_PATH: &str = "/.openshell/state"; +const SANDBOX_POD_IDENTITY_MOUNT_PATH: &str = "/.openshell/pod-identity"; +const SANDBOX_POD_UID_PATH: &str = "/.openshell/pod-identity/uid"; +const SANDBOX_PROXY_CA_VOLUME_NAME: &str = "openshell-run"; +const SANDBOX_PROXY_CA_MOUNT_PATH: &str = "/run"; +const SANDBOX_BOOTSTRAP_SCHEDULING_GATE: &str = "openshell.ai/bootstrap"; -fn supervisor_sidecar_env( - template_environment: &std::collections::HashMap, - spec_environment: &std::collections::HashMap, +/// Render the workload Pod that runs the `OpenShell` sandbox runtime. +/// +/// The pod receives no gateway credential or endpoint. Its non-root sandbox +/// owns the workload process and seccomp listener. The namespace workload +/// policy permits supervisor Pods to reach its TLS listener; authenticated +/// protocol identity binds the request to the exact supervisor generation. +fn apply_supervisor_sandbox_runtime_boundary( + pod_template: &mut serde_json::Value, params: &SandboxPodParams<'_>, -) -> Vec { - let mut env = Vec::new(); - apply_required_env( - &mut env, - params.sandbox_id, - params.sandbox_name, - params.grpc_endpoint, - "", - !params.client_tls_secret_name.is_empty(), - provider_spiffe_socket_path(params), +) { + let metadata = pod_template + .as_object_mut() + .expect("pod template must be an object") + .entry("metadata") + .or_insert_with(|| serde_json::json!({})); + let labels = metadata + .as_object_mut() + .expect("pod metadata must be an object") + .entry("labels") + .or_insert_with(|| serde_json::json!({})); + let labels = labels + .as_object_mut() + .expect("pod labels must be an object"); + labels.insert( + BOUNDARY_PAIR_LABEL.to_string(), + serde_json::json!(crate::sandbox_runtime::pair_label_value(params.sandbox_id)), ); - if !params.client_tls_secret_name.is_empty() { - upsert_env( - &mut env, - openshell_core::sandbox_env::TLS_CA, - &format!("{SIDECAR_CLIENT_TLS_MOUNT_PATH}/ca.crt"), - ); - upsert_env( - &mut env, - openshell_core::sandbox_env::TLS_CERT, - &format!("{SIDECAR_CLIENT_TLS_MOUNT_PATH}/tls.crt"), - ); - upsert_env( - &mut env, - openshell_core::sandbox_env::TLS_KEY, - &format!("{SIDECAR_CLIENT_TLS_MOUNT_PATH}/tls.key"), - ); - } - copy_log_level_env(&mut env, template_environment, spec_environment); - upsert_env( - &mut env, - openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY, - "sidecar", + labels.insert( + BOUNDARY_ROLE_LABEL.to_string(), + serde_json::json!("workload"), ); - upsert_env( - &mut env, - openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE, - "sidecar-nftables", + + let Some(spec) = pod_template + .get_mut("spec") + .and_then(serde_json::Value::as_object_mut) + else { + return; + }; + spec.insert("hostNetwork".to_string(), serde_json::json!(false)); + spec.insert("hostPID".to_string(), serde_json::json!(false)); + spec.insert("hostIPC".to_string(), serde_json::json!(false)); + spec.insert( + "shareProcessNamespace".to_string(), + serde_json::json!(false), ); - upsert_env( - &mut env, - openshell_core::sandbox_env::SIDECAR_CONTROL_SOCKET, - SIDECAR_CONTROL_SOCKET, + spec.insert("dnsPolicy".to_string(), serde_json::json!("None")); + spec.insert( + "schedulingGates".to_string(), + serde_json::json!([{"name": SANDBOX_BOOTSTRAP_SCHEDULING_GATE}]), ); - upsert_env( - &mut env, - openshell_core::sandbox_env::SSH_SOCKET_PATH, - SIDECAR_SSH_SOCKET_FILE, + spec.insert( + "dnsConfig".to_string(), + serde_json::json!({ + "nameservers": ["127.0.0.53"], + "options": [ + {"name": "ndots", "value": "5"}, + {"name": "timeout", "value": "2"}, + {"name": "attempts", "value": "2"} + ] + }), ); - upsert_env( - &mut env, - openshell_core::sandbox_env::PROXY_TLS_DIR, - SIDECAR_TLS_MOUNT_PATH, + spec.insert( + "securityContext".to_string(), + serde_json::json!({ + "runAsUser": params.sandbox_uid, + "runAsGroup": params.sandbox_gid, + "runAsNonRoot": true, + "fsGroup": params.sandbox_gid, + "fsGroupChangePolicy": "OnRootMismatch", + "supplementalGroups": [], + "supplementalGroupsPolicy": "Strict", + "seccompProfile": {"type": "RuntimeDefault"}, + "sysctls": [{"name": "net.ipv4.ip_unprivileged_port_start", "value": "0"}] + }), ); - apply_resolved_identity_env(&mut env, params.sandbox_uid, params.sandbox_gid); - if !params.process_binary_aware_network_policy { - upsert_env( - &mut env, - openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY, - "relaxed", - ); - } - env -} - -fn supervisor_sidecar_container( - template_environment: &std::collections::HashMap, - spec_environment: &std::collections::HashMap, - params: &SandboxPodParams<'_>, -) -> serde_json::Value { - let proxy_uid = effective_sidecar_proxy_uid(params); - let capabilities = if params.process_binary_aware_network_policy { + let volumes = spec + .entry("volumes") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut() + .expect("pod volumes must be an array"); + volumes.retain(|volume| { + !matches!( + volume.get("name").and_then(serde_json::Value::as_str), + Some( + "openshell-sa-token" + | "openshell-client-tls" + | "spiffe-workload-api" + | SANDBOX_RUNTIME_VOLUME_NAME + | SANDBOX_STATE_VOLUME_NAME + | SANDBOX_BOOTSTRAP_VOLUME_NAME + | SANDBOX_POD_IDENTITY_VOLUME_NAME + | SANDBOX_PROXY_CA_VOLUME_NAME + ) + ) + }); + volumes.extend([ + serde_json::json!({"name": SANDBOX_RUNTIME_VOLUME_NAME, "emptyDir": {"medium": "Memory"}}), + serde_json::json!({"name": SANDBOX_STATE_VOLUME_NAME, "emptyDir": {"medium": "Memory"}}), serde_json::json!({ - "drop": ["ALL"], - "add": ["SYS_PTRACE", "DAC_READ_SEARCH"] - }) - } else { + "name": SANDBOX_BOOTSTRAP_VOLUME_NAME, + "secret": {"secretName": params.sandbox_secret_name, "defaultMode": 0o440} + }), serde_json::json!({ - "drop": ["ALL"] - }) - }; - let mut container = serde_json::json!({ - "name": SUPERVISOR_NETWORK_SIDECAR_NAME, - "image": params.supervisor_image, - "command": [ - SUPERVISOR_IMAGE_BINARY_PATH, - "--mode=network", - ], - "env": supervisor_sidecar_env(template_environment, spec_environment, params), - "securityContext": { - "runAsUser": proxy_uid, - "runAsGroup": params.sandbox_gid, - "runAsNonRoot": proxy_uid != 0, - "allowPrivilegeEscalation": false, - "capabilities": capabilities - }, - "volumeMounts": [ - sidecar_state_volume_mount(), - sidecar_tls_volume_mount(), - { - "name": "openshell-sa-token", - "mountPath": "/var/run/secrets/openshell", - "readOnly": true + "name": SANDBOX_POD_IDENTITY_VOLUME_NAME, + "downwardAPI": { + "items": [{"path": "uid", "fieldRef": {"fieldPath": "metadata.uid"}}] } - ] - }); - container["command"] + }), + serde_json::json!({ + "name": SANDBOX_PROXY_CA_VOLUME_NAME, + "emptyDir": {"medium": "Memory"} + }), + ]); + + let init_containers = spec + .entry("initContainers") + .or_insert_with(|| serde_json::json!([])) .as_array_mut() - .expect("network supervisor command is an array") - .extend( - upstream_proxy_cli_args(params) - .into_iter() - .map(serde_json::Value::String), - ); - if let Some(policy) = params.supervisor_image_pull_policy { - container["imagePullPolicy"] = serde_json::json!(policy); - } - if params.provider_spiffe_enabled { - container["volumeMounts"] - .as_array_mut() - .expect("volumeMounts is an array") - .push(serde_json::json!({ - "name": SPIFFE_WORKLOAD_API_VOLUME_NAME, - "mountPath": spiffe_socket_mount_path(params.provider_spiffe_workload_api_socket_path), - "readOnly": true, - })); - } - if has_upstream_proxy_credentials(params) { - container["volumeMounts"] - .as_array_mut() - .expect("volumeMounts is an array") - .push(upstream_proxy_auth_volume_mount()); - } - if let Some(profile) = params.app_armor_profile { - container["securityContext"]["appArmorProfile"] = app_armor_profile_to_k8s(profile); - } - container -} - -fn effective_sidecar_proxy_uid(params: &SandboxPodParams<'_>) -> u32 { - if params.process_binary_aware_network_policy { - BINARY_AWARE_SIDECAR_PROXY_UID - } else { - params.proxy_uid - } -} - -fn supervisor_network_init_container(params: &SandboxPodParams<'_>) -> serde_json::Value { - let proxy_uid = effective_sidecar_proxy_uid(params); - let mut container = serde_json::json!({ - "name": SUPERVISOR_NETWORK_INIT_CONTAINER_NAME, - "image": params.supervisor_image, - "command": [ - SUPERVISOR_IMAGE_BINARY_PATH, - "--mode=network-init", - "--proxy-uid", - proxy_uid.to_string(), - "--proxy-gid", - params.sandbox_gid.to_string(), - "--sidecar-state-dir", - SIDECAR_STATE_MOUNT_PATH, - "--sidecar-tls-dir", - SIDECAR_TLS_MOUNT_PATH, - ], + .expect("pod init containers must be an array"); + let mut bootstrap = serde_json::json!({ + "name": "openshell-sandbox-bootstrap", + "image": params.sandbox_runtime_image, + "command": ["/openshell-sandbox", "bootstrap"], "securityContext": { - "runAsUser": 0, + "runAsUser": params.sandbox_uid, + "runAsGroup": params.sandbox_gid, + "runAsNonRoot": true, + "readOnlyRootFilesystem": true, "allowPrivilegeEscalation": false, - "capabilities": { - "drop": ["ALL"], - "add": ["NET_ADMIN", "NET_RAW", "CHOWN", "FOWNER"] - } + "capabilities": {"drop": ["ALL"]} }, "volumeMounts": [ - sidecar_state_volume_mount(), - sidecar_tls_volume_mount(), + {"name": SANDBOX_BOOTSTRAP_VOLUME_NAME, "mountPath": crate::sandbox_runtime::SANDBOX_BOOTSTRAP_INPUT_PATH, "readOnly": true}, + {"name": SANDBOX_RUNTIME_VOLUME_NAME, "mountPath": SANDBOX_RUNTIME_MOUNT_PATH}, + {"name": SANDBOX_STATE_VOLUME_NAME, "mountPath": SANDBOX_STATE_MOUNT_PATH} ] }); - if let Some(policy) = params.supervisor_image_pull_policy { - container["imagePullPolicy"] = serde_json::json!(policy); - } - if !params.client_tls_secret_name.is_empty() { - container["volumeMounts"] - .as_array_mut() - .expect("volumeMounts is an array") - .push(serde_json::json!({ - "name": "openshell-client-tls", - "mountPath": openshell_core::container_paths::CLIENT_TLS_DIR, - "readOnly": true - })); - } - if let Some(profile) = params.app_armor_profile { - container["securityContext"]["appArmorProfile"] = app_armor_profile_to_k8s(profile); - } - container -} - -fn apply_supervisor_sidecar_topology( - pod_template: &mut serde_json::Value, - template_environment: &std::collections::HashMap, - spec_environment: &std::collections::HashMap, - params: &SandboxPodParams<'_>, -) { - let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { - return; - }; - - let pod_security_context = spec - .entry("securityContext") - .or_insert_with(|| serde_json::json!({})); - if let Some(sc) = pod_security_context.as_object_mut() { - sc.insert("fsGroup".to_string(), serde_json::json!(params.sandbox_gid)); + if let Some(policy) = params.sandbox_runtime_image_pull_policy { + bootstrap["imagePullPolicy"] = serde_json::json!(policy.as_kubernetes_str()); } + init_containers.push(bootstrap); - spec.insert("shareProcessNamespace".to_string(), serde_json::json!(true)); - - apply_supervisor_binary_source( - spec, - params.supervisor_image, - params.supervisor_image_pull_policy, - params.supervisor_sideload_method, + let containers = spec + .get_mut("containers") + .and_then(serde_json::Value::as_array_mut) + .expect("pod containers must be an array"); + let index = containers + .iter() + .position(|container| { + container.get("name").and_then(serde_json::Value::as_str) == Some("agent") + }) + .unwrap_or(0); + let container = containers[index] + .as_object_mut() + .expect("agent container must be an object"); + container.insert( + "command".to_string(), + serde_json::json!([ + format!("{SANDBOX_RUNTIME_MOUNT_PATH}/openshell-sandbox"), + "--bootstrap", + BOUNDARY_CONFIG_PATH, + ]), ); - - let volumes = spec - .entry("volumes") + container.insert( + "securityContext".to_string(), + serde_json::json!({ + "runAsUser": params.sandbox_uid, + "runAsGroup": params.sandbox_gid, + "runAsNonRoot": true, + "allowPrivilegeEscalation": false, + "capabilities": {"drop": ["ALL"]} + }), + ); + container.insert( + "ports".to_string(), + serde_json::json!([{ + "name": "sandbox-control", + "containerPort": params.boundary_port, + "protocol": "TCP" + }]), + ); + let mounts = container + .entry("volumeMounts") .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(volumes) = volumes { - volumes.push(serde_json::json!({ - "name": SIDECAR_STATE_VOLUME_NAME, - "emptyDir": {} - })); - volumes.push(serde_json::json!({ - "name": SIDECAR_TLS_VOLUME_NAME, - "emptyDir": {} - })); - } - - let init_containers = spec - .entry("initContainers") + .as_array_mut() + .expect("agent volume mounts must be an array"); + mounts.retain(|mount| { + !matches!( + mount.get("name").and_then(serde_json::Value::as_str), + Some( + "openshell-sa-token" + | "openshell-client-tls" + | "spiffe-workload-api" + | SANDBOX_BOOTSTRAP_VOLUME_NAME + ) + ) + }); + mounts.extend([ + serde_json::json!({"name": SANDBOX_RUNTIME_VOLUME_NAME, "mountPath": SANDBOX_RUNTIME_MOUNT_PATH, "readOnly": true}), + serde_json::json!({"name": SANDBOX_STATE_VOLUME_NAME, "mountPath": SANDBOX_STATE_MOUNT_PATH}), + serde_json::json!({"name": SANDBOX_POD_IDENTITY_VOLUME_NAME, "mountPath": SANDBOX_POD_IDENTITY_MOUNT_PATH, "readOnly": true}), + serde_json::json!({"name": SANDBOX_PROXY_CA_VOLUME_NAME, "mountPath": SANDBOX_PROXY_CA_MOUNT_PATH}), + ]); + let env = container + .entry("env") .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(init_containers) = init_containers { - init_containers.push(supervisor_network_init_container(params)); - } - - let Some(containers) = spec.get_mut("containers").and_then(|v| v.as_array_mut()) else { - return; - }; - - let target_index = containers - .iter() - .position(|c| c.get("name").and_then(|v| v.as_str()) == Some("agent")) - .unwrap_or(0); - - if let Some(container) = containers - .get_mut(target_index) - .and_then(|v| v.as_object_mut()) - { - container.insert( - "command".to_string(), - serde_json::json!([ - format!("{}/openshell-sandbox", SUPERVISOR_MOUNT_PATH), - "--mode=process", - "--workdir", - driver_mounts::DEFAULT_WORKSPACE_ROOT - ]), - ); - - let security_context = container - .entry("securityContext") - .or_insert_with(|| serde_json::json!({})); - if let Some(sc) = security_context.as_object_mut() { - sc.insert( - "runAsUser".to_string(), - serde_json::json!(params.sandbox_uid), - ); - sc.insert( - "runAsGroup".to_string(), - serde_json::json!(params.sandbox_gid), - ); - sc.insert("runAsNonRoot".to_string(), serde_json::json!(true)); - sc.insert( - "allowPrivilegeEscalation".to_string(), - serde_json::json!(false), - ); - sc.insert( - "capabilities".to_string(), - serde_json::json!({ - "drop": ["ALL"] - }), - ); - } - - let volume_mounts = container - .entry("volumeMounts") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(volume_mounts) = volume_mounts { - remove_volume_mount(volume_mounts, "openshell-sa-token"); - remove_volume_mount(volume_mounts, "openshell-client-tls"); - remove_volume_mount(volume_mounts, SPIFFE_WORKLOAD_API_VOLUME_NAME); - volume_mounts.push(supervisor_volume_mount()); - volume_mounts.push(sidecar_state_volume_mount()); - volume_mounts.push(sidecar_tls_volume_mount()); - } - - let env = container - .entry("env") - .or_insert_with(|| serde_json::json!([])) - .as_array_mut(); - if let Some(env) = env { - remove_env(env, openshell_core::sandbox_env::ENDPOINT); - remove_env(env, openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME); - remove_env(env, openshell_core::sandbox_env::TLS_CA); - remove_env(env, openshell_core::sandbox_env::TLS_CERT); - remove_env(env, openshell_core::sandbox_env::TLS_KEY); - remove_env(env, openshell_core::sandbox_env::SANDBOX_TOKEN); - remove_env(env, openshell_core::sandbox_env::SANDBOX_TOKEN_FILE); - remove_env(env, openshell_core::sandbox_env::K8S_SA_TOKEN_FILE); - remove_env( - env, - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, - ); - upsert_env( - env, - openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY, - "sidecar", - ); - upsert_env( - env, - openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE, - "sidecar-nftables", - ); - upsert_env( - env, - openshell_core::sandbox_env::SSH_SOCKET_PATH, - SIDECAR_SSH_SOCKET_FILE, - ); - upsert_env( - env, - openshell_core::sandbox_env::SIDECAR_CONTROL_SOCKET, - SIDECAR_CONTROL_SOCKET, - ); - upsert_env( - env, - openshell_core::sandbox_env::PROXY_TLS_DIR, - SIDECAR_TLS_MOUNT_PATH, - ); - apply_resolved_identity_env(env, params.sandbox_uid, params.sandbox_gid); - } + .as_array_mut() + .expect("agent environment must be an array"); + for key in [ + openshell_core::sandbox_env::ENDPOINT, + openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME, + openshell_core::sandbox_env::TLS_CA, + openshell_core::sandbox_env::TLS_CERT, + openshell_core::sandbox_env::TLS_KEY, + openshell_core::sandbox_env::SANDBOX_TOKEN, + openshell_core::sandbox_env::SANDBOX_TOKEN_FILE, + openshell_core::sandbox_env::K8S_SA_TOKEN_FILE, + openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, + openshell_core::sandbox_env::MAIN_PROCESS_SPEC, + ] { + remove_env(env, key); } - - containers.push(supervisor_sidecar_container( - template_environment, - spec_environment, - params, - )); + apply_resolved_identity_env(env, params.sandbox_uid, params.sandbox_gid); } /// Apply workspace persistence transforms to an already-built pod template. @@ -3397,8 +5421,9 @@ fn apply_supervisor_sidecar_topology( fn apply_workspace_persistence( pod_template: &mut serde_json::Value, image: &str, - image_pull_policy: Option<&str>, - sandbox_gid: u32, + image_pull_policy: Option, + sandbox_gid: Option, + workspace_owner: Option<(u32, u32)>, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -3406,11 +5431,13 @@ fn apply_workspace_persistence( // fsGroup is a pod-level field — it instructs kubelet to chown mounted // volumes to this GID. It is invalid at the container securityContext level. - let pod_sc = spec - .entry("securityContext") - .or_insert_with(|| serde_json::json!({})); - if let Some(pod_sc_obj) = pod_sc.as_object_mut() { - pod_sc_obj.insert("fsGroup".to_string(), serde_json::json!(sandbox_gid)); + if let Some(sandbox_gid) = sandbox_gid { + let pod_sc = spec + .entry("securityContext") + .or_insert_with(|| serde_json::json!({})); + if let Some(pod_sc_obj) = pod_sc.as_object_mut() { + pod_sc_obj.insert("fsGroup".to_string(), serde_json::json!(sandbox_gid)); + } } // 1. Add workspace volume mount to the agent container @@ -3475,20 +5502,38 @@ fn apply_workspace_persistence( fi" ); - let mut init_spec = serde_json::json!({ - "name": WORKSPACE_INIT_CONTAINER_NAME, - "image": image, - "command": ["sh", "-c", copy_cmd], - "securityContext": { - "runAsUser": 0, - }, - "volumeMounts": [{ - "name": WORKSPACE_VOLUME_NAME, - "mountPath": WORKSPACE_INIT_MOUNT_PATH - }] - }); + let mut init_spec = if let Some((uid, gid)) = workspace_owner { + serde_json::json!({ + "name": WORKSPACE_INIT_CONTAINER_NAME, + "image": image, + "command": [format!("{SANDBOX_RUNTIME_MOUNT_PATH}/openshell-sandbox"), "seed-workspace"], + "securityContext": { + "runAsUser": uid, + "runAsGroup": gid, + "runAsNonRoot": true, + "readOnlyRootFilesystem": true, + "allowPrivilegeEscalation": false, + "capabilities": {"drop": ["ALL"]} + }, + "volumeMounts": [ + {"name": WORKSPACE_VOLUME_NAME, "mountPath": WORKSPACE_INIT_MOUNT_PATH}, + {"name": SANDBOX_RUNTIME_VOLUME_NAME, "mountPath": SANDBOX_RUNTIME_MOUNT_PATH, "readOnly": true} + ] + }) + } else { + serde_json::json!({ + "name": WORKSPACE_INIT_CONTAINER_NAME, + "image": image, + "command": ["sh", "-c", copy_cmd], + "securityContext": {"runAsUser": 0}, + "volumeMounts": [{ + "name": WORKSPACE_VOLUME_NAME, + "mountPath": WORKSPACE_INIT_MOUNT_PATH + }] + }) + }; if let Some(policy) = image_pull_policy { - init_spec["imagePullPolicy"] = serde_json::json!(policy); + init_spec["imagePullPolicy"] = serde_json::json!(policy.as_kubernetes_str()); } init_containers.push(init_spec); } @@ -3535,41 +5580,24 @@ fn default_workspace_volume_claim_templates( #[allow(clippy::struct_excessive_bools)] struct SandboxPodParams<'a> { default_image: &'a str, - image_pull_policy: Option<&'a str>, + image_pull_policy: Option, image_pull_secrets: &'a [String], - supervisor_image: &'a str, - supervisor_image_pull_policy: Option<&'a str>, - supervisor_sideload_method: SupervisorSideloadMethod, - topology: SupervisorTopology, - proxy_uid: u32, - process_binary_aware_network_policy: bool, - https_proxy: Option<&'a str>, - no_proxy: Option<&'a str>, - proxy_auth_secret_name: Option<&'a str>, - proxy_auth_secret_key: Option<&'a str>, - proxy_auth_allow_insecure: bool, - proxy_connect_by_hostname: bool, + sandbox_runtime_image: &'a str, + sandbox_runtime_image_pull_policy: Option, service_account_name: &'a str, sandbox_id: &'a str, - sandbox_name: &'a str, - grpc_endpoint: &'a str, - ssh_socket_path: &'a str, - client_tls_secret_name: &'a str, - host_gateway_ip: &'a str, enable_user_namespaces: bool, - app_armor_profile: Option<&'a AppArmorProfile>, workspace_default_storage_size: &'a str, workspace_storage_class: &'a str, default_runtime_class_name: &'a str, - /// Lifetime (seconds) of the projected `ServiceAccount` token used - /// for the bootstrap `IssueSandboxToken` exchange. - sa_token_ttl_secs: i64, - provider_spiffe_enabled: bool, - provider_spiffe_workload_api_socket_path: &'a str, /// Resolved sandbox UID for supervisor `runAsUser` and env var. sandbox_uid: u32, /// Resolved sandbox GID for PVC init container operations. sandbox_gid: u32, + /// TLS listener port exposed only to the paired supervisor Pod. + boundary_port: u16, + /// Immutable Secret name for this workload Pod generation. + sandbox_secret_name: &'a str, } impl Default for SandboxPodParams<'_> { @@ -3578,51 +5606,22 @@ impl Default for SandboxPodParams<'_> { default_image: "", image_pull_policy: None, image_pull_secrets: &[], - supervisor_image: "", - supervisor_image_pull_policy: None, - supervisor_sideload_method: SupervisorSideloadMethod::default(), - topology: SupervisorTopology::default(), - proxy_uid: DEFAULT_PROXY_UID, - process_binary_aware_network_policy: true, - https_proxy: None, - no_proxy: None, - proxy_auth_secret_name: None, - proxy_auth_secret_key: None, - proxy_auth_allow_insecure: false, - proxy_connect_by_hostname: false, + sandbox_runtime_image: "", + sandbox_runtime_image_pull_policy: None, service_account_name: DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, sandbox_id: "", - sandbox_name: "", - grpc_endpoint: "", - ssh_socket_path: "", - client_tls_secret_name: "", - host_gateway_ip: "", enable_user_namespaces: false, - app_armor_profile: None, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE, workspace_storage_class: "", default_runtime_class_name: "", - sa_token_ttl_secs: 3600, - provider_spiffe_enabled: false, - provider_spiffe_workload_api_socket_path: "", sandbox_uid: DEFAULT_SANDBOX_UID, sandbox_gid: DEFAULT_SANDBOX_UID, + boundary_port: 5500, + sandbox_secret_name: "os-sandbox-test-generation", } } } -fn validate_sidecar_proxy_identity( - params: &SandboxPodParams<'_>, -) -> Result<(), KubernetesDriverError> { - if params.topology == SupervisorTopology::Sidecar && params.proxy_uid == params.sandbox_uid { - return Err(KubernetesDriverError::Precondition(format!( - "proxy_uid ({}) must not match sandbox_uid ({}) in sidecar topology", - params.proxy_uid, params.sandbox_uid - ))); - } - Ok(()) -} - fn spec_pod_env(spec: Option<&SandboxSpec>) -> std::collections::HashMap { let mut env = spec.map_or_else(Default::default, |s| s.environment.clone()); if let Some(s) = spec.filter(|s| !s.log_level.is_empty()) { @@ -3636,22 +5635,15 @@ fn spec_pod_env(spec: Option<&SandboxSpec>) -> std::collections::HashMap, - provider_spiffe_workload_api_socket_path: Option<&str>, ) -> Result { let config = spec .and_then(|spec| spec.template.as_ref()) .map(KubernetesSandboxDriverConfig::from_template) .transpose()? .unwrap_or_default(); - let mut protected_paths = KUBERNETES_DRIVER_PROTECTED_MOUNT_PATHS.to_vec(); - let provider_spiffe_mount_path; - if let Some(socket_path) = provider_spiffe_workload_api_socket_path { - provider_spiffe_mount_path = spiffe_socket_mount_path(socket_path); - protected_paths.push(&provider_spiffe_mount_path); - } validate_kubernetes_protected_path_conflicts( &config.containers.agent.volume_mounts, - &protected_paths, + KUBERNETES_DRIVER_PROTECTED_MOUNT_PATHS, )?; Ok(config) } @@ -3660,8 +5652,7 @@ fn sandbox_to_k8s_spec( spec: Option<&SandboxSpec>, params: &SandboxPodParams<'_>, ) -> Result { - let driver_config = - kubernetes_driver_config_for_spec(spec, provider_spiffe_socket_path(params))?; + let driver_config = kubernetes_driver_config_for_spec(spec)?; let mut root = serde_json::Map::new(); // Determine early whether OpenShell should inject its default workspace @@ -3681,7 +5672,6 @@ fn sandbox_to_k8s_spec( template, driver_gpu_requirements(spec.resource_requirements.as_ref()), &pod_env, - Some(spec), &driver_config, inject_workspace, params, @@ -3715,7 +5705,6 @@ fn sandbox_to_k8s_spec( &SandboxTemplate::default(), driver_gpu_requirements(spec.and_then(|s| s.resource_requirements.as_ref())), &pod_env, - spec, &driver_config, inject_workspace, params, @@ -3743,7 +5732,6 @@ fn sandbox_template_to_k8s( template, gpu_requirements.as_ref(), spec_environment, - None, &driver_config, inject_workspace, params, @@ -3764,7 +5752,6 @@ fn sandbox_template_to_k8s_with_gpu_requirements( template, gpu_requirements, spec_environment, - None, &driver_config, inject_workspace, params, @@ -3775,29 +5762,16 @@ fn sandbox_template_to_k8s_with_validated_config( template: &SandboxTemplate, gpu_requirements: Option<&GpuResourceRequirements>, spec_environment: &std::collections::HashMap, - sandbox_spec: Option<&openshell_core::proto::compute::v1::DriverSandboxSpec>, driver_config: &KubernetesSandboxDriverConfig, inject_workspace: bool, params: &SandboxPodParams<'_>, ) -> serde_json::Value { let mut metadata = serde_json::Map::new(); - let mut pod_labels = template + let pod_labels = template .labels .iter() .map(|(key, value)| (key.clone(), serde_json::Value::String(value.clone()))) .collect::>(); - if params.provider_spiffe_enabled { - pod_labels.insert( - LABEL_MANAGED_BY.to_string(), - serde_json::Value::String(LABEL_MANAGED_BY_VALUE.to_string()), - ); - if !params.sandbox_id.is_empty() { - pod_labels.insert( - LABEL_SANDBOX_ID.to_string(), - serde_json::Value::String(params.sandbox_id.to_string()), - ); - } - } if !pod_labels.is_empty() { metadata.insert("labels".to_string(), serde_json::Value::Object(pod_labels)); } @@ -3905,75 +5879,24 @@ fn sandbox_template_to_k8s_with_validated_config( if !image.is_empty() { container.insert("image".to_string(), serde_json::json!(image)); if let Some(policy) = params.image_pull_policy { - container.insert("imagePullPolicy".to_string(), serde_json::json!(policy)); + container.insert( + "imagePullPolicy".to_string(), + serde_json::json!(policy.as_kubernetes_str()), + ); } } - // Build environment variables - start with OpenShell-required vars - let env = build_env_list( - None, - &template.environment, - spec_environment, - sandbox_spec, - params.sandbox_id, - params.sandbox_name, - params.grpc_endpoint, - params.ssh_socket_path, - !params.client_tls_secret_name.is_empty(), - provider_spiffe_socket_path(params), - ); + let env = build_sandbox_env(&template.environment, spec_environment); container.insert("env".to_string(), serde_json::Value::Array(env)); - let mut capabilities: Vec<&str> = vec!["SYS_ADMIN", "NET_ADMIN", "SYS_PTRACE", "SYSLOG"]; - if use_user_namespaces { - // In a user namespace the bounding set is reset. SETUID/SETGID are - // needed for the supervisor to drop privileges to the sandbox user. - // DAC_READ_SEARCH is needed for cross-UID /proc//fd/ access - // for process identity resolution in network policy enforcement. - capabilities.extend(["SETUID", "SETGID", "DAC_READ_SEARCH"]); - } - let mut security_context = serde_json::json!({ - "capabilities": { - "add": capabilities - } - }); - if let Some(profile) = params.app_armor_profile { - security_context["appArmorProfile"] = app_armor_profile_to_k8s(profile); - } - container.insert("securityContext".to_string(), security_context); - - // Mount client TLS secret for mTLS to the server. Gateway identity uses - // the projected ServiceAccount bootstrap token. Provider token grants may - // additionally mount the SPIFFE Workload API socket. - let mut volume_mounts: Vec = Vec::new(); - if !params.client_tls_secret_name.is_empty() { - volume_mounts.push(serde_json::json!({ - "name": CLIENT_TLS_VOLUME_NAME, - "mountPath": openshell_core::container_paths::CLIENT_TLS_DIR, - "readOnly": true - })); - } - if params.provider_spiffe_enabled { - volume_mounts.push(serde_json::json!({ - "name": SPIFFE_WORKLOAD_API_VOLUME_NAME, - "mountPath": spiffe_socket_mount_path(params.provider_spiffe_workload_api_socket_path), - "readOnly": true, - })); - } - volume_mounts.push(serde_json::json!({ - "name": SERVICE_ACCOUNT_TOKEN_VOLUME_NAME, - "mountPath": SERVICE_ACCOUNT_TOKEN_MOUNT_PATH, - "readOnly": true, - })); - volume_mounts.extend( - driver_config - .containers - .agent - .volume_mounts - .iter() - .map(kubernetes_driver_volume_mount_to_k8s), - ); + let volume_mounts = driver_config + .containers + .agent + .volume_mounts + .iter() + .map(kubernetes_driver_volume_mount_to_k8s) + .collect::>(); container.insert( "volumeMounts".to_string(), serde_json::Value::Array(volume_mounts), @@ -3984,103 +5907,16 @@ fn sandbox_template_to_k8s_with_validated_config( } apply_agent_driver_resources(&mut container, &driver_config.containers.agent.resources); spec.insert( - "containers".to_string(), - serde_json::Value::Array(vec![serde_json::Value::Object(container)]), - ); - - // Add TLS secret volume. Combined mode uses mode 0400 because the - // supervisor starts as root and drops privileges before running workload - // children. Sidecar mode keeps the process supervisor non-root, so it uses - // pod fsGroup + 0440 to preserve gateway session and SSH control behavior. - let mut volumes: Vec = Vec::new(); - if !params.client_tls_secret_name.is_empty() { - let client_tls_default_mode = match params.topology { - SupervisorTopology::Combined => 0o400, - SupervisorTopology::Sidecar => 0o440, - }; - volumes.push(serde_json::json!({ - "name": CLIENT_TLS_VOLUME_NAME, - "secret": { - "secretName": params.client_tls_secret_name, - "defaultMode": client_tls_default_mode - } - })); - } - if has_upstream_proxy_credentials(params) { - let secret_name = params - .proxy_auth_secret_name - .expect("complete proxy credential reference has a Secret name"); - let secret_key = params - .proxy_auth_secret_key - .expect("complete proxy credential reference has a Secret key"); - // The credential volume is mounted only into the container that runs - // network supervision. Sidecar mode uses the pod fsGroup already - // required for its non-root network supervisor. - let default_mode = match params.topology { - SupervisorTopology::Combined => 0o400, - SupervisorTopology::Sidecar => 0o440, - }; - volumes.push(serde_json::json!({ - "name": UPSTREAM_PROXY_AUTH_VOLUME_NAME, - "secret": { - "secretName": secret_name, - "defaultMode": default_mode, - "items": [{ - "key": secret_key, - "path": upstream_proxy_auth_file_name(), - }] - } - })); - } - if params.provider_spiffe_enabled { - volumes.push(serde_json::json!({ - "name": SPIFFE_WORKLOAD_API_VOLUME_NAME, - "csi": { - "driver": "csi.spiffe.io", - "readOnly": true - } - })); - } - // Projected ServiceAccountToken volume — kubelet writes a short-lived - // audience-bound JWT into /var/run/secrets/openshell/token and rotates - // it automatically. The supervisor exchanges this for a gateway-minted - // JWT via `IssueSandboxToken` once at startup. In sidecar topology both - // supervisor containers run with the sandbox GID and need group-read access. - let sa_token_default_mode = match params.topology { - SupervisorTopology::Combined => 0o400, - SupervisorTopology::Sidecar => 0o440, - }; - volumes.push(serde_json::json!({ - "name": SERVICE_ACCOUNT_TOKEN_VOLUME_NAME, - "projected": { - "sources": [{ - "serviceAccountToken": { - "audience": "openshell-gateway", - "expirationSeconds": params.sa_token_ttl_secs, - "path": "token" - } - }], - "defaultMode": sa_token_default_mode - } - })); - volumes.extend( - driver_config - .volumes - .iter() - .map(kubernetes_driver_volume_to_k8s), + "containers".to_string(), + serde_json::Value::Array(vec![serde_json::Value::Object(container)]), ); - spec.insert("volumes".to_string(), serde_json::Value::Array(volumes)); - // Add hostAliases so sandbox pods can reach the Docker host. - if !params.host_gateway_ip.is_empty() { - spec.insert( - "hostAliases".to_string(), - serde_json::json!([{ - "ip": params.host_gateway_ip, - "hostnames": ["host.docker.internal", "host.openshell.internal"] - }]), - ); - } + let volumes = driver_config + .volumes + .iter() + .map(kubernetes_driver_volume_to_k8s) + .collect::>(); + spec.insert("volumes".to_string(), serde_json::Value::Array(volumes)); let mut template_value = serde_json::Map::new(); if !metadata.is_empty() { @@ -4090,19 +5926,7 @@ fn sandbox_template_to_k8s_with_validated_config( let mut result = serde_json::Value::Object(template_value); - match params.topology { - SupervisorTopology::Combined => { - apply_supervisor_sideload_with_params(&mut result, params); - } - SupervisorTopology::Sidecar => { - apply_supervisor_sidecar_topology( - &mut result, - &template.environment, - spec_environment, - params, - ); - } - } + apply_supervisor_sandbox_runtime_boundary(&mut result, params); // Inject workspace persistence (init container + PVC volume mount) so // that /sandbox data survives pod rescheduling. Skipped when the user @@ -4112,7 +5936,8 @@ fn sandbox_template_to_k8s_with_validated_config( &mut result, image, params.image_pull_policy, - params.sandbox_gid, + None, + Some((params.sandbox_uid, params.sandbox_gid)), ); } @@ -4205,16 +6030,6 @@ fn image_pull_secret_refs(secrets: &[String]) -> Vec { .collect() } -fn app_armor_profile_to_k8s(profile: &AppArmorProfile) -> serde_json::Value { - let mut value = serde_json::json!({ - "type": profile.kubernetes_type() - }); - if let Some(localhost_profile) = profile.localhost_profile() { - value["localhostProfile"] = serde_json::json!(localhost_profile); - } - value -} - fn container_resources( template: &SandboxTemplate, gpu_requirements: Option<&GpuResourceRequirements>, @@ -4279,144 +6094,22 @@ fn apply_gpu_limit(resources: &mut serde_json::Value, quantity: &str) { limits_obj.insert(GPU_RESOURCE_NAME.to_string(), serde_json::json!(quantity)); } -#[allow(clippy::too_many_arguments)] -fn build_env_list( - existing_env: Option<&Vec>, +fn build_sandbox_env( template_environment: &std::collections::HashMap, spec_environment: &std::collections::HashMap, - sandbox_spec: Option<&openshell_core::proto::compute::v1::DriverSandboxSpec>, - sandbox_id: &str, - sandbox_name: &str, - grpc_endpoint: &str, - ssh_socket_path: &str, - tls_enabled: bool, - provider_spiffe_socket_path: Option<&str>, ) -> Vec { - let mut env = existing_env.cloned().unwrap_or_default(); - apply_env_map(&mut env, template_environment); - apply_env_map(&mut env, spec_environment); - let mut user_env = template_environment.clone(); - user_env.extend(spec_environment.clone()); - if !user_env.is_empty() - && let Ok(json) = serde_json::to_string(&user_env) - { - upsert_env( - &mut env, - openshell_core::sandbox_env::USER_ENVIRONMENT, - &json, - ); + let mut env = Vec::new(); + for (name, value) in template_environment.iter().chain(spec_environment) { + if !name.starts_with("OPENSHELL_") || name == openshell_core::sandbox_env::LOG_LEVEL { + upsert_env(&mut env, name, value); + } } - let main_process = - openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec(sandbox_spec) - .expect("main process config serialization cannot fail"); upsert_env( &mut env, - openshell_core::sandbox_env::MAIN_PROCESS_SPEC, - &main_process, - ); - apply_required_env( - &mut env, - sandbox_id, - sandbox_name, - grpc_endpoint, - ssh_socket_path, - tls_enabled, - provider_spiffe_socket_path, - ); - env -} - -fn apply_env_map( - env: &mut Vec, - values: &std::collections::HashMap, -) { - for (key, value) in values { - upsert_env(env, key, value); - } -} - -// Required env vars are passed individually for clarity at call sites; grouping into a struct -// would not improve readability for this internal helper. -fn apply_required_env( - env: &mut Vec, - sandbox_id: &str, - sandbox_name: &str, - grpc_endpoint: &str, - ssh_socket_path: &str, - tls_enabled: bool, - provider_spiffe_socket_path: Option<&str>, -) { - upsert_env(env, openshell_core::sandbox_env::SANDBOX_ID, sandbox_id); - upsert_env(env, openshell_core::sandbox_env::SANDBOX, sandbox_name); - upsert_env(env, openshell_core::sandbox_env::ENDPOINT, grpc_endpoint); - upsert_env( - env, openshell_core::sandbox_env::TELEMETRY_ENABLED, openshell_core::telemetry::enabled_env_value(), ); - // Runtime capabilities are driver-owned. Kubernetes topologies do not yet - // provide the complete policy DNS and transparent TCP substrate. - upsert_env( - env, - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, - "", - ); - if !ssh_socket_path.is_empty() { - upsert_env( - env, - openshell_core::sandbox_env::SSH_SOCKET_PATH, - ssh_socket_path, - ); - } - // TLS cert paths for sandbox-to-server mTLS. Only set when TLS is enabled - // and the client TLS secret is mounted into the sandbox pod. - if tls_enabled { - upsert_env( - env, - openshell_core::sandbox_env::TLS_CA, - "/etc/openshell-tls/client/ca.crt", - ); - upsert_env( - env, - openshell_core::sandbox_env::TLS_CERT, - "/etc/openshell-tls/client/tls.crt", - ); - upsert_env( - env, - openshell_core::sandbox_env::TLS_KEY, - "/etc/openshell-tls/client/tls.key", - ); - } - // Projected ServiceAccount token written by kubelet (see the volume - // definition in `sandbox_template_to_k8s`). The supervisor reads this - // and exchanges it for a gateway-minted JWT via `IssueSandboxToken`. - upsert_env( - env, - openshell_core::sandbox_env::K8S_SA_TOKEN_FILE, - "/var/run/secrets/openshell/token", - ); - if let Some(socket_path) = provider_spiffe_socket_path { - upsert_env( - env, - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, - socket_path, - ); - } -} - -fn provider_spiffe_socket_path<'a>(params: &'a SandboxPodParams<'a>) -> Option<&'a str> { - params - .provider_spiffe_enabled - .then_some(params.provider_spiffe_workload_api_socket_path) -} - -fn spiffe_socket_mount_path(socket_path: &str) -> String { - Path::new(socket_path) - .parent() - .and_then(Path::to_str) - .filter(|path| !path.is_empty() && *path != "/") - .expect("provider SPIFFE socket path should be validated before pod rendering") - .to_string() + env } fn upsert_env(env: &mut Vec, name: &str, value: &str) { @@ -4452,10 +6145,6 @@ fn remove_env(env: &mut Vec, name: &str) { env.retain(|item| item.get("name").and_then(|value| value.as_str()) != Some(name)); } -fn remove_volume_mount(volume_mounts: &mut Vec, name: &str) { - volume_mounts.retain(|mount| mount.get("name").and_then(|value| value.as_str()) != Some(name)); -} - /// Extract a string value from the template's `platform_config` Struct. fn platform_config_string(template: &SandboxTemplate, key: &str) -> Option { let config = template.platform_config.as_ref()?; @@ -4490,6 +6179,40 @@ fn platform_config_struct(template: &SandboxTemplate, key: &str) -> Option std::collections::HashMap { + object + .data + .pointer("/spec/podTemplate/spec/containers/0/env") + .and_then(serde_json::Value::as_array) + .into_iter() + .flatten() + .filter_map(|entry| { + let name = entry.get("name")?.as_str()?; + let value = entry.get("value")?.as_str()?; + (!name.starts_with("OPENSHELL_")).then(|| (name.to_string(), value.to_string())) + }) + .collect() +} + +fn required_sandbox_annotation( + object: &DynamicObject, + name: &str, +) -> Result { + object + .metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(name)) + .cloned() + .ok_or_else(|| { + KubernetesDriverError::Precondition(format!( + "Sandbox resource is missing required annotation {name}" + )) + }) +} + fn status_from_object(obj: &DynamicObject) -> Option { let status = obj.data.get("status")?; let status_obj = status.as_object()?; @@ -4528,9 +6251,121 @@ fn status_from_object(obj: &DynamicObject) -> Option { .to_string(), conditions, deleting: obj.metadata.deletion_timestamp.is_some(), + ..Default::default() }) } +async fn create_or_validate_sandbox_runtime_fence( + policies: &Api, + expected: &NetworkPolicy, +) -> Result<(), KubernetesDriverError> { + let name = expected.metadata.name.as_deref().unwrap_or_default(); + match tokio::time::timeout(KUBE_API_TIMEOUT, policies.get_opt(name)).await { + Ok(Ok(Some(existing))) => { + return validate_sandbox_runtime_fence(&existing, expected); + } + Ok(Ok(None)) => {} + Ok(Err(error)) => return Err(KubernetesDriverError::from_kube(error)), + Err(_) => { + return Err(KubernetesDriverError::Message( + "timed out reading sandbox-runtime workload fence".to_string(), + )); + } + } + + match tokio::time::timeout( + KUBE_API_TIMEOUT, + policies.create(&PostParams::default(), expected), + ) + .await + { + Ok(Ok(_)) => Ok(()), + Ok(Err(KubeError::Api(error))) if error.code == 409 => { + let existing = tokio::time::timeout(KUBE_API_TIMEOUT, policies.get(name)) + .await + .map_err(|_| { + KubernetesDriverError::Message( + "timed out validating existing sandbox-runtime workload fence".to_string(), + ) + })? + .map_err(KubernetesDriverError::from_kube)?; + validate_sandbox_runtime_fence(&existing, expected) + } + Ok(Err(error)) => Err(KubernetesDriverError::from_kube(error)), + Err(_) => Err(KubernetesDriverError::Message( + "timed out creating sandbox-runtime workload fence".to_string(), + )), + } +} + +fn validate_sandbox_runtime_fence( + existing: &NetworkPolicy, + expected: &NetworkPolicy, +) -> Result<(), KubernetesDriverError> { + if sandbox_runtime_fence_matches(existing, expected) { + Ok(()) + } else { + let name = expected.metadata.name.as_deref().unwrap_or_default(); + Err(KubernetesDriverError::Precondition(format!( + "sandbox-runtime workload fence {name} exists but does not match the intended enforcement" + ))) + } +} + +fn sandbox_runtime_fence_matches(existing: &NetworkPolicy, expected: &NetworkPolicy) -> bool { + fn normalized_spec(mut spec: Option) -> Option { + if let Some(spec) = spec.as_mut() { + // The Kubernetes API server omits explicitly empty rule arrays when it + // persists a NetworkPolicy. For a policy type named in `policyTypes`, + // an omitted rule array and an empty rule array both deny all traffic. + if spec.egress.as_ref().is_some_and(Vec::is_empty) { + spec.egress = None; + } + if spec.ingress.as_ref().is_some_and(Vec::is_empty) { + spec.ingress = None; + } + } + spec + } + + fn contains_required_metadata( + actual: &Option>, + required: &Option>, + ) -> bool { + required.as_ref().is_none_or(|required| { + actual.as_ref().is_some_and(|actual| { + required + .iter() + .all(|(key, value)| actual.get(key) == Some(value)) + }) + }) + } + + normalized_spec(existing.spec.clone()) == normalized_spec(expected.spec.clone()) + && contains_required_metadata(&existing.metadata.labels, &expected.metadata.labels) + && contains_required_metadata( + &existing.metadata.annotations, + &expected.metadata.annotations, + ) +} + +fn sandbox_runtime_namespace_fence_generation_matches( + policy: &NetworkPolicy, + sandbox: &DynamicObject, +) -> bool { + let Some(annotations) = sandbox.metadata.annotations.as_ref() else { + return false; + }; + policy.metadata.uid.as_deref() + == annotations + .get(ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID) + .map(String::as_str) + && policy.metadata.resource_version.as_deref() + == annotations + .get(ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_VERSION) + .map(String::as_str) +} + fn kubernetes_sandbox_has_stopped_condition(obj: &DynamicObject) -> bool { obj.data .get("status") @@ -4637,18 +6472,68 @@ fn sandbox_operating_state_patch( running: bool, ) -> serde_json::Value { if api_version == SANDBOX_VERSION_V1BETA1 { - serde_json::json!({ - "metadata": {"resourceVersion": resource_version}, - "spec": {"operatingMode": if running { "Running" } else { "Suspended" }} - }) + if running { + serde_json::json!({ + "metadata": {"resourceVersion": resource_version}, + "spec": {"operatingMode": "Running"} + }) + } else { + sandbox_runtime_rollback_patch(api_version, resource_version) + } } else { - serde_json::json!({ - "metadata": {"resourceVersion": resource_version}, - "spec": {"replicas": i32::from(running)} - }) + if running { + serde_json::json!({ + "metadata": {"resourceVersion": resource_version}, + "spec": {"replicas": 1} + }) + } else { + sandbox_runtime_rollback_patch(api_version, resource_version) + } } } +fn sandbox_runtime_rollback_patch(api_version: &str, resource_version: &str) -> serde_json::Value { + let desired_state = if api_version == SANDBOX_VERSION_V1BETA1 { + serde_json::json!({"operatingMode": "Suspended"}) + } else { + serde_json::json!({"replicas": 0}) + }; + serde_json::json!({ + "metadata": { + "resourceVersion": resource_version, + "annotations": { + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING: "true", + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT: openshell_core::time::now_ms().to_string(), + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION: "stop", + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE: SandboxRuntimeBootstrapPhase::RollingBack.as_str(), + ANNOTATION_SANDBOX_RUNTIME_READINESS: "unavailable", + }, + }, + "spec": desired_state, + }) +} + +fn sandbox_runtime_rollback_completion_patch(resource_version: &str) -> serde_json::Value { + serde_json::json!({ + "metadata": { + "resourceVersion": resource_version, + "annotations": { + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_SESSION: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_GENERATION: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_VERSION: serde_json::Value::Null, + ANNOTATION_SANDBOX_RUNTIME_READINESS: "unavailable", + }, + } + }) +} + fn condition_from_value(value: &serde_json::Value) -> Option { let obj = value.as_object()?; Some(SandboxCondition { @@ -4688,9 +6573,9 @@ fn spawn_namespace_label_watcher( tokio::spawn(async move { let mut retry_attempt = 0; + let mut relisted_names = std::collections::BTreeSet::new(); loop { let mut stream = watcher::watcher(ns_api.clone(), watcher_config.clone()).boxed(); - let mut relisted_names = BTreeSet::new(); loop { let event = tokio::select! { @@ -4703,9 +6588,43 @@ fn spawn_namespace_label_watcher( } }; match event { - Ok(Some(event)) => { + Ok(Some(Event::Apply(ns))) => { + retry_attempt = 0; + if let Some(name) = ns.metadata.name.as_deref() + && allowlist.insert(name.to_string()) + { + info!(namespace = name, "operator namespace added to allowlist"); + } + } + Ok(Some(Event::Delete(ns))) => { retry_attempt = 0; - apply_namespace_watch_event(&allowlist, &mut relisted_names, event); + if let Some(name) = ns.metadata.name.as_deref() + && allowlist.remove(name) + { + info!( + namespace = name, + "operator namespace removed from allowlist" + ); + } + } + Ok(Some(Event::Init)) => { + retry_attempt = 0; + relisted_names.clear(); + } + Ok(Some(Event::InitApply(ns))) => { + retry_attempt = 0; + if let Some(name) = ns.metadata.name { + relisted_names.insert(name); + } + } + Ok(Some(Event::InitDone)) => { + retry_attempt = 0; + let count = relisted_names.len(); + allowlist.replace(std::mem::take(&mut relisted_names)); + info!( + total = count, + "operator namespace allowlist replaced from full relist" + ); } Ok(None) => { warn!("operator namespace watcher stream ended unexpectedly"); @@ -4747,7 +6666,7 @@ fn namespace_watcher_retry_delay(attempt: u32, jitter_seed: u64) -> Duration { Duration::from_secs(base_secs + jitter_secs) } -fn load_namespace_file(path: &Path) -> Result, String> { +fn load_namespace_file(path: &Path) -> Result, String> { let contents = std::fs::read_to_string(path) .map_err(|e| format!("failed to read {}: {e}", path.display()))?; let names: Vec = serde_json::from_str(&contents) @@ -4755,47 +6674,6 @@ fn load_namespace_file(path: &Path) -> Result, String> { Ok(names.into_iter().collect()) } -fn apply_namespace_watch_event( - allowlist: &OperatorNamespaceAllowlist, - relisted_names: &mut BTreeSet, - event: Event, -) { - match event { - Event::Apply(ns) => { - if let Some(name) = ns.metadata.name - && allowlist.insert(name.clone()) - { - info!(namespace = name, "operator namespace added to allowlist"); - } - } - Event::Delete(ns) => { - if let Some(name) = ns.metadata.name - && allowlist.remove(&name) - { - info!( - namespace = name, - "operator namespace removed from allowlist" - ); - } - } - Event::Init => relisted_names.clear(), - Event::InitApply(ns) => { - if let Some(name) = ns.metadata.name { - relisted_names.insert(name); - } - } - Event::InitDone => { - // Readers must see a complete snapshot, including during interrupted relists. - let count = relisted_names.len(); - allowlist.replace(std::mem::take(relisted_names)); - info!( - total = count, - "operator namespace allowlist replaced from full relist" - ); - } - } -} - fn spawn_namespace_file_watcher( path: PathBuf, allowlist: OperatorNamespaceAllowlist, @@ -4935,6 +6813,16 @@ mod tests { static ENV_LOCK: std::sync::LazyLock> = std::sync::LazyLock::new(|| std::sync::Mutex::new(())); + #[test] + fn boundary_authority_uses_stable_service_dns_name() { + let names = SandboxRuntimeNames::new("sandbox-1"); + + assert_eq!( + boundary_service_authority("workspace-a", &names, 5500), + "os-boundary-sandbox-1.workspace-a.svc:5500" + ); + } + #[tokio::test] async fn tracing_create_sandbox_failure_exports_a_kubernetes_operation_span() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; @@ -4969,6 +6857,46 @@ mod tests { provider.shutdown().unwrap(); } + #[tokio::test] + async fn start_sandbox_span_does_not_capture_launch_authentication() { + use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); + let driver = KubernetesComputeDriver::new_for_test(KubernetesComputeConfig::default()); + + driver + .start_sandbox( + "sandbox-1", + "invalid-generation", + b"secret-launch-authentication", + ) + .with_subscriber(subscriber) + .await + .expect_err("invalid generation must fail before contacting Kubernetes"); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|span| span.name == "kubernetes.start_sandbox") + .expect("start operation span"); + assert!(span.attributes.iter().all(|attribute| { + !matches!( + attribute.key.as_str(), + "launch_authentication" | "encoded_authentication" + ) + })); + provider.shutdown().unwrap(); + } + #[tokio::test] async fn sandbox_annotation_propagates_the_active_w3c_trace_context() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; @@ -5033,6 +6961,32 @@ mod tests { }) } + #[test] + fn resource_version_conflicts_are_not_reported_as_duplicate_sandboxes() { + let conflict = KubeError::Api(kube::core::ErrorResponse { + status: "Failure".to_string(), + message: "the object has been modified".to_string(), + reason: "Conflict".to_string(), + code: 409, + }); + assert!(is_kube_resource_version_conflict(&conflict)); + assert!(matches!( + KubernetesDriverError::from_kube(conflict), + KubernetesDriverError::Message(_) + )); + + let duplicate = KubeError::Api(kube::core::ErrorResponse { + status: "Failure".to_string(), + message: "already exists".to_string(), + reason: "AlreadyExists".to_string(), + code: 409, + }); + assert!(matches!( + KubernetesDriverError::from_kube(duplicate), + KubernetesDriverError::AlreadyExists + )); + } + fn expired_watch_error() -> watcher::Error { watcher::Error::WatchError(kube::core::ErrorResponse { status: "Failure".to_string(), @@ -5043,7 +6997,7 @@ mod tests { } #[tokio::test] - async fn sandbox_watcher_error_does_not_hide_relist() { + async fn sandbox_watcher_error_does_not_hide_restarted_recovery_event() { let recovered = DynamicObject { types: None, metadata: ObjectMeta { @@ -5052,15 +7006,10 @@ mod tests { }, data: serde_json::json!({}), }; - let source = futures::stream::iter([ - Err(expired_watch_error()), - Ok(Event::Init), - Ok(Event::InitApply(recovered)), - Ok(Event::InitDone), - ]); + let source = + futures::stream::iter([Err(expired_watch_error()), Ok(Event::InitApply(recovered))]); let mut stream = continue_on_watcher_errors(source, "sandbox-resource"); - assert!(matches!(stream.next().await, Some(Event::Init))); let event = stream .next() .await @@ -5069,7 +7018,6 @@ mod tests { panic!("expected kube-runtime recovery to emit InitApply"); }; assert_eq!(object.metadata.name.as_deref(), Some("recovered-sandbox")); - assert!(matches!(stream.next().await, Some(Event::InitDone))); assert!( stream.next().await.is_none(), "source closure must be preserved" @@ -5096,15 +7044,17 @@ mod tests { }, data: serde_json::json!({}), }; - let source = futures::stream::iter([ - Err(expired_watch_error()), - Ok(Event::Init), - Ok(Event::InitApply(recovered)), - Ok(Event::InitDone), - ]) - .chain(futures::stream::pending()); + let source = + futures::stream::iter([Err(expired_watch_error()), Ok(Event::InitApply(recovered))]) + .chain(futures::stream::pending()); let sandbox_stream = recovering_watcher_stream(source, "sandbox-resource").boxed(); - let mut outward = cluster_wide_watch_stream(sandbox_stream, "default".to_string()); + let driver = KubernetesComputeDriver::new_for_test(KubernetesComputeConfig::default()); + let mut outward = cluster_wide_watch_stream( + sandbox_stream, + "default".to_string(), + driver.watch_client.clone(), + driver, + ); let event = outward .next() @@ -5127,76 +7077,24 @@ mod tests { } #[tokio::test] - async fn kubernetes_event_watcher_error_does_not_hide_relist() { + async fn kubernetes_event_watcher_error_does_not_hide_restarted_recovery_event() { let source = futures::stream::iter([ Err(expired_watch_error()), - Ok(Event::Init), Ok(Event::InitApply(KubeEventObj::default())), - Ok(Event::InitDone), ]); - let mut stream = continue_on_watcher_errors(source, "kubernetes-event"); - - let event = stream - .next() - .await - .expect("410 Expired must not terminate the watcher stream"); - assert!(matches!(event, Event::Init)); - assert!(matches!(stream.next().await, Some(Event::InitApply(_)))); - assert!(matches!(stream.next().await, Some(Event::InitDone))); - assert!( - stream.next().await.is_none(), - "source closure must be preserved" - ); - } - - #[test] - fn namespace_relist_replaces_only_completed_snapshots() { - let allowlist = OperatorNamespaceAllowlist::from_set(BTreeSet::from(["old".to_string()])); - let mut pending = BTreeSet::new(); - let namespace = |name: &str| Namespace { - metadata: ObjectMeta { - name: Some(name.to_string()), - ..Default::default() - }, - ..Default::default() - }; - let config = KubernetesComputeConfig { - workspace_mode: WorkspaceMode::Operator, - ..Default::default() - }; - - apply_namespace_watch_event(&allowlist, &mut pending, Event::Init); - apply_namespace_watch_event( - &allowlist, - &mut pending, - Event::InitApply(namespace("partial")), - ); - assert!(accepts_auth_namespace(&config, Some(&allowlist), "old")); - assert!(!accepts_auth_namespace( - &config, - Some(&allowlist), - "partial" - )); - - apply_namespace_watch_event(&allowlist, &mut pending, Event::Init); - apply_namespace_watch_event(&allowlist, &mut pending, Event::InitApply(namespace("new"))); - apply_namespace_watch_event(&allowlist, &mut pending, Event::InitDone); - assert!(accepts_auth_namespace(&config, Some(&allowlist), "new")); - assert!(!accepts_auth_namespace(&config, Some(&allowlist), "old")); - assert!(!accepts_auth_namespace( - &config, - Some(&allowlist), - "partial" - )); - - apply_namespace_watch_event(&allowlist, &mut pending, Event::Apply(namespace("live"))); - assert!(accepts_auth_namespace(&config, Some(&allowlist), "live")); - apply_namespace_watch_event(&allowlist, &mut pending, Event::Delete(namespace("live"))); - assert!(!accepts_auth_namespace(&config, Some(&allowlist), "live")); + let mut stream = continue_on_watcher_errors(source, "kubernetes-event"); - apply_namespace_watch_event(&allowlist, &mut pending, Event::Init); - apply_namespace_watch_event(&allowlist, &mut pending, Event::InitDone); - assert!(!accepts_auth_namespace(&config, Some(&allowlist), "new")); + let event = stream + .next() + .await + .expect("410 Expired must not terminate the watcher stream"); + let Event::InitApply(_event) = event else { + panic!("expected kube-runtime recovery to emit InitApply"); + }; + assert!( + stream.next().await.is_none(), + "source closure must be preserved" + ); } fn authenticated_token_review(username: &str) -> TokenReviewStatus { @@ -5322,7 +7220,7 @@ mod tests { } #[test] - fn pod_identity_requires_matching_uid_annotation_and_controlling_owner() { + fn pod_identity_requires_matching_uid_annotation_and_expected_owner_kind() { let owner = sandbox_owner_for_test("sandbox-a", "sandbox-uid-a"); let pod = Pod { metadata: ObjectMeta { @@ -5339,7 +7237,7 @@ mod tests { validate_pod_uid(&pod, "pod-uid-a").expect("matching pod UID"); assert_eq!(pod_sandbox_id(&pod).unwrap(), "sandbox-id-a"); - assert_eq!(sandbox_owner_reference(&pod).unwrap(), &owner); + assert_eq!(sandbox_owner_reference(&pod, true).unwrap(), &owner); let error = validate_pod_uid(&pod, "other-pod-uid").unwrap_err(); assert_eq!(error.code(), tonic::Code::PermissionDenied); @@ -5351,8 +7249,133 @@ mod tests { let mut non_controlling = pod; non_controlling.metadata.owner_references.as_mut().unwrap()[0].controller = Some(false); - let error = sandbox_owner_reference(&non_controlling).unwrap_err(); + let error = sandbox_owner_reference(&non_controlling, true).unwrap_err(); assert_eq!(error.code(), tonic::Code::PermissionDenied); + assert!(sandbox_owner_reference(&non_controlling, false).is_ok()); + } + + #[test] + fn proxy_control_identity_requires_exact_pair_and_role_labels() { + let sandbox_id = "sandbox-id-a"; + let mut pod = Pod { + metadata: ObjectMeta { + labels: Some(BTreeMap::from([ + (LABEL_SANDBOX_ID.to_string(), sandbox_id.to_string()), + ( + BOUNDARY_PAIR_LABEL.to_string(), + crate::sandbox_runtime::pair_label_value(sandbox_id), + ), + (BOUNDARY_ROLE_LABEL.to_string(), "supervisor".to_string()), + ])), + ..Default::default() + }, + ..Default::default() + }; + validate_proxy_control_labels(&pod, sandbox_id).unwrap(); + + pod.metadata + .labels + .as_mut() + .unwrap() + .insert(BOUNDARY_ROLE_LABEL.to_string(), "workload".to_string()); + assert_eq!( + validate_proxy_control_labels(&pod, sandbox_id) + .unwrap_err() + .code(), + tonic::Code::PermissionDenied + ); + } + + #[test] + fn sandbox_authentication_requires_the_paired_supervisor() { + require_proxy_control_authentication(true).expect("paired supervisor is trusted"); + assert_eq!( + require_proxy_control_authentication(false) + .expect_err("workload JWT must not authenticate directly") + .code(), + tonic::Code::PermissionDenied + ); + } + + #[test] + fn sandbox_runtime_fence_validation_accepts_api_normalization_and_injected_metadata() { + let names = SandboxRuntimeNames::new("sandbox-id-a"); + let mut expected = workload_fence("namespace-a", &names, 5000).workload_policy; + expected.metadata.labels = Some(BTreeMap::from([( + LABEL_MANAGED_BY.to_string(), + LABEL_MANAGED_BY_VALUE.to_string(), + )])); + + let mut persisted = expected.clone(); + persisted.spec.as_mut().unwrap().egress = None; + persisted + .metadata + .labels + .as_mut() + .unwrap() + .insert("admission.example/injected".to_string(), "true".to_string()); + assert!(sandbox_runtime_fence_matches(&persisted, &expected)); + assert!(validate_sandbox_runtime_fence(&persisted, &expected).is_ok()); + + persisted.spec.as_mut().unwrap().policy_types = Some(vec!["Ingress".to_string()]); + assert!(!sandbox_runtime_fence_matches(&persisted, &expected)); + assert!(matches!( + validate_sandbox_runtime_fence(&persisted, &expected), + Err(KubernetesDriverError::Precondition(_)) + )); + } + + #[test] + fn sandbox_runtime_bootstrap_marker_and_age_gate_rollback() { + let started = Duration::from_hours(490_896); + let mut object: DynamicObject = serde_json::from_value(serde_json::json!({ + "apiVersion": "agents.x-k8s.io/v1beta1", + "kind": "Sandbox", + "metadata": { + "name": "sandbox-a", + "creationTimestamp": "2020-01-01T00:00:00Z", + "annotations": { + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING: "true", + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT: started.as_millis().to_string(), + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION: "restart", + } + } + })) + .unwrap(); + assert!(sandbox_runtime_bootstrap_in_progress(&object)); + assert_eq!( + sandbox_runtime_bootstrap_operation(&object), + Some("restart") + ); + let started_at = SystemTime::UNIX_EPOCH + started; + assert!(!sandbox_runtime_bootstrap_is_stale( + &object, + started_at + SANDBOX_RUNTIME_BOOTSTRAP_GRACE - Duration::from_secs(1), + SANDBOX_RUNTIME_BOOTSTRAP_GRACE + )); + assert!(sandbox_runtime_bootstrap_is_stale( + &object, + started_at + SANDBOX_RUNTIME_BOOTSTRAP_GRACE, + SANDBOX_RUNTIME_BOOTSTRAP_GRACE + )); + object + .metadata + .annotations + .as_mut() + .unwrap() + .remove(ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT); + assert!(sandbox_runtime_bootstrap_is_stale( + &object, + started_at, + SANDBOX_RUNTIME_BOOTSTRAP_GRACE + )); + object + .metadata + .annotations + .as_mut() + .unwrap() + .remove(ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING); + assert!(!sandbox_runtime_bootstrap_in_progress(&object)); } #[test] @@ -5386,11 +7409,20 @@ mod tests { fn lifecycle_patch_uses_version_specific_operating_state() { let beta_stop = sandbox_operating_state_patch(SANDBOX_VERSION_V1BETA1, "42", false); assert_eq!(beta_stop["metadata"]["resourceVersion"], "42"); + assert_eq!( + beta_stop["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING], + "true" + ); + assert_eq!( + beta_stop["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE], + SandboxRuntimeBootstrapPhase::RollingBack.as_str() + ); assert_eq!(beta_stop["spec"]["operatingMode"], "Suspended"); assert!(beta_stop["spec"].get("replicas").is_none()); let alpha_start = sandbox_operating_state_patch(SANDBOX_VERSION_V1ALPHA1, "43", true); assert_eq!(alpha_start["metadata"]["resourceVersion"], "43"); + assert!(alpha_start["metadata"].get("annotations").is_none()); assert_eq!(alpha_start["spec"]["replicas"], 1); assert!(alpha_start["spec"].get("operatingMode").is_none()); } @@ -5592,7 +7624,7 @@ mod tests { ..Default::default() }; - let err = kubernetes_driver_config_for_spec(sandbox.spec.as_ref(), None).unwrap_err(); + let err = kubernetes_driver_config_for_spec(sandbox.spec.as_ref()).unwrap_err(); assert!(err.contains("unknown field")); assert!(err.contains("gpu_device_ids")); } @@ -5941,7 +7973,7 @@ mod tests { ..SandboxSpec::default() }; - let err = kubernetes_driver_config_for_spec(Some(&spec), None).unwrap_err(); + let err = kubernetes_driver_config_for_spec(Some(&spec)).unwrap_err(); assert!(err.contains("/var/run/secrets/openshell")); } @@ -5969,7 +8001,7 @@ mod tests { ..SandboxSpec::default() }; - kubernetes_driver_config_for_spec(Some(&spec), None) + kubernetes_driver_config_for_spec(Some(&spec)) .expect("SPIFFE workload path should only be protected when SPIFFE is enabled"); } @@ -6084,7 +8116,11 @@ mod tests { CLIENT_TLS_VOLUME_NAME, SERVICE_ACCOUNT_TOKEN_VOLUME_NAME, SPIFFE_WORKLOAD_API_VOLUME_NAME, - SUPERVISOR_VOLUME_NAME, + SANDBOX_RUNTIME_VOLUME_NAME, + SANDBOX_STATE_VOLUME_NAME, + SANDBOX_BOOTSTRAP_VOLUME_NAME, + SANDBOX_POD_IDENTITY_VOLUME_NAME, + SANDBOX_PROXY_CA_VOLUME_NAME, WORKSPACE_VOLUME_NAME, ] { let template = SandboxTemplate { @@ -6107,12 +8143,7 @@ mod tests { #[test] fn reserved_kubernetes_volume_names_cover_managed_pod_volumes() { - let params = SandboxPodParams { - client_tls_secret_name: "openshell-client-tls-secret", - provider_spiffe_enabled: true, - provider_spiffe_workload_api_socket_path: "/spiffe-workload-api/spire-agent.sock", - ..SandboxPodParams::default() - }; + let params = SandboxPodParams::default(); let pod_template = sandbox_template_to_k8s( &SandboxTemplate::default(), false, @@ -6135,36 +8166,6 @@ mod tests { } } - #[test] - fn driver_config_rejects_runtime_provider_spiffe_mount_path() { - let spec = SandboxSpec { - template: Some(SandboxTemplate { - driver_config: Some(json_struct(serde_json::json!({ - "volumes": [{ - "name": "user-data", - "persistent_volume_claim": {"claim_name": "pvc-user-data"} - }], - "containers": { - "agent": { - "volume_mounts": [{ - "name": "user-data", - "mount_path": "/custom-spiffe" - }] - } - } - }))), - ..SandboxTemplate::default() - }), - ..SandboxSpec::default() - }; - - let err = - kubernetes_driver_config_for_spec(Some(&spec), Some("/custom-spiffe/spire-agent.sock")) - .unwrap_err(); - - assert!(err.contains("/custom-spiffe")); - } - #[test] fn validate_rejects_zero_gpu_count() { let sandbox = Sandbox { @@ -6217,757 +8218,161 @@ mod tests { ); assert_eq!( - metadata.get(PROGRESS_COMPLETE_STEP_KEY).map(String::as_str), - Some(PROGRESS_STEP_PULLING_IMAGE) - ); - assert_eq!( - metadata - .get(PROGRESS_COMPLETE_LABEL_KEY) - .map(String::as_str), - Some("Image pulled (42 MB)") - ); - assert_eq!( - metadata.get(PROGRESS_ACTIVE_STEP_KEY).map(String::as_str), - Some(PROGRESS_STEP_STARTING_SANDBOX) - ); - } - - #[test] - fn supervisor_sideload_injects_run_as_user_zero() { - let mut pod_template = serde_json::json!({ - "spec": { - "containers": [{ - "name": "agent", - "image": "custom-image:latest", - "securityContext": { - "capabilities": { - "add": ["SYS_ADMIN", "NET_ADMIN", "SYS_PTRACE", "SYSLOG"] - } - } - }] - } - }); - - apply_supervisor_sideload( - &mut pod_template, - "custom-image:latest", - Some("IfNotPresent"), - SupervisorSideloadMethod::InitContainer, - 1500, // sandbox_uid - 1500, // sandbox_gid - ); - - let sc = &pod_template["spec"]["containers"][0]["securityContext"]; - assert_eq!(sc["runAsUser"], 0, "runAsUser must be 0 for supervisor"); - // Capabilities should be preserved - assert!( - sc["capabilities"]["add"] - .as_array() - .unwrap() - .contains(&serde_json::json!("SYS_ADMIN")) - ); - } - - #[test] - fn supervisor_sideload_replaces_spoofed_identity_environment() { - let mut pod_template = serde_json::json!({ - "spec": { - "containers": [{ - "name": "agent", - "image": "custom-image:latest", - "env": [ - {"name": openshell_core::sandbox_env::OCI_IMAGE_USER, "value": "spoofed"}, - {"name": openshell_core::sandbox_env::SANDBOX_UID, "value": "9999"}, - {"name": openshell_core::sandbox_env::SANDBOX_GID, "value": "9999"}, - {"name": openshell_core::sandbox_env::OCI_IMAGE_USER, "value": "duplicate"} - ] - }] - } - }); - - apply_supervisor_sideload( - &mut pod_template, - "supervisor-image:latest", - Some("IfNotPresent"), - SupervisorSideloadMethod::InitContainer, - 1500, - 1600, - ); - - let agent = &pod_template["spec"]["containers"][0]; - let env = agent["env"].as_array().unwrap(); - for name in [ - openshell_core::sandbox_env::OCI_IMAGE_USER, - openshell_core::sandbox_env::SANDBOX_UID, - openshell_core::sandbox_env::SANDBOX_GID, - ] { - assert_eq!( - env.iter().filter(|item| item["name"] == name).count(), - 1, - "{name} must have one driver-owned value" - ); - } - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::OCI_IMAGE_USER), - Some("") - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::SANDBOX_UID), - Some("1500") - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::SANDBOX_GID), - Some("1600") - ); - } - - #[test] - fn supervisor_sideload_adds_security_context_when_missing() { - let mut pod_template = serde_json::json!({ - "spec": { - "containers": [{ - "name": "agent", - "image": "custom-image:latest" - }] - } - }); - - apply_supervisor_sideload( - &mut pod_template, - "supervisor-image:latest", - Some("IfNotPresent"), - SupervisorSideloadMethod::InitContainer, - 1000, // sandbox_uid - 1000, // sandbox_gid - ); - - let sc = &pod_template["spec"]["containers"][0]["securityContext"]; - assert_eq!( - sc["runAsUser"], 0, - "runAsUser must be 0 even when no prior securityContext" - ); - } - - #[test] - fn supervisor_sideload_injects_emptydir_volume_init_container_and_mount() { - let mut pod_template = serde_json::json!({ - "spec": { - "containers": [{ - "name": "agent", - "image": "custom-image:latest" - }] - } - }); - - apply_supervisor_sideload( - &mut pod_template, - "supervisor-image:latest", - Some("IfNotPresent"), - SupervisorSideloadMethod::InitContainer, - 1000, // sandbox_uid - 1000, // sandbox_gid - ); - - // Volume should be an emptyDir - let volumes = pod_template["spec"]["volumes"] - .as_array() - .expect("volumes should exist"); - assert_eq!(volumes.len(), 1); - assert_eq!(volumes[0]["name"], SUPERVISOR_VOLUME_NAME); - assert!( - volumes[0]["emptyDir"].is_object(), - "volume should be emptyDir, not hostPath" - ); - - // Init container should use the supervisor image, not the sandbox image - let init_containers = pod_template["spec"]["initContainers"] - .as_array() - .expect("initContainers should exist"); - assert_eq!(init_containers.len(), 1); - assert_eq!(init_containers[0]["name"], SUPERVISOR_INIT_CONTAINER_NAME); - assert_eq!(init_containers[0]["image"], "supervisor-image:latest"); - assert_eq!(init_containers[0]["imagePullPolicy"], "IfNotPresent"); - - // The init container must invoke the binary directly with - // `copy-self ` rather than depending on shell utilities. - let init_command = init_containers[0]["command"] - .as_array() - .expect("init container command should be set"); - assert_eq!(init_command.len(), 3, "expected [binary, copy-self, dest]"); - assert_eq!(init_command[0], SUPERVISOR_IMAGE_BINARY_PATH); - assert_eq!(init_command[1], "copy-self"); - assert_eq!( - init_command[2].as_str().unwrap(), - format!("{SUPERVISOR_MOUNT_PATH}/openshell-sandbox") - ); - assert!( - !init_command.iter().any(|v| v == "sh"), - "init container must not depend on a shell" - ); - - // `--workdir` is optional for standalone supervisor invocations and - // has no implicit default, so Kubernetes must pass its fixed workspace. - let command = pod_template["spec"]["containers"][0]["command"] - .as_array() - .expect("command should be set"); - assert_eq!( - command[0].as_str().unwrap(), - format!("{SUPERVISOR_MOUNT_PATH}/openshell-sandbox") - ); - assert_eq!( - command, - serde_json::json!([ - format!("{SUPERVISOR_MOUNT_PATH}/openshell-sandbox"), - "--workdir", - driver_mounts::DEFAULT_WORKSPACE_ROOT - ]) - .as_array() - .unwrap() - ); - - // Agent volume mount should be read-only - let mounts = pod_template["spec"]["containers"][0]["volumeMounts"] - .as_array() - .expect("volumeMounts should exist"); - assert_eq!(mounts.len(), 1); - assert_eq!(mounts[0]["name"], SUPERVISOR_VOLUME_NAME); - assert_eq!(mounts[0]["mountPath"], SUPERVISOR_MOUNT_PATH); - assert_eq!(mounts[0]["readOnly"], true); - } - - #[test] - fn supervisor_sideload_image_volume_injects_image_source_without_init_container() { - let mut pod_template = serde_json::json!({ - "spec": { - "containers": [{ - "name": "agent", - "image": "custom-image:latest" - }] - } - }); - - apply_supervisor_sideload( - &mut pod_template, - "supervisor-image:latest", - Some("IfNotPresent"), - SupervisorSideloadMethod::ImageVolume, - 1000, // sandbox_uid - 1000, // sandbox_gid - ); - - let volumes = pod_template["spec"]["volumes"] - .as_array() - .expect("volumes should exist"); - assert_eq!(volumes.len(), 1); - assert_eq!(volumes[0]["name"], SUPERVISOR_VOLUME_NAME); - assert_eq!(volumes[0]["image"]["reference"], "supervisor-image:latest"); - assert_eq!(volumes[0]["image"]["pullPolicy"], "IfNotPresent"); - assert!( - volumes[0]["emptyDir"].is_null(), - "image volume method must not use emptyDir" - ); - - assert!( - pod_template["spec"]["initContainers"].is_null(), - "image volume method must not inject init containers" - ); - - let command = pod_template["spec"]["containers"][0]["command"] - .as_array() - .expect("command should be set"); - assert_eq!( - command[0].as_str().unwrap(), - format!("{SUPERVISOR_MOUNT_PATH}/openshell-sandbox") - ); - - let sc = &pod_template["spec"]["containers"][0]["securityContext"]; - assert_eq!(sc["runAsUser"], 0); - - let mounts = pod_template["spec"]["containers"][0]["volumeMounts"] - .as_array() - .expect("volumeMounts should exist"); - assert_eq!(mounts[0]["name"], SUPERVISOR_VOLUME_NAME); - assert_eq!(mounts[0]["mountPath"], SUPERVISOR_MOUNT_PATH); - assert_eq!(mounts[0]["readOnly"], true); - } - - #[test] - fn supervisor_image_volume_omits_pull_policy_when_unspecified() { - let mut pod_template = serde_json::json!({ - "spec": { - "containers": [{ - "name": "agent", - "image": "custom-image:latest" - }] - } - }); - - apply_supervisor_sideload( - &mut pod_template, - "supervisor-image:latest", - None, - SupervisorSideloadMethod::ImageVolume, - 1000, // sandbox_uid - 1000, // sandbox_gid + metadata.get(PROGRESS_COMPLETE_STEP_KEY).map(String::as_str), + Some(PROGRESS_STEP_PULLING_IMAGE) ); - - let volume = &pod_template["spec"]["volumes"][0]; - assert_eq!(volume["image"]["reference"], "supervisor-image:latest"); - assert!( - volume["image"].get("pullPolicy").is_none(), - "pullPolicy should be omitted when unspecified" + assert_eq!( + metadata + .get(PROGRESS_COMPLETE_LABEL_KEY) + .map(String::as_str), + Some("Image pulled (42 MB)") + ); + assert_eq!( + metadata.get(PROGRESS_ACTIVE_STEP_KEY).map(String::as_str), + Some(PROGRESS_STEP_STARTING_SANDBOX) ); } #[test] - fn sidecar_topology_renders_process_agent_and_network_sidecar() { + fn sandbox_runtime_renders_credential_free_boundary_workload() { let params = SandboxPodParams { - topology: SupervisorTopology::Sidecar, - supervisor_sideload_method: SupervisorSideloadMethod::InitContainer, - supervisor_image: "supervisor-image:latest", - supervisor_image_pull_policy: Some("IfNotPresent"), - grpc_endpoint: "https://openshell-gateway.openshell.svc:8080", - client_tls_secret_name: "openshell-client-tls", - proxy_uid: 2200, + default_image: "agent:latest", + image_pull_policy: Some(crate::KubernetesImagePullPolicy::Always), + sandbox_runtime_image: "sandbox-runtime-image:latest", + sandbox_runtime_image_pull_policy: Some(crate::KubernetesImagePullPolicy::Never), + sandbox_id: "sandbox-123", sandbox_uid: 1500, sandbox_gid: 1500, ..SandboxPodParams::default() }; let pod_template = sandbox_template_to_k8s( - &SandboxTemplate { - image: "agent-image:latest".to_string(), - environment: std::collections::HashMap::from([ - ( - openshell_core::sandbox_env::OCI_IMAGE_USER.to_string(), - "spoofed".to_string(), - ), - ( - openshell_core::sandbox_env::SANDBOX_UID.to_string(), - "9999".to_string(), - ), - ( - openshell_core::sandbox_env::SANDBOX_GID.to_string(), - "9999".to_string(), - ), - ]), - ..SandboxTemplate::default() - }, + &SandboxTemplate::default(), false, &std::collections::HashMap::new(), - false, + true, ¶ms, ); + let agent = &pod_template["spec"]["containers"][0]; + assert_eq!(agent["imagePullPolicy"], "Always"); - assert_eq!(pod_template["spec"]["shareProcessNamespace"], true); - assert_eq!(pod_template["spec"]["securityContext"]["fsGroup"], 1500); - let containers = pod_template["spec"]["containers"].as_array().unwrap(); - assert_eq!(containers.len(), 2); - - let agent = containers - .iter() - .find(|container| container["name"] == "agent") - .unwrap(); assert_eq!( agent["command"], serde_json::json!([ - format!("{SUPERVISOR_MOUNT_PATH}/openshell-sandbox"), - "--mode=process", - "--workdir", - driver_mounts::DEFAULT_WORKSPACE_ROOT + format!("{SANDBOX_RUNTIME_MOUNT_PATH}/openshell-sandbox"), + "--bootstrap", + BOUNDARY_CONFIG_PATH ]) ); assert_eq!(agent["securityContext"]["runAsUser"], 1500); assert_eq!(agent["securityContext"]["runAsGroup"], 1500); assert_eq!(agent["securityContext"]["runAsNonRoot"], true); - assert_eq!(agent["securityContext"]["allowPrivilegeEscalation"], false); assert_eq!( agent["securityContext"]["capabilities"], - serde_json::json!({ - "drop": ["ALL"] - }) - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::ENDPOINT), - None - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME), - None - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::TLS_CA), - None - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::K8S_SA_TOKEN_FILE), - None - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::SSH_SOCKET_PATH), - Some(SIDECAR_SSH_SOCKET_FILE) - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::SIDECAR_CONTROL_SOCKET), - Some(SIDECAR_CONTROL_SOCKET) - ); - assert_eq!(rendered_env(agent, "OPENSHELL_SUPERVISOR_READY_FILE"), None); - assert_eq!(rendered_env(agent, "OPENSHELL_ENTRYPOINT_PID_FILE"), None); - assert_eq!( - rendered_env(agent, "OPENSHELL_SIDECAR_POLICY_SNAPSHOT_FILE"), - None - ); - assert_eq!( - rendered_env(agent, "OPENSHELL_SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE"), - None - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::PROXY_TLS_DIR), - Some(SIDECAR_TLS_MOUNT_PATH) - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::SANDBOX_UID), - Some("1500") - ); - assert_eq!( - rendered_env(agent, openshell_core::sandbox_env::OCI_IMAGE_USER), - Some("") - ); - - let sidecar = containers - .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) - .unwrap(); - assert_eq!(sidecar["image"], "supervisor-image:latest"); - assert_eq!(sidecar["imagePullPolicy"], "IfNotPresent"); - assert_eq!( - sidecar["command"], - serde_json::json!([SUPERVISOR_IMAGE_BINARY_PATH, "--mode=network"]) - ); - assert_eq!(sidecar["securityContext"]["runAsUser"], 0); - assert_eq!(sidecar["securityContext"]["runAsGroup"], 1500); - assert_eq!(sidecar["securityContext"]["runAsNonRoot"], false); - assert_eq!( - sidecar["securityContext"]["allowPrivilegeEscalation"], - false - ); - assert_eq!( - sidecar["securityContext"]["capabilities"], - serde_json::json!({ - "drop": ["ALL"], - "add": ["SYS_PTRACE", "DAC_READ_SEARCH"] - }) - ); - assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::ENDPOINT), - Some("https://openshell-gateway.openshell.svc:8080") - ); - assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::SSH_SOCKET_PATH), - Some(SIDECAR_SSH_SOCKET_FILE) - ); - assert!( - SIDECAR_SSH_SOCKET_FILE.starts_with('@'), - "sidecar SSH relay must use a Linux abstract socket" - ); - assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::SANDBOX_UID), - Some("1500") - ); - assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::SANDBOX_GID), - Some("1500") - ); - assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::OCI_IMAGE_USER), - Some("") - ); - assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::SIDECAR_CONTROL_SOCKET), - Some(SIDECAR_CONTROL_SOCKET) - ); - assert_eq!( - rendered_env(sidecar, "OPENSHELL_SIDECAR_POLICY_SNAPSHOT_FILE"), - None - ); - assert_eq!( - rendered_env(sidecar, "OPENSHELL_SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE"), - None + serde_json::json!({"drop": ["ALL"]}) ); + assert!(agent["securityContext"]["capabilities"]["add"].is_null()); + assert_eq!(pod_template["spec"]["securityContext"]["fsGroup"], 1500); assert_eq!( - rendered_env( - sidecar, - openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY - ), - None + pod_template["spec"]["securityContext"]["seccompProfile"]["type"], + "RuntimeDefault" ); - assert_eq!(rendered_env(sidecar, "OPENSHELL_ENTRYPOINT_PID_FILE"), None); + assert_eq!(pod_template["spec"]["dnsPolicy"], "None"); assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::PROXY_TLS_DIR), - Some(SIDECAR_TLS_MOUNT_PATH) + pod_template["spec"]["dnsConfig"]["nameservers"], + serde_json::json!(["127.0.0.53"]) ); assert_eq!( - rendered_env(sidecar, openshell_core::sandbox_env::TLS_CA), - Some("/etc/openshell-tls/proxy/client/ca.crt") + pod_template["spec"]["schedulingGates"], + serde_json::json!([{"name": SANDBOX_BOOTSTRAP_SCHEDULING_GATE}]) ); - let sidecar_mounts = sidecar["volumeMounts"].as_array().unwrap(); - assert!( - !sidecar_mounts - .iter() - .any(|mount| mount["name"] == "openshell-client-tls"), - "runtime sidecar should use the init-copied TLS files, not the root-owned Secret mount" - ); - let agent_mounts = agent["volumeMounts"].as_array().unwrap(); - assert!( - !agent_mounts - .iter() - .any(|mount| mount["name"] == "openshell-sa-token"), - "agent container must not mount gateway bootstrap token in sidecar topology" - ); - assert!( - !agent_mounts - .iter() - .any(|mount| mount["name"] == "openshell-client-tls"), - "agent container must not mount gateway client TLS secret in sidecar topology" - ); - let volumes = pod_template["spec"]["volumes"].as_array().unwrap(); - let sa_token = volumes - .iter() - .find(|volume| volume["name"] == "openshell-sa-token") - .unwrap(); - assert_eq!(sa_token["projected"]["defaultMode"], 0o440); - let client_tls = volumes + let sandbox_bootstrap = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() .iter() - .find(|volume| volume["name"] == "openshell-client-tls") + .find(|container| container["name"] == "openshell-sandbox-bootstrap") .unwrap(); - assert_eq!(client_tls["secret"]["defaultMode"], 0o440); - - let init_containers = pod_template["spec"]["initContainers"].as_array().unwrap(); - let network_init = init_containers + assert_eq!(sandbox_bootstrap["image"], "sandbox-runtime-image:latest"); + assert_eq!(sandbox_bootstrap["imagePullPolicy"], "Never"); + let workspace_init = pod_template["spec"]["initContainers"] + .as_array() + .unwrap() .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_INIT_CONTAINER_NAME) + .find(|container| container["name"] == WORKSPACE_INIT_CONTAINER_NAME) .unwrap(); - assert_eq!(network_init["image"], "supervisor-image:latest"); - assert_eq!(network_init["imagePullPolicy"], "IfNotPresent"); assert_eq!( - network_init["command"], + workspace_init["command"], serde_json::json!([ - SUPERVISOR_IMAGE_BINARY_PATH, - "--mode=network-init", - "--proxy-uid", - "0", - "--proxy-gid", - "1500", - "--sidecar-state-dir", - SIDECAR_STATE_MOUNT_PATH, - "--sidecar-tls-dir", - SIDECAR_TLS_MOUNT_PATH + format!("{SANDBOX_RUNTIME_MOUNT_PATH}/openshell-sandbox"), + "seed-workspace" ]) ); + assert_eq!(workspace_init["securityContext"]["runAsUser"], 1500); + assert_eq!(workspace_init["imagePullPolicy"], "Always"); assert_eq!( - network_init["securityContext"]["capabilities"], - serde_json::json!({ - "drop": ["ALL"], - "add": ["NET_ADMIN", "NET_RAW", "CHOWN", "FOWNER"] - }) - ); - let network_init_mounts = network_init["volumeMounts"].as_array().unwrap(); - assert!(network_init_mounts.iter().any(|mount| { - mount["name"] == "openshell-client-tls" - && mount["mountPath"] == "/etc/openshell-tls/client" - })); - } - - #[test] - fn sidecar_topology_can_relax_process_binary_aware_network_policy() { - let params = SandboxPodParams { - topology: SupervisorTopology::Sidecar, - supervisor_sideload_method: SupervisorSideloadMethod::InitContainer, - supervisor_image: "supervisor-image:latest", - proxy_uid: 2200, - sandbox_uid: 1500, - sandbox_gid: 1500, - process_binary_aware_network_policy: false, - ..SandboxPodParams::default() - }; - let pod_template = sandbox_template_to_k8s( - &SandboxTemplate { - image: "agent-image:latest".to_string(), - ..SandboxTemplate::default() - }, - false, - &std::collections::HashMap::new(), - false, - ¶ms, + workspace_init["securityContext"]["capabilities"], + serde_json::json!({"drop": ["ALL"]}) ); - - let containers = pod_template["spec"]["containers"].as_array().unwrap(); - let sidecar = containers - .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) - .unwrap(); - assert_eq!(sidecar["securityContext"]["runAsUser"], 2200); - assert_eq!(sidecar["securityContext"]["runAsGroup"], 1500); - assert_eq!(sidecar["securityContext"]["runAsNonRoot"], true); assert_eq!( - sidecar["securityContext"]["allowPrivilegeEscalation"], - false + rendered_env(agent, openshell_core::sandbox_env::ENDPOINT), + None ); assert_eq!( - sidecar["securityContext"]["capabilities"], - serde_json::json!({ - "drop": ["ALL"] - }) + rendered_env(agent, openshell_core::sandbox_env::K8S_SA_TOKEN_FILE), + None ); assert_eq!( - rendered_env( - sidecar, - openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY - ), - Some("relaxed") + pod_template["metadata"]["labels"][BOUNDARY_ROLE_LABEL], + "workload" ); - let init_containers = pod_template["spec"]["initContainers"].as_array().unwrap(); - let network_init = init_containers - .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_INIT_CONTAINER_NAME) - .unwrap(); - assert_eq!(network_init["command"][3], "2200"); - } - #[test] - fn sidecar_topology_adds_shared_state_and_tls_volumes() { - let params = SandboxPodParams { - topology: SupervisorTopology::Sidecar, - supervisor_sideload_method: SupervisorSideloadMethod::ImageVolume, - supervisor_image: "supervisor-image:latest", - grpc_endpoint: "http://openshell-gateway.openshell.svc:8080", - ..SandboxPodParams::default() - }; - let pod_template = sandbox_template_to_k8s( - &SandboxTemplate::default(), - false, - &std::collections::HashMap::new(), - false, - ¶ms, + let mounts = agent["volumeMounts"].as_array().unwrap(); + assert!(mounts.iter().any(|mount| { + mount["name"] == SANDBOX_RUNTIME_VOLUME_NAME && mount["readOnly"] == true + })); + assert!(mounts.iter().any(|mount| { + mount["name"] == SANDBOX_STATE_VOLUME_NAME && mount["readOnly"].is_null() + })); + assert!(mounts.iter().any(|mount| { + mount["name"] == SANDBOX_POD_IDENTITY_VOLUME_NAME + && mount["mountPath"] == SANDBOX_POD_IDENTITY_MOUNT_PATH + && mount["readOnly"] == true + })); + assert!(mounts.iter().any(|mount| { + mount["name"] == SANDBOX_PROXY_CA_VOLUME_NAME + && mount["mountPath"] == SANDBOX_PROXY_CA_MOUNT_PATH + && mount["readOnly"].is_null() + })); + assert!( + !mounts + .iter() + .any(|mount| mount["name"] == SANDBOX_BOOTSTRAP_VOLUME_NAME) ); - - let volumes = pod_template["spec"]["volumes"].as_array().unwrap(); assert!( - volumes + !mounts .iter() - .any(|volume| volume["name"] == SIDECAR_STATE_VOLUME_NAME) + .any(|mount| mount["name"] == CLIENT_TLS_VOLUME_NAME) ); assert!( - volumes + !mounts .iter() - .any(|volume| volume["name"] == SIDECAR_TLS_VOLUME_NAME) + .any(|mount| mount["name"] == SERVICE_ACCOUNT_TOKEN_VOLUME_NAME) ); - assert!(volumes.iter().any(|volume| { - volume["name"] == SUPERVISOR_VOLUME_NAME && volume["image"].is_object() - })); - let containers = pod_template["spec"]["containers"].as_array().unwrap(); - let sidecar = containers + let volumes = pod_template["spec"]["volumes"].as_array().unwrap(); + let pod_identity = volumes .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) + .find(|volume| volume["name"] == SANDBOX_POD_IDENTITY_VOLUME_NAME) .unwrap(); assert_eq!( - sidecar["securityContext"]["capabilities"], - serde_json::json!({ - "drop": ["ALL"], - "add": ["SYS_PTRACE", "DAC_READ_SEARCH"] - }) - ); - assert_eq!(sidecar["securityContext"]["runAsUser"], 0); - assert_eq!(sidecar["securityContext"]["runAsGroup"], 1000); - assert_eq!(sidecar["securityContext"]["runAsNonRoot"], false); - assert_eq!( - sidecar["securityContext"]["allowPrivilegeEscalation"], - false + pod_identity["downwardAPI"]["items"], + serde_json::json!([{"path": "uid", "fieldRef": {"fieldPath": "metadata.uid"}}]) ); - - for container_name in ["agent", SUPERVISOR_NETWORK_SIDECAR_NAME] { - let container = containers - .iter() - .find(|container| container["name"] == container_name) - .unwrap(); - let mounts = container["volumeMounts"].as_array().unwrap(); - assert!(mounts.iter().any(|mount| { - mount["name"] == SIDECAR_STATE_VOLUME_NAME - && mount["mountPath"] == SIDECAR_STATE_MOUNT_PATH - })); - assert!(mounts.iter().any(|mount| { - mount["name"] == SIDECAR_TLS_VOLUME_NAME - && mount["mountPath"] == SIDECAR_TLS_MOUNT_PATH - })); - } - let init_containers = pod_template["spec"]["initContainers"].as_array().unwrap(); - let network_init = init_containers - .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_INIT_CONTAINER_NAME) - .unwrap(); - assert_eq!(network_init["command"][3], "0"); - } - - #[test] - fn sidecar_topology_rejects_proxy_uid_matching_sandbox_uid() { - let params = SandboxPodParams { - topology: SupervisorTopology::Sidecar, - proxy_uid: 1500, - sandbox_uid: 1500, - ..SandboxPodParams::default() - }; - - let err = validate_sidecar_proxy_identity(¶ms).unwrap_err(); - assert!(matches!(err, KubernetesDriverError::Precondition(_))); - assert!(err.to_string().contains("proxy_uid")); } /// Regression test: TLS mount path must match env var paths. /// The volume is mounted at a specific path and the env vars must point to /// files within that same path, otherwise the sandbox will fail to start /// with "No such file or directory" errors. - #[test] - fn tls_env_vars_match_volume_mount_path() { - // The mount path used in pod template construction - const TLS_MOUNT_PATH: &str = "/etc/openshell-tls/client"; - - // Build env with TLS enabled - let mut env = Vec::new(); - apply_required_env( - &mut env, - "sandbox-1", - "my-sandbox", - "https://endpoint:8080", - "0.0.0.0:2222", - true, // tls_enabled - None, - ); - - // Extract the TLS-related env vars - let get_env = |name: &str| -> Option { - env.iter() - .find(|e| e.get("name").and_then(|v| v.as_str()) == Some(name)) - .and_then(|e| e.get("value").and_then(|v| v.as_str()).map(String::from)) - }; - - let tls_ca = get_env("OPENSHELL_TLS_CA").expect("OPENSHELL_TLS_CA must be set"); - let tls_cert = get_env("OPENSHELL_TLS_CERT").expect("OPENSHELL_TLS_CERT must be set"); - let tls_key = get_env("OPENSHELL_TLS_KEY").expect("OPENSHELL_TLS_KEY must be set"); - - // All TLS paths must be within the mount path - assert!( - tls_ca.starts_with(TLS_MOUNT_PATH), - "OPENSHELL_TLS_CA path '{tls_ca}' must start with mount path '{TLS_MOUNT_PATH}'" - ); - assert!( - tls_cert.starts_with(TLS_MOUNT_PATH), - "OPENSHELL_TLS_CERT path '{tls_cert}' must start with mount path '{TLS_MOUNT_PATH}'" - ); - assert!( - tls_key.starts_with(TLS_MOUNT_PATH), - "OPENSHELL_TLS_KEY path '{tls_key}' must start with mount path '{TLS_MOUNT_PATH}'" - ); - } #[test] fn gpu_sandbox_adds_runtime_class_and_gpu_limit() { @@ -7302,87 +8707,9 @@ mod tests { let resources = &pod_template["spec"]["containers"][0]["resources"]; assert_eq!(resources["limits"]["cpu"], serde_json::json!("500m")); - assert_eq!(resources["limits"]["memory"], serde_json::json!("2Gi")); - assert_eq!(resources["requests"]["cpu"], serde_json::json!("500m")); - assert_eq!(resources["requests"]["memory"], serde_json::json!("2Gi")); - } - - #[test] - fn host_aliases_injected_when_gateway_ip_set() { - let pod_template = { - let params = SandboxPodParams { - host_gateway_ip: "172.17.0.1", - ..Default::default() - }; - sandbox_template_to_k8s( - &SandboxTemplate::default(), - false, - &std::collections::HashMap::new(), - true, - ¶ms, - ) - }; - - let host_aliases = pod_template["spec"]["hostAliases"] - .as_array() - .expect("hostAliases should exist"); - assert_eq!(host_aliases.len(), 1); - assert_eq!(host_aliases[0]["ip"], "172.17.0.1"); - let hostnames = host_aliases[0]["hostnames"] - .as_array() - .expect("hostnames should exist"); - assert!(hostnames.contains(&serde_json::json!("host.docker.internal"))); - assert!(hostnames.contains(&serde_json::json!("host.openshell.internal"))); - } - - #[test] - fn host_aliases_not_injected_when_gateway_ip_empty() { - let pod_template = { - let params = SandboxPodParams::default(); - sandbox_template_to_k8s( - &SandboxTemplate::default(), - false, - &std::collections::HashMap::new(), - true, - ¶ms, - ) - }; - - assert!( - pod_template["spec"]["hostAliases"].is_null(), - "hostAliases should not be present when host_gateway_ip is empty" - ); - } - - #[test] - fn tls_secret_volume_uses_restrictive_default_mode() { - let template = SandboxTemplate::default(); - let pod_template = { - let params = SandboxPodParams { - client_tls_secret_name: "my-tls-secret", - ..Default::default() - }; - sandbox_template_to_k8s( - &template, - false, - &std::collections::HashMap::new(), - true, - ¶ms, - ) - }; - - let volumes = pod_template["spec"]["volumes"] - .as_array() - .expect("volumes should exist"); - let tls_vol = volumes - .iter() - .find(|v| v["name"] == CLIENT_TLS_VOLUME_NAME) - .expect("TLS volume should exist"); - assert_eq!( - tls_vol["secret"]["defaultMode"], - 256, // 0o400 - "TLS secret volume must use mode 0400 to prevent sandbox user from reading the private key" - ); + assert_eq!(resources["limits"]["memory"], serde_json::json!("2Gi")); + assert_eq!(resources["requests"]["cpu"], serde_json::json!("500m")); + assert_eq!(resources["requests"]["memory"], serde_json::json!("2Gi")); } // ----------------------------------------------------------------------- @@ -7403,8 +8730,9 @@ mod tests { apply_workspace_persistence( &mut pod_template, "openshell/sandbox:latest", - Some("IfNotPresent"), - 1000, // sandbox_gid + Some(crate::KubernetesImagePullPolicy::IfNotPresent), + Some(1000), // sandbox_gid + None, ); // Init container @@ -7462,8 +8790,9 @@ mod tests { apply_workspace_persistence( &mut pod_template, "my-custom-image:v2", - Some("IfNotPresent"), - 1000, + Some(crate::KubernetesImagePullPolicy::IfNotPresent), + Some(1000), + None, ); let init_image = pod_template["spec"]["initContainers"][0]["image"] @@ -7486,7 +8815,13 @@ mod tests { } }); - apply_workspace_persistence(&mut pod_template, "img:latest", Some("Always"), 1000); + apply_workspace_persistence( + &mut pod_template, + "img:latest", + Some(crate::KubernetesImagePullPolicy::Always), + Some(1000), + None, + ); let cmd = pod_template["spec"]["initContainers"][0]["command"] .as_array() @@ -7514,10 +8849,7 @@ mod tests { #[test] fn workspace_persistence_skipped_when_inject_workspace_false() { - let params = SandboxPodParams { - supervisor_sideload_method: SupervisorSideloadMethod::InitContainer, - ..SandboxPodParams::default() - }; + let params = SandboxPodParams::default(); let pod_template = sandbox_template_to_k8s( &SandboxTemplate::default(), false, @@ -7565,65 +8897,6 @@ mod tests { ) } - #[test] - fn app_armor_profile_omitted_by_default() { - let pod_template = default_template_to_k8s(false); - assert!( - pod_template["spec"]["containers"][0]["securityContext"]["appArmorProfile"].is_null(), - "appArmorProfile must be omitted when no profile is configured" - ); - } - - #[test] - fn app_armor_profile_renders_unconfined() { - let profile = AppArmorProfile::Unconfined; - let params = SandboxPodParams { - app_armor_profile: Some(&profile), - ..Default::default() - }; - let pod_template = sandbox_template_to_k8s( - &SandboxTemplate::default(), - false, - &std::collections::HashMap::new(), - true, - ¶ms, - ); - - assert_eq!( - pod_template["spec"]["containers"][0]["securityContext"]["appArmorProfile"], - serde_json::json!({ "type": "Unconfined" }) - ); - assert_eq!( - pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["add"][0], - serde_json::json!("SYS_ADMIN"), - "AppArmor rendering must preserve required capabilities" - ); - } - - #[test] - fn app_armor_profile_renders_localhost_profile() { - let profile = AppArmorProfile::Localhost("openshell-supervisor".to_string()); - let params = SandboxPodParams { - app_armor_profile: Some(&profile), - ..Default::default() - }; - let pod_template = sandbox_template_to_k8s( - &SandboxTemplate::default(), - false, - &std::collections::HashMap::new(), - true, - ¶ms, - ); - - assert_eq!( - pod_template["spec"]["containers"][0]["securityContext"]["appArmorProfile"], - serde_json::json!({ - "type": "Localhost", - "localhostProfile": "openshell-supervisor" - }) - ); - } - #[test] fn user_namespaces_disabled_by_default() { let pod_template = default_template_to_k8s(false); @@ -7631,11 +8904,10 @@ mod tests { pod_template["spec"]["hostUsers"].is_null(), "hostUsers must not be set when user namespaces are disabled" ); - let caps = pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["add"] - .as_array() - .unwrap(); - assert_eq!(caps.len(), 4); - assert!(!caps.contains(&serde_json::json!("SETUID"))); + let capabilities = + &pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]; + assert!(capabilities["add"].is_null()); + assert_eq!(capabilities["drop"], serde_json::json!(["ALL"])); } #[test] @@ -7649,19 +8921,12 @@ mod tests { } #[test] - fn user_namespaces_adds_extra_capabilities() { + fn user_namespaces_preserve_capability_free_posture() { let pod_template = default_template_to_k8s(true); - let caps = pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["add"] - .as_array() - .unwrap(); - assert!(caps.contains(&serde_json::json!("SYS_ADMIN"))); - assert!(caps.contains(&serde_json::json!("NET_ADMIN"))); - assert!(caps.contains(&serde_json::json!("SYS_PTRACE"))); - assert!(caps.contains(&serde_json::json!("SYSLOG"))); - assert!(caps.contains(&serde_json::json!("SETUID"))); - assert!(caps.contains(&serde_json::json!("SETGID"))); - assert!(caps.contains(&serde_json::json!("DAC_READ_SEARCH"))); - assert_eq!(caps.len(), 7); + let capabilities = + &pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]; + assert!(capabilities["add"].is_null()); + assert_eq!(capabilities["drop"], serde_json::json!(["ALL"])); } #[test] @@ -7685,10 +8950,10 @@ mod tests { serde_json::json!(false), "per-sandbox user namespace intent must set hostUsers: false" ); - let caps = pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["add"] - .as_array() - .unwrap(); - assert!(caps.contains(&serde_json::json!("SETUID"))); + let capabilities = + &pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]; + assert!(capabilities["add"].is_null()); + assert_eq!(capabilities["drop"], serde_json::json!(["ALL"])); } #[test] @@ -7714,14 +8979,10 @@ mod tests { pod_template["spec"]["hostUsers"].is_null(), "per-sandbox user namespace intent must override the cluster default" ); - let caps = pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["add"] - .as_array() - .unwrap(); - assert_eq!( - caps.len(), - 4, - "extra capabilities must not be added when user namespaces are disabled" - ); + let capabilities = + &pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]; + assert!(capabilities["add"].is_null()); + assert_eq!(capabilities["drop"], serde_json::json!(["ALL"])); } #[test] @@ -7893,51 +9154,6 @@ mod tests { ); } - #[test] - fn provider_spiffe_mounts_csi_socket_and_keeps_sa_token_bootstrap() { - let params = SandboxPodParams { - sandbox_id: "sandbox-123", - sandbox_name: "sandbox", - provider_spiffe_enabled: true, - provider_spiffe_workload_api_socket_path: "/spiffe-workload-api/spire-agent.sock", - ..SandboxPodParams::default() - }; - let pod_template = sandbox_template_to_k8s( - &SandboxTemplate::default(), - false, - &std::collections::HashMap::new(), - true, - ¶ms, - ); - - let env = pod_template["spec"]["containers"][0]["env"] - .as_array() - .expect("env"); - assert!(env.iter().any(|e| { - e["name"] == openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET - && e["value"] == "/spiffe-workload-api/spire-agent.sock" - })); - assert!(env.iter().any(|e| { - e["name"] == openshell_core::sandbox_env::K8S_SA_TOKEN_FILE - && e["value"] == "/var/run/secrets/openshell/token" - })); - - let volumes = pod_template["spec"]["volumes"].as_array().expect("volumes"); - assert!(volumes.iter().any(|volume| { - volume["name"] == SPIFFE_WORKLOAD_API_VOLUME_NAME - && volume["csi"]["driver"] == "csi.spiffe.io" - })); - assert!(volumes.iter().any(|volume| { - volume["name"] == SERVICE_ACCOUNT_TOKEN_VOLUME_NAME - && volume["projected"]["sources"][0]["serviceAccountToken"]["path"] == "token" - })); - - assert_eq!( - pod_template["metadata"]["labels"][LABEL_MANAGED_BY], - serde_json::json!(LABEL_MANAGED_BY_VALUE) - ); - } - #[test] fn log_level_propagates_as_env_var_to_sandbox_pod() { let spec = SandboxSpec { @@ -7987,7 +9203,7 @@ mod tests { } #[test] - fn sandbox_pod_clears_unsupported_network_capabilities() { + fn sandbox_pod_drops_legacy_network_capability_environment() { let spec = SandboxSpec { environment: std::collections::HashMap::from([( openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES.to_string(), @@ -7999,14 +9215,9 @@ mod tests { let env = cr["spec"]["podTemplate"]["spec"]["containers"][0]["env"] .as_array() .unwrap(); - let entries = env - .iter() - .filter(|entry| { - entry["name"] == openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES - }) - .collect::>(); - assert_eq!(entries.len(), 1); - assert_eq!(entries[0]["value"], serde_json::json!("")); + assert!(!env.iter().any(|entry| { + entry["name"] == openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES + })); } #[test] @@ -8455,103 +9666,6 @@ mod tests { ); } - #[test] - fn upstream_proxy_is_injected_only_into_network_supervisors() { - let params = SandboxPodParams { - topology: SupervisorTopology::Sidecar, - supervisor_sideload_method: SupervisorSideloadMethod::InitContainer, - supervisor_image: "supervisor-image:latest", - https_proxy: Some("http://proxy.corp.example:8080"), - no_proxy: Some(".svc.cluster.local,10.96.0.0/12"), - proxy_auth_secret_name: Some("corporate-proxy-auth"), - proxy_auth_secret_key: Some("credentials"), - proxy_auth_allow_insecure: true, - proxy_connect_by_hostname: true, - sandbox_uid: 1500, - sandbox_gid: 1500, - ..SandboxPodParams::default() - }; - let pod = sandbox_template_to_k8s( - &SandboxTemplate::default(), - false, - &std::collections::HashMap::new(), - false, - ¶ms, - ); - let containers = pod["spec"]["containers"].as_array().unwrap(); - let network = containers - .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_SIDECAR_NAME) - .unwrap(); - let command = network["command"].as_array().unwrap(); - assert!(command.iter().any(|arg| arg == "--upstream-proxy")); - assert!(command.iter().any(|arg| arg == "--upstream-no-proxy")); - let auth_file_index = command - .iter() - .position(|arg| arg == "--upstream-proxy-auth-file") - .unwrap(); - assert_eq!( - command[auth_file_index + 1], - openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH - ); - assert!( - command - .iter() - .any(|arg| arg == "--upstream-proxy-auth-allow-insecure") - ); - assert!( - command - .iter() - .any(|arg| arg == "--upstream-proxy-connect-by-hostname") - ); - assert!( - network["volumeMounts"] - .as_array() - .unwrap() - .iter() - .any(|mount| mount["name"] == UPSTREAM_PROXY_AUTH_VOLUME_NAME) - ); - - let init = pod["spec"]["initContainers"] - .as_array() - .unwrap() - .iter() - .find(|container| container["name"] == SUPERVISOR_NETWORK_INIT_CONTAINER_NAME) - .unwrap(); - assert!(!init["command"].as_array().unwrap().iter().any(|arg| { - arg.as_str() - .is_some_and(|arg| arg.starts_with("--upstream-")) - })); - let agent = containers - .iter() - .find(|container| container["name"] == "agent") - .unwrap(); - assert!( - !agent["volumeMounts"] - .as_array() - .unwrap() - .iter() - .any(|mount| mount["name"] == UPSTREAM_PROXY_AUTH_VOLUME_NAME) - ); - assert!(!agent["env"].as_array().unwrap().iter().any(|entry| { - entry["value"] == "corporate-proxy-auth" || entry["value"] == "credentials" - })); - - let volume = pod["spec"]["volumes"] - .as_array() - .unwrap() - .iter() - .find(|volume| volume["name"] == UPSTREAM_PROXY_AUTH_VOLUME_NAME) - .unwrap(); - assert_eq!(volume["secret"]["secretName"], "corporate-proxy-auth"); - assert_eq!(volume["secret"]["items"][0]["key"], "credentials"); - assert_eq!( - volume["secret"]["items"][0]["path"], - upstream_proxy_auth_file_name() - ); - assert_eq!(volume["secret"]["defaultMode"], 0o440); - } - #[test] fn sandbox_lookup_selector_always_includes_gateway_id() { let sel = sandbox_lookup_selector_for("sb-123", "gw-42"); @@ -8782,4 +9896,240 @@ mod tests { assert!(gpu.default_selection_supported); assert!(gpu.count_selection_supported); } + + #[test] + fn sandbox_runtime_control_availability_requires_a_ready_pod() { + let mut pod = Pod::default(); + assert_eq!( + sandbox_runtime_control_availability_from_pod(&pod), + SandboxRuntimeControlAvailability::Unavailable + ); + pod.status = Some( + serde_json::from_value(serde_json::json!({ + "conditions": [{"type": "Ready", "status": "True"}] + })) + .expect("valid Pod status"), + ); + assert_eq!( + sandbox_runtime_control_availability_from_pod(&pod), + SandboxRuntimeControlAvailability::Available + ); + } + + #[test] + fn sandbox_runtime_readiness_transitions_bump_the_watched_cr() { + let unavailable = sandbox_runtime_readiness_transition_patch("42", "unavailable"); + let ready = sandbox_runtime_readiness_transition_patch("42", "ready"); + + assert_eq!(unavailable["metadata"]["resourceVersion"], "42"); + assert_eq!(ready["metadata"]["resourceVersion"], "42"); + assert_eq!( + unavailable["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_READINESS], + "unavailable" + ); + assert_eq!( + ready["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_READINESS], + "ready" + ); + assert_ne!(unavailable, ready); + } + + #[test] + fn sandbox_runtime_bootstrap_completion_is_resource_version_guarded_and_publishes_ready() { + let patch = sandbox_runtime_bootstrap_completion_patch("42"); + + assert_eq!(patch["metadata"]["resourceVersion"], "42"); + assert_eq!( + patch["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING], + serde_json::Value::Null + ); + assert_eq!( + patch["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT], + serde_json::Value::Null + ); + assert_eq!( + patch["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION], + serde_json::Value::Null + ); + assert_eq!( + patch["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE], + serde_json::Value::Null + ); + assert_eq!( + patch["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_SESSION], + serde_json::Value::Null + ); + assert_eq!( + patch["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_READINESS], + "ready" + ); + } + + #[test] + fn sandbox_runtime_rollback_is_durable_until_cleanup_completes() { + let rollback = sandbox_runtime_rollback_patch(SANDBOX_VERSION_V1BETA1, "42"); + assert_eq!(rollback["metadata"]["resourceVersion"], "42"); + assert_eq!( + rollback["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING], + "true" + ); + assert_eq!( + rollback["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION], + "stop" + ); + assert_eq!( + rollback["metadata"]["annotations"][ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE], + SandboxRuntimeBootstrapPhase::RollingBack.as_str() + ); + assert_eq!(rollback["spec"]["operatingMode"], "Suspended"); + + let complete = sandbox_runtime_rollback_completion_patch("43"); + assert_eq!(complete["metadata"]["resourceVersion"], "43"); + for annotation in [ + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAPPING, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_STARTED_AT, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_OPERATION, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_PHASE, + ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_SESSION, + ANNOTATION_SANDBOX_RUNTIME_GENERATION, + ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID, + ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_VERSION, + ] { + assert_eq!( + complete["metadata"]["annotations"][annotation], + serde_json::Value::Null + ); + } + } + + #[test] + fn sandbox_runtime_bootstrap_completion_waits_for_runtime_ready() { + let resource = ApiResource::from_gvk(&GroupVersionKind::gvk( + SANDBOX_GROUP, + SANDBOX_VERSION_V1BETA1, + SANDBOX_KIND, + )); + let mut sandbox = DynamicObject::new("sandbox", &resource); + sandbox.metadata.generation = Some(7); + sandbox.data = serde_json::json!({ + "status": { + "conditions": [{"type": "Suspended", "status": "True", "observedGeneration": 7}] + } + }); + assert!(!sandbox_runtime_runtime_is_ready(&sandbox)); + + sandbox.data["status"]["conditions"] = serde_json::json!([ + {"type": "Suspended", "status": "False", "observedGeneration": 6}, + {"type": "Ready", "status": "True", "observedGeneration": 6} + ]); + assert!(!sandbox_runtime_runtime_is_ready(&sandbox)); + + sandbox.data["status"]["conditions"] = serde_json::json!([ + {"type": "Suspended", "status": "False", "observedGeneration": 7}, + {"type": "Ready", "status": "True", "observedGeneration": 7} + ]); + assert!(sandbox_runtime_runtime_is_ready(&sandbox)); + + sandbox.data["status"]["conditions"] = serde_json::json!([ + {"type": "Suspended", "status": "True", "observedGeneration": 7}, + {"type": "Ready", "status": "True", "observedGeneration": 7} + ]); + assert!(!sandbox_runtime_runtime_is_ready(&sandbox)); + } + + #[test] + fn sandbox_runtime_readiness_is_downgraded_with_a_transient_reason() { + let mut sandbox = Sandbox { + status: Some(SandboxStatus { + conditions: vec![SandboxCondition { + r#type: "Ready".to_string(), + status: "True".to_string(), + ..Default::default() + }], + ..Default::default() + }), + ..Default::default() + }; + mark_sandbox_runtime_control_unavailable(&mut sandbox); + let ready = &sandbox.status.unwrap().conditions[0]; + assert_eq!(ready.status, "False"); + assert_eq!(ready.reason, "DependenciesNotReady"); + } + + #[test] + fn sandbox_runtime_bootstrap_does_not_publish_a_terminal_suspension() { + let mut sandbox = Sandbox { + status: Some(SandboxStatus { + conditions: vec![SandboxCondition { + r#type: SANDBOX_SUSPENDED_CONDITION.to_string(), + status: "True".to_string(), + reason: "PodTerminated".to_string(), + ..Default::default() + }], + ..Default::default() + }), + ..Default::default() + }; + + mark_sandbox_runtime_bootstrapping(&mut sandbox); + + let conditions = &sandbox.status.unwrap().conditions; + assert!( + !conditions + .iter() + .any(|condition| condition.r#type == SANDBOX_SUSPENDED_CONDITION) + ); + assert!(conditions.iter().any(|condition| { + condition.r#type == "Ready" + && condition.status == "False" + && condition.reason == "DependenciesNotReady" + })); + assert!(conditions.iter().any(|condition| { + condition.r#type == "Bootstrapping" + && condition.status == "True" + && condition.reason == "SandboxRuntimeGenerationStarting" + })); + } + + #[test] + fn completed_sandbox_runtime_bootstrap_preserves_real_suspension() { + let sandbox = Sandbox { + status: Some(SandboxStatus { + conditions: vec![SandboxCondition { + r#type: SANDBOX_SUSPENDED_CONDITION.to_string(), + status: "True".to_string(), + reason: "PodTerminated".to_string(), + ..Default::default() + }], + ..Default::default() + }), + ..Default::default() + }; + + assert!(sandbox.status.unwrap().conditions.iter().any(|condition| { + condition.r#type == SANDBOX_SUSPENDED_CONDITION && condition.status == "True" + })); + } + + #[test] + fn sandbox_runtime_should_run_tracks_both_sandbox_apis() { + let resource = ApiResource::from_gvk(&GroupVersionKind::gvk( + SANDBOX_GROUP, + SANDBOX_VERSION_V1BETA1, + SANDBOX_KIND, + )); + let mut beta = DynamicObject::new("beta", &resource); + beta.data = serde_json::json!({"spec": {"operatingMode": "Suspended"}}); + assert!(!sandbox_runtime_should_run(&beta)); + beta.data = serde_json::json!({"spec": {"operatingMode": "Running"}}); + assert!(sandbox_runtime_should_run(&beta)); + + let mut alpha = DynamicObject::new("alpha", &resource); + alpha.data = serde_json::json!({"spec": {"replicas": 0}}); + assert!(!sandbox_runtime_should_run(&alpha)); + alpha.data = serde_json::json!({"spec": {"replicas": 1}}); + assert!(sandbox_runtime_should_run(&alpha)); + } } diff --git a/crates/openshell-driver-kubernetes/src/grpc.rs b/crates/openshell-driver-kubernetes/src/grpc.rs index 095752d842..82d54ecea0 100644 --- a/crates/openshell-driver-kubernetes/src/grpc.rs +++ b/crates/openshell-driver-kubernetes/src/grpc.rs @@ -158,19 +158,21 @@ impl ComputeDriver for ComputeDriverService { &self, request: Request, ) -> Result, Status> { - self.rpc_tracer - .trace(openshell_otel::rpc::CREATE_SANDBOX, async { - let sandbox = request - .into_inner() - .sandbox - .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; - self.driver - .create_sandbox(&sandbox) - .await - .map_err(|e| Status::from(openshell_core::ComputeDriverError::from(e)))?; - Ok(Response::new(CreateSandboxResponse {})) - }) - .await + Box::pin( + self.rpc_tracer + .trace(openshell_otel::rpc::CREATE_SANDBOX, async { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + self.driver + .create_sandbox(&sandbox) + .await + .map_err(|e| Status::from(openshell_core::ComputeDriverError::from(e)))?; + Ok(Response::new(CreateSandboxResponse {})) + }), + ) + .await } async fn stop_sandbox( @@ -198,21 +200,26 @@ impl ComputeDriver for ComputeDriverService { &self, request: Request, ) -> Result, Status> { - self.rpc_tracer - .trace(openshell_otel::rpc::START_SANDBOX, async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - self.driver - .start_sandbox(&request.sandbox_id) + Box::pin( + self.rpc_tracer + .trace(openshell_otel::rpc::START_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + Box::pin(self.driver.start_sandbox( + &request.sandbox_id, + &request.generation_id, + &request.launch_authentication, + )) .await .map_err(|error| { Status::from(openshell_core::ComputeDriverError::from(error)) })?; - Ok(Response::new(StartSandboxResponse {})) - }) - .await + Ok(Response::new(StartSandboxResponse {})) + }), + ) + .await } async fn delete_sandbox( @@ -250,10 +257,12 @@ impl ComputeDriver for ComputeDriverService { let stream = stream.map(|item| item.map_err(|err| Status::internal(err.to_string()))); Ok::(Box::pin(stream)) }; - self.rpc_tracer - .trace_stream(openshell_otel::rpc::WATCH_SANDBOXES, create_stream) - .await - .map(Response::new) + Box::pin( + self.rpc_tracer + .trace_stream(openshell_otel::rpc::WATCH_SANDBOXES, create_stream), + ) + .await + .map(Response::new) } async fn ensure_workspace( diff --git a/crates/openshell-driver-kubernetes/src/isolation.rs b/crates/openshell-driver-kubernetes/src/isolation.rs new file mode 100644 index 0000000000..bb9ae8ff66 --- /dev/null +++ b/crates/openshell-driver-kubernetes/src/isolation.rs @@ -0,0 +1,447 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Kubernetes provisioning for the shared authenticated boundary protocol. +//! +//! This module deliberately contains no lifecycle, process, network, identity, +//! or wire implementation. The driver places the sandbox runtime, binds +//! immutable Kubernetes resource identities, and provisions TCP coordinates; +//! `openshell-isolation-interface` and `openshell-sandbox` provide the common +//! control and boundary behavior. + +use std::collections::{BTreeMap, HashMap}; +use std::net::{IpAddr, SocketAddr}; +use std::path::PathBuf; + +use k8s_openapi::api::networking::v1::{ + NetworkPolicy, NetworkPolicyEgressRule, NetworkPolicyIngressRule, NetworkPolicyPeer, + NetworkPolicyPort, NetworkPolicySpec, +}; +use k8s_openapi::apimachinery::pkg::apis::meta::v1::LabelSelector; +use k8s_openapi::apimachinery::pkg::util::intstr::IntOrString; +use kube::core::ObjectMeta; +use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity}; +use openshell_sandbox_backend::boundary_protocol::{ + BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor, + SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport, +}; + +/// Isolation backend implemented by the `OpenShell` sandbox runtime. +pub const BACKEND_NAME: &str = openshell_sandbox_backend::BACKEND_NAME; + +/// Label that binds the workload and supervisor pods in one unique pair. +pub const BOUNDARY_PAIR_LABEL: &str = "openshell.ai/boundary-pair"; + +/// Label distinguishing the two pods in a sandbox generation. +pub const BOUNDARY_ROLE_LABEL: &str = "openshell.ai/boundary-role"; + +const WORKLOAD_ROLE: &str = "workload"; +const SUPERVISOR_ROLE: &str = "supervisor"; + +/// Driver-owned inputs for the workload pod's Kubernetes network fence. +/// +/// This is the first phase of sandbox-runtime provisioning. The driver applies the +/// returned labels to the respective pods and creates the returned policy. It +/// then observes the policy UID and resourceVersion and supplies both to +/// `KubernetesSandboxRuntimeBoundarySpec`. +pub struct KubernetesSandboxRuntimeNetworkFenceSpec { + pub namespace: String, + pub policy_name: String, + pub supervisor_policy_name: String, + pub boundary_port: u16, +} + +/// Labels and policy needed to remove direct workload-pod egress. +pub struct KubernetesSandboxRuntimeNetworkFence { + pub workload_labels: BTreeMap, + pub control_labels: BTreeMap, + pub workload_policy: NetworkPolicy, + pub supervisor_policy: NetworkPolicy, +} + +impl KubernetesSandboxRuntimeNetworkFenceSpec { + /// Render the namespace-wide workload fence. + /// + /// Kubernetes `NetworkPolicy` is connection-aware: traffic returning over + /// the control-initiated boundary connection is allowed even though the + /// workload pod has no egress rules. The control pod remains responsible + /// for opening policy-approved upstream connections. + #[must_use] + pub fn provision(self) -> KubernetesSandboxRuntimeNetworkFence { + let workload_labels = role_labels(WORKLOAD_ROLE); + let control_labels = role_labels(SUPERVISOR_ROLE); + + let workload_policy = NetworkPolicy { + metadata: ObjectMeta { + name: Some(self.policy_name), + namespace: Some(self.namespace.clone()), + ..Default::default() + }, + spec: Some(NetworkPolicySpec { + pod_selector: LabelSelector { + match_labels: Some(workload_labels.clone()), + ..Default::default() + }, + policy_types: Some(vec!["Ingress".to_string(), "Egress".to_string()]), + // Any trusted OpenShell supervisor in this namespace may + // reach a sandbox listener. The Sandbox Protocol enforces the + // exact sandbox, generation, and Pod UID binding. + ingress: Some(vec![NetworkPolicyIngressRule { + from: Some(vec![NetworkPolicyPeer { + pod_selector: Some(LabelSelector { + match_labels: Some(control_labels.clone()), + ..Default::default() + }), + ..Default::default() + }]), + ports: Some(vec![NetworkPolicyPort { + port: Some(IntOrString::Int(i32::from(self.boundary_port))), + protocol: Some("TCP".to_string()), + ..Default::default() + }]), + }]), + // An explicit empty list selects the pod for egress and allows + // no new workload-initiated connections, including DNS and the + // Kubernetes API. Reply traffic for allowed ingress remains + // permitted by conforming NetworkPolicy implementations. + egress: Some(Vec::new()), + }), + }; + + // Namespace-wide default-deny policies are additive with this rule. + // Select only OpenShell supervisor pods and explicitly allow their + // policy-approved DNS and upstream connections. + let supervisor_policy = NetworkPolicy { + metadata: ObjectMeta { + name: Some(self.supervisor_policy_name), + namespace: Some(self.namespace), + ..Default::default() + }, + spec: Some(NetworkPolicySpec { + pod_selector: LabelSelector { + match_labels: Some(control_labels.clone()), + ..Default::default() + }, + policy_types: Some(vec!["Egress".to_string()]), + egress: Some(vec![NetworkPolicyEgressRule::default()]), + ..Default::default() + }), + }; + + KubernetesSandboxRuntimeNetworkFence { + workload_labels, + control_labels, + workload_policy, + supervisor_policy, + } + } +} + +fn role_labels(role: &str) -> BTreeMap { + BTreeMap::from([(BOUNDARY_ROLE_LABEL.to_string(), role.to_string())]) +} + +/// Driver-owned inputs that bind one workload/supervisor pair to one boundary. +/// +/// The driver constructs this only after Kubernetes has assigned every UID and +/// after it has observed the namespace workload-policy resource version. The +/// workload stays held until the matching boundary config and supervisor +/// resources have been installed. +pub struct KubernetesSandboxRuntimeBoundarySpec { + pub boundary_id: String, + pub generation: String, + pub session_id: openshell_core::SandboxSessionId, + pub session_rotation: openshell_core::jwt::SessionRotation, + pub auth_epoch: openshell_core::jwt::CredentialEpoch, + pub gateway_id: String, + pub verification_keys: Vec, + pub namespace_uid: String, + pub sandbox_resource_uid: String, + pub workload_pod_uid: String, + pub workload_pod_uid_path: PathBuf, + pub supervisor_pod_uid: String, + pub egress_policy_uid: String, + pub egress_policy_resource_version: String, + pub boundary_listener: SocketAddr, + pub control_authority: String, + pub control_address: SocketAddr, + pub sandbox_tls: SandboxTlsServerConfig, + pub supervisor_tls: SandboxTlsClientConfig, + pub host_gateway_ip: Option, + pub workload_identity: ResolvedWorkloadIdentity, + pub child_env: HashMap, +} + +/// Protected workload-pod config and matching sandbox-runtime descriptor. +pub struct KubernetesSandboxRuntimeBoundaryProvisioning { + pub boundary_config: BoundaryConfig, + pub runtime_descriptor: SandboxRuntimeDescriptor, +} + +impl KubernetesSandboxRuntimeBoundarySpec { + /// Produce both sides of the common protocol from one observed Kubernetes + /// resource set so a stale or recreated object cannot be attached. + #[must_use] + pub fn provision(self) -> KubernetesSandboxRuntimeBoundaryProvisioning { + let resource_claims = BTreeMap::from([ + ("kubernetes.namespace_uid".to_string(), self.namespace_uid), + ( + "kubernetes.sandbox_resource_uid".to_string(), + self.sandbox_resource_uid, + ), + ( + "kubernetes.workload_pod_uid".to_string(), + self.workload_pod_uid, + ), + ( + "kubernetes.supervisor_pod_uid".to_string(), + self.supervisor_pod_uid, + ), + ( + "kubernetes.egress_policy_uid".to_string(), + self.egress_policy_uid, + ), + ( + "kubernetes.egress_policy_resource_version".to_string(), + self.egress_policy_resource_version, + ), + ]); + let driver_fence = DriverFenceEvidence::Kubernetes { + network_policy_uid: resource_claims["kubernetes.egress_policy_uid"].clone(), + network_policy_resource_version: + resource_claims["kubernetes.egress_policy_resource_version"].clone(), + ingress_isolated: true, + egress_isolated: true, + egress_rule_count: 0, + }; + KubernetesSandboxRuntimeBoundaryProvisioning { + boundary_config: BoundaryConfig { + boundary_id: self.boundary_id.clone(), + generation: self.generation.clone(), + session_id: self.session_id, + session_rotation: self.session_rotation, + auth_epoch: self.auth_epoch, + gateway_id: self.gateway_id, + verification_keys: self.verification_keys, + listener: BoundaryListener::TlsTcp { + address: self.boundary_listener, + tls: self.sandbox_tls, + }, + resource_claims: resource_claims.clone(), + resource_claim_files: BTreeMap::from([( + "kubernetes.workload_pod_uid".to_string(), + self.workload_pod_uid_path, + )]), + workload_identity: self.workload_identity.clone(), + driver_fence: driver_fence.clone(), + child_env: self.child_env, + }, + runtime_descriptor: SandboxRuntimeDescriptor { + boundary_id: self.boundary_id, + generation: self.generation, + session_id: self.session_id, + workload_identity: self.workload_identity, + transport: SandboxTransport::Tcp { + authority: self.control_authority, + addresses: vec![self.control_address], + }, + tls: self.supervisor_tls, + host_gateway_ip: self.host_gateway_ip, + resource_claims, + driver_fence, + }, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn spec() -> KubernetesSandboxRuntimeBoundarySpec { + KubernetesSandboxRuntimeBoundarySpec { + boundary_id: "sandbox-1".to_string(), + generation: "generation-1".to_string(), + session_id: openshell_core::SandboxSessionId::new(), + session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(), + auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).unwrap(), + gateway_id: "gateway-1".to_string(), + verification_keys: vec![GatewayVerificationKey { + key_id: "key-1".to_string(), + public_key_pem: "public-key".to_string(), + }], + namespace_uid: "namespace-uid".to_string(), + sandbox_resource_uid: "sandbox-resource-uid".to_string(), + workload_pod_uid: "pod-uid".to_string(), + workload_pod_uid_path: PathBuf::from("/.openshell/pod-identity/uid"), + supervisor_pod_uid: "supervisor-pod-uid".to_string(), + egress_policy_uid: "network-policy-uid".to_string(), + egress_policy_resource_version: "1945".to_string(), + boundary_listener: "0.0.0.0:5500".parse().expect("valid listener"), + control_authority: "os-boundary-sandbox.default.svc:5500".to_string(), + control_address: "10.42.0.7:5500".parse().expect("valid target"), + sandbox_tls: SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from("/run/boundary/tls.crt"), + private_key_path: PathBuf::from("/run/boundary/tls.key"), + }, + supervisor_tls: SandboxTlsClientConfig { + server_name: "boundary.sandbox.openshell".to_string(), + trust_anchor_pem: "test-ca".to_string(), + }, + host_gateway_ip: Some("10.42.0.1".parse().expect("valid gateway IP")), + workload_identity: ResolvedWorkloadIdentity::new( + 1000, + 1000, + vec![1000], + "kubernetes-config".to_string(), + "sandbox:sandbox-resource-uid".to_string(), + ) + .unwrap(), + child_env: HashMap::new(), + } + } + + #[test] + fn provisioning_binds_identical_kubernetes_resource_claims() { + let provisioned = spec().provision(); + + assert_eq!( + provisioned.boundary_config.resource_claims, + provisioned.runtime_descriptor.resource_claims + ); + assert_eq!( + provisioned.runtime_descriptor.resource_claims["kubernetes.sandbox_resource_uid"], + "sandbox-resource-uid" + ); + assert_eq!( + provisioned.runtime_descriptor.resource_claims["kubernetes.egress_policy_resource_version"], + "1945" + ); + assert_eq!( + provisioned.boundary_config.driver_fence, + provisioned.runtime_descriptor.driver_fence + ); + assert!( + provisioned + .runtime_descriptor + .driver_fence + .validate() + .is_ok() + ); + } + + #[test] + fn provisioning_uses_one_shared_tcp_protocol_across_pods() { + let provisioned = spec().provision(); + + assert_eq!( + provisioned.boundary_config.listener, + BoundaryListener::TlsTcp { + address: "0.0.0.0:5500".parse().expect("valid listener"), + tls: SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from("/run/boundary/tls.crt"), + private_key_path: PathBuf::from("/run/boundary/tls.key"), + }, + } + ); + assert_eq!( + provisioned.runtime_descriptor.transport, + SandboxTransport::Tcp { + authority: "os-boundary-sandbox.default.svc:5500".to_string(), + addresses: vec!["10.42.0.7:5500".parse().expect("valid target")], + } + ); + assert_eq!( + provisioned.runtime_descriptor.tls, + SandboxTlsClientConfig { + server_name: "boundary.sandbox.openshell".to_string(), + trust_anchor_pem: "test-ca".to_string(), + } + ); + } + + #[test] + fn network_fence_denies_all_workload_initiated_egress() { + let fence = KubernetesSandboxRuntimeNetworkFenceSpec { + namespace: "sandbox-ns".to_string(), + policy_name: "openshell-boundary-sandbox-1".to_string(), + supervisor_policy_name: "openshell-sandbox-supervisors".to_string(), + boundary_port: 5500, + } + .provision(); + + let policy_spec = fence.workload_policy.spec.expect("policy has a spec"); + assert_eq!( + policy_spec.policy_types, + Some(vec!["Ingress".to_string(), "Egress".to_string()]) + ); + assert_eq!(policy_spec.egress, Some(Vec::new())); + assert_eq!( + policy_spec.pod_selector.match_labels, + Some(fence.workload_labels) + ); + } + + #[test] + fn network_fence_allows_namespace_supervisors_to_boundary_port() { + let fence = KubernetesSandboxRuntimeNetworkFenceSpec { + namespace: "sandbox-ns".to_string(), + policy_name: "openshell-boundary-sandbox-1".to_string(), + supervisor_policy_name: "openshell-sandbox-supervisors".to_string(), + boundary_port: 5500, + } + .provision(); + + let policy_spec = fence.workload_policy.spec.expect("policy has a spec"); + let ingress = policy_spec + .ingress + .expect("policy has ingress rules") + .pop() + .expect("policy has one ingress rule"); + let peer = ingress + .from + .expect("rule has peers") + .pop() + .expect("rule has one peer"); + assert_eq!( + peer.pod_selector + .expect("peer has a pod selector") + .match_labels, + Some(fence.control_labels) + ); + assert!(peer.namespace_selector.is_none()); + + let port = ingress + .ports + .expect("rule has ports") + .pop() + .expect("rule has one port"); + assert_eq!(port.protocol.as_deref(), Some("TCP")); + assert_eq!(port.port, Some(IntOrString::Int(5500))); + } + + #[test] + fn network_fence_keeps_supervisor_egress_available() { + let fence = KubernetesSandboxRuntimeNetworkFenceSpec { + namespace: "sandbox-ns".to_string(), + policy_name: "openshell-sandbox-workloads".to_string(), + supervisor_policy_name: "openshell-sandbox-supervisors".to_string(), + boundary_port: 5500, + } + .provision(); + + let policy_spec = fence + .supervisor_policy + .spec + .expect("supervisor policy has a spec"); + assert_eq!(policy_spec.policy_types, Some(vec!["Egress".to_string()])); + assert_eq!( + policy_spec.pod_selector.match_labels, + Some(fence.control_labels) + ); + assert_eq!( + policy_spec.egress, + Some(vec![NetworkPolicyEgressRule::default()]) + ); + } +} diff --git a/crates/openshell-driver-kubernetes/src/lib.rs b/crates/openshell-driver-kubernetes/src/lib.rs index 28d3c77a7d..bb861308a2 100644 --- a/crates/openshell-driver-kubernetes/src/lib.rs +++ b/crates/openshell-driver-kubernetes/src/lib.rs @@ -4,13 +4,14 @@ pub mod config; pub mod driver; pub mod grpc; +pub mod isolation; pub mod otel_tracing; +mod sandbox_runtime; pub use config::{ - AppArmorProfile, DEFAULT_GATEWAY_ID, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, - DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesSidecarConfig, - ManagedSshIngressConfig, SupervisorSideloadMethod, SupervisorTopology, WorkspaceMode, - managed_namespace_prefix, + DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_STORAGE_SIZE, + KubernetesComputeConfig, KubernetesImagePullPolicy, KubernetesSandboxRuntimeConfig, + ManagedSshIngressConfig, WorkspaceMode, managed_namespace_prefix, }; pub use driver::{KubernetesComputeDriver, KubernetesDriverError}; pub use grpc::ComputeDriverService; diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index d949d6c72c..fbcaf1d5f6 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -8,13 +8,12 @@ use std::net::SocketAddr; use std::path::PathBuf; use tracing::info; +use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; -use openshell_core::{ImagePullPolicy, VERSION}; use openshell_driver_kubernetes::{ - AppArmorProfile, ComputeDriverService, DEFAULT_GATEWAY_ID, DEFAULT_PROXY_UID, - DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, KubernetesComputeConfig, KubernetesComputeDriver, - KubernetesSidecarConfig, ManagedSshIngressConfig, SupervisorSideloadMethod, SupervisorTopology, - WorkspaceMode, + ComputeDriverService, DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, + KubernetesComputeConfig, KubernetesComputeDriver, KubernetesImagePullPolicy, + KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, WorkspaceMode, }; #[derive(Parser, Debug)] @@ -72,7 +71,7 @@ struct Args { sandbox_image: Option, #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE_PULL_POLICY")] - sandbox_image_pull_policy: Option, + sandbox_image_pull_policy: Option, #[arg( long, @@ -94,10 +93,8 @@ struct Args { )] managed_ssh_gateway_pod_selector: Vec, - /// Gateway callback endpoint reachable from sandbox pods. Kubernetes - /// service topology cannot be inferred from the sandbox namespace. #[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")] - grpc_endpoint: String, + grpc_endpoint: Option, #[arg( long, @@ -112,37 +109,31 @@ struct Args { #[arg(long, env = "OPENSHELL_HOST_GATEWAY_IP")] host_gateway_ip: Option, + #[arg(long, env = "OPENSHELL_SANDBOX_RUNTIME_IMAGE")] + sandbox_runtime_image: Option, + + #[arg(long, env = "OPENSHELL_SANDBOX_RUNTIME_IMAGE_PULL_POLICY")] + sandbox_runtime_image_pull_policy: Option, + #[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE")] supervisor_image: Option, #[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY")] - supervisor_image_pull_policy: Option, + supervisor_image_pull_policy: Option, #[arg( long, - env = "OPENSHELL_SUPERVISOR_SIDELOAD_METHOD", - default_value = "image-volume" - )] - supervisor_sideload_method: SupervisorSideloadMethod, - - #[arg(long, env = "OPENSHELL_K8S_TOPOLOGY", default_value = "combined")] - topology: SupervisorTopology, - - #[arg( - long = "sidecar-proxy-uid", - alias = "proxy-uid", - env = "OPENSHELL_K8S_SIDECAR_PROXY_UID", - default_value_t = DEFAULT_PROXY_UID + env = "OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED", + default_value_t = false )] - sidecar_proxy_uid: u32, + sandbox_runtime_network_policy_enforced: bool, #[arg( - long = "sidecar-process-binary-aware-network-policy", - env = "OPENSHELL_K8S_SIDECAR_PROCESS_BINARY_AWARE_NETWORK_POLICY", - default_value_t = true, - action = ArgAction::Set + long, + env = "OPENSHELL_K8S_SANDBOX_RUNTIME_BOUNDARY_PORT", + default_value_t = 5500 )] - sidecar_process_binary_aware_network_policy: bool, + sandbox_runtime_boundary_port: u16, /// Corporate HTTP forward proxy for policy-approved TLS CONNECT egress. #[arg(long, env = "OPENSHELL_UPSTREAM_PROXY")] @@ -171,9 +162,6 @@ struct Args { #[arg(long, env = "OPENSHELL_ENABLE_USER_NAMESPACES")] enable_user_namespaces: bool, - #[arg(long, env = "OPENSHELL_K8S_APP_ARMOR_PROFILE")] - app_armor_profile: Option, - /// Lifetime (seconds) of the projected `ServiceAccount` token /// kubelet writes into each sandbox pod for the `IssueSandboxToken` /// bootstrap exchange. Kubelet enforces a minimum of 600s; the @@ -257,16 +245,17 @@ async fn main() -> Result<()> { gateway_namespace: args.managed_ssh_gateway_namespace.unwrap_or_default(), gateway_pod_selector: managed_ssh_gateway_pod_selector, }, + sandbox_runtime_image: args + .sandbox_runtime_image + .unwrap_or_else(openshell_core::config::default_sandbox_runtime_image), + sandbox_runtime_image_pull_policy: args.sandbox_runtime_image_pull_policy, supervisor_image: args .supervisor_image .unwrap_or_else(openshell_core::config::default_supervisor_image), supervisor_image_pull_policy: args.supervisor_image_pull_policy, - supervisor_sideload_method: args.supervisor_sideload_method, - topology: args.topology, - sidecar: KubernetesSidecarConfig { - proxy_uid: args.sidecar_proxy_uid, - process_binary_aware_network_policy: args - .sidecar_process_binary_aware_network_policy, + sandbox_runtime: KubernetesSandboxRuntimeConfig { + network_policy_enforced: args.sandbox_runtime_network_policy_enforced, + boundary_port: args.sandbox_runtime_boundary_port, }, https_proxy: args.https_proxy, no_proxy: args.no_proxy, @@ -274,12 +263,11 @@ async fn main() -> Result<()> { proxy_auth_secret_key: args.proxy_auth_secret_key, proxy_auth_allow_insecure: args.proxy_auth_allow_insecure.then_some(true), proxy_connect_by_hostname: args.proxy_connect_by_hostname.then_some(true), - grpc_endpoint: args.grpc_endpoint, + grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), ssh_socket_path: args.sandbox_ssh_socket_path, client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(), host_gateway_ip: args.host_gateway_ip.unwrap_or_default(), enable_user_namespaces: args.enable_user_namespaces, - app_armor_profile: args.app_armor_profile, workspace_default_storage_size: std::env::var( "OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE", ) @@ -337,13 +325,6 @@ async fn main() -> Result<()> { mod tests { use super::*; - #[test] - fn requires_explicit_gateway_callback_endpoint() { - let error = Args::try_parse_from(["openshell-driver-kubernetes"]) - .expect_err("Kubernetes service topology must be explicit"); - assert!(error.to_string().contains("--grpc-endpoint")); - } - #[test] fn accepts_gateway_otlp_configuration() { let args = Args::try_parse_from([ @@ -352,8 +333,6 @@ mod tests { "http://collector.example:4317", "--gateway-name", "kubernetes-dev", - "--grpc-endpoint", - "http://openshell.example:8080", ]) .expect("OTLP endpoint should parse"); diff --git a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs new file mode 100644 index 0000000000..2653597ff3 --- /dev/null +++ b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs @@ -0,0 +1,822 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Kubernetes resources that place and protect the `OpenShell` sandbox runtime. + +use std::collections::BTreeMap; +use std::path::Path; + +use k8s_openapi::ByteString; +use k8s_openapi::api::core::v1::{ + CSIVolumeSource, Capabilities, Container, EmptyDirVolumeSource, EnvVar, ExecAction, KeyToPath, + LocalObjectReference, Pod, PodSchedulingGate, PodSecurityContext, PodSpec, Probe, + ProjectedVolumeSource, Secret, SecretVolumeSource, SecurityContext, Service, + ServiceAccountTokenProjection, ServicePort, ServiceSpec, Volume, VolumeMount, VolumeProjection, +}; +use k8s_openapi::apimachinery::pkg::apis::meta::v1::OwnerReference; +use k8s_openapi::apimachinery::pkg::util::intstr::IntOrString; +use kube::core::ObjectMeta; +use rcgen::{CertificateParams, DnType, IsCa, KeyPair, KeyUsagePurpose}; + +use crate::isolation::{ + BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesSandboxRuntimeNetworkFence, + KubernetesSandboxRuntimeNetworkFenceSpec, +}; + +pub const SANDBOX_SECRET_COMPONENT: &str = "sandbox-bootstrap"; +pub const SUPERVISOR_SECRET_COMPONENT: &str = "supervisor-bootstrap"; +pub const BOUNDARY_CONFIG_KEY: &str = "boundary.json"; +pub const BACKEND_DESCRIPTOR_KEY: &str = "runtime-descriptor.json"; +pub const BOUNDARY_CERTIFICATE_KEY: &str = "tls.crt"; +pub const BOUNDARY_PRIVATE_KEY: &str = "tls.key"; +pub const SUPERVISOR_AUTH_BUNDLE_KEY: &str = "auth.json"; +pub const PROXY_CA_CERTIFICATE_KEY: &str = "proxy-ca.crt"; +pub const PROXY_CA_PRIVATE_KEY: &str = "proxy-ca.key"; +pub const SANDBOX_BOOTSTRAP_INPUT_PATH: &str = "/.openshell/bootstrap-input"; +pub const BOUNDARY_CONFIG_PATH: &str = "/.openshell/state/bootstrap/boundary.json"; +pub const BOUNDARY_CERTIFICATE_PATH: &str = "/.openshell/state/bootstrap/tls.crt"; +pub const BOUNDARY_PRIVATE_KEY_PATH: &str = "/.openshell/state/bootstrap/tls.key"; +pub const BACKEND_DESCRIPTOR_PATH: &str = "/.openshell/supervisor/runtime-descriptor.json"; +pub const SUPERVISOR_AUTH_BUNDLE_PATH: &str = "/.openshell/supervisor/auth.json"; +pub const PROXY_CA_CERTIFICATE_PATH: &str = "/.openshell/supervisor/proxy-ca.crt"; +pub const PROXY_CA_PRIVATE_KEY_PATH: &str = "/.openshell/supervisor/proxy-ca.key"; +pub const CONTROL_HEALTH_SOCKET_PATH: &str = "/run/openshell/health.sock"; +pub const NAMESPACE_WORKLOAD_POLICY_NAME: &str = "openshell-sandbox-workloads"; +pub const NAMESPACE_SUPERVISOR_EGRESS_POLICY_NAME: &str = "openshell-sandbox-supervisors"; + +pub struct ProxyCaMaterial { + pub certificate_pem: String, + pub private_key_pem: String, +} + +pub fn generate_proxy_ca_material() -> Result { + let key = KeyPair::generate().map_err(|error| format!("generate proxy CA key: {error}"))?; + let mut params = CertificateParams::default(); + params.is_ca = IsCa::Ca(rcgen::BasicConstraints::Unconstrained); + params + .distinguished_name + .push(DnType::CommonName, "OpenShell Sandbox CA"); + params + .distinguished_name + .push(DnType::OrganizationName, "OpenShell"); + params.key_usages = vec![KeyUsagePurpose::KeyCertSign, KeyUsagePurpose::CrlSign]; + let certificate = params + .self_signed(&key) + .map_err(|error| format!("generate proxy CA certificate: {error}"))?; + Ok(ProxyCaMaterial { + certificate_pem: certificate.pem(), + private_key_pem: key.serialize_pem(), + }) +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SandboxRuntimeNames { + pub sandbox_secret: String, + pub supervisor_secret: String, + pub boundary_service: String, + pub supervisor_pod: String, + pub workload_policy: String, + pub supervisor_policy: String, +} + +impl SandboxRuntimeNames { + #[must_use] + pub fn new(sandbox_id: &str) -> Self { + let suffix = sandbox_id.to_ascii_lowercase(); + Self { + sandbox_secret: format!("os-sandbox-{suffix}"), + supervisor_secret: format!("os-supervisor-{suffix}"), + boundary_service: format!("os-boundary-{suffix}"), + supervisor_pod: format!("os-supervisor-{suffix}"), + workload_policy: NAMESPACE_WORKLOAD_POLICY_NAME.to_string(), + supervisor_policy: NAMESPACE_SUPERVISOR_EGRESS_POLICY_NAME.to_string(), + } + } + + /// Return stable companion names plus generation-specific immutable + /// bootstrap Secret names. + #[must_use] + pub fn for_generation(sandbox_id: &str, generation: &str) -> Self { + let mut names = Self::new(sandbox_id); + let generation = generation + .chars() + .filter(char::is_ascii_alphanumeric) + .take(12) + .collect::() + .to_ascii_lowercase(); + names.sandbox_secret = format!("{}-{generation}", names.sandbox_secret); + names.supervisor_secret = format!("{}-{generation}", names.supervisor_secret); + names + } +} + +#[must_use] +pub fn pair_label_value(sandbox_id: &str) -> String { + sandbox_id.to_ascii_lowercase() +} + +#[must_use] +pub fn workload_fence( + namespace: &str, + names: &SandboxRuntimeNames, + boundary_port: u16, +) -> KubernetesSandboxRuntimeNetworkFence { + KubernetesSandboxRuntimeNetworkFenceSpec { + namespace: namespace.to_string(), + policy_name: names.workload_policy.clone(), + supervisor_policy_name: names.supervisor_policy.clone(), + boundary_port, + } + .provision() +} + +#[must_use] +pub fn boundary_service( + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + boundary_port: u16, + owner: OwnerReference, +) -> Service { + Service { + metadata: ObjectMeta { + name: Some(names.boundary_service.clone()), + namespace: Some(namespace.to_string()), + owner_references: Some(vec![owner]), + labels: Some(common_labels(sandbox_id, "boundary-service")), + ..Default::default() + }, + spec: Some(ServiceSpec { + selector: Some(pair_labels(sandbox_id, "workload")), + ports: Some(vec![ServicePort { + name: Some("boundary".to_string()), + protocol: Some("TCP".to_string()), + port: i32::from(boundary_port), + target_port: Some(IntOrString::Int(i32::from(boundary_port))), + ..Default::default() + }]), + ..Default::default() + }), + ..Default::default() + } +} + +#[allow(clippy::too_many_arguments, clippy::similar_names)] +pub fn supervisor_pod( + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + sandbox_name: &str, + gateway_id: &str, + supervisor_image: &str, + supervisor_pull_policy: Option, + service_account_name: &str, + control_uid: u32, + control_gid: u32, + image_pull_secrets: &[String], + grpc_endpoint: &str, + client_tls_secret_name: &str, + main_process_spec: &str, + log_level: &str, + sa_token_ttl_secs: i64, + https_proxy: Option<&str>, + no_proxy: Option<&str>, + proxy_auth_secret: Option<(&str, &str)>, + proxy_auth_allow_insecure: bool, + proxy_connect_by_hostname: bool, + provider_spiffe_socket_path: Option<&str>, + owner: OwnerReference, +) -> Result { + let labels = control_labels(sandbox_id, gateway_id); + let mut environment = vec![ + env_var( + "OPENSHELL_ADMITTED_ISOLATION_BACKEND", + crate::isolation::BACKEND_NAME, + ), + env_var("OPENSHELL_ENDPOINT", grpc_endpoint), + env_var("OPENSHELL_SANDBOX_ID", sandbox_id), + env_var("OPENSHELL_SANDBOX", sandbox_name), + env_var("OPENSHELL_MAIN_PROCESS_SPEC", main_process_spec), + env_var( + "OPENSHELL_K8S_SA_TOKEN_FILE", + "/var/run/secrets/openshell/token", + ), + env_var("OPENSHELL_SSH_SOCKET_PATH", "/run/openshell/ssh.sock"), + env_var(openshell_core::sandbox_env::SSH_SOCKET_SHARED, "true"), + env_var("OPENSHELL_PROXY_TLS_DIR", "/run/openshell/proxy-tls"), + env_var( + openshell_core::sandbox_env::PROXY_CA_CERT, + PROXY_CA_CERTIFICATE_PATH, + ), + env_var( + openshell_core::sandbox_env::PROXY_CA_KEY, + PROXY_CA_PRIVATE_KEY_PATH, + ), + env_var("OPENSHELL_LOG_LEVEL", log_level), + env_var( + openshell_core::sandbox_env::TELEMETRY_ENABLED, + openshell_core::telemetry::enabled_env_value(), + ), + env_var( + openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, + "", + ), + ]; + let mut volume_mounts = vec![ + volume_mount("bootstrap", "/.openshell/supervisor", true), + volume_mount("sa-token", "/var/run/secrets/openshell", true), + volume_mount("run", "/run/openshell", false), + volume_mount("logs", "/var/log", false), + ]; + let mut volumes = vec![ + secret_volume("bootstrap", &names.supervisor_secret, None), + Volume { + name: "sa-token".to_string(), + projected: Some(ProjectedVolumeSource { + default_mode: Some(0o440), + sources: Some(vec![VolumeProjection { + service_account_token: Some(ServiceAccountTokenProjection { + audience: Some("openshell-gateway".to_string()), + expiration_seconds: Some(sa_token_ttl_secs), + path: "token".to_string(), + }), + ..Default::default() + }]), + }), + ..Default::default() + }, + empty_dir_volume("run"), + empty_dir_volume("logs"), + ]; + if !client_tls_secret_name.is_empty() { + environment.extend([ + env_var("OPENSHELL_TLS_CA", "/var/run/secrets/openshell-tls/ca.crt"), + env_var( + "OPENSHELL_TLS_CERT", + "/var/run/secrets/openshell-tls/tls.crt", + ), + env_var( + "OPENSHELL_TLS_KEY", + "/var/run/secrets/openshell-tls/tls.key", + ), + ]); + volume_mounts.push(volume_mount( + "client-tls", + "/var/run/secrets/openshell-tls", + true, + )); + volumes.push(secret_volume("client-tls", client_tls_secret_name, None)); + } + let mut command = vec![ + "/openshell-supervisor".to_string(), + "--backend-descriptor-file".to_string(), + BACKEND_DESCRIPTOR_PATH.to_string(), + "--auth-bundle-file".to_string(), + SUPERVISOR_AUTH_BUNDLE_PATH.to_string(), + "--workdir".to_string(), + "/sandbox".to_string(), + "--health-socket-path".to_string(), + CONTROL_HEALTH_SOCKET_PATH.to_string(), + ]; + if let Some(url) = https_proxy { + command.extend(["--upstream-proxy".to_string(), url.to_string()]); + } + if let Some(hosts) = no_proxy { + command.extend(["--upstream-no-proxy".to_string(), hosts.to_string()]); + } + if proxy_auth_secret.is_some() { + command.extend([ + "--upstream-proxy-auth-file".to_string(), + openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH.to_string(), + ]); + } + if proxy_auth_allow_insecure { + command.push("--upstream-proxy-auth-allow-insecure".to_string()); + } + if proxy_connect_by_hostname { + command.push("--upstream-proxy-connect-by-hostname".to_string()); + } + if let Some((secret_name, secret_key)) = proxy_auth_secret { + let auth_path = Path::new(openshell_core::container_paths::UPSTREAM_PROXY_AUTH_MOUNT_PATH); + let mount_path = auth_path + .parent() + .and_then(Path::to_str) + .ok_or_else(|| "upstream proxy authentication path has no UTF-8 parent".to_string())?; + let item_path = auth_path + .file_name() + .and_then(|name| name.to_str()) + .ok_or_else(|| { + "upstream proxy authentication path has no UTF-8 file name".to_string() + })?; + volume_mounts.push(volume_mount("upstream-proxy-auth", mount_path, true)); + volumes.push(secret_volume( + "upstream-proxy-auth", + secret_name, + Some(KeyToPath { + key: secret_key.to_string(), + path: item_path.to_string(), + ..Default::default() + }), + )); + } + if let Some(socket_path) = provider_spiffe_socket_path { + environment.push(env_var( + openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, + socket_path, + )); + let mount_path = Path::new(socket_path) + .parent() + .and_then(Path::to_str) + .ok_or_else(|| "SPIFFE socket path has no UTF-8 parent".to_string())?; + volume_mounts.push(volume_mount("spiffe-workload-api", mount_path, true)); + volumes.push(Volume { + name: "spiffe-workload-api".to_string(), + csi: Some(CSIVolumeSource { + driver: "csi.spiffe.io".to_string(), + read_only: Some(true), + ..Default::default() + }), + ..Default::default() + }); + } + let mut container = Container { + name: "supervisor".to_string(), + image: Some(supervisor_image.to_string()), + command: Some(command), + termination_message_policy: Some("FallbackToLogsOnError".to_string()), + env: Some(environment), + readiness_probe: Some(Probe { + exec: Some(ExecAction { + command: Some(vec![ + "/openshell-supervisor".to_string(), + "health".to_string(), + "--socket".to_string(), + CONTROL_HEALTH_SOCKET_PATH.to_string(), + ]), + }), + period_seconds: Some(1), + failure_threshold: Some(3), + ..Default::default() + }), + security_context: Some(SecurityContext { + run_as_user: Some(i64::from(control_uid)), + run_as_group: Some(i64::from(control_gid)), + run_as_non_root: Some(true), + read_only_root_filesystem: Some(true), + allow_privilege_escalation: Some(false), + capabilities: Some(Capabilities { + drop: Some(vec!["ALL".to_string()]), + ..Default::default() + }), + ..Default::default() + }), + volume_mounts: Some(volume_mounts), + ..Default::default() + }; + if let Some(policy) = supervisor_pull_policy { + container.image_pull_policy = Some(policy.as_kubernetes_str().to_string()); + } + Ok(Pod { + metadata: ObjectMeta { + name: Some(names.supervisor_pod.clone()), + namespace: Some(namespace.to_string()), + owner_references: Some(vec![owner]), + labels: Some(labels), + annotations: Some(BTreeMap::from([( + "openshell.ai/sandbox-id".to_string(), + sandbox_id.to_string(), + )])), + ..Default::default() + }, + spec: Some(PodSpec { + service_account_name: Some(service_account_name.to_string()), + image_pull_secrets: Some( + image_pull_secrets + .iter() + .map(|name| LocalObjectReference { name: name.clone() }) + .collect(), + ), + automount_service_account_token: Some(false), + scheduling_gates: Some(vec![PodSchedulingGate { + name: "openshell.ai/bootstrap".to_string(), + }]), + security_context: Some(PodSecurityContext { + fs_group: Some(i64::from(control_gid)), + fs_group_change_policy: Some("OnRootMismatch".to_string()), + seccomp_profile: Some(k8s_openapi::api::core::v1::SeccompProfile { + type_: "RuntimeDefault".to_string(), + ..Default::default() + }), + ..Default::default() + }), + restart_policy: Some("Never".to_string()), + containers: vec![container], + volumes: Some(volumes), + ..Default::default() + }), + ..Default::default() + }) +} + +#[allow(clippy::too_many_arguments)] +#[must_use] +pub fn sandbox_bootstrap_secret( + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + boundary_config: Vec, + boundary_certificate: Vec, + boundary_private_key: Vec, + owner: OwnerReference, +) -> Secret { + Secret { + metadata: ObjectMeta { + name: Some(names.sandbox_secret.clone()), + namespace: Some(namespace.to_string()), + owner_references: Some(vec![owner]), + labels: Some(common_labels(sandbox_id, SANDBOX_SECRET_COMPONENT)), + ..Default::default() + }, + data: Some(BTreeMap::from([ + (BOUNDARY_CONFIG_KEY.to_string(), ByteString(boundary_config)), + ( + BOUNDARY_CERTIFICATE_KEY.to_string(), + ByteString(boundary_certificate), + ), + ( + BOUNDARY_PRIVATE_KEY.to_string(), + ByteString(boundary_private_key), + ), + ])), + immutable: Some(true), + type_: Some("Opaque".to_string()), + ..Default::default() + } +} + +#[must_use] +#[allow(clippy::too_many_arguments)] +pub fn supervisor_bootstrap_secret( + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + backend_descriptor: Vec, + supervisor_auth_bundle: Vec, + proxy_ca_certificate: Vec, + proxy_ca_private_key: Vec, + owner: OwnerReference, +) -> Secret { + Secret { + metadata: ObjectMeta { + name: Some(names.supervisor_secret.clone()), + namespace: Some(namespace.to_string()), + owner_references: Some(vec![owner]), + labels: Some(common_labels(sandbox_id, SUPERVISOR_SECRET_COMPONENT)), + ..Default::default() + }, + data: Some(BTreeMap::from([ + ( + BACKEND_DESCRIPTOR_KEY.to_string(), + ByteString(backend_descriptor), + ), + ( + SUPERVISOR_AUTH_BUNDLE_KEY.to_string(), + ByteString(supervisor_auth_bundle), + ), + ( + PROXY_CA_CERTIFICATE_KEY.to_string(), + ByteString(proxy_ca_certificate), + ), + ( + PROXY_CA_PRIVATE_KEY.to_string(), + ByteString(proxy_ca_private_key), + ), + ])), + immutable: Some(true), + type_: Some("Opaque".to_string()), + ..Default::default() + } +} + +#[must_use] +pub fn sandbox_owner_reference( + name: &str, + uid: &str, + api_version: &str, + controller: bool, +) -> OwnerReference { + OwnerReference { + api_version: api_version.to_string(), + kind: "Sandbox".to_string(), + name: name.to_string(), + uid: uid.to_string(), + controller: controller.then_some(true), + // The driver's RBAC intentionally does not permit mutating Sandbox + // finalizers. Kubernetes garbage collection does not require this bit, + // and setting it would make admission fail under + // OwnerReferencesPermissionEnforcement. + block_owner_deletion: Some(false), + } +} + +fn pair_labels(sandbox_id: &str, role: &str) -> BTreeMap { + BTreeMap::from([ + ( + BOUNDARY_PAIR_LABEL.to_string(), + pair_label_value(sandbox_id), + ), + (BOUNDARY_ROLE_LABEL.to_string(), role.to_string()), + ]) +} + +fn common_labels(sandbox_id: &str, component: &str) -> BTreeMap { + BTreeMap::from([ + ( + "openshell.ai/managed-by".to_string(), + "openshell".to_string(), + ), + ( + "openshell.ai/sandbox-id".to_string(), + sandbox_id.to_string(), + ), + ("openshell.ai/component".to_string(), component.to_string()), + ]) +} + +fn control_labels(sandbox_id: &str, gateway_id: &str) -> BTreeMap { + let mut labels = common_labels(sandbox_id, "supervisor"); + labels.extend(pair_labels(sandbox_id, "supervisor")); + labels.insert( + "openshell.ai/gateway-id".to_string(), + gateway_id.to_string(), + ); + labels +} + +fn env_var(name: &str, value: &str) -> EnvVar { + EnvVar { + name: name.to_string(), + value: Some(value.to_string()), + ..Default::default() + } +} + +fn volume_mount(name: &str, mount_path: &str, read_only: bool) -> VolumeMount { + VolumeMount { + name: name.to_string(), + mount_path: mount_path.to_string(), + read_only: read_only.then_some(true), + ..Default::default() + } +} + +fn secret_volume(name: &str, secret_name: &str, item: Option) -> Volume { + Volume { + name: name.to_string(), + secret: Some(SecretVolumeSource { + secret_name: Some(secret_name.to_string()), + default_mode: Some(0o440), + items: item.map(|item| vec![item]), + ..Default::default() + }), + ..Default::default() + } +} + +fn empty_dir_volume(name: &str) -> Volume { + Volume { + name: name.to_string(), + empty_dir: Some(EmptyDirVolumeSource::default()), + ..Default::default() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn owner() -> OwnerReference { + sandbox_owner_reference("demo", "uid-1", "agents.x-k8s.io/v1beta1", false) + } + + #[test] + fn service_selects_only_the_workload_boundary() { + let names = SandboxRuntimeNames::new("4b67c0d0-1111-2222-3333-444444444444"); + let service = boundary_service("sandbox", &names, "pair", 5500, owner()); + assert_eq!( + service.spec.unwrap().selector.unwrap()[BOUNDARY_ROLE_LABEL], + "workload" + ); + } + + #[test] + fn owner_reference_does_not_require_finalizer_mutation_permission() { + assert_eq!(owner().block_owner_deletion, Some(false)); + } + + #[test] + fn supervisor_pod_is_gated_non_restarting_and_unprivileged() { + let names = SandboxRuntimeNames::new("pair"); + let pod = supervisor_pod( + "sandbox", + &names, + "pair", + "demo", + "gateway", + "supervisor:latest", + Some(crate::KubernetesImagePullPolicy::IfNotPresent), + "sandbox-sa", + 1000, + 1000, + &["registry-credentials".to_string()], + "https://gateway:8080", + "client-tls", + "{}", + "info", + 600, + None, + None, + None, + false, + false, + None, + owner(), + ) + .expect("render supervisor Pod"); + let pod_spec = pod.spec.as_ref().expect("Pod spec"); + assert_eq!( + pod.metadata + .owner_references + .as_ref() + .expect("owner references")[0] + .controller, + None + ); + let container = &pod_spec.containers[0]; + assert_eq!(container.image_pull_policy.as_deref(), Some("IfNotPresent")); + assert_eq!(pod_spec.automount_service_account_token, Some(false)); + assert_eq!(pod_spec.restart_policy.as_deref(), Some("Never")); + assert_eq!( + pod_spec + .scheduling_gates + .as_ref() + .and_then(|gates| gates.first()) + .map(|gate| gate.name.as_str()), + Some("openshell.ai/bootstrap") + ); + assert_eq!( + pod_spec + .image_pull_secrets + .as_ref() + .expect("image pull secrets")[0] + .name + .as_str(), + "registry-credentials" + ); + let pod_security = pod_spec + .security_context + .as_ref() + .expect("Pod security context"); + assert_eq!(pod_security.fs_group, Some(1000)); + assert_eq!( + pod_security + .seccomp_profile + .as_ref() + .map(|profile| profile.type_.as_str()), + Some("RuntimeDefault") + ); + let container_security = container + .security_context + .as_ref() + .expect("container security context"); + assert_eq!(container_security.run_as_user, Some(1000)); + assert_eq!(container_security.run_as_non_root, Some(true)); + assert_eq!(container_security.read_only_root_filesystem, Some(true)); + assert_eq!( + container_security + .capabilities + .as_ref() + .and_then(|capabilities| capabilities.drop.as_ref()), + Some(&vec!["ALL".to_string()]) + ); + assert_eq!( + container + .readiness_probe + .as_ref() + .and_then(|probe| probe.exec.as_ref()) + .and_then(|exec| exec.command.as_ref()), + Some(&vec![ + "/openshell-supervisor".to_string(), + "health".to_string(), + "--socket".to_string(), + CONTROL_HEALTH_SOCKET_PATH.to_string(), + ]) + ); + let command = container.command.as_ref().unwrap(); + assert!( + command + .windows(2) + .any(|args| args == ["--health-socket-path", CONTROL_HEALTH_SOCKET_PATH]) + ); + let env = container.env.as_ref().unwrap(); + let env_value = |name: &str| { + env.iter() + .find(|variable| variable.name == name) + .and_then(|variable| variable.value.as_deref()) + }; + assert_eq!( + env_value(openshell_core::sandbox_env::SSH_SOCKET_SHARED), + Some("true") + ); + assert_eq!( + env_value(openshell_core::sandbox_env::PROXY_CA_CERT), + Some(PROXY_CA_CERTIFICATE_PATH) + ); + assert_eq!( + env_value(openshell_core::sandbox_env::PROXY_CA_KEY), + Some(PROXY_CA_PRIVATE_KEY_PATH) + ); + let mount = container + .volume_mounts + .as_ref() + .unwrap() + .iter() + .find(|mount| mount.name == "bootstrap") + .expect("durable supervisor material is mounted into supervisor"); + assert_eq!(mount.mount_path, "/.openshell/supervisor"); + assert_eq!(mount.read_only, Some(true)); + } + + #[test] + fn bootstrap_secrets_are_immutable_and_split_by_trust_domain() { + let names = SandboxRuntimeNames::new("pair"); + let sandbox = sandbox_bootstrap_secret( + "sandbox", + &names, + "pair", + Vec::new(), + Vec::new(), + Vec::new(), + owner(), + ); + assert_eq!( + sandbox.metadata.labels.as_ref().unwrap()["openshell.ai/component"], + SANDBOX_SECRET_COMPONENT + ); + assert_eq!(sandbox.immutable, Some(true)); + let sandbox_keys = sandbox + .data + .unwrap() + .into_keys() + .collect::>(); + assert_eq!( + sandbox_keys, + std::collections::BTreeSet::from([ + BOUNDARY_CERTIFICATE_KEY.to_string(), + BOUNDARY_CONFIG_KEY.to_string(), + BOUNDARY_PRIVATE_KEY.to_string(), + ]) + ); + + let supervisor = supervisor_bootstrap_secret( + "sandbox", + &names, + "pair", + Vec::new(), + Vec::new(), + Vec::new(), + Vec::new(), + owner(), + ); + assert_eq!( + supervisor.metadata.labels.as_ref().unwrap()["openshell.ai/component"], + SUPERVISOR_SECRET_COMPONENT + ); + assert_eq!(supervisor.immutable, Some(true)); + let supervisor_keys = supervisor + .data + .unwrap() + .into_keys() + .collect::>(); + assert_eq!( + supervisor_keys, + std::collections::BTreeSet::from([ + PROXY_CA_CERTIFICATE_KEY.to_string(), + PROXY_CA_PRIVATE_KEY.to_string(), + SUPERVISOR_AUTH_BUNDLE_KEY.to_string(), + BACKEND_DESCRIPTOR_KEY.to_string(), + ]) + ); + } + + #[test] + fn generated_proxy_ca_material_is_pem_encoded() { + let material = generate_proxy_ca_material().unwrap(); + assert!( + material + .certificate_pem + .starts_with("-----BEGIN CERTIFICATE-----") + ); + assert!(material.private_key_pem.contains("PRIVATE KEY")); + } +} diff --git a/crates/openshell-driver-mxc/src/driver.rs b/crates/openshell-driver-mxc/src/driver.rs index 60f824516f..a58c8fdc83 100644 --- a/crates/openshell-driver-mxc/src/driver.rs +++ b/crates/openshell-driver-mxc/src/driver.rs @@ -1199,6 +1199,7 @@ fn make_sandbox_with_condition( sandbox_fd: String::new(), conditions: vec![condition.clone()], deleting, + ..Default::default() }), } } diff --git a/crates/openshell-driver-podman/Cargo.toml b/crates/openshell-driver-podman/Cargo.toml index 8b3e014e8c..c49c309f65 100644 --- a/crates/openshell-driver-podman/Cargo.toml +++ b/crates/openshell-driver-podman/Cargo.toml @@ -17,6 +17,10 @@ path = "src/main.rs" [dependencies] openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] } openshell-otel = { path = "../openshell-otel" } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } +tar = "0.4" +uuid = { workspace = true } tokio = { workspace = true } tonic = { workspace = true, features = ["transport"] } diff --git a/crates/openshell-driver-podman/NETWORKING.md b/crates/openshell-driver-podman/NETWORKING.md index 567abcbfcd..4ad9eedae6 100644 --- a/crates/openshell-driver-podman/NETWORKING.md +++ b/crates/openshell-driver-podman/NETWORKING.md @@ -1,463 +1,62 @@ -# Rootless Podman Networking +# Podman Networking -Deep-dive into how networking works in the Podman compute driver when running -rootless with pasta as the network backend. Covers the external tooling -(Podman, Netavark, pasta, aardvark-dns), the three nested namespace layers, and -the complete data paths for SSH, outbound traffic, and supervisor-to-gateway -communication. - -For the general Podman driver architecture, lifecycle, API surface, and driver -comparison, see [README.md](README.md). - -## Component Stack - -Podman's networking is composed of four independent projects: - -| Component | Language | Role | -|---|---|---| -| Podman | Go | Container runtime; orchestrates network lifecycle. | -| Netavark | Rust | Network backend; creates interfaces, bridges, firewall rules. | -| aardvark-dns | Rust | Authoritative DNS server for container name resolution. | -| pasta, part of passt | C | User-mode networking; L2-to-L4 socket translation for rootless containers. | - -The key split: rootful containers default to Netavark bridge networking with -real kernel interfaces, while rootless containers commonly use pasta user-mode -networking without needing host privileges. - -## How Netavark Works - -Netavark is invoked by Podman as an external binary. It reads a JSON network -configuration from STDIN and executes one of three commands: - -- `netavark setup ` creates interfaces, assigns IPs, and sets up - firewall rules for NAT and port-forwarding. -- `netavark teardown ` reverses setup and removes interfaces and - firewall rules. -- `netavark create` takes a partial network config and completes it by - assigning subnets and gateways. - -For rootful bridge networking: - -1. Podman creates a network namespace for the container. -2. Podman invokes `netavark setup` with the network config JSON. -3. Netavark creates a bridge, such as `podman0`, if it does not exist. The - default subnet is `10.88.0.0/16`. -4. Netavark creates a veth pair. One end goes into the container's netns and - the other attaches to the bridge. -5. Netavark assigns an IP from the subnet to the container's veth interface. -6. Netavark configures iptables or nftables rules for masquerade and port - mappings. -7. Netavark starts aardvark-dns when DNS is enabled, listening on the bridge - gateway address. - -```text -Host Kernel - | - +-- Bridge interface, such as "podman0" - | | - | +-- veth pair endpoint, host side, container 1 - | +-- veth pair endpoint, host side, container 2 - | - +-- Host physical interface, such as eth0 - | - +-- NAT, iptables or nftables rules managed by Netavark -``` - -Netavark also supports macvlan networks, where the container gets a -sub-interface of a physical host NIC with its own MAC address, and external -plugins via a documented JSON API. - -## How Pasta Works - -Unprivileged users cannot create network interfaces on the host. They cannot -create veth pairs, bridges, or iptables rules. Netavark's bridge approach -cannot work directly for rootless containers without an additional rootless -networking layer. - -Pasta, part of the `passt` project, operates in userspace and translates -between the container's L2 TAP interface and the host's L4 sockets. It requires -no capabilities or privileges. - -```text -Container Network Namespace - | - +-- TAP device, such as "eth0" - | ^ - | | L2 frames, Ethernet - | v - +-- pasta process, userspace - | - | Translation: L2 frames <-> L4 sockets - | - v - Host Network Stack, native TCP/UDP/ICMP sockets -``` - -For an outbound TCP connection from a container: - -1. The application calls `connect()` to an external address. -2. The kernel routes the packet through the default gateway to the TAP device. -3. Pasta reads the raw Ethernet frame from the TAP file descriptor. -4. Pasta parses L2/L3/L4 headers and identifies the TCP SYN. -5. Pasta opens a native TCP socket on the host and calls `connect()` to the - same destination. -6. When the host socket connects, pasta reflects the SYN-ACK back through the - TAP as an L2 frame. -7. For ongoing data transfer, pasta translates between TAP frames and the host - socket, coordinating TCP windows and acknowledgments between the two sides. - -Pasta does not maintain per-connection packet buffers. It reflects observed -sending windows and ACKs directly between peers. This is a thinner translation -layer than a full TCP/IP stack. - -### Built-in Services - -Pasta includes minimal network services so the container stack can -auto-configure: - -| Service | Purpose | -|---|---| -| ARP proxy | Resolves the gateway address to the host's MAC address. | -| DHCP server | Hands out a single IPv4 address, usually matching the host's upstream interface. | -| NDP proxy | Handles IPv6 neighbor discovery and SLAAC prefix advertisement. | -| DHCPv6 server | Hands out a single IPv6 address, usually matching the host's upstream interface. | - -By default there is no NAT. Pasta copies the host's IP addresses into the -container namespace. - -### Local Connection Bypass - -For connections between the container and the host, pasta implements a local -bypass path: - -- Packets with a local destination skip L2 translation. -- TCP uses `splice(2)`. -- UDP uses `recvmmsg(2)` and `sendmmsg(2)`. - -### Port Forwarding - -By default, pasta uses auto-detection. It scans `/proc/net/tcp` and -`/proc/net/tcp6` periodically and automatically forwards ports that are bound -and listening. Port forwarding is configurable through pasta options. - -### Security Properties - -Pasta is designed for rootless use: - -- No dynamic memory allocation after startup. -- All capabilities dropped, except `CAP_NET_BIND_SERVICE` when granted. -- Restrictive seccomp profile. -- Detaches into its own user, mount, IPC, UTS, and PID namespaces. -- No external dependencies beyond libc. - -### Inter-Container Limitation - -Unlike bridge networking, pasta containers are isolated from each other by -default. No virtual bridge connects them. Communication requires port mappings -through the host, pods with a shared network namespace, or opting into rootless -Netavark bridge networking with `podman network create`. - -## Three Nested Namespaces - -The Podman compute driver creates three layers of network isolation: +Only the external supervisor has external network connectivity. The workload +container uses `network=none`; its loopback DNS relay and TCP socket mediation +reach the supervisor through a protected Unix socket, not a veth or proxy +environment variable. ```text -Namespace 1: Host - | - pasta manages port forwarding, such as 127.0.0.1: - gateway listens on its configured bind address and port - | -Namespace 2: Rootless Podman network namespace, managed by pasta - | - Bridge "openshell", often 10.89.x.0/24 - aardvark-dns for container name resolution - | - Container netns - supervisor, proxy, and relay client run here - | -Namespace 3: Inner sandbox netns, created by supervisor - | - veth pair, such as 10.200.0.1 <-> 10.200.0.2 - nftables forces ordinary traffic through proxy - user workload runs here +workload container supervisor container +agent -> sandbox -- private UDS / gRPC -> policy proxy -> host network -> destination + | + +-- authenticated gateway callback ``` -Pasta bridges namespace 1 and 2. The veth pair bridges namespace 2 and 3. The -proxy at the boundary of namespace 2 and 3 enforces network policy. +## Outer network fence -### Layer 1 Pasta +The driver creates the workload without networks, host aliases, published +ports, or added capabilities. It checks the Podman inspect response before +launch and restart. The sandbox installs seccomp mediation and Landlock before +executing the agent. It does not create network namespaces, configure nftables, +or require `CAP_NET_ADMIN`. -At driver startup, the driver ensures a Podman bridge network exists: +TCP opens, TCP byte streams, DNS requests/replies, and lifecycle operations +share the authenticated gRPC channel. DNS is resolved and authorized by the +supervisor. General UDP is unsupported. -```rust -client.ensure_network(&config.network_name).await?; -``` +## Supervisor callback network -This creates a bridge network named `openshell` by default, with DNS enabled. -In rootless mode, this bridge can exist inside a user namespace managed by -pasta. The bridge IP range is not reliably routable from the host. - -```text -Host - | - 127.0.0.1:, pasta binds this on the host - | - pasta process, translates L4 sockets <-> L2 TAP frames - | - rootless network namespace - | - Bridge "openshell", such as 10.89.1.0/24 - | - +-- 10.89.1.1, bridge gateway and aardvark-dns - | - +-- veth to container netns - | - 10.89.1.2, container IP -``` - -### Layer 2 Container Networking - -The container spec configures: - -- `nsmode: "bridge"` to use the Podman bridge network. -- `networks` to attach to the configured bridge, `openshell` by default. -- `portmappings` with `host_port: 0`, `container_port: 2222`, and `protocol: - "tcp"` to publish the SSH compatibility port on an ephemeral host port. -- `hostadd` entries for `host.containers.internal` and - `host.openshell.internal`, using Podman's `host-gateway` resolver or the - configured `host_gateway_ip`. - -Pasta is not explicitly configured by the driver. The driver requests bridge -mode and logs the network backend that Podman reports at startup. - -The `host.containers.internal` hostname is injected into `/etc/hosts` so the -supervisor can reach the gateway on the host. Linux defaults to -`host-gateway`; macOS Podman machine defaults to `192.168.127.254`, gvproxy's -host-loopback IP, because older Podman machine images can fail to resolve -`host-gateway`. Override this with `host_gateway_ip` or -`OPENSHELL_PODMAN_HOST_GATEWAY_IP` when a Podman machine uses a non-standard -host-loopback address. - -If `OPENSHELL_GRPC_ENDPOINT` is empty, the driver auto-detects: - -```rust -if config.grpc_endpoint.is_empty() { - let scheme = if config.tls_enabled() { - "https" - } else { - "http" - }; - config.grpc_endpoint = - format!("{scheme}://host.containers.internal:{}", config.gateway_port); -} -``` - -The bridge gateway IP is not a stable substitute in rootless mode because it -can live inside the user namespace rather than on the host. - -Before the gateway binds its serving sockets, the driver reports the callback -listener required by the selected topology: - -- Rootful Linux Podman reports the configured bridge's gateway address exactly. -- Rootless Linux Podman explicitly reporting pasta requests the private IPv4 - source address selected by the host's default route. This avoids guessing - among private interfaces on a multihomed host. -- Rootless Linux Podman reporting slirp4netns, another named helper, or no - helper cannot use a direct local callback listener. The driver fails startup - unless `grpc_endpoint` names an explicitly remote endpoint. Supporting - slirp4netns requires a relay inside Podman's rootless network namespace. -- Podman Machine requests IPv4 loopback because gvproxy terminates the host - forwarding path there. -- An explicitly remote callback endpoint requests no additional local listener. - -On Linux, an explicit `host_gateway_ip` is reported exactly for rootful Podman -and rootless pasta because the driver maps both local callback aliases to that -literal. Other rootless helpers still fail closed. Podman Machine requests -gateway loopback because its configured address is guest-visible and gvproxy -terminates that route on host loopback. The gateway validates and binds every -accepted callback requirement. If the primary listener covers the requested -address, the gateway reuses it and relies on sandbox JWT authorization to limit -the supervisor's RPCs. Otherwise, it creates an additional listener that -exposes only the gateway's sandbox-callable gRPC methods. Operator, health, -reflection, and HTTP requests must use the primary listener. - -### Layer 3 Inner Sandbox Network Namespace - -Inside the container, the supervisor creates another network namespace for the -user workload: - -```text -Container on the Podman bridge - | - Supervisor process, running in container's default netns - | - +-- Proxy listener at the inner namespace gateway address - | - +-- veth pair - | - +-- Inner network namespace - | - sandbox-side veth address - | - default route -> supervisor-side veth address - | - user code runs here - | - nftables rules: - ACCEPT -> proxy TCP - ACCEPT -> loopback - ACCEPT -> established/related - LOG -> TCP SYN bypass attempts - REJECT -> TCP - LOG -> UDP bypass attempts - REJECT -> UDP -``` - -The supervisor uses `nsenter --net=` rather than `ip netns exec` to avoid sysfs -remount issues that arise under rootless Podman where real host -`CAP_SYS_ADMIN` is unavailable. - -For a policy with explicit `protocol: tcp` endpoints, this same inner namespace -also hosts policy DNS and transparent TCP capture. The supervisor answers only -policy-eligible names with epoch-scoped synthetic addresses, redirects TCP to -those synthetic ranges into its transparent listener, and leaves direct real-IP -dials subject to the terminal bypass fence. The Podman driver advertises this -substrate through its driver-owned runtime capability; sandbox image and policy -environment values cannot opt into it independently. - -The container spec preserves Podman's resolver search domains and options. -Policy DNS captures both UDP and TCP in the inner namespace, so it does not -depend on libc honoring `use-vc` and does not change ordinary short-name -resolution for sandboxes that do not use native TCP. - -A tmpfs is mounted at `/run/netns` in the container spec so the supervisor can -create named network namespaces. In rootless Podman this directory does not -exist on the host, so a private tmpfs gives the supervisor its own writable -`/run/netns` without needing host filesystem access. - -## Complete Data Paths - -### SSH Session - -```text -Client, openshell CLI - | - 1. gRPC: CreateSshSession -> gateway, returns token and connect_path - 2. HTTP CONNECT /connect/ssh to gateway - headers: x-sandbox-id, x-sandbox-token - | -Gateway - | - 3. Looks up SupervisorSession for sandbox_id - 4. Sends RelayOpen{channel_id} over ConnectSupervisor bidi stream - | - gRPC traverses host -> pasta translation -> container bridge - | -Supervisor inside container - | - 5. Receives RelayOpen, opens new RelayStream RPC back to gateway - 6. Sends RelayInit{channel_id} on the stream - 7. Connects to Unix socket /run/openshell/ssh.sock - 8. Bidirectional bridge: RelayStream <-> Unix socket - | -SSH daemon inside container, Unix socket only - | - 9. Authenticates. Access is gated by the relay chain. - 10. Spawns shell process - 11. Shell enters inner netns via setns(fd, CLONE_NEWNET) - | -User shell in sandbox netns -``` - -The SSH daemon listens on a Unix socket with restrictive permissions. The -published TCP port mapping exists in the container spec for compatibility and -health/debug paths. Normal SSH communication uses the gRPC reverse-connect relay -pattern. - -### Outbound HTTP Request - -```text -User code in inner netns - | - 1. curl https://api.example.com - HTTP_PROXY points at the local sandbox proxy - | - 2. TCP connect to proxy - allowed by nftables as the only ordinary egress destination - | - 3. HTTP CONNECT api.example.com:443 - | -Supervisor proxy in container netns - | - 4. Policy evaluation with process identity - 5. SSRF check - 6. Optional L7 TLS intercept and HTTP method/path inspection - | - 7. If allowed, TCP connect to api.example.com:443 - from the container netns - | - 8. Through Podman bridge -> pasta -> host -> internet -``` - -### Supervisor gRPC Callback - -The Podman driver auto-detects the callback endpoint scheme based on whether -TLS client certificates are configured. When the RPM's auto-generated PKI is in -place, the endpoint is `https://host.containers.internal:17670` and the -supervisor connects with mTLS. Without TLS configuration, it falls back to -`http://host.containers.internal:`. - -```text -Supervisor in container netns - | - 1. Connects to host.containers.internal: - with mTLS when OPENSHELL_TLS_* paths are set - | - 2. Routed through container default gateway - | - 3. Pasta translates L2 frame -> host L4 socket when rootless backend uses pasta - | - 4. Host TCP socket connects to gateway - | -Gateway - | - 5. TLS handshake when enabled - 6. ConnectSupervisor bidirectional stream established - 7. Heartbeats at the interval accepted by the gateway - 8. Reconnects with exponential backoff on failure - 9. Same gRPC channel reused for RelayStream calls -``` +The supervisor companion uses Podman's host network. Host-gateway aliases and +the upstream corporate proxy apply only to the supervisor. The gateway's SSH +tunnel uses the supervisor relay over its private Unix socket, so the driver +does not publish a supervisor port. -The gateway binds to `127.0.0.1:17670` by default in the RPM packaging. Client -certificates are auto-generated by `openshell-gateway generate-certs` on first -start and bind-mounted into sandbox containers by the Podman driver. +Rootful Podman uses the configured bridge and its gateway address. Rootless +local callbacks require the existing pasta path; slirp4netns or unknown helpers +require an explicitly remote `grpc_endpoint`. On macOS, Podman Machine provides +the runtime and host-loopback forwarding. -## Differences from the Kubernetes Driver +These runtime-managed network helpers are outside the workload trust boundary. +Sharing the workload's user namespace preserves volume UID/GID mapping; it +does not share the workload's PID, mount, or network namespaces. -| Aspect | Kubernetes | Podman, rootless pasta | -|---|---|---| -| Container or pod IP | Routable cluster-wide | Non-routable from the host in common rootless setups. | -| Network reachability | Pod IPs reachable from gateway | Bridge not reliably routable from host; requires host aliases or published ports. | -| Sandbox to gateway | Direct TCP to Kubernetes service or endpoint | `host.containers.internal` through bridge and rootless backend. | -| SSH transport | Reverse gRPC relay | Reverse gRPC relay. | -| Port publishing | Not needed for relay | Ephemeral host port remains in the container spec for compatibility and debug paths. | -| TLS | mTLS via Kubernetes secrets | mTLS via mounted client files, RPM defaults, or explicit configuration. | -| DNS | Kubernetes CoreDNS | Podman bridge DNS through aardvark-dns when DNS is enabled. | -| Network policy | Kubernetes network policy for pod ingress plus supervisor policy | nftables inside inner sandbox netns plus supervisor policy. | -| Supervisor delivery | Kubernetes driver managed pod image or template | OCI image volume mount. | -| Secrets | Kubernetes Secret volume and env vars | Per-sandbox JWT via Podman secret; TLS client materials from configured host files. | +## Troubleshooting -Both drivers use the same reverse gRPC relay for SSH transport. The most -important Podman-specific difference is network reachability: in rootless -Podman, the bridge network is not reliably routable from the host, so -host-to-container and container-to-host communication must use host aliases, -published ports, or the supervisor relay. +Inspect both containers with the same sandbox-ID label, distinguishing +`openshell.io/isolation-role=sandbox` from +`openshell.io/isolation-role=supervisor`. -## Port Assignments +- Sandbox fails its qualification probe: use its log to identify the denied + kernel/runtime primitive. Do not add capabilities or disable runtime seccomp. +- Sandbox cannot authenticate to supervisor: check the private channel volume, + matching user namespace mappings, and shared SELinux label. +- Supervisor cannot call back: inspect its configured gateway endpoint, + credentials, host network, and gateway callback listener. +- DNS or egress denied: inspect supervisor policy decisions. Do not add a + workload network, resolver bypass, or direct gateway route. +- Pair is not Ready: check the supervisor health socket and gateway session. + A running workload container alone does not establish readiness. -| Port | Component | Purpose | -|---|---|---| -| `17670` | Gateway | Default local gRPC and HTTP multiplexed server port. | -| `2222` | Sandbox | Container port mapping default for the SSH compatibility port. | -| `3128` | Sandbox proxy | HTTP CONNECT proxy inside the sandbox network model. | -| `0` | Host | Ephemeral host port requested for the container SSH compatibility port. | +See the [driver overview](README.md) and +[Podman runtime documentation](https://docs.podman.io/en/latest/markdown/podman-run.1.html) +for runtime options. diff --git a/crates/openshell-driver-podman/README.md b/crates/openshell-driver-podman/README.md index 5781594158..912a79ac7a 100644 --- a/crates/openshell-driver-podman/README.md +++ b/crates/openshell-driver-podman/README.md @@ -1,489 +1,122 @@ # openshell-driver-podman -The Podman compute driver manages sandbox containers via the Podman REST API -over a Unix socket. It targets single-machine and developer environments where -rootless container isolation is preferred over a full Kubernetes cluster. The -driver runs in-process within the gateway server and delegates all sandbox -isolation enforcement to the `openshell-sandbox` supervisor binary, which is -sideloaded into each container via an OCI image volume mount. +The Podman compute driver runs inside the gateway and uses the native libpod +REST API over a Unix socket. Each sandbox has two independent containers: -When the gateway configures `[openshell.gateway.otlp]`, Podman compute-driver -spans export to the same OTLP/gRPC collector with the service name -`openshell-driver-podman`. The driver preserves the gateway trace context and -uses the same compute-driver RPC span names in its in-process and standalone -forms. +- `openshell-sandbox` owns the agent process in the workload container. +- `openshell-supervisor` evaluates policy, holds gateway credentials, and + proxies approved egress in a separate companion container. -`mise run gateway:podman` enables this export only when a local collector is -listening on `127.0.0.1:4317`. Otherwise, it omits the gateway OTLP configuration -so the development gateway does not repeatedly report export failures. +The driver provisions placement, identity, credentials, transport, and lifecycle. +The shared isolation interface supplies exec, attach, signal, terminate, binary +identity, DNS, TCP, and loopback-forwarding semantics. -Before creating the container, the driver inspects the final sandbox image and -captures its immutable image ID and raw OCI `Config.User`. Container creation -uses that image ID with pulling disabled, preventing a mutable tag from changing -between inspection and launch. The supervisor runs as root, resolves omitted -policy identity fields from the image declaration, and drops only agent -children to the completed identity. Named OCI components remain names after -validation; a missing group is filled with the user's numeric primary GID. Explicit -`process.run_as_user` and `process.run_as_group` values take precedence -independently. +## Runtime posture -For a rootless networking deep dive, see [NETWORKING.md](NETWORKING.md). - -## Stop and Start - -Stop stops the managed container without deleting it. The per-sandbox named -workspace volume, token and proxy-auth secrets, labels, and container metadata -remain intact. Start starts the same container and reuses the same named -volume. Stopped managed containers remain visible through list and watch -reconciliation. Delete remains responsible for removing the container, -driver-owned secrets, and workspace volume. - -The stop call waits until Podman reports the container as stopped or exited. -This keeps an immediate start from racing a rootless Podman stop that is still -finishing after its API request returns. - -Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted -phase requires running compute without changing that persisted intent. On -startup, the gateway sends an idempotent `StartSandbox` request for the same -sandboxes, restarting their retained containers. Explicitly stopped sandboxes -remain excluded. - -## Architecture - -The Podman driver communicates with the Podman daemon over a Unix socket and -delegates sandbox isolation to the supervisor binary running inside each -container. - -```mermaid -graph TB - CLI["openshell CLI"] -->|gRPC| GW["Gateway Server
(openshell-server)"] - GW -->|in-process| PD["PodmanComputeDriver"] - PD -->|HTTP/1.1
Unix socket| PA["Podman API"] - PA -->|OCI runtime
crun/runc| C["Sandbox Container"] - C -->|image volume
read-only| SV["Supervisor Binary
/opt/openshell/bin/openshell-sandbox"] - SV -->|creates| NS["Nested Network Namespace
veth pair + proxy"] - SV -->|enforces| LL["Landlock + seccomp"] - SV -->|gRPC callback| GW -``` - -## Isolation Model - -The Podman driver provides the same protection layers as the other compute -drivers. The driver itself does not implement isolation primitives directly. It -configures the container so that the `openshell-sandbox` supervisor can enforce -them at runtime. - -### Container Security Configuration - -The container spec in `container.rs` sets these security-critical fields: - -| Setting | Value | Rationale | +| Property | Workload | Supervisor | |---|---|---| -| `user` | `0:0` | The supervisor needs root inside the container for namespace creation, proxy setup, Landlock, seccomp, and filesystem preparation. | -| `cap_drop` | Selected unneeded defaults | Podman's default capability set is already restricted. The driver drops capabilities the supervisor does not need. | -| `cap_add` | `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, `SYSLOG`, `DAC_READ_SEARCH`, `SETPCAP`, `KILL` | Grants supervisor-only capabilities required for namespace setup, process identity, bypass diagnostics, child bounding-set cleanup, and forwarding shutdown signals to a workload that runs as the sandbox user. Policy DNS binds an unprivileged supervisor port and does not require `NET_BIND_SERVICE`. | -| `no_new_privileges` | `true` | Prevents privilege escalation after exec. | -| `seccomp_profile_path` | `unconfined` | The supervisor installs its own policy-aware BPF filter. A container-level profile can block Landlock/seccomp syscalls during setup. | -| `mounts` | Private tmpfs at `/run/netns` | Lets the supervisor create named network namespaces in rootless Podman. | -| CDI GPU devices | Opaque `driver_config.cdi_devices` values when set, otherwise the requested count of NVIDIA CDI GPUs selected in round-robin order. Local `/dev/dxg` permits `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. | Exposes requested GPUs to GPU-enabled sandbox containers. Exact CDI device lists must not contain duplicates and must match the effective GPU count. | - -The restricted agent child does not retain these supervisor privileges. - -## Driver Config Mounts - -The gateway forwards the `podman` block from `--driver-config-json` to this -driver. The driver accepts user-supplied `mounts` entries with these Podman -mount types: - -- `bind`: mounts an absolute host path when `[openshell.drivers.podman]` - has `enable_bind_mounts = true`. -- `volume`: mounts an existing Podman named volume. The driver validates that - the volume exists before provisioning and never creates or removes it. Podman - local-driver volumes created with bind options are treated as host bind - mounts and require `enable_bind_mounts = true`. -- `tmpfs`: mounts an in-memory filesystem with optional `options`, - `size_bytes`, and `mode`. -- `image`: mounts an OCI image through Podman's image-volume API. The driver - pulls the image during provisioning using the sandbox image pull policy. - -Host bind mounts are disabled by default because they expose gateway host paths -to sandbox requests. The driver still uses internal bind mounts for configured -TLS material; per-sandbox gateway JWTs are delivered through Podman secrets. - -Podman `bind` mounts accept `source`, `target`, optional `read_only`, and an -optional `selinux_label` of `shared` (applies `:z`) or `private` (applies -`:Z`) for SELinux-enforcing hosts. User-supplied bind and volume mounts are -read-only by default; set `read_only: false` to make them writable. Podman -image and volume mounts do not support `subpath` in OpenShell driver config. -Mount `source` and `target` values must not contain surrounding whitespace. -Mount targets must be absolute container paths and must not replace -the workspace root (`/sandbox`) or overlap OpenShell supervisor files, -`/etc/openshell`, `/etc/openshell-tls`, or `/run/netns`. - -Example named-volume usage: - -```shell -podman volume create openshell-work - -openshell sandbox create \ - --driver-config-json '{"podman":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \ - -- claude -``` - -### Capability Breakdown - -| Capability | Purpose | -|---|---| -| `SYS_ADMIN` | seccomp filter installation, namespace creation, and Landlock setup. | -| `NET_ADMIN` | Network namespace veth setup, IP address assignment, routes, and nftables. | -| `SYS_PTRACE` | Reading `/proc//exe` and walking process ancestry for binary identity. | -| `SYSLOG` | Reading `/dev/kmsg` for bypass-detection diagnostics. | -| `DAC_READ_SEARCH` | Reading `/proc//fd/` across UIDs so the proxy can resolve the binary responsible for a connection. | -| `SETPCAP` | Clearing the restricted child process capability bounding set before exec. | - -The driver intentionally keeps Podman's default `SETUID`, `SETGID`, `CHOWN`, -and `FOWNER` capabilities because the supervisor needs them to drop privileges -and prepare writable sandbox directories. It also keeps `SETPCAP` until child -setup so `drop_privileges()` can clear the child capability bounding set before -exec. It drops unneeded defaults such as -`DAC_OVERRIDE`, `FSETID`, `KILL`, `NET_RAW`, `SETFCAP`, -and `SYS_CHROOT`. - -## Supervisor Sideloading - -The supervisor binary is delivered to sandbox containers via Podman's OCI image -volume mechanism, distinct from both the Kubernetes pod-volume approach and the -VM's embedded guest bundle. - -```mermaid -sequenceDiagram - participant D as PodmanComputeDriver - participant P as Podman API - participant C as Sandbox Container - - D->>P: pull_image(supervisor, "missing") - D->>P: create_container(spec with image_volumes) - Note over P: Podman resolves image_volumes at
libpod layer before OCI spec generation - P->>C: Mount supervisor image at /opt/openshell/bin (read-only) - D->>P: start_container - C->>C: entrypoint: /opt/openshell/bin/openshell-sandbox -``` - -The supervisor image from `deploy/docker/Dockerfile.supervisor` provides the -static `openshell-sandbox` binary at `/openshell-sandbox`. -Mounting that image at `/opt/openshell/bin` makes the binary available as -`/opt/openshell/bin/openshell-sandbox`. - -The container spec sets that binary as the entrypoint. This avoids relying on -the sandbox image entrypoint or command, which might otherwise append the -supervisor path as an argument to an image-provided shell. - -## TLS - -When all three Podman TLS paths are set, the driver treats sandbox callbacks as -mTLS callbacks: - -- `OPENSHELL_PODMAN_TLS_CA` -- `OPENSHELL_PODMAN_TLS_CERT` -- `OPENSHELL_PODMAN_TLS_KEY` - -The driver validates that the TLS paths are provided as a complete set. Partial -configuration fails early instead of silently falling back to plaintext. - -When enabled, the driver: - -1. Switches the auto-detected endpoint scheme from `http://` to `https://`. -2. Bind-mounts the client cert files read-only into the container at - `/etc/openshell/tls/client/`. -3. Sets `OPENSHELL_TLS_CA`, `OPENSHELL_TLS_CERT`, and `OPENSHELL_TLS_KEY` to - the container-side paths. - -The supervisor reads these env vars and uses them to establish an mTLS -connection back to the gateway. On SELinux systems, the bind mounts include -Podman's shared relabel option so the container process can read the files. - -The RPM packaging auto-generates a self-signed PKI on first start via -`openshell-gateway generate-certs`. Client certs are placed in the CLI -auto-discovery directory (`~/.config/openshell/gateways/openshell/mtls/`) so -the CLI connects with mTLS without manual configuration. See -`deploy/rpm/CONFIGURATION.md` for the full RPM configuration reference. - -## Network Model - -Sandbox network isolation uses a two-layer approach: a Podman bridge network -for container-to-host communication, and a nested network namespace created by -the supervisor for sandbox process isolation. - -```mermaid -graph TB - subgraph Host - GW["Gateway Server
127.0.0.1:17670"] - PS["Podman Socket"] - end - - subgraph Bridge["Podman Bridge Network (10.89.x.x)"] - subgraph Container["Sandbox Container"] - SV["Supervisor
(root in user ns)"] - subgraph NestedNS["Nested Network Namespace"] - SP["Sandbox Process
(resolved non-root identity)"] - VE2["veth1: 10.200.0.2"] - end - VE1["veth0: 10.200.0.1
(CONNECT proxy)"] - SV --- VE1 - VE1 ---|veth pair| VE2 - end - end - - GW -.->|SSH via supervisor relay
gRPC session| SV - SV -->|gRPC callback via
host.containers.internal| GW - SP -->|all egress via proxy| VE1 +| UID/GID | Pinned non-root workload identity | Same mapped identity | +| Capabilities | Drop all; add none | Drop all; add none | +| Seccomp | Runtime default plus sandbox-installed filters | Runtime default | +| Network | `none`; loopback only | Podman host network | +| Gateway JWT and upstream credentials | Never mounted | Podman secrets | +| User volumes and CDI devices | Workload only | Never mounted | +| Channel | Private named volume, writable | Same volume, read-only | + +Podman creates the namespaces and volume ownership before the workload runs. +Rootless operation uses the operator's Podman service and subordinate-ID +configuration; it does not require adding capabilities to either container. +The supervisor joins the workload's **user namespace only** to preserve UID/GID +mapping for shared-volume access. PID, mount, and network namespaces remain +separate. The channel volume uses shared SELinux relabeling (`:z`). + +Before starting either container, the driver uploads volume-relative archives +directly to the channel and workspace volume destinations. A rootfs upload on a +stopped Podman container does not populate nested named volumes. Restart restores +only the channel bootstrap into the existing channel volume, preserving the +workspace. The workload starts before the supervisor so its user namespace exists +when the supervisor joins it; a stopped supervisor resolves that namespace again +on its next start. + +The runtime must pass the sandbox's unprivileged enforcement probe, including +nested seccomp notification and Landlock. Unsupported runtime defaults fail +closed; do not switch to an unconfined profile or add capabilities. + +## Protected channel and network enforcement + +```text +agent -> openshell-sandbox === authenticated gRPC / private UDS === supervisor -> network + network=none TCP, DNS, control streams policy + gateway JWT ``` -Key points: - -- Bridge network: created by `client.ensure_network()` with DNS enabled. - Containers on the bridge can see each other at L3, but sandbox processes - cannot because they are isolated inside the nested netns. -- Nested netns: the supervisor creates a private `NetworkNamespace` with a veth - pair. Sandbox processes enter this netns via `setns(fd, CLONE_NEWNET)` in the - `pre_exec` hook, forcing ordinary traffic through the CONNECT proxy. -- Policy DNS and transparent TCP: the driver advertises the complete - `policy-dns-transparent-tcp` substrate. For explicit `protocol: tcp` - endpoints, the supervisor installs namespace-local DNS listeners, synthetic - routes, and TCP redirect rules before starting the workload. The container - disables Podman's implicit DNS search suffix so policy DNS evaluates the - exact endpoint name requested by the workload, and asks libc to use the - policy DNS TCP listener to avoid rootless Podman's nested UDP NAT return - path. -- Port publishing: the container spec still requests `host_port: 0` for the - configured SSH port. The gateway SSH tunnel uses the supervisor relay rather - than connecting directly to the published port. -- Host gateway: `host.containers.internal` and `host.openshell.internal` are - injected into `/etc/hosts` so containers can reach services on the gateway - host. Linux defaults to Podman's `host-gateway` resolver. macOS Podman - machine defaults to gvproxy's host-loopback IP, `192.168.127.254`, because - stale Podman machines may fail to resolve `host-gateway`. -- nsenter: the supervisor uses `nsenter --net=` instead of `ip netns exec` for - namespace operations, avoiding the sysfs remount path that fails in rootless - containers. - -See [NETWORKING.md](NETWORKING.md) for the rootless Podman networking deep dive. - -## Supervisor Relay - -Podman follows the same end-to-end contract as the Kubernetes and VM drivers -for the in-container SSH relay: gateway config to `PodmanComputeConfig` to -sandbox environment to supervisor session registration on that path. - -1. `[openshell.drivers.podman].ssh_socket_path` is deserialized into - `PodmanComputeConfig::ssh_socket_path` when the gateway builds the in-process - driver. The field defaults to `/run/openshell/ssh.sock` when omitted. -2. `build_env()` in `container.rs` sets `OPENSHELL_SSH_SOCKET_PATH` to that - value, alongside required vars such as `OPENSHELL_ENDPOINT` and - `OPENSHELL_SANDBOX_ID`. These driver-controlled entries overwrite template - environment variables to prevent spoofing. -3. The supervisor reads `OPENSHELL_SSH_SOCKET_PATH` and uses it for the Unix - socket the gateway's SSH stack bridges to. - -The standalone `openshell-driver-podman` binary sets the same struct field from -`OPENSHELL_SANDBOX_SSH_SOCKET_PATH`. - -## Credential Injection - -Sandboxes authenticate to the gateway via mTLS using client materials bind- -mounted into the container from a Podman secret. No shared per-request secret -is injected as an environment variable. - -| Credential | Mechanism | Visible in `inspect`? | Visible in `/proc//environ`? | -|---|---|---|---| -| mTLS client cert/key | Bind-mounted file paths (`OPENSHELL_TLS_*` env vars point at them) | Yes (paths only) | Yes (paths only) | -| Sandbox identity | Plaintext env var | Yes | Yes | -| gRPC endpoint | Plaintext env var, override-protected | Yes | Yes | -| Supervisor relay socket path | Plaintext env var, override-protected | Yes | Yes | - -The `build_env()` function inserts user-supplied variables first, then -unconditionally overwrites all security-critical variables to prevent spoofing -via sandbox templates: - -- `OPENSHELL_SANDBOX` -- `OPENSHELL_SANDBOX_ID` -- `OPENSHELL_ENDPOINT` -- `OPENSHELL_SSH_SOCKET_PATH` -- `OPENSHELL_CONTAINER_IMAGE` -- `OPENSHELL_MAIN_PROCESS_SPEC` - -## Sandbox Lifecycle - -### Creation Flow - -```mermaid -sequenceDiagram - participant GW as Gateway - participant D as PodmanComputeDriver - participant P as Podman API - - GW->>D: create_sandbox(DriverSandbox) - D->>D: validate name + id - D->>D: validated_container_name() - - D->>P: pull_image(supervisor, "missing") - D->>P: pull_image(sandbox_image, policy) - - D->>P: create_volume(workspace) - Note over D: On failure below, rollback volume - - D->>P: create_container(spec) - alt Conflict (409) - D->>P: remove_volume - D-->>GW: AlreadyExists - end - Note over D: On failure below, rollback container + volume - - D->>P: start_container - D-->>GW: Ok -``` - -Each step rolls back previously-created resources on failure. The Conflict path -cleans up the volume because it is keyed by the new sandbox's ID, not the -conflicting container's ID. - -### Readiness and Health - -The container `healthconfig` marks the sandbox healthy when any of these -signals succeeds: - -- Legacy marker file `/var/run/openshell-ssh-ready`. -- `test -S` on the configured supervisor Unix socket path. -- The prior TCP check for a listener on the in-container SSH port. - -The Unix socket check allows relay-only backend readiness when the supervisor -exposes the socket without the old marker or published-port signal. Omitting -`health_check_interval_secs` disables these Podman/conmon probes, but it does -not bypass public readiness gating: the gateway keeps a backend-ready sandbox -in `Provisioning` with `SupervisorNotConnected` until its supervisor control -session is connected. - -### Deletion Flow - -1. Validate `sandbox_name` and stable `sandbox_id` from `DeleteSandboxRequest`. -2. Best-effort inspect cross-checks the container label when present, but - cleanup remains keyed by the request `sandbox_id`. -3. Best-effort stop, ignoring the stop result. -4. Force-remove the container. -5. Remove workspace volume derived from the request `sandbox_id`, warning on - failure and continuing. - -If the container is already gone during inspect or remove, the driver still -performs idempotent volume cleanup using the request `sandbox_id` and -returns `Ok(false)` for the container-delete result. This prevents leaked -Podman resources after out-of-band container removal or label drift. - -## Configuration - -| Environment Variable | CLI Flag | Default | Description | -|---|---|---|---| -| `OPENSHELL_PODMAN_SOCKET` | `--podman-socket` | Probes known local Podman API sockets and uses the first responsive socket, then falls back to asking the `podman` CLI for the host-side socket. Fails to start if neither finds one. | Podman API Unix socket path. | -| `OPENSHELL_SANDBOX_IMAGE` | `--sandbox-image` | From gateway config | Default OCI image for sandboxes. | -| `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY` | `--sandbox-image-pull-policy` | `if_not_present` | Pull policy: `always`, `if_not_present`, `never`, or `newer`. | -| `OPENSHELL_GRPC_ENDPOINT` | `--grpc-endpoint` | Auto-detected via `host.containers.internal` | Gateway gRPC endpoint for sandbox callbacks. | -| `OPENSHELL_GATEWAY_PORT` | `--gateway-port` | `17670` | Gateway port used for endpoint auto-detection by the standalone binary. | -| `OPENSHELL_NETWORK_NAME` | `--network-name` | `openshell` | Podman bridge network name. | -| `OPENSHELL_PODMAN_HOST_GATEWAY_IP` | `--host-gateway-ip` | empty on Linux, `192.168.127.254` on macOS | Host gateway IP used for sandbox host aliases. Empty uses Podman's `host-gateway` resolver. | -| `OPENSHELL_SANDBOX_SSH_SOCKET_PATH` | `--sandbox-ssh-socket-path` | `/run/openshell/ssh.sock` | Supervisor Unix socket path in `PodmanComputeConfig`. | -| `OPENSHELL_STOP_TIMEOUT` | `--stop-timeout` | `45` | Container stop timeout in seconds. | -| `OPENSHELL_SANDBOX_PIDS_LIMIT` | `--sandbox-pids-limit` | `2048` | Podman cgroup PID limit for sandbox containers. Omission uses OpenShell's `2048` default; explicit `0` is invalid. | -| `OPENSHELL_SUPERVISOR_IMAGE` | `--supervisor-image` | `ghcr.io/nvidia/openshell/supervisor:latest` through the gateway, required standalone | OCI image containing the supervisor binary. | -| `OPENSHELL_PODMAN_TLS_CA` | `--podman-tls-ca` | unset | Host path to the CA certificate mounted for sandbox mTLS. | -| `OPENSHELL_PODMAN_TLS_CERT` | `--podman-tls-cert` | unset | Host path to the client certificate mounted for sandbox mTLS. | -| `OPENSHELL_PODMAN_TLS_KEY` | `--podman-tls-key` | unset | Host path to the client private key mounted for sandbox mTLS. | -| `OPENSHELL_SANDBOX_HTTPS_PROXY` | `--sandbox-https-proxy` | unset | Corporate forward proxy URL for the supervisor's upstream TLS dials, chained with HTTP CONNECT. Credential-free `http://host:port` and `https://host:port` URLs are supported (scheme and port required). For an `https://` proxy the supervisor TLS-wraps the proxy connection, verifying the proxy certificate against the built-in and system roots plus `--sandbox-proxy-ca-bundle`. Plain-HTTP requests always dial directly. | -| `OPENSHELL_SANDBOX_NO_PROXY` | `--sandbox-no-proxy` | unset | Comma-separated `NO_PROXY` list (hostnames, domain suffixes, IPs, CIDRs, each with an optional `:port` qualifier) dialed directly instead of through the corporate proxy. IP/CIDR entries also match hostnames through their validated DNS resolution. | -| `OPENSHELL_SANDBOX_PROXY_AUTH_FILE` | `--sandbox-proxy-auth-file` | unset | Path to a file containing the proxy credentials as `user:pass`. Staged as a root-only Podman secret so credentials never appear in config or container metadata. Requires the insecure-auth acknowledgement below. | -| `OPENSHELL_SANDBOX_PROXY_AUTH_ALLOW_INSECURE` | `--sandbox-proxy-auth-allow-insecure` | unset | Explicit acknowledgement (`true`) that the credential is sent as cleartext Basic auth over the plain-TCP connection to the `http://` proxy. Required when the auth file is set with an `http://` proxy; not required for `https://` proxies (the credential travels inside the verified TLS session) but tolerated if set. Rejected when no auth file is configured. | -| `OPENSHELL_SANDBOX_PROXY_CONNECT_BY_HOSTNAME` | `--sandbox-proxy-connect-by-hostname` | unset | Send the destination hostname in CONNECT requests instead of a validated IP. Last resort for proxies whose ACLs filter on hostnames: the proxy then resolves the name itself, so sandbox SSRF/`allowed_ips` validation no longer binds the connection. | -| `OPENSHELL_PODMAN_USERNS` | `--userns` | unset | User namespace mode for sandbox containers (e.g. `auto`). When unset, containers use the default user namespace. | -| `OPENSHELL_SANDBOX_PROXY_CA_BUNDLE` | `--sandbox-proxy-ca-bundle` | unset | Path (on the gateway host) to a PEM CA bundle trusted for the corporate proxy. Bind-mounted read-only into the sandbox (a CA certificate is not secret). Trusted for the `https://` proxy TLS handshake and, because TLS-intercepting proxies re-sign tunneled certificates, folded into the sandbox trust bundle and upstream verification. Requires a proxy URL; the file must exist and hold at least one certificate. | - -Through the gateway, the same settings are the `https_proxy`, `no_proxy`, -`proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, -and `proxy_ca_bundle` keys under `[openshell.drivers.podman]`; see -`docs/reference/gateway-config.mdx`. - -`provider_spiffe_workload_api_socket` accepts either an absolute host UNIX -Workload API socket, projected through a dedicated read-only mount, or an -explicit container-reachable `tcp:IP:port` endpoint. The driver sets the -supervisor's `OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET` accordingly. -`app_armor_profile` shares the canonical -`RuntimeDefault`, `Unconfined`, or `Localhost/` model with Docker and -Kubernetes. When omitted, the driver sends no override and preserves Podman's -runtime-selected profile. Set `Unconfined` explicitly only when the deployment -requires the supervisor's mount setup to bypass that profile. Explicit confined -choices fail early when Podman reports AppArmor unavailable. - -This is an operator-owned egress boundary: the driver passes the settings on -the supervisor's command line, so sandbox and template environment — and any -`ENV` baked into the sandbox image — cannot override them, and the -conventional `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` variables a sandbox -controls do not steer it. Credentials must be supplied through -`proxy_auth_file`; an inline `user:pass@` in the URL is rejected at startup. - -Basic auth over an `http://` proxy is cleartext on the wire: anyone on the -network path between the sandbox host and the proxy can recover the -credential. Setting `proxy_auth_file` therefore requires -`proxy_auth_allow_insecure = true`; both the driver and the in-container -supervisor reject credentials without that explicit acknowledgement. - -CONNECT requests target a validated resolved IP by default, so the proxy -performs no DNS resolution and the tunnel stays bound to the address that -passed the sandbox's SSRF and `allowed_ips` checks; the hostname still -travels inside the tunnel (TLS SNI, application `Host`). In split-horizon -networks, point the gateway host at the corporate resolver. Set -`proxy_connect_by_hostname = true` only when the proxy's ACLs filter on -hostnames and reject IP CONNECT targets — it re-opens proxy-side DNS -resolution, making the proxy's ACLs the effective egress control. - -## Rootless-Specific Adaptations - -The Podman driver is designed for rootless operation. The following adaptations -matter compared to cluster or rootful runtimes: - -1. subuid/subgid preflight check: on non-macOS hosts, `check_subuid_range()` in - `driver.rs` warns operators if `/etc/subuid` or `/etc/subgid` entries are - missing for the current user. This is not a hard error because some systems - use LDAP or other mechanisms. macOS skips the check because `podman machine` - runs the Podman service inside a Linux VM. -2. cgroups v2 requirement: the driver refuses to start if cgroups v1 is - detected. Rootless Podman requires the unified cgroup hierarchy. -3. `nsenter` for namespace operations: `openshell-sandbox` uses - `nsenter --net=` instead of `ip netns exec` to avoid the sysfs remount path - that requires real `CAP_SYS_ADMIN` in the host user namespace. -4. `DAC_READ_SEARCH` capability: required for the proxy to read - `/proc//fd/` across UIDs within the user namespace. -5. `SETUID` and `SETGID` capabilities: kept from Podman's default capability - set so `drop_privileges()` can call `setuid()` and `setgid()`. -6. `host.containers.internal`: used instead of Docker's `host.docker.internal` - for container-to-host communication. The driver also injects the - OpenShell-owned `host.openshell.internal` alias. -7. Ephemeral port publishing: the SSH compatibility port uses `host_port: 0` - because the bridge network IP is not reliably routable from the host in - rootless mode. -8. tmpfs at `/run/netns`: a private tmpfs lets the supervisor create named - network namespaces via `ip netns add`. - -## Implementation References - -- Gateway integration: `crates/openshell-gateway/src/lib.rs` registers the - driver factory and constructs `PodmanComputeConfig` from the generic server - build context. -- Server configuration: `crates/openshell-server/src/lib.rs` exposes the - backend-agnostic registry and factory context. -- Gateway relay path: `openshell-core` `Config::sandbox_ssh_socket_path` in - `crates/openshell-core/src/config.rs`. -- SSRF mitigation: `crates/openshell-core/src/net.rs`, - `crates/openshell-sandbox/src/proxy.rs`, and - `crates/openshell-server/src/grpc/policy.rs`. -- Sandbox supervisor: `crates/openshell-sandbox/src/` for Landlock, seccomp, - netns, proxy, and relay behavior shared by all drivers. -- Container engine abstraction: `tasks/scripts/container-engine.sh` for - build/deploy support across Docker and Podman. -- Supervisor image build: `deploy/docker/Dockerfile.supervisor`. +The workload has no external interface or published port. Seccomp socket +mediation carries TCP and DNS through one authenticated gRPC connection. +DNS remains supervisor-mediated; general UDP is unsupported. The driver sets +`net.ipv4.ip_unprivileged_port_start=0` in the isolated workload network +namespace so the sandbox's loopback DNS relay can bind port 53 without a +capability. No nftables or nested network namespace setup runs in the sandbox. + +The channel contains the sandbox bootstrap and sandbox-side TLS identity only. +Supervisor private keys and the runtime descriptor stay in the supervisor's private filesystem. +Landlock denies agent access to the top-level `/.openshell` control hierarchy. +The driver verifies Podman's reported `network=none` fence before launch and +restart. `host.containers.internal` and callback networking apply to the +supervisor, not the agent. + +Gateway callbacks use the existing sandbox JWT and optional configured mTLS +bundle. The sandbox/supervisor channel always uses its separate, per-sandbox +mutual TLS material. These are distinct authentication relationships. + +## Identity and trusted binaries + +Both workload and supervisor images are pinned by immutable image ID. The +driver reads account files from a stopped workload-image container; it never +executes the image to resolve an account. Policy identity fields override OCI +`USER` independently. Named users/groups resolve against that image, including +supplementary groups. Root and unresolved identities fail before provisioning. +Images must not prepopulate the reserved `/.openshell` hierarchy; this prevents +image-controlled symlinks from aliasing private control state into user mounts. + +`sandbox_runtime_image` supplies the statically linked musl +`/openshell-sandbox` binary. Podman's read-only image volume delivers it to the +workload; user-namespace modes that cannot use image volumes retain the trusted +binary extraction path. `supervisor_image` supplies the dynamically linked +glibc `/openshell-supervisor` binary outside the workload. Image and request +environment belong to agent children, never the supervisor process. + +## Lifecycle and readiness + +Create builds both stopped containers and stages the private archives before +starting either container. The sandbox does not execute the agent until the +supervisor authenticates and confirms the common boundary contract. Failed +creation removes only containers created by that attempt, then cleans up +driver-owned volumes and secrets. + +Stop retains both containers, workspace, channel, and secrets. Start restores +the consumed sandbox bootstrap from a copy in the supervisor's private +filesystem, verifies the fence, and starts the same pair. A failed supervisor +start stops the workload. Delete removes the companion first, then the workload, +channel, workspace, and driver-owned secrets. User-owned volumes are retained. + +Only workload containers appear in sandbox list/watch results. Readiness uses +the supervisor's private health socket; there is no shell, legacy marker, or +TCP-listener shortcut. Watch reconciliation and supervisor exit/removal events +stop a running workload whose companion is unavailable. The gateway also +requires the authenticated supervisor session before publishing Ready. + +## Mounts, GPUs, and configuration + +User `bind`, `volume`, `tmpfs`, and `image` mounts and CDI GPU selection remain +native Podman features and apply only to the workload. Bind mounts require the +operator's `enable_bind_mounts` opt-in. Reserved control paths and the workspace +root cannot be replaced. User-owned volumes are never created or deleted. + +See [gateway configuration](../../docs/reference/gateway-config.mdx) for +operator settings and [NETWORKING.md](NETWORKING.md) for callback networking. +The supervisor uses Podman's host network and owns the upstream proxy settings. +Omit `health_check_interval_secs` to disable Podman's periodic health command. +Explicit zero is invalid. OpenShell still gates readiness on the supervisor's +authenticated health signal. + +Gateway OTLP configuration continues to export compute-driver spans under the +`openshell-driver-podman` service, preserving gateway trace context. diff --git a/crates/openshell-driver-podman/src/client.rs b/crates/openshell-driver-podman/src/client.rs index 0670a026dd..be5dcea0b9 100644 --- a/crates/openshell-driver-podman/src/client.rs +++ b/crates/openshell-driver-podman/src/client.rs @@ -177,6 +177,8 @@ pub struct ImageInspect { pub struct ImageConfig { #[serde(default)] pub user: String, + #[serde(default)] + pub env: Vec, } /// A container summary returned by the list API. @@ -450,6 +452,93 @@ impl PodmanClient { .await } + pub(crate) async fn create_typed_container( + &self, + spec: &(impl serde::Serialize + Sync), + ) -> Result { + #[derive(serde::Deserialize)] + struct Created { + #[serde(rename = "Id", alias = "ID")] + id: String, + } + let body = + serde_json::to_vec(spec).map_err(|error| PodmanApiError::Json(error.to_string()))?; + let (status, bytes) = self + .request_raw( + hyper::Method::POST, + "/libpod/containers/create", + "application/json", + body.into(), + ) + .await?; + if !status.is_success() { + return Err(error_from_response(status.as_u16(), &bytes)); + } + let created: Created = serde_json::from_slice(&bytes) + .map_err(|error| PodmanApiError::Json(error.to_string()))?; + validate_name(&created.id)?; + Ok(created.id) + } + + pub(crate) async fn copy_to_container( + &self, + name: &str, + destination: &str, + archive: Vec, + ) -> Result<(), PodmanApiError> { + validate_name(name)?; + let (status, bytes) = self + .request_raw( + hyper::Method::PUT, + &format!( + "/libpod/containers/{name}/archive?path={}", + url_encode(destination) + ), + "application/x-tar", + archive.into(), + ) + .await?; + if status.is_success() { + Ok(()) + } else { + Err(error_from_response(status.as_u16(), &bytes)) + } + } + + pub(crate) async fn verify_isolation_fence(&self, id: &str) -> Result<(), PodmanApiError> { + #[derive(serde::Deserialize)] + #[serde(rename_all = "PascalCase")] + struct HostConfig { + network_mode: String, + privileged: bool, + } + #[derive(serde::Deserialize)] + #[serde(rename_all = "PascalCase")] + struct FenceInspect { + host_config: HostConfig, + network_settings: NetworkSettings, + } + validate_name(id)?; + let inspected: FenceInspect = self + .request_json( + hyper::Method::GET, + &format!("/libpod/containers/{id}/json"), + None, + ) + .await?; + if inspected.host_config.network_mode != "none" + || inspected.host_config.privileged + || inspected + .network_settings + .networks + .keys() + .any(|name| name != "none") + { + return Err(PodmanApiError::InvalidInput("sandbox requires an unprivileged container with network mode none and no attached networks".into())); + } + Ok(()) + } + /// Start a container by name or ID. pub async fn start_container(&self, name: &str) -> Result<(), PodmanApiError> { validate_name(name)?; diff --git a/crates/openshell-driver-podman/src/config.rs b/crates/openshell-driver-podman/src/config.rs index 9f139cde94..d0ca11669e 100644 --- a/crates/openshell-driver-podman/src/config.rs +++ b/crates/openshell-driver-podman/src/config.rs @@ -60,9 +60,11 @@ pub struct PodmanComputeConfig { pub host_gateway_ip: String, /// Container stop timeout in seconds (SIGTERM → SIGKILL). pub stop_timeout_secs: u32, - /// OCI image containing the openshell-sandbox supervisor binary. - /// Mounted read-only into sandbox containers at /opt/openshell/bin - /// using Podman's `type=image` mount. + /// OCI image containing the statically linked `openshell-sandbox` binary. + /// The driver extracts the binary from this image into a verified host + /// cache and mounts it read-only into each sandbox container. + pub sandbox_runtime_image: String, + /// OCI image containing the dynamically linked `openshell-supervisor` binary. pub supervisor_image: String, /// Host path to the CA certificate for sandbox mTLS. /// @@ -91,11 +93,11 @@ pub struct PodmanComputeConfig { /// Host path to a SPIFFE Workload API Unix socket exposed to sandbox /// supervisors for provider token exchange client assertions. pub provider_spiffe_workload_api_socket: Option, - /// `AppArmor` confinement requested for sandbox containers. Omission sends - /// no override and preserves Podman's runtime-selected profile. + /// `AppArmor` confinement requested for the workload container. Omission + /// sends no override and preserves Podman's runtime-selected profile. #[serde(default, skip_serializing_if = "Option::is_none")] pub app_armor_profile: Option, - /// Health check interval in seconds for sandbox containers. + /// Health check interval in seconds for supervisor containers. /// /// Podman runs the health check command at this interval to determine /// container readiness. Lower values detect readiness faster but @@ -469,6 +471,7 @@ impl Default for PodmanComputeConfig { network_name: DEFAULT_NETWORK_NAME.to_string(), host_gateway_ip: Self::default_host_gateway_ip(), stop_timeout_secs: DEFAULT_PODMAN_STOP_TIMEOUT_SECS, + sandbox_runtime_image: openshell_core::config::default_sandbox_runtime_image(), supervisor_image: openshell_core::config::default_supervisor_image(), guest_tls_ca: None, guest_tls_cert: None, @@ -503,6 +506,7 @@ impl std::fmt::Debug for PodmanComputeConfig { .field("network_name", &self.network_name) .field("host_gateway_ip", &self.host_gateway_ip) .field("stop_timeout_secs", &self.stop_timeout_secs) + .field("sandbox_runtime_image", &self.sandbox_runtime_image) .field("supervisor_image", &self.supervisor_image) .field("guest_tls_ca", &self.guest_tls_ca) .field("guest_tls_cert", &self.guest_tls_cert) diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index 541cb2f756..4378d0a6bb 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -197,7 +197,7 @@ pub fn short_id(id: &str) -> String { // --------------------------------------------------------------------------- #[derive(Serialize)] -struct ContainerSpec { +pub struct ContainerSpec { name: String, image: String, labels: BTreeMap, @@ -212,18 +212,21 @@ struct ContainerSpec { entrypoint: Vec, command: Vec, user: String, + #[serde(skip_serializing_if = "Vec::is_empty")] + groups: Vec, + #[serde(skip_serializing_if = "Vec::is_empty")] + unsetenv: Vec, cap_drop: Vec, cap_add: Vec, no_new_privileges: bool, - seccomp_profile_path: String, - /// Podman's container create API accepts `AppArmor` through the dedicated - /// `apparmor_profile` `SpecGenerator` field. This is not Docker's - /// `security_opt` representation. #[serde(skip_serializing_if = "Option::is_none")] apparmor_profile: Option, + #[serde(skip_serializing_if = "String::is_empty")] + seccomp_profile_path: String, + #[serde(skip_serializing_if = "BTreeMap::is_empty")] + sysctl: BTreeMap, image_pull_policy: String, - #[serde(skip_serializing_if = "Option::is_none")] - healthconfig: Option, + healthconfig: HealthConfig, resource_limits: ResourceLimits, /// Env-type secrets: map of `ENV_VAR_NAME → secret_name`. /// Podman's libpod `SpecGenerator` uses `secret_env` (a flat map) for @@ -340,8 +343,6 @@ struct SecretMount { struct ResourceLimits { cpu: CpuLimits, memory: MemoryLimits, - // Podman's libpod API consumes the OCI LinuxResources shape. A Docker-style - // scalar PidsLimit is silently ignored and leaves the runtime default. #[serde(skip_serializing_if = "Option::is_none")] pids: Option, } @@ -486,7 +487,7 @@ fn build_env( config: &PodmanComputeConfig, image: &str, oci_user: &str, -) -> BTreeMap { +) -> Result, ComputeDriverError> { let spec = sandbox.spec.as_ref(); let template = spec.and_then(|s| s.template.as_ref()); @@ -511,10 +512,11 @@ fn build_env( user_env.insert(k.clone(), v.clone()); } } - env.extend(user_env.clone()); - if !user_env.is_empty() - && let Ok(json) = serde_json::to_string(&user_env) - { + // User environment belongs exclusively to mediated workload children. In + // particular, never activate loader or policy overrides in the supervisor. + if !user_env.is_empty() { + let json = serde_json::to_string(&user_env) + .map_err(|error| ComputeDriverError::Precondition(error.to_string()))?; env.insert(openshell_core::sandbox_env::USER_ENVIRONMENT.into(), json); } @@ -543,7 +545,7 @@ fn build_env( ); env.insert("OPENSHELL_CONTAINER_IMAGE".into(), image.to_string()); let main_process = openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec(spec) - .expect("main process config serialization cannot fail"); + .map_err(|error| ComputeDriverError::Precondition(error.to_string()))?; env.insert( openshell_core::sandbox_env::MAIN_PROCESS_SPEC.into(), main_process, @@ -615,7 +617,7 @@ fn build_env( ); } - env + Ok(env) } /// Merge labels from the sandbox template with required managed labels. @@ -665,12 +667,14 @@ fn build_resource_limits(sandbox: &DriverSandbox, config: &PodmanComputeConfig) period: DEFAULT_CPU_PERIOD, }, memory: MemoryLimits { limit: mem_bytes }, - pids: config - .sandbox_pids_limit - .map(|limit| PidsLimits { limit: limit.get() }), + pids: podman_pids_limit(config.sandbox_pids_limit).map(|limit| PidsLimits { limit }), } } +fn podman_pids_limit(value: Option) -> Option { + value.map(std::num::NonZeroI64::get) +} + pub fn podman_driver_volume_mount_sources( sandbox: &DriverSandbox, enable_bind_mounts: bool, @@ -880,6 +884,7 @@ fn validate_podman_driver_mounts( } }; driver_mounts::validate_container_mount_target(target)?; + driver_mounts::validate_mount_control_path(target, "/.openshell")?; let normalized_target = driver_mounts::normalize_mount_target(target); if !targets.insert(normalized_target.clone()) { return Err(format!( @@ -951,14 +956,6 @@ fn validate_tmpfs_options(options: &[String]) -> Result, String> { .collect() } -fn podman_apparmor_profile(profile: Option<&openshell_core::AppArmorProfile>) -> Option { - match profile { - None | Some(openshell_core::AppArmorProfile::RuntimeDefault) => None, - Some(openshell_core::AppArmorProfile::Unconfined) => Some("unconfined".to_string()), - Some(openshell_core::AppArmorProfile::Localhost(profile)) => Some(profile.clone()), - } -} - /// Build the Podman container creation JSON spec. #[cfg(test)] #[must_use] @@ -1025,6 +1022,7 @@ pub fn build_container_spec_with_token_and_gpu_devices( } #[allow(clippy::too_many_arguments)] +#[cfg(test)] pub fn build_container_spec_for_image( sandbox: &DriverSandbox, config: &PodmanComputeConfig, @@ -1036,10 +1034,36 @@ pub fn build_container_spec_for_image( supervisor_bin_path: Option<&Path>, tls_secret_names: Option<&[String; 3]>, ) -> Result { + serde_json::to_value(build_base_spec( + sandbox, + config, + token_secret_name, + gpu_device_ids, + requested_image, + image_id, + oci_user, + supervisor_bin_path, + tls_secret_names, + )?) + .map_err(|error| ComputeDriverError::Message(format!("encode Podman spec: {error}"))) +} + +#[allow(clippy::too_many_arguments)] +fn build_base_spec( + sandbox: &DriverSandbox, + config: &PodmanComputeConfig, + token_secret_name: Option<&str>, + gpu_device_ids: Option<&[String]>, + requested_image: &str, + image_id: &str, + oci_user: &str, + supervisor_bin_path: Option<&Path>, + tls_secret_names: Option<&[String; 3]>, +) -> Result { let name = container_name(&sandbox.workspace, &sandbox.name, &sandbox.id); let vol = volume_name(&sandbox.id); - let env = build_env(sandbox, config, requested_image, oci_user); + let env = build_env(sandbox, config, requested_image, oci_user)?; let labels = build_labels(sandbox); let resource_limits = build_resource_limits(sandbox, config); let user_mounts = podman_user_mounts(sandbox, config.enable_bind_mounts) @@ -1080,7 +1104,7 @@ pub fn build_container_spec_for_image( Vec::new() } else { vec![ImageVolume { - source: config.supervisor_image.clone(), + source: config.sandbox_runtime_image.clone(), destination: SUPERVISOR_MOUNT_DIR.into(), rw: false, }] @@ -1098,10 +1122,10 @@ pub fn build_container_spec_for_image( labels, env, volumes, - // Side-load the supervisor binary from a standalone OCI image. + // Side-load the sandbox runtime binary from its standalone OCI image. // Podman resolves image_volumes at the libpod layer, mounting the // image's filesystem at the destination path without starting a - // container from it. The supervisor image exposes the binary at + // container from it. The sandbox runtime image exposes the binary at // /openshell-sandbox, so it appears at /opt/openshell/bin/openshell-sandbox. image_volumes, hostname: format!("sandbox-{}", sandbox.name), @@ -1117,86 +1141,19 @@ pub fn build_container_spec_for_image( // corporate proxy flags follow it; the workload command comes from // the reserved environment variable. command, - // Force the supervisor to run as root (UID 0). Sandbox images may - // set a non-root USER directive (e.g. `USER sandbox`), but the - // supervisor needs root to create network namespaces, set up the - // proxy, and configure Landlock/seccomp. This matches the K8s - // driver's runAsUser: 0. - user: "0:0".into(), - // Podman's default container capability set is already restricted: - // CHOWN DAC_OVERRIDE FOWNER FSETID KILL SETGID SETUID SETPCAP - // NET_BIND_SERVICE SYS_CHROOT SETFCAP - // We add what the supervisor needs and drop what it doesn't. - cap_drop: vec![ - // Not needed: standard file permission bits are sufficient; dropping - // prevents the supervisor from bypassing DAC checks it shouldn't need. - "DAC_OVERRIDE".into(), - // Not needed: the supervisor does not create setuid/setgid executables. - "FSETID".into(), - // Not needed: the supervisor does not bind privileged ports (<1024). - "NET_BIND_SERVICE".into(), - // Not in Podman's default set but explicitly denied in case the image - // or runtime adds it; raw sockets are not required. - "NET_RAW".into(), - // Not needed: the supervisor does not manipulate file capabilities. - "SETFCAP".into(), - // Not needed: the supervisor does not call chroot(). - "SYS_CHROOT".into(), - ], - cap_add: vec![ - // seccomp filter installation, namespace creation, Landlock setup. - "SYS_ADMIN".into(), - // Network namespace veth setup, IP/route configuration. - "NET_ADMIN".into(), - // Reading /proc//exe and ancestor walk for process identity in policy. - "SYS_PTRACE".into(), - // Reading /dev/kmsg for bypass-detection diagnostics. - "SYSLOG".into(), - // Reading /proc//fd/ across UIDs for process identity resolution. - // In rootless Podman the supervisor runs as UID 0 inside a user namespace - // while sandbox processes run as the sandbox user. The kernel's - // proc_fd_permission() calls generic_permission() which denies cross-UID - // access to the dr-x------ fd directory unless this cap is present. - // Without it the proxy cannot determine which binary made each outbound - // connection and all traffic is denied. - "DAC_READ_SEARCH".into(), - // Child setup clears the capability bounding set before exec, which - // requires CAP_SETPCAP in the supervisor until drop_privileges(). - "SETPCAP".into(), - // Forwarding shutdown signals to the canonical workload process - // group after it drops to the sandbox UID requires CAP_KILL. - "KILL".into(), - ], - // SETUID, SETGID, SETPCAP, CHOWN, and FOWNER are intentionally kept from - // Podman's default set and not dropped: - // SETUID/SETGID – drop_privileges(): setuid()/setgid()/initgroups() to the - // sandbox user. In rootless Podman cap_drop:ALL removes them - // from the bounding set even though uid=0 owns the user - // namespace — so we keep them by not dropping them explicitly. - // SETPCAP – drop_privileges(): clears the child capability - // bounding set before the sandbox user execs. - // CHOWN – prepare_filesystem(): chown(path, uid, gid) on newly - // created read_write directories so the sandbox user can - // write to them. - // FOWNER – chown on files where the supervisor is not the owner - // (e.g. pre-existing directories owned by another user). - // - // Disable the container-level seccomp profile. The sandbox supervisor The sandbox supervisor - // installs its own policy-aware BPF seccomp filter at runtime via - // seccompiler (two-phase: clone3 blocker + main filter). The runtime - // filter is more restrictive than Podman's default — it blocks 20+ - // dangerous syscalls and conditionally restricts socket domains based - // on network policy. The filter self-seals by blocking further - // seccomp(SET_MODE_FILTER) calls after installation. - // - // A container-level profile would interfere by blocking the landlock - // and seccomp syscalls the supervisor needs during setup, before it - // locks itself down. + // The paired builder supplies the immutable non-root identity. + user: String::new(), + groups: Vec::new(), + unsetenv: Vec::new(), + cap_drop: vec!["ALL".into()], + cap_add: Vec::new(), no_new_privileges: true, - seccomp_profile_path: "unconfined".into(), - apparmor_profile: podman_apparmor_profile(config.app_armor_profile.as_ref()), + apparmor_profile: None, + // Omission selects the runtime default, never an unconfined profile. + seccomp_profile_path: String::new(), + sysctl: BTreeMap::new(), image_pull_policy: "never".to_string(), - healthconfig: config.health_check_interval_secs.map(|interval_secs| HealthConfig { + healthconfig: HealthConfig { test: vec![ "CMD-SHELL".into(), format!( @@ -1205,11 +1162,14 @@ pub fn build_container_spec_for_image( openshell_core::config::DEFAULT_SSH_PORT ), ], - interval: interval_secs.get() * 1_000_000_000, + interval: config + .health_check_interval_secs + .map_or(10, std::num::NonZeroU64::get) + * 1_000_000_000, timeout: 2_000_000_000, retries: 10, start_period: 5_000_000_000, - }), + }, resource_limits, secret_env: BTreeMap::new(), secrets: { @@ -1409,7 +1369,211 @@ pub fn build_container_spec_for_image( }, }; - Ok(serde_json::to_value(container_spec).expect("ContainerSpec serialization cannot fail")) + Ok(container_spec) +} + +/// Driver-owned inputs for the two independent runtime containers. +pub struct IsolationSpecInput<'a> { + pub sandbox: &'a DriverSandbox, + pub config: &'a PodmanComputeConfig, + pub token_secret: Option<&'a str>, + pub gpu_devices: Option<&'a [String]>, + pub requested_image: &'a str, + pub image_id: &'a str, + pub image_user: &'a str, + pub image_env: &'a [String], + pub supervisor_bin: Option<&'a Path>, + pub tls_secrets: Option<&'a [String; 3]>, + pub identity: &'a openshell_isolation_interface::contract::ResolvedWorkloadIdentity, +} + +pub struct IsolationSpecs { + pub workload: ContainerSpec, + pub supervisor: ContainerSpec, +} + +pub fn build_isolation_specs( + input: IsolationSpecInput<'_>, +) -> Result { + let base = || { + build_base_spec( + input.sandbox, + input.config, + input.token_secret, + input.gpu_devices, + input.requested_image, + input.image_id, + input.image_user, + input.supervisor_bin, + input.tls_secrets, + ) + }; + let mut workload = base()?; + let mut supervisor = base()?; + let user = format!("{}:{}", input.identity.uid, input.identity.gid); + let channel = crate::isolation::channel_volume_name(&input.sandbox.id); + + workload + .labels + .insert(crate::isolation::LABEL_ROLE.into(), "sandbox".into()); + workload.env = BTreeMap::new(); + workload.unsetenv = input + .image_env + .iter() + .filter_map(|entry| entry.split_once('=').map(|(key, _)| key.to_string())) + .collect(); + workload.command = vec![ + "--bootstrap".into(), + crate::isolation::BOOTSTRAP_PATH.into(), + ]; + workload.user.clone_from(&user); + workload.groups = input + .identity + .supplementary_gids + .iter() + .map(ToString::to_string) + .collect(); + workload.cap_drop = vec!["ALL".into()]; + workload.cap_add.clear(); + workload.apparmor_profile = input + .config + .app_armor_profile + .as_ref() + .and_then(openshell_core::config::AppArmorProfile::oci_security_opt) + .and_then(|option| option.strip_prefix("apparmor=").map(str::to_string)); + workload.seccomp_profile_path.clear(); + workload + .sysctl + .insert("net.ipv4.ip_unprivileged_port_start".into(), "0".into()); + workload.netns.nsmode = "none".into(); + workload.networks.clear(); + workload.portmappings.clear(); + workload.hostadd.clear(); + workload.secret_env.clear(); + workload.secrets.clear(); + workload.healthconfig.test = vec!["NONE".into()]; + workload + .mounts + .retain(|mount| !trusted_mount(&mount.destination)); + workload.mounts.push(Mount { + kind: "tmpfs".into(), + source: "tmpfs".into(), + destination: openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_ROOT.into(), + options: vec![ + "rw".into(), + "noexec".into(), + "nosuid".into(), + "nodev".into(), + // Libpod's OCI `mounts` API rejects tmpfs uid/gid options. The + // unprivileged runtime creates the owned material subdirectory. + "mode=0777".into(), + "size=1m".into(), + ], + }); + workload.volumes.push(NamedVolume { + name: channel.clone(), + dest: crate::isolation::CHANNEL_ROOT.into(), + options: vec!["rw".into(), "z".into()], + }); + + supervisor.name = crate::isolation::supervisor_name(&input.sandbox.id); + supervisor + .labels + .insert(crate::isolation::LABEL_ROLE.into(), "supervisor".into()); + supervisor.image.clone_from(&input.config.supervisor_image); + supervisor.entrypoint = vec!["/openshell-supervisor".into()]; + supervisor.command.extend([ + format!( + "--backend-descriptor-file={}", + crate::isolation::RUNTIME_DESCRIPTOR_PATH + ), + format!("--auth-bundle-file={}", crate::isolation::AUTH_BUNDLE_PATH), + "--health-socket-path=/run/openshell/supervisor-health.sock".into(), + ]); + supervisor.env.insert( + openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND.into(), + openshell_sandbox_backend::BACKEND_NAME.into(), + ); + supervisor.user = user; + supervisor.groups = input + .identity + .supplementary_gids + .iter() + .map(ToString::to_string) + .collect(); + supervisor.cap_drop = vec!["ALL".into()]; + supervisor.cap_add.clear(); + supervisor.seccomp_profile_path.clear(); + // The trusted supervisor originates approved egress from the Podman host + // network. Keep it in the caller's user namespace as well: joining the + // workload's user namespace is incompatible with host networking under + // rootless Podman, and the authenticated channel does not require a shared + // user namespace. The workload remains fenced by network=none. + supervisor.userns = Some(UserNS { + nsmode: "host".into(), + value: None, + }); + supervisor.idmappings = None; + supervisor.netns.nsmode = "host".into(); + supervisor.networks.clear(); + supervisor.portmappings.clear(); + supervisor.devices = None; + supervisor.image_volumes.clear(); + supervisor.volumes = vec![NamedVolume { + name: channel, + dest: crate::isolation::CHANNEL_ROOT.into(), + options: vec!["ro".into(), "z".into()], + }]; + supervisor.mounts.retain(|mount| { + trusted_mount(&mount.destination) + && mount.destination != openshell_core::container_paths::NETNS_MOUNT_ROOT + }); + for destination in ["/run", "/var/log", "/tmp"] { + supervisor.mounts.push(Mount { + kind: "tmpfs".into(), + source: "tmpfs".into(), + destination: destination.into(), + options: vec![ + "rw".into(), + "noexec".into(), + "nosuid".into(), + "nodev".into(), + "mode=0777".into(), + "size=64m".into(), + ], + }); + } + for secret in &mut supervisor.secrets { + secret.uid = input.identity.uid; + secret.gid = input.identity.gid; + } + if let Some(interval) = input.config.health_check_interval_secs { + supervisor.healthconfig.test = vec![ + "CMD".into(), + "/openshell-supervisor".into(), + "health".into(), + "--socket".into(), + "/run/openshell/supervisor-health.sock".into(), + ]; + supervisor.healthconfig.interval = interval.get() * 1_000_000_000; + } else { + supervisor.healthconfig.test = vec!["NONE".into()]; + } + Ok(IsolationSpecs { + workload, + supervisor, + }) +} + +fn trusted_mount(destination: &str) -> bool { + matches!( + destination, + TLS_CA_MOUNT_PATH + | TLS_CERT_MOUNT_PATH + | TLS_KEY_MOUNT_PATH + | PROXY_CA_MOUNT_PATH + | PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR + ) || destination == openshell_core::container_paths::NETNS_MOUNT_ROOT } fn provider_spiffe_workload_api_socket_env_value(config: &PodmanComputeConfig) -> Option { @@ -1516,6 +1680,129 @@ mod tests { static ENV_LOCK: std::sync::LazyLock> = std::sync::LazyLock::new(|| std::sync::Mutex::new(())); + #[test] + fn isolated_pair_keeps_privileges_network_and_secrets_out_of_workload() { + let sandbox = DriverSandbox { + id: "pair".into(), + name: "agent".into(), + ..Default::default() + }; + let mut config = PodmanComputeConfig::default(); + config.app_armor_profile = Some(openshell_core::config::AppArmorProfile::Localhost( + "openshell-sandbox".into(), + )); + config.userns = Some("auto".into()); + let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new( + 1000, + 1001, + vec![2000], + "image".into(), + "sha256:image".into(), + ) + .unwrap(); + let env = vec![ + "LD_PRELOAD=/hostile.so".into(), + "HTTP_PROXY=http://bypass".into(), + ]; + let specs = build_isolation_specs(IsolationSpecInput { + sandbox: &sandbox, + config: &config, + token_secret: Some("jwt"), + gpu_devices: None, + requested_image: "image:latest", + image_id: "sha256:image", + image_user: "1000:1001", + image_env: &env, + supervisor_bin: None, + tls_secrets: None, + identity: &identity, + }) + .unwrap(); + for spec in [&specs.workload, &specs.supervisor] { + assert_eq!(spec.user, "1000:1001"); + assert_eq!(spec.groups, vec!["2000"]); + assert_eq!(spec.cap_drop, vec!["ALL"]); + assert!(spec.cap_add.is_empty()); + assert!(spec.seccomp_profile_path.is_empty()); + assert!(spec.no_new_privileges); + } + assert_eq!(specs.workload.netns.nsmode, "none"); + assert_eq!( + specs + .workload + .userns + .as_ref() + .map(|userns| userns.nsmode.as_str()), + Some("auto") + ); + assert_eq!( + specs.workload.apparmor_profile.as_deref(), + Some("openshell-sandbox") + ); + assert_eq!(specs.supervisor.apparmor_profile, None); + let workload_json = serde_json::to_string(&specs.workload).unwrap(); + assert!(workload_json.contains("\"apparmor_profile\":\"openshell-sandbox\"")); + assert_eq!(specs.supervisor.healthconfig.test, vec!["NONE"]); + assert!(specs.workload.networks.is_empty()); + assert!(specs.workload.portmappings.is_empty()); + assert_eq!(specs.supervisor.netns.nsmode, "host"); + assert_eq!( + specs + .supervisor + .userns + .as_ref() + .map(|userns| userns.nsmode.as_str()), + Some("host") + ); + assert!(specs.supervisor.idmappings.is_none()); + assert!(specs.supervisor.networks.is_empty()); + assert!(specs.supervisor.portmappings.is_empty()); + assert!(specs.workload.env.is_empty()); + assert_eq!(specs.workload.unsetenv, vec!["LD_PRELOAD", "HTTP_PROXY"]); + assert!(specs.workload.secrets.is_empty()); + assert!( + specs + .workload + .mounts + .iter() + .all(|mount| !trusted_mount(&mount.destination)) + ); + let supervisor_ca_mount = specs + .workload + .mounts + .iter() + .find(|mount| { + mount.destination == openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_ROOT + }) + .expect("workload supervisor CA mount"); + assert_eq!(supervisor_ca_mount.kind, "tmpfs"); + assert_eq!(supervisor_ca_mount.source, "tmpfs"); + for option in ["rw", "noexec", "nosuid", "nodev", "mode=0777", "size=1m"] { + assert!(supervisor_ca_mount.options.contains(&option.to_string())); + } + assert!( + specs + .workload + .mounts + .iter() + .all(|mount| mount.destination != "/run") + ); + assert_eq!(specs.supervisor.secrets.len(), 1); + assert_eq!(specs.supervisor.secrets[0].source, "jwt"); + assert_eq!(specs.supervisor.secrets[0].uid, 1000); + assert_eq!(specs.supervisor.volumes.len(), 1); + assert!(specs.workload.volumes[0].options.contains(&"rw".into())); + assert!(!specs.workload.volumes[0].options.contains(&"nocopy".into())); + assert!(specs.supervisor.volumes[0].options.contains(&"ro".into())); + assert!(specs.supervisor.volumes[0].options.contains(&"z".into())); + assert!( + !specs.supervisor.volumes[0] + .options + .contains(&"nocopy".into()) + ); + assert_eq!(specs.supervisor.entrypoint, vec!["/openshell-supervisor"]); + } + fn json_struct(value: Value) -> prost_types::Struct { let Value::Object(object) = value else { panic!("expected JSON object"); @@ -1564,7 +1851,7 @@ mod tests { } #[test] - fn container_spec_applies_resource_limits() { + fn container_spec_applies_cpu_and_memory_limits() { use openshell_core::proto::compute::v1::{ DriverResourceRequirements, DriverSandboxSpec, DriverSandboxTemplate, }; @@ -1581,22 +1868,18 @@ mod tests { }), ..Default::default() }); - let mut config = test_config(); - config.sandbox_pids_limit = std::num::NonZeroI64::new(2048); - let spec = build_container_spec(&sandbox, &config); + let config = test_config(); + let limits = build_resource_limits(&sandbox, &config); + assert_eq!(limits.cpu.quota, 50_000); + assert_eq!(limits.memory.limit, 2 * 1024 * 1024 * 1024); assert_eq!( - spec["resource_limits"]["cpu"]["quota"].as_u64(), - Some(50_000) - ); - assert_eq!( - spec["resource_limits"]["memory"]["limit"].as_u64(), - Some(2 * 1024 * 1024 * 1024) - ); - assert_eq!( - spec["resource_limits"]["pids"]["limit"].as_i64(), - Some(2048) + limits.pids.as_ref().map(|pids| pids.limit), + openshell_core::config::default_sandbox_pids_limit().map(std::num::NonZeroI64::get) ); + let serialized = serde_json::to_string(&limits).unwrap(); + assert!(serialized.contains("\"pids\":{\"limit\":2048}")); + assert!(!serialized.contains("PidsLimit")); } #[test] @@ -1604,43 +1887,9 @@ mod tests { let sandbox = test_sandbox("test-id", "test-name"); let mut config = test_config(); config.sandbox_pids_limit = None; - let spec = build_container_spec(&sandbox, &config); - - assert!(spec["resource_limits"].get("pids").is_none()); - } - - #[test] - fn container_spec_uses_podman_apparmor_profile_field() { - let sandbox = test_sandbox("test-id", "test-name"); - - for (profile, expected) in [ - (openshell_core::AppArmorProfile::Unconfined, "unconfined"), - ( - openshell_core::AppArmorProfile::Localhost("openshell-supervisor".to_string()), - "openshell-supervisor", - ), - ] { - let mut config = test_config(); - config.app_armor_profile = Some(profile); - let spec = build_container_spec(&sandbox, &config); - - assert_eq!(spec["apparmor_profile"].as_str(), Some(expected)); - assert!(spec.get("security_opt").is_none()); - } - } - - #[test] - fn container_spec_omits_podman_apparmor_profile_for_runtime_default() { - let sandbox = test_sandbox("test-id", "test-name"); - - for profile in [None, Some(openshell_core::AppArmorProfile::RuntimeDefault)] { - let mut config = test_config(); - config.app_armor_profile = profile; - let spec = build_container_spec(&sandbox, &config); + let limits = build_resource_limits(&sandbox, &config); - assert!(spec.get("apparmor_profile").is_none()); - assert!(spec.get("security_opt").is_none()); - } + assert!(limits.pids.is_none()); } #[test] @@ -1681,7 +1930,8 @@ mod tests { container["env"]["OPENSHELL_CONTAINER_IMAGE"].as_str(), Some("registry.example/app:latest") ); - assert_eq!(container["user"].as_str(), Some("0:0")); + // Only the paired builder materializes the immutable non-root user. + assert_eq!(container["user"].as_str(), Some("")); assert_eq!(container["image_pull_policy"].as_str(), Some("never")); assert_eq!(container["dns_search"], serde_json::json!([])); assert_eq!(container["dns_option"], serde_json::json!([])); @@ -1921,66 +2171,25 @@ mod tests { } #[test] - fn container_spec_includes_required_capabilities() { + fn container_spec_defaults_drop_capabilities_and_keep_runtime_seccomp() { let sandbox = test_sandbox("test-id", "test-name"); let config = test_config(); - let spec = build_container_spec(&sandbox, &config); - - let added: Vec<&str> = spec["cap_add"] - .as_array() - .expect("cap_add should be an array") - .iter() - .filter_map(|v| v.as_str()) - .collect(); - assert!(added.contains(&"SYS_ADMIN"), "missing SYS_ADMIN"); - assert!(added.contains(&"NET_ADMIN"), "missing NET_ADMIN"); - assert!(added.contains(&"SYS_PTRACE"), "missing SYS_PTRACE"); - assert!(added.contains(&"SYSLOG"), "missing SYSLOG"); - assert!( - added.contains(&"DAC_READ_SEARCH"), - "missing DAC_READ_SEARCH" - ); - assert!(added.contains(&"SETPCAP"), "missing SETPCAP"); - assert!(added.contains(&"KILL"), "missing KILL"); - - // SETUID and SETGID are NOT in cap_add — they remain available from the - // default bounding set because we no longer use cap_drop:ALL. Verify they - // are also not explicitly dropped. Similarly SETPCAP, CHOWN and FOWNER - // must not be dropped because child setup clears the bounding set and - // prepare_filesystem() calls chown() on newly created read_write - // directories before the supervisor drops privileges. - let dropped: Vec<&str> = spec["cap_drop"] - .as_array() - .expect("cap_drop should be an array") - .iter() - .filter_map(|v| v.as_str()) - .collect(); - assert!(!dropped.contains(&"SETUID"), "SETUID must not be dropped"); - assert!(!dropped.contains(&"SETGID"), "SETGID must not be dropped"); - assert!( - dropped.contains(&"NET_BIND_SERVICE"), - "NET_BIND_SERVICE must stay dropped; policy DNS binds an unprivileged port" - ); - assert!( - !dropped.contains(&"CHOWN"), - "CHOWN must not be dropped (needed for prepare_filesystem chown)" - ); - assert!( - !dropped.contains(&"FOWNER"), - "FOWNER must not be dropped (needed for chown on non-owned files)" - ); - assert!( - !dropped.contains(&"SETPCAP"), - "SETPCAP must not be dropped (needed for child bounding-set clear)" - ); - assert!( - !dropped.contains(&"KILL"), - "KILL must not be dropped (needed to signal the sandbox workload on shutdown)" - ); - assert!( - !dropped.contains(&"ALL"), - "must not use cap_drop:ALL in rootless Podman" - ); + let spec = build_base_spec( + &sandbox, + &config, + None, + None, + "image", + "sha256:image", + "", + None, + None, + ) + .unwrap(); + assert_eq!(spec.cap_drop, vec!["ALL"]); + assert!(spec.cap_add.is_empty()); + assert!(spec.seccomp_profile_path.is_empty()); + assert!(spec.no_new_privileges); } #[test] @@ -2016,8 +2225,7 @@ mod tests { #[test] fn container_spec_healthcheck_accepts_supervisor_socket() { let sandbox = test_sandbox("test-id", "test-name"); - let mut config = test_config(); - config.health_check_interval_secs = std::num::NonZeroU64::new(10); + let config = test_config(); let spec = build_container_spec(&sandbox, &config); let healthcheck = spec["healthconfig"]["test"] @@ -2033,13 +2241,6 @@ mod tests { ); } - #[test] - fn container_spec_omits_healthcheck_when_disabled() { - let sandbox = test_sandbox("test-id", "test-name"); - let spec = build_container_spec(&sandbox, &test_config()); - assert!(spec.get("healthconfig").is_none()); - } - #[test] fn container_spec_healthcheck_interval_from_config() { let sandbox = test_sandbox("test-id", "test-name"); @@ -2476,7 +2677,7 @@ mod tests { } #[test] - fn container_spec_includes_supervisor_image_volume() { + fn container_spec_includes_sandbox_runtime_image_volume() { let sandbox = test_sandbox("test-id", "test-name"); let config = test_config(); let spec = build_container_spec(&sandbox, &config); @@ -2493,8 +2694,8 @@ mod tests { let vol = &image_volumes[0]; assert_eq!( vol["source"].as_str(), - Some(openshell_core::config::default_supervisor_image().as_str()), - "image volume source should be the supervisor image" + Some(openshell_core::config::default_sandbox_runtime_image().as_str()), + "image volume source should be the sandbox runtime image" ); assert_eq!( vol["destination"].as_str(), @@ -2579,9 +2780,9 @@ mod tests { let image_volumes = spec["image_volumes"] .as_array() .expect("image_volumes should be an array"); - let expected_supervisor = openshell_core::config::default_supervisor_image(); + let expected_sandbox_runtime = openshell_core::config::default_sandbox_runtime_image(); assert!(image_volumes.iter().any(|volume| { - volume["source"].as_str() == Some(expected_supervisor.as_str()) + volume["source"].as_str() == Some(expected_sandbox_runtime.as_str()) && volume["destination"].as_str() == Some("/opt/openshell/bin") })); assert!(image_volumes.iter().any(|volume| { @@ -2930,6 +3131,27 @@ mod tests { assert!(err.to_string().contains("reserved OpenShell path")); } + #[test] + fn user_mounts_cannot_replace_private_channel_hierarchy() { + for target in [ + "/.openshell", + "/.openshell/channel", + "/.openshell/channel/sandbox", + "/.openshell/supervisor", + ] { + let mount = PodmanDriverMountConfig::Volume { + source: "user-owned".into(), + target: target.into(), + read_only: false, + subpath: None, + }; + assert!( + validate_podman_driver_mounts(&[mount], false).is_err(), + "{target}" + ); + } + } + #[test] fn container_spec_uses_configured_host_gateway_ip() { let sandbox = test_sandbox("test-id", "test-name"); @@ -3367,7 +3589,7 @@ mod tests { !image_volumes .iter() .any(|v| v["destination"].as_str() == Some(SUPERVISOR_MOUNT_DIR)), - "supervisor image volume should not be present when bind path is provided" + "sandbox runtime image volume should not be present when bind path is provided" ); let mounts = spec["mounts"] @@ -3401,7 +3623,7 @@ mod tests { image_volumes .iter() .any(|v| v["destination"].as_str() == Some(SUPERVISOR_MOUNT_DIR)), - "supervisor image volume should be present by default" + "sandbox runtime image volume should be present by default" ); let mounts = spec["mounts"] diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index 2852c43328..62142a3157 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -13,7 +13,7 @@ use crate::watcher::{ use openshell_core::ComputeDriverError; use openshell_core::config::CDI_GPU_DEVICE_ALL; use openshell_core::driver_utils::{ - GatewayCallbackTopology, SUPERVISOR_IMAGE_BINARY_PATH, extract_first_tar_entry, + GatewayCallbackRoute, SANDBOX_RUNTIME_IMAGE_BINARY_PATH, extract_first_tar_entry, gateway_callback_endpoint, supervisor_image_should_refresh, temp_extract_container_name, validate_linux_elf_binary, write_cache_binary_atomic, }; @@ -30,6 +30,7 @@ use openshell_core::proto::compute::v1::{ GpuResourceCapabilities, GpuResourceRequirements, MemoryResourceCapabilities, ResourceCapabilities, gateway_listener_requirement::Selector, }; +use std::collections::HashMap; #[cfg(target_os = "linux")] use std::net::{IpAddr, SocketAddr}; use std::path::{Path, PathBuf}; @@ -41,6 +42,24 @@ use url::Url; const STOP_COMPLETION_POLL_INTERVAL: Duration = Duration::from_millis(50); const STOP_COMPLETION_TIMEOUT_HEADROOM: Duration = Duration::from_secs(5); +fn decode_launch_authentication( + encoded: &[u8], +) -> Result { + let authentication = + serde_json::from_slice::(encoded) + .map_err(|error| { + ComputeDriverError::Precondition(format!( + "decode Podman sandbox launch authentication: {error}" + )) + })?; + authentication.validate().map_err(|error| { + ComputeDriverError::Precondition(format!( + "validate Podman sandbox launch authentication: {error}" + )) + })?; + Ok(authentication) +} + impl From for ComputeDriverError { fn from(value: PodmanApiError) -> Self { match value { @@ -437,10 +456,10 @@ impl PodmanComputeDriver { } // Auto-detect the gRPC callback endpoint before deciding whether this - // topology needs the Podman bridge gateway address. + // callback route needs the Podman bridge gateway address. if config.grpc_endpoint.is_empty() { config.grpc_endpoint = gateway_callback_endpoint( - GatewayCallbackTopology::Podman, + GatewayCallbackRoute::Podman, config.gateway_port, config.tls_enabled(), ); @@ -452,7 +471,7 @@ impl PodmanComputeDriver { } // Ensure the bridge network exists. Inspect its gateway only when the - // selected Linux callback topology will bind that exact address. + // selected Linux callback route will bind that exact address. client.ensure_network(&config.network_name).await?; let uses_local_callback_alias = Url::parse(&config.grpc_endpoint) .ok() @@ -777,13 +796,29 @@ impl PodmanComputeDriver { "Creating sandbox container" ); - let (image, immutable_image_id, image_user) = async { + let (image, immutable_image_id, image_user, image_env) = async { let phase_status = openshell_otel::ErrorStatusGuard::current(); let result = async { - // The supervisor binary is shipped in a standalone OCI image and - // mounted into sandbox containers via Podman's type=image mount. + // The sandbox runtime is shipped in a standalone OCI image. + // The driver extracts and verifies its binary before bind-mounting + // it into the workload container. + let sandbox_runtime_pull_policy = + runtime_image_pull_policy(&self.config.sandbox_runtime_image); + info!( + image = %self.config.sandbox_runtime_image, + policy = sandbox_runtime_pull_policy, + "Ensuring sandbox runtime image" + ); + self.client + .pull_image( + &self.config.sandbox_runtime_image, + sandbox_runtime_pull_policy, + ) + .await + .map_err(ComputeDriverError::from)?; + let supervisor_pull_policy = - supervisor_image_pull_policy(&self.config.supervisor_image); + runtime_image_pull_policy(&self.config.supervisor_image); info!( image = %self.config.supervisor_image, policy = supervisor_pull_policy, @@ -837,7 +872,8 @@ impl PodmanComputeDriver { .map_err(ComputeDriverError::from)?; } - Ok((image.to_string(), inspected_image.id, image_user)) + let image_env = inspected_image.config.as_ref().map_or_else(Vec::new, |config| config.env.clone()); + Ok((image.to_string(), inspected_image.id, image_user, image_env)) } .await; phase_status.finish(result) @@ -856,6 +892,32 @@ impl PodmanComputeDriver { // content at startup. validate_sandbox_proxy_ca_bundle(&self.config).await?; + let identity = self + .resolve_workload_identity(sandbox, &immutable_image_id, &image_user) + .await?; + let channel_volume = crate::isolation::channel_volume_name(&sandbox.id); + let mut runtime_config = self.config.clone(); + runtime_config.sandbox_runtime_image = self + .client + .inspect_image(&self.config.sandbox_runtime_image) + .await? + .id; + if runtime_config.sandbox_runtime_image.is_empty() { + return Err(ComputeDriverError::Precondition( + "sandbox runtime image inspection returned no immutable image ID".into(), + )); + } + runtime_config.supervisor_image = self + .client + .inspect_image(&self.config.supervisor_image) + .await? + .id; + if runtime_config.supervisor_image.is_empty() { + return Err(ComputeDriverError::Precondition( + "supervisor image inspection returned no immutable image ID".into(), + )); + } + // Create workspace volume and per-sandbox token secret. let (token_secret_name, proxy_auth_secret_name) = async { let phase_status = openshell_otel::ErrorStatusGuard::current(); @@ -901,6 +963,7 @@ impl PodmanComputeDriver { // Clean up the volume and both per-sandbox secrets on any failure past // this point. let cleanup_created = || async { + let _ = self.client.remove_volume(&channel_volume).await; let _ = self.client.remove_volume(&vol_name).await; if let Some(secret) = token_secret_name.as_deref() { cleanup_sandbox_token_secret(&self.client, secret).await; @@ -911,7 +974,7 @@ impl PodmanComputeDriver { }; // Prepare and create the container. - let tls_secret_names = async { + async { let phase_status = openshell_otel::ErrorStatusGuard::current(); let result = async { let gpu_devices = match self.resolve_gpu_cdi_devices( @@ -925,22 +988,19 @@ impl PodmanComputeDriver { return Err(e); } }; - let supervisor_bin_path = if userns_needs_extraction(self.config.userns.as_deref()) - { - match extract_supervisor_bin(&self.client, &self.config).await { + // Podman's image-volume support varies across libpod/runtime + // combinations. Always use the verified extraction cache so + // workload startup does not depend on type=image mounts. + let supervisor_bin_path = + match extract_sandbox_bin(&self.client, &runtime_config).await { Ok(path) => Some(path), Err(e) => { cleanup_created().await; return Err(e); } - } - } else { - None - }; + }; - let tls_secret_names = if userns_remaps_uids(self.config.userns.as_deref()) - && self.config.tls_enabled() - { + let tls_secret_names = if self.config.tls_enabled() { let names = container::tls_secret_names(&sandbox.id); if let Err(e) = create_tls_secrets(&self.client, &self.config, &names).await { cleanup_created().await; @@ -958,32 +1018,90 @@ impl PodmanComputeDriver { } }; - let spec = match container::build_container_spec_for_image( + let specs = container::build_isolation_specs(container::IsolationSpecInput { sandbox, - &self.config, - token_secret_name.as_deref(), - gpu_devices.as_deref(), - &image, - &immutable_image_id, - &image_user, - supervisor_bin_path.as_deref(), - tls_secret_names.as_ref(), - ) { + config: &runtime_config, + token_secret: token_secret_name.as_deref(), + gpu_devices: gpu_devices.as_deref(), + requested_image: &image, + image_id: &immutable_image_id, + image_user: &image_user, + image_env: &image_env, + supervisor_bin: supervisor_bin_path.as_deref(), + tls_secrets: tls_secret_names.as_ref(), + identity: &identity, + }); + let specs = match specs { Ok(spec) => spec, Err(e) => { cleanup_all().await; return Err(e); } }; - match self.client.create_container(&spec).await { - Ok(_) => Ok(tls_secret_names), - Err(PodmanApiError::Conflict(_)) => { - cleanup_all().await; - Err(ComputeDriverError::AlreadyExists) + let mut created_workload = None; + let mut created_supervisor = None; + let create_result = async { + self.client.create_volume(&channel_volume).await?; + let workload_id = self.client.create_typed_container(&specs.workload).await?; + created_workload = Some(workload_id.clone()); + self.client.verify_isolation_fence(&workload_id).await?; + let child_env = podman_child_environment(sandbox, &image_env); + let launch_authentication = sandbox + .spec + .as_ref() + .filter(|spec| !spec.launch_authentication.is_empty()) + .ok_or_else(|| { + ComputeDriverError::Precondition( + "Podman sandbox launch authentication is required".to_string(), + ) + }) + .and_then(|spec| { + decode_launch_authentication(&spec.launch_authentication) + })?; + let archives = crate::isolation::bootstrap_archives( + &sandbox.id, + &workload_id, + &uuid::Uuid::new_v4().to_string(), + &identity, + child_env, + &launch_authentication, + )?; + self.client + .copy_to_container( + &workload_id, + crate::isolation::CHANNEL_ROOT, + archives.channel, + ) + .await?; + self.client + .copy_to_container(&workload_id, "/sandbox", archives.workspace) + .await?; + let supervisor_id = self + .client + .create_typed_container(&specs.supervisor) + .await?; + created_supervisor = Some(supervisor_id.clone()); + self.client + .copy_to_container(&supervisor_id, "/", archives.supervisor) + .await?; + // Start the sandbox only after both containers and their + // bootstrap material exist. It keeps the agent stopped + // until the authenticated supervisor confirms the boundary. + self.client.start_container(&workload_id).await?; + self.client.start_container(&supervisor_id).await?; + Ok::<(), ComputeDriverError>(()) + } + .await; + if create_result.is_err() { + for id in [created_supervisor, created_workload].into_iter().flatten() { + let _ = self.client.remove_container(&id, 0).await; } + } + match create_result { + Ok(()) => Ok(()), Err(e) => { cleanup_all().await; - Err(ComputeDriverError::from(e)) + Err(e) } } } @@ -998,44 +1116,6 @@ impl PodmanComputeDriver { )) .await?; - let cleanup_all = || async { - cleanup_created().await; - if let Some(names) = &tls_secret_names { - cleanup_tls_secrets(&self.client, names).await; - } - }; - - // Start container. - let start_result = async { - let phase_status = openshell_otel::ErrorStatusGuard::current(); - let result = self - .client - .start_container(&name) - .await - .map_err(ComputeDriverError::from); - phase_status.finish(result) - } - .instrument(tracing::info_span!( - "podman.start_container", - otel.name = "podman.start_container", - otel.status_code = tracing::field::Empty, - container.name = %name, - )) - .await; - if let Err(e) = start_result { - warn!( - sandbox_name = %sandbox.name, - error = %e, - "Failed to start container; cleaning up" - ); - let _ = self - .client - .remove_container(&name, self.config.stop_timeout_secs) - .await; - cleanup_all().await; - return Err(e); - } - info!( sandbox_id = %sandbox.id, sandbox_name = %sandbox.name, @@ -1045,7 +1125,65 @@ impl PodmanComputeDriver { span_status.finish(Ok(())) } - /// Find the Podman container ID for a sandbox by its sandbox ID using label lookup. + /// Resolve image accounts without executing any image-supplied program. + async fn resolve_workload_identity( + &self, + sandbox: &DriverSandbox, + image: &str, + image_user: &str, + ) -> Result + { + #[derive(serde::Serialize)] + struct InspectionSpec<'a> { + name: String, + image: &'a str, + } + // Inspect a stopped, unexecuted container pinned to the final image ID. + let id = self + .client + .create_typed_container(&InspectionSpec { + name: format!("openshell-identity-{}", uuid::Uuid::new_v4()), + image, + }) + .await?; + let result = + async { + // Do not let image-controlled symlinks alias the protected channel + // into an agent-readable subtree before Podman mounts it. + match self.client.copy_from_container(&id, "/.openshell").await { + Err(PodmanApiError::NotFound(_)) => {} + Ok(_) => return Err(ComputeDriverError::Precondition( + "workload images must not prepopulate the reserved /.openshell hierarchy" + .into(), + )), + Err(error) => return Err(error.into()), + } + let mut accounts = Vec::new(); + for path in ["/etc/passwd", "/etc/group"] { + let content = match self.client.copy_from_container(&id, path).await { + Ok(archive) => extract_first_tar_entry(&archive) + .map_err(ComputeDriverError::Precondition)?, + Err(PodmanApiError::NotFound(_)) => Vec::new(), + Err(error) => return Err(error.into()), + }; + accounts.push(content); + } + let [passwd, group] = accounts.as_slice() else { + return Err(ComputeDriverError::Precondition( + "image account inspection was incomplete".into(), + )); + }; + crate::isolation::resolve_identity(sandbox, image, image_user, passwd, group) + } + .await; + let cleanup = self.client.remove_container(&id, 0).await; + if let Err(error) = cleanup { + warn!(container = %id, %error, "Failed to remove stopped identity inspection container"); + } + result + } + + /// Find only the workload, never its supervisor companion. async fn find_container_id( &self, sandbox_id: &str, @@ -1060,7 +1198,11 @@ impl PodmanComputeDriver { let id_filter = format!("{LABEL_SANDBOX_ID}={sandbox_id}"); let entries = self .client - .list_containers(&[LABEL_MANAGED_FILTER, &id_filter]) + .list_containers(&[ + LABEL_MANAGED_FILTER, + &id_filter, + crate::isolation::WORKLOAD_FILTER, + ]) .await .map_err(ComputeDriverError::from)?; Ok(entries.into_iter().next()) @@ -1113,6 +1255,15 @@ impl PodmanComputeDriver { .await? .ok_or(ComputeDriverError::NotFound)?; let container_id = container.id; + let supervisor = crate::isolation::supervisor_name(sandbox_id); + match self + .client + .stop_container(&supervisor, self.config.stop_timeout_secs) + .await + { + Ok(()) | Err(PodmanApiError::NotFound(_)) => {} + Err(error) => return Err(error.into()), + } if container.state == "stopping" { let result = async { let finished_at = self @@ -1162,21 +1313,64 @@ impl PodmanComputeDriver { /// Start a previously stopped sandbox container. #[tracing::instrument( name = "podman.start_sandbox", - skip(self), + skip_all, fields( otel.name = "podman.start_sandbox", otel.status_code = tracing::field::Empty, sandbox.id = %sandbox_id, ) )] - pub async fn start_sandbox(&self, sandbox_id: &str) -> Result<(), ComputeDriverError> { + pub async fn start_sandbox( + &self, + sandbox_id: &str, + generation_id: &str, + encoded_authentication: &[u8], + ) -> Result<(), ComputeDriverError> { let span_status = openshell_otel::ErrorStatusGuard::current(); + let generation = openshell_core::sandbox_generation::SandboxGenerationId::parse( + generation_id.to_string(), + ) + .map_err(|error| ComputeDriverError::InvalidArgument(error.to_string()))?; + let launch_authentication = decode_launch_authentication(encoded_authentication)?; let container = self .find_container(sandbox_id) .await? .ok_or(ComputeDriverError::NotFound)?; if container.state == "running" { - return span_status.finish(Ok(())); + let supervisor = self + .client + .inspect_container(&crate::isolation::supervisor_name(sandbox_id)) + .await; + if supervisor + .as_ref() + .is_ok_and(|inspect| inspect.state.running) + { + let archive = self + .client + .copy_from_container( + &crate::isolation::supervisor_name(sandbox_id), + crate::isolation::RESTART_METADATA_PATH, + ) + .await?; + let bundle = + extract_first_tar_entry(&archive).map_err(ComputeDriverError::Precondition)?; + let metadata = crate::isolation::restart_metadata_from_slice(&bundle)?; + if metadata.generation == generation.as_str() && encoded_authentication.is_empty() { + return span_status.finish(Ok(())); + } + if metadata.generation != generation.as_str() { + return span_status.finish(Err(ComputeDriverError::Precondition(format!( + "Podman sandbox is already running generation {}", + metadata.generation + )))); + } + // A non-empty bundle for the same generation comes from + // gateway startup recovery. Restart both containers so the + // in-memory launch session changes atomically on both sides. + } + self.client.stop_container(&container.id, 0).await?; + self.wait_for_container_stopped(sandbox_id, &container.id) + .await?; } let container_id = container.id; info!(sandbox_id = %sandbox_id, container = %container_id, "Starting sandbox container"); @@ -1193,11 +1387,47 @@ impl PodmanComputeDriver { .map_err(ComputeDriverError::from)?; self.lifecycle_event_fences .record_previous_exit(sandbox_id, previous.state.finished_at.as_deref()); - let result = self - .client - .start_container(&container_id) - .await - .map_err(ComputeDriverError::from); + let result = async { + let supervisor = crate::isolation::supervisor_name(sandbox_id); + self.client + .stop_container(&supervisor, self.config.stop_timeout_secs) + .await?; + self.wait_for_container_stopped(sandbox_id, &supervisor) + .await?; + let archive = self + .client + .copy_from_container(&supervisor, crate::isolation::RESTART_METADATA_PATH) + .await?; + let bundle = + extract_first_tar_entry(&archive).map_err(ComputeDriverError::Precondition)?; + let restart_metadata = crate::isolation::restart_metadata_from_slice(&bundle)?; + let archives = crate::isolation::bootstrap_archives( + sandbox_id, + &container_id, + generation.as_str(), + &restart_metadata.workload_identity, + restart_metadata.child_env, + &launch_authentication, + )?; + self.client + .copy_to_container( + &container_id, + crate::isolation::CHANNEL_ROOT, + archives.channel, + ) + .await?; + self.client + .copy_to_container(&supervisor, "/", archives.supervisor) + .await?; + self.client.verify_isolation_fence(&container_id).await?; + self.client.start_container(&container_id).await?; + if let Err(error) = self.client.start_container(&supervisor).await { + let _ = self.client.stop_container(&container_id, 0).await; + return Err(error.into()); + } + Ok(()) + } + .await; span_status.finish(result) } @@ -1219,6 +1449,25 @@ impl PodmanComputeDriver { )); } + let supervisor = crate::isolation::supervisor_name(sandbox_id); + match self + .client + .remove_container(&supervisor, self.config.stop_timeout_secs) + .await + { + Ok(()) | Err(PodmanApiError::NotFound(_)) => {} + Err(error) => return Err(error.into()), + } + match self + .client + .remove_volume(&crate::isolation::channel_volume_name(sandbox_id)) + .await + { + Ok(()) | Err(PodmanApiError::NotFound(_)) => {} + // The workload still owns the volume until its removal below. + Err(error) => debug!(%error, "Channel volume is still attached to workload"), + } + let Some(container_id) = self.find_container_id(sandbox_id).await? else { debug!(sandbox_id = %sandbox_id, "Sandbox container not found (already deleted)"); let vol = container::volume_name(sandbox_id); @@ -1252,6 +1501,13 @@ impl PodmanComputeDriver { }; // Remove workspace volume. + if let Err(error) = self + .client + .remove_volume(&crate::isolation::channel_volume_name(sandbox_id)) + .await + { + warn!(%sandbox_id, %error, "Failed to remove private channel volume"); + } let vol = container::volume_name(sandbox_id); if let Err(e) = self.client.remove_volume(&vol).await { warn!( @@ -1278,7 +1534,11 @@ impl PodmanComputeDriver { let id_filter = format!("{LABEL_SANDBOX_ID}={sandbox_id}"); let entries = self .client - .list_containers(&[LABEL_MANAGED_FILTER, &id_filter]) + .list_containers(&[ + LABEL_MANAGED_FILTER, + &id_filter, + crate::isolation::WORKLOAD_FILTER, + ]) .await .map_err(ComputeDriverError::from)?; Ok(!entries.is_empty()) @@ -1292,16 +1552,18 @@ impl PodmanComputeDriver { let id_filter = format!("{LABEL_SANDBOX_ID}={sandbox_id}"); let entries = self .client - .list_containers(&[LABEL_MANAGED_FILTER, &id_filter]) + .list_containers(&[ + LABEL_MANAGED_FILTER, + &id_filter, + crate::isolation::WORKLOAD_FILTER, + ]) .await .map_err(ComputeDriverError::from)?; let Some(entry) = entries.first() else { return Ok(None); }; if entry.state == "running" { - Ok(self - .client - .inspect_container(&entry.id) + Ok(watcher::inspect_workload(&self.client, &entry.id) .await .ok() .and_then(|inspect| driver_sandbox_from_inspect(&inspect)) @@ -1318,7 +1580,7 @@ impl PodmanComputeDriver { pub async fn list_sandboxes(&self) -> Result, ComputeDriverError> { let entries = self .client - .list_containers(&[LABEL_MANAGED_FILTER]) + .list_containers(&[LABEL_MANAGED_FILTER, crate::isolation::WORKLOAD_FILTER]) .await .map_err(ComputeDriverError::from)?; @@ -1326,7 +1588,7 @@ impl PodmanComputeDriver { for entry in &entries { if entry.state == "running" { // Running containers need inspect for health check status. - match self.client.inspect_container(&entry.id).await { + match watcher::inspect_workload(&self.client, &entry.id).await { Ok(inspect) => { if let Some(sandbox) = driver_sandbox_from_inspect(&inspect) { sandboxes.push(sandbox); @@ -1419,7 +1681,7 @@ fn validate_apparmor_support( Ok(()) } -fn supervisor_image_pull_policy(image: &str) -> &'static str { +fn runtime_image_pull_policy(image: &str) -> &'static str { if supervisor_image_should_refresh(image) { "newer" } else { @@ -1491,34 +1753,37 @@ fn validate_rootless_local_callback_helper( ))) } -// ── Supervisor binary extraction (userns fallback) ───────────────────── +// ── Sandbox binary extraction (userns fallback) ──────────────────────── -async fn extract_supervisor_bin( +async fn extract_sandbox_bin( client: &PodmanClient, config: &PodmanComputeConfig, ) -> Result { let mut inspect = client - .inspect_image(&config.supervisor_image) + .inspect_image(&config.sandbox_runtime_image) .await .map_err(ComputeDriverError::from)?; - if supervisor_image_should_refresh(&config.supervisor_image) { + if supervisor_image_should_refresh(&config.sandbox_runtime_image) { info!( - image = %config.supervisor_image, - "Refreshing mutable podman supervisor image" + image = %config.sandbox_runtime_image, + "Refreshing mutable Podman sandbox runtime image" ); - match client.pull_image(&config.supervisor_image, "always").await { + match client + .pull_image(&config.sandbox_runtime_image, "always") + .await + { Ok(()) => { inspect = client - .inspect_image(&config.supervisor_image) + .inspect_image(&config.sandbox_runtime_image) .await .map_err(ComputeDriverError::from)?; } Err(err) => { warn!( - image = %config.supervisor_image, + image = %config.sandbox_runtime_image, error = %err, - "Failed to refresh mutable podman supervisor image; \ + "Failed to refresh mutable Podman sandbox runtime image; \ falling back to local image if present", ); } @@ -1527,36 +1792,38 @@ async fn extract_supervisor_bin( let digest = if inspect.id.is_empty() { return Err(ComputeDriverError::Precondition(format!( - "supervisor image '{}' has no ID", - config.supervisor_image, + "sandbox runtime image '{}' has no ID", + config.sandbox_runtime_image, ))); } else { &inspect.id }; - let cache_path = - openshell_core::driver_utils::supervisor_cache_path("podman-supervisor", digest) - .map_err(ComputeDriverError::Precondition)?; + let cache_path = openshell_core::driver_utils::supervisor_cache_path("podman-sandbox", digest) + .map_err(ComputeDriverError::Precondition)?; + // Unit tests use ordered Podman API stubs and intentionally exercise the + // extraction path on every create. Production reuses the immutable cache. + #[cfg(not(test))] if cache_path.is_file() { validate_linux_elf_binary(&cache_path).map_err(ComputeDriverError::Precondition)?; info!( cache_path = %cache_path.display(), - "Using cached supervisor binary" + "Using cached sandbox binary" ); return Ok(cache_path); } info!( - image = %config.supervisor_image, + image = %config.sandbox_runtime_image, cache_path = %cache_path.display(), - "Extracting supervisor binary from image" + "Extracting sandbox binary from image" ); let container_name = temp_extract_container_name(); let spec = serde_json::json!({ - "image": config.supervisor_image, + "image": config.sandbox_runtime_image, "name": container_name, - "entrypoint": [SUPERVISOR_IMAGE_BINARY_PATH], + "entrypoint": [SANDBOX_RUNTIME_IMAGE_BINARY_PATH], "command": [], }); client @@ -1570,7 +1837,7 @@ async fn extract_supervisor_bin( warn!( container = container_name, error = %err, - "Failed to remove supervisor extractor container" + "Failed to remove sandbox runtime extractor container" ); } @@ -1583,13 +1850,13 @@ async fn extract_binary_from_container( cache_path: &Path, ) -> Result { let tar_bytes = client - .copy_from_container(container_name, SUPERVISOR_IMAGE_BINARY_PATH) + .copy_from_container(container_name, SANDBOX_RUNTIME_IMAGE_BINARY_PATH) .await .map_err(ComputeDriverError::from)?; let binary_bytes = extract_first_tar_entry(&tar_bytes).map_err(|err| { ComputeDriverError::Precondition(format!( - "failed to extract supervisor binary from tar: {err}" + "failed to extract sandbox binary from tar: {err}" )) })?; @@ -1599,16 +1866,32 @@ async fn extract_binary_from_container( Ok(cache_path.to_path_buf()) } -fn userns_needs_extraction(userns: Option<&str>) -> bool { - userns.is_some_and(|mode| { - let base = mode.split(':').next().unwrap_or(mode); - !base.eq_ignore_ascii_case("host") - }) +fn podman_child_environment( + sandbox: &DriverSandbox, + image_env: &[String], +) -> HashMap { + let mut environment = image_env + .iter() + .filter_map(|entry| { + entry + .split_once('=') + .map(|(key, value)| (key.to_string(), value.to_string())) + }) + .collect::>(); + if let Some(spec) = sandbox.spec.as_ref() { + if let Some(template) = spec.template.as_ref() { + environment.extend(template.environment.clone()); + } + environment.extend(spec.environment.clone()); + } + environment.retain(|key, _| !key.starts_with("OPENSHELL_")); + environment } /// Returns `true` when userns remaps all UIDs, making host-owned bind mounts /// unreadable from inside the container. `auto` and `no-map` remap every UID; /// `keep-id` preserves the host user's UID; `host` uses the host namespace. +#[cfg(test)] fn userns_remaps_uids(userns: Option<&str>) -> bool { userns.is_some_and(|mode| { let base = mode.split(':').next().unwrap_or(mode); @@ -1621,6 +1904,10 @@ mod tests { use super::*; use crate::test_utils::{StubResponse, spawn_podman_stub}; use hyper::StatusCode; + use openshell_core::jwt::{ + CredentialEpoch, SandboxLaunchAuthentication, SecretJwt, SessionVerificationKey, + SupervisorAuthBundle, + }; use openshell_core::proto::compute::v1::{ DriverSandboxSpec, DriverSandboxTemplate, ResourceRequirements, }; @@ -1628,6 +1915,33 @@ mod tests { use std::fs; use std::path::{Path, PathBuf}; + fn launch_authentication() -> SandboxLaunchAuthentication { + SandboxLaunchAuthentication { + supervisor: SupervisorAuthBundle { + session_id: openshell_core::SandboxSessionId::new(), + runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-1", + ) + .unwrap(), + session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(), + auth_epoch: CredentialEpoch::new(1).unwrap(), + gateway_token: SecretJwt::parse("gateway.token.value").unwrap(), + gateway_expires_at: i64::MAX, + sandbox_token: SecretJwt::parse("sandbox.token.value").unwrap(), + sandbox_expires_at: i64::MAX, + }, + gateway_id: "gateway-test".to_string(), + verification_keys: vec![SessionVerificationKey { + key_id: "test-key".to_string(), + public_key_pem: b"public-key".to_vec(), + }], + } + } + + fn encoded_launch_authentication() -> Vec { + serde_json::to_vec(&launch_authentication()).unwrap() + } + // ── socket resolution ─────────────────────────────────────────────── // // These test resolve_socket_path directly with an injected detector, so @@ -1705,6 +2019,7 @@ mod tests { "lifecycle-stop", vec![ StubResponse::new(StatusCode::OK, r#"[{"Id":"ctr-1","State":"running"}]"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), // companion stop StubResponse::new(StatusCode::NO_CONTENT, ""), StubResponse::new( StatusCode::OK, @@ -1720,7 +2035,7 @@ mod tests { assert_eq!( stop_requests .lock() - .expect("request log lock should not be poisoned")[1], + .expect("request log lock should not be poisoned")[2], format!( "POST {}", api_path("/libpod/containers/ctr-1/stop?timeout=10") @@ -1729,7 +2044,7 @@ mod tests { assert_eq!( stop_requests .lock() - .expect("request log lock should not be poisoned")[2], + .expect("request log lock should not be poisoned")[3], format!("GET {}", api_path("/libpod/containers/ctr-1/json")) ); @@ -1741,14 +2056,38 @@ mod tests { StatusCode::OK, r#"{"Id":"ctr-1","Name":"sandbox","State":{"Status":"exited","Running":false,"FinishedAt":"2026-08-12T16:39:13Z"},"Config":{}}"#, ), - StubResponse::new(StatusCode::NO_CONTENT, ""), - ], + ].into_iter().chain(restart_responses()).collect(), ); + let authentication = encoded_launch_authentication(); test_driver(start_socket.clone()) - .start_sandbox("sandbox-1") + .start_sandbox("sandbox-1", "generation-1", &authentication) .await .expect("start should succeed"); start_handle.await.expect("start stub should finish"); + let restart_requests = start_requests.lock().unwrap().clone(); + assert_eq!( + restart_requests + .iter() + .filter(|request| request.starts_with("PUT ")) + .cloned() + .collect::>(), + vec![ + format!( + "PUT {}", + api_path("/libpod/containers/ctr-1/archive?path=%2F.openshell%2Fchannel") + ), + format!( + "PUT {}", + api_path("/libpod/containers/openshell-supervisor-sandbox-1/archive?path=%2F") + ), + ] + ); + assert!( + !restart_requests + .iter() + .any(|request| request.contains("/volumes/create") + || request.contains("/containers/create")) + ); assert_eq!( start_requests .lock() @@ -1759,7 +2098,10 @@ mod tests { start_requests .lock() .expect("request log lock should not be poisoned")[2], - format!("POST {}", api_path("/libpod/containers/ctr-1/start")) + format!( + "POST {}", + api_path("/libpod/containers/openshell-supervisor-sandbox-1/stop?timeout=10") + ) ); let _ = fs::remove_file(stop_socket); @@ -1772,6 +2114,7 @@ mod tests { "lifecycle-stop-wait", vec![ StubResponse::new(StatusCode::OK, r#"[{"Id":"ctr-1","State":"running"}]"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), // companion stop StubResponse::new(StatusCode::NO_CONTENT, ""), StubResponse::new( StatusCode::OK, @@ -1793,12 +2136,12 @@ mod tests { let requests = requests .lock() .expect("request log lock should not be poisoned"); - assert_eq!(requests.len(), 4); + assert_eq!(requests.len(), 5); assert_eq!( - requests[2], + requests[3], format!("GET {}", api_path("/libpod/containers/ctr-1/json")) ); - assert_eq!(requests[3], requests[2]); + assert_eq!(requests[4], requests[3]); let _ = fs::remove_file(socket); } @@ -1809,6 +2152,7 @@ mod tests { "lifecycle-stop-retry", vec![ StubResponse::new(StatusCode::OK, r#"[{"Id":"ctr-1","State":"stopping"}]"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), // companion stop StubResponse::new( StatusCode::OK, r#"{"Id":"ctr-1","Name":"sandbox","State":{"Status":"exited","Running":false,"FinishedAt":"2026-08-12T16:39:13Z"},"Config":{}}"#, @@ -1825,9 +2169,9 @@ mod tests { let requests = requests .lock() .expect("request log lock should not be poisoned"); - assert_eq!(requests.len(), 2); + assert_eq!(requests.len(), 3); assert_eq!( - requests[1], + requests[2], format!("GET {}", api_path("/libpod/containers/ctr-1/json")) ); @@ -1845,6 +2189,7 @@ mod tests { "trace-stop", vec![ StubResponse::new(StatusCode::OK, r#"[{"Id":"ctr-1","State":"running"}]"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), // companion stop StubResponse::new(StatusCode::NO_CONTENT, ""), StubResponse::new( StatusCode::OK, @@ -1884,6 +2229,45 @@ mod tests { let _ = fs::remove_file(socket_path); } + #[tokio::test] + async fn start_sandbox_span_does_not_capture_launch_authentication() { + use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); + + test_driver(PathBuf::from("/nonexistent/podman.sock")) + .start_sandbox( + "sandbox-1", + "invalid-generation", + b"secret-launch-authentication", + ) + .with_subscriber(subscriber) + .await + .expect_err("invalid generation must fail before contacting Podman"); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|span| span.name == "podman.start_sandbox") + .expect("start operation span"); + assert!(span.attributes.iter().all(|attribute| { + !matches!( + attribute.key.as_str(), + "launch_authentication" | "encoded_authentication" + ) + })); + provider.shutdown().unwrap(); + } + #[tokio::test] async fn create_sandbox_exports_nested_preparation_spans() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; @@ -1891,19 +2275,12 @@ mod tests { use tracing_subscriber::layer::SubscriberExt as _; let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; - let (socket_path, _requests, handle) = spawn_podman_stub( + let (socket_path, requests, handle) = spawn_podman_stub( "trace-create", - vec![ - StubResponse::new(StatusCode::OK, "{}"), - StubResponse::new(StatusCode::OK, "{}"), - StubResponse::new( - StatusCode::OK, - r#"{"Id":"sha256:sandbox","Config":{"User":"1234:1235"}}"#, - ), - StubResponse::new(StatusCode::CREATED, "{}"), - StubResponse::new(StatusCode::CREATED, "{}"), - StubResponse::new(StatusCode::NO_CONTENT, ""), - ], + create_setup_responses(false) + .into_iter() + .chain(create_launch_responses()) + .collect(), ); let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() @@ -1912,12 +2289,33 @@ mod tests { let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); + let mut sandbox = plain_sandbox("sandbox-trace", "demo"); + sandbox.spec = Some(DriverSandboxSpec { + launch_authentication: encoded_launch_authentication(), + ..DriverSandboxSpec::default() + }); test_driver(socket_path.clone()) - .create_sandbox(&plain_sandbox("sandbox-trace", "demo")) + .create_sandbox(&sandbox) .with_subscriber(subscriber) .await .expect("create should succeed"); handle.await.expect("stub should finish"); + let uploads: Vec<_> = requests + .lock() + .unwrap() + .iter() + .filter(|request| request.starts_with("PUT ")) + .cloned() + .collect(); + assert_eq!( + uploads, + [ + "/libpod/containers/workload/archive?path=%2F.openshell%2Fchannel", + "/libpod/containers/workload/archive?path=%2Fsandbox", + "/libpod/containers/supervisor/archive?path=%2F", + ] + .map(|path| format!("PUT {}", api_path(path))) + ); provider.force_flush().unwrap(); let spans = exporter.get_finished_spans().unwrap(); @@ -1929,7 +2327,6 @@ mod tests { "podman.prepare_images", "podman.prepare_storage", "podman.prepare_container", - "podman.start_container", ] { let child = spans .iter() @@ -2009,11 +2406,11 @@ mod tests { StatusCode::OK, r#"{"Id":"ctr-1","Name":"sandbox","State":{"Status":"exited","Running":false,"FinishedAt":"2026-08-12T16:39:13Z"},"Config":{}}"#, ), - StubResponse::new(StatusCode::NO_CONTENT, ""), - ], + ].into_iter().chain(restart_responses()).collect(), ); + let authentication = encoded_launch_authentication(); test_driver(start_socket.clone()) - .start_sandbox("sandbox-1") + .start_sandbox("sandbox-1", "generation-1", &authentication) .with_subscriber(subscriber) .await .expect("start should succeed"); @@ -2022,6 +2419,8 @@ mod tests { let (delete_socket, _requests, delete_handle) = spawn_podman_stub( "trace-delete", vec![ + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove companion + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove channel if detached StubResponse::new(StatusCode::OK, "[]"), StubResponse::new(StatusCode::NO_CONTENT, ""), ], @@ -2664,25 +3063,25 @@ mod tests { #[test] fn supervisor_pull_policy_refreshes_mutable_tags_only() { assert_eq!( - supervisor_image_pull_policy("ghcr.io/nvidia/openshell/supervisor:dev"), + runtime_image_pull_policy("ghcr.io/nvidia/openshell/supervisor:dev"), "newer" ); assert_eq!( - supervisor_image_pull_policy("ghcr.io/nvidia/openshell/supervisor:latest"), + runtime_image_pull_policy("ghcr.io/nvidia/openshell/supervisor:latest"), "newer" ); assert_eq!( - supervisor_image_pull_policy("ghcr.io/nvidia/openshell/supervisor"), + runtime_image_pull_policy("ghcr.io/nvidia/openshell/supervisor"), "newer" ); assert_eq!( - supervisor_image_pull_policy( + runtime_image_pull_policy( "ghcr.io/nvidia/openshell/supervisor:0.0.47-dev.13-g57b71c68f" ), "missing" ); assert_eq!( - supervisor_image_pull_policy("ghcr.io/nvidia/openshell/supervisor@sha256:abc123"), + runtime_image_pull_policy("ghcr.io/nvidia/openshell/supervisor@sha256:abc123"), "missing" ); } @@ -2892,6 +3291,8 @@ mod tests { let (socket_path, request_log, handle) = spawn_podman_stub( "delete-not-found", vec![ + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove companion + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove channel if detached // list_containers returns empty (container already gone) StubResponse::new(StatusCode::OK, "[]"), // remove_volume @@ -2911,9 +3312,9 @@ mod tests { .lock() .expect("request log lock should not be poisoned") .clone(); - assert!(requests[0].contains("/libpod/containers/json")); + assert!(requests[2].contains("/libpod/containers/json")); assert_eq!( - requests[1], + requests[3], format!( "DELETE {}", api_path(&format!("/libpod/volumes/{volume_name}")) @@ -2946,11 +3347,59 @@ mod tests { name: name.to_string(), namespace: String::new(), workspace: String::new(), - spec: None, + spec: Some(DriverSandboxSpec { + launch_authentication: encoded_launch_authentication(), + ..Default::default() + }), status: None, } } + #[test] + fn child_environment_preserves_precedence_and_strips_control_keys() { + let mut sandbox = plain_sandbox("sandbox", "agent"); + sandbox.spec = Some(DriverSandboxSpec { + template: Some(DriverSandboxTemplate { + environment: HashMap::from([ + ("TEMPLATE_ONLY".to_string(), "template".to_string()), + ("OVERRIDE".to_string(), "template".to_string()), + ]), + ..Default::default() + }), + environment: HashMap::from([ + ("REQUEST_ONLY".to_string(), "request".to_string()), + ("OVERRIDE".to_string(), "request".to_string()), + ("OPENSHELL_SANDBOX_TOKEN".to_string(), "spoofed".to_string()), + ]), + ..Default::default() + }); + let image_env = vec![ + "IMAGE_ONLY=image".to_string(), + "OVERRIDE=image".to_string(), + "OPENSHELL_ENDPOINT=spoofed".to_string(), + ]; + + let environment = podman_child_environment(&sandbox, &image_env); + + assert_eq!( + environment.get("IMAGE_ONLY").map(String::as_str), + Some("image") + ); + assert_eq!( + environment.get("TEMPLATE_ONLY").map(String::as_str), + Some("template") + ); + assert_eq!( + environment.get("REQUEST_ONLY").map(String::as_str), + Some("request") + ); + assert_eq!( + environment.get("OVERRIDE").map(String::as_str), + Some("request") + ); + assert!(!environment.keys().any(|key| key.starts_with("OPENSHELL_"))); + } + fn secret_delete_request(sandbox_id: &str) -> String { format!( "DELETE {}", @@ -2961,6 +3410,209 @@ mod tests { ) } + fn restart_responses() -> Vec { + let mut archive = tar::Builder::new(Vec::new()); + let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new( + 1000, + 1001, + vec![], + "image".into(), + "sha256:image".into(), + ) + .unwrap(); + let bundle = serde_json::to_vec(&crate::isolation::RestartMetadata { + generation: "generation-1".to_string(), + workload_identity: identity, + child_env: HashMap::new(), + }) + .unwrap(); + let mut header = tar::Header::new_gnu(); + header.set_size(bundle.len() as u64); + header.set_mode(0o600); + header.set_cksum(); + archive + .append_data(&mut header, "restart-metadata.json", bundle.as_slice()) + .unwrap(); + vec![ + StubResponse::new(StatusCode::NO_CONTENT, ""), // supervisor stop + StubResponse::new( + StatusCode::OK, + r#"{"Id":"supervisor","Name":"supervisor","State":{"Status":"exited","Running":false},"Config":{}}"#, + ), + StubResponse::new(StatusCode::OK, archive.into_inner().unwrap()), + StubResponse::new(StatusCode::OK, "").with_archive_members(channel_archive_members()), + StubResponse::new(StatusCode::OK, ""), // refreshed supervisor auth and runtime descriptor + fence_response(), + StubResponse::new(StatusCode::NO_CONTENT, ""), // workload start + StubResponse::new(StatusCode::NO_CONTENT, ""), // supervisor start + ] + } + + fn fence_response() -> StubResponse { + #[derive(serde::Serialize)] + #[serde(rename_all = "PascalCase")] + struct HostConfig { + network_mode: &'static str, + privileged: bool, + } + #[derive(serde::Serialize)] + #[serde(rename_all = "PascalCase")] + struct Networks { + networks: std::collections::BTreeMap, + } + #[derive(serde::Serialize)] + #[serde(rename_all = "PascalCase")] + struct Fence { + host_config: HostConfig, + network_settings: Networks, + } + StubResponse::new( + StatusCode::OK, + serde_json::to_vec(&Fence { + host_config: HostConfig { + network_mode: "none", + privileged: false, + }, + network_settings: Networks { + networks: std::collections::BTreeMap::default(), + }, + }) + .unwrap(), + ) + } + + fn created_response(id: &'static str) -> StubResponse { + #[derive(serde::Serialize)] + struct Created { + #[serde(rename = "Id")] + id: &'static str, + } + StubResponse::new( + StatusCode::CREATED, + serde_json::to_vec(&Created { id }).unwrap(), + ) + } + + fn image_response(id: &'static str) -> StubResponse { + #[derive(serde::Serialize)] + #[serde(rename_all = "PascalCase")] + struct Config { + user: &'static str, + } + #[derive(serde::Serialize)] + #[serde(rename_all = "PascalCase")] + struct Image { + id: &'static str, + config: Config, + } + StubResponse::new( + StatusCode::OK, + serde_json::to_vec(&Image { + id, + config: Config { user: "1234:1235" }, + }) + .unwrap(), + ) + } + + fn sandbox_binary_archive_response() -> StubResponse { + let mut archive = Vec::new(); + { + let mut builder = tar::Builder::new(&mut archive); + let payload = b"\x7fELF-test-sandbox"; + let mut header = tar::Header::new_gnu(); + header.set_path("openshell-sandbox").unwrap(); + header.set_size(payload.len() as u64); + header.set_mode(0o755); + header.set_cksum(); + builder.append(&header, payload.as_slice()).unwrap(); + builder.finish().unwrap(); + } + StubResponse::new(StatusCode::OK, archive) + } + + fn create_setup_responses(proxy_secret: bool) -> Vec { + let mut responses = vec![ + StubResponse::new(StatusCode::OK, "{}"), // sandbox runtime pull + StubResponse::new(StatusCode::OK, "{}"), // supervisor pull + StubResponse::new(StatusCode::OK, "{}"), // workload pull + image_response("sha256:sandbox"), + created_response("identity-reader"), + StubResponse::new(StatusCode::NOT_FOUND, ""), // reserved hierarchy absent + StubResponse::new(StatusCode::NOT_FOUND, ""), // optional passwd + StubResponse::new(StatusCode::NOT_FOUND, ""), // optional group + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove stopped reader + image_response("sha256:sandbox-runtime"), + image_response("sha256:supervisor"), + StubResponse::new(StatusCode::CREATED, "{}"), // workspace volume + ]; + if proxy_secret { + responses.push(StubResponse::new(StatusCode::CREATED, "{}")); + } + responses.extend([ + image_response("sha256:sandbox-runtime"), + created_response("sandbox-runtime-extractor"), + sandbox_binary_archive_response(), + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove extractor + ]); + responses.push(StubResponse::new(StatusCode::CREATED, "{}")); // channel volume + responses + } + + fn create_launch_responses() -> Vec { + vec![ + created_response("workload"), + fence_response(), + StubResponse::new(StatusCode::OK, "").with_archive_members(channel_archive_members()), + StubResponse::new(StatusCode::OK, "").with_archive_members(&["."]), + created_response("supervisor"), + StubResponse::new(StatusCode::OK, ""), // supervisor archive + StubResponse::new(StatusCode::NO_CONTENT, ""), // workload start + StubResponse::new(StatusCode::NO_CONTENT, ""), // supervisor start + ] + } + + fn channel_archive_members() -> &'static [&'static str] { + &[ + ".", + "sandbox", + "sandbox/bootstrap.json", + "sandbox/server.crt", + "sandbox/server.key", + ] + } + + #[tokio::test] + async fn reserved_image_control_root_fails_before_workload_or_secrets() { + let (path, requests, handle) = spawn_podman_stub( + "reserved-control-root", + vec![ + StubResponse::new(StatusCode::OK, "{}"), + StubResponse::new(StatusCode::OK, "{}"), + StubResponse::new(StatusCode::OK, "{}"), + image_response("sha256:image"), + created_response("identity-reader"), + StubResponse::new(StatusCode::OK, "existing reserved path"), + StubResponse::new(StatusCode::NO_CONTENT, ""), + ], + ); + let error = test_driver(path.clone()) + .create_sandbox(&plain_sandbox("id", "name")) + .await + .unwrap_err(); + assert!(error.to_string().contains("reserved /.openshell")); + handle.await.unwrap(); + assert!( + !requests + .lock() + .unwrap() + .iter() + .any(|request| request.contains("/libpod/volumes") + || request.contains("/libpod/secrets")) + ); + let _ = fs::remove_file(path); + } + #[tokio::test] async fn create_sandbox_removes_proxy_auth_secret_on_container_create_failure() { // A credential secret is staged before the container is created, so a @@ -2969,24 +3621,25 @@ mod tests { let auth_file = write_proxy_auth_file("create-fail"); let (socket_path, request_log, handle) = spawn_podman_stub( "create-container-fail", - vec![ - StubResponse::new(StatusCode::OK, "{}"), // pull supervisor image - StubResponse::new(StatusCode::OK, "{}"), // pull sandbox image - StubResponse::new( - StatusCode::OK, - r#"{"Id":"sha256:sandbox","Config":{"User":"1234:1235"}}"#, - ), // inspect sandbox image - StubResponse::new(StatusCode::CREATED, "{}"), // create volume - StubResponse::new(StatusCode::CREATED, "{}"), // create proxy-auth secret - StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, r#"{"message":"boom"}"#), // create container - StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove volume - StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove proxy-auth secret - ], + create_setup_responses(true) + .into_iter() + .chain([ + StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, "create failed"), + StubResponse::new(StatusCode::NO_CONTENT, ""), // channel + StubResponse::new(StatusCode::NO_CONTENT, ""), // workspace + StubResponse::new(StatusCode::NO_CONTENT, ""), // proxy secret + ]) + .collect(), ); let driver = test_driver_with_config(proxy_auth_config(socket_path.clone(), &auth_file)); + let mut sandbox = plain_sandbox(sandbox_id, "demo"); + sandbox.spec = Some(DriverSandboxSpec { + launch_authentication: encoded_launch_authentication(), + ..DriverSandboxSpec::default() + }); driver - .create_sandbox(&plain_sandbox(sandbox_id, "demo")) + .create_sandbox(&sandbox) .await .expect_err("container create should fail"); @@ -3011,26 +3664,28 @@ mod tests { let auth_file = write_proxy_auth_file("start-fail"); let (socket_path, request_log, handle) = spawn_podman_stub( "create-start-fail", - vec![ - StubResponse::new(StatusCode::OK, "{}"), // pull supervisor image - StubResponse::new(StatusCode::OK, "{}"), // pull sandbox image - StubResponse::new( - StatusCode::OK, - r#"{"Id":"sha256:sandbox","Config":{"User":"1234:1235"}}"#, - ), // inspect sandbox image - StubResponse::new(StatusCode::CREATED, "{}"), // create volume - StubResponse::new(StatusCode::CREATED, "{}"), // create proxy-auth secret - StubResponse::new(StatusCode::CREATED, "{}"), // create container - StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, r#"{"message":"boom"}"#), // start container - StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove container - StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove volume - StubResponse::new(StatusCode::NO_CONTENT, ""), // cleanup: remove proxy-auth secret - ], + create_setup_responses(true) + .into_iter() + .chain(create_launch_responses().into_iter().take(7)) + .chain([ + StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, "supervisor start failed"), + StubResponse::new(StatusCode::NO_CONTENT, ""), // supervisor + StubResponse::new(StatusCode::NO_CONTENT, ""), // workload + StubResponse::new(StatusCode::NO_CONTENT, ""), // channel + StubResponse::new(StatusCode::NO_CONTENT, ""), // workspace + StubResponse::new(StatusCode::NO_CONTENT, ""), // proxy secret + ]) + .collect(), ); let driver = test_driver_with_config(proxy_auth_config(socket_path.clone(), &auth_file)); + let mut sandbox = plain_sandbox(sandbox_id, "demo"); + sandbox.spec = Some(DriverSandboxSpec { + launch_authentication: encoded_launch_authentication(), + ..DriverSandboxSpec::default() + }); driver - .create_sandbox(&plain_sandbox(sandbox_id, "demo")) + .create_sandbox(&sandbox) .await .expect_err("container start should fail"); @@ -3055,7 +3710,9 @@ mod tests { let (socket_path, request_log, handle) = spawn_podman_stub( "delete-proxy-auth", vec![ - StubResponse::new(StatusCode::OK, "[]"), // list_containers (not found) + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove companion + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove channel if detached + StubResponse::new(StatusCode::OK, "[]"), // list_containers (not found) StubResponse::new(StatusCode::NO_CONTENT, ""), // remove volume StubResponse::new(StatusCode::NO_CONTENT, ""), // remove token secret StubResponse::new(StatusCode::NO_CONTENT, ""), // remove proxy-auth secret @@ -3098,10 +3755,14 @@ mod tests { let (socket_path, request_log, handle) = spawn_podman_stub( "delete-label-lookup", vec![ + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove companion + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove channel if detached // list_containers by label StubResponse::new(StatusCode::OK, list_body), // single timed remove_container operation StubResponse::new(StatusCode::NO_CONTENT, ""), + // channel volume, now detached + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove_volume StubResponse::new(StatusCode::NO_CONTENT, ""), ], @@ -3119,9 +3780,9 @@ mod tests { .lock() .expect("request log lock should not be poisoned") .clone(); - assert!(requests[0].contains("/libpod/containers/json")); + assert!(requests[2].contains("/libpod/containers/json")); assert_eq!( - requests[1], + requests[3], format!( "DELETE {}", api_path(&format!( @@ -3130,7 +3791,7 @@ mod tests { ) ); assert_eq!( - requests[2], + requests[5], format!( "DELETE {}", api_path(&format!("/libpod/volumes/{volume_name}")) @@ -3139,19 +3800,6 @@ mod tests { let _ = fs::remove_file(socket_path); } - #[test] - fn userns_needs_extraction_cases() { - assert!(!userns_needs_extraction(None)); - assert!(!userns_needs_extraction(Some("host"))); - assert!(!userns_needs_extraction(Some("Host"))); - assert!(userns_needs_extraction(Some("auto"))); - assert!(userns_needs_extraction(Some("auto:size=65536"))); - assert!(userns_needs_extraction(Some("keep-id"))); - assert!(userns_needs_extraction(Some("keep-id:uid=1000"))); - assert!(userns_needs_extraction(Some("no-map"))); - assert!(userns_needs_extraction(Some("private"))); - } - #[test] fn userns_remaps_uids_cases() { assert!(!userns_remaps_uids(None)); diff --git a/crates/openshell-driver-podman/src/grpc.rs b/crates/openshell-driver-podman/src/grpc.rs index fedeea3068..b8605d9f49 100644 --- a/crates/openshell-driver-podman/src/grpc.rs +++ b/crates/openshell-driver-podman/src/grpc.rs @@ -162,8 +162,7 @@ impl ComputeDriver for ComputeDriverService { .into_inner() .sandbox .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; - self.driver - .create_sandbox(&sandbox) + Box::pin(self.driver.create_sandbox(&sandbox)) .await .map_err(Status::from)?; Ok(Response::new(CreateSandboxResponse {})) @@ -201,7 +200,11 @@ impl ComputeDriver for ComputeDriverService { return Err(Status::invalid_argument("sandbox_id is required")); } self.driver - .start_sandbox(&request.sandbox_id) + .start_sandbox( + &request.sandbox_id, + &request.generation_id, + &request.launch_authentication, + ) .await .map_err(Status::from)?; Ok(Response::new(StartSandboxResponse {})) @@ -680,6 +683,8 @@ mod tests { let (socket_path, request_log, handle) = spawn_podman_stub( "forward-id", vec![ + StubResponse::new(StatusCode::NO_CONTENT, ""), // companion + StubResponse::new(StatusCode::NO_CONTENT, ""), // channel // list_containers returns empty (container already gone) StubResponse::new(StatusCode::OK, "[]"), // remove_volume @@ -708,9 +713,9 @@ mod tests { .lock() .expect("request log lock should not be poisoned") .clone(); - assert!(requests[0].contains("/libpod/containers/json")); + assert!(requests[2].contains("/libpod/containers/json")); assert_eq!( - requests[1], + requests[3], format!( "DELETE {}", api_path(&format!("/libpod/volumes/{volume_name}")) diff --git a/crates/openshell-driver-podman/src/isolation.rs b/crates/openshell-driver-podman/src/isolation.rs new file mode 100644 index 0000000000..b616fa6474 --- /dev/null +++ b/crates/openshell-driver-podman/src/isolation.rs @@ -0,0 +1,463 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Podman-owned provisioning for the common authenticated isolation channel. + +use std::collections::{BTreeMap, HashMap}; +#[cfg(test)] +use std::io::Read; +use std::path::PathBuf; + +use openshell_core::ComputeDriverError; +use openshell_core::proto::compute::v1::DriverSandbox; +use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity}; +use openshell_sandbox_backend::boundary_protocol::{ + BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor, + SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport, + generate_sandbox_tls_material, +}; +use serde::{Deserialize, Serialize}; + +pub const LABEL_ROLE: &str = "openshell.io/isolation-role"; +pub const WORKLOAD_FILTER: &str = "openshell.io/isolation-role=sandbox"; +pub const CHANNEL_ROOT: &str = "/.openshell/channel"; +pub const BOOTSTRAP_PATH: &str = "/.openshell/channel/sandbox/bootstrap.json"; +pub const RUNTIME_DESCRIPTOR_PATH: &str = "/.openshell/supervisor/runtime-descriptor.json"; +pub const AUTH_BUNDLE_PATH: &str = "/.openshell/supervisor/auth.json"; +pub const RESTART_METADATA_PATH: &str = "/.openshell/supervisor/restart-metadata.json"; +const SOCKET_PATH: &str = "/.openshell/channel/sandbox/control.sock"; + +pub fn supervisor_name(id: &str) -> String { + format!("openshell-supervisor-{id}") +} +pub fn channel_volume_name(id: &str) -> String { + format!("openshell-channel-{id}") +} + +fn invalid(error: impl std::fmt::Display) -> ComputeDriverError { + ComputeDriverError::Precondition(error.to_string()) +} + +/// Resolve policy names against the pinned workload image, never the gateway. +pub fn resolve_identity( + sandbox: &DriverSandbox, + image_id: &str, + image_user: &str, + passwd: &[u8], + group: &[u8], +) -> Result { + let passwd = std::str::from_utf8(passwd).map_err(invalid)?; + let group = std::str::from_utf8(group).map_err(invalid)?; + let accounts: Vec<_> = passwd + .lines() + .filter_map(|line| { + let mut fields = line.split(':'); + let name = fields.next()?; + fields.next()?; + Some(( + name, + fields.next()?.parse::().ok()?, + fields.next()?.parse::().ok()?, + )) + }) + .collect(); + let groups: Vec<_> = group + .lines() + .filter_map(|line| { + let mut fields = line.split(':'); + let name = fields.next()?; + fields.next()?; + Some((name, fields.next()?.parse::().ok()?, fields.next()?)) + }) + .collect(); + let request = sandbox + .spec + .as_ref() + .and_then(|spec| spec.workload_identity.as_ref()); + let requested_user = request.map_or("", |identity| identity.user.trim()); + let requested_group = request.map_or("", |identity| identity.group.trim()); + let (image_user, image_group) = image_user.split_once(':').unwrap_or((image_user, "")); + let user = if requested_user.is_empty() { + image_user + } else { + requested_user + }; + let group = if requested_group.is_empty() { + image_group + } else { + requested_group + }; + let account = accounts + .iter() + .find(|(name, uid, _)| *name == user || user.parse::().ok() == Some(*uid)); + let uid = user + .parse() + .ok() + .or_else(|| account.map(|(_, uid, _)| *uid)) + .ok_or_else(|| invalid("configure a non-root workload user present in the pinned image"))?; + let gid = if group.is_empty() { + account.map(|(_, _, gid)| *gid) + } else { + group.parse().ok().or_else(|| { + groups + .iter() + .find(|(name, _, _)| *name == group) + .map(|(_, gid, _)| *gid) + }) + } + .ok_or_else(|| { + invalid("configure an explicit workload group for a UID without an image passwd entry") + })?; + let supplemental = account.map_or_else(Vec::new, |(username, _, _)| { + groups + .iter() + .filter(|(_, id, members)| { + *id != gid && members.split(',').any(|member| member == *username) + }) + .map(|(_, gid, _)| *gid) + .collect() + }); + let source = if requested_user.is_empty() && requested_group.is_empty() { + "image" + } else { + "policy" + }; + ResolvedWorkloadIdentity::new(uid, gid, supplemental, source.into(), image_id.into()) + .map_err(invalid) +} + +pub struct BootstrapArchives { + pub channel: Vec, + pub workspace: Vec, + pub supervisor: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct RestartMetadata { + pub(crate) generation: String, + pub(crate) workload_identity: ResolvedWorkloadIdentity, + pub(crate) child_env: HashMap, +} + +/// The shared volume contains only sandbox credentials. Supervisor credentials, +/// gateway authorization, and the restart copy never enter that volume. +pub fn bootstrap_archives( + sandbox_id: &str, + container_id: &str, + generation: &str, + identity: &ResolvedWorkloadIdentity, + child_env: HashMap, + launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication, +) -> Result { + launch_authentication.validate().map_err(invalid)?; + let session_id = launch_authentication.supervisor.session_id; + let tls = generate_sandbox_tls_material(session_id).map_err(invalid)?; + let resource_claims = BTreeMap::from([ + ("podman.container_id".into(), container_id.into()), + ( + "podman.image_identity".into(), + identity.resource_digest.clone(), + ), + ]); + let driver_fence = DriverFenceEvidence::Podman { + container_id: container_id.into(), + network_mode: "none".into(), + unexpected_networks: Vec::new(), + }; + let runtime_generation = launch_authentication + .supervisor + .runtime_generation + .to_string(); + let verification_keys = launch_authentication + .verification_keys + .iter() + .map(|key| { + String::from_utf8(key.public_key_pem.clone()) + .map(|public_key_pem| GatewayVerificationKey { + key_id: key.key_id.clone(), + public_key_pem, + }) + .map_err(invalid) + }) + .collect::, _>>()?; + let config = BoundaryConfig { + boundary_id: sandbox_id.into(), + generation: runtime_generation.clone(), + session_id, + session_rotation: launch_authentication.supervisor.session_rotation, + auth_epoch: launch_authentication.supervisor.auth_epoch, + gateway_id: launch_authentication.gateway_id.clone(), + verification_keys, + listener: BoundaryListener::Unix { + socket_path: PathBuf::from(SOCKET_PATH), + tls: SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from("/.openshell/channel/sandbox/server.crt"), + private_key_path: PathBuf::from("/.openshell/channel/sandbox/server.key"), + }, + }, + resource_claims: resource_claims.clone(), + resource_claim_files: BTreeMap::new(), + workload_identity: identity.clone(), + driver_fence: driver_fence.clone(), + child_env: child_env.clone(), + }; + let runtime_descriptor = SandboxRuntimeDescriptor { + boundary_id: sandbox_id.into(), + generation: runtime_generation, + session_id, + transport: SandboxTransport::Unix { + socket_path: PathBuf::from(SOCKET_PATH), + }, + tls: SandboxTlsClientConfig { + server_name: tls.server_name, + trust_anchor_pem: tls.trust_anchor_pem, + }, + host_gateway_ip: None, + resource_claims, + workload_identity: identity.clone(), + driver_fence, + }; + // Libpod resolves the requested upload destination once for a stopped + // container. Archive entries must be relative to the selected named volume, + // not rootfs paths that the volume would shadow on container start. + let mut channel = Archive::new(identity); + channel.directory(".", 0o755, false)?; + channel.directory("sandbox", 0o711, true)?; + channel.file( + "sandbox/bootstrap.json", + &serde_json::to_vec(&config).map_err(invalid)?, + )?; + channel.file("sandbox/server.crt", tls.certificate_chain_pem.as_bytes())?; + channel.file("sandbox/server.key", tls.private_key_pem.as_bytes())?; + let channel = channel.finish()?; + let mut workspace = Archive::new(identity); + workspace.directory(".", 0o700, true)?; + let mut supervisor = Archive::new(identity); + supervisor.directory(".openshell", 0o755, false)?; + supervisor.directory(".openshell/supervisor", 0o700, true)?; + supervisor.file( + RUNTIME_DESCRIPTOR_PATH, + &serde_json::to_vec(&runtime_descriptor).map_err(invalid)?, + )?; + supervisor.file( + AUTH_BUNDLE_PATH, + &serde_json::to_vec(&launch_authentication.supervisor).map_err(invalid)?, + )?; + let restart_metadata = RestartMetadata { + generation: generation.to_string(), + workload_identity: identity.clone(), + child_env, + }; + supervisor.file( + RESTART_METADATA_PATH, + &serde_json::to_vec(&restart_metadata).map_err(invalid)?, + )?; + Ok(BootstrapArchives { + channel, + workspace: workspace.finish()?, + supervisor: supervisor.finish()?, + }) +} + +pub fn restart_metadata_from_slice(bytes: &[u8]) -> Result { + serde_json::from_slice(bytes).map_err(invalid) +} + +struct Archive<'a> { + builder: tar::Builder>, + identity: &'a ResolvedWorkloadIdentity, +} +impl<'a> Archive<'a> { + fn new(identity: &'a ResolvedWorkloadIdentity) -> Self { + Self { + builder: tar::Builder::new(Vec::new()), + identity, + } + } + fn directory(&mut self, path: &str, mode: u32, owned: bool) -> Result<(), ComputeDriverError> { + self.append(path, mode, owned, tar::EntryType::Directory, &[]) + } + fn file(&mut self, path: &str, content: &[u8]) -> Result<(), ComputeDriverError> { + self.append( + path.trim_start_matches('/'), + 0o600, + true, + tar::EntryType::Regular, + content, + ) + } + fn append( + &mut self, + path: &str, + mode: u32, + owned: bool, + kind: tar::EntryType, + content: &[u8], + ) -> Result<(), ComputeDriverError> { + let mut header = tar::Header::new_gnu(); + header.set_entry_type(kind); + header.set_mode(mode); + header.set_uid(if owned { + u64::from(self.identity.uid) + } else { + 0 + }); + header.set_gid(if owned { + u64::from(self.identity.gid) + } else { + 0 + }); + header.set_size(content.len() as u64); + header.set_mtime(0); + header.set_cksum(); + self.builder + .append_data(&mut header, path, content) + .map_err(invalid) + } + fn finish(self) -> Result, ComputeDriverError> { + self.builder.into_inner().map_err(invalid) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use openshell_core::jwt::{ + CredentialEpoch, SandboxLaunchAuthentication, SecretJwt, SessionVerificationKey, + SupervisorAuthBundle, + }; + + fn authentication() -> SandboxLaunchAuthentication { + SandboxLaunchAuthentication { + supervisor: SupervisorAuthBundle { + session_id: openshell_core::SandboxSessionId::new(), + runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-1", + ) + .unwrap(), + session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(), + auth_epoch: CredentialEpoch::new(1).unwrap(), + gateway_token: SecretJwt::parse("gateway.token.value").unwrap(), + gateway_expires_at: i64::MAX, + sandbox_token: SecretJwt::parse("sandbox.token.value").unwrap(), + sandbox_expires_at: i64::MAX, + }, + gateway_id: "gateway-test".to_string(), + verification_keys: vec![SessionVerificationKey { + key_id: "test-key".to_string(), + public_key_pem: b"public-key".to_vec(), + }], + } + } + + #[test] + fn identity_uses_pinned_image_accounts_and_rejects_root() { + let sandbox = DriverSandbox::default(); + let passwd = b"root:x:0:0:root:/root:/bin/sh\nagent:x:1000:1001::/home/agent:/bin/sh\n"; + let groups = b"agent:x:1001:\ndata:x:2000:agent\n"; + let identity = + resolve_identity(&sandbox, "sha256:pinned", "agent", passwd, groups).unwrap(); + assert_eq!((identity.uid, identity.gid), (1000, 1001)); + assert_eq!(identity.supplementary_gids, vec![2000]); + assert_eq!(identity.resource_digest, "sha256:pinned"); + assert!(resolve_identity(&sandbox, "sha256:pinned", "root", passwd, groups).is_err()); + assert!(resolve_identity(&sandbox, "sha256:pinned", "", passwd, groups).is_err()); + assert!(resolve_identity(&sandbox, "sha256:pinned", "2000", passwd, groups).is_err()); + } + + fn files(bytes: &[u8]) -> BTreeMap> { + tar::Archive::new(bytes) + .entries() + .unwrap() + .filter_map(|entry| { + let mut entry = entry.unwrap(); + if !entry.header().entry_type().is_file() { + return None; + } + let path = entry.path().unwrap().into_owned(); + assert_eq!(entry.header().mode().unwrap(), 0o600); + assert_eq!(entry.header().uid().unwrap(), 1000); + let mut content = Vec::new(); + entry.read_to_end(&mut content).unwrap(); + Some((path, content)) + }) + .collect() + } + + #[test] + fn archives_separate_supervisor_credentials_and_bind_one_channel() { + let identity = ResolvedWorkloadIdentity::new( + 1000, + 1001, + vec![], + "image".into(), + "sha256:image".into(), + ) + .unwrap(); + let authentication = authentication(); + let child_env = HashMap::from([("PATH".to_string(), "/agent/bin".to_string())]); + let archives = bootstrap_archives( + "sandbox", + "container", + "generation-1", + &identity, + child_env.clone(), + &authentication, + ) + .unwrap(); + let workload = files(&archives.channel); + let supervisor = files(&archives.supervisor); + let mut workspace = tar::Archive::new(archives.workspace.as_slice()); + let mut entries = workspace.entries().unwrap(); + let root = entries.next().unwrap().unwrap(); + assert_eq!(root.path().unwrap().as_ref(), std::path::Path::new(".")); + assert!(root.header().entry_type().is_dir()); + assert_eq!(root.header().uid().unwrap(), u64::from(identity.uid)); + assert_eq!(root.header().gid().unwrap(), u64::from(identity.gid)); + assert_eq!(root.header().mode().unwrap(), 0o700); + assert!(entries.next().is_none()); + assert_eq!(workload.len(), 3); + assert_eq!(supervisor.len(), 3); + assert!(workload.keys().all(|path| path.starts_with("sandbox"))); + assert!( + supervisor + .keys() + .all(|path| path.starts_with(".openshell/supervisor")) + ); + let config: BoundaryConfig = serde_json::from_slice( + workload + .get(&PathBuf::from("sandbox/bootstrap.json")) + .unwrap(), + ) + .unwrap(); + let runtime_descriptor: SandboxRuntimeDescriptor = serde_json::from_slice( + supervisor + .get(&PathBuf::from( + RUNTIME_DESCRIPTOR_PATH.trim_start_matches('/'), + )) + .unwrap(), + ) + .unwrap(); + assert_eq!(config.boundary_id, runtime_descriptor.boundary_id); + assert_eq!(config.session_id, runtime_descriptor.session_id); + assert_eq!(config.driver_fence, runtime_descriptor.driver_fence); + assert_eq!(config.workload_identity, identity); + runtime_descriptor.driver_fence.validate().unwrap(); + let restart_metadata: RestartMetadata = serde_json::from_slice( + supervisor + .get(&PathBuf::from( + RESTART_METADATA_PATH.trim_start_matches('/'), + )) + .unwrap(), + ) + .unwrap(); + assert_eq!(restart_metadata.workload_identity, identity); + assert_eq!(restart_metadata.child_env, child_env); + let restart_bytes = serde_json::to_vec(&restart_metadata).unwrap(); + assert!( + !restart_bytes + .windows(b"PRIVATE KEY".len()) + .any(|window| window == b"PRIVATE KEY") + ); + } +} diff --git a/crates/openshell-driver-podman/src/lib.rs b/crates/openshell-driver-podman/src/lib.rs index 115e64eb2f..fa06cf4864 100644 --- a/crates/openshell-driver-podman/src/lib.rs +++ b/crates/openshell-driver-podman/src/lib.rs @@ -6,6 +6,7 @@ pub mod config; pub(crate) mod container; pub mod driver; pub mod grpc; +mod isolation; pub mod otel_tracing; mod socket_discovery; #[cfg(test)] diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs index e4554602f4..62a2dde5a1 100644 --- a/crates/openshell-driver-podman/src/main.rs +++ b/crates/openshell-driver-podman/src/main.rs @@ -101,7 +101,11 @@ struct Args { )] health_check_interval_secs: Option, - /// OCI image containing the openshell-sandbox supervisor binary. + /// OCI image containing the `openshell-sandbox` runtime binary. + #[arg(long, env = "OPENSHELL_SANDBOX_RUNTIME_IMAGE")] + sandbox_runtime_image: Option, + + /// OCI image containing the `openshell-supervisor` control binary. #[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE")] supervisor_image: Option, @@ -207,6 +211,9 @@ async fn main() -> Result<()> { ssh_socket_path: args.sandbox_ssh_socket_path, network_name: args.network_name, stop_timeout_secs: args.stop_timeout, + sandbox_runtime_image: args + .sandbox_runtime_image + .unwrap_or_else(openshell_core::config::default_sandbox_runtime_image), supervisor_image: args .supervisor_image .unwrap_or_else(openshell_core::config::default_supervisor_image), diff --git a/crates/openshell-driver-podman/src/test_utils.rs b/crates/openshell-driver-podman/src/test_utils.rs index ec5c8f7f11..25cbcb4ac2 100644 --- a/crates/openshell-driver-podman/src/test_utils.rs +++ b/crates/openshell-driver-podman/src/test_utils.rs @@ -3,7 +3,7 @@ //! Shared test helpers for openshell-driver-podman unit tests. -use http_body_util::Full; +use http_body_util::{BodyExt as _, Full}; use hyper::StatusCode; use hyper::body::Bytes; use hyper::server::conn::http1; @@ -20,16 +20,18 @@ use tokio::net::UnixListener; #[derive(Clone)] pub struct StubResponse { pub status: StatusCode, - pub body: String, + pub body: Bytes, pub delay: Duration, + pub archive_members: Option>, } impl StubResponse { - pub fn new(status: StatusCode, body: impl Into) -> Self { + pub fn new(status: StatusCode, body: impl Into) -> Self { Self { status, body: body.into(), delay: Duration::ZERO, + archive_members: None, } } @@ -37,6 +39,11 @@ impl StubResponse { self.delay = delay; self } + + pub fn with_archive_members(mut self, members: &[&str]) -> Self { + self.archive_members = Some(members.iter().map(PathBuf::from).collect()); + self + } } /// Generate a unique Unix socket path for a test. @@ -88,7 +95,7 @@ pub fn spawn_podman_stub( let result = http1::Builder::new() .serve_connection( TokioIo::new(stream), - service_fn(move |req| { + service_fn(move |req: hyper::Request| { let log = log.clone(); let queue = queue.clone(); async move { @@ -104,11 +111,22 @@ pub fn spawn_podman_stub( .expect("response queue lock should not be poisoned") .pop_front() .expect("stub response should exist"); + if let Some(expected_members) = &response.archive_members { + assert_eq!(req.method(), hyper::Method::PUT); + let body = req.into_body().collect().await.unwrap().to_bytes(); + let mut archive = tar::Archive::new(body.as_ref()); + let members: Vec<_> = archive + .entries() + .unwrap() + .map(|entry| entry.unwrap().path().unwrap().into_owned()) + .collect(); + assert_eq!(&members, expected_members); + } tokio::time::sleep(response.delay).await; Ok::<_, Infallible>( hyper::Response::builder() .status(response.status) - .body(Full::new(Bytes::from(response.body))) + .body(Full::new(response.body)) .expect("stub response should build"), ) } diff --git a/crates/openshell-driver-podman/src/watcher.rs b/crates/openshell-driver-podman/src/watcher.rs index f8f97af705..6e5eb52f27 100644 --- a/crates/openshell-driver-podman/src/watcher.rs +++ b/crates/openshell-driver-podman/src/watcher.rs @@ -139,14 +139,16 @@ pub async fn start_watch( let mut event_rx = client.events_stream(LABEL_MANAGED_FILTER).await?; // 2. List existing containers for initial state sync. - let existing = client.list_containers(&[LABEL_MANAGED_FILTER]).await?; + let existing = client + .list_containers(&[LABEL_MANAGED_FILTER, crate::isolation::WORKLOAD_FILTER]) + .await?; for entry in &existing { // For running containers, use inspect to get full state including // health check status — matching the same condition derivation used // for live events. if entry.state == "running" { - match client.inspect_container(&entry.id).await { + match inspect_workload(&client, &entry.id).await { Ok(inspect) => { if let Some(sandbox) = driver_sandbox_from_inspect(&inspect) { if tx.send(Ok(sandbox_event(sandbox))).await.is_err() { @@ -246,11 +248,34 @@ async fn map_podman_event( return None; } + if event + .actor + .attributes + .get(crate::isolation::LABEL_ROLE) + .is_some_and(|role| role == "supervisor") + { + let id_filter = format!("{LABEL_SANDBOX_ID}={sandbox_id}"); + let workloads = client + .list_containers(&[ + LABEL_MANAGED_FILTER, + &id_filter, + crate::isolation::WORKLOAD_FILTER, + ]) + .await + .ok()?; + let workload = workloads.first()?; + return inspect_workload(client, &workload.id) + .await + .ok() + .and_then(|inspect| driver_sandbox_from_inspect(&inspect)) + .map(sandbox_event); + } + match event.action.as_str() { "remove" => Some(deleted_event(sandbox_id.clone())), "create" | "start" | "stop" | "die" | "health_status" => { // Inspect the container to get current state. - match client.inspect_container(container_id).await { + match inspect_workload(client, container_id).await { Ok(inspect) => { if lifecycle_event_fences.matches_previous_exit( event, @@ -327,6 +352,54 @@ async fn map_podman_event( } } +/// A workload is ready only when its independent supervisor is healthy. This +/// check runs both on watch reconciliation and on events, and contains a lost +/// supervisor even when the gateway missed the original exit event. +pub async fn inspect_workload( + client: &PodmanClient, + id: &str, +) -> Result { + let mut workload = client.inspect_container(id).await?; + if workload + .config + .labels + .get(crate::isolation::LABEL_ROLE) + .is_none_or(|role| role != "sandbox") + { + return Ok(workload); + } + let Some(sandbox_id) = workload.config.labels.get(LABEL_SANDBOX_ID) else { + return Ok(workload); + }; + let supervisor = client + .inspect_container(&crate::isolation::supervisor_name(sandbox_id)) + .await; + if workload.state.running { + match supervisor { + Ok(supervisor) if supervisor.state.running => { + workload.state.health = supervisor.state.health; + } + Ok(supervisor) + if supervisor.state.status == "configured" + || supervisor.state.status == "created" => + { + workload.state.health = Some(HealthState { + status: "starting".into(), + }); + } + // Both containers exist before initial start. A missing or exited + // companion therefore requires containment, including after a + // gateway restart that missed the original Podman exit event. + Ok(_) | Err(PodmanApiError::NotFound(_)) => { + client.stop_container(&workload.id, 0).await?; + workload = client.inspect_container(&workload.id).await?; + } + Err(error) => return Err(error), + } + } + Ok(workload) +} + /// Construct a `DriverSandbox` from common fields. /// /// Centralises the boilerplate that every event/inspect/list path shares: @@ -353,6 +426,7 @@ fn build_driver_sandbox( sandbox_fd: String::new(), conditions: vec![condition], deleting, + ..Default::default() }), workspace, } @@ -509,6 +583,65 @@ fn condition_from_state(state: &ContainerState) -> DriverCondition { mod tests { use super::*; + #[tokio::test] + async fn missing_supervisor_stops_workload_during_reconciliation() { + use crate::test_utils::{StubResponse, spawn_podman_stub}; + use hyper::StatusCode; + let (path, requests, handle) = spawn_podman_stub( + "lost-supervisor", + vec![ + StubResponse::new( + StatusCode::OK, + r#"{"Id":"workload","Name":"workload","State":{"Status":"running","Running":true},"Config":{"Labels":{"openshell.ai/sandbox-id":"test","openshell.io/isolation-role":"sandbox"}}}"#, + ), + StubResponse::new(StatusCode::NOT_FOUND, "missing companion"), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new( + StatusCode::OK, + r#"{"Id":"workload","Name":"workload","State":{"Status":"exited","Running":false},"Config":{}}"#, + ), + ], + ); + let client = PodmanClient::new(path.clone()); + let inspected = inspect_workload(&client, "workload").await.unwrap(); + assert!(!inspected.state.running); + handle.await.unwrap(); + assert!( + requests + .lock() + .unwrap() + .iter() + .any(|request| request.ends_with("/libpod/containers/workload/stop?timeout=0")) + ); + let _ = std::fs::remove_file(path); + } + + #[tokio::test] + async fn created_supervisor_keeps_bootstrapping_workload_starting() { + use crate::test_utils::{StubResponse, spawn_podman_stub}; + use hyper::StatusCode; + let (path, requests, handle) = spawn_podman_stub( + "starting-supervisor", + vec![ + StubResponse::new( + StatusCode::OK, + r#"{"Id":"workload","Name":"workload","State":{"Status":"running","Running":true},"Config":{"Labels":{"openshell.ai/sandbox-id":"test","openshell.io/isolation-role":"sandbox"}}}"#, + ), + StubResponse::new( + StatusCode::OK, + r#"{"Id":"supervisor","Name":"supervisor","State":{"Status":"configured","Running":false},"Config":{}}"#, + ), + ], + ); + let client = PodmanClient::new(path.clone()); + let inspected = inspect_workload(&client, "workload").await.unwrap(); + assert!(inspected.state.running); + assert_eq!(inspected.state.health.unwrap().status, "starting"); + handle.await.unwrap(); + assert_eq!(requests.lock().unwrap().len(), 2); + let _ = std::fs::remove_file(path); + } + fn podman_event(action: &str, sandbox_id: &str, time_nano: i64) -> PodmanEvent { PodmanEvent { event_type: "container".to_string(), @@ -761,6 +894,7 @@ mod tests { sandbox_fd: String::new(), conditions: vec![condition], deleting: false, + ..Default::default() }), workspace: String::new(), }; diff --git a/crates/openshell-driver-vm/Cargo.toml b/crates/openshell-driver-vm/Cargo.toml index 0d250b6e71..9c2ef8c81d 100644 --- a/crates/openshell-driver-vm/Cargo.toml +++ b/crates/openshell-driver-vm/Cargo.toml @@ -17,45 +17,88 @@ path = "src/lib.rs" [[bin]] name = "openshell-driver-vm" path = "src/main.rs" +required-features = ["compute-driver"] + +[[bin]] +name = "openshell-vm-init" +path = "src/bin/openshell-vm-init.rs" [dependencies] openshell-core = { path = "../openshell-core", default-features = false } -openshell-otel = { path = "../openshell-otel" } -openshell-policy = { path = "../openshell-policy" } -openshell-driver-podman = { path = "../openshell-driver-podman" } -openshell-vfio = { path = "../openshell-vfio" } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } +openshell-otel = { path = "../openshell-otel", optional = true } +openshell-policy = { path = "../openshell-policy", optional = true } +openshell-driver-podman = { path = "../openshell-driver-podman", optional = true } +openshell-vfio = { path = "../openshell-vfio", optional = true } -bollard = { version = "0.20", features = ["ssh"] } +base64 = { workspace = true, optional = true } +bollard = { version = "0.20", features = ["ssh"], optional = true } tokio = { workspace = true } -tonic = { workspace = true, features = ["transport"] } -tower-http = { workspace = true } -http = { workspace = true } -prost = { workspace = true } -prost-types = { workspace = true } -futures = { workspace = true } -tokio-stream = { workspace = true, features = ["net"] } -nix = { workspace = true } -clap = { workspace = true } -tracing = { workspace = true } -tracing-subscriber = { workspace = true } -opentelemetry = { workspace = true } -opentelemetry_sdk = { workspace = true } -tracing-opentelemetry = { workspace = true } -miette = { workspace = true } -url = { workspace = true } +tonic = { workspace = true, features = ["transport"], optional = true } +tower-http = { workspace = true, optional = true } +http = { workspace = true, optional = true } +prost = { workspace = true, optional = true } +prost-types = { workspace = true, optional = true } +futures = { workspace = true, optional = true } +tokio-stream = { workspace = true, features = ["net"], optional = true } +nix = { workspace = true, optional = true } +clap = { workspace = true, optional = true } +tracing = { workspace = true, optional = true } +tracing-subscriber = { workspace = true, optional = true } +opentelemetry = { workspace = true, optional = true } +opentelemetry_sdk = { workspace = true, optional = true } +tracing-opentelemetry = { workspace = true, optional = true } +miette = { workspace = true, optional = true } +rand = { workspace = true, optional = true } +url = { workspace = true, optional = true } serde = { workspace = true } serde_json = { workspace = true } -oci-client = "0.16" +oci-client = { version = "0.16", optional = true } libc = "0.2" -rustix = { workspace = true } -libloading = "0.8" -tar = "0.4" -flate2 = "1" -sha2 = "0.10" -zstd = "0.13" +rustix = { workspace = true, optional = true } +libloading = { version = "0.8", optional = true } +tar = { version = "0.4", optional = true } +flate2 = { version = "1", optional = true } +sha2 = { version = "0.10", optional = true } +zstd = { version = "0.13", optional = true } [features] -default = ["telemetry"] +default = ["compute-driver", "telemetry"] +## Build the standalone compute driver and its host runtime implementation. +compute-driver = [ + "dep:base64", + "dep:bollard", + "dep:clap", + "dep:flate2", + "dep:futures", + "dep:http", + "dep:libloading", + "dep:miette", + "dep:nix", + "dep:oci-client", + "dep:openshell-otel", + "dep:openshell-policy", + "dep:openshell-driver-podman", + "dep:openshell-vfio", + "dep:opentelemetry", + "dep:opentelemetry_sdk", + "dep:polling", + "dep:prost", + "dep:prost-types", + "dep:rand", + "dep:rustix", + "dep:sha2", + "dep:tar", + "dep:tokio-stream", + "dep:tonic", + "dep:tower-http", + "dep:tracing", + "dep:tracing-opentelemetry", + "dep:tracing-subscriber", + "dep:url", + "dep:zstd", +] ## Compile in telemetry support (forwards to openshell-core/telemetry). On by ## default; build with `--no-default-features` for a telemetry-free VM driver ## that reports telemetry disabled to the sandboxes it launches. @@ -68,7 +111,7 @@ telemetry = ["openshell-core/telemetry"] ## enabling it alongside `telemetry` is a compile error rather than a silent ## telemetry-on build. Kept in sync with `default` by ## `rust:verify:defaults-without-telemetry`. -defaults-without-telemetry = [] +defaults-without-telemetry = ["compute-driver"] [dev-dependencies] openshell-otel-test-support = { path = "../openshell-otel-test-support" } @@ -82,7 +125,7 @@ opentelemetry_sdk = { workspace = true, features = ["testing"] } # nix::sys::prctl::set_pdeathsig there keeps the Linux path a single # syscall with no helper thread. [target.'cfg(any(target_os = "macos", target_os = "ios", target_os = "freebsd", target_os = "netbsd", target_os = "openbsd", target_os = "dragonfly"))'.dependencies] -polling = "3.11" +polling = { version = "3.11", optional = true } [lints] workspace = true diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index e71b11a2e8..0dc41498c2 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -1,33 +1,40 @@ # openshell-driver-vm -> Status: Experimental. The VM compute driver is under active development and the interface still has VM-specific plumbing that will be generalized. +> Status: Experimental. The VM compute driver is under active development. -Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess, talks to it over a Unix domain socket with the `openshell.compute.v1.ComputeDriver` gRPC surface, and lets it manage per-sandbox microVMs. The runtime (libkrun + libkrunfw + gvproxy), guest OCI unpacker, and sandbox supervisor are embedded directly in the binary; each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir. +Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess and talks to it over the `openshell.compute.v1.ComputeDriver` Unix-socket surface. `openshell-supervisor` runs as a native host process, while `openshell-sandbox` runs as capability-free PID 1 inside each microVM and applies guest-local isolation over virtio-vsock. + +The driver embeds libkrun, libkrunfw, the guest OCI unpacker, the portable guest sandbox, and the custom kernel runtime. Each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir. ## How it fits together ```mermaid flowchart LR - subgraph host["Host process"] + subgraph host["Host"] gateway["openshell-gateway
(vm::spawn)"] - driver["openshell-driver-vm
├── libkrun (VM)
├── gvproxy (net)
└── openshell-sandbox.zst"] + driver["openshell-driver-vm
libkrun"] + supervisor["openshell-supervisor
host policy supervisor"] gateway <-->|"gRPC over UDS
compute-driver.sock"| driver + supervisor <-->|"authenticated gRPC
policy + relay"| gateway end subgraph guest["Per-sandbox microVM"] init["/srv/openshell-vm-
sandbox-init.sh"] - supervisor["/opt/openshell/bin/
openshell-sandbox
(PID 1)"] - init --> supervisor + sandbox["openshell-sandbox
capability-free guest PID 1"] + workload["sandbox workload"] + init --> sandbox --> workload end driver -->|"CreateSandbox
boots via libkrun"| guest - supervisor -.->|"gRPC callback
--grpc-endpoint"| gateway + supervisor <-->|"mutual TLS RFC 0012
over virtio-vsock"| sandbox - client["openshell-cli"] -->|"SSH proxy
127.0.0.1:<port>"| supervisor + client["openshell-cli"] -->|"connect / exec / forward"| gateway client -->|"CreateSandbox / Watch"| gateway ``` -Sandbox guests execute `/opt/openshell/bin/openshell-sandbox` as PID 1 inside the VM. gvproxy exposes a single inbound SSH port (`host:` → `guest:2222`) and provides virtio-net egress. +The supervisor owns gateway credentials, admitted policy, provider resolution, middleware, the network proxy, and relay registration. The sandbox receives no gateway JWT. Each VM generation receives distinct sandbox and supervisor channel keys; the guest consumes and unlinks its private bootstrap files before launching the workload. + +VM-specific RFC 0012 code under `src/isolation/` only chooses the vsock transport and binds immutable VM generation and image claims into the protected guest config and host descriptor. Lifecycle, authentication, process control, binary identity, forwarding, and streaming come from `openshell-isolation-interface` and `openshell-sandbox`. ## Quick start (recommended) @@ -35,7 +42,7 @@ Sandbox guests execute `/opt/openshell/bin/openshell-sandbox` as PID 1 inside th mise run gateway:vm ``` -First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/gvproxy/umoci and `mise run vm:supervisor` builds the bundled guest supervisor. Subsequent runs are cached. +First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/umoci and `mise run vm:supervisor` builds the portable Linux guest sandbox plus its small static guest-init helper. The development task also builds the native host supervisor. Subsequent runs are cached. By default `mise run gateway:vm`: @@ -96,13 +103,13 @@ rm -rf "${XDG_CONFIG_HOME:-$HOME/.config}/openshell/gateways/vm-dev" If you want to drive the launch yourself instead of using `mise run gateway:vm` (i.e. `tasks/scripts/gateway-vm.sh`): ```shell -# 1. Stage runtime artifacts + supervisor bundle into target/vm-runtime-compressed/ +# 1. Stage runtime artifacts + guest sandbox into target/vm-runtime-compressed/ mise run vm:setup -mise run vm:supervisor # if openshell-sandbox.zst is not already present +mise run vm:supervisor # builds the Linux guest sandbox and static guest-init helper -# 2. Build both binaries with the staged artifacts embedded +# 2. Build gateway, native host supervisor, and driver OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ - cargo build -p openshell-gateway -p openshell-driver-vm + cargo build -p openshell-gateway -p openshell-supervisor -p openshell-driver-vm # 3. macOS only: codesign the driver for Hypervisor.framework codesign \ @@ -121,8 +128,7 @@ disable_tls = true [openshell.drivers.vm] default_image = "" -# Optional override; the gateway derives host.openshell.internal:18081 when omitted. -grpc_endpoint = "http://host.openshell.internal:18081" +grpc_endpoint = "http://127.0.0.1:18081" driver_dir = "$PWD/target/debug" state_dir = "/tmp/openshell-vm-driver-$USER-vm-dev" EOF @@ -143,8 +149,8 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`, | Configuration key | Default | Purpose | |---|---|---| -| `grpc_endpoint` | topology-derived | Optional override for the URL the sandbox guest dials to reach the gateway. The gateway derives `http(s)://host.openshell.internal:` when absent. Use `host.containers.internal`, `host.docker.internal`, or another routable host only for a non-standard topology. Loopback URLs are rewritten automatically by the driver. The bare gateway IP (`192.168.127.1`) only carries gvproxy's own services and will not reach host-bound ports. | -| `state_dir` | `target/openshell-vm-driver` | Per-sandbox overlay disks, console logs, image cache, and private `run/compute-driver.sock` UDS. | +| `grpc_endpoint` | empty | Required. URL the native host supervisor uses to reach the gateway. Host loopback such as `http://127.0.0.1:` is valid. Legacy guest aliases are normalized to host loopback. This endpoint is never sent into the VM. | +| `state_dir` | `target/openshell-vm-driver` | Per-sandbox overlay disks, console logs, image cache, and private `run/compute-driver.sock` UDS. Relative paths are resolved to absolute paths at driver startup. | | `driver_dir` | unset | Override the directory searched for `openshell-driver-vm`. | | `default_image` | OpenShell base image | Sandbox image used when a create request omits one. | | `bootstrap_image` | unset | VM runtime image used as the immutable bootstrap root disk. Defaults to the sandbox image when unset. | @@ -152,8 +158,10 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`, | `mem_mib` | `2048` | Memory per sandbox, in MiB. | | `overlay_disk_mib` | `4096` | Sparse writable overlay disk size per sandbox, in MiB. | | `krun_log_level` | `1` | libkrun verbosity (0-5). | -| `sandbox_uid` / `sandbox_gid` | image `sandbox` account, otherwise `1000` / UID | Explicit values override the image account; when both are omitted, a supplied image `sandbox` account is preserved and an image without one gets `1000:1000`. Each overlay records its effective UID/GID. During migration, an unmarked overlay recovers identity from its upper layer or prepared rootfs, an explicit override, or the current image. Legacy `10001:10001` is retained only when persisted state reports it. | -| `https_proxy` | unset | Corporate forward proxy (`http://host:port` or `https://host:port`) the in-guest supervisor chains policy-approved TLS CONNECT egress through. On the libkrun backend a proxy on the gateway host's loopback must be addressed as `http://host.openshell.internal:` — guest egress leaves through gvproxy, which NATs `192.168.127.254` to the host's `127.0.0.1`. The QEMU/TAP backend has no such NAT, so a gateway-host proxy URL is rejected before GPU sandbox launch; use an address routable from the guest's masqueraded egress. | +| `guest_tls_ca` | unset | Historical key name for the host supervisor's gateway CA certificate. Required when `grpc_endpoint` uses `https://`; never copied into the guest. | +| `guest_tls_cert` | unset | Historical key name for the host supervisor's client certificate; never copied into the guest. | +| `guest_tls_key` | unset | Historical key name for the host supervisor's client private key; never copied into the guest. | +| `https_proxy` | unset | Corporate forward proxy (`http://host:port` or `https://host:port`) that host control chains policy-approved TLS CONNECT egress through. Host-loopback proxy URLs work because control runs on the gateway host. | | `no_proxy` | unset | Comma-separated bypass list for the corporate proxy only. OpenShell policy evaluation still applies. | | `proxy_auth_file` | unset | Gateway-host path to a validated `user:pass` credential file. Staged root-only into the per-sandbox overlay and removed with the sandbox; credentials never enter logs or process arguments. | | `proxy_auth_allow_insecure` | unset | Required with `proxy_auth_file` against an `http://` proxy: acknowledges that Basic auth is cleartext on the connection to the proxy. | @@ -161,18 +169,7 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`, | `proxy_ca_bundle` | unset | Gateway-host PEM CA bundle trusted for the corporate proxy and TLS-intercepted server certificates. The driver validates it and stages it at a fixed non-secret guest path in the protected overlay. Requires `https_proxy`. | | `provider_spiffe_workload_api_tcp_endpoint` | unset | Explicit guest-reachable `tcp:IP:port` SPIFFE Workload API listener for provider token exchange. It requires `provider_spiffe_allow_guest_tcp = true`; a host UNIX socket is never silently exposed to a VM guest. | -The proxy settings are operator-owned and deployment-level: they are not accepted through `template.driver_config.vm`, and they reach the supervisor through a protected per-sandbox argument file the driver writes into the overlay upperdir on every launch, so a sandbox image cannot forge or shadow them. Every present-but-invalid value is fatal at gateway or sandbox startup rather than degrading to a direct dial. - -For gateway-managed VM drivers, configure `guest_tls_ca`, `guest_tls_cert`, and -`guest_tls_key` together under `[openshell.gateway]`; the gateway validates and -injects that bundle into only the selected local driver. The standalone -`openshell-driver-vm` CLI retains its `--guest-tls-*` inputs for independent -operation. Standalone invocations use `--grpc-endpoint` and the -`--upstream-proxy`, `--upstream-no-proxy`, `--upstream-proxy-auth-file`, -`--upstream-proxy-auth-allow-insecure`, `--upstream-proxy-connect-by-hostname`, -and `--upstream-proxy-ca-bundle` flags. Replace the removed -`--openshell-endpoint`, `--https-proxy`, `--no-proxy`, and `--proxy-*` spellings -in existing scripts. +The proxy settings are operator-owned and deployment-level: they are not accepted through `template.driver_config.vm`, and the driver passes them only to native host control. Every present-but-invalid value is fatal at gateway or sandbox startup rather than degrading to a direct dial. See [`openshell-gateway --help`](../openshell-server/src/cli.rs) for the gateway process flag surface. @@ -230,9 +227,17 @@ marked sandboxes without launching compute. Start removes the marker and uses the normal persisted restore path with the existing overlay. Delete removes the entire sandbox state directory, including a stop marker and overlay. -The driver records a terminal tombstone when the canonical main process exits. -Driver startup reports that sandbox as terminal instead of relaunching the VM, -even when the process exited successfully. +The host control writes and syncs a terminal tombstone when the canonical main +process exits, before it reports completion and while it retains the boundary +for exec and forwarding. Driver startup reports that sandbox as terminal +instead of relaunching the VM, even when the process exited successfully. + +The driver embeds a platform-native host supervisor and extracts it into +`/host-runtime`. It accepts a cached binary only when its SHA-256 +content matches the embedded supervisor and it remains an executable regular +file. Replacement is written and synced under a temporary name, then atomically +renamed into place. `OPENSHELL_VM_SUPERVISOR_BIN` remains an explicit +development override. ## Logs and debugging @@ -244,34 +249,23 @@ RUST_LOG=openshell_server=debug,openshell_driver_vm=debug \ ``` The VM guest's serial console is appended to `//console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions and removes same-owner stale sockets. On clean shutdown, the gateway sends the managed driver `SIGTERM`, waits up to five seconds for it to flush telemetry and exit, then force-kills it if necessary and removes the socket. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development. +The VM serial console is appended to `/sandboxes//rootfs-console.log`. Host-supervisor stdout and stderr are written beside it as `supervisor.log` and `supervisor.err.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions, removes same-owner stale sockets, and the gateway removes the socket on clean shutdown via `ManagedDriverProcess::drop`. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development. -## Host-side nftables rules - -The VM driver creates a per-VM nftables table on the host (`openshell_vm_vmtap_`) with three chains. These rules serve two purposes: NAT infrastructure (required for VM connectivity) and defense-in-depth host isolation. Primary security enforcement — proxy-only egress and bypass detection — is handled by the sandbox supervisor's own nftables rules inside the VM guest. - -**`postrouting` (NAT):** Masquerades outbound VM traffic so it can be routed from the VM's private subnet to the external network. This chain handles forwarded traffic (VM → internet), not traffic destined for the host. - -**`forward` (defense-in-depth):** Accepts all outbound traffic from the VM (security enforcement happens guest-side) and accepts established/related response traffic back to the VM. Drops unsolicited inbound connections to the VM from the broader network. This chain handles forwarded traffic only — packets transiting the host between the TAP interface and other interfaces. - -**`input` (defense-in-depth):** Accepts traffic from the VM to the gateway port on the host. Drops all other traffic from the VM destined for the host itself. This limits what a compromised guest can reach on the host to the gateway service only. - -The `input` and `postrouting` chains handle different traffic paths: `input` covers packets addressed to the host (VM → host), while `postrouting` covers packets the host is forwarding on behalf of the VM (VM → internet). A packet from the VM goes through one path or the other, never both. - -All chains use `policy accept`, so non-TAP traffic is unaffected. Because nftables evaluates multiple base chains on the same hook independently, host firewalls interact with these rules as follows: - -- **Open host (no other firewall):** Our chains are the only filter. The defense-in-depth drop rules block unsolicited inbound and non-gateway host access. Non-TAP traffic passes through. -- **Restrictive host firewall (e.g. firewalld):** The host firewall's chains may additionally drop TAP traffic that our chains accept. A `drop` verdict from any chain is final — our `accept` cannot override it. If VM connectivity fails, verify that the host firewall allows forwarding and input for `vmtap-*` interfaces. +## Network isolation -Each table is created atomically via `nft -f` on VM start and torn down atomically via `nft delete table` when the VM is destroyed. +VM sandboxes boot without a virtual NIC. The guest exposes only the protected +vsock channel used by `openshell-sandbox`; `openshell-supervisor` performs DNS, +policy evaluation, and external networking on the host. The driver does not +create TAP devices or install nftables/iptables rules. ## Prerequisites - macOS on Apple Silicon, or Linux on aarch64/x86_64 with KVM - Rust toolchain -- e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk image creation, identity inspection, and QEMU environment injection. Explicit `sandbox_uid`/`sandbox_gid` values do not remove this runtime prerequisite. -- Guest-supervisor cross-compile toolchain (needed on macOS, and on Linux when host arch ≠ guest arch): - - Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or `x86_64-unknown-linux-gnu` for an amd64 guest) - - `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` uses `cargo zigbuild` to cross-compile the in-VM `openshell-sandbox` supervisor binary. +- e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk image creation and QEMU environment injection +- Guest-sandbox cross-compile toolchain (needed on macOS, and on Linux when host arch differs from the guest): + - Matching static guest target: `rustup target add aarch64-unknown-linux-musl` (or `x86_64-unknown-linux-musl` for an amd64 guest) + - `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` cross-compiles the Linux guest `openshell-sandbox` and its matching `openshell-supervisor`. - [mise](https://mise.jdx.dev/) task runner - Docker or Podman socket on the local CLI/gateway host when building an image before `openshell sandbox create --from `; the VM driver exports the @@ -302,11 +296,11 @@ The RPM gateway package is configured for the Podman driver. On Apple Silicon macOS, `install.sh` stages the generated `openshell.rb` formula from the selected release in the `nvidia/openshell` Homebrew tap. -Homebrew installs `openshell`, `openshell-gateway`, and -`openshell-driver-vm`, ad-hoc signs the driver with the Hypervisor entitlement -in `post_install`, and owns the `brew services` gateway lifecycle. The service -also leaves `OPENSHELL_COMPUTE_DRIVER` unset so driver choice remains automatic unless -the user explicitly overrides it. +Homebrew installs `openshell`, `openshell-gateway`, and the self-contained +`openshell-driver-vm` with its embedded native supervisor. It ad-hoc signs the +driver with the Hypervisor entitlement in `post_install` and owns the `brew +services` gateway lifecycle. The service also leaves `OPENSHELL_DRIVERS` unset +so driver choice remains automatic unless the user explicitly overrides it. ## TODOs diff --git a/crates/openshell-driver-vm/build.rs b/crates/openshell-driver-vm/build.rs index 92532ed7b2..92525927d8 100644 --- a/crates/openshell-driver-vm/build.rs +++ b/crates/openshell-driver-vm/build.rs @@ -3,13 +3,17 @@ //! Build script for openshell-driver-vm. //! -//! This crate embeds the sandbox supervisor plus the minimal libkrun runtime +//! This crate embeds the sandbox, host supervisor, and minimal libkrun runtime //! artifacts it needs to boot VMs without a separate VM runtime binary. use std::path::{Path, PathBuf}; use std::{env, fs}; fn main() { + if env::var_os("CARGO_FEATURE_COMPUTE_DRIVER").is_none() { + return; + } + println!("cargo:rerun-if-env-changed=OPENSHELL_VM_RUNTIME_COMPRESSED_DIR"); if let Ok(dir) = env::var("OPENSHELL_VM_RUNTIME_COMPRESSED_DIR") { @@ -19,8 +23,9 @@ fn main() { "libkrunfw.so.5.zst", "libkrun.dylib.zst", "libkrunfw.5.dylib.zst", - "gvproxy.zst", "openshell-sandbox.zst", + "openshell-supervisor.zst", + "openshell-vm-init.zst", "umoci.zst", ] { println!("cargo:rerun-if-changed={dir}/{name}"); @@ -38,7 +43,14 @@ fn main() { println!("cargo:warning=VM runtime not available for {target_os}-{target_arch}"); generate_stub_resources( &out_dir, - &["libkrun", "libkrunfw", "openshell-sandbox.zst", "umoci.zst"], + &[ + "libkrun", + "libkrunfw", + "openshell-sandbox.zst", + "openshell-supervisor.zst", + "openshell-vm-init.zst", + "umoci.zst", + ], ); return; } @@ -54,8 +66,9 @@ fn main() { &[ &format!("{libkrun_name}.zst"), &format!("{libkrunfw_name}.zst"), - "gvproxy.zst", "openshell-sandbox.zst", + "openshell-supervisor.zst", + "openshell-vm-init.zst", "umoci.zst", ], ); @@ -74,11 +87,18 @@ fn main() { format!("{libkrunfw_name}.zst"), format!("{libkrunfw_name}.zst"), ), - ("gvproxy.zst".to_string(), "gvproxy.zst".to_string()), ( "openshell-sandbox.zst".to_string(), "openshell-sandbox.zst".to_string(), ), + ( + "openshell-supervisor.zst".to_string(), + "openshell-supervisor.zst".to_string(), + ), + ( + "openshell-vm-init.zst".to_string(), + "openshell-vm-init.zst".to_string(), + ), ("umoci.zst".to_string(), "umoci.zst".to_string()), ]; diff --git a/crates/openshell-driver-vm/runtime/README.md b/crates/openshell-driver-vm/runtime/README.md index b686874ba2..bb02232f92 100644 --- a/crates/openshell-driver-vm/runtime/README.md +++ b/crates/openshell-driver-vm/runtime/README.md @@ -11,15 +11,18 @@ runtime/ openshell.kconfig ``` -`openshell-driver-vm` embeds libkrun, libkrunfw, gvproxy, umoci for guest-side -OCI image unpacking, and the bundled `openshell-sandbox` supervisor. +`openshell-driver-vm` embeds libkrun, libkrunfw, umoci for guest-side OCI image +unpacking, and the portable capability-free `openshell-sandbox` role. +VMs do not attach a guest NIC. The boundary carries control, mediated network, +and DNS streams over the authenticated vsock channel. ## Why -The stock `libkrunfw` kernel does not include the bridge, netfilter, -conntrack, cgroup, seccomp, and Landlock features the sandbox supervisor needs -inside each microVM. `kernel/openshell.kconfig` extends the libkrunfw kernel so -VM sandboxes can run the same supervisor enforcement path as other backends. +The stock `libkrunfw` kernel does not include every cgroup, seccomp, and +Landlock feature the sandbox needs inside each microVM. +`kernel/openshell.kconfig` extends the libkrunfw kernel so VM sandboxes retain +guest-local process, network-syscall, and filesystem enforcement while the +supervisor runs on the host. ## Build Scripts @@ -27,7 +30,7 @@ VM sandboxes can run the same supervisor enforcement path as other backends. |---|---|---| | `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw and libkrun from source with the custom kernel config | | `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Builds portable libkrunfw and libkrun from a prebuilt `kernel.c` | -| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages `vm-runtime-.tar.zst` with libraries, gvproxy, umoci, and provenance | +| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages `vm-runtime-.tar.zst` with libraries, umoci, and provenance | | `tasks/scripts/vm/download-kernel-runtime.sh` | Any | Downloads runtime tarballs from the `vm-runtime` release and stages compressed files | ## Local Flow @@ -36,12 +39,12 @@ VM sandboxes can run the same supervisor enforcement path as other backends. # Download the current pre-built runtime and stage compressed artifacts mise run vm:setup -# Build the bundled guest supervisor +# Build the portable Linux guest sandbox and static guest-init helper mise run vm:supervisor -# Build the gateway and VM driver with embedded runtime artifacts +# Build the gateway, native host supervisor, and VM driver OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ - cargo build -p openshell-gateway -p openshell-driver-vm + cargo build -p openshell-gateway -p openshell-supervisor -p openshell-driver-vm ``` Use `FROM_SOURCE=1 mise run vm:setup` to build the runtime from source instead @@ -62,7 +65,7 @@ publish the driver binary next to `openshell-gateway`. ## Provenance `package-vm-runtime.sh` writes `provenance.json` into each runtime tarball with -the platform, libkrunfw commit, kernel version, gvproxy and umoci versions, +the platform, libkrunfw commit, kernel version, and umoci version, GitHub SHA, and build time. The driver logs this metadata when it extracts and loads a runtime bundle. diff --git a/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig b/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig index e8d826c53e..4249e71121 100644 --- a/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig +++ b/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig @@ -1,136 +1,37 @@ # Custom kernel config fragment for libkrunfw (OpenShell VM) # -# This fragment is applied on top of libkrunfw's base kernel config -# to enable bridge CNI, netfilter/iptables, and conntrack support -# required for Kubernetes pod networking in the VM. +# This fragment is applied on top of libkrunfw's base kernel config. VM +# sandboxes have no virtual NIC; the base loopback, TCP/UDP, Unix-socket, +# route-netlink, and vsock support is enough for the capability-free sandbox. # # Apply with: scripts/merge-kconfig.sh # # See also: check-vm-capabilities.sh for runtime verification. -# ── Root disk transport and filesystem ───────────────────────────────── +# Root disk transport and filesystem. CONFIG_BLOCK=y CONFIG_BLK_DEV=y CONFIG_VIRTIO_BLK=y CONFIG_EXT4_FS=y CONFIG_EXT4_USE_FOR_EXT2=y -# ── Network Namespaces (required for pod isolation) ───────────────────── -CONFIG_NET_NS=y -CONFIG_NAMESPACES=y - -# ── Virtual Ethernet (veth pairs for pod networking) ──────────────────── -CONFIG_VETH=y - -# ── Linux Bridge (required for bridge CNI plugin) ────────────────────── -CONFIG_BRIDGE=y -CONFIG_BRIDGE_NETFILTER=y -CONFIG_BRIDGE_IGMP_SNOOPING=y - -# ── Netfilter framework ──────────────────────────────────────────────── -CONFIG_NETFILTER=y -CONFIG_NETFILTER_ADVANCED=y -CONFIG_NETFILTER_INGRESS=y -CONFIG_NETFILTER_NETLINK=y -CONFIG_NETFILTER_NETLINK_QUEUE=y -CONFIG_NETFILTER_NETLINK_LOG=y - -# ── Connection tracking (required for NAT and kube-proxy) ────────────── -CONFIG_NF_CONNTRACK=y -CONFIG_NF_CT_NETLINK=y -CONFIG_NF_CONNTRACK_EVENTS=y -CONFIG_NF_CONNTRACK_TIMEOUT=y -CONFIG_NF_CONNTRACK_TIMESTAMP=y - -# ── Netfilter xtables match modules (required by kube-proxy & kubelet) ─ -# kube-proxy uses xt_conntrack for stateful rules and xt_comment for -# labeling chains. Without these, iptables fails with: -# "Couldn't load match 'conntrack': No such file or directory" -CONFIG_NETFILTER_XTABLES=y -CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y -CONFIG_NETFILTER_XT_MATCH_COMMENT=y -CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y -CONFIG_NETFILTER_XT_MATCH_MARK=y -CONFIG_NETFILTER_XT_MATCH_STATISTIC=y -CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y -CONFIG_NETFILTER_XT_MATCH_RECENT=y -CONFIG_NETFILTER_XT_MATCH_LIMIT=y -CONFIG_NETFILTER_XT_TARGET_LOG=y -CONFIG_NETFILTER_XT_TARGET_MARK=y -CONFIG_NETFILTER_XT_TARGET_CONNMARK=y -CONFIG_NETFILTER_XT_MATCH_CONNMARK=y - -# ── NAT (required for service VIP / DNAT / SNAT) ────────────────────── -CONFIG_NF_NAT=y -CONFIG_NF_NAT_MASQUERADE_IPV4=y - -# ── iptables (CNI bridge masquerade + compat) ────────────────────────── -CONFIG_IP_NF_IPTABLES=y -CONFIG_IP_NF_FILTER=y -CONFIG_IP_NF_NAT=y -CONFIG_IP_NF_MANGLE=y -CONFIG_IP_NF_TARGET_MASQUERADE=y -CONFIG_IP_NF_TARGET_REJECT=y - -# ── nftables (kube-proxy nftables mode — primary proxy backend) ───────── -# kube-proxy nftables proxier requires: numgen (random LB), fib (local -# address detection), counter, ct, nat, masq, reject, limit, redir. -CONFIG_NF_TABLES=y -CONFIG_NF_TABLES_INET=y -CONFIG_NFT_CT=y -CONFIG_NFT_NAT=y -CONFIG_NFT_MASQ=y -CONFIG_NFT_REJECT=y -CONFIG_NFT_COMPAT=y -CONFIG_NFT_NUMGEN=y -CONFIG_NFT_FIB_IPV4=y -CONFIG_NFT_FIB_IPV6=y -CONFIG_NFT_LIMIT=y -CONFIG_NFT_LOG=y -CONFIG_NFT_REDIR=y -CONFIG_NFT_TPROXY=y - -# ── IP forwarding and routing (required for pod-to-pod) ──────────────── -CONFIG_IP_ADVANCED_ROUTER=y -CONFIG_IP_MULTIPLE_TABLES=y -CONFIG_IP_ROUTE_MULTIPATH=y -CONFIG_NET_IP_TUNNEL=y - -# ── IPVS (optional: kube-proxy IPVS mode) ───────────────────────────── -CONFIG_IP_VS=y -CONFIG_IP_VS_PROTO_TCP=y -CONFIG_IP_VS_PROTO_UDP=y -CONFIG_IP_VS_RR=y -CONFIG_IP_VS_WRR=y -CONFIG_IP_VS_SH=y -CONFIG_IP_VS_NFCT=y - -# ── Misc networking required by Kubernetes ───────────────────────────── -CONFIG_NET_SCH_HTB=y -CONFIG_NET_CLS_CGROUP=y -CONFIG_CGROUP_NET_PRIO=y -CONFIG_CGROUP_NET_CLASSID=y - -# ── Dummy interface (fallback networking) ────────────────────────────── -CONFIG_DUMMY=y - -# ── TUN/TAP (used by some CNI plugins) ──────────────────────────────── -CONFIG_TUN=y - -# ── Cgroups (already in base, ensure v2 is available) ────────────────── +# Cgroups used for process supervision and resource limits. CONFIG_CGROUPS=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_PIDS=y CONFIG_MEMCG=y -# ── Disable kernel headers archive (avoids cpio issues in CI) ────────── +# Disable the kernel headers archive to avoid cpio issues in CI. # CONFIG_IKHEADERS is not set -# ── POSIX message queues (required by runc to mount /dev/mqueue in containers) ─ +# POSIX message queues used by OCI workloads. CONFIG_POSIX_MQUEUE=y CONFIG_POSIX_MQUEUE_SYSCTL=y -# ── Security features required by the sandbox runtime ─────────────────── +# Capability-free sandbox enforcement. +CONFIG_SECURITY=y CONFIG_SECURITY_LANDLOCK=y +CONFIG_LSM="landlock,lockdown,yama,loadpin,safesetid,ipe,bpf" +CONFIG_SECCOMP=y CONFIG_SECCOMP_FILTER=y diff --git a/crates/openshell-driver-vm/runtime/pins.env b/crates/openshell-driver-vm/runtime/pins.env index 34a9f0bf33..9977eddcc1 100644 --- a/crates/openshell-driver-vm/runtime/pins.env +++ b/crates/openshell-driver-vm/runtime/pins.env @@ -29,10 +29,6 @@ COMMUNITY_SANDBOX_IMAGE="${COMMUNITY_SANDBOX_IMAGE:-ghcr.io/nvidia/openshell-com # during local development. Override via IMAGE_REPO_BASE and IMAGE_TAG # environment variables (defaults: openshell/gateway:dev). -# ── gvproxy (networking proxy) ────────────────────────────────────────── -# Repo: https://github.com/containers/gvisor-tap-vsock -GVPROXY_VERSION="${GVPROXY_VERSION:-v0.8.9}" - # ── umoci (guest OCI unpacker) ────────────────────────────────────────── # Repo: https://github.com/opencontainers/umoci UMOCI_VERSION="${UMOCI_VERSION:-v0.6.0}" diff --git a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh index df8fc3dad6..0f63316195 100644 --- a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh +++ b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh @@ -3,9 +3,9 @@ # SPDX-License-Identifier: Apache-2.0 # Minimal init for sandbox VMs. Runs as PID 1 inside the guest, mounts the -# essential filesystems, configures networking (gvproxy DHCP or TAP static), -# optionally loads NVIDIA GPU drivers, then execs the OpenShell sandbox -# supervisor. +# essential filesystems, optionally loads NVIDIA GPU drivers, then execs the +# portable VM sandbox. Workload networking crosses the authenticated +# boundary channel; the VM does not receive a network interface. set -euo pipefail @@ -14,26 +14,9 @@ set -euo pipefail unset KRUN_INIT_PID1 BOOT_START=$(date +%s%3N 2>/dev/null || date +%s) -# gvisor-tap-vsock subnet layout: -# 192.168.127.1 — gateway: gvproxy's DNS / DHCP / HTTP API. Does NOT -# proxy arbitrary host ports. -# 192.168.127.254 — host-loopback: NAT-rewritten to host's 127.0.0.1 by -# gvproxy's TCP/UDP/ICMP forwarder. Use this address -# (or any of the host.* hostnames below) to reach a -# service the host is listening on. -# The host.openshell.internal / host.containers.internal / -# host.docker.internal DNS records served by gvproxy's embedded resolver -# point at 192.168.127.254. We mirror that in /etc/hosts so the supervisor -# can reach the gateway even when gvproxy's DNS is not in resolv.conf -# (e.g. DHCP failed and we fell back to 8.8.8.8). -GVPROXY_GATEWAY_IP="192.168.127.1" -GVPROXY_HOST_LOOPBACK_IP="192.168.127.254" -GATEWAY_IP="$GVPROXY_GATEWAY_IP" +SANDBOX_OWNER_NORMALIZED_MARKER="/opt/openshell/.sandbox-owner-normalized" GPU_ENABLED="${GPU_ENABLED:-false}" -VM_NET_IP="${VM_NET_IP:-}" -VM_NET_GW="${VM_NET_GW:-}" -VM_NET_DNS="${VM_NET_DNS:-}" ts() { local now @@ -133,6 +116,10 @@ ensure_target_runtime() { cp /opt/openshell/bin/openshell-sandbox "$image_root/opt/openshell/bin/openshell-sandbox" chmod 0755 "$image_root/opt/openshell/bin/openshell-sandbox" fi + if [ -x /opt/openshell/bin/openshell-vm-init ]; then + cp /opt/openshell/bin/openshell-vm-init "$image_root/opt/openshell/bin/openshell-vm-init" + chmod 0755 "$image_root/opt/openshell/bin/openshell-vm-init" + fi touch "$image_root/etc/passwd" "$image_root/etc/group" "$image_root/etc/shadow" "$image_root/etc/gshadow" if [ "$replace_account" -eq 1 ]; then @@ -208,67 +195,6 @@ prepare_guest_image_rootfs() { rm -rf "$payload_dir" } -# Driver-owned arguments appended to the supervisor's command line. -# -# The VM driver cannot build the supervisor's argv the way the container -# drivers do, so it writes the arguments it chose into the overlay upperdir -# and this script appends them verbatim. Populated by -# read_supervisor_extra_args; empty until then. -SUPERVISOR_EXTRA_ARGS=() - -# Upper bound on driver-supplied supervisor arguments. -# -# The corporate proxy settings are the only producer today and top out at ten -# entries. The cap exists so a corrupt or oversized file cannot expand into an -# unbounded command line. -SUPERVISOR_EXTRA_ARGS_MAX=32 - -read_supervisor_extra_args() { - # Read the driver-authored supervisor argument list, one argument per - # line, verbatim -- no word splitting, globbing, or expansion, so values - # containing spaces (e.g. a NO_PROXY list) survive intact. - # - # Security: this is the operator-owned egress boundary. The driver writes - # this file into the overlay upperdir on every launch, including an empty - # file when it has no arguments to pass, so the upperdir copy always - # shadows the read-only image layer. A sandbox image can therefore neither - # supply its own supervisor arguments by baking a file at this path nor - # disable the operator's by omitting one. A missing file means the driver - # passed nothing; a file it cannot read means the overlay is broken, and - # we fail closed rather than start a supervisor with a silently truncated - # egress configuration. - local args_file - args_file="$(root_path /opt/openshell/supervisor-args)" - - SUPERVISOR_EXTRA_ARGS=() - if [ ! -f "$args_file" ]; then - return 0 - fi - if [ ! -r "$args_file" ]; then - ts "FATAL: supervisor argument list ${args_file} is not readable" - exit 1 - fi - - local arg - while IFS= read -r arg; do - # render_guest_supervisor_args never emits a blank line, so one means - # the file was truncated or tampered with after the driver wrote it. - if [ -z "$arg" ]; then - ts "FATAL: empty entry in supervisor argument list" - exit 1 - fi - if [ "${#SUPERVISOR_EXTRA_ARGS[@]}" -ge "$SUPERVISOR_EXTRA_ARGS_MAX" ]; then - ts "FATAL: supervisor argument list exceeds ${SUPERVISOR_EXTRA_ARGS_MAX} entries" - exit 1 - fi - SUPERVISOR_EXTRA_ARGS+=("$arg") - done < "$args_file" - - if [ "${#SUPERVISOR_EXTRA_ARGS[@]}" -gt 0 ]; then - ts "supervisor arguments from driver: ${#SUPERVISOR_EXTRA_ARGS[@]} entries" - fi -} - exec_supervisor_in_newroot() { local chroot_bin local bootstrap="/.openshell-bootstrap" @@ -369,170 +295,6 @@ setup_overlay_root() { run_post_overlay_setup } -parse_endpoint() { - local endpoint="$1" - local scheme rest authority path host port - - case "$endpoint" in - *://*) - scheme="${endpoint%%://*}" - rest="${endpoint#*://}" - ;; - *) - return 1 - ;; - esac - - authority="${rest%%/*}" - path="${rest#"$authority"}" - if [ "$path" = "$rest" ]; then - path="" - fi - - if [[ "$authority" =~ ^\[([^]]+)\]:(.+)$ ]]; then - host="${BASH_REMATCH[1]}" - port="${BASH_REMATCH[2]}" - elif [[ "$authority" =~ ^\[([^]]+)\]$ ]]; then - host="${BASH_REMATCH[1]}" - port="" - elif [[ "$authority" == *:* ]]; then - host="${authority%%:*}" - port="${authority##*:}" - else - host="$authority" - port="" - fi - - if [ -z "$port" ]; then - case "$scheme" in - https) port="443" ;; - *) port="80" ;; - esac - fi - - printf '%s\n%s\n%s\n%s\n' "$scheme" "$host" "$port" "$path" -} - -tcp_probe() { - local host="$1" - local port="$2" - - if command -v timeout >/dev/null 2>&1; then - timeout 2 bash -c "exec 3<>/dev/tcp/\$1/\$2" _ "$host" "$port" >/dev/null 2>&1 - else - bash -c "exec 3<>/dev/tcp/\$1/\$2" _ "$host" "$port" >/dev/null 2>&1 - fi -} - -ensure_host_gateway_aliases() { - # Seed /etc/hosts with the well-known gvproxy hostnames so the supervisor - # can reach the OpenShell server even when gvproxy's built-in DNS is not - # in resolv.conf (e.g. when DHCP fails and we fall back to 8.8.8.8). - # - # Critical distinction: host.* aliases point at the gvproxy *host-loopback* - # IP (192.168.127.254), not the gateway IP (192.168.127.1). Only the - # host-loopback IP carries NAT rewriting to the host's 127.0.0.1 — the - # gateway IP only listens on gvproxy's own service ports (DNS:53, DHCP, - # HTTP API:80). Pinning host.containers.internal to the gateway IP - # silently breaks guest→host port reachability for arbitrary ports. - local host_aliases="host.openshell.internal host.containers.internal host.docker.internal" - local gateway_aliases="gateway.containers.internal" - local filter='(^|[[:space:]])(host\.openshell\.internal|host\.containers\.internal|host\.docker\.internal|gateway\.containers\.internal)([[:space:]]|$)' - - write_host_gateway_aliases "$(root_path /etc/hosts)" "$(root_path "/tmp/openshell-hosts.$$.tmp")" || true - if [ -n "${ROOT_PREFIX:-}" ]; then - write_host_gateway_aliases "/etc/hosts" "/tmp/openshell-hosts.$$.tmp" || true - fi -} - -write_host_gateway_aliases() { - local hosts_path="$1" - local hosts_tmp="$2" - mkdir -p "$(dirname "$hosts_path")" 2>/dev/null || true - mkdir -p "$(dirname "$hosts_tmp")" 2>/dev/null || true - if [ -f "$hosts_path" ]; then - grep -vE "$filter" "$hosts_path" > "$hosts_tmp" || true - else - : > "$hosts_tmp" - fi - - # In TAP/GPU mode, GATEWAY_IP is overridden to VM_NET_GW (the host-side - # of the TAP), and the gateway is reachable directly there. In gvproxy - # mode, host.openshell.internal etc. need GVPROXY_HOST_LOOPBACK_IP - # (192.168.127.254) which is gvproxy's host-NAT entry, while - # gateway.containers.internal points at the gvproxy gateway itself. - if [ "${GATEWAY_IP}" = "${GVPROXY_GATEWAY_IP}" ]; then - printf '%s %s\n' "$GVPROXY_HOST_LOOPBACK_IP" "$host_aliases" >> "$hosts_tmp" - printf '%s %s\n' "$GVPROXY_GATEWAY_IP" "$gateway_aliases" >> "$hosts_tmp" - else - # TAP networking: gateway and host are both reachable at GATEWAY_IP. - printf '%s %s %s\n' "$GATEWAY_IP" "$host_aliases" "$gateway_aliases" >> "$hosts_tmp" - fi - if ! cat "$hosts_tmp" > "$hosts_path" 2>/dev/null; then - rm -f "$hosts_tmp" - ts "WARNING: could not update ${hosts_path}" - return 1 - fi - rm -f "$hosts_tmp" -} - -rewrite_openshell_endpoint_if_needed() { - local endpoint="${OPENSHELL_ENDPOINT:-}" - [ -n "$endpoint" ] || return 0 - - local parsed - if ! parsed="$(parse_endpoint "$endpoint")"; then - ts "WARNING: could not parse OPENSHELL_ENDPOINT=$endpoint" - return 0 - fi - - local scheme host port path - scheme="$(printf '%s\n' "$parsed" | sed -n '1p')" - host="$(printf '%s\n' "$parsed" | sed -n '2p')" - port="$(printf '%s\n' "$parsed" | sed -n '3p')" - path="$(printf '%s\n' "$parsed" | sed -n '4p')" - - if tcp_probe "$host" "$port"; then - return 0 - fi - - # Probe candidates in preference order. Hostnames first for informative - # log output, then a bare IP as a final safety net. In gvproxy mode the - # bare IP is the host-loopback (192.168.127.254). In TAP/GPU mode it's - # the TAP host gateway. - local fallback_ip="$GVPROXY_HOST_LOOPBACK_IP" - if [ "${GATEWAY_IP}" != "${GVPROXY_GATEWAY_IP}" ]; then - fallback_ip="$GATEWAY_IP" - fi - local candidates="host.openshell.internal host.containers.internal host.docker.internal" - if [ "$scheme" != "https" ]; then - candidates="${candidates} ${fallback_ip}" - fi - - for candidate in $candidates; do - if [ "$candidate" = "$host" ]; then - continue - fi - if tcp_probe "$candidate" "$port"; then - local authority="$candidate" - if ! { [ "$scheme" = "http" ] && [ "$port" = "80" ]; } \ - && ! { [ "$scheme" = "https" ] && [ "$port" = "443" ]; }; then - authority="${authority}:${port}" - fi - export OPENSHELL_ENDPOINT="${scheme}://${authority}${path}" - ts "rewrote OPENSHELL_ENDPOINT to ${OPENSHELL_ENDPOINT}" - return 0 - fi - done - - if [ "$scheme" = "https" ]; then - ts "WARNING: could not preflight HTTPS OpenShell endpoint ${host}:${port}; preserving hostname for TLS verification" - return 0 - fi - - ts "WARNING: could not reach OpenShell endpoint ${host}:${port}" -} - create_gpu_device_nodes_mknod() { # Mode 666 is intentional: single-tenant microVM with the VM itself as the # isolation boundary. The sandbox user is the only non-root user. @@ -801,117 +563,28 @@ run_post_overlay_setup() { mount -t cgroup2 cgroup2 "$(root_path /sys/fs/cgroup)" 2>/dev/null & wait - # Allow nftables LOG rules to work in non-init network namespaces. - # Without this, the kernel's nf_log_syslog silently suppresses output - # from the sandbox's network namespace. - if [ -f /proc/sys/net/netfilter/nf_log_all_netns ]; then - echo 1 > /proc/sys/net/netfilter/nf_log_all_netns 2>/dev/null || true - fi - reconcile_sandbox_account setup_sandbox_workdir configure_hostname - ip link set lo up 2>/dev/null || true - -# Networking: use TAP static config if VM_NET_IP is set (QEMU path), -# otherwise fall back to gvproxy DHCP on eth0 (libkrun path). -if [ -n "${VM_NET_IP}" ] && [ -n "${VM_NET_GW}" ]; then - ts "configuring TAP networking (static ${VM_NET_IP} gw ${VM_NET_GW})" - GATEWAY_IP="${VM_NET_GW}" - - TAP_NIC="" - NIC_WAIT=0 - while [ -z "$TAP_NIC" ] && [ "$NIC_WAIT" -lt 10 ]; do - for candidate in eth0 ens3 enp0s2; do - if ip link show "$candidate" >/dev/null 2>&1 && [ "$candidate" != "lo" ]; then - TAP_NIC="$candidate" - break - fi - done - if [ -z "$TAP_NIC" ]; then - for sys_nic in /sys/class/net/*; do - [ -e "$sys_nic" ] || continue - candidate="${sys_nic##*/}" - if ip link show "$candidate" >/dev/null 2>&1 && [ "$candidate" != "lo" ]; then - TAP_NIC="$candidate" - break - fi - done - fi - if [ -z "$TAP_NIC" ]; then - sleep 1 - NIC_WAIT=$((NIC_WAIT + 1)) - fi - done - - if [ -n "$TAP_NIC" ]; then - ts "using NIC ${TAP_NIC} for TAP networking" - ip link set "$TAP_NIC" up 2>/dev/null || true - ip addr add "${VM_NET_IP}/30" dev "$TAP_NIC" 2>/dev/null || true - ip route add default via "${VM_NET_GW}" 2>/dev/null || true - else - ts "WARNING: no network interface found for TAP networking" - fi - - if [ -n "${VM_NET_DNS}" ]; then - echo "nameserver ${VM_NET_DNS}" > "$(root_path /etc/resolv.conf)" - elif [ ! -s "$(root_path /etc/resolv.conf)" ]; then - echo "nameserver 8.8.8.8" > "$(root_path /etc/resolv.conf)" - echo "nameserver 8.8.4.4" >> "$(root_path /etc/resolv.conf)" - fi - - ensure_host_gateway_aliases -elif ip link show eth0 >/dev/null 2>&1; then - ts "detected eth0 (gvproxy networking)" - ip link set eth0 up 2>/dev/null || true - - if command -v udhcpc >/dev/null 2>&1; then - UDHCPC_SCRIPT="$(root_path /run/openshell-udhcpc.script)" - mkdir -p "$(dirname "$UDHCPC_SCRIPT")" - cat > "$UDHCPC_SCRIPT" <<'DHCP_SCRIPT' -#!/bin/sh -case "$1" in - bound|renew) - ip addr flush dev "$interface" - ip addr add "$ip/$mask" dev "$interface" - if [ -n "$router" ]; then - ip route add default via "$router" dev "$interface" - fi - if [ -n "$dns" ]; then - resolv_conf="${OPENSHELL_RESOLV_CONF:-/etc/resolv.conf}" - mkdir -p "$(dirname "$resolv_conf")" 2>/dev/null || true - : > "$resolv_conf" 2>/dev/null || true - for d in $dns; do - echo "nameserver $d" >> "$resolv_conf" 2>/dev/null || true - done - fi - ;; -esac -DHCP_SCRIPT - chmod +x "$UDHCPC_SCRIPT" - - if ! OPENSHELL_RESOLV_CONF="$(root_path /etc/resolv.conf)" \ - udhcpc -i eth0 -f -q -n -T 1 -t 3 -A 1 -s "$UDHCPC_SCRIPT" 2>&1; then - ts "WARNING: DHCP failed, falling back to static config" - ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true - ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true - fi - else - ts "no DHCP client, using static config" - ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true - ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true + if ! /opt/openshell/bin/openshell-vm-init prepare-network; then + ts "FATAL: failed to bring up the loopback interface" + exit 1 fi - if [ ! -s "$(root_path /etc/resolv.conf)" ]; then - echo "nameserver 8.8.8.8" > "$(root_path /etc/resolv.conf)" - echo "nameserver 8.8.4.4" >> "$(root_path /etc/resolv.conf)" + # The capability-free sandbox owns a loopback-only DNS relay. Guest init + # grants the low port before handing control to the zero-capability UID. + if ! echo 0 > /proc/sys/net/ipv4/ip_unprivileged_port_start; then + ts "FATAL: failed to permit the unprivileged DNS relay to bind port 53" + exit 1 fi + cat >"$(root_path /etc/resolv.conf)" <<'EOF' +nameserver 127.0.0.53 +options timeout:2 attempts:2 +EOF - ensure_host_gateway_aliases -else - ts "WARNING: no network interface found; supervisor will start without guest egress" -fi +# The boundary transport mediates network and DNS requests. Only loopback is +# configured in the guest; no public resolver or guest NIC is needed. export HOME=/sandbox export USER=sandbox @@ -938,34 +611,42 @@ fi run_openshell_init_dropins -rewrite_openshell_endpoint_if_needed - -# Log supervisor connectivity state for debugging stuck-in-Provisioning issues -if [ -n "${OPENSHELL_ENDPOINT:-}" ]; then - _ep_parsed="$(parse_endpoint "$OPENSHELL_ENDPOINT" 2>/dev/null || true)" - if [ -n "$_ep_parsed" ]; then - _ep_host="$(printf '%s\n' "$_ep_parsed" | sed -n '2p')" - _ep_port="$(printf '%s\n' "$_ep_parsed" | sed -n '3p')" - if tcp_probe "$_ep_host" "$_ep_port"; then - ts "gateway reachable at ${_ep_host}:${_ep_port}" - else - ts "WARNING: gateway NOT reachable at ${_ep_host}:${_ep_port} — supervisor may fail to connect" - fi - fi - ts "OPENSHELL_ENDPOINT=${OPENSHELL_ENDPOINT}" -fi if [ -n "${OPENSHELL_SANDBOX_ID:-}" ]; then ts "OPENSHELL_SANDBOX_ID=${OPENSHELL_SANDBOX_ID}" fi -read_supervisor_extra_args -set -- --workdir /sandbox "${SUPERVISOR_EXTRA_ARGS[@]+"${SUPERVISOR_EXTRA_ARGS[@]}"}" - -ts "starting openshell-sandbox supervisor" +ts "starting OpenShell VM sandbox" +_sandbox_owner="$(sandbox_owner)" +_sandbox_uid="${_sandbox_owner%%:*}" +_sandbox_gid="${_sandbox_owner##*:}" +_sandbox_bootstrap_guest="${OPENSHELL_VM_SANDBOX_BOOTSTRAP:-/.openshell/state/bootstrap.json}" +_sandbox_bootstrap="$(root_path "$_sandbox_bootstrap_guest")" +_sandbox_state_dir="${_sandbox_bootstrap%/*}" +if [ ! -f "$_sandbox_bootstrap" ]; then + ts "FATAL: capability-free sandbox bootstrap is missing" + exit 1 +fi +chown "${_sandbox_uid}:${_sandbox_gid}" "$_sandbox_state_dir" +chmod 0700 "$_sandbox_state_dir" +for _sandbox_private_file in "$_sandbox_state_dir"/*; do + [ -f "$_sandbox_private_file" ] || continue + chown "${_sandbox_uid}:${_sandbox_gid}" "$_sandbox_private_file" + chmod 0600 "$_sandbox_private_file" +done +if [ "${OPENSHELL_VM_INIT_MODE:-sandbox}" = "capability-probe" ]; then + ts "starting capability-free VM qualification as ${_sandbox_uid}:${_sandbox_gid}" + if [ "${ROOT_PREFIX:-}" = "/newroot" ]; then + exec_supervisor_in_newroot capability-probe-launch "$_sandbox_uid" "$_sandbox_gid" + fi + exec /opt/openshell/bin/openshell-sandbox \ + capability-probe-launch "$_sandbox_uid" "$_sandbox_gid" +fi if [ "${ROOT_PREFIX:-}" = "/newroot" ]; then - exec_supervisor_in_newroot "$@" + exec_supervisor_in_newroot \ + launch-capability-free "$_sandbox_uid" "$_sandbox_gid" "$_sandbox_bootstrap_guest" fi -exec /opt/openshell/bin/openshell-sandbox "$@" +exec /opt/openshell/bin/openshell-sandbox \ + launch-capability-free "$_sandbox_uid" "$_sandbox_gid" "$_sandbox_bootstrap_guest" } if [ "${1:-}" != "--post-overlay" ]; then diff --git a/crates/openshell-driver-vm/src/bin/openshell-vm-init.rs b/crates/openshell-driver-vm/src/bin/openshell-vm-init.rs new file mode 100644 index 0000000000..a8eb52ce1b --- /dev/null +++ b/crates/openshell-driver-vm/src/bin/openshell-vm-init.rs @@ -0,0 +1,240 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Narrow, one-shot guest preparation used by the VM compute driver. + +use std::ffi::OsString; +use std::fmt; +#[cfg(target_os = "linux")] +use std::mem::size_of; +#[cfg(target_os = "linux")] +use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; +use std::process::ExitCode; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum Command { + PrepareNetwork, + Version, +} + +#[cfg(target_os = "linux")] +#[derive(Clone, Copy)] +enum InterfaceFlagOperation { + Read, + Write, +} + +#[derive(Debug)] +struct InitError(String); + +impl fmt::Display for InitError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(&self.0) + } +} + +fn main() -> ExitCode { + match run(std::env::args_os().skip(1)) { + Ok(()) => ExitCode::SUCCESS, + Err(error) => { + eprintln!("openshell-vm-init: {error}"); + ExitCode::FAILURE + } + } +} + +fn run(args: impl IntoIterator) -> Result<(), InitError> { + match parse_command(args)? { + Command::PrepareNetwork => prepare_network(), + Command::Version => { + println!("openshell-vm-init {}", env!("CARGO_PKG_VERSION")); + Ok(()) + } + } +} + +fn parse_command(args: impl IntoIterator) -> Result { + let mut args = args.into_iter(); + let command = args + .next() + .ok_or_else(|| InitError("expected the prepare-network command".to_string()))?; + if args.next().is_some() { + return Err(InitError("command does not accept arguments".to_string())); + } + match command.to_str() { + Some("prepare-network") => Ok(Command::PrepareNetwork), + Some("--version") => Ok(Command::Version), + _ => Err(InitError(format!( + "unknown command '{}'; expected prepare-network or --version", + command.to_string_lossy() + ))), + } +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn prepare_network() -> Result<(), InitError> { + // The helper is only invoked by trusted VM guest init, before it hands the + // workload to the capability-free sandbox identity. + if unsafe { libc::geteuid() } != 0 { + return Err(InitError( + "prepare-network must run as the VM guest root user".to_string(), + )); + } + + // SAFETY: socket returns a fresh descriptor or -1 and does not borrow any + // caller-owned memory. + let raw_fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_DGRAM | libc::SOCK_CLOEXEC, 0) }; + if raw_fd < 0 { + return Err(last_os_error("open loopback control socket")); + } + // SAFETY: raw_fd was returned successfully above and ownership transfers + // exactly once to OwnedFd. + let socket = unsafe { OwnedFd::from_raw_fd(raw_fd) }; + + let mut request = InterfaceRequest::loopback(); + ioctl_interface_flags( + socket.as_raw_fd(), + InterfaceFlagOperation::Read, + &mut request, + ) + .map_err(|error| InitError(format!("read loopback flags: {error}")))?; + let up_flag = libc::c_short::try_from(libc::IFF_UP) + .map_err(|_| InitError("platform IFF_UP value does not fit in ifreq flags".to_string()))?; + let flags = request.flags(); + if flags & up_flag == 0 { + request.set_flags(flags | up_flag); + ioctl_interface_flags( + socket.as_raw_fd(), + InterfaceFlagOperation::Write, + &mut request, + ) + .map_err(|error| InitError(format!("enable loopback: {error}")))?; + } + + request.set_flags(0); + ioctl_interface_flags( + socket.as_raw_fd(), + InterfaceFlagOperation::Read, + &mut request, + ) + .map_err(|error| InitError(format!("verify loopback flags: {error}")))?; + if request.flags() & up_flag == 0 { + return Err(InitError( + "loopback remained down after successful configuration".to_string(), + )); + } + Ok(()) +} + +#[cfg(not(target_os = "linux"))] +fn prepare_network() -> Result<(), InitError> { + Err(InitError( + "prepare-network is only supported in Linux VM guests".to_string(), + )) +} + +#[cfg(target_os = "linux")] +#[repr(C)] +union InterfaceRequestData { + flags: libc::c_short, + // Linux's ifreq union is 24 bytes on the supported 64-bit guest targets. + storage: [u8; 24], +} + +#[cfg(target_os = "linux")] +#[repr(C)] +struct InterfaceRequest { + name: [libc::c_char; libc::IFNAMSIZ], + data: InterfaceRequestData, +} + +#[cfg(target_os = "linux")] +const _: () = assert!(size_of::() == size_of::()); + +#[cfg(target_os = "linux")] +impl InterfaceRequest { + fn loopback() -> Self { + let mut request = Self { + name: [0; libc::IFNAMSIZ], + data: InterfaceRequestData { storage: [0; 24] }, + }; + request.name[0] = 108; + request.name[1] = 111; + request + } + + #[allow(unsafe_code)] + fn flags(&self) -> libc::c_short { + // SAFETY: the request was most recently populated by SIOCGIFFLAGS or + // set_flags, both of which initialize the flags union member. + unsafe { self.data.flags } + } + + fn set_flags(&mut self, flags: libc::c_short) { + self.data.flags = flags; + } +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn ioctl_interface_flags( + fd: std::os::fd::RawFd, + operation: InterfaceFlagOperation, + interface: &mut InterfaceRequest, +) -> std::io::Result<()> { + let request = match operation { + InterfaceFlagOperation::Read => libc::SIOCGIFFLAGS, + InterfaceFlagOperation::Write => libc::SIOCSIFFLAGS, + }; + let request = libc::Ioctl::try_from(request).map_err(|_| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "interface flag ioctl request does not fit the platform ABI", + ) + })?; + let interface_pointer: *mut InterfaceRequest = interface; + // SAFETY: interface points to an ifreq-compatible buffer that remains + // valid and exclusively borrowed for the duration of ioctl. + if unsafe { libc::ioctl(fd, request, interface_pointer) } < 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } +} + +#[cfg(target_os = "linux")] +fn last_os_error(operation: &str) -> InitError { + InitError(format!("{operation}: {}", std::io::Error::last_os_error())) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn accepts_only_prepare_network_without_arguments() { + assert_eq!( + parse_command([OsString::from("prepare-network")]).expect("valid command"), + Command::PrepareNetwork + ); + assert_eq!( + parse_command([OsString::from("--version")]).expect("valid version command"), + Command::Version + ); + assert!(parse_command(Vec::::new()).is_err()); + assert!(parse_command([OsString::from("other")]).is_err()); + assert!( + parse_command([OsString::from("prepare-network"), OsString::from("eth0")]).is_err() + ); + } + + #[cfg(target_os = "linux")] + #[test] + fn loopback_request_has_a_fixed_interface_name() { + let request = InterfaceRequest::loopback(); + assert_eq!(request.name[0], 108); + assert_eq!(request.name[1], 111); + assert!(request.name[2..].iter().all(|byte| *byte == 0)); + } +} diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index d012ffb99e..a9ab72b2c5 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -1,18 +1,22 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -use crate::gpu::{ - GpuInventory, SubnetAllocator, allocate_vsock_cid, mac_from_sandbox_id, tap_device_name, -}; +#![allow(unsafe_code)] + +use crate::gpu::{GpuInventory, allocate_vsock_cid}; + +use crate::isolation::VmBoundarySpec; use crate::lifecycle::{ BackendFeature, GuestInitDropin, LaunchAbortReason, LaunchPlan, LifecycleExtensionRegistry, RestoreContext, extension_state_dir, }; use crate::rootfs::{ clone_or_copy_sparse_file, create_ext4_image_from_dir_with_size, create_rootfs_image_from_dir, - extract_rootfs_archive_to, prepare_sandbox_rootfs_from_image_root, sandbox_guest_init_path, - sandbox_guest_user_ids_from_image, sandbox_guest_user_ids_from_overlay_image, - set_rootfs_image_file_mode, write_rootfs_image_file, + extract_host_supervisor, extract_rootfs_archive_to, prepare_sandbox_rootfs_from_image_root, + recover_rootfs_image, remove_rootfs_image_file, sandbox_guest_init_path, + sandbox_guest_runtime_identity, sandbox_guest_user_ids_from_image, + sandbox_guest_user_ids_from_overlay_image, set_rootfs_image_file_mode, + validate_host_supervisor, write_rootfs_image_file, }; use crate::runtime::VmBackend; use bollard::Docker; @@ -57,15 +61,22 @@ use openshell_core::proto::compute::v1::{ use openshell_core::proto_struct::{ deserialize_optional_non_empty_string_list, struct_to_json_value, }; +use openshell_sandbox_backend::boundary_protocol::{ + BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor, + SandboxTlsClientConfig, SandboxTlsMaterial, SandboxTlsServerConfig, SandboxTransport, + generate_sandbox_tls_material, +}; use openshell_vfio::SysfsRoot; use opentelemetry::trace::TraceContextExt as _; use prost::Message; use sha2::{Digest, Sha256}; use std::collections::{HashMap, HashSet}; +use std::fmt::Write as _; use std::fs; use std::future::Future; -use std::io::{BufRead, BufReader, BufWriter, Read, Write}; -use std::net::{IpAddr, Ipv4Addr}; +use std::io::{BufRead, BufReader, BufWriter, Read, Seek, SeekFrom, Write}; +#[cfg(unix)] +use std::os::fd::AsRawFd as _; #[cfg(unix)] use std::os::unix::fs::PermissionsExt; use std::path::{Component, Path, PathBuf}; @@ -97,6 +108,7 @@ const REGISTRY_RETRY_MAX_DELAY: Duration = Duration::from_secs(1); /// 10 GiB — configurable via `rootfs_tar_max_bytes`. const DEFAULT_ROOTFS_TAR_MAX_BYTES: u64 = 10 * 1024 * 1024 * 1024; const ROOTFS_TAR_STAGING_DIR: &str = "rootfs-tar-staging"; +const VM_CONSOLE_DIAGNOSTIC_BYTES: u64 = 8 * 1024; #[derive(Debug, Clone, Default, serde::Deserialize)] #[serde(default, deny_unknown_fields)] @@ -132,39 +144,34 @@ impl VmSandboxDriverConfig { } } -/// gvproxy host-loopback IP — gvproxy's TCP/UDP/ICMP forwarder NAT-rewrites -/// this destination to the host's `127.0.0.1` and dials out from the host -/// process. This is the only address that transparently reaches host-bound -/// services without explicit `expose` rules. -/// -/// See gvisor-tap-vsock `cmd/gvproxy/config.go` (default NAT entry -/// `HostIP -> 127.0.0.1`) and `pkg/services/forwarder/tcp.go` (NAT lookup -/// before `net.Dial`). -/// -/// Code paths route via `GVPROXY_HOST_LOOPBACK_ALIAS` (DNS / /etc/hosts) -/// instead so logs stay readable; this constant is kept for documentation -/// and parity with the guest init script. -#[allow(dead_code)] // Documentation/parity anchor; all routing goes via the alias. -const GVPROXY_HOST_LOOPBACK_IP: &str = "192.168.127.254"; const OPENSHELL_HOST_GATEWAY_ALIAS: &str = "host.openshell.internal"; -/// Hostname gvproxy resolves (via its embedded DNS) to the host-loopback IP. -/// -/// We rewrite loopback URLs to this hostname rather than the bare IP because: -/// * the guest init script seeds /etc/hosts with the same mapping, so it -/// resolves even when gvproxy's DNS is not in resolv.conf; -/// * keeping a recognisable hostname makes log messages clearer than a bare -/// 192.168.127.254 reference; -/// * package-managed gateway certificates include this SAN for guest mTLS. -/// -/// Both names ultimately route through the gvproxy NAT path on -/// `GVPROXY_HOST_LOOPBACK_IP` — they do **not** go through the gateway IP. -const GVPROXY_HOST_LOOPBACK_ALIAS: &str = OPENSHELL_HOST_GATEWAY_ALIAS; +const HOST_LOOPBACK_ALIASES: &[&str] = &[ + OPENSHELL_HOST_GATEWAY_ALIAS, + "host.containers.internal", + "host.docker.internal", +]; +#[allow(dead_code)] const GUEST_SSH_SOCKET_PATH: &str = openshell_core::container_paths::SSH_SOCKET_PATH; +#[allow(dead_code)] const GUEST_TLS_CA_PATH: &str = openshell_core::container_paths::VM_GUEST_TLS_CA_PATH; +#[allow(dead_code)] const GUEST_TLS_CERT_PATH: &str = openshell_core::container_paths::VM_GUEST_TLS_CERT_PATH; +#[allow(dead_code)] const GUEST_TLS_KEY_PATH: &str = openshell_core::container_paths::VM_GUEST_TLS_KEY_PATH; +#[allow(dead_code)] const GUEST_SANDBOX_TOKEN_PATH: &str = openshell_core::container_paths::VM_GUEST_SANDBOX_TOKEN_PATH; const GUEST_INIT_DROPIN_DIR: &str = openshell_core::container_paths::VM_GUEST_INIT_DROPIN_DIR; +const GUEST_BOUNDARY_CONFIG_DIR: &str = "/.openshell/state"; +const GUEST_BOUNDARY_CONFIG_ENV: &str = "OPENSHELL_VM_SANDBOX_BOOTSTRAP"; +const HOST_AUTH_BUNDLE_FILE: &str = "supervisor-auth.json"; +const HOST_RUNTIME_DESCRIPTOR_FILE: &str = "runtime-descriptor.json"; +const HOST_BOUNDARY_GENERATION_FILE: &str = "boundary-generation"; +/// The backend this driver admits. VM-specific placement remains inside the +/// opaque runtime descriptor. +const DRIVER_ADMITTED_BACKEND: &str = openshell_sandbox_backend::BACKEND_NAME; +const HOST_SUPERVISOR_BINARY: &str = "host-runtime/openshell-supervisor"; +const VM_CONTROL_SOCKET: &str = "control.sock"; +const VM_CONTROL_PORT: u32 = 5500; /// Guest path of the driver-authored manifest enumerating which /// `init.d` drop-ins the guest init script is allowed to execute. /// @@ -174,19 +181,6 @@ const GUEST_INIT_DROPIN_DIR: &str = openshell_core::container_paths::VM_GUEST_IN /// upperdir on every launch, so the image cannot forge or shadow it. const GUEST_INIT_DROPIN_MANIFEST: &str = openshell_core::container_paths::VM_GUEST_INIT_DROPIN_MANIFEST; -/// Guest path of the root-only corporate proxy credential staged by the driver. -const GUEST_UPSTREAM_PROXY_AUTH_PATH: &str = - openshell_core::container_paths::VM_GUEST_UPSTREAM_PROXY_AUTH_PATH; -/// Guest path of the corporate proxy CA bundle staged by the driver. -const GUEST_PROXY_CA_PATH: &str = openshell_core::container_paths::VM_GUEST_PROXY_CA_PATH; -/// Guest path of the driver-authored supervisor argument list. -/// -/// The counterpart of [`GUEST_INIT_DROPIN_MANIFEST`] for the supervisor's own -/// command line: written into the overlay upperdir on every launch (empty -/// when there is nothing to pass) so the guest appends exactly the arguments -/// the driver chose and a sandbox image cannot forge or shadow them. -const GUEST_SUPERVISOR_ARGS_PATH: &str = - openshell_core::container_paths::VM_GUEST_SUPERVISOR_ARGS_PATH; const IMAGE_CACHE_ROOT_DIR: &str = "images"; const IMAGE_CACHE_ROOTFS_IMAGE: &str = "rootfs.ext4"; const OVERLAY_TEMPLATE_CACHE_DIR: &str = "overlay-templates"; @@ -204,7 +198,7 @@ const GUEST_IMAGE_CONFIG_DIR: &str = "openshell-image"; const GUEST_IMAGE_OCI_LAYOUT_DIR: &str = "oci"; const GUEST_IMAGE_OCI_REF: &str = "openshell"; const IMAGE_EXPORT_ROOTFS_ARCHIVE: &str = "source-rootfs.tar"; -const BOOTSTRAP_IMAGE_CACHE_LAYOUT_VERSION: &str = "sandbox-bootstrap-rootfs-ext4-v3"; +const BOOTSTRAP_IMAGE_CACHE_LAYOUT_VERSION: &str = "sandbox-bootstrap-rootfs-ext4-v4"; const PREPARED_IMAGE_CACHE_LAYOUT_VERSION: &str = "sandbox-prepared-rootfs-ext4-umoci-v3"; const IMAGE_IDENTITY_FILE: &str = "image-identity"; const IMAGE_REFERENCE_FILE: &str = "image-reference"; @@ -486,7 +480,7 @@ impl VmDriverConfig { if provided.iter().all(Option::is_none) { return if self.requires_tls_materials() { Err( - "https:// openshell endpoint requires OPENSHELL_VM_TLS_CA, OPENSHELL_VM_TLS_CERT, and OPENSHELL_VM_TLS_KEY so sandbox VMs can authenticate to the gateway" + "https:// openshell endpoint requires OPENSHELL_VM_TLS_CA, OPENSHELL_VM_TLS_CERT, and OPENSHELL_VM_TLS_KEY so the host supervisor can authenticate to the gateway" .to_string(), ) } else { @@ -545,9 +539,28 @@ fn validate_openshell_endpoint(endpoint: &str) -> Result<(), String> { Ok(()) } +fn host_control_openshell_endpoint(endpoint: &str) -> Result<(String, Option), String> { + let mut url = Url::parse(endpoint) + .map_err(|err| format!("invalid openshell endpoint '{endpoint}': {err}"))?; + let Some(host) = url.host_str().map(str::to_string) else { + return Ok((endpoint.to_string(), None)); + }; + if !HOST_LOOPBACK_ALIASES.contains(&host.as_str()) { + return Ok((endpoint.to_string(), None)); + } + + // The supervisor runs on the host, so guest aliases dial loopback while + // retaining the configured hostname for TLS certificate verification. + url.set_host(Some("127.0.0.1")) + .map_err(|error| format!("failed to rewrite host endpoint '{endpoint}': {error}"))?; + Ok((url.into(), Some(host))) +} + #[derive(Debug)] struct VmProcess { child: Child, + supervisor: Child, + supervisor_liveness: Option, deleting: bool, } @@ -557,7 +570,6 @@ struct SandboxRecord { process: Option>>, provisioning_task: Option>, gpu_bdf: Option, - qemu_network_allocated: bool, deleting: bool, } @@ -615,7 +627,6 @@ pub struct VmDriver { image_cache_lock: Arc>, events: broadcast::Sender, gpu_inventory: Option>>, - subnet_allocator: Arc>, lifecycle_extensions: Arc, } @@ -625,7 +636,7 @@ impl VmDriver { } pub async fn new_with_extensions( - config: VmDriverConfig, + mut config: VmDriverConfig, lifecycle_extensions: LifecycleExtensionRegistry, ) -> Result { lifecycle_extensions @@ -639,13 +650,11 @@ impl VmDriver { } validate_openshell_endpoint(&config.grpc_endpoint)?; let _ = config.tls_paths()?; + config.state_dir = absolute_state_dir(&config.state_dir)?; #[cfg(target_os = "linux")] if config.gpu_enabled { check_gpu_privileges()?; - tokio::task::spawn_blocking(crate::cleanup_stale_tap_interfaces) - .await - .map_err(|e| format!("cleanup stale TAP interfaces panicked: {e}"))?; } let state_root = sandboxes_root_dir(&config.state_dir); @@ -691,11 +700,6 @@ impl VmDriver { None }; - let subnet_allocator = Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))); - let (events, _) = broadcast::channel(WATCH_BUFFER); let driver = Self { config, @@ -704,7 +708,6 @@ impl VmDriver { image_cache_lock: Arc::new(Mutex::new(())), events, gpu_inventory, - subnet_allocator, lifecycle_extensions: Arc::new(lifecycle_extensions), }; driver.restore_persisted_sandboxes().await; @@ -773,6 +776,205 @@ impl VmDriver { Ok(canonical) } + async fn host_supervisor_binary(&self) -> Result { + if let Some(configured) = std::env::var_os("OPENSHELL_VM_SUPERVISOR_BIN") { + let configured = PathBuf::from(configured); + if configured.is_file() { + return Ok(configured); + } + return Err(Status::failed_precondition(format!( + "configured host supervisor does not exist: {}", + configured.display() + ))); + } + + let destination = self.config.state_dir.join(HOST_SUPERVISOR_BINARY); + if validate_host_supervisor(&destination).is_ok() { + return Ok(destination); + } + let _cache_guard = self.image_cache_lock.lock().await; + if validate_host_supervisor(&destination).is_ok() { + return Ok(destination); + } + let destination_for_extract = destination.clone(); + tokio::task::spawn_blocking(move || extract_host_supervisor(&destination_for_extract)) + .await + .map_err(|error| { + Status::internal(format!("host supervisor extraction panicked: {error}")) + })? + .map_err(Status::failed_precondition)?; + validate_host_supervisor(&destination).map_err(Status::failed_precondition)?; + Ok(destination) + } + + async fn spawn_host_supervisor( + &self, + sandbox: &Sandbox, + state_dir: &Path, + tls_paths: Option<&VmDriverTlsPaths>, + runtime_descriptor: &SandboxRuntimeDescriptor, + auth_bundle: &openshell_core::jwt::SupervisorAuthBundle, + sandbox_owner: SandboxOwnerIdentity, + ) -> Result<(Child, Option), Status> { + let supervisor_binary = self.host_supervisor_binary().await?; + let (openshell_endpoint, gateway_tls_server_name) = + host_control_openshell_endpoint(&self.config.grpc_endpoint) + .map_err(Status::failed_precondition)?; + let auth_bundle_path = state_dir.join(HOST_AUTH_BUNDLE_FILE); + let encoded_auth_bundle = serde_json::to_vec(auth_bundle) + .map_err(|error| Status::internal(format!("encode supervisor auth bundle: {error}")))?; + tokio::fs::write(&auth_bundle_path, encoded_auth_bundle) + .await + .map_err(|error| Status::internal(format!("write supervisor auth bundle: {error}")))?; + #[cfg(unix)] + tokio::fs::set_permissions(&auth_bundle_path, fs::Permissions::from_mode(0o600)) + .await + .map_err(|error| { + Status::internal(format!("restrict supervisor auth bundle: {error}")) + })?; + + let descriptor = runtime_descriptor + .backend_descriptor() + .map_err(|error| Status::internal(error.to_string()))?; + // The payload carries the boundary bootstrap token, so it must not + // appear in the world-readable process cmdline; deliver it through a + // driver-owned 0600 file like the gateway token. + let payload_path = state_dir.join(HOST_RUNTIME_DESCRIPTOR_FILE); + tokio::fs::write(&payload_path, &descriptor.payload) + .await + .map_err(|error| Status::internal(format!("write host runtime descriptor: {error}")))?; + #[cfg(unix)] + tokio::fs::set_permissions(&payload_path, fs::Permissions::from_mode(0o600)) + .await + .map_err(|error| { + Status::internal(format!("restrict host runtime descriptor: {error}")) + })?; + let main_process_spec = openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec( + sandbox.spec.as_ref(), + ) + .map_err(|error| Status::internal(format!("encode main process spec: {error}")))?; + let upstream_proxy_args = upstream_proxy_cli_args(&self.config) + .map_err(|error| Status::invalid_argument(format!("render upstream proxy: {error}")))?; + let mut command = Command::new(&supervisor_binary); + isolate_host_control_environment(&mut command); + command + .kill_on_drop(true) + .stdin(Stdio::null()) + .stdout(Stdio::from( + fs::File::create(state_dir.join("supervisor.log")) + .map_err(|error| Status::internal(format!("create supervisor log: {error}")))?, + )) + .stderr(Stdio::from( + fs::File::create(state_dir.join("supervisor.err.log")).map_err(|error| { + Status::internal(format!("create supervisor error log: {error}")) + })?, + )) + .arg("--backend-descriptor-file") + .arg(&payload_path) + .arg("--auth-bundle-file") + .arg(&auth_bundle_path) + .arg("--workdir") + .arg("/sandbox") + .args(upstream_proxy_args) + .env( + openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND, + DRIVER_ADMITTED_BACKEND, + ) + .env( + openshell_core::sandbox_env::MAIN_PROCESS_SPEC, + main_process_spec, + ) + .env(openshell_core::sandbox_env::ENDPOINT, openshell_endpoint) + .env(openshell_core::sandbox_env::SANDBOX_ID, &sandbox.id) + .env(openshell_core::sandbox_env::SANDBOX, &sandbox.name) + .env( + openshell_core::sandbox_env::SSH_SOCKET_PATH, + state_dir.join("ssh.sock"), + ) + .env( + openshell_core::sandbox_env::PROXY_TLS_DIR, + state_dir.join("proxy-tls"), + ) + .env( + openshell_core::sandbox_env::SANDBOX_UID, + sandbox_owner.uid.to_string(), + ) + .env( + openshell_core::sandbox_env::SANDBOX_GID, + sandbox_owner.gid.to_string(), + ) + .env(openshell_core::sandbox_env::OCI_IMAGE_USER, "") + .env( + openshell_core::sandbox_env::LOG_LEVEL, + openshell_core::driver_utils::sandbox_log_level(sandbox, &self.config.log_level), + ) + .env( + openshell_core::sandbox_env::TELEMETRY_ENABLED, + openshell_core::telemetry::enabled_env_value(), + ); + if let Some(server_name) = gateway_tls_server_name { + command.env( + openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME, + server_name, + ); + } + configure_main_exit_marker(&mut command, state_dir); + if let Some(tls) = tls_paths { + command + .env(openshell_core::sandbox_env::TLS_CA, &tls.ca) + .env(openshell_core::sandbox_env::TLS_CERT, &tls.cert) + .env(openshell_core::sandbox_env::TLS_KEY, &tls.key); + } + #[cfg(unix)] + let (liveness_read, liveness_write) = nix::unistd::pipe().map_err(|error| { + Status::internal(format!("create supervisor parent-liveness pipe: {error}")) + })?; + #[cfg(unix)] + for fd in [&liveness_read, &liveness_write] { + nix::fcntl::fcntl( + fd.as_raw_fd(), + nix::fcntl::FcntlArg::F_SETFD(nix::fcntl::FdFlag::FD_CLOEXEC), + ) + .map_err(|error| { + Status::internal(format!( + "protect supervisor parent-liveness descriptor: {error}" + )) + })?; + } + #[cfg(unix)] + let liveness_read_fd = liveness_read.as_raw_fd(); + #[cfg(unix)] + command + .arg("--parent-liveness-fd") + .arg(liveness_read_fd.to_string()); + #[cfg(unix)] + unsafe { + command.pre_exec(move || { + nix::fcntl::fcntl( + liveness_read_fd, + nix::fcntl::FcntlArg::F_SETFD(nix::fcntl::FdFlag::empty()), + ) + .map_err(std::io::Error::other)?; + #[cfg(target_os = "linux")] + nix::sys::prctl::set_pdeathsig(Signal::SIGKILL).map_err(std::io::Error::other)?; + Ok(()) + }); + } + let child = command.spawn().map_err(|error| { + Status::internal(format!( + "start host supervisor '{}': {error}", + supervisor_binary.display() + )) + })?; + #[cfg(unix)] + { + drop(liveness_read); + Ok((child, Some(fs::File::from(liveness_write)))) + } + #[cfg(not(unix))] + Ok((child, None)) + } + #[must_use] pub fn capabilities(&self) -> GetCapabilitiesResponse { GetCapabilitiesResponse { @@ -865,7 +1067,6 @@ impl VmDriver { process: None, provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }, ); @@ -922,15 +1123,14 @@ impl VmDriver { let provisioning_span = provisioning_span(&parent, &sandbox_id, &image_ref); let task = tokio::spawn( async move { - driver - .provision_sandbox( - sandbox_for_task, - image_ref_for_task, - state_dir_for_task, - tls_paths, - OverlayPreparation::Fresh, - ) - .await; + Box::pin(driver.provision_sandbox( + sandbox_for_task, + image_ref_for_task, + state_dir_for_task, + tls_paths, + OverlayPreparation::Fresh, + )) + .await; } .instrument(provisioning_span), ); @@ -1068,6 +1268,41 @@ impl VmDriver { let image_disk = image_plan.image_disk; let owner_source_disk = image_disk.as_ref().unwrap_or(&root_disk).clone(); let overlay_disk = disk_paths.overlay_disk; + let boundary_generation = + match tokio::fs::read_to_string(state_dir.join(HOST_BOUNDARY_GENERATION_FILE)).await { + Ok(generation) if !generation.trim().is_empty() => generation.trim().to_string(), + Ok(_) => random_boundary_token(), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + random_boundary_token() + } + Err(error) => { + return Err(Status::internal(format!( + "read VM boundary generation: {error}" + ))); + } + }; + let launch_authentication = sandbox + .spec + .as_ref() + .filter(|spec| !spec.launch_authentication.is_empty()) + .ok_or_else(|| { + Status::failed_precondition("VM sandbox launch authentication is required") + }) + .and_then(|spec| { + serde_json::from_slice::( + &spec.launch_authentication, + ) + .map_err(|error| { + Status::failed_precondition(format!( + "decode VM sandbox launch authentication: {error}" + )) + }) + })?; + launch_authentication.validate().map_err(|error| { + Status::failed_precondition(format!( + "validate VM sandbox launch authentication: {error}" + )) + })?; self.publish_platform_event( sandbox.id.clone(), @@ -1083,12 +1318,6 @@ impl VmDriver { &state_dir, &overlay_disk, &owner_source_disk, - tls_paths.as_ref(), - sandbox - .spec - .as_ref() - .map(|spec| spec.sandbox_token.as_str()) - .filter(|token| !token.is_empty()), overlay_preparation, ) .await @@ -1119,24 +1348,11 @@ impl VmDriver { match self.build_vm_launch_plan(&sandbox.id, needs_qemu, is_gpu, gpu_bdf.clone()) { Ok(plan) => plan, Err(err) => { - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); return Err(err); } }; - // `build_vm_launch_plan` already allocated the QEMU subnet, so record - // it as allocated now — before the cancellable `configure_launch` / - // `before_launch` hooks run. If a delete aborts provisioning while - // one of those hooks is awaiting, the aborted future never runs its - // own release path, and the delete cleanup is gated on this flag; if - // the flag were still unset the subnet would leak. - if plan.backend == VmBackend::Qemu - && let Err(err) = self.mark_qemu_network_allocated(&sandbox.id).await - { - self.release_gpu_and_subnet(&sandbox.id); - return Err(err); - } - if let Err(err) = self .lifecycle_extensions .configure_launch(&sandbox, &state_dir, &mut plan) @@ -1149,7 +1365,7 @@ impl VmDriver { LaunchAbortReason::BeforeLaunchHookFailed, ) .await; - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); let message = format!( "vm lifecycle extension rejected sandbox launch plan: {}", err.message() @@ -1163,8 +1379,8 @@ impl VmDriver { // Resolve and validate the backend from the requirements that // `configure_launch` extensions contributed. After this point the - // plan's backend, sizing, and host allocations (subnet, tap, vsock) - // are final; the `before_launch` hook below may still mutate + // plan's backend, sizing, and host allocations are final; the + // `before_launch` hook below may still mutate // `plan.env` and `plan.guest_init_dropins` and may abort the launch, // but it MUST NOT change `plan.backend`, `plan.required_backends`, // or `plan.required_backend_features` -- those are enforced as a @@ -1179,7 +1395,7 @@ impl VmDriver { LaunchAbortReason::BeforeLaunchHookFailed, ) .await; - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); return Err(err); } @@ -1191,7 +1407,7 @@ impl VmDriver { LaunchAbortReason::BeforeLaunchHookFailed, ) .await; - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); return Err(err); } @@ -1207,7 +1423,7 @@ impl VmDriver { LaunchAbortReason::BeforeLaunchHookFailed, ) .await; - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); let message = format!( "vm lifecycle extension rejected sandbox launch: {}", err.message() @@ -1223,29 +1439,85 @@ impl VmDriver { self.lifecycle_extensions .after_launch_failed(&sandbox, &state_dir, LaunchAbortReason::GuestPrepareFailed) .await; - self.release_gpu_and_subnet(&sandbox.id); - return Err(err); - } - - // Staged on every launch, including a restart onto a preserved - // overlay, so the driver's copy always shadows the image layer. - if let Err(err) = inject_guest_upstream_proxy(&overlay_disk, &self.config).await { - self.lifecycle_extensions - .after_launch_failed(&sandbox, &state_dir, LaunchAbortReason::GuestPrepareFailed) - .await; - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); return Err(err); } - let endpoint_override = if plan.backend == VmBackend::Qemu { - plan.host_ip.as_deref().map(|host_ip| { - guest_visible_openshell_endpoint_for_tap(&self.config.grpc_endpoint, host_ip) - }) + let console_output = state_dir.join("rootfs-console.log"); + let control_socket = state_dir.join(VM_CONTROL_SOCKET); + let session_id = launch_authentication.supervisor.session_id; + let channel_tls = generate_sandbox_tls_material(session_id) + .map_err(|error| Status::internal(error.to_string()))?; + let supervisor_tls = SandboxTlsClientConfig { + server_name: channel_tls.server_name.clone(), + trust_anchor_pem: channel_tls.trust_anchor_pem.clone(), + }; + let transport = if plan.backend == VmBackend::Qemu { + SandboxTransport::Vsock { + guest_cid: plan.vsock_cid.ok_or_else(|| { + Status::internal("QEMU launch plan is missing a guest vsock CID") + })?, + port: VM_CONTROL_PORT, + } } else { - None + SandboxTransport::Unix { + socket_path: control_socket.clone(), + } }; - - let console_output = state_dir.join("rootfs-console.log"); + let verification_keys = launch_authentication + .verification_keys + .iter() + .map(|key| { + String::from_utf8(key.public_key_pem.clone()) + .map(|public_key_pem| GatewayVerificationKey { + key_id: key.key_id.clone(), + public_key_pem, + }) + .map_err(|error| { + Status::failed_precondition(format!( + "VM sandbox verification key is not UTF-8 PEM: {error}" + )) + }) + }) + .collect::, _>>()?; + let provisioning = VmBoundarySpec { + boundary_id: sandbox.id.clone(), + generation: launch_authentication + .supervisor + .runtime_generation + .to_string(), + session_id, + session_rotation: launch_authentication.supervisor.session_rotation, + auth_epoch: launch_authentication.supervisor.auth_epoch, + gateway_id: launch_authentication.gateway_id.clone(), + verification_keys, + image_identity, + transport, + supervisor_tls, + sandbox_tls: guest_boundary_tls_paths(&boundary_generation), + control_port: VM_CONTROL_PORT, + agent_uid: sandbox_owner_state.uid, + agent_gid: sandbox_owner_state.gid, + child_env: merged_environment(&sandbox), + } + .provision() + .map_err(|error| Status::failed_precondition(error.to_string()))?; + let guest_boundary_config_path = + guest_boundary_config_path(&provisioning.boundary_config.generation); + inject_guest_boundary_bundle( + &overlay_disk, + &guest_boundary_config_path, + &provisioning.boundary_config, + &channel_tls, + ) + .map_err(|error| Status::internal(format!("inject VM boundary configuration: {error}")))?; + write_private_file( + &state_dir.join(HOST_BOUNDARY_GENERATION_FILE), + boundary_generation.as_bytes().to_vec(), + ) + .await + .map_err(|error| Status::internal(format!("persist VM boundary generation: {error}")))?; + let runtime_descriptor = provisioning.runtime_descriptor; let mut command = Command::new(&self.launcher_bin); command.kill_on_drop(true); command.stdin(Stdio::null()); @@ -1271,24 +1543,16 @@ impl VmDriver { if let Some(bdf) = plan.gpu_bdf.as_deref() { command.arg("--vm-gpu-bdf").arg(bdf); } - if let Some(tap) = plan.tap_device.as_deref() { - command.arg("--vm-tap-device").arg(tap); - } - if let Some(guest_ip) = plan.guest_ip.as_deref() { - command.arg("--vm-guest-ip").arg(guest_ip); - } - if let Some(host_ip) = plan.host_ip.as_deref() { - command.arg("--vm-host-ip").arg(host_ip); - } if let Some(vsock_cid) = plan.vsock_cid { command.arg("--vm-vsock-cid").arg(vsock_cid.to_string()); } - if let Some(guest_mac) = plan.guest_mac.as_deref() { - command.arg("--vm-guest-mac").arg(guest_mac); - } - if let Some(port) = plan.gateway_port { - command.arg("--vm-gateway-port").arg(port.to_string()); - } + } else { + let _ = tokio::fs::remove_file(&control_socket).await; + command + .arg("--vm-vsock-control-port") + .arg(VM_CONTROL_PORT.to_string()) + .arg("--vm-vsock-control-socket") + .arg(&control_socket); } self.ensure_provisioning_active(&sandbox.id).await?; @@ -1297,9 +1561,12 @@ impl VmDriver { .arg("--vm-krun-log-level") .arg(self.config.krun_log_level.to_string()); - for env in build_guest_environment(&sandbox, &self.config, endpoint_override.as_deref()) { + for env in build_guest_environment(&sandbox, &self.config) { command.arg("--vm-env").arg(env); } + command.arg("--vm-env").arg(format!( + "{GUEST_BOUNDARY_CONFIG_ENV}={guest_boundary_config_path}" + )); for env in &plan.env { command.arg("--vm-env").arg(env); } @@ -1313,7 +1580,7 @@ impl VmDriver { console_output = %console_output.display(), "vm driver: spawning VM launcher" ); - let child = match spawn_vm_launcher(&mut command, &sandbox.id, &plan.backend) { + let mut child = match command.spawn() { Ok(child) => child, Err(err) => { warn!( @@ -1328,7 +1595,7 @@ impl VmDriver { LaunchAbortReason::LauncherSpawnFailed, ) .await; - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); return Err(Status::internal(format!( "failed to launch vm helper '{}': {err}", self.launcher_bin.display() @@ -1340,8 +1607,35 @@ impl VmDriver { launcher_pid = child.id().unwrap_or(0), "vm driver: launcher spawned" ); + let (supervisor, supervisor_liveness) = match self + .spawn_host_supervisor( + &sandbox, + &state_dir, + tls_paths.as_ref(), + &runtime_descriptor, + &launch_authentication.supervisor, + sandbox_owner_state, + ) + .await + { + Ok(supervisor) => supervisor, + Err(error) => { + let _ = terminate_vm_process(&mut child).await; + self.lifecycle_extensions + .after_launch_failed( + &sandbox, + &state_dir, + LaunchAbortReason::LauncherSpawnFailed, + ) + .await; + self.release_gpu(&sandbox.id); + return Err(error); + } + }; let process = Arc::new(Mutex::new(VmProcess { child, + supervisor, + supervisor_liveness, deleting: false, })); @@ -1353,7 +1647,6 @@ impl VmDriver { Some(record) if !record.deleting => { record.process = Some(process.clone()); record.gpu_bdf.clone_from(&gpu_bdf); - record.qemu_network_allocated = plan.backend == VmBackend::Qemu; snapshot_to_publish = Some(record.snapshot.clone()); } _ => { @@ -1366,11 +1659,11 @@ impl VmDriver { { let mut process = process.lock().await; process.deleting = true; - terminate_vm_process(&mut process.child) + terminate_sandbox_processes(&mut process) .await - .map_err(|err| Status::internal(format!("failed to stop vm: {err}")))?; + .map_err(|err| Status::internal(format!("failed to stop sandbox: {err}")))?; } - self.release_gpu_and_subnet(&sandbox.id); + self.release_gpu(&sandbox.id); return Err(Status::cancelled("sandbox provisioning cancelled")); } @@ -1431,7 +1724,7 @@ impl VmDriver { .await .map_err(|err| Status::internal(format!("persist stop marker failed: {err}")))?; - let (process, provisioning_task, has_gpu, has_qemu_network, snapshot) = { + let (process, provisioning_task, has_gpu, snapshot) = { let mut registry = self.registry.lock().await; let record = registry .get_mut(&record_id) @@ -1440,7 +1733,6 @@ impl VmDriver { record.process.take(), record.provisioning_task.take(), record.gpu_bdf.take().is_some(), - std::mem::take(&mut record.qemu_network_allocated), record.snapshot.clone(), ) }; @@ -1451,14 +1743,23 @@ impl VmDriver { if let Some(process) = process { let mut process = process.lock().await; process.deleting = true; - terminate_vm_process(&mut process.child) + terminate_sandbox_processes(&mut process) .await - .map_err(|err| Status::internal(format!("failed to stop vm: {err}")))?; + .map_err(|err| Status::internal(format!("failed to stop sandbox: {err}")))?; } + remove_runtime_generation_material(&state_dir) + .await + .map_err(|error| { + Status::internal(format!( + "remove stopped VM authentication material: {error}" + )) + })?; self.lifecycle_extensions .after_launch_failed(&snapshot, &state_dir, LaunchAbortReason::Stopped) .await; - self.release_allocations(&record_id, has_gpu, has_qemu_network); + if has_gpu { + self.release_gpu(&record_id); + } if let Some(snapshot) = self .set_snapshot_condition(&record_id, stopped_condition(), false) @@ -1473,10 +1774,20 @@ impl VmDriver { Ok(()) } - pub async fn start_sandbox(&self, sandbox_id: &str, sandbox_name: &str) -> Result<(), Status> { + pub async fn start_sandbox( + &self, + sandbox_id: &str, + sandbox_name: &str, + generation_id: &str, + launch_authentication: Vec, + ) -> Result<(), Status> { if !sandbox_id.is_empty() { validate_sandbox_id(sandbox_id)?; } + let generation = openshell_core::sandbox_generation::SandboxGenerationId::parse( + generation_id.to_string(), + ) + .map_err(|error| Status::invalid_argument(error.to_string()))?; let (record_id, state_dir, already_running) = { let registry = self.registry.lock().await; let (id, record) = if let Some(entry) = registry.get_key_value(sandbox_id) { @@ -1494,14 +1805,71 @@ impl VmDriver { ) }; if already_running { - return Ok(()); + let active_generation = + tokio::fs::read_to_string(state_dir.join(HOST_BOUNDARY_GENERATION_FILE)) + .await + .map_err(|error| { + Status::failed_precondition(format!( + "read active VM sandbox generation: {error}" + )) + })?; + if active_generation.trim() != generation.as_str() { + return Err(Status::failed_precondition(format!( + "VM sandbox is already running generation {}", + active_generation.trim() + ))); + } + if launch_authentication.is_empty() { + return Ok(()); + } + // The gateway keeps launch sessions in memory. A non-empty bundle + // during startup recovery represents a new gateway session, so + // restart the VM before installing it rather than leaving the old + // supervisor connected with invalid credentials. + self.stop_sandbox(&record_id, sandbox_name).await?; } - let sandbox = read_sandbox_request(&state_dir.join(SANDBOX_REQUEST_FILE)) + remove_runtime_generation_material(&state_dir) + .await + .map_err(|error| { + Status::internal(format!( + "remove previous VM authentication material: {error}" + )) + })?; + write_private_file( + &state_dir.join(HOST_BOUNDARY_GENERATION_FILE), + generation.as_str().as_bytes().to_vec(), + ) + .await + .map_err(|error| Status::internal(format!("persist VM start generation: {error}")))?; + let mut sandbox = read_sandbox_request(&state_dir.join(SANDBOX_REQUEST_FILE)) .await .map_err(|err| { Status::internal(format!("read sandbox start metadata failed: {err}")) })?; + let authentication = serde_json::from_slice::< + openshell_core::jwt::SandboxLaunchAuthentication, + >(&launch_authentication) + .map_err(|error| { + Status::failed_precondition(format!("decode VM sandbox launch authentication: {error}")) + })?; + authentication.validate().map_err(|error| { + Status::failed_precondition(format!( + "validate VM sandbox launch authentication: {error}" + )) + })?; + let spec = sandbox + .spec + .as_mut() + .ok_or_else(|| Status::failed_precondition("persisted VM sandbox spec is missing"))?; + spec.launch_authentication = launch_authentication; + write_sandbox_request(&state_dir, &sandbox) + .await + .map_err(|error| { + Status::internal(format!( + "persist refreshed VM launch authentication: {error}" + )) + })?; let stopped_record = self .registry .lock() @@ -1558,14 +1926,7 @@ impl VmDriver { return span_status.finish(Ok(DeleteSandboxResponse { deleted: false })); }; - let ( - state_dir, - process, - gpu_bdf, - qemu_network_allocated, - provisioning_task, - sandbox_snapshot, - ) = { + let (state_dir, process, gpu_bdf, provisioning_task, sandbox_snapshot) = { let mut registry = self.registry.lock().await; let Some(record) = registry.get_mut(&record_id) else { return span_status.finish(Ok(DeleteSandboxResponse { deleted: false })); @@ -1575,7 +1936,6 @@ impl VmDriver { record.state_dir.clone(), record.process.clone(), record.gpu_bdf.clone(), - record.qemu_network_allocated, record.provisioning_task.take(), record.snapshot.clone(), ) @@ -1595,16 +1955,18 @@ impl VmDriver { if let Some(process) = process { let mut process = process.lock().await; process.deleting = true; - terminate_vm_process(&mut process.child) + terminate_sandbox_processes(&mut process) .await - .map_err(|err| Status::internal(format!("failed to stop vm: {err}")))?; + .map_err(|err| Status::internal(format!("failed to stop sandbox: {err}")))?; } self.lifecycle_extensions .after_delete(&sandbox_snapshot, &state_dir) .await; - self.release_allocations(&record_id, gpu_bdf.is_some(), qemu_network_allocated); + if gpu_bdf.is_some() { + self.release_gpu(&record_id); + } remove_sandbox_state_dir(&self.config.state_dir, &state_dir).await?; @@ -1740,7 +2102,6 @@ impl VmDriver { process: None, provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }); drop(registry); @@ -1768,7 +2129,6 @@ impl VmDriver { process: None, provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }); drop(registry); @@ -1864,7 +2224,6 @@ impl VmDriver { process: None, provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }, ); @@ -1912,15 +2271,14 @@ impl VmDriver { provisioning_span(&restoration_span.context(), &sandbox_id, &image_ref); let task = tokio::spawn( async move { - driver - .provision_sandbox( - sandbox, - image_ref, - state_dir, - tls_paths, - OverlayPreparation::PreserveExisting, - ) - .await; + Box::pin(driver.provision_sandbox( + sandbox, + image_ref, + state_dir, + tls_paths, + OverlayPreparation::PreserveExisting, + )) + .await; drop(reconciliation_span); } .instrument(provisioning_span) @@ -1948,26 +2306,6 @@ impl VmDriver { } } - fn release_subnet(&self, sandbox_id: &str) { - if let Ok(mut alloc) = self.subnet_allocator.lock() { - alloc.release(sandbox_id); - } - } - - fn release_allocations(&self, sandbox_id: &str, has_gpu: bool, has_qemu_network: bool) { - if has_gpu { - self.release_gpu(sandbox_id); - } - if has_qemu_network { - self.release_subnet(sandbox_id); - } - } - - fn release_gpu_and_subnet(&self, sandbox_id: &str) { - self.release_gpu(sandbox_id); - self.release_subnet(sandbox_id); - } - async fn ensure_extension_state_dirs(&self, state_dir: &Path) -> Result<(), Status> { for extension_name in self.lifecycle_extensions.names() { let extension_dir = extension_state_dir(state_dir, &extension_name).map_err(|err| { @@ -2017,10 +2355,12 @@ impl VmDriver { Ok(()) } - #[allow(clippy::result_large_err)] + // Keep the fallible shape used by launch-plan resolution: driver-local + // backends may add allocation failures here without changing callers. + #[allow(clippy::result_large_err, clippy::unnecessary_wraps)] fn configure_qemu_launch_plan( &self, - sandbox_id: &str, + _sandbox_id: &str, is_gpu: bool, gpu_bdf: Option, plan: &mut LaunchPlan, @@ -2033,44 +2373,10 @@ impl VmDriver { if plan.gpu_bdf.is_none() { plan.gpu_bdf = gpu_bdf; } - if !has_complete_qemu_network(plan) { - let subnet = self - .subnet_allocator - .lock() - .map_err(|e| Status::internal(format!("subnet allocator lock poisoned: {e}")))? - .allocate(sandbox_id) - .map_err(Status::failed_precondition)?; - let mac = mac_from_sandbox_id(sandbox_id); - plan.tap_device = Some(tap_device_name(sandbox_id)); - plan.guest_ip = Some(subnet.guest_ip.to_string()); - plan.host_ip = Some(subnet.host_ip.to_string()); - plan.vsock_cid = Some(allocate_vsock_cid()); - plan.guest_mac = Some(format!( - "{:02x}:{:02x}:{:02x}:{:02x}:{:02x}:{:02x}", - mac[0], mac[1], mac[2], mac[3], mac[4], mac[5] - )); - plan.gateway_port = gateway_port_from_endpoint(&self.config.grpc_endpoint); - } - - // The corporate-proxy host-loopback recipe is a libkrun/gvproxy - // property and has no QEMU/TAP equivalent (see - // `proxy_url_targets_gateway_host`). Run it here, after the subnet - // allocation above has settled `plan.host_ip`, because the address to - // compare against is this sandbox's own TAP host address. Fail the - // create with the reason rather than boot a sandbox whose - // policy-approved CONNECTs all time out against an unreachable proxy. - if let Some(url) = self.config.upstream_proxy.https_proxy.as_deref() - && proxy_url_targets_gateway_host(url, plan.host_ip.as_deref()) - { - let tap_host = plan.host_ip.as_deref().unwrap_or("the TAP host address"); - return Err(Status::failed_precondition(format!( - "https_proxy '{url}' addresses the gateway host, which a QEMU/TAP sandbox \ - (GPU sandboxes) cannot reach: host.openshell.internal resolves to this \ - sandbox's TAP host address {tap_host} and the driver's nftables rules allow \ - only the gateway port from the guest. Configure a proxy address routable \ - from the guest's masqueraded egress, or run this sandbox without a GPU" - ))); + if plan.vsock_cid.is_some() { + return Ok(()); } + plan.vsock_cid = Some(allocate_vsock_cid()); Ok(()) } @@ -2149,21 +2455,12 @@ impl VmDriver { Ok(()) } - async fn mark_qemu_network_allocated(&self, sandbox_id: &str) -> Result<(), Status> { - let mut registry = self.registry.lock().await; - match registry.get_mut(sandbox_id) { - Some(record) if !record.deleting => { - record.qemu_network_allocated = true; - Ok(()) - } - _ => Err(Status::cancelled("sandbox provisioning cancelled")), - } - } - - #[allow(clippy::result_large_err)] + // Keep the fallible shape used by provisioning and lifecycle tests even + // though NIC/subnet allocation no longer introduces a failure today. + #[allow(clippy::result_large_err, clippy::unnecessary_wraps)] fn build_vm_launch_plan( &self, - sandbox_id: &str, + _sandbox_id: &str, needs_qemu: bool, is_gpu: bool, gpu_bdf: Option, @@ -2178,32 +2475,13 @@ impl VmDriver { kernel_profile: None, kernel_image: None, gpu_bdf: None, - tap_device: None, - guest_ip: None, - host_ip: None, vsock_cid: None, - guest_mac: None, - gateway_port: None, guest_init_dropins: Vec::new(), env: Vec::new(), }); } - let subnet = self - .subnet_allocator - .lock() - .map_err(|e| Status::internal(format!("subnet allocator lock poisoned: {e}")))? - .allocate(sandbox_id) - .map_err(Status::failed_precondition)?; let vsock_cid = allocate_vsock_cid(); - let mac = mac_from_sandbox_id(sandbox_id); - let mac_str = format!( - "{:02x}:{:02x}:{:02x}:{:02x}:{:02x}:{:02x}", - mac[0], mac[1], mac[2], mac[3], mac[4], mac[5] - ); - let tap = tap_device_name(sandbox_id); - let gateway_port = gateway_port_from_endpoint(&self.config.grpc_endpoint); - let (vcpus, mem_mib) = if is_gpu { (self.config.gpu_vcpus, self.config.gpu_mem_mib) } else { @@ -2219,12 +2497,7 @@ impl VmDriver { kernel_profile: None, kernel_image: None, gpu_bdf, - tap_device: Some(tap), - guest_ip: Some(subnet.guest_ip.to_string()), - host_ip: Some(subnet.host_ip.to_string()), vsock_cid: Some(vsock_cid), - guest_mac: Some(mac_str), - gateway_port, guest_init_dropins: Vec::new(), env: Vec::new(), }) @@ -2296,7 +2569,7 @@ impl VmDriver { message: &str, remove_state: bool, ) { - self.release_gpu_and_subnet(sandbox_id); + self.release_gpu(sandbox_id); let snapshot = { let mut registry = self.registry.lock().await; let Some(record) = registry.get_mut(sandbox_id) else { @@ -2307,7 +2580,6 @@ impl VmDriver { } record.process = None; record.gpu_bdf = None; - record.qemu_network_allocated = false; record.snapshot.status = Some(status_with_condition( &record.snapshot, error_condition(reason, message), @@ -2420,16 +2692,9 @@ impl VmDriver { state_dir: &Path, overlay_disk: &Path, owner_source_disk: &Path, - tls_paths: Option<&VmDriverTlsPaths>, - sandbox_token: Option<&str>, preparation: OverlayPreparation, ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); - let tls_materials = match tls_paths { - Some(paths) => Some(read_guest_tls_materials(paths).await?), - None => None, - }; - let sandbox_token = sandbox_token.map(str::to_string); let overlay_disk = overlay_disk.to_path_buf(); let overlay_size_bytes = self .config @@ -2459,6 +2724,10 @@ impl VmDriver { } let template_path = overlay_template_image(&self.config.state_dir, overlay_size_bytes); + let recover_preserved_overlay = preparation == OverlayPreparation::PreserveExisting + && tokio::fs::metadata(&overlay_disk) + .await + .is_ok_and(|metadata| metadata.is_file()); if !overlay_template_image_ready(&template_path, overlay_size_bytes).await? { let _cache_guard = self.image_cache_lock.lock().await; let template_path = template_path.clone(); @@ -2469,12 +2738,11 @@ impl VmDriver { .map_err(|err| format!("overlay template preparation panicked: {err}"))??; } + let overlay_to_recover = overlay_disk.clone(); let result = tokio::task::spawn_blocking(move || { prepare_sandbox_overlay_image( &template_path, &overlay_disk, - tls_materials.as_ref(), - sandbox_token.as_deref(), preparation, overlay_size_bytes, ) @@ -2482,6 +2750,11 @@ impl VmDriver { .await .map_err(|err| format!("overlay image preparation panicked: {err}"))?; result?; + if recover_preserved_overlay { + tokio::task::spawn_blocking(move || recover_rootfs_image(&overlay_to_recover)) + .await + .map_err(|error| format!("overlay recovery panicked: {error}"))??; + } if write_owner_state && !owner_state_written_before_prepare { write_sandbox_owner_state(state_dir, owner_state).await?; } @@ -3726,62 +3999,109 @@ impl VmDriver { process.clone() }; - let exit_status = { + let poll_result = { let mut process = process.lock().await; if process.deleting { return; } match process.child.try_wait() { - Ok(status) => status, - Err(err) => { - if let Some(snapshot) = self - .set_snapshot_condition( - &sandbox_id, - error_condition("ProcessPollFailed", &err.to_string()), - false, - ) - .await - { - self.publish_snapshot(snapshot); - } - self.publish_platform_event( - sandbox_id.clone(), - platform_event( - "vm", - "Warning", - "ProcessPollFailed", - format!("Failed to poll VM helper process: {err}"), - ), - ); - return; + Ok(Some(status)) => Ok(Some(("VM", status))), + Ok(None) => process + .supervisor + .try_wait() + .map(|status| status.map(|status| ("host supervisor", status))), + Err(error) => Err(error), + } + }; + + let exit_status = match poll_result { + Ok(status) => status, + Err(err) => { + if let Some(snapshot) = self + .set_snapshot_condition( + &sandbox_id, + error_condition("ProcessPollFailed", &err.to_string()), + false, + ) + .await + { + self.publish_snapshot(snapshot); } + self.publish_platform_event( + sandbox_id.clone(), + platform_event( + "vm", + "Warning", + "ProcessPollFailed", + format!("Failed to poll VM sandbox process: {err}"), + ), + ); + return; } }; - if let Some(status) = exit_status { + if let Some((component, status)) = exit_status { let state_dir = { let registry = self.registry.lock().await; registry .get(&sandbox_id) .map(|record| record.state_dir.clone()) }; - if let Some(state_dir) = state_dir - && let Err(error) = write_private_file( - &state_dir.join(MAIN_PROCESS_EXITED_FILE), - b"terminal\n".to_vec(), - ) - .await + if let Some(ref state_dir) = state_dir { + let marker = state_dir.join(MAIN_PROCESS_EXITED_FILE); + if !tokio::fs::try_exists(&marker).await.unwrap_or(false) + && let Err(error) = + write_private_file(&marker, b"terminal\n".to_vec()).await + { + warn!( + sandbox_id = %sandbox_id, + %error, + "vm driver: failed to persist canonical-process exit tombstone" + ); + } + } { - warn!( - sandbox_id = %sandbox_id, - %error, - "vm driver: failed to persist canonical-process exit tombstone" - ); + let mut process = process.lock().await; + if component == "VM" { + let _ = terminate_vm_process(&mut process.supervisor).await; + } else { + let _ = terminate_vm_process(&mut process.child).await; + } } - let message = status.code().map_or_else( - || "VM process exited".to_string(), - |code| format!("VM process exited with status {code}"), + let mut message = status.code().map_or_else( + || format!("{component} process exited"), + |code| format!("{component} process exited with status {code}"), ); + if component == "VM" + && let Some(state_dir) = state_dir.as_deref() + && let Some(console) = read_vm_console_tail( + &state_dir.join("rootfs-console.log"), + VM_CONSOLE_DIAGNOSTIC_BYTES, + ) + { + write!(message, "; guest console tail:\n{console}") + .expect("writing to String cannot fail"); + } + if component == "host supervisor" + && let Some(state_dir) = state_dir.as_deref() + && let Some(stderr) = read_vm_console_tail( + &state_dir.join("supervisor.err.log"), + VM_CONSOLE_DIAGNOSTIC_BYTES, + ) + { + write!(message, "; supervisor stderr tail:\n{stderr}") + .expect("writing to String cannot fail"); + } + if component == "host supervisor" + && let Some(state_dir) = state_dir.as_deref() + && let Some(console) = read_vm_console_tail( + &state_dir.join("rootfs-console.log"), + VM_CONSOLE_DIAGNOSTIC_BYTES, + ) + { + write!(message, "; guest console tail:\n{console}") + .expect("writing to String cannot fail"); + } if let Some(snapshot) = self .set_snapshot_condition( &sandbox_id, @@ -3796,17 +4116,14 @@ impl VmDriver { sandbox_id.clone(), platform_event("vm", "Warning", "ProcessExited", message), ); - let (has_gpu, has_qemu_network, cleanup_ctx) = { + let (has_gpu, cleanup_ctx) = { let registry = self.registry.lock().await; - registry - .get(&sandbox_id) - .map_or((false, false, None), |record| { - ( - record.gpu_bdf.is_some(), - record.qemu_network_allocated, - Some((record.snapshot.clone(), record.state_dir.clone())), - ) - }) + registry.get(&sandbox_id).map_or((false, None), |record| { + ( + record.gpu_bdf.is_some(), + Some((record.snapshot.clone(), record.state_dir.clone())), + ) + }) }; // Give lifecycle extensions a chance to release host // resources they allocated in `before_launch` (e.g. device @@ -3820,7 +4137,9 @@ impl VmDriver { .after_launch_failed(&sandbox, &state_dir, LaunchAbortReason::ProcessExited) .await; } - self.release_allocations(&sandbox_id, has_gpu, has_qemu_network); + if has_gpu { + self.release_gpu(&sandbox_id); + } return; } @@ -3883,6 +4202,27 @@ impl VmDriver { } } +fn read_vm_console_tail(path: &Path, limit: u64) -> Option { + if limit == 0 { + return None; + } + let mut file = fs::File::open(path).ok()?; + let length = file.metadata().ok()?.len(); + file.seek(SeekFrom::Start(length.saturating_sub(limit))) + .ok()?; + let mut bytes = Vec::with_capacity(usize::try_from(length.min(limit)).ok()?); + file.read_to_end(&mut bytes).ok()?; + let text = String::from_utf8_lossy(&bytes); + let text = text.trim_matches(['\0', '\n', '\r']); + (!text.is_empty()).then(|| text.to_string()) +} + +fn configure_main_exit_marker(command: &mut Command, state_dir: &Path) { + command + .arg("--main-exit-marker") + .arg(state_dir.join(MAIN_PROCESS_EXITED_FILE)); +} + #[tonic::async_trait] impl ComputeDriver for VmDriver { async fn authenticate_sandbox( @@ -3986,8 +4326,13 @@ impl ComputeDriver for VmDriver { request: Request, ) -> Result, Status> { let request = request.into_inner(); - self.start_sandbox(&request.sandbox_id, &request.sandbox_name) - .await?; + self.start_sandbox( + &request.sandbox_id, + &request.sandbox_name, + &request.generation_id, + request.launch_authentication, + ) + .await?; Ok(Response::new(StartSandboxResponse {})) } @@ -4078,8 +4423,8 @@ impl ComputeDriver for VmDriver { fn check_gpu_privileges() -> Result<(), String> { if !rustix::process::geteuid().is_root() { return Err( - "GPU support requires root privileges for VFIO bind/unbind and TAP networking. \ - Run with sudo or ensure CAP_SYS_ADMIN + CAP_NET_ADMIN capabilities are set." + "GPU support requires root privileges for VFIO bind/unbind. \ + Run with sudo or grant the host device-management capabilities required by VFIO." .to_string(), ); } @@ -5204,147 +5549,25 @@ fn merged_environment(sandbox: &Sandbox) -> HashMap { environment } -/// Rewrites loopback host references in a gateway URL to a hostname the guest -/// can reach via gvproxy. -/// -/// The driver receives the gateway endpoint from `--grpc-endpoint`, which -/// in local/dev/e2e setups is typically `http://127.0.0.1:`. That URL is -/// useless inside the guest because the guest's loopback interface is its own, -/// not the host's. Inside the guest we need a name that gvproxy will translate -/// into the host's loopback address. -/// -/// We rewrite to `host.openshell.internal`, which gvproxy's embedded DNS resolves -/// to the host-loopback IP `192.168.127.254`. gvproxy installs a default NAT entry -/// rewriting that destination to the host's `127.0.0.1` and dialing out from the -/// host process, so any port the host is listening on becomes reachable. The -/// gateway IP `192.168.127.1` does **not** do this — it only listens on gvproxy's -/// own service ports (DNS, DHCP, HTTP API). The guest init script also seeds the -/// hostname in `/etc/hosts` so resolution works even if gvproxy's DNS isn't in -/// resolv.conf (e.g. when DHCP fails). -/// -/// Non-loopback URLs are returned unchanged. -fn guest_visible_openshell_endpoint(endpoint: &str) -> String { - let Ok(mut url) = Url::parse(endpoint) else { - return endpoint.to_string(); - }; - - let should_rewrite = match url.host() { - Some(Host::Ipv4(ip)) => ip.is_loopback(), - Some(Host::Ipv6(ip)) => ip.is_loopback(), - Some(Host::Domain(host)) => host.eq_ignore_ascii_case("localhost"), - None => false, - }; - - if should_rewrite && url.set_host(Some(GVPROXY_HOST_LOOPBACK_ALIAS)).is_ok() { - return url.to_string(); - } - - endpoint.to_string() -} - -/// Whether a corporate proxy URL points at the gateway host itself, as seen -/// from a QEMU/TAP guest whose TAP host address is `tap_host_ip`. -/// -/// On the libkrun backend gvproxy NATs the host-loopback alias -/// `host.openshell.internal` (and any loopback URL, which the driver rewrites -/// to that alias) to the gateway host's `127.0.0.1`, so a proxy bound to host -/// loopback is reachable from the guest. The QEMU/TAP backend used for GPU -/// sandboxes has no equivalent: `host.openshell.internal` resolves to the TAP -/// host address, and the driver's own nftables `input` chain accepts only the -/// gateway port from the guest and drops the rest, so no proxy on the gateway -/// host is reachable regardless of the address it binds. -/// -/// The gateway host is therefore reached from a QEMU guest under exactly three -/// spellings: the guest's own loopback (never the host's, but a configuration -/// that plainly means the host), the documented host aliases that -/// `write_host_gateway_aliases` seeds to the TAP host address, and that TAP -/// host address written literally. `tap_host_ip` is this sandbox's allocated -/// address, so the comparison must be made after the launch plan's subnet -/// allocation; `None` means the plan carries no TAP host and only the -/// address-independent spellings are classified. -/// -/// gvproxy's `GVPROXY_HOST_LOOPBACK_IP` is deliberately **not** matched here. -/// It is special only to libkrun; on QEMU/TAP it is an ordinary address that -/// may well be routable through the guest's masqueraded egress, and rejecting -/// it would refuse a working configuration. -/// -/// Used to reject an unreachable configuration up front on the QEMU path -/// instead of letting every policy-approved CONNECT time out. -fn proxy_url_targets_gateway_host(url: &str, tap_host_ip: Option<&str>) -> bool { - let Ok(parsed) = Url::parse(url) else { - // Unparseable URLs are rejected by shared validation before launch. - return false; - }; - let tap_host = tap_host_ip.and_then(|ip| ip.parse::().ok()); - match parsed.host() { - Some(Host::Ipv4(ip)) => ip.is_loopback() || tap_host == Some(IpAddr::V4(ip)), - Some(Host::Ipv6(ip)) => ip.is_loopback() || tap_host == Some(IpAddr::V6(ip)), - Some(Host::Domain(host)) => { - host.eq_ignore_ascii_case("localhost") - || host.eq_ignore_ascii_case(OPENSHELL_HOST_GATEWAY_ALIAS) - || host.eq_ignore_ascii_case("host.containers.internal") - || host.eq_ignore_ascii_case("host.docker.internal") - } - None => false, - } -} - -fn gateway_port_from_endpoint(endpoint: &str) -> Option { - Url::parse(endpoint).ok().and_then(|url| url.port()) -} - -fn has_complete_qemu_network(plan: &LaunchPlan) -> bool { - plan.tap_device.is_some() - && plan.guest_ip.is_some() - && plan.host_ip.is_some() - && plan.vsock_cid.is_some() - && plan.guest_mac.is_some() -} - -fn guest_visible_openshell_endpoint_for_tap(endpoint: &str, host_ip: &str) -> String { - let Ok(mut url) = Url::parse(endpoint) else { - return endpoint.to_string(); - }; - if url.set_host(Some(host_ip)).is_ok() { - url.to_string() - } else { - endpoint.to_string() +fn random_boundary_token() -> String { + let mut token = String::with_capacity(64); + for byte in rand::random::<[u8; 32]>() { + write!(&mut token, "{byte:02x}").expect("writing to String cannot fail"); } + token } -fn build_guest_environment( - sandbox: &Sandbox, - config: &VmDriverConfig, - endpoint_override: Option<&str>, -) -> Vec { - let openshell_endpoint = endpoint_override.map_or_else( - || guest_visible_openshell_endpoint(&config.grpc_endpoint), - String::from, - ); - // 1. User-supplied environment (lowest priority). - let user_env = merged_environment(sandbox); +fn build_guest_environment(sandbox: &Sandbox, config: &VmDriverConfig) -> Vec { + // The guest receives only driver-owned boot metadata. Gateway credentials, + // TLS material, and logical-supervisor configuration remain on the host; + // workload environment is carried in the authenticated BoundaryConfig. let mut environment: HashMap = HashMap::new(); - environment.extend(user_env.clone()); - if !user_env.is_empty() - && let Ok(json) = serde_json::to_string(&user_env) - { - environment.insert( - openshell_core::sandbox_env::USER_ENVIRONMENT.to_string(), - json, - ); - } - - // 2. Required driver vars (highest priority -- always overwrite). environment.insert("HOME".to_string(), "/root".to_string()); environment.insert( "PATH".to_string(), "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin".to_string(), ); environment.insert("TERM".to_string(), "xterm".to_string()); - environment.insert( - openshell_core::sandbox_env::ENDPOINT.to_string(), - openshell_endpoint, - ); environment.insert( openshell_core::sandbox_env::SANDBOX_ID.to_string(), sandbox.id.clone(), @@ -5353,74 +5576,14 @@ fn build_guest_environment( openshell_core::sandbox_env::SANDBOX.to_string(), sandbox.name.clone(), ); - environment.insert( - openshell_core::sandbox_env::SSH_SOCKET_PATH.to_string(), - GUEST_SSH_SOCKET_PATH.to_string(), - ); - // The libkrun guest environment path does not preserve spaces in values - // before guest startup. Use a whitespace-free base64url envelope so - // command arguments remain lossless. - let main_process = - openshell_core::sandbox_env::MainProcessConfig::encode_driver_spec_base64url( - sandbox.spec.as_ref(), - ) - .expect("main process config serialization cannot fail"); - environment.insert( - openshell_core::sandbox_env::MAIN_PROCESS_SPEC.to_string(), - main_process, - ); environment.insert( openshell_core::sandbox_env::LOG_LEVEL.to_string(), openshell_core::driver_utils::sandbox_log_level(sandbox, &config.log_level), ); - if config.requires_tls_materials() { - environment.insert( - openshell_core::sandbox_env::TLS_CA.to_string(), - GUEST_TLS_CA_PATH.to_string(), - ); - environment.insert( - openshell_core::sandbox_env::TLS_CERT.to_string(), - GUEST_TLS_CERT_PATH.to_string(), - ); - environment.insert( - openshell_core::sandbox_env::TLS_KEY.to_string(), - GUEST_TLS_KEY_PATH.to_string(), - ); - } - if let Some(endpoint) = config.provider_spiffe_workload_api_tcp_endpoint.as_ref() { - environment.insert( - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET.to_string(), - endpoint.clone(), - ); - } environment.insert( openshell_core::sandbox_env::TELEMETRY_ENABLED.to_string(), openshell_core::telemetry::enabled_env_value().to_string(), ); - // Runtime capabilities are driver-owned. The VM driver does not yet - // provide policy DNS and transparent TCP interception. - environment.insert( - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES.to_string(), - String::new(), - ); - environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN); - environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN_FILE); - // Prevent user-supplied environment from overriding the TLS server name - // the supervisor verifies — a sandbox user who can redirect the gateway - // hostname could otherwise present a certificate for a name they control - // and intercept the sandbox JWT. - environment.remove(openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME); - if sandbox - .spec - .as_ref() - .is_some_and(|spec| !spec.sandbox_token.is_empty()) - { - environment.insert( - openshell_core::sandbox_env::SANDBOX_TOKEN_FILE.to_string(), - GUEST_SANDBOX_TOKEN_PATH.to_string(), - ); - } - let mut pairs = environment.into_iter().collect::>(); pairs.sort_by(|left, right| left.0.cmp(&right.0)); pairs @@ -5826,8 +5989,9 @@ fn write_oci_layout_for_manifest( fn bootstrap_image_cache_identity(image_identity: &str) -> String { format!( - "{BOOTSTRAP_IMAGE_CACHE_LAYOUT_VERSION}:openshell-{}:{image_identity}", - openshell_core::VERSION + "{BOOTSTRAP_IMAGE_CACHE_LAYOUT_VERSION}:openshell-{}:guest-{}:{image_identity}", + openshell_core::VERSION, + sandbox_guest_runtime_identity() ) } @@ -5907,11 +6071,22 @@ async fn read_persisted_image_identity(state_dir: &Path) -> Result Result<(), std::io::Error> { restrict_owner_only_dir(state_dir).await?; - write_private_file( - &state_dir.join(SANDBOX_REQUEST_FILE), - sandbox.encode_to_vec(), - ) - .await + let destination = state_dir.join(SANDBOX_REQUEST_FILE); + let sequence = IMAGE_CACHE_BUILD_COUNTER.fetch_add(1, Ordering::Relaxed); + let temporary = state_dir.join(format!( + ".{SANDBOX_REQUEST_FILE}.{}.{}.tmp", + std::process::id(), + sequence + )); + if let Err(error) = write_private_file(&temporary, sandbox.encode_to_vec()).await { + let _ = tokio::fs::remove_file(&temporary).await; + return Err(error); + } + if let Err(error) = tokio::fs::rename(&temporary, destination).await { + let _ = tokio::fs::remove_file(&temporary).await; + return Err(error); + } + Ok(()) } async fn read_sandbox_request(path: &Path) -> Result { @@ -5929,6 +6104,47 @@ async fn write_private_file(path: &Path, bytes: Vec) -> Result<(), std::io:: restrict_owner_read_write(path).await } +async fn remove_runtime_generation_material(state_dir: &Path) -> Result<(), String> { + let generation_path = state_dir.join(HOST_BOUNDARY_GENERATION_FILE); + let generation = match tokio::fs::read_to_string(&generation_path).await { + Ok(generation) => generation.trim().to_string(), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => String::new(), + Err(error) => return Err(format!("read boundary generation marker: {error}")), + }; + if !generation.is_empty() { + let overlay = sandbox_runtime_disk_paths(state_dir).overlay_disk; + let generation_for_cleanup = generation.clone(); + tokio::task::spawn_blocking(move || { + let tls = guest_boundary_tls_paths(&generation_for_cleanup); + for guest_path in [ + PathBuf::from(guest_boundary_config_path(&generation_for_cleanup)), + tls.certificate_chain_path, + tls.private_key_path, + ] { + remove_rootfs_image_file( + &overlay, + &overlay_upper_path(&guest_path.to_string_lossy()), + )?; + } + Ok::<(), String>(()) + }) + .await + .map_err(|error| format!("guest authentication cleanup task failed: {error}"))??; + } + for path in [ + state_dir.join(HOST_AUTH_BUNDLE_FILE), + state_dir.join(HOST_RUNTIME_DESCRIPTOR_FILE), + generation_path, + ] { + match tokio::fs::remove_file(&path).await { + Ok(()) => {} + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => return Err(format!("remove {}: {error}", path.display())), + } + } + Ok(()) +} + #[cfg(unix)] async fn restrict_owner_read_write(path: &Path) -> Result<(), std::io::Error> { tokio::fs::set_permissions(path, fs::Permissions::from_mode(0o600)).await @@ -5962,26 +6178,6 @@ fn validate_restored_sandbox_state( Ok(()) } -#[derive(Debug, Clone)] -struct GuestTlsMaterials { - ca: Vec, - cert: Vec, - key: Vec, -} - -async fn read_guest_tls_materials(paths: &VmDriverTlsPaths) -> Result { - let ca = tokio::fs::read(&paths.ca) - .await - .map_err(|err| format!("read {}: {err}", paths.ca.display()))?; - let cert = tokio::fs::read(&paths.cert) - .await - .map_err(|err| format!("read {}: {err}", paths.cert.display()))?; - let key = tokio::fs::read(&paths.key) - .await - .map_err(|err| format!("read {}: {err}", paths.key.display()))?; - Ok(GuestTlsMaterials { ca, cert, key }) -} - async fn overlay_template_image_ready(path: &Path, size_bytes: u64) -> Result { match tokio::fs::metadata(path).await { Ok(metadata) => Ok(metadata.is_file() && metadata.len() == size_bytes), @@ -6066,36 +6262,19 @@ fn create_empty_sandbox_overlay_image(overlay_disk: &Path, size_bytes: u64) -> R fn create_sandbox_overlay_image_from_template( template_path: &Path, overlay_disk: &Path, - tls_materials: Option<&GuestTlsMaterials>, - sandbox_token: Option<&str>, ) -> Result<(), String> { - clone_or_copy_sparse_file(template_path, overlay_disk)?; - if let Some(tls) = tls_materials { - inject_guest_tls_materials(overlay_disk, tls)?; - } - if let Some(token) = sandbox_token { - inject_guest_sandbox_token(overlay_disk, token)?; - } - Ok(()) + clone_or_copy_sparse_file(template_path, overlay_disk) } fn prepare_sandbox_overlay_image( template_path: &Path, overlay_disk: &Path, - tls_materials: Option<&GuestTlsMaterials>, - sandbox_token: Option<&str>, preparation: OverlayPreparation, expected_size_bytes: u64, ) -> Result<(), String> { if preparation == OverlayPreparation::PreserveExisting { match fs::metadata(overlay_disk) { Ok(metadata) if metadata.is_file() && metadata.len() == expected_size_bytes => { - if let Some(tls) = tls_materials { - inject_guest_tls_materials(overlay_disk, tls)?; - } - if let Some(token) = sandbox_token { - inject_guest_sandbox_token(overlay_disk, token)?; - } return Ok(()); } Ok(metadata) if metadata.is_file() => { @@ -6122,37 +6301,53 @@ fn prepare_sandbox_overlay_image( } } - create_sandbox_overlay_image_from_template( - template_path, - overlay_disk, - tls_materials, - sandbox_token, - ) + create_sandbox_overlay_image_from_template(template_path, overlay_disk) } -fn inject_guest_tls_materials( +fn inject_guest_boundary_bundle( overlay_disk: &Path, - materials: &GuestTlsMaterials, + guest_path: &str, + config: &BoundaryConfig, + material: &SandboxTlsMaterial, ) -> Result<(), String> { - write_rootfs_image_file( - overlay_disk, - &overlay_upper_path(GUEST_TLS_CA_PATH), - &materials.ca, - )?; - write_rootfs_image_file( - overlay_disk, - &overlay_upper_path(GUEST_TLS_CERT_PATH), - &materials.cert, - )?; - let key_path = overlay_upper_path(GUEST_TLS_KEY_PATH); - write_rootfs_image_file(overlay_disk, &key_path, &materials.key)?; - set_rootfs_image_file_mode(overlay_disk, &key_path, 0o600) -} - -fn inject_guest_sandbox_token(overlay_disk: &Path, token: &str) -> Result<(), String> { - let token_path = overlay_upper_path(GUEST_SANDBOX_TOKEN_PATH); - write_rootfs_image_file(overlay_disk, &token_path, format!("{token}\n").as_bytes())?; - set_rootfs_image_file_mode(overlay_disk, &token_path, 0o600) + let tls = match &config.listener { + BoundaryListener::Unix { tls, .. } + | BoundaryListener::TlsTcp { tls, .. } + | BoundaryListener::Vsock { tls, .. } => tls.clone(), + }; + let encoded_config = config + .encode() + .map_err(|error| format!("encode VM boundary configuration: {error}"))?; + let config_path = overlay_upper_path(guest_path); + write_rootfs_image_file(overlay_disk, &config_path, &encoded_config)?; + set_rootfs_image_file_mode(overlay_disk, &config_path, 0o600)?; + for (guest_path, contents) in [ + ( + tls.certificate_chain_path, + material.certificate_chain_pem.as_bytes(), + ), + (tls.private_key_path, material.private_key_pem.as_bytes()), + ] { + let path = overlay_upper_path(guest_path.to_string_lossy().as_ref()); + write_rootfs_image_file(overlay_disk, &path, contents)?; + set_rootfs_image_file_mode(overlay_disk, &path, 0o600)?; + } + Ok(()) +} + +fn guest_boundary_config_path(generation: &str) -> String { + format!("{GUEST_BOUNDARY_CONFIG_DIR}/bootstrap-{generation}.json") +} + +fn guest_boundary_tls_paths(generation: &str) -> SandboxTlsServerConfig { + SandboxTlsServerConfig { + certificate_chain_path: PathBuf::from(format!( + "{GUEST_BOUNDARY_CONFIG_DIR}/sandbox-{generation}.crt" + )), + private_key_path: PathBuf::from(format!( + "{GUEST_BOUNDARY_CONFIG_DIR}/sandbox-{generation}.key" + )), + } } #[allow(clippy::result_large_err)] @@ -6202,26 +6397,34 @@ fn inject_guest_init_dropins( span_status.finish(Ok(())) } -/// Build the corporate upstream-proxy arguments passed to the guest supervisor. +/// Build the corporate upstream-proxy arguments passed to host control. /// /// This operator-owned egress boundary travels on the supervisor's argv, /// which sandbox spec/template environment and image `ENV` cannot influence. -/// Credentials are never on argv — only the root-only guest path is passed; -/// the supervisor reads the credential from that file. -fn upstream_proxy_cli_args(config: &VmDriverConfig) -> Vec { +/// Credentials are never on argv; the supervisor reads them from the +/// operator-owned host file. +fn upstream_proxy_cli_args(config: &VmDriverConfig) -> Result, String> { let mut args = Vec::new(); if let Some(url) = &config.upstream_proxy.https_proxy { args.push("--upstream-proxy".to_string()); args.push(url.clone()); + let proxy_url = Url::parse(url) + .map_err(|error| format!("invalid upstream proxy endpoint '{url}': {error}"))?; + if proxy_url + .host_str() + .is_some_and(|host| HOST_LOOPBACK_ALIASES.contains(&host)) + { + args.push("--upstream-proxy-dial-ip".to_string()); + args.push("127.0.0.1".to_string()); + } } if let Some(list) = &config.upstream_proxy.no_proxy { args.push("--upstream-no-proxy".to_string()); args.push(list.clone()); } - if config.upstream_proxy.proxy_auth_file.is_some() { + if let Some(path) = &config.upstream_proxy.proxy_auth_file { args.push("--upstream-proxy-auth-file".to_string()); - // The guest path, never the gateway-host path the operator configured. - args.push(GUEST_UPSTREAM_PROXY_AUTH_PATH.to_string()); + args.push(path.display().to_string()); } // Config validation guarantees the acknowledgement is `true` whenever an // auth file is configured against an http:// proxy; the supervisor @@ -6234,156 +6437,11 @@ fn upstream_proxy_cli_args(config: &VmDriverConfig) -> Vec { if config.upstream_proxy.proxy_connect_by_hostname == Some(true) { args.push("--upstream-proxy-connect-by-hostname".to_string()); } - if config.proxy_ca_bundle.is_some() { + if let Some(path) = &config.proxy_ca_bundle { args.push("--upstream-proxy-ca-bundle".to_string()); - // The guest path, never the gateway-host path the operator configured. - args.push(GUEST_PROXY_CA_PATH.to_string()); - } - args -} - -/// Render the supervisor argument list as newline-separated arguments. -/// -/// One argument per line, verbatim: the guest reads the lines into an array -/// without word splitting or globbing, so values containing spaces survive -/// intact. An empty list renders an empty file, which the guest reads as "no -/// extra arguments". -fn render_guest_supervisor_args(args: &[String]) -> Vec { - let mut body = args.join("\n"); - if !body.is_empty() { - body.push('\n'); - } - body.into_bytes() -} - -/// Reject argument values the newline-delimited guest file cannot represent. -/// -/// Every value here is operator-supplied config, so this is a guard against -/// misconfiguration rather than an attack: a stray newline would otherwise -/// split one value into two arguments in the guest. -fn validate_guest_supervisor_args(args: &[String]) -> Result<(), String> { - for arg in args { - if arg.contains('\n') || arg.contains('\r') || arg.contains('\0') { - return Err( - "corporate proxy settings must not contain newline or NUL characters".to_string(), - ); - } + args.push(path.display().to_string()); } - Ok(()) -} - -/// Read and validate the corporate proxy credential from the gateway host. -/// -/// Uses the validators shared with the supervisor, so a credential accepted -/// here is never rejected inside the guest. The error never carries the file -/// contents. -async fn read_sandbox_proxy_credential(path: &Path) -> Result { - let path_owned = path.to_path_buf(); - let display_path = path.display().to_string(); - let raw = tokio::task::spawn_blocking(move || { - let path = path_owned - .to_str() - .ok_or_else(|| "proxy_auth_file path is not valid UTF-8".to_string())?; - openshell_core::driver_utils::read_upstream_proxy_credential_file(path) - }) - .await - .map_err(|err| Status::internal(format!("proxy_auth_file read task failed: {err}")))? - .map_err(Status::invalid_argument)?; - let credential = - openshell_core::driver_utils::parse_upstream_proxy_credential(&raw).map_err(|err| { - Status::invalid_argument(format!("proxy_auth_file '{display_path}': {err}")) - })?; - Ok(credential.to_string()) -} - -/// Read and validate the corporate proxy CA bundle from the gateway host. -/// -/// The validation rejects symlinks and non-regular files, bounds the read, -/// requires PEM certificate markers, and never includes file contents in an -/// error. -async fn read_sandbox_proxy_ca_bundle(path: &Path) -> Result, Status> { - let path_owned = path.to_path_buf(); - let display_path = path.display().to_string(); - tokio::task::spawn_blocking(move || { - let path = path_owned - .to_str() - .ok_or_else(|| "proxy_ca_bundle path is not valid UTF-8".to_string())?; - openshell_core::driver_utils::read_upstream_proxy_ca_bundle_file(path, "proxy_ca_bundle") - .map(String::into_bytes) - }) - .await - .map_err(|err| Status::internal(format!("proxy_ca_bundle read task failed: {err}")))? - .map_err(|err| { - Status::invalid_argument(format!( - "proxy_ca_bundle '{display_path}' could not be read: {err}" - )) - }) -} - -/// Stage the corporate upstream-proxy configuration into the guest overlay. -/// -/// Writes three files into the overlay upperdir the driver owns: -/// -/// * the credential at [`GUEST_UPSTREAM_PROXY_AUTH_PATH`], mode `0600`; -/// * the CA bundle at [`GUEST_PROXY_CA_PATH`], mode `0644` (a CA certificate -/// is not secret); -/// * the supervisor argument list at [`GUEST_SUPERVISOR_ARGS_PATH`], mode -/// `0644`. -/// -/// Both are written on every launch, empty when the corresponding -/// setting is absent. Writing rather than skipping is what makes the channel -/// unforgeable: the upperdir copy always shadows the read-only image layer, so -/// a sandbox image cannot supply its own arguments or credential by baking a -/// file at these paths, and cannot disable the operator's by omitting one. It -/// also clears material a previous launch staged into a preserved overlay -/// after the operator removed the setting. -/// -/// A microVM has no bind mounts or container secrets, so the credential lives -/// at rest inside the per-sandbox overlay disk on the host — the same -/// delivery the per-sandbox gateway JWT already uses. It is removed with the -/// sandbox when the state directory is deleted. -#[allow(clippy::result_large_err)] -async fn inject_guest_upstream_proxy( - overlay_disk: &Path, - config: &VmDriverConfig, -) -> Result<(), Status> { - // Written whether or not they are configured. Writing empty files when - // the operator removed a setting clears material a previous launch staged - // into a preserved overlay, and shadows anything an image baked at these - // paths, so a staged file is only ever the one this launch produced. - let credential = match config.upstream_proxy.proxy_auth_file.as_deref() { - Some(path) => format!("{}\n", read_sandbox_proxy_credential(path).await?).into_bytes(), - None => Vec::new(), - }; - let credential_path = overlay_upper_path(GUEST_UPSTREAM_PROXY_AUTH_PATH); - write_rootfs_image_file(overlay_disk, &credential_path, &credential) - .map_err(|err| Status::internal(format!("write VM guest proxy credential: {err}")))?; - set_rootfs_image_file_mode(overlay_disk, &credential_path, 0o600) - .map_err(|err| Status::internal(format!("set VM guest proxy credential mode: {err}")))?; - - let ca_bundle = match config.proxy_ca_bundle.as_deref() { - Some(path) => read_sandbox_proxy_ca_bundle(path).await?, - None => Vec::new(), - }; - let ca_path = overlay_upper_path(GUEST_PROXY_CA_PATH); - write_rootfs_image_file(overlay_disk, &ca_path, &ca_bundle) - .map_err(|err| Status::internal(format!("write VM guest proxy CA bundle: {err}")))?; - set_rootfs_image_file_mode(overlay_disk, &ca_path, 0o644) - .map_err(|err| Status::internal(format!("set VM guest proxy CA bundle mode: {err}")))?; - - let args = upstream_proxy_cli_args(config); - validate_guest_supervisor_args(&args).map_err(Status::failed_precondition)?; - let guest_path = overlay_upper_path(GUEST_SUPERVISOR_ARGS_PATH); - write_rootfs_image_file( - overlay_disk, - &guest_path, - &render_guest_supervisor_args(&args), - ) - .map_err(|err| Status::internal(format!("write VM guest supervisor arguments: {err}")))?; - set_rootfs_image_file_mode(overlay_disk, &guest_path, 0o644).map_err(|err| { - Status::internal(format!("set VM guest supervisor arguments mode: {err}")) - })?; - Ok(()) + Ok(args) } /// Render the drop-in allow-list as newline-separated, ASCII-sorted, @@ -6583,47 +6641,6 @@ fn dir_size_bytes(path: &Path) -> Result { Ok(total) } -#[cfg(test)] -fn stage_guest_tls_materials( - staging_dir: &Path, - materials: &GuestTlsMaterials, -) -> Result<(), String> { - let tls_dir = staging_dir - .join("upper") - .join(GUEST_TLS_CA_PATH.trim_start_matches('/')) - .parent() - .ok_or_else(|| "guest TLS CA path has no parent".to_string())? - .to_path_buf(); - fs::create_dir_all(&tls_dir) - .map_err(|err| format!("create guest TLS dir {}: {err}", tls_dir.display()))?; - - let ca_path = staging_dir - .join("upper") - .join(GUEST_TLS_CA_PATH.trim_start_matches('/')); - let cert_path = staging_dir - .join("upper") - .join(GUEST_TLS_CERT_PATH.trim_start_matches('/')); - let key_path = staging_dir - .join("upper") - .join(GUEST_TLS_KEY_PATH.trim_start_matches('/')); - fs::write(&ca_path, &materials.ca) - .map_err(|err| format!("write guest TLS CA {}: {err}", ca_path.display()))?; - fs::write(&cert_path, &materials.cert) - .map_err(|err| format!("write guest TLS cert {}: {err}", cert_path.display()))?; - fs::write(&key_path, &materials.key) - .map_err(|err| format!("write guest TLS key {}: {err}", key_path.display()))?; - - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt as _; - - fs::set_permissions(&key_path, fs::Permissions::from_mode(0o600)) - .map_err(|err| format!("chmod guest TLS key {}: {err}", key_path.display()))?; - } - - Ok(()) -} - fn overlay_staging_dir(overlay_disk: &Path) -> PathBuf { let parent = overlay_disk.parent().unwrap_or_else(|| Path::new(".")); parent.join(format!( @@ -6653,6 +6670,34 @@ async fn terminate_vm_process(child: &mut Child) -> Result<(), std::io::Error> { } } +async fn terminate_sandbox_processes(process: &mut VmProcess) -> Result<(), std::io::Error> { + process.supervisor_liveness.take(); + let supervisor_error = terminate_vm_process(&mut process.supervisor).await.err(); + let vm_error = terminate_vm_process(&mut process.child).await.err(); + + match (supervisor_error, vm_error) { + (None, None) => Ok(()), + (Some(error), None) => Err(std::io::Error::other(format!("stop supervisor: {error}"))), + (None, Some(error)) => Err(std::io::Error::other(format!("stop vm: {error}"))), + (Some(supervisor), Some(vm)) => Err(std::io::Error::other(format!( + "stop supervisor: {supervisor}; stop vm: {vm}" + ))), + } +} + +fn absolute_state_dir(state_dir: &Path) -> Result { + if state_dir.is_absolute() { + return Ok(state_dir.to_path_buf()); + } + std::env::current_dir() + .map(|working_dir| working_dir.join(state_dir)) + .map_err(|err| format!("failed to resolve VM driver state directory: {err}")) +} + +fn isolate_host_control_environment(command: &mut Command) { + command.env_clear(); +} + #[tracing::instrument( name = "vm.launch", skip(command), @@ -6663,6 +6708,7 @@ async fn terminate_vm_process(child: &mut Child) -> Result<(), std::io::Error> { vm.backend = ?backend, ) )] +#[allow(dead_code)] fn spawn_vm_launcher( command: &mut Command, sandbox_id: &str, @@ -6684,6 +6730,7 @@ fn sandbox_snapshot(sandbox: &Sandbox, condition: SandboxCondition, deleting: bo sandbox_fd: String::new(), conditions: vec![condition], deleting, + ..Default::default() }), ..Default::default() } @@ -6701,6 +6748,7 @@ fn status_with_condition( sandbox_fd: String::new(), conditions: vec![condition], deleting, + ..Default::default() } } @@ -6845,7 +6893,7 @@ fn pulling_layer_detail(metadata: &HashMap) -> Option { #[cfg(test)] mod tests { use super::*; - use crate::gpu::{SubnetAllocator, allocate_vsock_cid, mac_from_sandbox_id, tap_device_name}; + use crate::gpu::allocate_vsock_cid; use openshell_core::progress::{ PROGRESS_ACTIVE_DETAIL_KEY, PROGRESS_ACTIVE_STEP_KEY, PROGRESS_COMPLETE_LABEL_KEY, PROGRESS_COMPLETE_STEP_KEY, @@ -6864,6 +6912,23 @@ mod tests { static ENV_LOCK: std::sync::LazyLock> = std::sync::LazyLock::new(|| std::sync::Mutex::new(())); + #[test] + fn vm_console_diagnostic_is_bounded_to_the_tail() { + let directory = tempfile::tempdir().unwrap(); + let console = directory.path().join("rootfs-console.log"); + fs::write(&console, b"discard-this\nFATAL: sandbox startup failed\n").unwrap(); + + assert_eq!( + read_vm_console_tail(&console, 30).as_deref(), + Some("FATAL: sandbox startup failed") + ); + assert_eq!(read_vm_console_tail(&console, 0), None); + assert_eq!( + read_vm_console_tail(&directory.path().join("missing"), 30), + None + ); + } + #[test] fn registry_throttling_errors_are_retryable() { let error = OciDistributionError::RegistryError { @@ -7402,8 +7467,6 @@ mod tests { Path::new("/unused"), Path::new("/unused"), Path::new("/unused"), - None, - None, OverlayPreparation::Fresh, ) .instrument(parent) @@ -8462,6 +8525,10 @@ mod tests { let sandbox = Sandbox { id: "sandbox-stopped".to_string(), name: "stopped".to_string(), + spec: Some(SandboxSpec { + launch_authentication: test_launch_authentication("old").0, + ..Default::default() + }), ..Default::default() }; let state_dir = temp.path().join("sandboxes").join(&sandbox.id); @@ -8479,13 +8546,18 @@ mod tests { process: None, provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }, ); + let (fresh_authentication, fresh_session) = test_launch_authentication("fresh"); let err = driver - .start_sandbox(&sandbox.id, &sandbox.name) + .start_sandbox( + &sandbox.id, + &sandbox.name, + "g0000000000000001", + fresh_authentication, + ) .await .expect_err("start without an image should fail"); @@ -8508,6 +8580,54 @@ mod tests { .expect("stopped condition"); assert_eq!(condition.r#type, "Stopped"); assert_eq!(condition.status, "True"); + let persisted = read_sandbox_request(&state_dir.join(SANDBOX_REQUEST_FILE)) + .await + .expect("persisted sandbox request"); + let persisted_authentication = + serde_json::from_slice::( + &persisted + .spec + .expect("persisted sandbox spec") + .launch_authentication, + ) + .expect("persisted launch authentication"); + assert_eq!( + persisted_authentication.supervisor.session_id, + fresh_session + ); + } + + fn test_launch_authentication(label: &str) -> (Vec, openshell_core::SandboxSessionId) { + use openshell_core::jwt::{ + SandboxLaunchAuthentication, SecretJwt, SessionVerificationKey, SupervisorAuthBundle, + }; + + let session_id = openshell_core::SandboxSessionId::new(); + let authentication = SandboxLaunchAuthentication { + supervisor: SupervisorAuthBundle { + session_id, + runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-1", + ) + .expect("runtime generation"), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).expect("auth epoch"), + gateway_token: SecretJwt::parse(format!("gateway-{label}")).expect("gateway token"), + gateway_expires_at: 1, + sandbox_token: SecretJwt::parse(format!("sandbox-{label}")).expect("sandbox token"), + sandbox_expires_at: 1, + }, + gateway_id: "gateway-a".to_string(), + verification_keys: vec![SessionVerificationKey { + key_id: "key-a".to_string(), + public_key_pem: b"public-key".to_vec(), + }], + }; + ( + serde_json::to_vec(&authentication).expect("encode launch authentication"), + session_id, + ) } #[test] @@ -8522,8 +8642,6 @@ mod tests { prepare_sandbox_overlay_image( &template, &overlay, - None, - None, OverlayPreparation::PreserveExisting, "saved-overlay".len() as u64, ) @@ -8545,8 +8663,6 @@ mod tests { prepare_sandbox_overlay_image( &template, &overlay, - None, - None, OverlayPreparation::PreserveExisting, "fresh-overlay".len() as u64, ) @@ -8560,8 +8676,8 @@ mod tests { #[test] fn overlay_upper_path_targets_overlay_upperdir() { assert_eq!( - overlay_upper_path(GUEST_TLS_KEY_PATH), - "/upper/opt/openshell/tls/tls.key" + overlay_upper_path(&guest_boundary_config_path("generation-123")), + "/upper/.openshell/state/bootstrap-generation-123.json" ); } @@ -8577,16 +8693,30 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events: broadcast::channel(WATCH_BUFFER).0, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; assert_eq!(driver.capabilities().default_image, "openshell/sandbox:dev"); } + #[test] + fn host_control_receives_driver_owned_completion_marker() { + let mut command = Command::new("openshell-sandbox"); + configure_main_exit_marker(&mut command, Path::new("/private/sandboxes/sb-1")); + let args = command + .as_std() + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + assert_eq!( + args, + [ + "--main-exit-marker".to_string(), + "/private/sandboxes/sb-1/main-process-exited".to_string(), + ] + ); + } + #[test] fn resolved_sandbox_image_prefers_template_image() { let driver = VmDriver { @@ -8599,10 +8729,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events: broadcast::channel(WATCH_BUFFER).0, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; let sandbox = Sandbox { @@ -8634,10 +8760,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events: broadcast::channel(WATCH_BUFFER).0, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; let sandbox = Sandbox { @@ -8663,10 +8785,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events: broadcast::channel(WATCH_BUFFER).0, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; let sandbox = Sandbox { @@ -8693,10 +8811,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events: broadcast::channel(WATCH_BUFFER).0, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; @@ -8718,10 +8832,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events: broadcast::channel(WATCH_BUFFER).0, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; @@ -8740,10 +8850,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events: broadcast::channel(WATCH_BUFFER).0, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; @@ -8775,7 +8881,7 @@ mod tests { } #[test] - fn build_guest_environment_sets_supervisor_defaults() { + fn build_guest_environment_sets_sandbox_boot_metadata() { let config = VmDriverConfig { grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() @@ -8787,16 +8893,18 @@ mod tests { ..Default::default() }; - let env = build_guest_environment(&sandbox, &config, None); + let env = build_guest_environment(&sandbox, &config); assert!(env.contains(&"HOME=/root".to_string())); - assert!(env.contains(&format!( - "OPENSHELL_ENDPOINT=http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080/" - ))); assert!(env.contains(&"OPENSHELL_SANDBOX_ID=sandbox-123".to_string())); assert!(env.contains(&"OPENSHELL_SANDBOX=breezy-rhinoceros".to_string())); - assert!(env.contains(&format!( - "OPENSHELL_SSH_SOCKET_PATH={GUEST_SSH_SOCKET_PATH}" - ))); + assert!( + !env.iter() + .any(|entry| entry.starts_with("OPENSHELL_ENDPOINT=")) + ); + assert!( + !env.iter() + .any(|entry| entry.starts_with("OPENSHELL_SSH_SOCKET_PATH=")) + ); } #[test] @@ -8820,78 +8928,45 @@ mod tests { } #[test] - fn persisted_legacy_sandbox_without_command_uses_scratch_main() { + fn build_guest_environment_keeps_user_values_in_child_channel() { let config = VmDriverConfig { grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() }; - // Requests persisted before the canonical-main contract have a - // present DriverSandboxSpec but no command or tty fields. let sandbox = Sandbox { - id: "legacy-sandbox".to_string(), - name: "legacy-sandbox".to_string(), - spec: Some(SandboxSpec::default()), - ..Default::default() - }; - - let env = build_guest_environment(&sandbox, &config, None); - let encoded = env - .iter() - .find_map(|entry| { - entry.strip_prefix(&format!( - "{}=", - openshell_core::sandbox_env::MAIN_PROCESS_SPEC - )) - }) - .expect("main process environment"); - let main = openshell_core::sandbox_env::MainProcessConfig::decode(encoded) - .expect("legacy persisted request should produce a valid main config"); - - assert_eq!( - main, - openshell_core::sandbox_env::MainProcessConfig::scratch() - ); - } - - #[test] - fn build_guest_environment_preserves_main_command_spaces() { - let config = VmDriverConfig { - grpc_endpoint: "https://127.0.0.1:8080".to_string(), - ..Default::default() - }; - let command = vec![ - "sh".to_string(), - "-lc".to_string(), - "echo ready; while true; do sleep 1; done".to_string(), - ]; - let sandbox = Sandbox { - id: "space-command".to_string(), - name: "space-command".to_string(), + id: "sandbox-123".to_string(), + name: "sandbox-123".to_string(), spec: Some(SandboxSpec { - command: command.clone(), + environment: HashMap::from([ + ("LD_PRELOAD".to_string(), "/workload/evil.so".to_string()), + ("BAD;touch /root/pwned".to_string(), "value".to_string()), + ]), ..Default::default() }), ..Default::default() }; - let env = build_guest_environment(&sandbox, &config, None); - let encoded = env - .iter() - .find_map(|entry| { - entry.strip_prefix(&format!( - "{}=", - openshell_core::sandbox_env::MAIN_PROCESS_SPEC - )) - }) - .expect("main process environment"); + let env = build_guest_environment(&sandbox, &config); - assert!(!encoded.contains(char::is_whitespace)); - let main = openshell_core::sandbox_env::MainProcessConfig::decode(encoded).unwrap(); - assert_eq!(main.command, command); + assert!(!env.iter().any(|entry| entry.starts_with("LD_PRELOAD="))); + assert!(!env.iter().any(|entry| entry.starts_with("BAD;"))); + assert!( + !env.iter() + .any(|entry| { entry.starts_with(openshell_core::sandbox_env::USER_ENVIRONMENT) }) + ); + let child_env = merged_environment(&sandbox); + assert_eq!( + child_env.get("LD_PRELOAD"), + Some(&"/workload/evil.so".to_string()) + ); + assert_eq!( + child_env.get("BAD;touch /root/pwned"), + Some(&"value".to_string()) + ); } #[test] - fn build_guest_environment_uses_token_file_without_raw_token_env() { + fn build_guest_environment_excludes_all_gateway_credentials() { let config = VmDriverConfig { grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() @@ -8910,16 +8985,16 @@ mod tests { ..Default::default() }; - let env = build_guest_environment(&sandbox, &config, None); + let env = build_guest_environment(&sandbox, &config); assert!(!env.iter().any(|v| v.starts_with(&format!( "{}=", openshell_core::sandbox_env::SANDBOX_TOKEN )))); - assert!(env.contains(&format!( - "{}={GUEST_SANDBOX_TOKEN_PATH}", + assert!(!env.iter().any(|v| v.starts_with(&format!( + "{}=", openshell_core::sandbox_env::SANDBOX_TOKEN_FILE - ))); + )))); } #[test] @@ -8941,7 +9016,7 @@ mod tests { ..Default::default() }; - let env = build_guest_environment(&sandbox, &config, None); + let env = build_guest_environment(&sandbox, &config); assert!( !env.iter().any(|v| v.starts_with(&format!( @@ -8978,7 +9053,7 @@ mod tests { ..Default::default() }; - let env = build_guest_environment(&sandbox, &config, None); + let env = build_guest_environment(&sandbox, &config); let telemetry_entries = env .iter() .filter(|entry| { @@ -8998,102 +9073,6 @@ mod tests { ); } - #[test] - fn build_guest_environment_clears_unsupported_network_capabilities() { - let config = VmDriverConfig { - grpc_endpoint: "http://127.0.0.1:8080".to_string(), - ..Default::default() - }; - let sandbox = Sandbox { - id: "sandbox-123".to_string(), - name: "sandbox-123".to_string(), - spec: Some(SandboxSpec { - environment: HashMap::from([( - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES.to_string(), - openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY.to_string(), - )]), - ..Default::default() - }), - ..Default::default() - }; - let env = build_guest_environment(&sandbox, &config, None); - assert!(env.contains(&format!( - "{}=", - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES - ))); - assert!(!env.contains(&format!( - "{}={}", - openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, - openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY - ))); - } - - #[test] - fn build_guest_environment_uses_endpoint_override_for_tap() { - let config = VmDriverConfig { - grpc_endpoint: "http://127.0.0.1:8080".to_string(), - ..Default::default() - }; - let sandbox = Sandbox { - id: "sandbox-123".to_string(), - name: "sandbox-123".to_string(), - spec: Some(SandboxSpec::default()), - ..Default::default() - }; - - let env = build_guest_environment(&sandbox, &config, Some("http://10.0.128.1:8080")); - assert!( - env.contains(&"OPENSHELL_ENDPOINT=http://10.0.128.1:8080".to_string()), - "TAP endpoint override must replace the default" - ); - let endpoint_count = env - .iter() - .filter(|e| e.starts_with("OPENSHELL_ENDPOINT=")) - .count(); - assert_eq!( - endpoint_count, 1, - "must have exactly one OPENSHELL_ENDPOINT" - ); - } - - #[test] - fn guest_visible_openshell_endpoint_rewrites_loopback_hosts_to_gvproxy_host_alias() { - assert_eq!( - guest_visible_openshell_endpoint("http://127.0.0.1:8080"), - format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080/") - ); - assert_eq!( - guest_visible_openshell_endpoint("http://localhost:8080"), - format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080/") - ); - assert_eq!( - guest_visible_openshell_endpoint("https://[::1]:8443"), - format!("https://{GVPROXY_HOST_LOOPBACK_ALIAS}:8443/") - ); - } - - #[test] - fn guest_visible_openshell_endpoint_preserves_non_loopback_hosts() { - assert_eq!( - guest_visible_openshell_endpoint(&format!( - "http://{OPENSHELL_HOST_GATEWAY_ALIAS}:8080" - )), - format!("http://{OPENSHELL_HOST_GATEWAY_ALIAS}:8080") - ); - assert_eq!( - guest_visible_openshell_endpoint(&format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080")), - format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080") - ); - assert_eq!( - guest_visible_openshell_endpoint("http://192.168.127.1:8080"), - "http://192.168.127.1:8080" - ); - assert_eq!( - guest_visible_openshell_endpoint("https://gateway.internal:8443"), - "https://gateway.internal:8443" - ); - } - #[test] fn image_reference_registry_host_defaults_to_docker_hub() { assert_eq!(image_reference_registry_host("ubuntu:24.04"), "docker.io"); @@ -9231,78 +9210,7 @@ mod tests { } #[test] - fn vm_driver_rejects_invalid_rootfs_tar_limits_before_startup() { - let config = VmDriverConfig { - rootfs_tar_max_bytes: Some(0), - ..Default::default() - }; - let error = config.validate_rootfs_tar_config().unwrap_err(); - assert!(error.contains("rootfs_tar_max_bytes")); - - let config = VmDriverConfig { - rootfs_tar_staging_dir: Some(PathBuf::new()), - ..Default::default() - }; - let error = config.validate_rootfs_tar_config().unwrap_err(); - assert!(error.contains("rootfs_tar_staging_dir")); - } - - #[test] - fn vm_proxy_and_spiffe_config_require_explicit_safe_acknowledgements() { - let config = VmDriverConfig { - upstream_proxy: UpstreamProxyConfig { - https_proxy: Some("http://proxy.example:8080".to_string()), - no_proxy: Some(".svc".to_string()), - proxy_auth_file: Some(PathBuf::from("/run/secrets/proxy-auth")), - proxy_auth_allow_insecure: Some(true), - proxy_connect_by_hostname: None, - }, - provider_spiffe_workload_api_tcp_endpoint: Some("tcp:192.0.2.10:8081".to_string()), - provider_spiffe_allow_guest_tcp: false, - ..Default::default() - }; - let error = config.validate_runtime_security_config().unwrap_err(); - assert!(error.contains("provider_spiffe_allow_guest_tcp")); - - let config = VmDriverConfig { - provider_spiffe_workload_api_tcp_endpoint: Some("tcp:192.0.2.10:8081".to_string()), - provider_spiffe_allow_guest_tcp: true, - ..Default::default() - }; - assert!(config.validate_runtime_security_config().is_ok()); - } - - #[test] - fn build_guest_environment_projects_spiffe_endpoint_without_operator_proxy() { - let config = VmDriverConfig { - upstream_proxy: UpstreamProxyConfig { - https_proxy: Some("https://proxy.example:8443".to_string()), - no_proxy: Some(".svc".to_string()), - proxy_auth_file: Some(PathBuf::from("/run/secrets/proxy-auth")), - proxy_auth_allow_insecure: None, - proxy_connect_by_hostname: Some(true), - }, - provider_spiffe_workload_api_tcp_endpoint: Some("tcp:192.0.2.10:8081".to_string()), - provider_spiffe_allow_guest_tcp: true, - ..Default::default() - }; - let sandbox = Sandbox { - id: "vm-spiffe".to_string(), - name: "vm-spiffe".to_string(), - ..Default::default() - }; - let env = build_guest_environment(&sandbox, &config, None); - assert!(env.contains( - &"OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET=tcp:192.0.2.10:8081".to_string() - )); - assert!( - !env.iter().any(|value| value.contains("UPSTREAM_PROXY")), - "operator proxy settings must use protected guest argument staging: {env:?}" - ); - } - - #[test] - fn build_guest_environment_includes_tls_paths_for_https_endpoint() { + fn build_guest_environment_keeps_tls_paths_host_side() { let config = VmDriverConfig { grpc_endpoint: "https://127.0.0.1:8443".to_string(), guest_tls_ca: Some(PathBuf::from("/host/ca.crt")), @@ -9317,10 +9225,8 @@ mod tests { ..Default::default() }; - let env = build_guest_environment(&sandbox, &config, None); - assert!(env.contains(&format!("OPENSHELL_TLS_CA={GUEST_TLS_CA_PATH}"))); - assert!(env.contains(&format!("OPENSHELL_TLS_CERT={GUEST_TLS_CERT_PATH}"))); - assert!(env.contains(&format!("OPENSHELL_TLS_KEY={GUEST_TLS_KEY_PATH}"))); + let env = build_guest_environment(&sandbox, &config); + assert!(!env.iter().any(|entry| entry.starts_with("OPENSHELL_TLS_"))); } #[test] @@ -9350,10 +9256,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; @@ -9385,6 +9287,8 @@ mod tests { record.state_dir = retry_state_dir; record.process = Some(Arc::new(Mutex::new(VmProcess { child: spawn_exited_child(), + supervisor: spawn_exited_child(), + supervisor_liveness: None, deleting: false, }))); } @@ -9414,10 +9318,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; @@ -9437,7 +9337,6 @@ mod tests { process: None, provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }, ); @@ -9470,10 +9369,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; @@ -9494,7 +9389,6 @@ mod tests { process: None, provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }, ); @@ -9581,7 +9475,62 @@ mod tests { } #[test] - fn prepared_image_cache_identity_includes_layout_version_and_owner_contract() { + fn host_control_endpoint_rewrites_guest_host_aliases() { + for host in HOST_LOOPBACK_ALIASES { + assert_eq!( + host_control_openshell_endpoint(&format!("https://{host}:8443/control")) + .expect("guest alias should be rewritten"), + ( + "https://127.0.0.1:8443/control".to_string(), + Some((*host).to_string()), + ), + "host alias {host}" + ); + } + } + + #[test] + fn host_control_endpoint_preserves_remote_gateway() { + assert_eq!( + host_control_openshell_endpoint("https://gateway.internal:8443") + .expect("remote gateway should be preserved"), + ("https://gateway.internal:8443".to_string(), None) + ); + } + + #[test] + fn relative_state_dir_is_resolved_from_the_working_directory() { + let working_dir = std::env::current_dir().expect("working directory"); + assert_eq!( + absolute_state_dir(Path::new("target/driver-state")).expect("resolve state dir"), + working_dir.join("target/driver-state") + ); + + let absolute = working_dir.join("existing-absolute-state"); + assert_eq!( + absolute_state_dir(&absolute).expect("preserve absolute state dir"), + absolute + ); + } + + #[test] + fn host_control_environment_contains_only_explicit_values() { + let mut command = Command::new("openshell-sandbox"); + command.env("UNTRUSTED_PARENT_VALUE", "must-not-leak"); + isolate_host_control_environment(&mut command); + command.env("DRIVER_OWNED_VALUE", "kept"); + + let environment = command.as_std().get_envs().collect::>(); + assert_eq!(environment.len(), 1); + assert_eq!(environment[0].0, "DRIVER_OWNED_VALUE"); + assert_eq!( + environment[0].1.and_then(std::ffi::OsStr::to_str), + Some("kept") + ); + } + + #[test] + fn prepared_image_cache_identity_includes_rootfs_layout_and_openshell_version() { let image = "sha256:local-image"; let image_account = prepared_image_cache_identity(image, &VmDriverConfig::default()); assert_eq!( @@ -9626,14 +9575,14 @@ mod tests { } #[test] - fn bootstrap_image_cache_identity_includes_rootfs_layout_and_openshell_version() { - assert_eq!( - bootstrap_image_cache_identity("sha256:bootstrap-image"), - format!( - "sandbox-bootstrap-rootfs-ext4-v3:openshell-{}:sha256:bootstrap-image", - openshell_core::VERSION - ) - ); + fn bootstrap_image_cache_identity_includes_rootfs_layout_version_and_guest_runtime() { + let identity = bootstrap_image_cache_identity("sha256:bootstrap-image"); + assert!(identity.starts_with(&format!( + "sandbox-bootstrap-rootfs-ext4-v4:openshell-{}:guest-", + openshell_core::VERSION + ))); + assert!(identity.ends_with(":sha256:bootstrap-image")); + assert!(identity.contains(&sandbox_guest_runtime_identity())); } #[test] @@ -9736,96 +9685,6 @@ mod tests { ); } - #[tokio::test] - async fn read_guest_tls_materials_reports_missing_input() { - let base = unique_temp_dir(); - let source_dir = base.join("missing-source"); - - let err = read_guest_tls_materials(&VmDriverTlsPaths { - ca: source_dir.join("ca.crt"), - cert: source_dir.join("tls.crt"), - key: source_dir.join("tls.key"), - }) - .await - .expect_err("missing TLS materials should fail before image injection"); - - assert!(err.contains("ca.crt")); - - let _ = std::fs::remove_dir_all(base); - } - - #[cfg(unix)] - #[test] - fn stage_guest_tls_materials_places_files_in_overlay_upper_with_private_key_mode() { - use std::os::unix::fs::PermissionsExt as _; - - let base = unique_temp_dir(); - let materials = GuestTlsMaterials { - ca: b"ca".to_vec(), - cert: b"cert".to_vec(), - key: b"key".to_vec(), - }; - - stage_guest_tls_materials(&base, &materials).expect("stage TLS materials"); - - assert_eq!( - fs::read( - base.join("upper") - .join(GUEST_TLS_CA_PATH.trim_start_matches('/')) - ) - .unwrap(), - b"ca" - ); - assert_eq!( - fs::read( - base.join("upper") - .join(GUEST_TLS_CERT_PATH.trim_start_matches('/')) - ) - .unwrap(), - b"cert" - ); - let key_path = base - .join("upper") - .join(GUEST_TLS_KEY_PATH.trim_start_matches('/')); - assert_eq!(fs::read(&key_path).unwrap(), b"key"); - assert_eq!( - fs::metadata(&key_path).unwrap().permissions().mode() & 0o777, - 0o600 - ); - - let _ = std::fs::remove_dir_all(base); - } - - #[test] - fn subnet_allocator_assigns_and_releases() { - let mut alloc = SubnetAllocator::new(Ipv4Addr::new(10, 0, 128, 0), 17); - let s1 = alloc.allocate("sandbox-1").unwrap(); - assert_eq!(s1.host_ip, Ipv4Addr::new(10, 0, 128, 1)); - assert_eq!(s1.guest_ip, Ipv4Addr::new(10, 0, 128, 2)); - assert_eq!(s1.prefix_len, 30); - - let s2 = alloc.allocate("sandbox-2").unwrap(); - assert_ne!(s1.host_ip, s2.host_ip); - - alloc.release("sandbox-1"); - let s3 = alloc.allocate("sandbox-3").unwrap(); - assert!(s3.host_ip != s2.host_ip); - } - - #[test] - fn tap_device_name_fits_ifnamsiz() { - let name = tap_device_name("sandbox-abc-def-ghi"); - assert!(name.len() <= 15); - assert!(name.starts_with("vmtap-")); - } - - #[test] - fn mac_address_is_locally_administered() { - let mac = mac_from_sandbox_id("test-sandbox"); - assert_eq!(mac[0] & 0x02, 0x02); - assert_eq!(mac[0] & 0x01, 0x00); - } - #[test] fn vsock_cid_monotonically_increases() { let cid1 = allocate_vsock_cid(); @@ -9870,6 +9729,8 @@ mod tests { }; let process = Arc::new(Mutex::new(VmProcess { child, + supervisor: spawn_exited_child(), + supervisor_liveness: None, deleting: false, })); @@ -9882,7 +9743,6 @@ mod tests { process: Some(process), provisioning_task: None, gpu_bdf: None, - qemu_network_allocated: false, deleting: false, }, ); @@ -9908,10 +9768,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), } } @@ -10236,10 +10092,6 @@ mod tests { image_cache_lock: Arc::new(Mutex::new(())), events, gpu_inventory: None, - subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( - Ipv4Addr::new(10, 0, 128, 0), - 17, - ))), lifecycle_extensions: Arc::new(extensions), } } @@ -10318,11 +10170,7 @@ mod tests { assert_eq!(plan.backend, VmBackend::Libkrun); assert_eq!(plan.vcpus, 2); assert_eq!(plan.mem_mib, 2048); - assert!(plan.tap_device.is_none()); - assert!(plan.guest_ip.is_none()); - assert!(plan.host_ip.is_none()); assert!(plan.vsock_cid.is_none()); - assert!(plan.guest_mac.is_none()); assert!(plan.gpu_bdf.is_none()); assert!(plan.env.is_empty()); } @@ -10345,11 +10193,7 @@ mod tests { assert_eq!(plan.vcpus, 8); assert_eq!(plan.mem_mib, 16384); assert_eq!(plan.gpu_bdf.as_deref(), Some("0000:01:00.0")); - assert!(plan.tap_device.is_some()); - assert!(plan.guest_ip.is_some()); - assert!(plan.host_ip.is_some()); assert!(plan.vsock_cid.is_some()); - assert!(plan.guest_mac.is_some()); } #[test] @@ -10363,12 +10207,7 @@ mod tests { kernel_profile: None, kernel_image: Some(PathBuf::from("/tmp/openshell-test-kernel")), gpu_bdf: None, - tap_device: None, - guest_ip: None, - host_ip: None, vsock_cid: None, - guest_mac: None, - gateway_port: None, guest_init_dropins: Vec::new(), env: Vec::new(), }; @@ -10401,13 +10240,7 @@ mod tests { .expect("backend feature should resolve"); assert_eq!(plan.backend, VmBackend::Qemu); - assert!(plan.tap_device.is_some()); - assert!(plan.guest_ip.is_some()); - assert!(plan.host_ip.is_some()); assert!(plan.vsock_cid.is_some()); - assert!(plan.guest_mac.is_some()); - - driver.release_subnet("sandbox-vfio"); } #[test] @@ -10423,11 +10256,7 @@ mod tests { .expect("backend requirement should resolve"); assert_eq!(plan.backend, VmBackend::Qemu); - assert!(plan.tap_device.is_some()); - assert!(plan.guest_ip.is_some()); - assert!(plan.host_ip.is_some()); - - driver.release_subnet("sandbox-qemu"); + assert!(plan.vsock_cid.is_some()); } #[test] @@ -10443,7 +10272,6 @@ mod tests { .expect("guest init feature should resolve"); assert_eq!(plan.backend, VmBackend::Libkrun); - assert!(plan.tap_device.is_none()); } #[test] @@ -10506,12 +10334,7 @@ mod tests { kernel_profile: None, kernel_image: None, gpu_bdf: None, - tap_device: None, - guest_ip: None, - host_ip: None, vsock_cid: None, - guest_mac: None, - gateway_port: None, guest_init_dropins: Vec::new(), env: Vec::new(), }; @@ -10530,12 +10353,7 @@ mod tests { kernel_profile: None, kernel_image: None, gpu_bdf: None, - tap_device: Some("vmtap-x".to_string()), - guest_ip: Some("10.0.0.2".to_string()), - host_ip: Some("10.0.0.1".to_string()), vsock_cid: Some(7), - guest_mac: Some("02:00:00:00:00:01".to_string()), - gateway_port: Some(8080), guest_init_dropins: Vec::new(), env: Vec::new(), }; @@ -10563,12 +10381,7 @@ mod tests { kernel_profile: None, kernel_image: None, gpu_bdf: None, - tap_device: Some("vmtap-x".to_string()), - guest_ip: Some("10.0.0.2".to_string()), - host_ip: Some("10.0.0.1".to_string()), vsock_cid: Some(7), - guest_mac: Some("02:00:00:00:00:01".to_string()), - gateway_port: Some(8080), guest_init_dropins: Vec::new(), env: Vec::new(), }; @@ -10581,7 +10394,11 @@ mod tests { } /// A driver config carrying only corporate proxy settings. - fn proxy_config(https_proxy: Option<&str>, auth_file: Option<&str>) -> VmDriverConfig { + fn proxy_config( + https_proxy: Option<&str>, + auth_file: Option<&str>, + ca_bundle: Option<&str>, + ) -> VmDriverConfig { VmDriverConfig { grpc_endpoint: "http://127.0.0.1:8080".to_string(), upstream_proxy: UpstreamProxyConfig { @@ -10590,6 +10407,7 @@ mod tests { proxy_auth_allow_insecure: auth_file.map(|_| true), ..UpstreamProxyConfig::default() }, + proxy_ca_bundle: ca_bundle.map(PathBuf::from), ..Default::default() } } @@ -10603,6 +10421,7 @@ mod tests { proxy_config( Some("http://user:secret@proxy.corp.test:3128"), Some("/etc/openshell/secrets/proxy-auth"), + None, ) ); assert!( @@ -10620,72 +10439,43 @@ mod tests { ); } - #[test] - fn proxy_material_is_staged_inside_the_per_sandbox_overlay() { - // Everything the driver stages lands in the overlay upperdir, which - // lives in the sandbox's own state directory. That is what makes the - // credential removable with the sandbox (remove_sandbox_state_dir - // deletes the whole directory) and unforgeable by the guest image - // (the upperdir shadows the read-only image layer). - for guest_path in [ - GUEST_UPSTREAM_PROXY_AUTH_PATH, - GUEST_PROXY_CA_PATH, - GUEST_SUPERVISOR_ARGS_PATH, - ] { - assert!( - guest_path.starts_with("/opt/openshell/"), - "{guest_path} must be under the reserved guest control root" - ); - assert_eq!( - overlay_upper_path(guest_path), - format!("/upper{guest_path}"), - "{guest_path} must be staged into the overlay upperdir" - ); - } - } - #[test] fn upstream_proxy_args_are_empty_without_a_configured_proxy() { - assert!(upstream_proxy_cli_args(&VmDriverConfig::default()).is_empty()); - // The file is still written, empty, so the guest cannot fall back to - // an image-baked argument list. - assert!(render_guest_supervisor_args(&[]).is_empty()); + assert!( + upstream_proxy_cli_args(&VmDriverConfig::default()) + .unwrap() + .is_empty() + ); } #[test] - fn upstream_proxy_args_pass_guest_paths_not_host_paths() { - let mut config = proxy_config( + fn upstream_proxy_args_pass_host_paths_to_host_control() { + let config = proxy_config( Some("http://proxy.corp.test:3128"), Some("/etc/openshell/secrets/proxy-auth"), + Some("/etc/openshell/tls/corp-ca.pem"), ); - config.proxy_ca_bundle = Some(PathBuf::from("/etc/openshell/tls/corp-ca.pem")); - let args = upstream_proxy_cli_args(&config); + let args = upstream_proxy_cli_args(&config).unwrap(); - // The credential and CA live at fixed guest paths; the gateway-host - // paths the operator configured must never reach the guest argv. + // Control runs on the gateway host and receives the operator-owned + // paths directly; neither path is copied into the guest. let auth = args .iter() .position(|arg| arg == "--upstream-proxy-auth-file") .map(|i| args[i + 1].as_str()); - assert_eq!(auth, Some(GUEST_UPSTREAM_PROXY_AUTH_PATH)); + assert_eq!(auth, Some("/etc/openshell/secrets/proxy-auth")); let ca = args .iter() .position(|arg| arg == "--upstream-proxy-ca-bundle") .map(|i| args[i + 1].as_str()); - assert_eq!(ca, Some(GUEST_PROXY_CA_PATH)); - assert!( - !args - .iter() - .any(|arg| arg.contains("/etc/openshell/secrets") || arg.contains("corp-ca.pem")), - "host paths leaked into the guest argv: {args:?}" - ); + assert_eq!(ca, Some("/etc/openshell/tls/corp-ca.pem")); } #[test] fn upstream_proxy_args_pass_only_explicit_opt_ins() { - let mut config = proxy_config(Some("https://proxy.corp.test:3130"), None); + let mut config = proxy_config(Some("https://proxy.corp.test:3130"), None, None); config.upstream_proxy.no_proxy = Some("10.0.0.0/8,.svc.cluster.local".to_string()); - let args = upstream_proxy_cli_args(&config); + let args = upstream_proxy_cli_args(&config).unwrap(); assert_eq!( args, vec![ @@ -10701,44 +10491,35 @@ mod tests { config.upstream_proxy.proxy_connect_by_hostname = Some(false); assert!( !upstream_proxy_cli_args(&config) + .unwrap() .iter() .any(|arg| arg == "--upstream-proxy-connect-by-hostname") ); config.upstream_proxy.proxy_connect_by_hostname = Some(true); assert!( upstream_proxy_cli_args(&config) + .unwrap() .iter() .any(|arg| arg == "--upstream-proxy-connect-by-hostname") ); } #[test] - fn guest_supervisor_args_render_one_argument_per_line() { - let args = vec![ - "--upstream-proxy".to_string(), - "http://proxy.corp.test:3128".to_string(), - "--upstream-no-proxy".to_string(), - "a.example, b.example".to_string(), - ]; - // A value containing a space stays one line, so the guest reads it - // back as a single argument rather than word-splitting it. - assert_eq!( - String::from_utf8(render_guest_supervisor_args(&args)).unwrap(), - "--upstream-proxy\nhttp://proxy.corp.test:3128\n--upstream-no-proxy\na.example, b.example\n" - ); - } - - #[test] - fn guest_supervisor_args_reject_line_breaking_values() { - // A newline would split one operator value into two guest arguments. - for bad in ["a\nb", "a\rb", "a\0b"] { - assert!( - validate_guest_supervisor_args(&[bad.to_string()]).is_err(), - "{bad:?} must be rejected" + fn upstream_proxy_args_route_vm_host_aliases_to_host_loopback() { + for alias in HOST_LOOPBACK_ALIASES { + let config = proxy_config(Some(&format!("http://{alias}:3128")), None, None); + let args = upstream_proxy_cli_args(&config).unwrap(); + assert_eq!( + args, + vec![ + "--upstream-proxy".to_string(), + format!("http://{alias}:3128"), + "--upstream-proxy-dial-ip".to_string(), + "127.0.0.1".to_string(), + ], + "host alias {alias} must dial host loopback without changing its TLS identity" ); } - validate_guest_supervisor_args(&["--upstream-proxy".to_string()]) - .expect("ordinary arguments are accepted"); } #[test] @@ -10755,7 +10536,7 @@ mod tests { .expect_err("a bypass list without a proxy would hide a fail-open state"); assert!(err.contains("no_proxy"), "{err}"); - let config = proxy_config(Some("http://proxy.corp.test:3128"), None); + let config = proxy_config(Some("http://proxy.corp.test:3128"), None, None); config .validate_runtime_security_config() .expect("a lone proxy URL is a complete configuration"); @@ -10775,6 +10556,7 @@ mod tests { let mut config = proxy_config( Some("http://proxy.corp.test:3128"), Some("/etc/openshell/secrets/proxy-auth"), + None, ); config.upstream_proxy.proxy_auth_allow_insecure = None; let err = config @@ -10784,128 +10566,15 @@ mod tests { } #[test] - fn qemu_backend_rejects_a_gateway_host_proxy() { - // gvproxy's host-loopback NAT has no QEMU/TAP equivalent, so a proxy - // on the gateway host is unreachable from a GPU sandbox and must be - // rejected rather than time out on every CONNECT. The address that - // reaches the gateway host from a QEMU guest is this sandbox's own - // TAP host address, so the classifier is parameterized by it. - let tap_host = Some("10.0.128.1"); - for url in [ - "http://host.openshell.internal:8080", - "http://host.containers.internal:8080", - "http://host.docker.internal:8080", - "http://127.0.0.1:8080", - "http://localhost:8080", - "https://[::1]:8080", - // The address the aliases above resolve to inside the guest. - "http://10.0.128.1:8080", - ] { - assert!(proxy_url_targets_gateway_host(url, tap_host), "{url}"); - } - for url in [ - "http://proxy.corp.example:8080", - "https://10.1.2.3:3128", - // Special only to libkrun/gvproxy. On QEMU/TAP it is an ordinary - // address that may be routable through the guest's masqueraded - // egress, so rejecting it would refuse a working configuration. - "http://192.168.127.254:8080", - // Another sandbox's TAP host, not this one's. - "http://10.0.128.5:8080", - "not a url", - ] { - assert!(!proxy_url_targets_gateway_host(url, tap_host), "{url}"); - } - - // Without an allocated TAP host only the address-independent - // spellings classify; the loopback and alias guards still hold. - assert!(proxy_url_targets_gateway_host( - "http://127.0.0.1:8080", - None - )); - assert!(proxy_url_targets_gateway_host( - "http://host.openshell.internal:8080", - None - )); - assert!(!proxy_url_targets_gateway_host( - "http://10.0.128.1:8080", - None - )); - } - - #[test] - fn qemu_launch_plan_rejects_a_proxy_at_the_allocated_tap_host() { - // The preflight has to run against the address this sandbox actually - // got, which only exists once the launch plan's subnet is allocated. - // A proxy there is what `host.openshell.internal` resolves to in the - // guest, and the driver's own nftables input chain drops the port. - let probe = test_driver_with_extensions(LifecycleExtensionRegistry::new()); - let tap_host = probe - .build_vm_launch_plan("sandbox-proxy-tap", true, true, None) - .expect("gpu plan should build") - .host_ip - .expect("a QEMU plan carries a TAP host address"); - probe.release_subnet("sandbox-proxy-tap"); - - let driver = test_driver_with_proxy(&format!("http://{tap_host}:8080")); - let mut plan = driver - .build_vm_launch_plan("sandbox-proxy-tap", true, true, None) - .expect("gpu plan should build"); - assert_eq!(plan.host_ip.as_deref(), Some(tap_host.as_str())); - - let err = driver - .resolve_launch_plan_backend("sandbox-proxy-tap", true, None, &mut plan) - .expect_err("a proxy at the TAP host address is unreachable from the guest"); - assert_eq!(err.code(), Code::FailedPrecondition); - assert!(err.message().contains(&tap_host), "{err}"); - - driver.release_subnet("sandbox-proxy-tap"); - } - - #[test] - fn qemu_launch_plan_allows_a_proxy_at_the_gvproxy_host_loopback_address() { - // 192.168.127.254 carries no meaning on QEMU/TAP, so a launch must - // proceed rather than be refused for a libkrun-only reason. - let driver = test_driver_with_proxy(&format!("http://{GVPROXY_HOST_LOOPBACK_IP}:8080")); + fn qemu_launch_plan_uses_vsock_only_with_host_proxy() { + let driver = test_driver_with_proxy("http://127.0.0.1:8080"); let mut plan = driver - .build_vm_launch_plan("sandbox-proxy-gvproxy", true, true, None) + .build_vm_launch_plan("sandbox-proxy-vsock", true, true, None) .expect("gpu plan should build"); - assert_ne!(plan.host_ip.as_deref(), Some(GVPROXY_HOST_LOOPBACK_IP)); - driver - .resolve_launch_plan_backend("sandbox-proxy-gvproxy", true, None, &mut plan) - .expect("a routable proxy address must not block a GPU launch"); - assert_eq!(plan.backend, VmBackend::Qemu); - - driver.release_subnet("sandbox-proxy-gvproxy"); - } - - #[tokio::test] - async fn proxy_credential_is_validated_against_the_supervisor_rules() { - let dir = std::env::temp_dir().join(format!("openshell-vm-cred-{}", std::process::id())); - std::fs::create_dir_all(&dir).unwrap(); - let path = dir.join("proxy-auth"); - - std::fs::write(&path, "proxyuser:proxypass\n").unwrap(); - assert_eq!( - read_sandbox_proxy_credential(&path) - .await - .expect("a well-formed credential is accepted"), - "proxyuser:proxypass" - ); - - // Rejected here rather than inside every sandbox's supervisor. - std::fs::write(&path, "no-separator\n").unwrap(); - let err = read_sandbox_proxy_credential(&path) - .await - .expect_err("a malformed credential must fail closed"); - assert_eq!(err.code(), Code::InvalidArgument); - assert!( - !err.message().contains("no-separator"), - "the error must not echo credential file contents: {err}" - ); - - std::fs::remove_dir_all(&dir).unwrap(); + .resolve_launch_plan_backend("sandbox-proxy-vsock", true, None, &mut plan) + .expect("host control can reach a host-loopback proxy"); + assert!(plan.vsock_cid.is_some()); } #[test] @@ -10916,6 +10585,7 @@ mod tests { let config = proxy_config( Some("http://proxy.corp.test:3128"), Some("/etc/openshell/secrets/proxy-auth"), + None, ); let sandbox = Sandbox { id: "sb-proxy".to_string(), @@ -10935,7 +10605,7 @@ mod tests { ..Default::default() }; - let env = build_guest_environment(&sandbox, &config, None); + let env = build_guest_environment(&sandbox, &config); assert!( !env.iter().any(|entry| entry.starts_with("--upstream")), "driver environment must never carry supervisor arguments: {env:?}" @@ -10952,7 +10622,7 @@ mod tests { #[test] fn sandbox_driver_config_cannot_carry_proxy_settings() { - // The upstream proxy is host network topology, not a per-sandbox + // The upstream proxy is host network infrastructure, not a per-sandbox // setting: the caller-supplied envelope must reject it outright // rather than silently ignoring it. for key in [ diff --git a/crates/openshell-driver-vm/src/embedded_runtime.rs b/crates/openshell-driver-vm/src/embedded_runtime.rs index 70626edd98..4b75bd4ba8 100644 --- a/crates/openshell-driver-vm/src/embedded_runtime.rs +++ b/crates/openshell-driver-vm/src/embedded_runtime.rs @@ -10,7 +10,6 @@ use std::path::{Path, PathBuf}; mod resources { pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.dylib.zst")); pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.5.dylib.zst")); - pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst")); pub const LIBKRUN_NAME: &str = "libkrun.dylib"; pub const LIBKRUNFW_NAME: &str = "libkrunfw.5.dylib"; } @@ -19,7 +18,6 @@ mod resources { mod resources { pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.so.zst")); pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.so.5.zst")); - pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst")); pub const LIBKRUN_NAME: &str = "libkrun.so"; pub const LIBKRUNFW_NAME: &str = "libkrunfw.so.5"; } @@ -28,7 +26,6 @@ mod resources { mod resources { pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.so.zst")); pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.so.5.zst")); - pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst")); pub const LIBKRUN_NAME: &str = "libkrun.so"; pub const LIBKRUNFW_NAME: &str = "libkrunfw.so.5"; } @@ -41,7 +38,6 @@ mod resources { mod resources { pub const LIBKRUN: &[u8] = &[]; pub const LIBKRUNFW: &[u8] = &[]; - pub const GVPROXY: &[u8] = &[]; pub const LIBKRUN_NAME: &str = "libkrun"; pub const LIBKRUNFW_NAME: &str = "libkrunfw"; } @@ -82,7 +78,6 @@ pub fn ensure_runtime_extracted() -> Result { resources::LIBKRUNFW, &cache_dir.join(resources::LIBKRUNFW_NAME), )?; - extract_resource(resources::GVPROXY, &cache_dir.join("gvproxy"))?; #[cfg(target_os = "macos")] { @@ -96,22 +91,13 @@ pub fn ensure_runtime_extracted() -> Result { fs::write(&version_marker, cache_key) .map_err(|e| format!("write runtime marker {}: {e}", version_marker.display()))?; - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt as _; - fs::set_permissions(cache_dir.join("gvproxy"), fs::Permissions::from_mode(0o755)) - .map_err(|e| format!("chmod gvproxy: {e}"))?; - } - Ok(cache_dir) } pub fn validate_runtime_dir(dir: &Path) -> Result<(), String> { let libkrun = dir.join(resources::LIBKRUN_NAME); let libkrunfw = dir.join(resources::LIBKRUNFW_NAME); - let gvproxy = dir.join("gvproxy"); - - for path in [&libkrun, &libkrunfw, &gvproxy] { + for path in [&libkrun, &libkrunfw] { if !path.is_file() { return Err(format!("missing runtime file: {}", path.display())); } @@ -128,7 +114,6 @@ fn runtime_cache_key() -> String { let mut fp: u64 = 0; for (index, chunk) in [resources::LIBKRUN, resources::LIBKRUNFW] .into_iter() - .chain(std::iter::once(resources::GVPROXY)) .enumerate() { let sample = &chunk[..chunk.len().min(64)]; diff --git a/crates/openshell-driver-vm/src/ffi.rs b/crates/openshell-driver-vm/src/ffi.rs index 423ad6f05b..f84ea35743 100644 --- a/crates/openshell-driver-vm/src/ffi.rs +++ b/crates/openshell-driver-vm/src/ffi.rs @@ -52,23 +52,8 @@ type KrunSetConsoleOutput = unsafe extern "C" fn(ctx_id: u32, filepath: *const c type KrunStartEnter = unsafe extern "C" fn(ctx_id: u32) -> i32; type KrunDisableImplicitVsock = unsafe extern "C" fn(ctx_id: u32) -> i32; type KrunAddVsock = unsafe extern "C" fn(ctx_id: u32, tsi_features: u32) -> i32; -#[cfg(target_os = "macos")] -type KrunAddNetUnixgram = unsafe extern "C" fn( - ctx_id: u32, - c_path: *const c_char, - fd: i32, - c_mac: *const u8, - features: u32, - flags: u32, -) -> i32; -type KrunAddNetUnixstream = unsafe extern "C" fn( - ctx_id: u32, - c_path: *const c_char, - fd: i32, - c_mac: *const u8, - features: u32, - flags: u32, -) -> i32; +type KrunAddVsockPort2 = + unsafe extern "C" fn(ctx_id: u32, port: u32, filepath: *const c_char, listen: bool) -> i32; // Field names mirror the libkrun C API symbol names (`krun_*`); preserving // the prefix keeps the FFI binding 1:1 with the upstream library. @@ -86,10 +71,7 @@ pub struct LibKrun { pub krun_start_enter: KrunStartEnter, pub krun_disable_implicit_vsock: KrunDisableImplicitVsock, pub krun_add_vsock: KrunAddVsock, - #[cfg(target_os = "macos")] - pub krun_add_net_unixgram: KrunAddNetUnixgram, - #[allow(dead_code)] // Used on Linux when gvproxy runs in qemu/unixstream mode. - pub krun_add_net_unixstream: KrunAddNetUnixstream, + pub krun_add_vsock_port2: KrunAddVsockPort2, } static LIBKRUN: OnceLock = OnceLock::new(); @@ -151,13 +133,7 @@ impl LibKrun { &libkrun_path, )?, krun_add_vsock: load_symbol(library, b"krun_add_vsock\0", &libkrun_path)?, - #[cfg(target_os = "macos")] - krun_add_net_unixgram: load_symbol(library, b"krun_add_net_unixgram\0", &libkrun_path)?, - krun_add_net_unixstream: load_symbol( - library, - b"krun_add_net_unixstream\0", - &libkrun_path, - )?, + krun_add_vsock_port2: load_symbol(library, b"krun_add_vsock_port2\0", &libkrun_path)?, }) } } diff --git a/crates/openshell-driver-vm/src/gpu.rs b/crates/openshell-driver-vm/src/gpu.rs index dc5883b5ba..e7b0233fdf 100644 --- a/crates/openshell-driver-vm/src/gpu.rs +++ b/crates/openshell-driver-vm/src/gpu.rs @@ -5,8 +5,6 @@ use openshell_vfio::{ GpuBindGuard, GpuBindState, GpuBinding, GpuInfo, SysfsRoot, prepare_gpu_for_passthrough, probe_host_nvidia_vfio_readiness, reconcile_stale_bindings, validate_bdf, }; -use std::collections::HashMap; -use std::net::Ipv4Addr; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicU32, Ordering}; @@ -156,164 +154,16 @@ pub struct GpuAssignment { pub iommu_group: u32, } -// --------------------------------------------------------------------------- -// Subnet allocation for per-sandbox TAP networking -// --------------------------------------------------------------------------- - -/// Allocates /30 subnets from a pool for per-sandbox TAP networking. -pub struct SubnetAllocator { - base: Ipv4Addr, - prefix_len: u8, - next_offset: u32, - allocated: HashMap, -} - -pub struct SubnetAllocation { - pub host_ip: Ipv4Addr, - pub guest_ip: Ipv4Addr, - pub prefix_len: u8, - pub offset: u32, -} - static NEXT_VSOCK_CID: AtomicU32 = AtomicU32::new(3); -impl SubnetAllocator { - pub fn new(base: Ipv4Addr, prefix_len: u8) -> Self { - Self { - base, - prefix_len, - next_offset: 0, - allocated: HashMap::new(), - } - } - - pub fn allocate(&mut self, sandbox_id: &str) -> Result { - let pool_size = 1u32 << (32 - self.prefix_len); - let max_subnets = pool_size / 4; - - if u32::try_from(self.allocated.len()).unwrap_or(u32::MAX) >= max_subnets { - return Err("subnet pool exhausted".to_string()); - } - - while self - .allocated - .values() - .any(|a| a.offset == self.next_offset) - { - self.next_offset = (self.next_offset + 1) % max_subnets; - } - - let base_u32 = u32::from(self.base); - let subnet_base = base_u32 + (self.next_offset * 4); - let host_ip = Ipv4Addr::from(subnet_base + 1); - let guest_ip = Ipv4Addr::from(subnet_base + 2); - - let allocation = SubnetAllocation { - host_ip, - guest_ip, - prefix_len: 30, - offset: self.next_offset, - }; - - self.allocated.insert(sandbox_id.to_string(), allocation); - self.next_offset = (self.next_offset + 1) % max_subnets; - - let alloc = &self.allocated[sandbox_id]; - Ok(SubnetAllocation { - host_ip: alloc.host_ip, - guest_ip: alloc.guest_ip, - prefix_len: alloc.prefix_len, - offset: alloc.offset, - }) - } - - pub fn release(&mut self, sandbox_id: &str) { - self.allocated.remove(sandbox_id); - } -} - pub fn allocate_vsock_cid() -> u32 { NEXT_VSOCK_CID.fetch_add(1, Ordering::Relaxed) } -/// Generate a locally-administered MAC from sandbox ID using FNV-1a. -pub fn mac_from_sandbox_id(sandbox_id: &str) -> [u8; 6] { - let mut hash: u64 = 0xcbf2_9ce4_8422_2325; - for byte in sandbox_id.as_bytes() { - hash ^= u64::from(*byte); - hash = hash.wrapping_mul(0x0000_0100_0000_01b3); - } - let bytes = hash.to_le_bytes(); - let mut mac = [bytes[0], bytes[1], bytes[2], bytes[3], bytes[4], bytes[5]]; - mac[0] = (mac[0] & 0xFE) | 0x02; - mac -} - -/// TAP device name from sandbox ID (fits `IFNAMSIZ=16`). -pub fn tap_device_name(sandbox_id: &str) -> String { - let mut end = sandbox_id.len().min(8); - // Walk back to a UTF-8 char boundary (str::floor_char_boundary requires - // Rust 1.91 — we still build on older toolchains). - while end > 0 && !sandbox_id.is_char_boundary(end) { - end -= 1; - } - let prefix = &sandbox_id[..end]; - format!("vmtap-{prefix}") -} - #[cfg(test)] mod tests { use super::*; - #[test] - fn subnet_allocator_assigns_sequential_blocks() { - let mut alloc = SubnetAllocator::new(Ipv4Addr::new(10, 0, 128, 0), 17); - - let s1 = alloc.allocate("sandbox-1").unwrap(); - assert_eq!(s1.host_ip, Ipv4Addr::new(10, 0, 128, 1)); - assert_eq!(s1.guest_ip, Ipv4Addr::new(10, 0, 128, 2)); - assert_eq!(s1.prefix_len, 30); - - let s2 = alloc.allocate("sandbox-2").unwrap(); - assert_eq!(s2.host_ip, Ipv4Addr::new(10, 0, 128, 5)); - assert_eq!(s2.guest_ip, Ipv4Addr::new(10, 0, 128, 6)); - } - - #[test] - fn subnet_allocator_recycles_after_release() { - let mut alloc = SubnetAllocator::new(Ipv4Addr::new(10, 0, 128, 0), 17); - - let _s1 = alloc.allocate("sandbox-1").unwrap(); - let _s2 = alloc.allocate("sandbox-2").unwrap(); - alloc.release("sandbox-1"); - - let s3 = alloc.allocate("sandbox-3").unwrap(); - assert_eq!(s3.host_ip, Ipv4Addr::new(10, 0, 128, 9)); - } - - #[test] - fn tap_device_name_truncates_long_ids() { - assert_eq!(tap_device_name("abc"), "vmtap-abc"); - assert_eq!(tap_device_name("abcdefghijklmnop"), "vmtap-abcdefgh"); - } - - #[test] - fn mac_from_sandbox_id_sets_locally_administered_bit() { - let mac = mac_from_sandbox_id("sandbox-123"); - assert_eq!(mac[0] & 0x02, 0x02, "locally-administered bit must be set"); - assert_eq!(mac[0] & 0x01, 0x00, "multicast bit must be clear"); - } - - #[test] - fn mac_from_sandbox_id_deterministic() { - let mac1 = mac_from_sandbox_id("sandbox-x"); - let mac2 = mac_from_sandbox_id("sandbox-x"); - assert_eq!(mac1, mac2); - - let mac3 = mac_from_sandbox_id("sandbox-y"); - assert_ne!(mac1, mac3); - } - #[test] fn vsock_cid_increments() { let cid1 = allocate_vsock_cid(); diff --git a/crates/openshell-driver-vm/src/isolation/mod.rs b/crates/openshell-driver-vm/src/isolation/mod.rs new file mode 100644 index 0000000000..49c1135310 --- /dev/null +++ b/crates/openshell-driver-vm/src/isolation/mod.rs @@ -0,0 +1,169 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! VM provisioning for the shared authenticated boundary protocol. +//! +//! This module deliberately contains no lifecycle, process, network, or wire +//! implementation. The driver chooses the host transport and binds immutable +//! VM claims; `openshell-isolation-interface` and `openshell-sandbox` provide +//! the common control and boundary behavior. + +use openshell_isolation_interface::contract::{ + BackendError, DriverFenceEvidence, ResolvedWorkloadIdentity, +}; +use openshell_sandbox_backend::boundary_protocol::{ + BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor, + SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport, +}; +use std::collections::{BTreeMap, HashMap}; + +/// Driver-owned inputs that bind one VM generation to one supervisor boundary. +pub struct VmBoundarySpec { + pub boundary_id: String, + pub generation: String, + pub session_id: openshell_core::SandboxSessionId, + pub session_rotation: openshell_core::jwt::SessionRotation, + pub auth_epoch: openshell_core::jwt::CredentialEpoch, + pub gateway_id: String, + pub verification_keys: Vec, + pub image_identity: String, + pub transport: SandboxTransport, + pub supervisor_tls: SandboxTlsClientConfig, + pub sandbox_tls: SandboxTlsServerConfig, + pub control_port: u32, + pub agent_uid: u32, + pub agent_gid: u32, + pub child_env: HashMap, +} + +/// The protected guest config and matching host descriptor for one VM. +pub struct VmBoundaryProvisioning { + pub boundary_config: BoundaryConfig, + pub runtime_descriptor: SandboxRuntimeDescriptor, +} + +impl VmBoundarySpec { + /// Produce both sides of the common protocol from one set of immutable + /// driver inputs so their identity claims cannot drift. + pub fn provision(self) -> Result { + let workload_identity = ResolvedWorkloadIdentity::new( + self.agent_uid, + self.agent_gid, + Vec::new(), + "vm-config".to_string(), + self.image_identity.clone(), + )?; + let resource_claims = BTreeMap::from([ + ("vm.generation".to_string(), self.generation.clone()), + ("vm.image_identity".to_string(), self.image_identity), + ]); + let driver_fence = DriverFenceEvidence::Vm { + generation: self.generation.clone(), + network_device_count: 0, + }; + Ok(VmBoundaryProvisioning { + boundary_config: BoundaryConfig { + boundary_id: self.boundary_id.clone(), + generation: self.generation.clone(), + session_id: self.session_id, + session_rotation: self.session_rotation, + auth_epoch: self.auth_epoch, + gateway_id: self.gateway_id, + verification_keys: self.verification_keys, + listener: BoundaryListener::Vsock { + control_port: self.control_port, + tls: self.sandbox_tls, + }, + resource_claims: resource_claims.clone(), + resource_claim_files: BTreeMap::new(), + workload_identity: workload_identity.clone(), + driver_fence: driver_fence.clone(), + child_env: self.child_env, + }, + runtime_descriptor: SandboxRuntimeDescriptor { + boundary_id: self.boundary_id, + generation: self.generation, + session_id: self.session_id, + workload_identity, + transport: self.transport, + tls: self.supervisor_tls, + // The host-side control process is the network broker, so + // reserved host aliases terminate at its loopback address + // after crossing the authenticated boundary channel. + host_gateway_ip: Some(std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST)), + resource_claims, + driver_fence, + }, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use openshell_sandbox_backend::boundary_protocol::{ + SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport, + generate_sandbox_tls_material, + }; + + #[test] + fn provisioning_binds_identical_resource_claims() { + let session_id = openshell_core::SandboxSessionId::new(); + let material = generate_sandbox_tls_material(session_id).unwrap(); + let provisioned = VmBoundarySpec { + boundary_id: "sandbox-1".to_string(), + generation: "generation-1".to_string(), + session_id, + session_rotation: openshell_core::jwt::SessionRotation::new(1).unwrap(), + auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).unwrap(), + gateway_id: "gateway-1".to_string(), + verification_keys: vec![GatewayVerificationKey { + key_id: "key-1".to_string(), + public_key_pem: "public-key".to_string(), + }], + image_identity: "sha256:image".to_string(), + transport: SandboxTransport::Vsock { + guest_cid: 42, + port: 5500, + }, + supervisor_tls: SandboxTlsClientConfig { + server_name: material.server_name, + trust_anchor_pem: material.trust_anchor_pem, + }, + sandbox_tls: SandboxTlsServerConfig { + certificate_chain_path: "/.openshell/state/sandbox.crt".into(), + private_key_path: "/.openshell/state/sandbox.key".into(), + }, + control_port: 5500, + agent_uid: 1000, + agent_gid: 1000, + child_env: HashMap::new(), + } + .provision() + .unwrap(); + + assert_eq!( + provisioned.boundary_config.resource_claims, + provisioned.runtime_descriptor.resource_claims + ); + assert_eq!( + provisioned.runtime_descriptor.resource_claims["vm.generation"], + "generation-1" + ); + assert_eq!( + provisioned.runtime_descriptor.host_gateway_ip, + Some(std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST)) + ); + assert_eq!( + provisioned.boundary_config.driver_fence, + provisioned.runtime_descriptor.driver_fence + ); + assert!( + provisioned + .runtime_descriptor + .driver_fence + .validate() + .is_ok() + ); + } +} diff --git a/crates/openshell-driver-vm/src/lib.rs b/crates/openshell-driver-vm/src/lib.rs index f34c7dda8d..8e3f41a15d 100644 --- a/crates/openshell-driver-vm/src/lib.rs +++ b/crates/openshell-driver-vm/src/lib.rs @@ -11,24 +11,36 @@ compile_error!( build a telemetry-free VM driver with `--no-default-features --features defaults-without-telemetry`" ); +#[cfg(feature = "compute-driver")] pub mod driver; +#[cfg(feature = "compute-driver")] mod embedded_runtime; +#[cfg(feature = "compute-driver")] mod ffi; +#[cfg(feature = "compute-driver")] pub mod gpu; +#[cfg(feature = "compute-driver")] +mod isolation; +#[cfg(feature = "compute-driver")] pub mod lifecycle; -mod nft_ruleset; +#[cfg(feature = "compute-driver")] pub mod otel_tracing; +#[cfg(feature = "compute-driver")] pub mod procguard; +#[cfg(feature = "compute-driver")] mod rootfs; +#[cfg(feature = "compute-driver")] mod runtime; +#[cfg(feature = "compute-driver")] pub use driver::{VmDriver, VmDriverConfig}; +#[cfg(feature = "compute-driver")] pub use lifecycle::{ BackendFeature, ExtensionCapabilities, ExtensionDescriptor, GuestInitDropin, LaunchAbortReason, LaunchPlan, LifecycleError, LifecycleExtension, LifecycleExtensionRegistry, LifecycleResult, RestoreContext, }; +#[cfg(feature = "compute-driver")] pub use runtime::{ - VM_RUNTIME_DIR_ENV, VmBackend, VmLaunchConfig, cleanup_stale_tap_interfaces, - configured_runtime_dir, run_vm, + VM_RUNTIME_DIR_ENV, VmBackend, VmLaunchConfig, VsockPortMap, configured_runtime_dir, run_vm, }; diff --git a/crates/openshell-driver-vm/src/lifecycle.rs b/crates/openshell-driver-vm/src/lifecycle.rs index 25ec91db67..51977ccef2 100644 --- a/crates/openshell-driver-vm/src/lifecycle.rs +++ b/crates/openshell-driver-vm/src/lifecycle.rs @@ -108,9 +108,6 @@ pub enum BackendFeature { /// QEMU-only and currently rejected for non-GPU sandboxes pending the /// non-GPU QEMU launch path landing. PciPassthrough, - /// Extension needs a host TAP device wired into the guest. Currently - /// QEMU-only (libkrun does not expose a TAP transport). - TapNetworking, } impl BackendFeature { @@ -120,7 +117,6 @@ impl BackendFeature { Self::ExternalKernelImage => "external-kernel-image", Self::GuestInitDropins => "guest-init-dropins", Self::PciPassthrough => "pci-passthrough", - Self::TapNetworking => "tap-networking", } } @@ -130,10 +126,7 @@ impl BackendFeature { /// exists. #[must_use] pub fn requires_qemu(self) -> bool { - matches!( - self, - Self::ExternalKernelImage | Self::PciPassthrough | Self::TapNetworking - ) + matches!(self, Self::ExternalKernelImage | Self::PciPassthrough) } } @@ -226,12 +219,7 @@ pub struct LaunchPlan { pub kernel_profile: Option, pub kernel_image: Option, pub gpu_bdf: Option, - pub tap_device: Option, - pub guest_ip: Option, - pub host_ip: Option, pub vsock_cid: Option, - pub guest_mac: Option, - pub gateway_port: Option, pub guest_init_dropins: Vec, pub env: Vec, } @@ -296,7 +284,7 @@ pub enum ExtensionActivation { /// (kernel profile, guest init drop-ins, etc.). Called before the driver /// has resolved the final backend. /// 2. Driver resolves [`LaunchPlan::backend`] from declared requirements -/// and allocates backend-specific host resources (subnet, tap, vsock). +/// and allocates backend-specific host resources such as a vsock CID. /// 3. [`before_launch`](Self::before_launch) — perform host-side /// side effects with the resolved plan in hand, optionally append /// additional guest env via [`LaunchPlan::env`]. @@ -353,7 +341,7 @@ pub trait LifecycleExtension: std::fmt::Debug + Send + Sync { /// At this point [`LaunchPlan::backend`], /// [`LaunchPlan::required_backends`], and /// [`LaunchPlan::required_backend_features`] are finalized and any - /// backend-specific host resources (subnet, tap, vsock) have been + /// backend-specific host resources have been /// allocated. This hook is the right place to bind PCI devices, set /// up filesystem state, or otherwise prepare the host. /// @@ -953,12 +941,7 @@ mod tests { kernel_profile: None, kernel_image: None, gpu_bdf: None, - tap_device: None, - guest_ip: None, - host_ip: None, vsock_cid: None, - guest_mac: None, - gateway_port: None, guest_init_dropins: Vec::new(), env: Vec::new(), } diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index 8f1cc649d7..65f37eae56 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -8,7 +8,9 @@ use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; #[cfg(target_os = "macos")] use openshell_driver_vm::{VM_RUNTIME_DIR_ENV, configured_runtime_dir}; -use openshell_driver_vm::{VmBackend, VmDriver, VmDriverConfig, VmLaunchConfig, procguard, run_vm}; +use openshell_driver_vm::{ + VmBackend, VmDriver, VmDriverConfig, VmLaunchConfig, VsockPortMap, procguard, run_vm, +}; use std::io; use std::net::SocketAddr; use std::os::unix::fs::{FileTypeExt, MetadataExt, PermissionsExt}; @@ -189,6 +191,30 @@ struct Args { #[arg(long, env = "OPENSHELL_VM_SANDBOX_GID")] sandbox_gid: Option, + // Corporate forward proxy for sandbox egress. Operator-owned: these reach + // the host supervisor on its argv, which the sandbox image and the + // user-supplied environment cannot influence. + #[arg(long, env = "OPENSHELL_VM_HTTPS_PROXY")] + https_proxy: Option, + + #[arg(long, env = "OPENSHELL_VM_NO_PROXY")] + no_proxy: Option, + + #[arg(long, env = "OPENSHELL_VM_PROXY_AUTH_FILE")] + proxy_auth_file: Option, + + // Value-taking rather than a presence flag so an explicit `false` in + // `[openshell.drivers.vm]` survives the gateway -> driver hop and still + // trips the "acknowledgement without a credential" check. + #[arg(long, env = "OPENSHELL_VM_PROXY_AUTH_ALLOW_INSECURE")] + proxy_auth_allow_insecure: Option, + + #[arg(long, env = "OPENSHELL_VM_PROXY_CONNECT_BY_HOSTNAME")] + proxy_connect_by_hostname: Option, + + #[arg(long, env = "OPENSHELL_VM_PROXY_CA_BUNDLE")] + proxy_ca_bundle: Option, + #[arg(long, env = "OPENSHELL_VM_ROOTFS_TAR_STAGING_DIR")] rootfs_tar_staging_dir: Option, @@ -201,37 +227,22 @@ struct Args { #[arg(long, hide = true)] vm_gpu_bdf: Option, - #[arg(long, hide = true)] - vm_tap_device: Option, - - #[arg(long, hide = true)] - vm_guest_ip: Option, - - #[arg(long, hide = true)] - vm_host_ip: Option, - #[arg(long, hide = true)] vm_vsock_cid: Option, #[arg(long, hide = true)] - vm_guest_mac: Option, + vm_vsock_control_port: Option, #[arg(long, hide = true)] - vm_gateway_port: Option, + vm_vsock_control_socket: Option, } #[tokio::main] async fn main() -> Result<()> { let args = Args::parse(); if args.internal_run_vm { - // We intentionally defer procguard arming until `run_vm()` so - // that the only arm is the one that knows how to clean up - // gvproxy. Racing two watchers against the same parent-death - // event causes the bare arm's `exit(1)` to win, skipping the - // gvproxy cleanup and leaking the helper. The risk window - // before `run_vm` arms procguard is ~a few syscalls long - // (`build_vm_launch_config`, `configured_runtime_dir`), which - // is negligible next to the parent gRPC server's uptime. + // The VM launcher arms procguard after resolving its runtime so its + // libkrun worker cannot outlive the launcher. maybe_reexec_internal_vm_with_runtime_env()?; let config = build_vm_launch_config(&args).map_err(|err| miette::miette!("{err}"))?; run_vm(&config).map_err(|err| miette::miette!("{err}"))?; @@ -254,7 +265,7 @@ async fn main() -> Result<()> { // we also die. Without this the driver is reparented to init and // keeps its per-sandbox VM launchers alive forever. Launchers have // their own procguards (armed in `run_vm`) which cascade cleanup of - // gvproxy and the libkrun worker the moment this driver exits. + // the libkrun worker the moment this driver exits. if let Err(err) = procguard::die_with_parent() { tracing::warn!( error = %err, @@ -598,12 +609,24 @@ fn build_vm_launch_config(args: &Args) -> std::result::Result Some(VsockPortMap { + guest_port, + host_socket, + host_initiated: true, + }), + (None, None) => None, + _ => { + return Err( + "--vm-vsock-control-port and --vm-vsock-control-socket must be set together" + .to_string(), + ); + } + }, }) } diff --git a/crates/openshell-driver-vm/src/nft_ruleset.rs b/crates/openshell-driver-vm/src/nft_ruleset.rs deleted file mode 100644 index fe3e86c902..0000000000 --- a/crates/openshell-driver-vm/src/nft_ruleset.rs +++ /dev/null @@ -1,92 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use std::fmt::Write; - -/// Sanitize a TAP device name for use as an nftables table name suffix. -/// Assumes device names match `vmtap-[a-f0-9]+` (driver-controlled). -fn sanitize_table_name(device: &str) -> String { - device.replace('-', "_") -} - -/// Return the nftables table name for a TAP device. -pub fn teardown_table_name(device: &str) -> String { - format!("openshell_vm_{}", sanitize_table_name(device)) -} - -/// Generate the nftables ruleset for VM TAP networking. -pub fn generate_tap_ruleset(tap_device: &str, subnet: &str, gateway_port: u16) -> String { - let table_name = teardown_table_name(tap_device); - let mut ruleset = String::with_capacity(512); - - writeln!(ruleset, "table ip {table_name} {{").unwrap(); - writeln!(ruleset, " chain postrouting {{").unwrap(); - writeln!( - ruleset, - " type nat hook postrouting priority 100; policy accept;" - ) - .unwrap(); - writeln!(ruleset, " ip saddr {subnet} masquerade").unwrap(); - writeln!(ruleset, " }}").unwrap(); - writeln!(ruleset, " chain forward {{").unwrap(); - writeln!( - ruleset, - " type filter hook forward priority 0; policy accept;" - ) - .unwrap(); - writeln!(ruleset, " iifname \"{tap_device}\" accept").unwrap(); - writeln!( - ruleset, - " oifname \"{tap_device}\" ct state related,established accept" - ) - .unwrap(); - writeln!(ruleset, " oifname \"{tap_device}\" drop").unwrap(); - writeln!(ruleset, " }}").unwrap(); - writeln!(ruleset, " chain input {{").unwrap(); - writeln!( - ruleset, - " type filter hook input priority 0; policy accept;" - ) - .unwrap(); - writeln!( - ruleset, - " iifname \"{tap_device}\" tcp dport {gateway_port} accept" - ) - .unwrap(); - writeln!(ruleset, " iifname \"{tap_device}\" drop").unwrap(); - writeln!(ruleset, " }}").unwrap(); - writeln!(ruleset, "}}").unwrap(); - - ruleset -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn generates_tap_setup_ruleset() { - let ruleset = generate_tap_ruleset("vmtap-abcd", "10.0.128.0/30", 8080); - assert!(ruleset.contains("table ip openshell_vm_vmtap_abcd {")); - assert!(ruleset.contains("type nat hook postrouting priority 100; policy accept;")); - assert!(ruleset.contains("ip saddr 10.0.128.0/30 masquerade")); - assert!(ruleset.contains("type filter hook forward priority 0; policy accept;")); - assert!(ruleset.contains("iifname \"vmtap-abcd\" accept")); - assert!(ruleset.contains("oifname \"vmtap-abcd\" ct state related,established accept")); - assert!(ruleset.contains("oifname \"vmtap-abcd\" drop")); - assert!(ruleset.contains("type filter hook input priority 0; policy accept;")); - assert!(ruleset.contains("iifname \"vmtap-abcd\" tcp dport 8080 accept")); - } - - #[test] - fn table_name_sanitizes_device_name() { - let ruleset = generate_tap_ruleset("vmtap-abc-123", "10.0.128.0/30", 8080); - assert!(ruleset.contains("table ip openshell_vm_vmtap_abc_123 {")); - } - - #[test] - fn teardown_command_targets_correct_table() { - let cmd = teardown_table_name("vmtap-abcd"); - assert_eq!(cmd, "openshell_vm_vmtap_abcd"); - } -} diff --git a/crates/openshell-driver-vm/src/procguard.rs b/crates/openshell-driver-vm/src/procguard.rs index fd4d3c872c..5f89f0848c 100644 --- a/crates/openshell-driver-vm/src/procguard.rs +++ b/crates/openshell-driver-vm/src/procguard.rs @@ -4,10 +4,9 @@ //! Cross-platform "die when my parent dies" primitive. //! //! The VM driver spawns a chain of subprocesses (compute driver → `--internal-run-vm` -//! launcher → gvproxy + libkrun fork). If any link in that chain is killed +//! launcher → libkrun fork). If any link in that chain is killed //! with SIGKILL — or simply crashes — the children are reparented to init -//! and survive indefinitely, leaking libkrun workers and gvproxy -//! instances. +//! and survive indefinitely, leaking libkrun workers. //! //! This module exposes two functions: //! * [`die_with_parent`] — configure the kernel (Linux) or a helper @@ -17,7 +16,7 @@ //! the runtime.rs comment at the single call site). //! * [`die_with_parent_cleanup`] — same as above, but on the BSD path a //! best-effort cleanup callback runs *before* this process exits. -//! This matters when we own a non-Rust child (e.g. gvproxy) that +//! This matters when we own a non-Rust child that //! cannot arm its own procguard; the callback lets us SIGTERM it //! first. //! diff --git a/crates/openshell-driver-vm/src/rootfs.rs b/crates/openshell-driver-vm/src/rootfs.rs index 5bd7d80f35..c868049343 100644 --- a/crates/openshell-driver-vm/src/rootfs.rs +++ b/crates/openshell-driver-vm/src/rootfs.rs @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 use flate2::read::MultiGzDecoder; +use sha2::{Digest, Sha256}; use std::fs; use std::fs::File; #[cfg(test)] @@ -9,9 +10,14 @@ use std::io::BufWriter; use std::io::{BufRead, BufReader, Cursor, Read, Seek, SeekFrom, Write}; use std::path::{Path, PathBuf}; use std::process::Command; +use std::sync::OnceLock; use std::sync::atomic::{AtomicU64, Ordering}; -const SUPERVISOR: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/openshell-sandbox.zst")); +use crate::driver::DEFAULT_SANDBOX_UID; + +const SANDBOX: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/openshell-sandbox.zst")); +const SUPERVISOR: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/openshell-supervisor.zst")); +const VM_INIT: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/openshell-vm-init.zst")); const UMOCI: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/umoci.zst")); const ROOTFS_VARIANT_MARKER: &str = ".openshell-rootfs-variant"; /// Leading bytes of a gzip stream, used to recognize `.tar.gz`/`.tgz` input @@ -20,7 +26,7 @@ pub const GZIP_MAGIC: [u8; 2] = [0x1f, 0x8b]; const SANDBOX_GUEST_INIT_PATH: &str = "/srv/openshell-vm-sandbox-init.sh"; const SANDBOX_SUPERVISOR_PATH: &str = openshell_core::driver_utils::SUPERVISOR_CONTAINER_BINARY; const SANDBOX_UMOCI_PATH: &str = openshell_core::container_paths::VM_UMOCI_PATH; -const DEFAULT_SANDBOX_UID: u32 = 1000; +const SANDBOX_VM_INIT_PATH: &str = "/opt/openshell/bin/openshell-vm-init"; const ROOTFS_IMAGE_MIN_SIZE_BYTES: u64 = 512 * 1024 * 1024; const ROOTFS_IMAGE_MIN_HEADROOM_BYTES: u64 = 256 * 1024 * 1024; const EXT4_IMAGE_MIN_HEADROOM_BYTES: u64 = 16 * 1024 * 1024; @@ -30,6 +36,141 @@ pub const fn sandbox_guest_init_path() -> &'static str { SANDBOX_GUEST_INIT_PATH } +/// Identity of every embedded artifact materialized into a bootstrap rootfs. +/// +/// Including this in the image-cache key makes local, uncommitted guest-sandbox +/// changes invalidate the cache even when the `OpenShell` version is unchanged. +pub fn sandbox_guest_runtime_identity() -> String { + let mut hasher = Sha256::new(); + hasher.update(SANDBOX); + hasher.update(VM_INIT); + hasher.update(UMOCI); + hasher.update(include_bytes!("../scripts/openshell-vm-sandbox-init.sh")); + format!("{:x}", hasher.finalize()) +} + +/// Materialize the supervisor embedded in the VM driver for host-side use. +pub fn extract_host_supervisor(path: &Path) -> Result<(), String> { + if SUPERVISOR.is_empty() { + return Err( + "host supervisor is not embedded; run `mise run vm:supervisor` and rebuild openshell-driver-vm" + .to_string(), + ); + } + let supervisor = embedded_host_supervisor()?; + install_host_supervisor_atomically(path, &supervisor)?; + validate_host_supervisor(path) +} + +pub fn validate_host_supervisor(path: &Path) -> Result<(), String> { + validate_host_supervisor_digest(path, embedded_host_supervisor_digest()?) +} + +fn validate_host_supervisor_digest(path: &Path, expected: [u8; 32]) -> Result<(), String> { + let metadata = fs::symlink_metadata(path) + .map_err(|error| format!("inspect cached host supervisor {}: {error}", path.display()))?; + if !metadata.file_type().is_file() { + return Err(format!( + "cached host supervisor is not a regular file: {}", + path.display() + )); + } + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + if metadata.permissions().mode() & 0o111 == 0 { + return Err(format!( + "cached host supervisor is not executable: {}", + path.display() + )); + } + } + let actual = sha256_reader( + File::open(path) + .map_err(|error| format!("open cached host supervisor {}: {error}", path.display()))?, + ) + .map_err(|error| format!("hash cached host supervisor {}: {error}", path.display()))?; + if actual != expected { + return Err(format!( + "cached host supervisor content does not match embedded runtime: {}", + path.display() + )); + } + Ok(()) +} + +fn embedded_host_supervisor() -> Result, String> { + zstd::decode_all(Cursor::new(SUPERVISOR)) + .map_err(|error| format!("decompress host supervisor: {error}")) +} + +fn embedded_host_supervisor_digest() -> Result<[u8; 32], String> { + static DIGEST: OnceLock> = OnceLock::new(); + DIGEST + .get_or_init(|| embedded_host_supervisor().map(|bytes| sha256_bytes(&bytes))) + .clone() +} + +fn sha256_bytes(bytes: &[u8]) -> [u8; 32] { + Sha256::digest(bytes).into() +} + +fn sha256_reader(mut reader: impl Read) -> std::io::Result<[u8; 32]> { + let mut hasher = Sha256::new(); + let mut buffer = [0_u8; 16 * 1024]; + loop { + let read = reader.read(&mut buffer)?; + if read == 0 { + break; + } + hasher.update(&buffer[..read]); + } + Ok(hasher.finalize().into()) +} + +fn install_host_supervisor_atomically(path: &Path, bytes: &[u8]) -> Result<(), String> { + let parent = path + .parent() + .filter(|parent| !parent.as_os_str().is_empty()) + .ok_or_else(|| format!("host supervisor path has no parent: {}", path.display()))?; + fs::create_dir_all(parent).map_err(|error| format!("create {}: {error}", parent.display()))?; + let temporary = parent.join(format!( + ".openshell-sandbox.tmp-{}-{}", + std::process::id(), + INJECTION_COUNTER.fetch_add(1, Ordering::Relaxed) + )); + let result = (|| { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(0o755); + } + let mut file = options + .open(&temporary) + .map_err(|error| format!("create {}: {error}", temporary.display()))?; + file.write_all(bytes) + .map_err(|error| format!("write {}: {error}", temporary.display()))?; + file.sync_all() + .map_err(|error| format!("sync {}: {error}", temporary.display()))?; + fs::rename(&temporary, path).map_err(|error| { + format!( + "commit cached host supervisor {} to {}: {error}", + temporary.display(), + path.display() + ) + })?; + File::open(parent) + .and_then(|directory| directory.sync_all()) + .map_err(|error| format!("sync host supervisor cache {}: {error}", parent.display())) + })(); + if result.is_err() { + let _ = fs::remove_file(&temporary); + } + result +} + #[allow(clippy::similar_names)] pub fn prepare_sandbox_rootfs_from_image_root( rootfs: &Path, @@ -204,6 +345,13 @@ pub fn write_rootfs_image_file( result } +pub fn remove_rootfs_image_file(image_path: &Path, guest_path: &str) -> Result<(), String> { + let Some(quoted_guest_path) = debugfs_quote_absolute_path(guest_path) else { + return Err(format!("invalid debugfs guest path '{guest_path}'")); + }; + run_debugfs(image_path, &format!("rm {quoted_guest_path}")) +} + pub fn set_rootfs_image_file_mode( image_path: &Path, guest_path: &str, @@ -219,6 +367,41 @@ pub fn set_rootfs_image_file_mode( ) } +/// Replay the ext4 journal and repair automatically correctable filesystem +/// state before the driver mutates a preserved guest disk offline. +pub fn recover_rootfs_image(image_path: &Path) -> Result<(), String> { + let mut failures = Vec::new(); + let mut unavailable = Vec::new(); + + for candidate in e2fs_tool_candidates("e2fsck") { + let label = candidate.display().to_string(); + match Command::new(&candidate) + .arg("-p") + .arg("-f") + .arg(image_path) + .output() + { + Ok(output) if matches!(output.status.code(), Some(0..=2)) => return Ok(()), + Ok(output) => failures.push(format!( + "{label} failed with status {}\nstdout: {}\nstderr: {}", + output.status, + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + )), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + unavailable.push(format!("{label} not found")); + } + Err(error) => failures.push(format!("run {label}: {error}")), + } + } + + Err(if failures.is_empty() { + unavailable.join("\n") + } else { + failures.join("\n") + }) +} + #[cfg(target_os = "macos")] fn try_clone_file(source: &Path, dest: &Path) -> Result<(), String> { let output = Command::new("cp") @@ -401,6 +584,7 @@ fn prepare_sandbox_rootfs( } ensure_supervisor_binary(rootfs)?; + ensure_vm_init_binary(rootfs)?; ensure_umoci_binary(rootfs)?; let opt_dir = rootfs.join("opt/openshell"); @@ -420,6 +604,7 @@ fn prepare_sandbox_rootfs( pub fn validate_sandbox_rootfs(rootfs: &Path) -> Result<(), String> { require_rootfs_path(rootfs, SANDBOX_GUEST_INIT_PATH)?; require_rootfs_path(rootfs, SANDBOX_SUPERVISOR_PATH)?; + require_rootfs_path(rootfs, SANDBOX_VM_INIT_PATH)?; require_rootfs_path(rootfs, SANDBOX_UMOCI_PATH)?; require_any_rootfs_path(rootfs, &["/bin/bash"])?; require_any_rootfs_path(rootfs, &["/bin/mount", "/usr/bin/mount"])?; @@ -432,10 +617,6 @@ pub fn validate_sandbox_rootfs(rootfs: &Path) -> Result<(), String> { "/bin/chroot", ], )?; - require_any_rootfs_path( - rootfs, - &["/sbin/ip", "/usr/sbin/ip", "/bin/ip", "/usr/bin/ip"], - )?; require_any_rootfs_path(rootfs, &["/bin/sed", "/usr/bin/sed"])?; Ok(()) } @@ -908,20 +1089,20 @@ fn ensure_sandbox_guest_user( let etc_dir = rootfs.join("etc"); fs::create_dir_all(&etc_dir).map_err(|e| format!("create {}: {e}", etc_dir.display()))?; - ensure_line_in_file( + replace_or_append_line( &etc_dir.join("group"), &format!("sandbox:x:{sandbox_gid}:"), |line| line.starts_with("sandbox:"), )?; - ensure_line_in_file(&etc_dir.join("gshadow"), "sandbox:!::", |line| { + replace_or_append_line(&etc_dir.join("gshadow"), "sandbox:!::", |line| { line.starts_with("sandbox:") })?; - ensure_line_in_file( + replace_or_append_line( &etc_dir.join("passwd"), &format!("sandbox:x:{sandbox_uid}:{sandbox_gid}:OpenShell Sandbox:/sandbox:/bin/bash"), |line| line.starts_with("sandbox:"), )?; - ensure_line_in_file( + replace_or_append_line( &etc_dir.join("shadow"), "sandbox:!:20123:0:99999:7:::", |line| line.starts_with("sandbox:"), @@ -930,37 +1111,36 @@ fn ensure_sandbox_guest_user( Ok(()) } -fn ensure_line_in_file( +fn replace_or_append_line( path: &Path, line: &str, - exists: impl Fn(&str) -> bool, + matches: impl Fn(&str) -> bool, ) -> Result<(), String> { let contents = if path.exists() { fs::read_to_string(path).map_err(|e| format!("read {}: {e}", path.display()))? } else { String::new() }; - + let mut output = String::with_capacity(contents.len().max(line.len() + 1)); let mut replaced = false; - let mut updated = String::new(); for existing in contents.lines() { - if exists(existing) { - if !replaced { - updated.push_str(line); - updated.push('\n'); - replaced = true; + if matches(existing) { + if replaced { + continue; } + output.push_str(line); + replaced = true; } else { - updated.push_str(existing); - updated.push('\n'); + output.push_str(existing); } + output.push('\n'); } if !replaced { - updated.push_str(line); - updated.push('\n'); + output.push_str(line); + output.push('\n'); } - fs::write(path, updated).map_err(|e| format!("write {}: {e}", path.display())) + fs::write(path, output).map_err(|e| format!("write {}: {e}", path.display())) } fn ensure_supervisor_binary(rootfs: &Path) -> Result<(), String> { @@ -977,9 +1157,38 @@ fn ensure_supervisor_binary(rootfs: &Path) -> Result<(), String> { fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?; } - let supervisor = zstd::decode_all(Cursor::new(SUPERVISOR)) - .map_err(|e| format!("decompress supervisor: {e}"))?; - fs::write(&path, supervisor).map_err(|e| format!("write {}: {e}", path.display()))?; + let sandbox = zstd::decode_all(Cursor::new(SANDBOX)) + .map_err(|e| format!("decompress sandbox: {e}"))?; + fs::write(&path, sandbox).map_err(|e| format!("write {}: {e}", path.display()))?; + } + + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + + fs::set_permissions(&path, fs::Permissions::from_mode(0o755)) + .map_err(|e| format!("chmod {}: {e}", path.display()))?; + } + + Ok(()) +} + +fn ensure_vm_init_binary(rootfs: &Path) -> Result<(), String> { + let path = rootfs.join(SANDBOX_VM_INIT_PATH.trim_start_matches('/')); + if VM_INIT.is_empty() { + if !path.exists() { + return Err( + "VM guest init helper not embedded. Build openshell-driver-vm with OPENSHELL_VM_RUNTIME_COMPRESSED_DIR set and run `mise run vm:supervisor` first" + .to_string(), + ); + } + } else { + if let Some(parent) = path.parent() { + fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?; + } + let helper = zstd::decode_all(Cursor::new(VM_INIT)) + .map_err(|e| format!("decompress VM guest init helper: {e}"))?; + fs::write(&path, helper).map_err(|e| format!("write {}: {e}", path.display()))?; } #[cfg(unix)] @@ -1066,10 +1275,36 @@ fn remove_rootfs_path(rootfs: &Path, relative: &str) -> Result<(), String> { #[cfg(test)] mod tests { use super::*; + #[cfg(target_os = "linux")] + use std::os::unix::fs::PermissionsExt as _; use std::path::PathBuf; use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{SystemTime, UNIX_EPOCH}; + #[cfg(target_os = "linux")] + #[test] + fn host_supervisor_cache_rejects_wrong_content_and_installs_atomically() { + let directory = tempfile::tempdir().expect("cache directory"); + let destination = directory.path().join("openshell-sandbox"); + fs::write(&destination, b"stale executable").expect("write stale cache"); + fs::set_permissions(&destination, fs::Permissions::from_mode(0o755)) + .expect("make stale cache executable"); + let expected = sha256_bytes(b"trusted supervisor"); + + assert!(validate_host_supervisor_digest(&destination, expected).is_err()); + install_host_supervisor_atomically(&destination, b"trusted supervisor") + .expect("atomically replace cache"); + validate_host_supervisor_digest(&destination, expected).expect("validate installed cache"); + assert_eq!(fs::read(&destination).unwrap(), b"trusted supervisor"); + assert!(fs::read_dir(directory.path()).unwrap().all(|entry| { + !entry + .unwrap() + .file_name() + .to_string_lossy() + .contains(".tmp-") + })); + } + #[test] fn prepare_sandbox_rootfs_rewrites_guest_layout() { let dir = unique_temp_dir(); @@ -1081,10 +1316,10 @@ mod tests { write_fake_runtime_binaries(&rootfs); fs::write( rootfs.join("etc/passwd"), - "root:x:0:0:root:/root:/bin/bash\nsandbox:x:10001:10001:OpenShell Sandbox:/sandbox:/bin/sh\n", + "root:x:0:0:root:/root:/bin/bash\nsandbox:x:998:997:Sandbox:/sandbox:/bin/sh\n", ) .expect("write passwd"); - fs::write(rootfs.join("etc/group"), "root:x:0:\nsandbox:x:10001:\n").expect("write group"); + fs::write(rootfs.join("etc/group"), "root:x:0:\nsandbox:x:997:\n").expect("write group"); fs::write(rootfs.join("etc/hosts"), "127.0.0.1 localhost\n").expect("write hosts"); fs::create_dir_all(rootfs.join("bin")).expect("create bin"); fs::create_dir_all(rootfs.join("sbin")).expect("create sbin"); @@ -1101,6 +1336,25 @@ mod tests { assert!(rootfs.join("srv/openshell-vm-sandbox-init.sh").is_file()); assert!(rootfs.join("opt/openshell/bin/umoci").is_file()); + assert!(rootfs.join("opt/openshell/bin/openshell-vm-init").is_file()); + let init_script = fs::read_to_string(rootfs.join("srv/openshell-vm-sandbox-init.sh")) + .expect("read guest init"); + assert!( + init_script.contains( + "launch-capability-free \"$_sandbox_uid\" \"$_sandbox_gid\" \"$_sandbox_bootstrap_guest\"" + ) + ); + assert!(init_script.contains("OPENSHELL_VM_SANDBOX_BOOTSTRAP")); + assert!(init_script.contains(" reconcile_sandbox_account\n setup_sandbox_workdir")); + assert!( + init_script + .contains("chown \"${_sandbox_uid}:${_sandbox_gid}\" \"$_sandbox_state_dir\"") + ); + assert!(init_script.contains("chmod 0700 \"$_sandbox_state_dir\"")); + assert!(!init_script.contains("--backend-name=in-pod")); + assert!(!init_script.contains("@ISOLATION_INTERFACE_VERSION@")); + assert!(!init_script.contains("8.8.8.8")); + assert!(!init_script.contains("VM_NET_")); assert!(rootfs.join("sandbox").is_dir()); assert!(rootfs.join("image-cache").is_dir()); assert!(rootfs.join("lower").is_dir()); @@ -1112,24 +1366,14 @@ mod tests { .next() .is_none() ); - assert!( - fs::read_to_string(rootfs.join("etc/passwd")) - .expect("read passwd") - .contains(&format!( - "sandbox:x:{uid}:{uid}:OpenShell Sandbox:/sandbox:/bin/bash" - )) - ); - assert!( - fs::read_to_string(rootfs.join("etc/group")) - .expect("read group") - .contains(&format!("sandbox:x:{uid}:")) - ); - assert!( - !fs::read_to_string(rootfs.join("etc/passwd")) - .expect("read passwd") - .contains("sandbox:x:10001:"), - "newly prepared rootfs must replace the legacy sandbox account" - ); + let passwd = fs::read_to_string(rootfs.join("etc/passwd")).expect("read passwd"); + assert!(passwd.contains(&format!( + "sandbox:x:{uid}:{uid}:OpenShell Sandbox:/sandbox:/bin/bash" + ))); + assert!(!passwd.contains("sandbox:x:998:997:")); + let group = fs::read_to_string(rootfs.join("etc/group")).expect("read group"); + assert!(group.contains(&format!("sandbox:x:{uid}:"))); + assert!(!group.contains("sandbox:x:997:")); assert_eq!( fs::read_to_string(rootfs.join("etc/hosts")).expect("read hosts"), "127.0.0.1 localhost\n" @@ -1259,58 +1503,25 @@ mod tests { } #[test] - fn sandbox_user_preserves_image_account_when_identity_is_omitted() { - let dir = unique_temp_dir(); - let rootfs = dir.join("rootfs"); - fs::create_dir_all(rootfs.join("etc")).unwrap(); - fs::write( - rootfs.join("etc/passwd"), - "sandbox:x:4242:4343:Image:/image-home:/bin/false\n", - ) - .unwrap(); - fs::write(rootfs.join("etc/group"), "sandbox:x:4343:\n").unwrap(); - - ensure_sandbox_guest_user(&rootfs, None, None).unwrap(); - - assert_eq!(sandbox_guest_user_ids(&rootfs).unwrap(), Some((4242, 4343))); - assert!( - fs::read_to_string(rootfs.join("etc/passwd")) - .unwrap() - .contains("Image:/image-home:/bin/false") - ); - let _ = fs::remove_dir_all(dir); - } - - #[test] - fn sandbox_user_defaults_to_1000_when_image_has_no_account() { - let dir = unique_temp_dir(); - let rootfs = dir.join("rootfs"); - ensure_sandbox_guest_user(&rootfs, None, None).unwrap(); - assert_eq!(sandbox_guest_user_ids(&rootfs).unwrap(), Some((1000, 1000))); - let _ = fs::remove_dir_all(dir); - } + fn recover_rootfs_image_accepts_clean_ext4_image() { + if !e2fs_tool_candidates("e2fsck") + .iter() + .any(|candidate| Command::new(candidate).arg("-V").output().is_ok()) + { + return; + } - #[test] - fn sandbox_user_explicit_identity_overrides_image_account() { let dir = unique_temp_dir(); - let rootfs = dir.join("rootfs"); - fs::create_dir_all(rootfs.join("etc")).unwrap(); - fs::write( - rootfs.join("etc/passwd"), - "sandbox:x:4242:4343:Image:/image-home:/bin/false\n", - ) - .unwrap(); - fs::write(rootfs.join("etc/group"), "sandbox:x:4343:\n").unwrap(); + let source = dir.join("source"); + let image = dir.join("overlay.ext4"); + fs::create_dir_all(source.join("upper")).expect("create source upperdir"); + fs::create_dir_all(source.join("work")).expect("create source workdir"); + create_ext4_image_from_dir_with_size(&source, &image, 64 * 1024 * 1024) + .expect("create ext4 image"); - ensure_sandbox_guest_user(&rootfs, Some(2000), Some(3000)).unwrap(); + recover_rootfs_image(&image).expect("recover clean ext4 image"); - assert_eq!(sandbox_guest_user_ids(&rootfs).unwrap(), Some((2000, 3000))); - assert!( - fs::read_to_string(rootfs.join("etc/group")) - .unwrap() - .contains("sandbox:x:3000:") - ); - let _ = fs::remove_dir_all(dir); + let _ = fs::remove_dir_all(&dir); } #[test] @@ -1448,6 +1659,11 @@ mod tests { b"sandbox", ) .expect("write openshell-sandbox"); + fs::write( + rootfs.join("opt/openshell/bin/openshell-vm-init"), + b"vm-init", + ) + .expect("write openshell-vm-init"); fs::write(rootfs.join("opt/openshell/bin/umoci"), b"umoci").expect("write umoci"); } } diff --git a/crates/openshell-driver-vm/src/runtime.rs b/crates/openshell-driver-vm/src/runtime.rs index f6020af829..4d0680bffe 100644 --- a/crates/openshell-driver-vm/src/runtime.rs +++ b/crates/openshell-driver-vm/src/runtime.rs @@ -5,12 +5,12 @@ use std::ffi::CString; use std::path::{Path, PathBuf}; -use std::process::{Child as StdChild, Command as StdCommand, Stdio}; +use std::process::{Command as StdCommand, Stdio}; use std::ptr; use std::sync::atomic::{AtomicI32, Ordering}; -use std::time::{Duration, Instant}; +use std::time::Duration; -use crate::{embedded_runtime, ffi, nft_ruleset, procguard, rootfs}; +use crate::{embedded_runtime, ffi, procguard, rootfs}; pub const VM_RUNTIME_DIR_ENV: &str = "OPENSHELL_VM_RUNTIME_DIR"; const KRUN_INIT_PID1_ENV: &str = "KRUN_INIT_PID1=1"; @@ -19,31 +19,18 @@ const KRUN_INIT_PID1_ENV: &str = "KRUN_INIT_PID1=1"; /// Used by the SIGTERM/SIGINT handler to forward signals to the VM. static CHILD_PID: AtomicI32 = AtomicI32::new(0); -/// PID of the helper process (gvproxy for libkrun; zero for QEMU). -/// Zero when not running. Used by the SIGTERM/SIGINT handler and -/// procguard cleanup callback to ensure the helper doesn't outlive the -/// launcher (especially on macOS where `PR_SET_PDEATHSIG` is absent). -static GVPROXY_PID: AtomicI32 = AtomicI32::new(0); - #[derive(Debug, Clone, PartialEq, Eq)] pub enum VmBackend { Libkrun, Qemu, } -// virtio-net feature bits (see Linux `include/uapi/linux/virtio_net.h`). -const NET_FEATURE_CSUM: u32 = 1 << 0; -const NET_FEATURE_GUEST_CSUM: u32 = 1 << 1; -const NET_FEATURE_GUEST_TSO4: u32 = 1 << 7; -const NET_FEATURE_GUEST_UFO: u32 = 1 << 10; -const NET_FEATURE_HOST_TSO4: u32 = 1 << 11; -const NET_FEATURE_HOST_UFO: u32 = 1 << 14; -const COMPAT_NET_FEATURES: u32 = NET_FEATURE_CSUM - | NET_FEATURE_GUEST_CSUM - | NET_FEATURE_GUEST_TSO4 - | NET_FEATURE_GUEST_UFO - | NET_FEATURE_HOST_TSO4 - | NET_FEATURE_HOST_UFO; +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct VsockPortMap { + pub guest_port: u32, + pub host_socket: PathBuf, + pub host_initiated: bool, +} pub struct VmLaunchConfig { pub root_disk: PathBuf, @@ -60,12 +47,8 @@ pub struct VmLaunchConfig { pub console_output: PathBuf, pub backend: VmBackend, pub gpu_bdf: Option, - pub tap_device: Option, - pub guest_ip: Option, - pub host_ip: Option, pub vsock_cid: Option, - pub guest_mac: Option, - pub gateway_port: Option, + pub vsock_port_map: Option, } pub fn run_vm(config: &VmLaunchConfig) -> Result<(), String> { @@ -80,25 +63,9 @@ fn run_qemu_vm(config: &VmLaunchConfig) -> Result<(), String> { .gpu_bdf .as_deref() .ok_or("gpu_bdf is required for QEMU backend")?; - let tap_device = config - .tap_device - .as_deref() - .ok_or("tap_device is required for QEMU backend")?; - let guest_mac = config - .guest_mac - .as_deref() - .ok_or("guest_mac is required for QEMU backend")?; let vsock_cid = config .vsock_cid .ok_or("vsock_cid is required for QEMU backend")?; - let _guest_ip = config - .guest_ip - .as_deref() - .ok_or("guest_ip is required for QEMU backend")?; - let host_ip = config - .host_ip - .as_deref() - .ok_or("host_ip is required for QEMU backend")?; if !config.root_disk.is_file() { return Err(format!( @@ -125,13 +92,9 @@ fn run_qemu_vm(config: &VmLaunchConfig) -> Result<(), String> { #[cfg(target_os = "linux")] check_kvm_access()?; - let guest_env = qemu_guest_env_vars(config, host_dns_server()); + let guest_env = qemu_guest_env_vars(config); write_guest_env_file(&config.overlay_disk, &guest_env)?; - let gw_port = config.gateway_port.unwrap_or(0); - setup_tap_networking(tap_device, host_ip, gw_port)?; - let mut tap_guard = TapGuard::new(tap_device.to_string(), host_ip.to_string(), gw_port); - let vmlinux = if let Some(kernel_image) = &config.kernel_image { kernel_image.clone() } else { @@ -155,21 +118,12 @@ fn run_qemu_vm(config: &VmLaunchConfig) -> Result<(), String> { .arg(format!("{}M", config.mem_mib)) .arg("-nographic") .arg("-no-reboot") + .args(qemu_network_args()) .arg("-kernel") .arg(&vmlinux) .arg("-append") .arg(&kernel_cmdline) .args(qemu_disk_args(config)) - .arg("-netdev") - .arg(format!( - "tap,id=net0,ifname={tap_device},script=no,downscript=no" - )) - .arg("-device") - .arg("pcie-root-port,id=net_root,slot=3") - .arg("-device") - .arg(format!( - "virtio-net-pci-non-transitional,netdev=net0,mac={guest_mac},bus=net_root" - )) .arg("-device") .arg("pcie-root-port,id=vsock_root,slot=1") .arg("-device") @@ -211,8 +165,6 @@ fn run_qemu_vm(config: &VmLaunchConfig) -> Result<(), String> { .map_err(|e| format!("failed to wait for QEMU: {e}"))?; CHILD_PID.store(0, Ordering::Relaxed); - teardown_tap_networking(tap_device, host_ip, gw_port); - tap_guard.disarm(); if status.success() { Ok(()) @@ -221,6 +173,10 @@ fn run_qemu_vm(config: &VmLaunchConfig) -> Result<(), String> { } } +fn qemu_network_args() -> [&'static str; 2] { + ["-nic", "none"] +} + fn qemu_disk_args(config: &VmLaunchConfig) -> Vec { let mut args = vec![ "-drive".to_string(), @@ -271,20 +227,8 @@ fn write_guest_env_file(overlay_disk: &Path, env_vars: &[String]) -> Result<(), ) } -fn qemu_guest_env_vars(config: &VmLaunchConfig, dns_server: Option) -> Vec { +fn qemu_guest_env_vars(config: &VmLaunchConfig) -> Vec { let mut env_vars = config.env.clone(); - - if let Some(ip) = &config.guest_ip - && let Some(host_ip) = &config.host_ip - { - env_vars.push(format!("VM_NET_IP={ip}")); - env_vars.push(format!("VM_NET_GW={host_ip}")); - } - - if let Some(dns) = dns_server { - env_vars.push(format!("VM_NET_DNS={dns}")); - } - if config.gpu_bdf.is_some() { env_vars.push("GPU_ENABLED=true".to_string()); } @@ -312,12 +256,6 @@ fn build_kernel_cmdline(config: &VmLaunchConfig) -> String { format!("init={}", config.exec_path), ]; - if let Some(ip) = &config.guest_ip - && let Some(host_ip) = &config.host_ip - { - parts.push(format!("ip={ip}::{host_ip}:255.255.255.252:sandbox::off")); - } - if config.gpu_bdf.is_some() { parts.push("firmware_class.path=/lib/firmware".to_string()); } @@ -325,326 +263,16 @@ fn build_kernel_cmdline(config: &VmLaunchConfig) -> String { parts.join(" ") } -fn host_dns_server() -> Option { - // Prefer systemd-resolved upstream config (skips the 127.0.0.53 - // stub listener which is unreachable from inside QEMU/TAP guests). - for path in &["/run/systemd/resolve/resolv.conf", "/etc/resolv.conf"] { - let Ok(resolv) = std::fs::read_to_string(path) else { - continue; - }; - for line in resolv.lines() { - let line = line.trim(); - if let Some(server) = line.strip_prefix("nameserver") { - let server = server.trim(); - if server == "127.0.0.53" || server.starts_with("127.") { - continue; - } - if !server.is_empty() { - return Some(server.to_string()); - } - } - } - } - None -} - -/// Remove leftover `vmtap-*` interfaces from previous driver runs. -/// -/// Called once at driver startup for interfaces that were not torn down -/// (e.g. the launcher was `SIGKILL`-ed before teardown), so stale -/// interfaces cannot cause subnet routing conflicts with newly allocated TAPs. -pub fn cleanup_stale_tap_interfaces() { - let Ok(entries) = std::fs::read_dir("/sys/class/net") else { - return; - }; - for entry in entries.flatten() { - let name = entry.file_name(); - let Some(name) = name.to_str() else { - continue; - }; - if !name.starts_with("vmtap-") { - continue; - } - // Read the IP address so we can clean up iptables rules too. - // Port 0 tells teardown we don't know the original gateway port; - // the blanket legacy rule is still cleaned up best-effort. - let ip = read_tap_host_ip(name); - if let Some(ref host_ip) = ip { - teardown_tap_networking(name, host_ip, 0); - } else { - let _ = run_cmd("ip", &["link", "set", name, "down"]); - let _ = run_cmd("ip", &["tuntap", "del", "dev", name, "mode", "tap"]); - } - tracing::warn!(interface = %name, "removed stale TAP interface from previous run"); - } -} - -/// Read the first IPv4 address assigned to a network interface. -fn read_tap_host_ip(device: &str) -> Option { - let output = StdCommand::new("ip") - .args(["-4", "-o", "addr", "show", "dev", device]) - .stdin(Stdio::null()) - .stdout(Stdio::piped()) - .stderr(Stdio::null()) - .output() - .ok()?; - let stdout = String::from_utf8_lossy(&output.stdout); - // Format: "28: vmtap-xxx inet 10.0.128.1/30 ..." - for token in stdout.split_whitespace() { - if let Some((ip, _prefix)) = token.split_once('/') - && ip.parse::().is_ok() - { - return Some(ip.to_string()); - } - } - None -} - -fn setup_tap_networking(tap_device: &str, host_ip: &str, gateway_port: u16) -> Result<(), String> { - run_cmd("ip", &["tuntap", "add", "dev", tap_device, "mode", "tap"])?; - run_cmd( - "ip", - &["addr", "add", &format!("{host_ip}/30"), "dev", tap_device], - )?; - run_cmd("ip", &["link", "set", tap_device, "up"])?; - - // Deprioritize routes through down interfaces so a stale vmtap-* - // that somehow survives cleanup cannot shadow the active one. - let _ = std::fs::write( - format!("/proc/sys/net/ipv4/conf/{tap_device}/ignore_routes_with_linkdown"), - "1", - ); - - enable_ip_forwarding()?; - - let subnet = tap_subnet_from_host_ip(host_ip); - let table_name = nft_ruleset::teardown_table_name(tap_device); - - // Delete any stale nftables table from a previous driver run. - let _ = run_cmd("nft", &["delete", "table", "ip", &table_name]); - - // Clean up legacy iptables rules from older driver versions. - let _ = run_cmd( - "iptables", - &[ - "-t", - "nat", - "-D", - "POSTROUTING", - "-s", - &subnet, - "-j", - "MASQUERADE", - ], - ); - let _ = run_cmd( - "iptables", - &["-D", "FORWARD", "-i", tap_device, "-j", "ACCEPT"], - ); - let _ = run_cmd( - "iptables", - &[ - "-D", - "FORWARD", - "-o", - tap_device, - "-m", - "state", - "--state", - "RELATED,ESTABLISHED", - "-j", - "ACCEPT", - ], - ); - let port_str = gateway_port.to_string(); - let _ = run_cmd( - "iptables", - &[ - "-D", "INPUT", "-i", tap_device, "-p", "tcp", "--dport", &port_str, "-j", "ACCEPT", - ], - ); - let _ = run_cmd( - "iptables", - &["-D", "INPUT", "-i", tap_device, "-j", "ACCEPT"], - ); - - // Load nftables ruleset atomically. - let ruleset = nft_ruleset::generate_tap_ruleset(tap_device, &subnet, gateway_port); - run_nft_stdin(&ruleset)?; - - Ok(()) -} - -fn teardown_tap_networking(tap_device: &str, host_ip: &str, gateway_port: u16) { - // Delete the entire nftables table — single atomic operation. - let table_name = nft_ruleset::teardown_table_name(tap_device); - let _ = run_cmd("nft", &["delete", "table", "ip", &table_name]); - - // Clean up legacy iptables rules from older driver versions. - let subnet = tap_subnet_from_host_ip(host_ip); - let _ = run_cmd( - "iptables", - &[ - "-D", - "FORWARD", - "-o", - tap_device, - "-m", - "state", - "--state", - "RELATED,ESTABLISHED", - "-j", - "ACCEPT", - ], - ); - let _ = run_cmd( - "iptables", - &["-D", "FORWARD", "-i", tap_device, "-j", "ACCEPT"], - ); - if gateway_port > 0 { - let port_str = gateway_port.to_string(); - let _ = run_cmd( - "iptables", - &[ - "-D", "INPUT", "-i", tap_device, "-p", "tcp", "--dport", &port_str, "-j", "ACCEPT", - ], - ); - } - let _ = run_cmd( - "iptables", - &["-D", "INPUT", "-i", tap_device, "-j", "ACCEPT"], - ); - let _ = run_cmd( - "iptables", - &[ - "-t", - "nat", - "-D", - "POSTROUTING", - "-s", - &subnet, - "-j", - "MASQUERADE", - ], - ); - - let _ = run_cmd("ip", &["link", "set", tap_device, "down"]); - let _ = run_cmd("ip", &["tuntap", "del", "dev", tap_device, "mode", "tap"]); -} - -fn tap_subnet_from_host_ip(host_ip: &str) -> String { - host_ip.parse::().map_or_else( - |_| format!("{host_ip}/30"), - |ip| { - let base = u32::from(ip) & !3; - let base_ip = std::net::Ipv4Addr::from(base); - format!("{base_ip}/30") - }, - ) -} - -fn enable_ip_forwarding() -> Result<(), String> { - std::fs::write("/proc/sys/net/ipv4/ip_forward", "1") - .map_err(|e| format!("enable ip_forward: {e}")) -} - -fn run_cmd(cmd: &str, args: &[&str]) -> Result<(), String> { - let output = StdCommand::new(cmd) - .args(args) - .stdin(Stdio::null()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .output() - .map_err(|e| format!("failed to run {cmd}: {e}"))?; - if output.status.success() { - Ok(()) - } else { - let stderr = String::from_utf8_lossy(&output.stderr); - Err(format!("{cmd} {} failed: {stderr}", args.join(" "))) - } -} - -fn run_nft_stdin(ruleset: &str) -> Result<(), String> { - use std::io::Write; - - let mut child = StdCommand::new("nft") - .args(["-f", "-"]) - .stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .spawn() - .map_err(|e| format!("failed to run nft: {e}"))?; - - if let Some(mut stdin) = child.stdin.take() { - stdin - .write_all(ruleset.as_bytes()) - .map_err(|e| format!("failed to write nft ruleset: {e}"))?; - } - - let output = child - .wait_with_output() - .map_err(|e| format!("failed to wait for nft: {e}"))?; - - if output.status.success() { - Ok(()) - } else { - let stderr = String::from_utf8_lossy(&output.stderr); - Err(format!("nft -f - failed: {stderr}")) - } -} - -/// RAII guard that tears down TAP networking on drop. -struct TapGuard { - tap_device: String, - host_ip: String, - gateway_port: u16, - disarmed: bool, -} - -impl TapGuard { - fn new(tap_device: String, host_ip: String, gateway_port: u16) -> Self { - Self { - tap_device, - host_ip, - gateway_port, - disarmed: false, - } - } - - fn disarm(&mut self) { - self.disarmed = true; - } -} - -impl Drop for TapGuard { - fn drop(&mut self) { - if !self.disarmed { - teardown_tap_networking(&self.tap_device, &self.host_ip, self.gateway_port); - } - } -} - /// Shared procguard cleanup callback for both libkrun and QEMU paths. /// Only async-signal-safe calls: atomic loads and `kill(2)`. fn procguard_kill_children() { - let helper_pid = GVPROXY_PID.load(Ordering::Relaxed); let child_pid = CHILD_PID.load(Ordering::Relaxed); - if helper_pid > 0 { - unsafe { - libc::kill(helper_pid, libc::SIGTERM); - } - } if child_pid > 0 { unsafe { libc::kill(child_pid, libc::SIGTERM); } } std::thread::sleep(Duration::from_millis(200)); - if helper_pid > 0 { - unsafe { - libc::kill(helper_pid, libc::SIGKILL); - } - } if child_pid > 0 { unsafe { libc::kill(child_pid, libc::SIGKILL); @@ -677,13 +305,9 @@ fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { return Err(format!("image disk not found: {}", image_disk.display())); } - // Arm procguard first, BEFORE we spawn gvproxy or fork libkrun, so - // that the launcher can't be orphaned during setup. The cleanup - // callback reads the GVPROXY_PID atomic (initially 0 — no-op) and - // the CHILD_PID atomic (the libkrun fork), so it stays correct as - // those slots get populated later in this function. Only ONE arm - // per process: racing two watchers for the same NOTE_EXIT event - // would cause whichever wins to skip the cleanup. + // Arm procguard before forking libkrun so the VM worker cannot outlive + // the launcher. No network helper is started: the only host/guest data + // path is the protected vsock mapping below. if let Err(err) = procguard::die_with_parent_cleanup(procguard_kill_children) { return Err(format!("procguard arm failed: {err}")); } @@ -705,132 +329,12 @@ fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { )?; vm.set_workdir(&config.workdir)?; - // Run gvproxy strictly as the guest's virtual NIC / DHCP / router. - // - // After the supervisor-initiated relay migration (#867), the driver - // no longer forwards any host-side ports into the guest — all ingress - // traffic for SSH and exec rides the outbound `ConnectSupervisor` - // gRPC stream the guest opens to the gateway. What gvproxy still - // provides here is the TCP/IP *plane* the guest kernel needs: - // - // * a virtio-net backend attached to libkrun via a Unix - // SOCK_STREAM (Linux) or SOCK_DGRAM (macOS vfkit), which - // surfaces as `eth0` inside the guest; - // * the DHCP server + default router the guest's udhcpc client - // talks to on boot (IPs 192.168.127.1 / .2, defaults for - // gvisor-tap-vsock); - // * the host-facing gateway identity the guest uses for callbacks: - // gvproxy installs a default NAT entry rewriting `192.168.127.254` - // (the subnet's HostIP) to the host's `127.0.0.1`, and serves - // `host.containers.internal` / `host.docker.internal` / - // `host.openshell.internal` in its embedded DNS pointing at that - // same HostIP. The guest init script seeds /etc/hosts with the - // same mapping so the supervisor reaches the host gateway even - // when gvproxy's DNS isn't in resolv.conf. The gateway IP - // (192.168.127.1) is NOT a host-loopback proxy — it only listens - // on its own service ports (DNS:53, DHCP, HTTP API:80). - // - // That network plane is also what the sandbox supervisor's - // per-sandbox netns (veth pair + nftables, see - // `openshell-sandbox/src/sandbox/linux/netns.rs`) branches off of; - // libkrun's built-in TSI socket impersonation would not satisfy - // those kernel-level primitives. - // - // The `-listen` API socket and `-ssh-port` forwarder are both - // deliberately omitted: nothing in the driver enqueues port - // forwards on the API any more, and the host-side SSH listener is - // dead plumbing. - let gvproxy_guard = { - let gvproxy_binary = runtime_dir.join("gvproxy"); - if !gvproxy_binary.is_file() { - return Err(format!( - "missing runtime file: {}", - gvproxy_binary.display() - )); - } - - let sock_base = gvproxy_socket_base(&config.overlay_disk)?; - let net_sock = sock_base.with_extension("v"); - let _ = std::fs::remove_file(&net_sock); - let _ = std::fs::remove_file(sock_base.with_extension("v-krun.sock")); - - let run_dir = config.overlay_disk.parent().unwrap_or(&config.overlay_disk); - let gvproxy_log = run_dir.join("gvproxy.log"); - let gvproxy_log_file = std::fs::File::create(&gvproxy_log) - .map_err(|e| format!("create gvproxy log {}: {e}", gvproxy_log.display()))?; - - #[cfg(target_os = "linux")] - let (gvproxy_net_flag, gvproxy_net_url) = - ("-listen-qemu", format!("unix://{}", net_sock.display())); - #[cfg(target_os = "macos")] - let (gvproxy_net_flag, gvproxy_net_url) = ( - "-listen-vfkit", - format!("unixgram://{}", net_sock.display()), - ); - - // `-ssh-port -1` tells gvproxy to skip its default SSH forward - // (127.0.0.1:2222 → guest:22). We don't use it — all gateway - // ingress rides the supervisor-initiated relay — and leaving - // the default on would bind a host-side TCP listener per - // sandbox, racing concurrent sandboxes for port 2222 and - // surfacing a misleading "sshd is reachable" endpoint. See - // https://github.com/containers/gvisor-tap-vsock `cmd/gvproxy/main.go` - // (`getForwardsMap` returns an empty map when `sshPort == -1`). - let mut gvproxy_cmd = StdCommand::new(&gvproxy_binary); - gvproxy_cmd - .arg(gvproxy_net_flag) - .arg(&gvproxy_net_url) - .arg("-ssh-port") - .arg("-1") - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(gvproxy_log_file); - - // On Linux the kernel will SIGKILL gvproxy the moment this - // launcher dies (or is SIGKILLed). `pre_exec` runs in the child - // between fork and execve, so the PR_SET_PDEATHSIG flag is - // inherited across execve and applies to gvproxy proper. On - // macOS/BSDs there is no equivalent; we fall back to killing - // gvproxy explicitly from the launcher's procguard cleanup - // callback (see `run_vm` above) and SIGTERM handler - // (see `install_signal_forwarding` below). - #[cfg(target_os = "linux")] - { - use nix::sys::signal::Signal; - use std::os::unix::process::CommandExt as _; - unsafe { - gvproxy_cmd.pre_exec(|| { - nix::sys::prctl::set_pdeathsig(Signal::SIGKILL) - .map_err(|err| std::io::Error::other(format!("pdeathsig: {err}"))) - }); - } - } - - let child = gvproxy_cmd - .spawn() - .map_err(|e| format!("failed to start gvproxy {}: {e}", gvproxy_binary.display()))?; - // The procguard cleanup reads GVPROXY_PID atomically. Storing it - // here makes the callback able to SIGTERM gvproxy if the driver - // dies from this moment onward. - GVPROXY_PID.store(child.id().cast_signed(), Ordering::Relaxed); - - wait_for_path(&net_sock, Duration::from_secs(5), "gvproxy data socket")?; - - vm.disable_implicit_vsock()?; - vm.add_vsock(0)?; - - let mac: [u8; 6] = [0x5a, 0x94, 0xef, 0xe4, 0x0c, 0xee]; - - #[cfg(target_os = "linux")] - vm.add_net_unixstream(&net_sock, &mac, COMPAT_NET_FEATURES)?; - #[cfg(target_os = "macos")] - { - const NET_FLAG_VFKIT: u32 = 1 << 0; - vm.add_net_unixgram(&net_sock, &mac, COMPAT_NET_FEATURES, NET_FLAG_VFKIT)?; - } - - Some(GvproxyGuard::new(child)) - }; + vm.disable_implicit_vsock()?; + vm.add_vsock(0)?; + if let Some(port_map) = &config.vsock_port_map { + let _ = std::fs::remove_file(&port_map.host_socket); + vm.add_vsock_port(port_map)?; + } vm.set_console_output(&config.console_output)?; @@ -846,8 +350,7 @@ fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { // fires). Arm procguard so this fork is SIGKILLed if the // parent launcher dies abruptly. On Linux this uses // `PR_SET_PDEATHSIG`; on macOS this spawns a kqueue - // NOTE_EXIT watcher thread. Either way it closes the same - // leak gvproxy does above. + // NOTE_EXIT watcher thread. // // We also SIGKILL ourselves if arming fails — there's no // safe way to continue if we can't guarantee cleanup. @@ -864,9 +367,6 @@ fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { let status = wait_for_child(pid)?; CHILD_PID.store(0, Ordering::Relaxed); - cleanup_gvproxy(gvproxy_guard); - GVPROXY_PID.store(0, Ordering::Relaxed); - if libc::WIFEXITED(status) { match libc::WEXITSTATUS(status) { 0 => Ok(()), @@ -1118,50 +618,18 @@ impl VmContext { ) } - #[cfg(target_os = "macos")] - fn add_net_unixgram( - &self, - socket_path: &Path, - mac: &[u8; 6], - features: u32, - flags: u32, - ) -> Result<(), String> { - let sock_c = path_to_cstring(socket_path)?; - check( - unsafe { - (self.krun.krun_add_net_unixgram)( - self.ctx_id, - sock_c.as_ptr(), - -1, - mac.as_ptr(), - features, - flags, - ) - }, - "krun_add_net_unixgram", - ) - } - - #[allow(dead_code)] // Used on Linux when gvproxy runs in qemu/unixstream mode. - fn add_net_unixstream( - &self, - socket_path: &Path, - mac: &[u8; 6], - features: u32, - ) -> Result<(), String> { - let sock_c = path_to_cstring(socket_path)?; + fn add_vsock_port(&self, port_map: &VsockPortMap) -> Result<(), String> { + let socket_c = path_to_cstring(&port_map.host_socket)?; check( unsafe { - (self.krun.krun_add_net_unixstream)( + (self.krun.krun_add_vsock_port2)( self.ctx_id, - sock_c.as_ptr(), - -1, - mac.as_ptr(), - features, - 0, + port_map.guest_port, + socket_c.as_ptr(), + port_map.host_initiated, ) }, - "krun_add_net_unixstream", + "krun_add_vsock_port2", ) } @@ -1210,109 +678,6 @@ impl Drop for VmContext { } } -struct GvproxyGuard { - child: Option, -} - -impl GvproxyGuard { - fn new(child: StdChild) -> Self { - Self { child: Some(child) } - } - - fn disarm(&mut self) -> Option { - self.child.take() - } -} - -impl Drop for GvproxyGuard { - fn drop(&mut self) { - if let Some(mut child) = self.child.take() { - let _ = child.kill(); - let _ = child.wait(); - } - } -} - -fn wait_for_path(path: &Path, timeout: Duration, label: &str) -> Result<(), String> { - let deadline = Instant::now() + timeout; - let mut interval = Duration::from_millis(5); - while !path.exists() { - if Instant::now() >= deadline { - return Err(format!( - "{label} did not appear within {:.1}s: {}", - timeout.as_secs_f64(), - path.display() - )); - } - std::thread::sleep(interval); - interval = (interval * 2).min(Duration::from_millis(200)); - } - Ok(()) -} - -fn hash_path_id(path: &Path) -> String { - let mut hash: u64 = 0xcbf2_9ce4_8422_2325; - for byte in path.to_string_lossy().as_bytes() { - hash ^= u64::from(*byte); - hash = hash.wrapping_mul(0x0100_0000_01b3); - } - format!("{:012x}", hash & 0x0000_ffff_ffff_ffff) -} - -fn secure_socket_base(subdir: &str) -> Result { - let base = std::env::var_os("XDG_RUNTIME_DIR").map_or_else( - || { - let fallback = PathBuf::from("/tmp"); - if fallback.is_dir() { - fallback - } else { - std::env::temp_dir() - } - }, - PathBuf::from, - ); - let dir = base.join(subdir); - - if dir.exists() { - let meta = dir - .symlink_metadata() - .map_err(|e| format!("lstat {}: {e}", dir.display()))?; - if meta.file_type().is_symlink() { - return Err(format!( - "socket directory {} is a symlink; refusing to use it", - dir.display() - )); - } - #[cfg(unix)] - { - use std::os::unix::fs::MetadataExt as _; - let uid = unsafe { libc::getuid() }; - if meta.uid() != uid { - return Err(format!( - "socket directory {} is owned by uid {} but we are uid {}", - dir.display(), - meta.uid(), - uid - )); - } - } - } else { - std::fs::create_dir_all(&dir) - .map_err(|e| format!("create socket dir {}: {e}", dir.display()))?; - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt as _; - let _ = std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)); - } - } - - Ok(dir) -} - -fn gvproxy_socket_base(overlay_disk: &Path) -> Result { - Ok(secure_socket_base("osd-gv")?.join(hash_path_id(overlay_disk))) -} - fn install_signal_forwarding(pid: i32) { unsafe { libc::signal( @@ -1327,12 +692,7 @@ fn install_signal_forwarding(pid: i32) { CHILD_PID.store(pid, Ordering::Relaxed); } -/// Async-signal-safe handler that forwards SIGTERM to every process we -/// own: the libkrun VM worker and the gvproxy helper. We cannot rely on -/// Rust destructors (`GvproxyGuard::drop`, `ManagedDriverProcess::drop`) -/// running on signal-driven exit, so we explicitly deliver the signal -/// here. The `wait_for_child` loop reaps libkrun and `cleanup_gvproxy` -/// reaps gvproxy before `run_vm` returns. +/// Async-signal-safe handler that forwards SIGTERM to the VM worker. /// /// Only async-signal-safe libc calls are used — `kill(2)` is listed in /// POSIX.1-2017 as async-signal-safe, atomic loads are lock-free on the @@ -1344,13 +704,6 @@ extern "C" fn forward_signal(_sig: libc::c_int) { libc::kill(vm_pid, libc::SIGTERM); } } - let gv_pid = GVPROXY_PID.load(Ordering::Relaxed); - if gv_pid > 0 { - // gvproxy handles SIGTERM cleanly; no need for SIGKILL. - unsafe { - libc::kill(gv_pid, libc::SIGTERM); - } - } } fn wait_for_child(pid: i32) -> Result { @@ -1365,15 +718,6 @@ fn wait_for_child(pid: i32) -> Result { Ok(status) } -fn cleanup_gvproxy(mut guard: Option) { - if let Some(mut guard) = guard.take() - && let Some(mut child) = guard.disarm() - { - let _ = child.kill(); - let _ = child.wait(); - } -} - fn check(ret: i32, func: &'static str) -> Result<(), String> { if ret < 0 { Err(format!("{func} failed with error code {ret}")) @@ -1431,23 +775,17 @@ mod tests { console_output: PathBuf::from("/console.log"), backend: VmBackend::Qemu, gpu_bdf: Some("0000:01:00.0".to_string()), - tap_device: Some("vmtap-test".to_string()), - guest_ip: Some("10.0.128.2".to_string()), - host_ip: Some("10.0.128.1".to_string()), vsock_cid: Some(4), - guest_mac: Some("02:00:00:00:00:01".to_string()), - gateway_port: Some(8080), + vsock_port_map: None, } } #[test] - fn qemu_guest_env_vars_include_driver_runtime_metadata() { - let env = qemu_guest_env_vars(&qemu_config(), Some("1.1.1.1".to_string())); + fn qemu_guest_env_vars_omit_network_metadata() { + let env = qemu_guest_env_vars(&qemu_config()); assert!(env.contains(&"OPENSHELL_ENDPOINT=http://10.0.128.1:8080".to_string())); - assert!(env.contains(&"VM_NET_IP=10.0.128.2".to_string())); - assert!(env.contains(&"VM_NET_GW=10.0.128.1".to_string())); - assert!(env.contains(&"VM_NET_DNS=1.1.1.1".to_string())); + assert!(!env.iter().any(|value| value.starts_with("VM_NET_"))); assert!(env.contains(&"GPU_ENABLED=true".to_string())); } @@ -1490,13 +828,13 @@ mod tests { } #[test] - fn kernel_cmdline_keeps_guest_init_metadata_out_of_proc_cmdline() { + fn kernel_cmdline_has_no_guest_network_configuration() { let cmdline = build_kernel_cmdline(&qemu_config()); assert!(cmdline.contains("root=/dev/vda")); assert!(cmdline.contains("rootfstype=ext4")); assert!(cmdline.contains(" ro")); - assert!(cmdline.contains("ip=10.0.128.2::10.0.128.1:255.255.255.252:sandbox::off")); + assert!(!cmdline.contains("ip=")); assert!(cmdline.contains("firmware_class.path=/lib/firmware")); assert!(!cmdline.contains("VM_NET_IP=")); assert!(!cmdline.contains("VM_NET_GW=")); @@ -1524,6 +862,11 @@ mod tests { assert!(args.contains(&"virtio-blk-pci,drive=overlay".to_string())); } + #[test] + fn qemu_explicitly_disables_implicit_network_devices() { + assert_eq!(qemu_network_args(), ["-nic", "none"]); + } + #[test] fn qemu_disk_args_attach_prepared_image_readonly_when_present() { let mut config = qemu_config(); @@ -1536,29 +879,4 @@ mod tests { )); assert!(args.contains(&"virtio-blk-pci,drive=image".to_string())); } - - #[test] - fn gvproxy_socket_base_is_per_sandbox_overlay_path() { - let first = - gvproxy_socket_base(Path::new("/tmp/openshell-vm/sandboxes/first/overlay.ext4")) - .expect("first socket base"); - let second = - gvproxy_socket_base(Path::new("/tmp/openshell-vm/sandboxes/second/overlay.ext4")) - .expect("second socket base"); - - assert_ne!(first, second); - } - - #[test] - fn tap_subnet_from_host_ip_calculates_slash30_base() { - assert_eq!(tap_subnet_from_host_ip("10.0.128.1"), "10.0.128.0/30"); - assert_eq!(tap_subnet_from_host_ip("10.0.128.2"), "10.0.128.0/30"); - assert_eq!(tap_subnet_from_host_ip("10.0.128.5"), "10.0.128.4/30"); - } - - #[test] - fn tap_subnet_from_host_ip_handles_invalid_ip() { - let result = tap_subnet_from_host_ip("not-an-ip"); - assert_eq!(result, "not-an-ip/30"); - } } diff --git a/crates/openshell-isolation-interface/Cargo.toml b/crates/openshell-isolation-interface/Cargo.toml new file mode 100644 index 0000000000..94220dda73 --- /dev/null +++ b/crates/openshell-isolation-interface/Cargo.toml @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-isolation-interface" +description = "OpenShell Isolation Backend interface (RFC 0012)" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +openshell-core = { path = "../openshell-core", default-features = false } +async-trait = "0.1" +serde = { workspace = true } +tokio = { workspace = true } + +[target.'cfg(unix)'.dependencies] +libc = "0.2" + +[target.'cfg(target_os = "linux")'.dependencies] +rustix = { workspace = true, features = ["fs", "process"] } + +[dev-dependencies] +tokio = { workspace = true } + +[lints] +workspace = true diff --git a/crates/openshell-isolation-interface/src/contract.rs b/crates/openshell-isolation-interface/src/contract.rs new file mode 100644 index 0000000000..f671dd99e9 --- /dev/null +++ b/crates/openshell-isolation-interface/src/contract.rs @@ -0,0 +1,1031 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Runtime-selectable Isolation Backend contract (RFC 0012). +//! +//! This module is the object-safe, runtime-selectable contract the supervisor +//! role drives. A backend registers an [`IsolationBackend`] under a +//! `backend_name`; the supervisor resolves it from a [`BackendRegistry`] +//! against the admitted backend name and advances the boundary through a fixed +//! chain of boxed states: +//! +//! ```text +//! attach backend descriptor + sandbox context -> Bound -> confirm -> Ready +//! -> start_agent -> Running +//! ``` +//! +//! Each transition consumes the prior state by value (`self: Box`). +//! Trusted backend implementations construct confirmation through a validating +//! constructor; the supervisor cannot obtain a ready boundary without evidence. +//! The supervisor holds no `match`/downcast on concrete backends: the +//! registry is the only lookup by `backend_name`, and everything past it is a +//! `Box` / `Arc`. +//! +//! `attach` is atomic from the caller's perspective: it establishes and binds +//! the boundary, returns `Bound`, or fails closed. It never binds a resource +//! already bound to an active boundary. Binary identity travels on every +//! [`PendingTcpOpen`], resolved for that exact socket and process +//! generation; an unresolved identity denies the open. +//! +//! The contract is transport-neutral. Compute drivers keep runtime placement +//! and coordination details behind these interfaces. + +use std::collections::{BTreeMap, HashMap}; +use std::fmt; +use std::net::{IpAddr, SocketAddr}; +use std::path::PathBuf; +use std::str::FromStr; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use async_trait::async_trait; +use serde::{Deserialize, Serialize}; +use tokio::io::{AsyncRead, AsyncWrite}; +use tokio::sync::oneshot; + +pub use openshell_core::SandboxSessionId; +pub use openshell_core::policy::SandboxPolicy; + +// ============================================================================ +// Errors +// ============================================================================ + +/// Classified failures at the common contract boundary. +/// +/// An error never advances the lifecycle or authorizes an operation. +#[derive(Debug)] +pub enum BackendError { + /// Descriptor missing, malformed, unsupported, or mismatched against admission. + Descriptor(String), + /// No backend registered for the resolved `backend_name`. + NotRegistered(String), + /// Authenticated attachment rejection (incompatible or already-bound resource). + Denied(String), + /// Boundary temporarily unavailable. + Unavailable(String), + /// The selected backend does not implement an optional contract operation. + Unsupported(String), + /// Attachment-phase failure (establishment or mediation bring-up). + Attach(String), + /// Readiness confirmation failed (do not start workload code). + Confirm(String), + /// Process start or exec failure. + Process(String), + /// Abnormal boundary or workload loss, or an operation against an inactive + /// boundary. + Terminated(String), +} + +/// Coarse, machine-readable classification of a [`BackendError`] for supervisor +/// status mapping. The error's variant and message carry the structured context +/// (which operation failed). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BackendErrorKind { + /// Descriptor or backend mismatch. + Invalid, + /// Authenticated attachment rejection. + Denied, + /// Transient inability to serve an operation. + Unavailable, + /// The selected backend does not implement the requested optional operation. + Unsupported, + /// Attachment, confirmation, start, or runtime operation failure. + Failed, + /// Abnormal boundary/workload loss, or an operation against an inactive + /// boundary. + Terminated, +} + +impl BackendError { + /// The machine-readable kind for this error. + #[must_use] + pub fn kind(&self) -> BackendErrorKind { + match self { + Self::Descriptor(_) | Self::NotRegistered(_) => BackendErrorKind::Invalid, + Self::Denied(_) => BackendErrorKind::Denied, + Self::Unavailable(_) => BackendErrorKind::Unavailable, + Self::Unsupported(_) => BackendErrorKind::Unsupported, + Self::Attach(_) | Self::Confirm(_) | Self::Process(_) => BackendErrorKind::Failed, + Self::Terminated(_) => BackendErrorKind::Terminated, + } + } +} + +impl fmt::Display for BackendError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Descriptor(m) => write!(f, "descriptor error: {m}"), + Self::NotRegistered(m) => write!(f, "backend not registered: {m}"), + Self::Denied(m) => write!(f, "attachment denied: {m}"), + Self::Unavailable(m) => write!(f, "boundary unavailable: {m}"), + Self::Unsupported(m) => write!(f, "operation unsupported: {m}"), + Self::Attach(m) => write!(f, "attachment failed: {m}"), + Self::Confirm(m) => write!(f, "confirmation failed: {m}"), + Self::Process(m) => write!(f, "process error: {m}"), + Self::Terminated(m) => write!(f, "boundary terminated: {m}"), + } + } +} + +impl std::error::Error for BackendError {} + +/// Why an identity resolution failed. Resolution failure fails closed: the +/// mediation service denies and audits the connection; it never authorizes. +#[derive(Debug, Clone)] +pub enum ResolveError { + /// No process owns the connection (stale or unknown attribution). + NotFound, + /// Resolution attempted but could not produce trustworthy identity. + Failed(String), +} + +impl fmt::Display for ResolveError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::NotFound => write!(f, "connection owner not found"), + Self::Failed(m) => write!(f, "identity resolution failed: {m}"), + } + } +} + +impl std::error::Error for ResolveError {} + +// ============================================================================ +// Descriptor and registry +// ============================================================================ + +/// The common isolation backend descriptor envelope. +/// +/// The compute driver supplies one for the selected isolation backend. The opaque +/// payload identifies an existing resource or carries the trusted prepared +/// inputs the backend needs to establish one during `attach`; its protection +/// and resource lifecycle remain owned by the compute driver. +#[derive(Debug, Clone)] +pub struct BackendDescriptor { + /// The backend the supervisor must instantiate. + pub backend_name: String, + /// Backend-specific attachment data. + pub payload: Vec, +} + +/// A descriptor whose common envelope has passed registry verification. +/// +/// Minted only by [`BackendRegistry::resolve`]; no public constructor, so an +/// unverified descriptor cannot reach a backend. The type does not imply that +/// the opaque payload has been validated: the backend validates the payload and +/// atomically binds it to the sandbox context during `attach`. +pub struct VerifiedBackendDescriptor { + descriptor: BackendDescriptor, +} + +impl VerifiedBackendDescriptor { + /// The verified backend name. + #[must_use] + pub fn backend_name(&self) -> &str { + &self.descriptor.backend_name + } + /// The backend-specific payload (validated by the backend at `attach`). + #[must_use] + pub fn payload(&self) -> &[u8] { + &self.descriptor.payload + } +} + +/// Exact non-root identity selected before the immutable workload is created. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ResolvedWorkloadIdentity { + /// Effective and real user ID used by sandbox and all workload children. + pub uid: u32, + /// Primary group ID used by sandbox and all workload children. + pub gid: u32, + /// Sorted, unique supplementary groups inherited unchanged by children. + pub supplementary_gids: Vec, + /// Driver-defined resolution source (`policy`, `template`, or `image`). + pub source: String, + /// Digest of the immutable image/rootfs/config used for resolution. + pub resource_digest: String, +} + +impl ResolvedWorkloadIdentity { + /// Validate and construct a final workload identity. + pub fn new( + uid: u32, + gid: u32, + mut supplementary_gids: Vec, + source: String, + resource_digest: String, + ) -> Result { + if uid == 0 || gid == 0 || supplementary_gids.contains(&0) { + return Err(BackendError::Descriptor( + "workload identity must not contain UID or GID zero".to_string(), + )); + } + if source.trim().is_empty() || resource_digest.trim().is_empty() { + return Err(BackendError::Descriptor( + "workload identity source and resource digest are required".to_string(), + )); + } + supplementary_gids.retain(|supplementary_gid| *supplementary_gid != gid); + supplementary_gids.sort_unstable(); + supplementary_gids.dedup(); + Ok(Self { + uid, + gid, + supplementary_gids, + source, + resource_digest, + }) + } +} + +/// The trusted sandbox context, constructed by trusted common code after the +/// control plane assigns the resource to the admitted sandbox. +/// +/// Carries the admitted launch-time policy. Approved network-policy revisions +/// are made effective by supervisor-owned network mediation, outside the +/// backend lifecycle. +pub struct SandboxContext { + /// Which sandbox this is. + pub sandbox_id: String, + /// Which create or start-from-stopped launch this attachment belongs to. + /// + /// Retries of one durable launch reuse this identity. A later launch gets + /// a new identity even when the compute platform reuses its outer resource. + pub session_id: SandboxSessionId, + /// The admitted launch-time policy. + pub policy: SandboxPolicy, + /// The admitted agent workload. + pub agent: AgentSpec, + /// Immutable identity already applied by the driver to sandbox and agent. + pub identity: ResolvedWorkloadIdentity, +} + +/// The agent workload to run inside the boundary. +use crate::AgentSpec; + +/// Maps backend name to its implementation. This is the only lookup by name; +/// supervisor lifecycle never branches on a concrete backend, and resolution +/// never falls back to another backend. +#[derive(Default)] +pub struct BackendRegistry { + backends: HashMap>, +} + +impl BackendRegistry { + /// An empty registry. + #[must_use] + pub fn new() -> Self { + Self { + backends: HashMap::new(), + } + } + + /// Register a backend. Rejects a duplicate name. + /// + /// # Errors + /// + /// Returns [`BackendError::Descriptor`] for a duplicate `backend_name`. + pub fn register(&mut self, backend: Arc) -> Result<(), BackendError> { + let name = backend.backend_name().to_string(); + if self.backends.contains_key(&name) { + return Err(BackendError::Descriptor(format!( + "duplicate backend name {name:?}" + ))); + } + self.backends.insert(name, backend); + Ok(()) + } + + /// Verify the descriptor's common envelope against the admitted backend name + /// and resolve its backend. Fails closed and never falls back: + /// + /// - the descriptor's `backend_name` must equal the admitted name; + /// - a backend must be registered under that name. + /// + /// # Errors + /// + /// Returns [`BackendError::Descriptor`] for an admission mismatch, and + /// [`BackendError::NotRegistered`] when no backend is + /// registered for the admitted name. + pub fn resolve( + &self, + descriptor: BackendDescriptor, + admitted_backend_name: &str, + ) -> Result<(Arc, VerifiedBackendDescriptor), BackendError> { + if descriptor.backend_name != admitted_backend_name { + return Err(BackendError::Descriptor(format!( + "descriptor backend {:?} does not match admitted backend {admitted_backend_name:?}", + descriptor.backend_name + ))); + } + let backend = self + .backends + .get(&descriptor.backend_name) + .ok_or_else(|| BackendError::NotRegistered(descriptor.backend_name.clone()))? + .clone(); + if backend.backend_name() != descriptor.backend_name { + return Err(BackendError::Descriptor(format!( + "registry returned backend {:?} for name {:?}", + backend.backend_name(), + descriptor.backend_name + ))); + } + Ok((backend, VerifiedBackendDescriptor { descriptor })) + } +} + +/// Establishes and operates boundaries for one admitted backend implementation. +#[async_trait] +pub trait IsolationBackend: Send + Sync { + /// The stable registered backend name. + fn backend_name(&self) -> &str; + + /// Validate the opaque payload, establish any boundary-local resources, + /// and atomically bind them to the trusted sandbox context: returns `Bound` + /// or fails closed. Never binds a resource already bound to an active + /// boundary. The authenticated runtime session must match + /// `sandbox.session_id`; a session from an earlier launch is rejected. + /// Durable resource lifecycle remains owned by the compute driver or + /// external orchestrator that supplied the descriptor. + async fn attach( + &self, + descriptor: VerifiedBackendDescriptor, + sandbox: SandboxContext, + ) -> Result, BackendError>; +} + +// ============================================================================ +// Lifecycle states +// ============================================================================ + +/// Bound: the backend descriptor and trusted sandbox context are bound to the +/// same resource, and the mediation source is available. No untrusted workload +/// code is running. +#[async_trait] +pub trait BoundBoundary: Send { + /// The mediation service's backend-neutral source of workload network + /// requests. TCP and DNS remain typed operations so consumers cannot mix + /// their framing, decisions, or response semantics. + /// Retained by the supervisor before consuming `Bound`. + fn network_mediation_source(&self) -> Arc; + + /// Trusted host-side dial target for the well-known host-gateway aliases. + /// + /// Backends return this when the mediation service runs outside the + /// workload boundary and therefore cannot use the boundary's resolver + /// view. The supervisor preserves the original hostname for policy, HTTP, + /// and TLS while dialing this backend-provided address. Returning `None` + /// leaves host-gateway discovery to the supervisor's local environment. + fn host_gateway_ip(&self) -> Option { + None + } + + /// Confirm standing enforcement and return measured sandbox evidence. + /// Confirmation fails closed and does not execute untrusted workload code. + async fn confirm(self: Box) -> Result; +} + +/// Capability masks measured from `/proc//status`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub struct CapabilityEvidence { + pub inheritable: u64, + pub permitted: u64, + pub effective: u64, + pub bounding: u64, + pub ambient: u64, +} + +impl CapabilityEvidence { + /// True only when every Linux capability set is empty. + #[must_use] + pub const fn is_empty(self) -> bool { + self.inheritable == 0 + && self.permitted == 0 + && self.effective == 0 + && self.bounding == 0 + && self.ambient == 0 + } +} + +/// Active seccomp notification and socket-broker evidence. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[allow( + clippy::struct_excessive_bools, + reason = "each independently measured kernel operation is reported explicitly" +)] +pub struct SeccompEvidence { + pub new_listener: bool, + pub notification_round_trip: bool, + pub id_validation: bool, + pub addfd_send: bool, + pub retained_socket_operation: bool, + pub proc_fd_identity: bool, + pub task_memory_read: bool, + pub task_memory_write: bool, + pub cancellation: bool, +} + +/// Driver-owned evidence that the mandatory outer network fence is installed. +/// +/// The sandbox cannot observe the Docker daemon, Kubernetes API, or VM device +/// model directly. Drivers therefore bind the exact fence they validated into +/// both protected bootstrap halves. The sandbox reports that value back during +/// confirmation, and the supervisor rejects any mismatch before agent launch. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "backend", rename_all = "kebab-case", deny_unknown_fields)] +pub enum DriverFenceEvidence { + Docker { + container_id: String, + network_mode: String, + unexpected_networks: Vec, + }, + Podman { + container_id: String, + network_mode: String, + unexpected_networks: Vec, + }, + Kubernetes { + network_policy_uid: String, + network_policy_resource_version: String, + ingress_isolated: bool, + egress_isolated: bool, + egress_rule_count: u32, + }, + Vm { + generation: String, + network_device_count: u32, + }, +} + +impl DriverFenceEvidence { + #[must_use] + pub const fn driver_name(&self) -> &'static str { + match self { + Self::Docker { .. } => "docker", + Self::Podman { .. } => "podman", + Self::Kubernetes { .. } => "kubernetes", + Self::Vm { .. } => "vm", + } + } + + /// Validate the concrete outer-fence properties reported by the compute driver. + pub fn validate(&self) -> Result<(), BackendError> { + let valid = match self { + Self::Docker { + container_id, + network_mode, + unexpected_networks, + } + | Self::Podman { + container_id, + network_mode, + unexpected_networks, + } => { + !container_id.is_empty() && network_mode == "none" && unexpected_networks.is_empty() + } + Self::Kubernetes { + network_policy_uid, + network_policy_resource_version, + ingress_isolated, + egress_isolated, + egress_rule_count, + } => { + !network_policy_uid.is_empty() + && !network_policy_resource_version.is_empty() + && *ingress_isolated + && *egress_isolated + && *egress_rule_count == 0 + } + Self::Vm { + generation, + network_device_count, + } => !generation.is_empty() && *network_device_count == 0, + }; + if valid { + Ok(()) + } else { + Err(BackendError::Confirm(format!( + "{} driver fence evidence is incomplete", + self.driver_name() + ))) + } + } +} + +/// Measured sandbox-owned evidence produced before agent launch. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[allow( + clippy::struct_excessive_bools, + reason = "confirmation preserves independently measured security results" +)] +pub struct SandboxConfirmEvidence { + pub generation: String, + pub identity: ResolvedWorkloadIdentity, + pub capabilities: CapabilityEvidence, + pub no_new_privileges: bool, + pub sandbox_dumpable: bool, + pub child_dumpable: bool, + pub core_limit_zero: bool, + pub native_architecture: String, + pub kernel_release: String, + pub seccomp: SeccompEvidence, + pub landlock_abi: u32, + pub landlock_allow_deny: bool, + pub udp_dns_round_trip: bool, + pub tcp_dns_round_trip: bool, + pub tcp_allow_round_trip: bool, + pub tcp_deny_round_trip: bool, + pub authenticated_supervisor: bool, + pub session_id: SandboxSessionId, + pub driver_fence: DriverFenceEvidence, + /// The driver-owned containment primitive terminates the workload when its + /// Sandbox Runtime exits. + pub runtime_exit_terminates_workload: bool, + pub resource_claims: BTreeMap, +} + +impl SandboxConfirmEvidence { + /// Validate the security-critical evidence required before launch. + pub fn validate(&self, expected: &ResolvedWorkloadIdentity) -> Result<(), BackendError> { + self.driver_fence.validate()?; + let complete = &self.identity == expected + && self.capabilities.is_empty() + && self.no_new_privileges + && !self.sandbox_dumpable + && self.child_dumpable + && self.core_limit_zero + && self.seccomp.new_listener + && self.seccomp.notification_round_trip + && self.seccomp.id_validation + && self.seccomp.addfd_send + && self.seccomp.retained_socket_operation + && self.seccomp.proc_fd_identity + && self.seccomp.task_memory_read + && self.seccomp.task_memory_write + && self.seccomp.cancellation + && self.landlock_abi >= 3 + && self.landlock_allow_deny + && self.udp_dns_round_trip + && self.tcp_dns_round_trip + && self.tcp_allow_round_trip + && self.tcp_deny_round_trip + && self.authenticated_supervisor + && self.runtime_exit_terminates_workload + && !self.generation.is_empty(); + if complete { + Ok(()) + } else { + Err(BackendError::Confirm( + "sandbox confirmation evidence is incomplete or mismatched".to_string(), + )) + } + } +} + +/// Ready boundary paired with the evidence measured by `confirm`. +pub struct ConfirmedBoundary { + boundary: Box, + evidence: SandboxConfirmEvidence, +} + +impl ConfirmedBoundary { + /// Construct confirmation after checking measured evidence against the + /// immutable identity admitted at attach time. + /// + /// Backend implementations are trusted to collect this evidence and bind + /// it to their resource. This constructor enforces the common requirements + /// without requiring those implementations to live in the interface crate. + /// + /// # Errors + /// + /// Returns an error if evidence is incomplete or the identity does not match. + pub fn try_new( + boundary: Box, + evidence: SandboxConfirmEvidence, + expected: &ResolvedWorkloadIdentity, + ) -> Result { + evidence.validate(expected)?; + Ok(Self { boundary, evidence }) + } + + /// Return the measured evidence carried by this confirmed state. + pub fn evidence(&self) -> &SandboxConfirmEvidence { + &self.evidence + } + + /// Consume confirmation and advance to the sole launch-capable state. + pub fn into_boundary(self) -> Box { + self.boundary + } +} + +/// Ready: standing enforcement is confirmed, and the backend is prepared to +/// ensure the admitted launch-time controls are in force +/// before untrusted execution. Only agent activation is possible from here. +#[async_trait] +pub trait ReadyBoundary: Send { + /// Make the admitted agent runnable behind the boundary and return its + /// handle. `start_agent` is the sole operation that may make the admitted + /// agent runnable, and it fails closed if any `Ready` condition no longer + /// holds. Whether the backend creates the agent process or releases a held, + /// driver-provisioned execution object is backend-specific; every + /// applicable launch-time control is in force before the first untrusted + /// instruction. + async fn start_agent(self: Box) -> Result, BackendError>; +} + +/// Running: the agent is runnable behind the boundary and the returned agent +/// handle represents the admitted agent process. Exec and forwarding are available. +/// +/// All interface accessors return owned `Arc`s so a consumer can retain them +/// past any later state consumption. +#[async_trait] +pub trait RunningBoundary: Send + Sync { + /// The admitted agent process handle. + fn agent(&self) -> Arc; + /// The in-boundary exec interface. + fn exec(&self) -> Arc; + /// The loopback connection interface used by port forwarding and service exposure. + fn loopback_connector(&self) -> Arc; + /// Permanently terminate the boundary's owned process tree and return only + /// after the backend has acknowledged terminal state. A driver may use + /// destruction of the outer runtime as fallback proof when this operation + /// cannot complete. + async fn terminate(&self) -> Result<(), BackendError>; +} + +// ============================================================================ +// Process and exec +// ============================================================================ + +/// Placement-neutral terminal status of a boundary process. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BoundaryExitStatus { + /// Exited with a code. + Exited(i32), + /// Killed by a signal. + Signaled(i32), +} + +/// Placement-neutral signal to deliver to a boundary process. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BoundarySignal { + /// Graceful terminate. + Term, + /// Forceful kill. + Kill, + /// Interrupt. + Int, + /// Hangup. + Hup, +} + +/// A process running inside the boundary. `wait` returns one stable status +/// however many times it is called; a local PID is never the process handle. +#[async_trait] +pub trait BoundaryProcess: Send + Sync { + /// Attach to the admitted process's retained standard I/O. The boundary + /// remains the process owner and may permit only one control attachment. + async fn attach(&self) -> Result { + Err(BackendError::Unsupported( + "process attachment is not supported".to_string(), + )) + } + /// Await terminal status (stable across repeated calls). + async fn wait(&self) -> Result; + /// Deliver a signal to the process or its group. + async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError>; + /// Terminate the process and its backend-owned process group. + async fn terminate(&self) -> Result<(), BackendError>; +} + +/// A boxed async writer into a boundary process's stdin. +pub type BoundaryInput = Box; +/// A boxed async reader from a boundary process's stdout or stderr. +pub type BoundaryOutput = Box; + +/// A control-side attachment to the admitted process's retained I/O. +pub struct ProcessAttachment { + /// Stdin writer. + pub stdin: BoundaryInput, + /// Stdout reader, or the PTY-merged output stream. + pub stdout: BoundaryOutput, + /// Stderr reader, distinct from stdout for non-PTY processes. + pub stderr: Option, + /// PTY control, present when the admitted process owns a terminal. + pub terminal: Option>, +} + +/// A PTY attached to an exec session. +#[async_trait] +pub trait BoundaryTerminal: Send + Sync { + /// Resize the terminal. + async fn resize(&self, cols: u16, rows: u16) -> Result<(), BackendError>; +} + +/// An owned exec session: the process handle plus its stdio and optional PTY. +/// Owning the process keeps it alive after `exec` returns. +pub struct ExecSession { + /// The spawned process. + pub process: Arc, + /// Stdin writer, if not a PTY-merged stream. + pub stdin: Option, + /// Stdout reader. + pub stdout: BoundaryOutput, + /// Stderr reader, distinct from stdout for non-PTY exec. + pub stderr: Option, + /// PTY control, present when a terminal was requested. + pub terminal: Option>, +} + +/// What to run inside the boundary via [`BoundaryExec`]. +#[derive(Debug, Clone)] +pub struct ExecSpec { + /// Program to run. + pub program: String, + /// Program arguments. + pub args: Vec, + /// Extra environment over the boundary's base. + pub env: Vec<(String, String)>, + /// Working directory, if any. + pub workdir: Option, + /// Whether to allocate a PTY. + pub pty: bool, +} + +/// In-boundary process entry, consumed by the SSH server and supervisor session. +/// +/// Like `start_agent`, every exec ensures the applicable launch-time controls +/// are in force before the new process executes its first untrusted instruction +/// and preserves the provisioned execution environment. +#[async_trait] +pub trait BoundaryExec: Send + Sync { + /// Spawn `spec` inside the boundary, returning an owned session. + async fn exec(&self, spec: ExecSpec) -> Result; +} + +// ============================================================================ +// Port forward +// ============================================================================ + +/// A loopback-only target inside the boundary, validated at construction. +#[derive(Debug, Clone)] +pub struct LoopbackTarget { + host: IpAddr, + port: u16, +} + +impl LoopbackTarget { + /// Build a loopback target, rejecting any non-loopback host. + /// + /// # Errors + /// + /// Returns [`BackendError::Process`] when `host` is not a loopback address. + pub fn new(host: IpAddr, port: u16) -> Result { + if !host.is_loopback() { + return Err(BackendError::Process(format!( + "port-forward target {host} is not loopback" + ))); + } + Ok(Self { host, port }) + } + /// The loopback host. + #[must_use] + pub fn host(&self) -> IpAddr { + self.host + } + /// The target port. + #[must_use] + pub fn port(&self) -> u16 { + self.port + } +} + +/// A bidirectional byte stream into the boundary. +pub trait DuplexStream: AsyncRead + AsyncWrite + Send + Unpin {} +impl DuplexStream for T {} + +/// An open connection into a boundary loopback target. +pub type BoundaryDuplexStream = Box; + +/// Protected connector to services listening inside the boundary. +/// +/// Higher layers use this primitive for both end-user port forwarding and +/// service exposure. Authentication, public listeners, routing, and exposure +/// lifecycle remain outside the isolation backend. +#[async_trait] +pub trait BoundaryLoopbackConnector: Send + Sync { + /// Connect to `target` inside the boundary. + async fn connect(&self, target: LoopbackTarget) -> Result; +} + +// ============================================================================ +// Mediation and binary identity +// ============================================================================ + +/// Executable identity for one accepted connection, resolved by the backend and +/// delivered on [`PendingTcpOpen`] before the mediation service evaluates +/// policy. +/// +/// A missing digest is `None`, never an empty value; policy that requires an +/// unavailable identity field cannot authorize the connection. How a backend +/// resolves identity is private to that backend; the shape and the fail-closed +/// semantics do not change. +#[derive(Debug, Clone)] +pub struct BinaryIdentity { + /// Absolute path of the executable resolved for the accepted connection. + pub binary_path: PathBuf, + /// Digest of the resolved executable object. `None` when unavailable. + pub binary_digest: Option, + /// Ancestor process binaries, nearest first. + pub ancestors: Vec, + /// Absolute script/interpreter paths drawn from the process cmdlines. + /// Diagnostic context; never authorizes. + pub cmdline_paths: Vec, +} + +/// A SHA-256 digest, kept typed so the identity field is not coupled to its +/// textual encoding or forced to repeat the algorithm in its name. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(try_from = "String", into = "String")] +pub struct Sha256Digest([u8; 32]); + +impl TryFrom for Sha256Digest { + type Error = ResolveError; + + fn try_from(value: String) -> Result { + value.parse() + } +} + +impl From for String { + fn from(value: Sha256Digest) -> Self { + value.to_string() + } +} + +impl Sha256Digest { + /// Return the raw digest bytes. + #[must_use] + pub fn as_bytes(&self) -> &[u8; 32] { + &self.0 + } +} + +impl fmt::Display for Sha256Digest { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + for byte in self.0 { + write!(f, "{byte:02x}")?; + } + Ok(()) + } +} + +impl FromStr for Sha256Digest { + type Err = ResolveError; + + fn from_str(value: &str) -> Result { + if value.len() != 64 || !value.bytes().all(|byte| byte.is_ascii_hexdigit()) { + return Err(ResolveError::Failed( + "SHA-256 digest must contain 64 hexadecimal characters".to_string(), + )); + } + let mut bytes = [0_u8; 32]; + for (index, byte) in bytes.iter_mut().enumerate() { + *byte = u8::from_str_radix(&value[index * 2..index * 2 + 2], 16).map_err(|_| { + ResolveError::Failed("SHA-256 digest contains non-hexadecimal data".to_string()) + })?; + } + Ok(Self(bytes)) + } +} + +/// Immutable socket metadata supplied with a pending external TCP open. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub struct NetworkSocketMetadata { + /// Kernel socket cookie captured for the exact open-file description. + pub socket_cookie: u64, + /// Whether the workload requested nonblocking operation. + pub nonblocking: bool, + /// Workload process generation that owns the open. + pub process_generation: u64, +} + +/// Typed supervisor decision for one pending TCP open. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum TcpOpenDecision { + /// L4 authorization and a bounded relay handler are ready. L7 policy still + /// applies to bytes after the local connection commits. + RelayReady, + /// The socket remains unchanged and connect returns this positive errno. + Denied(TcpOpenDenial), +} + +/// Placement-neutral reason why a staged TCP open was not committed. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum TcpOpenDenial { + /// The admitted network policy rejected the request. + PolicyDenied, + /// The backend could not resolve authoritative executable identity. + IdentityUnavailable, + /// The requested destination could not be validated. + InvalidDestination, + /// A bounded mediation resource was exhausted. + ResourceExhausted, + /// The mediation path became unavailable before commit. + MediationUnavailable, +} + +/// Timing captured while one mediated operation crosses the sandbox boundary. +/// +/// The durations are measured in the sandbox's monotonic clock. The supervisor +/// timestamp is local to the supervisor and is intentionally not serialized. +#[derive(Debug, Clone)] +pub struct MediationTiming { + /// Time from receiving the sandbox syscall notification to queueing it for + /// the transport. + pub sandbox_notification_to_queue: Duration, + /// Time spent waiting in the sandbox-side mediation queue. + pub sandbox_queue_wait: Duration, + /// Time at which the supervisor received the operation. + pub supervisor_received_at: Instant, +} + +impl Default for MediationTiming { + fn default() -> Self { + Self { + sandbox_notification_to_queue: Duration::ZERO, + sandbox_queue_wait: Duration::ZERO, + supervisor_received_at: Instant::now(), + } + } +} + +/// A staged workload TCP open delivered before its local relay is committed. +/// +/// An `Err` identity must be denied and audited. The supervisor owns +/// `result`; dropping it cancels the open without changing the workload socket. +pub struct PendingTcpOpen { + /// Staged byte stream whose workload side is committed only after + /// [`TcpOpenDecision::RelayReady`]. + pub stream: BoundaryDuplexStream, + /// Executable identity, resolved by the backend for this connection. + pub binary_identity: Result, + /// Original external destination captured from the blocked syscall. + pub destination: SocketAddr, + /// Socket and process identity bound to this request. + pub socket: NetworkSocketMetadata, + /// Policy generation under which the request was created. + pub policy_generation: u64, + /// Monotonic stage timing for performance diagnostics. + pub timing: MediationTiming, + /// Single-use completion channel back to the sandbox broker. + pub decision: oneshot::Sender, +} + +/// A logical per-boundary stream of workload connections, consumed by the +/// mediation service wherever that service runs. +/// +/// It may wrap a dedicated listener or a demultiplexed view over shared +/// transport; how it reaches a co-located proxy, a sidecar, or a shared +/// mediation service is backend-private. A trusted backend component associates +/// every returned request with its active boundary without relying solely on a +/// transport tuple or workload-provided identifier. TCP and DNS use separate +/// accepts so they can be consumed concurrently with independent backpressure. +/// An `Err` means that mediation lane is unusable and fails closed. +#[async_trait] +pub trait NetworkMediationSource: Send + Sync { + /// Await the next staged workload TCP open. + async fn accept_tcp(&self) -> Result; + + /// Await the next workload DNS query. + async fn accept_dns(&self) -> Result; +} + +/// DNS transport used by one workload exchange. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum DnsTransport { + /// One DNS wire datagram without a TCP length prefix. + Udp, + /// One DNS message received over a TCP resolver connection. + Tcp, +} + +/// One workload DNS request and its fail-closed response channel. +pub struct PendingDnsQuery { + /// Exactly one DNS wire message, without a DNS-over-TCP length prefix. + /// The backend removes and restores transport framing. + pub message: Vec, + /// Workload DNS transport. + pub transport: DnsTransport, + /// Identity of the process that issued the DNS request when the backend + /// can observe it authoritatively. Native socket-write adapters report a + /// resolution error because the relay cannot prove which descriptor + /// holder sent a datagram. Consumers must never treat unavailable + /// identity as a binary-policy grant. + pub binary_identity: Result, + /// Monotonic stage timing for performance diagnostics. + pub timing: MediationTiming, + /// Single-use response channel owned by the backend adapter. + pub response: oneshot::Sender, BackendError>>, +} diff --git a/crates/openshell-isolation-interface/src/lib.rs b/crates/openshell-isolation-interface/src/lib.rs new file mode 100644 index 0000000000..a7740d8a7d --- /dev/null +++ b/crates/openshell-isolation-interface/src/lib.rs @@ -0,0 +1,53 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! The `OpenShell` **Isolation Backend** runtime contract (RFC 0012). +//! +//! An isolation backend establishes and enforces a workload's isolation boundary; +//! the supervisor role drives it through one contract. The supervisor-facing +//! contract lives in [`contract`]: an object-safe, runtime-selectable backend +//! plus a fixed chain of boxed lifecycle states the supervisor advances without +//! branching on placement. Each driver places a sandbox boundary beside the +//! workload and connects it to a separate supervisor. +//! +//! The backend establishes standing enforcement before untrusted code runs and +//! ensures launch-time controls are in force before each process's first +//! untrusted instruction. It also exposes process operations and supplies +//! workload egress to supervisor-owned network mediation. +//! +//! # Ordering is a security property +//! +//! The lifecycle states run in order: attach -> Bound -> confirm -> Ready -> +//! `start_agent` -> Running. Nothing untrusted runs inside the boundary until it +//! is confirmed ready. This is enforced *by construction*: each transition +//! consumes the prior state by value. Trusted backends construct confirmation +//! through [`contract::ConfirmedBoundary::try_new`], which checks common evidence +//! before the supervisor can obtain a [`contract::ReadyBoundary`]. +//! +//! [`AgentSpec`] is shared between the workload definition the supervisor +//! submits and the [`contract::SandboxContext`] that `attach` binds to a +//! boundary. + +/// The agent workload to run inside the boundary. +/// +/// Carried by [`contract::SandboxContext`] so a backend's `start_agent` takes no +/// spec; the bound boundary already carries what runs inside it. +#[derive(Debug, Clone)] +pub struct AgentSpec { + /// Entrypoint program. + pub program: String, + /// Entrypoint arguments. + pub args: Vec, + /// Working directory for the entrypoint, if any. + pub workdir: Option, + /// Wall-clock timeout for the entrypoint in seconds (0 = no timeout). + pub timeout_secs: u64, + /// Whether the entrypoint runs interactively (inherits the parent pgrp). + pub interactive: bool, +} + +pub mod contract; + +/// Linux-only primitives shared by capability-free sandbox implementations. +#[cfg(target_os = "linux")] +pub mod linux; diff --git a/crates/openshell-isolation-interface/src/linux/child_seccomp.rs b/crates/openshell-isolation-interface/src/linux/child_seccomp.rs new file mode 100644 index 0000000000..4ba4eefafb --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/child_seccomp.rs @@ -0,0 +1,528 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Prepared seccomp self-protection for same-UID workload children. +//! +//! The program is built before `fork` and installed by the workload launcher +//! after the child has inherited the network user-notification filter. It does +//! not allocate while installing and deliberately leaves mediated networking +//! syscalls alone so the older `USER_NOTIF` action can still win. + +#![allow(unsafe_code)] + +use std::io; + +const SECCOMP_SET_MODE_FILTER: libc::c_uint = 1; +const SECCOMP_RET_KILL_PROCESS: u32 = 0x8000_0000; +const SECCOMP_RET_ERRNO: u32 = 0x0005_0000; +const SECCOMP_RET_ALLOW: u32 = 0x7fff_0000; + +const BPF_LD_W_ABS: u16 = 0x20; +const BPF_JMP_JEQ_K: u16 = 0x15; +const BPF_JMP_JSET_K: u16 = 0x45; +const BPF_RET_K: u16 = 0x06; + +const SECCOMP_DATA_NR_OFFSET: u32 = 0; +const SECCOMP_DATA_ARCH_OFFSET: u32 = 4; +const SECCOMP_DATA_ARGS_OFFSET: u32 = 16; +#[cfg(target_arch = "x86_64")] +const X32_SYSCALL_BIT: u32 = 0x4000_0000; + +const CLOSE_RANGE_UNSHARE_FLAG: u32 = 1 << 1; +const F_SETOWN_COMMAND: u32 = 8; +const F_SETSIG_COMMAND: u32 = 10; +const F_SETOWN_EX_COMMAND: u32 = 15; +const FIOSETOWN_REQUEST: u32 = 0x8901; +const SIOCSPGRP_REQUEST: u32 = 0x8902; +const CLONE_NAMESPACE_FLAGS: u32 = (libc::CLONE_NEWCGROUP + | libc::CLONE_NEWIPC + | libc::CLONE_NEWNET + | libc::CLONE_NEWNS + | libc::CLONE_NEWPID + | libc::CLONE_NEWUSER + | libc::CLONE_NEWUTS) as u32; + +/// A prebuilt child filter that can be installed without heap allocation. +pub struct ChildHardeningProgram { + instructions: Vec, +} + +impl ChildHardeningProgram { + /// Install this filter on the calling thread only. + /// + /// The caller must invoke this from the post-fork child after all + /// sandbox-wide TSYNC work and the launcher's `NEW_LISTENER` filter. + pub fn install(&mut self) -> io::Result<()> { + set_no_new_privileges()?; + let len = u16::try_from(self.instructions.len()).map_err(|_| { + io::Error::new( + io::ErrorKind::InvalidInput, + "child seccomp filter is too large", + ) + })?; + let mut program = libc::sock_fprog { + len, + filter: self.instructions.as_mut_ptr(), + }; + // SAFETY: `program` references the prebuilt cBPF instruction vector + // for the complete syscall. No TSYNC flag is used. + let result = unsafe { + libc::syscall( + libc::SYS_seccomp, + SECCOMP_SET_MODE_FILTER, + 0, + std::ptr::addr_of_mut!(program), + ) + }; + if result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } + } + + /// Number of cBPF instructions, exposed for admission diagnostics. + #[must_use] + pub fn instruction_count(&self) -> usize { + self.instructions.len() + } +} + +/// Build the same-UID workload self-protection program before `fork`. +/// +/// `sandbox_tgid` is the sandbox PID as visible from its workload namespace. +/// The filter blocks thread-targeting operations that name the trusted sandbox +/// leader and blocks process-directed operations with the same target. The +/// ordinary workload listener additionally mediates `kill`, `tkill`, and +/// `rt_sigqueueinfo`: Linux accepts nonleader TIDs for these operations, so a +/// static TGID comparison alone cannot protect future sandbox worker threads. +pub fn prepare(sandbox_tgid: u32) -> io::Result { + if sandbox_tgid == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "sandbox TGID must be nonzero", + )); + } + + let mut instructions = vec![ + stmt(BPF_LD_W_ABS, SECCOMP_DATA_ARCH_OFFSET), + jump(BPF_JMP_JEQ_K, native_audit_arch(), 1, 0), + stmt(BPF_RET_K, SECCOMP_RET_KILL_PROCESS), + stmt(BPF_LD_W_ABS, SECCOMP_DATA_NR_OFFSET), + ]; + #[cfg(target_arch = "x86_64")] + instructions.extend([ + jump(BPF_JMP_JSET_K, X32_SYSCALL_BIT, 0, 1), + stmt(BPF_RET_K, SECCOMP_RET_KILL_PROCESS), + ]); + + for syscall in [ + libc::SYS_ptrace, + libc::SYS_process_vm_readv, + libc::SYS_process_vm_writev, + libc::SYS_pidfd_getfd, + libc::SYS_pidfd_send_signal, + libc::SYS_kcmp, + libc::SYS_process_madvise, + libc::SYS_process_mrelease, + libc::SYS_unshare, + libc::SYS_setns, + libc::SYS_mount, + libc::SYS_umount2, + libc::SYS_pivot_root, + libc::SYS_chroot, + libc::SYS_fsopen, + libc::SYS_fsconfig, + libc::SYS_fsmount, + libc::SYS_fspick, + libc::SYS_move_mount, + libc::SYS_open_tree, + libc::SYS_bpf, + libc::SYS_perf_event_open, + libc::SYS_userfaultfd, + libc::SYS_io_uring_setup, + libc::SYS_io_uring_enter, + libc::SYS_io_uring_register, + libc::SYS_capset, + libc::SYS_setuid, + libc::SYS_setgid, + libc::SYS_setreuid, + libc::SYS_setregid, + libc::SYS_setresuid, + libc::SYS_setresgid, + libc::SYS_setfsuid, + libc::SYS_setfsgid, + libc::SYS_setgroups, + libc::SYS_sethostname, + libc::SYS_setdomainname, + libc::SYS_setpriority, + libc::SYS_ioprio_set, + ] { + append_unconditional_deny(&mut instructions, syscall)?; + } + + // Modern launchers fall back from clone3 and pidfd_open only for ENOSYS. + // Returning EPERM here breaks otherwise portable process creation. The + // fallback paths remain constrained: namespace creation is denied from + // clone's scalar flags and no pidfd can be acquired. + append_unconditional_errno(&mut instructions, libc::SYS_clone3, libc::ENOSYS)?; + append_unconditional_errno(&mut instructions, libc::SYS_pidfd_open, libc::ENOSYS)?; + append_argument_masked_deny(&mut instructions, libc::SYS_clone, 0, CLONE_NAMESPACE_FLAGS)?; + + for (syscall, argument) in [ + (libc::SYS_kill, 0), + (libc::SYS_tkill, 0), + (libc::SYS_tgkill, 0), + (libc::SYS_rt_sigqueueinfo, 0), + (libc::SYS_rt_tgsigqueueinfo, 0), + ] { + append_argument_equal_deny(&mut instructions, syscall, argument, sandbox_tgid)?; + } + for syscall in [libc::SYS_kill, libc::SYS_rt_sigqueueinfo] { + // PID zero targets the caller's process group. Deny it even though + // OpenShell normally gives each workload a dedicated process group: + // an untrusted child can otherwise rejoin a trusted group first. + append_argument_equal_deny(&mut instructions, syscall, 0, 0)?; + } + // Negative PID arguments target process groups or every signalable + // process. The workload never needs that authority and must not be able + // to include the trusted sandbox workers in a broad signal operation. + append_argument_masked_deny(&mut instructions, libc::SYS_kill, 0, 1 << 31)?; + append_argument_masked_deny(&mut instructions, libc::SYS_rt_sigqueueinfo, 0, 1 << 31)?; + + for syscall in [ + libc::SYS_prlimit64, + libc::SYS_sched_setaffinity, + libc::SYS_sched_setattr, + libc::SYS_sched_setparam, + libc::SYS_sched_setscheduler, + ] { + append_argument_nonzero_deny(&mut instructions, syscall, 0)?; + } + + // The ordinary workload filter is installed after this program and owns + // the final ban on further seccomp installation. Blocking it here would + // prevent the sandbox from completing the prepared filter stack. + append_argument_masked_deny( + &mut instructions, + libc::SYS_close_range, + 2, + CLOSE_RANGE_UNSHARE_FLAG, + )?; + + for command in [F_SETOWN_COMMAND, F_SETSIG_COMMAND, F_SETOWN_EX_COMMAND] { + append_argument_equal_deny(&mut instructions, libc::SYS_fcntl, 1, command)?; + } + for request in [FIOSETOWN_REQUEST, SIOCSPGRP_REQUEST] { + append_argument_equal_deny(&mut instructions, libc::SYS_ioctl, 1, request)?; + } + + instructions.push(stmt(BPF_RET_K, SECCOMP_RET_ALLOW)); + Ok(ChildHardeningProgram { instructions }) +} + +fn append_unconditional_deny( + instructions: &mut Vec, + syscall: i64, +) -> io::Result<()> { + let syscall = syscall_number(syscall)?; + instructions.extend([ + stmt(BPF_LD_W_ABS, SECCOMP_DATA_NR_OFFSET), + jump(BPF_JMP_JEQ_K, syscall, 0, 1), + errno(libc::EPERM), + ]); + Ok(()) +} + +fn append_unconditional_errno( + instructions: &mut Vec, + syscall: i64, + error: i32, +) -> io::Result<()> { + let syscall = syscall_number(syscall)?; + instructions.extend([ + stmt(BPF_LD_W_ABS, SECCOMP_DATA_NR_OFFSET), + jump(BPF_JMP_JEQ_K, syscall, 0, 1), + errno(error), + ]); + Ok(()) +} + +fn append_argument_equal_deny( + instructions: &mut Vec, + syscall: i64, + argument: u32, + value: u32, +) -> io::Result<()> { + let syscall = syscall_number(syscall)?; + instructions.extend([ + stmt(BPF_LD_W_ABS, SECCOMP_DATA_NR_OFFSET), + jump(BPF_JMP_JEQ_K, syscall, 0, 3), + stmt(BPF_LD_W_ABS, argument_word_offset(argument)), + jump(BPF_JMP_JEQ_K, value, 0, 1), + errno(libc::EPERM), + ]); + Ok(()) +} + +fn append_argument_nonzero_deny( + instructions: &mut Vec, + syscall: i64, + argument: u32, +) -> io::Result<()> { + let syscall = syscall_number(syscall)?; + instructions.extend([ + stmt(BPF_LD_W_ABS, SECCOMP_DATA_NR_OFFSET), + jump(BPF_JMP_JEQ_K, syscall, 0, 3), + stmt(BPF_LD_W_ABS, argument_word_offset(argument)), + jump(BPF_JMP_JEQ_K, 0, 1, 0), + errno(libc::EPERM), + ]); + Ok(()) +} + +fn append_argument_masked_deny( + instructions: &mut Vec, + syscall: i64, + argument: u32, + mask: u32, +) -> io::Result<()> { + let syscall = syscall_number(syscall)?; + instructions.extend([ + stmt(BPF_LD_W_ABS, SECCOMP_DATA_NR_OFFSET), + jump(BPF_JMP_JEQ_K, syscall, 0, 3), + stmt(BPF_LD_W_ABS, argument_word_offset(argument)), + jump(BPF_JMP_JSET_K, mask, 0, 1), + errno(libc::EPERM), + ]); + Ok(()) +} + +fn syscall_number(syscall: i64) -> io::Result { + u32::try_from(syscall) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "negative syscall number")) +} + +const fn argument_word_offset(argument: u32) -> u32 { + SECCOMP_DATA_ARGS_OFFSET + argument * 8 +} + +const fn errno(value: i32) -> libc::sock_filter { + stmt(BPF_RET_K, SECCOMP_RET_ERRNO | value.cast_unsigned()) +} + +#[cfg(target_arch = "x86_64")] +const fn native_audit_arch() -> u32 { + 0xc000_003e +} + +#[cfg(target_arch = "aarch64")] +const fn native_audit_arch() -> u32 { + 0xc000_00b7 +} + +const fn stmt(code: u16, value: u32) -> libc::sock_filter { + libc::sock_filter { + code, + jt: 0, + jf: 0, + k: value, + } +} + +const fn jump(code: u16, value: u32, jt: u8, jf: u8) -> libc::sock_filter { + libc::sock_filter { + code, + jt, + jf, + k: value, + } +} + +fn set_no_new_privileges() -> io::Result<()> { + // SAFETY: PR_SET_NO_NEW_PRIVS is a one-way scalar transition. + if unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) } < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rejects_zero_sandbox_tgid() { + assert_eq!( + prepare(0).err().expect("zero TGID must fail").kind(), + io::ErrorKind::InvalidInput + ); + } + + #[test] + fn filter_blocks_same_uid_sandbox_control() { + // SAFETY: the child uses only raw syscalls after fork and exits with + // `_exit`, so it does not run copied Rust cleanup state. + let child = unsafe { libc::fork() }; + assert!(child >= 0, "fork: {}", io::Error::last_os_error()); + if child == 0 { + let sandbox_tgid = unsafe { libc::getppid() }; + let Ok(mut filter) = prepare(u32::try_from(sandbox_tgid).unwrap_or(0)) else { + unsafe { libc::_exit(1) }; + }; + if filter.install().is_err() { + unsafe { libc::_exit(2) }; + } + let mut local = 0_u8; + let mut remote = 0_u8; + let local_iov = libc::iovec { + iov_base: std::ptr::addr_of_mut!(local).cast(), + iov_len: 1, + }; + let remote_iov = libc::iovec { + iov_base: std::ptr::addr_of_mut!(remote).cast(), + iov_len: 1, + }; + let process_vm = unsafe { + libc::process_vm_readv( + sandbox_tgid, + std::ptr::addr_of!(local_iov), + 1, + std::ptr::addr_of!(remote_iov), + 1, + 0, + ) + }; + if process_vm != -1 || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) { + unsafe { libc::_exit(3) }; + } + if unsafe { libc::kill(sandbox_tgid, 0) } != -1 + || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) + { + unsafe { libc::_exit(4) }; + } + if unsafe { libc::kill(0, 0) } != -1 + || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) + { + unsafe { libc::_exit(10) }; + } + let sandbox_group = -unsafe { libc::getpgrp() }; + if unsafe { libc::kill(sandbox_group, 0) } != -1 + || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) + { + unsafe { libc::_exit(7) }; + } + if unsafe { libc::syscall(libc::SYS_prlimit64, sandbox_tgid, libc::RLIMIT_CORE, 0, 0) } + != -1 + || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) + { + unsafe { libc::_exit(5) }; + } + if unsafe { + libc::syscall( + libc::SYS_sched_setattr, + sandbox_tgid, + std::ptr::null::(), + 0, + ) + } != -1 + || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) + { + unsafe { libc::_exit(11) }; + } + if unsafe { libc::fcntl(libc::STDIN_FILENO, libc::F_SETOWN, sandbox_tgid) } != -1 + || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) + { + unsafe { libc::_exit(8) }; + } + let mut owner = sandbox_tgid; + if unsafe { + libc::ioctl( + libc::STDIN_FILENO, + libc::c_ulong::from(FIOSETOWN_REQUEST), + &raw mut owner, + ) + } != -1 + || io::Error::last_os_error().raw_os_error() != Some(libc::EPERM) + { + unsafe { libc::_exit(9) }; + } + unsafe { libc::_exit(0) }; + } + + let mut status = 0; + // SAFETY: `child` names our live direct child and status is writable. + assert_eq!(unsafe { libc::waitpid(child, &raw mut status, 0) }, child); + assert!(libc::WIFEXITED(status)); + assert_eq!(libc::WEXITSTATUS(status), 0); + } + + #[test] + fn filter_preserves_thread_and_process_creation() { + if std::env::var_os("OPENSHELL_CHILD_SECCOMP_CREATION_PROBE").is_some() { + let mut filter = prepare(std::process::id().saturating_add(1)) + .expect("prepare child hardening filter"); + filter.install().expect("install child hardening filter"); + + // A direct clone3 request must report ENOSYS so libc can use its + // established clone fallback. + let result = + unsafe { libc::syscall(libc::SYS_clone3, std::ptr::null::(), 0) }; + assert_eq!(result, -1); + assert_eq!( + io::Error::last_os_error().raw_os_error(), + Some(libc::ENOSYS) + ); + + // Process launchers such as uv also probe pidfd_open and require + // ENOSYS to select their non-pidfd fallback. + let result = unsafe { libc::syscall(libc::SYS_pidfd_open, libc::getpid(), 0) }; + assert_eq!(result, -1); + assert_eq!( + io::Error::last_os_error().raw_os_error(), + Some(libc::ENOSYS) + ); + + let joined = std::thread::spawn(|| 17_u8) + .join() + .expect("pthread-style child must start"); + assert_eq!(joined, 17); + assert!( + std::process::Command::new("/bin/true") + .status() + .expect("posix-spawn-style child must start") + .success() + ); + + let namespaced = unsafe { + libc::syscall( + libc::SYS_clone, + u64::from(CLONE_NAMESPACE_FLAGS & libc::CLONE_NEWUSER as u32) + | u64::from(libc::SIGCHLD as u32), + 0, + 0, + 0, + 0, + ) + }; + assert_eq!(namespaced, -1); + assert_eq!(io::Error::last_os_error().raw_os_error(), Some(libc::EPERM)); + return; + } + + let output = + std::process::Command::new(std::env::current_exe().expect("current test executable")) + .arg("--exact") + .arg("linux::child_seccomp::tests::filter_preserves_thread_and_process_creation") + .arg("--nocapture") + .env("OPENSHELL_CHILD_SECCOMP_CREATION_PROBE", "1") + .output() + .expect("run isolated child-hardening probe"); + assert!( + output.status.success(), + "isolated probe failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/landlock.rs b/crates/openshell-isolation-interface/src/linux/landlock.rs new file mode 100644 index 0000000000..28823ecba5 --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/landlock.rs @@ -0,0 +1,201 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Race-resistant handles for an explicit Landlock root allow-list. + +#![allow(unsafe_code)] +use std::collections::BTreeSet; +use std::ffi::{OsStr, OsString}; +use std::io; +use std::os::fd::OwnedFd; +use std::os::unix::ffi::OsStrExt; +use std::path::Path; + +use rustix::fs::{AtFlags, Mode, OFlags, Stat, fstat, open, openat, statat}; + +const LANDLOCK_CREATE_RULESET_VERSION: libc::c_uint = 1; + +/// Query the Landlock ABI admitted by the active kernel and outer seccomp +/// profile without installing a ruleset. +pub fn abi_version() -> io::Result { + // SAFETY: the VERSION operation requires a null ruleset pointer and zero + // size and returns one scalar ABI version. + let result = unsafe { + libc::syscall( + libc::SYS_landlock_create_ruleset, + std::ptr::null::(), + 0, + LANDLOCK_CREATE_RULESET_VERSION, + ) + }; + if result < 0 { + Err(io::Error::last_os_error()) + } else { + u32::try_from(result).map_err(|_| io::Error::other("Landlock ABI does not fit u32")) + } +} +/// One verified immediate child of the sandbox root. +pub struct RootEntryHandle { + name: OsString, + fd: OwnedFd, + stat: Stat, +} + +impl RootEntryHandle { + /// Immediate-root entry name. + #[must_use] + pub fn name(&self) -> &OsStr { + &self.name + } + + /// Open, no-follow handle suitable for a later Landlock `PathBeneath` rule. + #[must_use] + pub fn fd(&self) -> &OwnedFd { + &self.fd + } + + /// Device number captured when the entry was opened. + #[must_use] + pub fn device(&self) -> u64 { + self.stat.st_dev + } + + /// Inode number captured when the entry was opened. + #[must_use] + pub fn inode(&self) -> u64 { + self.stat.st_ino + } +} + +/// Open exactly the named root entries while proving that none is the private +/// sandbox hierarchy, a symlink, or a raced replacement. +/// +/// Unnamed root entries are deliberately not returned and therefore cannot be +/// admitted accidentally. The caller obtains the allow-list names from trusted +/// image/driver policy, not by blindly allowing everything present in `/`. +pub fn open_root_allowlist( + root: &Path, + allowed_names: &BTreeSet, + private_name: &OsStr, +) -> io::Result> { + validate_component(private_name)?; + if allowed_names.contains(private_name) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "private sandbox root cannot appear in the Landlock allow-list", + )); + } + + let root_fd = open( + root, + OFlags::PATH | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + )?; + let mut result = Vec::with_capacity(allowed_names.len()); + for name in allowed_names { + validate_component(name)?; + let before = statat(&root_fd, name, AtFlags::SYMLINK_NOFOLLOW)?; + if before.st_mode & libc::S_IFMT == libc::S_IFLNK { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!( + "Landlock root entry {} is a symlink", + Path::new(name).display() + ), + )); + } + let fd = openat( + &root_fd, + name, + OFlags::PATH | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + )?; + let after = fstat(&fd)?; + if before.st_dev != after.st_dev + || before.st_ino != after.st_ino + || before.st_mode & libc::S_IFMT != after.st_mode & libc::S_IFMT + { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!( + "Landlock root entry {} changed while it was opened", + Path::new(name).display() + ), + )); + } + result.push(RootEntryHandle { + name: name.clone(), + fd, + stat: after, + }); + } + Ok(result) +} + +fn validate_component(name: &OsStr) -> io::Result<()> { + let bytes = name.as_bytes(); + if bytes.is_empty() || bytes == b"." || bytes == b".." || bytes.contains(&b'/') { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("invalid immediate-root entry {}", Path::new(name).display()), + )); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use std::fs; + use std::os::unix::fs::symlink; + use std::sync::atomic::{AtomicU64, Ordering}; + + use super::*; + + static NEXT_TEMP: AtomicU64 = AtomicU64::new(0); + + fn temp_root() -> std::path::PathBuf { + let path = std::env::temp_dir().join(format!( + "openshell-landlock-root-{}-{}", + std::process::id(), + NEXT_TEMP.fetch_add(1, Ordering::Relaxed) + )); + fs::create_dir(&path).expect("create temp root"); + path + } + + #[test] + fn opens_only_explicit_entries_and_omits_private_root() { + let root = temp_root(); + fs::create_dir(root.join("bin")).expect("create bin"); + fs::create_dir(root.join("sandbox")).expect("create workspace"); + fs::create_dir(root.join(".openshell")).expect("create private root"); + fs::create_dir(root.join("unexpected")).expect("create unexpected root"); + + let allowed = BTreeSet::from([OsString::from("bin"), OsString::from("sandbox")]); + let entries = open_root_allowlist(&root, &allowed, OsStr::new(".openshell")) + .expect("open allow-list"); + assert_eq!( + entries + .iter() + .map(|entry| entry.name().to_owned()) + .collect::>(), + vec![OsString::from("bin"), OsString::from("sandbox")] + ); + + fs::remove_dir_all(root).expect("remove temp root"); + } + + #[test] + fn rejects_private_entry_and_symlink() { + let root = temp_root(); + fs::create_dir(root.join(".openshell")).expect("create private root"); + symlink(".openshell", root.join("runtime")).expect("create symlink"); + + let private = BTreeSet::from([OsString::from(".openshell")]); + assert!(open_root_allowlist(&root, &private, OsStr::new(".openshell")).is_err()); + let symlinked = BTreeSet::from([OsString::from("runtime")]); + assert!(open_root_allowlist(&root, &symlinked, OsStr::new(".openshell")).is_err()); + + fs::remove_dir_all(root).expect("remove temp root"); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/mod.rs b/crates/openshell-isolation-interface/src/linux/mod.rs new file mode 100644 index 0000000000..bad3d329dc --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/mod.rs @@ -0,0 +1,16 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Audited Linux primitives used by the capability-free sandbox. +//! +//! This module intentionally contains mechanisms, not sandbox orchestration. +//! The in-workload sandbox owns lifecycle, policy, and failure handling. + +pub mod child_seccomp; +pub mod landlock; +pub mod proc_fd; +pub mod process_signal; +pub mod seccomp_notify; +pub mod socket_registry; +pub mod task_memory; +pub mod workload_launcher; diff --git a/crates/openshell-isolation-interface/src/linux/proc_fd.rs b/crates/openshell-isolation-interface/src/linux/proc_fd.rs new file mode 100644 index 0000000000..13f1856e54 --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/proc_fd.rs @@ -0,0 +1,165 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Strict `/proc//fd` socket identity helpers. + +#![allow(unsafe_code)] + +use std::fs; +use std::io; +use std::os::fd::RawFd; + +/// Snapshot socket inodes installed in process descriptor tables other than +/// `excluded_pid`. +/// +/// Inaccessible or concurrently disappearing entries are +/// skipped. Callers use this only to reclaim bounded mediation metadata; a +/// later operation on an unregistered descriptor fails closed. +pub fn installed_socket_inodes_excluding( + excluded_pid: u32, +) -> io::Result> { + let mut inodes = std::collections::BTreeSet::new(); + for process in fs::read_dir("/proc")? { + let Ok(process) = process else { continue }; + let Some(name) = process.file_name().to_str().map(str::to_owned) else { + continue; + }; + let Ok(pid) = name.parse::() else { + continue; + }; + if pid == excluded_pid { + continue; + } + let Ok(descriptors) = fs::read_dir(process.path().join("fd")) else { + continue; + }; + for descriptor in descriptors.flatten() { + let Ok(target) = fs::read_link(descriptor.path()) else { + continue; + }; + let Some(target) = target.to_str() else { + continue; + }; + let Some(digits) = target + .strip_prefix("socket:[") + .and_then(|value| value.strip_suffix(']')) + else { + continue; + }; + if let Ok(inode) = digits.parse::() { + inodes.insert(inode); + } + } + } + Ok(inodes) +} + +/// Return the socket inode currently installed at `fd` in `tid`'s descriptor +/// table. +/// +/// The result is only a snapshot. Callers must revalidate the seccomp +/// notification, task generation, and any retained socket cookie before a +/// state-changing operation. +pub fn socket_inode(tid: u32, fd: RawFd) -> io::Result { + if tid == 0 || fd < 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "TID must be nonzero and FD must be nonnegative", + )); + } + let target = fs::read_link(format!("/proc/{tid}/fd/{fd}"))?; + let target = target.to_str().ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidData, + "procfs descriptor target is not UTF-8", + ) + })?; + let digits = target + .strip_prefix("socket:[") + .and_then(|value| value.strip_suffix(']')) + .ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidInput, + "procfs descriptor is not a socket", + ) + })?; + if digits.is_empty() || !digits.bytes().all(|byte| byte.is_ascii_digit()) { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "procfs socket inode has an invalid representation", + )); + } + digits.parse::().map_err(|error| { + io::Error::new( + io::ErrorKind::InvalidData, + format!("procfs socket inode does not fit u64: {error}"), + ) + }) +} + +#[cfg(test)] +mod tests { + use std::fs::File; + use std::os::fd::{AsRawFd, FromRawFd, OwnedFd}; + + use super::*; + + #[test] + fn identifies_socket_and_rejects_regular_file() { + let mut pair = [-1; 2]; + // SAFETY: pair points to storage for exactly two returned descriptors. + let result = unsafe { + libc::socketpair( + libc::AF_UNIX, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + 0, + pair.as_mut_ptr(), + ) + }; + assert_eq!(result, 0, "socketpair: {}", io::Error::last_os_error()); + // SAFETY: successful socketpair returned two independently owned FDs. + let left = unsafe { OwnedFd::from_raw_fd(pair[0]) }; + // SAFETY: successful socketpair returned two independently owned FDs. + let _right = unsafe { OwnedFd::from_raw_fd(pair[1]) }; + assert!(socket_inode(std::process::id(), left.as_raw_fd()).unwrap() > 0); + + let file = File::open("/dev/null").expect("open regular descriptor"); + assert_eq!( + socket_inode(std::process::id(), file.as_raw_fd()) + .expect_err("regular descriptor") + .kind(), + io::ErrorKind::InvalidInput + ); + } + + #[test] + fn installed_socket_snapshot_can_exclude_the_broker() { + let mut pair = [-1; 2]; + // SAFETY: pair points to storage for exactly two returned descriptors. + let result = unsafe { + libc::socketpair( + libc::AF_UNIX, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + 0, + pair.as_mut_ptr(), + ) + }; + assert_eq!(result, 0, "socketpair: {}", io::Error::last_os_error()); + // SAFETY: successful socketpair returned two independently owned FDs. + let left = unsafe { OwnedFd::from_raw_fd(pair[0]) }; + // SAFETY: successful socketpair returned two independently owned FDs. + let _right = unsafe { OwnedFd::from_raw_fd(pair[1]) }; + let inode = socket_inode(std::process::id(), left.as_raw_fd()).unwrap(); + + assert!( + installed_socket_inodes_excluding(u32::MAX) + .unwrap() + .contains(&inode) + ); + assert!( + !installed_socket_inodes_excluding(std::process::id()) + .unwrap() + .contains(&inode) + ); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/process_signal.rs b/crates/openshell-isolation-interface/src/linux/process_signal.rs new file mode 100644 index 0000000000..796f548037 --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/process_signal.rs @@ -0,0 +1,252 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Process-directed workload signals delivered through retained pidfds. +//! +//! Linux accepts a worker TID for `kill()`, so scalar seccomp checks against the +//! sandbox leader do not suffice. Resolve the thread group, exclude the live +//! sandbox, and retain the target before delivery. Never continue an inspected +//! numeric PID: it could be reused by a newly created sandbox worker. + +#![allow(unsafe_code)] + +use std::io; +use std::os::fd::{AsRawFd, FromRawFd, OwnedFd}; + +use crate::linux::seccomp_notify::{Notification, NotificationListener}; +use crate::linux::task_memory; + +/// Emulate one positive-target kill or `rt_sigqueueinfo` notification. +/// +/// The sandbox and workload must use the same procfs/PID namespace. The +/// sandbox TGID remains live throughout delivery, so an excluded numeric TGID +/// cannot be reused. Group/broadcast signaling remains deliberately denied. +/// Plain kill is broker-originated; queued signals retain their supplied +/// siginfo. Kernel signal permission and siginfo checks still apply. +pub fn mediate_process_signal( + listener: &NotificationListener, + notification: Notification, + sandbox_tgid: u32, +) -> io::Result<()> { + listener.validate_id(notification.id)?; + let target = scalar_int(notification.args[0]); + let signal = scalar_int(notification.args[1]); + if target <= 0 || !(0..=64).contains(&signal) { + return Err(io::Error::from_raw_os_error(if target <= 0 { + libc::EPERM + } else { + libc::EINVAL + })); + } + let target = u32::try_from(target).map_err(|_| io::Error::from_raw_os_error(libc::ESRCH))?; + let retained = retain_signal_target(target, sandbox_tgid)?; + // SAFETY: all-zero siginfo consists of valid integer/pointer fields. A + // queued operation copies the entire object once before it is consumed. + let mut info: libc::siginfo_t = unsafe { std::mem::zeroed() }; + let info_ptr = match i64::from(notification.syscall) { + libc::SYS_kill => std::ptr::null(), + libc::SYS_rt_sigqueueinfo => { + // SAFETY: bytes exclusively spans the live siginfo object; all bit + // patterns are valid and task_memory requires a complete copy. + let bytes = unsafe { + std::slice::from_raw_parts_mut( + (&raw mut info).cast::(), + size_of::(), + ) + }; + task_memory::read_exact(notification.tid, notification.args[2], bytes)?; + // Positive/kernel-origin and SI_TKILL codes cannot be impersonated. + if info.si_code >= 0 || info.si_code == libc::SI_TKILL { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } + &raw const info + } + _ => return Err(io::Error::from_raw_os_error(libc::ENOSYS)), + }; + listener.validate_id(notification.id)?; + // SAFETY: retained owns a live pidfd; info is null or a complete trusted + // copy. The kernel targets that process object, never a reused numeric PID. + let result = unsafe { + libc::syscall( + libc::SYS_pidfd_send_signal, + retained.as_raw_fd(), + signal, + info_ptr, + 0, + ) + }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + listener.respond_value(notification.id, 0) +} + +/// Continue a positive-target `tkill` only when the target thread belongs to +/// an untrusted workload process rather than the sandbox runtime itself. +/// +/// Continuing preserves Linux's thread-directed signal semantics, including +/// the cancellation signal used by musl. A target that exits between the +/// ownership check and continuation can only be reused inside the same PID +/// namespace; the static child filter still rejects the sandbox leader. +pub fn mediate_thread_signal( + listener: &NotificationListener, + notification: Notification, + sandbox_tgid: u32, +) -> io::Result<()> { + listener.validate_id(notification.id)?; + let target = scalar_int(notification.args[0]); + let signal = scalar_int(notification.args[1]); + if target <= 0 || !(0..=64).contains(&signal) { + return Err(io::Error::from_raw_os_error(if target <= 0 { + libc::EPERM + } else { + libc::EINVAL + })); + } + let target = u32::try_from(target).map_err(|_| io::Error::from_raw_os_error(libc::ESRCH))?; + let target_group = thread_group_id(target)?; + if target_group == sandbox_tgid || target_group == 0 { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } + listener.validate_id(notification.id)?; + listener.respond_continue(notification.id) +} + +fn scalar_int(value: u64) -> i32 { + let bytes = value.to_ne_bytes(); + #[cfg(target_endian = "little")] + let scalar = [bytes[0], bytes[1], bytes[2], bytes[3]]; + #[cfg(target_endian = "big")] + let scalar = [bytes[4], bytes[5], bytes[6], bytes[7]]; + i32::from_ne_bytes(scalar) +} + +fn retain_signal_target(tid: u32, sandbox_tgid: u32) -> io::Result { + if sandbox_tgid == 0 { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); + } + let target_group = thread_group_id(tid)?; + if target_group == sandbox_tgid || target_group == 0 { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } + // SAFETY: pidfd_open takes only scalar arguments and returns a new owned FD. + let fd = unsafe { libc::syscall(libc::SYS_pidfd_open, target_group, 0) }; + if fd < 0 { + return Err(io::Error::last_os_error()); + } + let fd = i32::try_from(fd).map_err(|_| io::Error::other("pidfd does not fit RawFd"))?; + // SAFETY: successful pidfd_open transferred this descriptor to the caller. + Ok(unsafe { OwnedFd::from_raw_fd(fd) }) +} + +fn thread_group_id(tid: u32) -> io::Result { + let status = std::fs::read_to_string(format!("/proc/{tid}/status")).map_err(|error| { + if error.kind() == io::ErrorKind::NotFound { + io::Error::from_raw_os_error(libc::ESRCH) + } else { + error + } + })?; + status + .lines() + .find_map(|line| { + line.strip_prefix("Tgid:") + .and_then(|value| value.trim().parse::().ok()) + }) + .ok_or_else(|| io::Error::from_raw_os_error(libc::ESRCH)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn filtered_worker_cannot_signal_the_sandbox_through_a_tid() { + let (sender, receiver) = std::sync::mpsc::sync_channel(1); + let worker = std::thread::spawn(move || { + let listener = crate::linux::seccomp_notify::install_workload_listener().unwrap(); + sender.send(listener).unwrap(); + // SAFETY: gettid has no arguments; signal zero checks permission + // without delivering a signal to the disposable test thread. + let tid = unsafe { libc::syscall(libc::SYS_gettid) }; + let result = unsafe { libc::syscall(libc::SYS_kill, tid, 0) }; + (result, io::Error::last_os_error().raw_os_error()) + }); + let listener = receiver + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + let notification = listener.receive().unwrap(); + let error = + mediate_process_signal(&listener, notification, std::process::id()).unwrap_err(); + assert_eq!(error.raw_os_error(), Some(libc::EPERM)); + listener + .respond_errno(notification.id, libc::EPERM) + .unwrap(); + assert_eq!(worker.join().unwrap(), (-1, Some(libc::EPERM))); + } + + #[test] + fn retained_child_can_be_signaled_without_numeric_pid_delivery() { + let mut child = std::process::Command::new("sleep") + .arg("30") + .spawn() + .unwrap(); + let retained = retain_signal_target(child.id(), std::process::id()).unwrap(); + // SAFETY: this pidfd owns the disposable child launched by this test. + assert_eq!( + unsafe { + libc::syscall( + libc::SYS_pidfd_send_signal, + retained.as_raw_fd(), + libc::SIGTERM, + std::ptr::null::(), + 0, + ) + }, + 0 + ); + assert!(!child.wait().unwrap().success()); + assert_eq!( + unsafe { + libc::syscall( + libc::SYS_pidfd_send_signal, + retained.as_raw_fd(), + 0, + std::ptr::null::(), + 0, + ) + }, + -1 + ); + assert_eq!(io::Error::last_os_error().raw_os_error(), Some(libc::ESRCH)); + } + + #[test] + fn rejects_live_sandbox_worker_tid() { + let sandbox_tgid = std::process::id(); + std::thread::spawn(move || { + // SAFETY: gettid has no arguments or side effects. + let tid = u32::try_from(unsafe { libc::syscall(libc::SYS_gettid) }).unwrap(); + assert_ne!(tid, sandbox_tgid); + assert_eq!( + retain_signal_target(tid, sandbox_tgid) + .unwrap_err() + .raw_os_error(), + Some(libc::EPERM) + ); + }) + .join() + .unwrap(); + } + + #[test] + fn rejects_leader_and_preserves_scalar_pid_semantics() { + let pid = std::process::id(); + assert_eq!( + retain_signal_target(pid, pid).unwrap_err().raw_os_error(), + Some(libc::EPERM) + ); + assert_eq!(scalar_int(u64::MAX), -1); + assert_eq!(scalar_int(1 << 32 | 0x7b), 123); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/seccomp_notify.rs b/crates/openshell-isolation-interface/src/linux/seccomp_notify.rs new file mode 100644 index 0000000000..7d24dfa4d9 --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/seccomp_notify.rs @@ -0,0 +1,938 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Minimal, typed wrappers for Linux seccomp user notification. +//! +//! The wrappers validate notification IDs around every operation and keep raw +//! UAPI structures private. Production policy and queueing belong to the +//! sandbox crate; this module owns only the kernel ABI and active conformance +//! probe. + +#![allow(unsafe_code)] + +use std::io; +use std::mem::size_of; +use std::os::fd::{AsRawFd, FromRawFd, OwnedFd, RawFd}; +use std::sync::mpsc; +use std::thread; +use std::time::Duration; + +const SECCOMP_SET_MODE_FILTER: libc::c_uint = 1; +const SECCOMP_GET_NOTIF_SIZES: libc::c_uint = 3; +const SECCOMP_FILTER_FLAG_NEW_LISTENER: libc::c_ulong = 1 << 3; +const SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV: libc::c_ulong = 1 << 5; + +const SECCOMP_RET_KILL_PROCESS: u32 = 0x8000_0000; +const SECCOMP_RET_USER_NOTIF: u32 = 0x7fc0_0000; +const SECCOMP_RET_ALLOW: u32 = 0x7fff_0000; + +const BPF_LD_W_ABS: u16 = 0x20; +const BPF_JMP_JEQ_K: u16 = 0x15; +#[cfg(target_arch = "x86_64")] +const BPF_JMP_JSET_K: u16 = 0x45; +const BPF_ALU_AND_K: u16 = 0x54; +const BPF_RET_K: u16 = 0x06; + +const SECCOMP_DATA_NR_OFFSET: u32 = 0; +const SECCOMP_DATA_ARCH_OFFSET: u32 = 4; +const SECCOMP_DATA_ARGS_OFFSET: u32 = 16; +#[cfg(target_arch = "x86_64")] +const X32_SYSCALL_BIT: u32 = 0x4000_0000; + +const SECCOMP_ADDFD_FLAG_SEND: u32 = 1 << 1; +const SECCOMP_USER_NOTIF_FLAG_CONTINUE: u32 = 1; + +const CONNECTED_SEND_FLAGS: u32 = + (libc::MSG_DONTWAIT | libc::MSG_EOR | libc::MSG_MORE | libc::MSG_NOSIGNAL | libc::MSG_OOB) + as u32; +const PROBE_NOTIFICATION_TIMEOUT: Duration = Duration::from_secs(5); + +const IOC_NRBITS: u32 = 8; +const IOC_TYPEBITS: u32 = 8; +const IOC_SIZEBITS: u32 = 14; +const IOC_NRSHIFT: u32 = 0; +const IOC_TYPESHIFT: u32 = IOC_NRSHIFT + IOC_NRBITS; +const IOC_SIZESHIFT: u32 = IOC_TYPESHIFT + IOC_TYPEBITS; +const IOC_DIRSHIFT: u32 = IOC_SIZESHIFT + IOC_SIZEBITS; +const IOC_WRITE: u32 = 1; +const IOC_READ: u32 = 2; +const SECCOMP_IOC_MAGIC: u32 = b'!' as u32; + +#[allow(clippy::cast_possible_truncation)] +const fn ioc(direction: u32, number: u32, size: usize) -> libc::c_ulong { + ((direction << IOC_DIRSHIFT) + | (SECCOMP_IOC_MAGIC << IOC_TYPESHIFT) + | (number << IOC_NRSHIFT) + | ((size as u32) << IOC_SIZESHIFT)) as libc::c_ulong +} + +const fn iowr(number: u32) -> libc::c_ulong { + ioc(IOC_READ | IOC_WRITE, number, size_of::()) +} + +const fn iow(number: u32) -> libc::c_ulong { + ioc(IOC_WRITE, number, size_of::()) +} + +#[repr(C)] +#[derive(Clone, Copy, Debug, Default)] +struct SeccompData { + nr: i32, + arch: u32, + instruction_pointer: u64, + args: [u64; 6], +} + +#[repr(C)] +#[derive(Clone, Copy, Debug, Default)] +struct RawNotification { + id: u64, + pid: u32, + flags: u32, + data: SeccompData, +} + +#[repr(C)] +#[derive(Clone, Copy, Debug, Default)] +struct RawResponse { + id: u64, + val: i64, + error: i32, + flags: u32, +} + +#[repr(C)] +#[derive(Clone, Copy, Debug, Default)] +struct RawAddFd { + id: u64, + flags: u32, + srcfd: u32, + newfd: u32, + newfd_flags: u32, +} + +#[repr(C)] +#[derive(Clone, Copy, Debug, Default)] +struct RawNotificationSizes { + notification: u16, + response: u16, + data: u16, +} + +const SECCOMP_IOCTL_NOTIF_RECV: libc::c_ulong = iowr::(0); +const SECCOMP_IOCTL_NOTIF_SEND: libc::c_ulong = iowr::(1); +const SECCOMP_IOCTL_NOTIF_ID_VALID: libc::c_ulong = iow::(2); +const SECCOMP_IOCTL_NOTIF_ADDFD: libc::c_ulong = iow::(3); + +/// One validated seccomp user-notification request. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Notification { + /// Kernel-unique notification identifier. + pub id: u64, + /// Notifying Linux thread ID. + pub tid: u32, + /// Native syscall number. + pub syscall: i32, + /// Raw syscall arguments. + pub args: [u64; 6], +} + +/// Result of exercising the unprivileged notification API under the active +/// kernel, outer seccomp profile, and LSM posture. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct NotificationProbeReport { + /// Whether `SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` was accepted. + pub wait_killable_recv: bool, + features: NotificationProbeFeatures, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct NotificationProbeFeatures(u8); + +impl NotificationProbeReport { + /// Whether ID validation and response delivery completed. + #[must_use] + pub fn notification_round_trip(self) -> bool { + self.features.0 & 1 != 0 + } + + /// Whether atomic ADDFD-SEND injected a close-on-exec descriptor. + #[must_use] + pub fn addfd_send(self) -> bool { + self.features.0 & 2 != 0 + } + + /// Whether process-VM read and write syscalls are admitted for same-process + /// memory, before the stronger child-credential probe runs in a driver. + #[must_use] + pub fn task_memory_copy(self) -> bool { + self.features.0 & 4 != 0 + } + + /// Whether connected null-destination `sendto` bypassed notification while + /// destination-bearing and unsafe-flag variants remained mediated. + #[must_use] + pub fn connected_send_fast_path(self) -> bool { + self.features.0 & 8 != 0 + } +} + +/// Owned listener returned by `SECCOMP_FILTER_FLAG_NEW_LISTENER`. +pub struct NotificationListener { + fd: OwnedFd, + wait_killable_recv: bool, +} + +impl NotificationListener { + /// Raw listener descriptor for readiness integration and diagnostics. + #[must_use] + pub fn as_raw_fd(&self) -> RawFd { + self.fd.as_raw_fd() + } + + /// Whether the listener was installed with killable receive waits. + #[must_use] + pub fn wait_killable_recv(&self) -> bool { + self.wait_killable_recv + } + + /// Receive the next kernel notification. + pub fn receive(&self) -> io::Result { + let mut raw = RawNotification::default(); + ioctl_ptr( + self.fd.as_raw_fd(), + SECCOMP_IOCTL_NOTIF_RECV, + std::ptr::addr_of_mut!(raw).cast(), + )?; + Ok(Notification { + id: raw.id, + tid: raw.pid, + syscall: raw.data.nr, + args: raw.data.args, + }) + } + + /// Verify that a notification still refers to a blocked live task. + pub fn validate_id(&self, id: u64) -> io::Result<()> { + let mut id = id; + ioctl_ptr( + self.fd.as_raw_fd(), + SECCOMP_IOCTL_NOTIF_ID_VALID, + std::ptr::addr_of_mut!(id).cast(), + )?; + Ok(()) + } + + /// Return a successful scalar result to the notifying syscall. + pub fn respond_value(&self, id: u64, value: i64) -> io::Result<()> { + self.validate_id(id)?; + let mut response = RawResponse { + id, + val: value, + error: 0, + flags: 0, + }; + ioctl_ptr( + self.fd.as_raw_fd(), + SECCOMP_IOCTL_NOTIF_SEND, + std::ptr::addr_of_mut!(response).cast(), + )?; + Ok(()) + } + + /// Return `errno` to the notifying syscall. + pub fn respond_errno(&self, id: u64, errno: i32) -> io::Result<()> { + if errno <= 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "seccomp response errno must be positive", + )); + } + self.validate_id(id)?; + let mut response = RawResponse { + id, + val: 0, + error: -errno, + flags: 0, + }; + ioctl_ptr( + self.fd.as_raw_fd(), + SECCOMP_IOCTL_NOTIF_SEND, + std::ptr::addr_of_mut!(response).cast(), + )?; + Ok(()) + } + + /// Continue a verified local-kernel operation in the notifying task. + /// + /// Callers must not use this for an external INET operation or where a + /// mutable workload pointer is part of the authorization decision. + pub fn respond_continue(&self, id: u64) -> io::Result<()> { + self.validate_id(id)?; + let mut response = RawResponse { + id, + val: 0, + error: 0, + flags: SECCOMP_USER_NOTIF_FLAG_CONTINUE, + }; + ioctl_ptr( + self.fd.as_raw_fd(), + SECCOMP_IOCTL_NOTIF_SEND, + std::ptr::addr_of_mut!(response).cast(), + )?; + Ok(()) + } + + /// Atomically inject `source` and complete the notifying syscall with the + /// allocated target FD. The target receives `O_CLOEXEC` when requested. + pub fn add_fd_and_send( + &self, + notification_id: u64, + source: RawFd, + close_on_exec: bool, + ) -> io::Result { + self.validate_id(notification_id)?; + let srcfd = u32::try_from(source) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "source FD is negative"))?; + let mut addfd = RawAddFd { + id: notification_id, + flags: SECCOMP_ADDFD_FLAG_SEND, + srcfd, + newfd: 0, + newfd_flags: if close_on_exec { + u32::try_from(libc::O_CLOEXEC).map_err(io::Error::other)? + } else { + 0 + }, + }; + ioctl_ptr( + self.fd.as_raw_fd(), + SECCOMP_IOCTL_NOTIF_ADDFD, + std::ptr::addr_of_mut!(addfd).cast(), + ) + .and_then(|fd| { + RawFd::try_from(fd).map_err(|_| io::Error::other("injected FD does not fit RawFd")) + }) + } +} + +/// Install a non-TSYNC listener filter on the calling thread. +/// +/// Only the named syscalls notify. Unexpected architectures are killed, x32 +/// syscalls are killed on x86-64, and all other native syscalls are allowed. +pub fn install_listener(syscalls: &[i64]) -> io::Result { + if syscalls.is_empty() { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "at least one notified syscall is required", + )); + } + verify_notification_sizes()?; + set_no_new_privileges()?; + + install_listener_with_flags(syscalls, true).map_err(|error| { + if error.raw_os_error() == Some(libc::EINVAL) { + io::Error::new( + io::ErrorKind::Unsupported, + "seccomp WAIT_KILLABLE_RECV is required (Linux 5.19 or newer)", + ) + } else { + error + } + }) +} + +/// Install the capability-free workload networking listener on the calling +/// launcher thread. +/// +/// The filter mediates every syscall that can create, select, or materially +/// reconfigure an INET endpoint. Connected `send()`/null-destination +/// `sendto()` retains the audited cBPF fast path. +pub fn install_workload_listener() -> io::Result { + install_listener(&[ + libc::SYS_socket, + libc::SYS_connect, + libc::SYS_bind, + libc::SYS_listen, + libc::SYS_accept, + libc::SYS_accept4, + libc::SYS_sendto, + libc::SYS_sendmsg, + libc::SYS_sendmmsg, + libc::SYS_getpeername, + libc::SYS_setsockopt, + libc::SYS_kill, + libc::SYS_tkill, + libc::SYS_rt_sigqueueinfo, + ]) +} + +/// Run a no-capability conformance probe. +/// +/// This uses the production launcher-thread shape: the listener is created on +/// one dedicated thread and moved to an unfiltered broker thread through an +/// in-process channel. +pub fn probe_notification_api() -> io::Result { + let wait_killable_recv = probe_scalar_round_trip()?; + probe_addfd_send()?; + probe_task_memory_copy()?; + probe_connected_sendto_fast_path()?; + Ok(NotificationProbeReport { + wait_killable_recv, + features: NotificationProbeFeatures(1 | 2 | 4 | 8), + }) +} + +fn probe_scalar_round_trip() -> io::Result { + const PROBE_VALUE: libc::c_long = 0x5a17; + let (sender, receiver) = mpsc::sync_channel(1); + let launcher = thread::spawn(move || -> io::Result { + let listener = install_listener(&[libc::SYS_getppid])?; + let wait_killable = listener.wait_killable_recv(); + sender + .send((listener, wait_killable)) + .map_err(|_| io::Error::other("notification broker disappeared"))?; + // SAFETY: getppid has no pointer arguments. The installed filter causes + // the kernel to block here until the broker validates and responds. + Ok(unsafe { libc::syscall(libc::SYS_getppid) }) + }); + + let (listener, wait_killable) = receiver + .recv() + .map_err(|_| io::Error::other("notification launcher disappeared"))?; + let notification = match receive_probe_notification(&listener) { + Ok(notification) => notification, + Err(error) => { + let _ = launcher.join(); + return Err(error); + } + }; + if i64::from(notification.syscall) != libc::SYS_getppid { + return Err(io::Error::other("unexpected scalar probe syscall")); + } + listener.respond_value(notification.id, PROBE_VALUE)?; + let observed = launcher + .join() + .map_err(|_| io::Error::other("notification launcher panicked"))??; + if observed != PROBE_VALUE { + return Err(io::Error::other("seccomp response value was not delivered")); + } + Ok(wait_killable) +} + +fn probe_addfd_send() -> io::Result<()> { + let (sender, receiver) = mpsc::sync_channel(1); + let launcher = thread::spawn(move || -> io::Result<()> { + let listener = install_listener(&[libc::SYS_socket])?; + sender + .send(listener) + .map_err(|_| io::Error::other("ADDFD broker disappeared"))?; + // SAFETY: arguments are scalar constants; the intercepted syscall is + // completed by ADDFD-SEND and returns the injected descriptor number. + let injected = unsafe { + libc::syscall( + libc::SYS_socket, + libc::AF_INET, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_TCP, + ) + }; + if injected < 0 { + return Err(io::Error::last_os_error()); + } + let injected = RawFd::try_from(injected) + .map_err(|_| io::Error::other("injected descriptor does not fit RawFd"))?; + // SAFETY: ADDFD-SEND returned one newly owned descriptor to this task. + let injected = unsafe { OwnedFd::from_raw_fd(injected) }; + // SAFETY: `injected` was returned as an open descriptor by the kernel. + let descriptor_flags = unsafe { libc::fcntl(injected.as_raw_fd(), libc::F_GETFD) }; + if descriptor_flags < 0 { + return Err(io::Error::last_os_error()); + } + if descriptor_flags & libc::FD_CLOEXEC == 0 { + return Err(io::Error::other("ADDFD did not preserve close-on-exec")); + } + let mut value = 0_u64; + // SAFETY: eventfd reads exactly one u64 into a valid aligned pointer. + let read = unsafe { + libc::read( + injected.as_raw_fd(), + std::ptr::addr_of_mut!(value).cast(), + size_of::(), + ) + }; + let word_size = isize::try_from(size_of::()).map_err(io::Error::other)?; + if read != word_size || value != 7 { + return Err(io::Error::other("injected eventfd was not usable")); + } + Ok(()) + }); + + let listener = receiver + .recv() + .map_err(|_| io::Error::other("ADDFD launcher disappeared"))?; + let notification = match receive_probe_notification(&listener) { + Ok(notification) => notification, + Err(error) => { + let _ = launcher.join(); + return Err(error); + } + }; + if i64::from(notification.syscall) != libc::SYS_socket { + return Err(io::Error::other("unexpected ADDFD probe syscall")); + } + // SAFETY: eventfd has no pointer arguments and returns an owned descriptor. + let source = unsafe { libc::eventfd(7, libc::EFD_CLOEXEC) }; + if source < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: eventfd returned a new owned descriptor. + let source = unsafe { OwnedFd::from_raw_fd(source) }; + listener.add_fd_and_send(notification.id, source.as_raw_fd(), true)?; + launcher + .join() + .map_err(|_| io::Error::other("ADDFD launcher panicked"))??; + Ok(()) +} + +fn probe_task_memory_copy() -> io::Result<()> { + let source = 0x1122_3344_5566_7788_u64; + let tid = std::process::id(); + let mut source_bytes = [0_u8; size_of::()]; + crate::linux::task_memory::read_exact( + tid, + std::ptr::addr_of!(source) as u64, + &mut source_bytes, + )?; + let mut copied = u64::from_ne_bytes(source_bytes); + if copied != source { + return Err(io::Error::other("task-memory probe read wrong value")); + } + + let replacement = 0xaabb_ccdd_eeff_0011_u64; + crate::linux::task_memory::write_exact( + tid, + std::ptr::addr_of_mut!(copied) as u64, + &replacement.to_ne_bytes(), + )?; + if copied != replacement { + return Err(io::Error::other("task-memory probe wrote wrong value")); + } + Ok(()) +} + +fn probe_connected_sendto_fast_path() -> io::Result<()> { + let mut pair = [-1; 2]; + // SAFETY: `pair` points to storage for exactly two returned descriptors. + let result = unsafe { + libc::socketpair( + libc::AF_UNIX, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + 0, + pair.as_mut_ptr(), + ) + }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: successful socketpair returned two independently owned FDs. + let sender_fd = unsafe { OwnedFd::from_raw_fd(pair[0]) }; + // SAFETY: successful socketpair returned two independently owned FDs. + let receiver_fd = unsafe { OwnedFd::from_raw_fd(pair[1]) }; + + let (sender, receiver) = mpsc::sync_channel(1); + let launcher = thread::spawn(move || -> io::Result<()> { + let listener = install_listener(&[libc::SYS_sendto])?; + sender + .send(listener) + .map_err(|_| io::Error::other("sendto broker disappeared"))?; + + let direct = b"direct"; + // SAFETY: the buffer is live and a null destination on this connected + // socket is equivalent to send(). The filter must allow this call + // without a broker round trip. + let sent = unsafe { + libc::sendto( + sender_fd.as_raw_fd(), + direct.as_ptr().cast(), + direct.len(), + libc::MSG_NOSIGNAL, + std::ptr::null(), + 0, + ) + }; + if sent != isize::try_from(direct.len()).map_err(io::Error::other)? { + return Err(io::Error::last_os_error()); + } + + let mut payload = [0_u8; 6]; + // SAFETY: the receive buffer is live for its full declared length. + let read = unsafe { + libc::read( + receiver_fd.as_raw_fd(), + payload.as_mut_ptr().cast(), + payload.len(), + ) + }; + if read != isize::try_from(payload.len()).map_err(io::Error::other)? || &payload != direct { + return Err(io::Error::other( + "connected sendto fast path did not relay data", + )); + } + + let destination = libc::sockaddr_un { + sun_family: libc::sa_family_t::try_from(libc::AF_UNIX).map_err(io::Error::other)?, + sun_path: [0; 108], + }; + // SAFETY: all pointers refer to live values. This deliberately + // destination-bearing call must be denied by the broker. + let result = unsafe { + libc::sendto( + sender_fd.as_raw_fd(), + direct.as_ptr().cast(), + direct.len(), + 0, + std::ptr::addr_of!(destination).cast(), + libc::socklen_t::try_from(size_of::()) + .map_err(io::Error::other)?, + ) + }; + if result != -1 || io::Error::last_os_error().raw_os_error() != Some(libc::EACCES) { + return Err(io::Error::other( + "destination-bearing sendto bypassed notification", + )); + } + + // A null destination with Fast Open must not use the connected-send + // fast path either. + // SAFETY: the live buffer and null address form a valid syscall; the + // broker supplies the expected denial. + let result = unsafe { + libc::sendto( + sender_fd.as_raw_fd(), + direct.as_ptr().cast(), + direct.len(), + libc::MSG_FASTOPEN, + std::ptr::null(), + 0, + ) + }; + if result != -1 || io::Error::last_os_error().raw_os_error() != Some(libc::EOPNOTSUPP) { + return Err(io::Error::other( + "MSG_FASTOPEN sendto bypassed notification", + )); + } + Ok(()) + }); + + let listener = receiver + .recv() + .map_err(|_| io::Error::other("sendto launcher disappeared"))?; + let destination = match receive_probe_notification(&listener) { + Ok(notification) => notification, + Err(error) => { + let _ = launcher.join(); + return Err(error); + } + }; + if i64::from(destination.syscall) != libc::SYS_sendto + || destination.args[4] == 0 + || destination.args[5] == 0 + { + return Err(io::Error::other( + "destination-bearing sendto notification was malformed", + )); + } + listener.respond_errno(destination.id, libc::EACCES)?; + + let fast_open = match receive_probe_notification(&listener) { + Ok(notification) => notification, + Err(error) => { + let _ = launcher.join(); + return Err(error); + } + }; + if i64::from(fast_open.syscall) != libc::SYS_sendto + || fast_open.args[4] != 0 + || fast_open.args[5] != 0 + || fast_open.args[3] & u64::from(libc::MSG_FASTOPEN as u32) == 0 + { + return Err(io::Error::other( + "Fast Open sendto notification was malformed", + )); + } + listener.respond_errno(fast_open.id, libc::EOPNOTSUPP)?; + + launcher + .join() + .map_err(|_| io::Error::other("sendto launcher panicked"))??; + Ok(()) +} + +fn receive_probe_notification(listener: &NotificationListener) -> io::Result { + let mut descriptor = libc::pollfd { + fd: listener.as_raw_fd(), + events: libc::POLLIN | libc::POLLHUP, + revents: 0, + }; + let timeout = + i32::try_from(PROBE_NOTIFICATION_TIMEOUT.as_millis()).map_err(io::Error::other)?; + // SAFETY: descriptor points to one live pollfd for the duration of poll. + let ready = unsafe { libc::poll(&raw mut descriptor, 1, timeout) }; + if ready < 0 { + return Err(io::Error::last_os_error()); + } + if ready == 0 { + return Err(io::Error::new( + io::ErrorKind::TimedOut, + "seccomp notification probe timed out", + )); + } + if descriptor.revents & libc::POLLIN == 0 { + return Err(io::Error::new( + io::ErrorKind::BrokenPipe, + "seccomp notification probe listener closed", + )); + } + listener.receive() +} + +fn install_listener_with_flags( + syscalls: &[i64], + wait_killable_recv: bool, +) -> io::Result { + let mut program = build_filter(syscalls)?; + let length = u16::try_from(program.len()) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "seccomp filter is too large"))?; + let mut fprog = libc::sock_fprog { + len: length, + filter: program.as_mut_ptr(), + }; + let flags = SECCOMP_FILTER_FLAG_NEW_LISTENER + | if wait_killable_recv { + SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV + } else { + 0 + }; + // SAFETY: `fprog` points to a live classic-BPF program for the duration of + // the syscall. The returned nonnegative value is a newly owned FD. + let result = unsafe { + libc::syscall( + libc::SYS_seccomp, + SECCOMP_SET_MODE_FILTER, + flags, + std::ptr::addr_of_mut!(fprog), + ) + }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + let fd = RawFd::try_from(result) + .map_err(|_| io::Error::other("seccomp listener FD does not fit RawFd"))?; + // SAFETY: successful NEW_LISTENER returns one newly owned descriptor. + let fd = unsafe { OwnedFd::from_raw_fd(fd) }; + Ok(NotificationListener { + fd, + wait_killable_recv, + }) +} + +fn build_filter(syscalls: &[i64]) -> io::Result> { + let mut program = vec![ + stmt(BPF_LD_W_ABS, SECCOMP_DATA_ARCH_OFFSET), + jump(BPF_JMP_JEQ_K, native_audit_arch(), 1, 0), + stmt(BPF_RET_K, SECCOMP_RET_KILL_PROCESS), + stmt(BPF_LD_W_ABS, SECCOMP_DATA_NR_OFFSET), + ]; + + #[cfg(target_arch = "x86_64")] + program.extend([ + jump(BPF_JMP_JSET_K, X32_SYSCALL_BIT, 0, 1), + stmt(BPF_RET_K, SECCOMP_RET_KILL_PROCESS), + ]); + + let mut syscalls = syscalls.to_vec(); + syscalls.sort_unstable(); + syscalls.dedup(); + for syscall in syscalls { + if syscall == libc::SYS_sendto { + append_sendto_filter(&mut program)?; + continue; + } + let syscall = u32::try_from(syscall) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "negative syscall number"))?; + program.extend([ + jump(BPF_JMP_JEQ_K, syscall, 0, 1), + stmt(BPF_RET_K, SECCOMP_RET_USER_NOTIF), + ]); + } + program.push(stmt(BPF_RET_K, SECCOMP_RET_ALLOW)); + Ok(program) +} + +fn append_sendto_filter(program: &mut Vec) -> io::Result<()> { + const SPECIAL_LENGTH: u8 = 20; + let syscall = u32::try_from(libc::SYS_sendto) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "negative sendto syscall"))?; + program.push(jump(BPF_JMP_JEQ_K, syscall, 0, SPECIAL_LENGTH)); + + for offset in [ + argument_word_offset(4, 0), + argument_word_offset(4, 1), + argument_word_offset(5, 0), + argument_word_offset(5, 1), + argument_word_offset(3, 1), + ] { + program.extend([ + stmt(BPF_LD_W_ABS, offset), + jump(BPF_JMP_JEQ_K, 0, 1, 0), + stmt(BPF_RET_K, SECCOMP_RET_USER_NOTIF), + ]); + } + program.extend([ + stmt(BPF_LD_W_ABS, argument_word_offset(3, 0)), + stmt(BPF_ALU_AND_K, !CONNECTED_SEND_FLAGS), + jump(BPF_JMP_JEQ_K, 0, 1, 0), + stmt(BPF_RET_K, SECCOMP_RET_USER_NOTIF), + stmt(BPF_RET_K, SECCOMP_RET_ALLOW), + ]); + Ok(()) +} + +const fn argument_word_offset(argument: u32, word: u32) -> u32 { + SECCOMP_DATA_ARGS_OFFSET + argument * 8 + word * 4 +} + +#[cfg(target_arch = "x86_64")] +const fn native_audit_arch() -> u32 { + 0xc000_003e +} + +#[cfg(target_arch = "aarch64")] +const fn native_audit_arch() -> u32 { + 0xc000_00b7 +} + +const fn stmt(code: u16, value: u32) -> libc::sock_filter { + libc::sock_filter { + code, + jt: 0, + jf: 0, + k: value, + } +} + +const fn jump(code: u16, value: u32, jt: u8, jf: u8) -> libc::sock_filter { + libc::sock_filter { + code, + jt, + jf, + k: value, + } +} + +fn set_no_new_privileges() -> io::Result<()> { + // SAFETY: PR_SET_NO_NEW_PRIVS accepts scalar arguments and only tightens + // the calling thread's privilege behavior. + let result = unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; + if result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } +} + +fn verify_notification_sizes() -> io::Result<()> { + let mut sizes = RawNotificationSizes::default(); + // SAFETY: the kernel writes only the fixed-size `RawNotificationSizes` + // object supplied here. + let result = unsafe { + libc::syscall( + libc::SYS_seccomp, + SECCOMP_GET_NOTIF_SIZES, + 0, + std::ptr::addr_of_mut!(sizes), + ) + }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + for (name, kernel, local) in [ + ( + "notification", + usize::from(sizes.notification), + size_of::(), + ), + ( + "response", + usize::from(sizes.response), + size_of::(), + ), + ("data", usize::from(sizes.data), size_of::()), + ] { + if kernel != local { + return Err(io::Error::new( + io::ErrorKind::Unsupported, + format!("kernel seccomp {name} size {kernel} differs from supported size {local}"), + )); + } + } + Ok(()) +} + +fn ioctl_ptr(fd: RawFd, request: libc::c_ulong, argument: *mut libc::c_void) -> io::Result { + // SAFETY: every caller supplies the UAPI structure encoded into `request`, + // alive and writable for the ioctl duration. + // `libc::ioctl` models the request as `c_ulong` for glibc and `c_int` + // for musl. Linux UAPI request values fit both representations. + #[cfg(target_env = "musl")] + let request = u32::try_from(request).map_err(|_| { + io::Error::new(io::ErrorKind::InvalidInput, "ioctl request exceeds 32 bits") + })?; + #[cfg(target_env = "musl")] + let request = libc::c_int::from_ne_bytes(request.to_ne_bytes()); + let result = unsafe { libc::ioctl(fd, request, argument) }; + if result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(i64::from(result)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn filter_rejects_empty_syscall_set() { + let error = install_listener(&[]).err().expect("empty filter must fail"); + assert_eq!(error.kind(), io::ErrorKind::InvalidInput); + } + + #[test] + fn active_notification_probe_passes() { + let report = probe_notification_api().expect("active notification probe"); + assert!(report.notification_round_trip()); + assert!(report.addfd_send()); + assert!(report.task_memory_copy()); + assert!(report.connected_send_fast_path()); + } + + #[test] + fn errno_response_rejects_nonpositive_values() { + // The input validation occurs before the listener FD is used. + // SAFETY: dup takes one valid descriptor and returns a new descriptor + // or a negative error without modifying memory. + let duplicated = unsafe { libc::dup(libc::STDERR_FILENO) }; + assert!(duplicated >= 0, "duplicate stderr for validation test"); + let listener = NotificationListener { + // SAFETY: successful dup returned a new owned descriptor. + fd: unsafe { OwnedFd::from_raw_fd(duplicated) }, + wait_killable_recv: false, + }; + let error = listener + .respond_errno(1, 0) + .expect_err("zero errno must fail"); + assert_eq!(error.kind(), io::ErrorKind::InvalidInput); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/socket_registry.rs b/crates/openshell-isolation-interface/src/linux/socket_registry.rs new file mode 100644 index 0000000000..81a71920e3 --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/socket_registry.rs @@ -0,0 +1,461 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Bounded registry for socket-time seccomp virtualization. + +#![allow(unsafe_code)] + +use std::collections::BTreeMap; +use std::io; +use std::mem::size_of; +use std::net::SocketAddr; +use std::os::fd::{AsRawFd, BorrowedFd, OwnedFd, RawFd}; + +use rustix::fs::fstat; + +use crate::linux::proc_fd; + +/// Stable identity for one mediated socket within a listener generation. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct SocketIdentity { + /// Generation of the seccomp listener that created the socket. + pub listener_generation: u64, + /// Socket inode observed from the source descriptor. + pub inode: u64, + /// Kernel `SO_COOKIE` value. + pub cookie: u64, +} + +/// Supported INET address family. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum InetFamily { + /// `AF_INET`. + V4, + /// `AF_INET6`. + V6, +} + +/// Supported INET socket kind and protocol. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum InetKind { + /// TCP stream socket. + Tcp, + /// UDP datagram socket restricted to the DNS relay. + DnsUdp, +} + +/// Immutable socket metadata captured before ADDFD-SEND. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct SocketMetadata { + /// Address family. + pub family: InetFamily, + /// Socket kind/protocol. + pub kind: InetKind, + /// Whether the injected descriptor must be close-on-exec. + pub close_on_exec: bool, + /// Whether the socket's open-file description is nonblocking. + pub nonblocking: bool, + /// Task generation that created the socket. + pub creator_generation: u64, +} + +/// Stable state of one socket open-file description. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum SocketState { + /// Created but not bound or connected. + Created, + /// Explicitly bound by the workload. + Bound { local: SocketAddr }, + /// Connected through an external supervisor relay. + Connected { original_peer: SocketAddr }, + /// Connected directly to an allowed workload loopback endpoint. + Local { peer: SocketAddr }, + /// UDP socket pinned to the exact local DNS relay. + DnsUdp { relay: SocketAddr }, + /// TCP socket pinned to the exact local DNS relay. + DnsTcp { relay: SocketAddr }, + /// Workload-owned listening socket. + Listening { local: SocketAddr }, + /// Stream accepted from a verified local peer. + AcceptedLocal { peer: SocketAddr }, + /// A committed relay failed after connection. + Failed { errno: i32 }, +} + +/// One committed registry entry. +#[derive(Debug)] +pub struct SocketEntry { + identity: SocketIdentity, + metadata: SocketMetadata, + state: SocketState, + retained_preconnect: Option, +} + +impl SocketEntry { + /// Stable socket identity. + #[must_use] + pub fn identity(&self) -> SocketIdentity { + self.identity + } + + /// Immutable creation metadata. + #[must_use] + pub fn metadata(&self) -> SocketMetadata { + self.metadata + } + + /// Current stable state. + #[must_use] + pub fn state(&self) -> &SocketState { + &self.state + } + + /// Retained source descriptor used to perform pre-connect operations on + /// the exact injected open-file description. + pub fn retained_preconnect(&self) -> io::Result<&OwnedFd> { + self.retained_preconnect.as_ref().ok_or_else(|| { + io::Error::new( + io::ErrorKind::NotConnected, + "socket no longer has a retained pre-connect descriptor", + ) + }) + } + + /// Replace the stable state. Callers perform policy and notification + /// revalidation before invoking this commit primitive. + pub fn set_state(&mut self, state: SocketState) { + self.state = state; + } + + /// Close the temporary source descriptor after a connection commits. + pub fn release_preconnect(&mut self) { + self.retained_preconnect = None; + } + + /// Verify that the retained source still has the registered cookie and + /// inode. + pub fn validate_retained_identity(&self) -> io::Result<()> { + let retained = self.retained_preconnect()?; + let identity = socket_identity(retained.as_raw_fd(), self.identity.listener_generation)?; + if identity == self.identity { + Ok(()) + } else { + Err(io::Error::new( + io::ErrorKind::InvalidData, + "retained socket identity changed", + )) + } + } +} + +/// Tentative socket metadata that is invisible until ADDFD-SEND succeeds. +#[derive(Debug)] +pub struct TentativeSocket { + identity: SocketIdentity, + metadata: SocketMetadata, + source: OwnedFd, +} + +impl TentativeSocket { + /// Stable identity used to correlate the ADDFD transaction. + #[must_use] + pub fn identity(&self) -> SocketIdentity { + self.identity + } + + /// Source descriptor passed to ADDFD-SEND. + #[must_use] + pub fn source_fd(&self) -> RawFd { + self.source.as_raw_fd() + } +} + +/// Bounded committed socket registry. +pub struct SocketRegistry { + listener_generation: u64, + capacity: usize, + entries: BTreeMap, +} + +impl SocketRegistry { + /// Create an empty registry for one nonzero listener generation. + pub fn new(listener_generation: u64, capacity: usize) -> io::Result { + if listener_generation == 0 || capacity == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "listener generation and registry capacity must be nonzero", + )); + } + Ok(Self { + listener_generation, + capacity, + entries: BTreeMap::new(), + }) + } + + /// Number of committed entries. + #[must_use] + pub fn len(&self) -> usize { + self.entries.len() + } + + /// Whether no sockets are committed. + #[must_use] + pub fn is_empty(&self) -> bool { + self.entries.is_empty() + } + + /// Whether another socket would exceed the configured bound. + #[must_use] + pub fn is_full(&self) -> bool { + self.entries.len() >= self.capacity + } + + /// Retain only sockets that remain installed in a workload descriptor + /// table. The trusted broker's temporary source descriptors are excluded + /// from `installed` by the caller. + pub fn retain_installed(&mut self, installed: &std::collections::BTreeSet) { + self.entries + .retain(|inode, _entry| installed.contains(inode)); + } + + /// Stage a newly created source descriptor without publishing it. + pub fn stage(&self, source: OwnedFd, metadata: SocketMetadata) -> io::Result { + if self.entries.len() >= self.capacity { + return Err(io::Error::from_raw_os_error(libc::EMFILE)); + } + let identity = socket_identity(source.as_raw_fd(), self.listener_generation)?; + if self.entries.contains_key(&identity.inode) { + return Err(io::Error::new( + io::ErrorKind::AlreadyExists, + "socket inode is already registered", + )); + } + Ok(TentativeSocket { + identity, + metadata, + source, + }) + } + + /// Publish a tentative socket only after ADDFD-SEND has succeeded. + pub fn commit(&mut self, tentative: TentativeSocket) -> io::Result { + self.commit_with_state(tentative, SocketState::Created) + } + + /// Publish a tentative socket in a caller-proven initial state. + /// + /// Accepted sockets are created and classified by the trusted broker, so + /// they enter the registry directly as [`SocketState::AcceptedLocal`] + /// rather than pretending to be unconnected. + pub fn commit_with_state( + &mut self, + tentative: TentativeSocket, + state: SocketState, + ) -> io::Result { + if self.entries.len() >= self.capacity { + return Err(io::Error::from_raw_os_error(libc::EMFILE)); + } + if tentative.identity.listener_generation != self.listener_generation + || self.entries.contains_key(&tentative.identity.inode) + { + return Err(io::Error::new( + io::ErrorKind::AlreadyExists, + "socket identity cannot be committed to this registry", + )); + } + let identity = tentative.identity; + let retain_source = matches!( + state, + SocketState::Created | SocketState::Bound { .. } | SocketState::Listening { .. } + ); + self.entries.insert( + identity.inode, + SocketEntry { + identity, + metadata: tentative.metadata, + state, + retained_preconnect: retain_source.then_some(tentative.source), + }, + ); + Ok(identity) + } + + /// Resolve a notifying task's installed descriptor to a committed entry. + pub fn resolve(&self, tid: u32, fd: RawFd) -> io::Result<&SocketEntry> { + let inode = proc_fd::socket_inode(tid, fd)?; + let entry = self.entries.get(&inode).ok_or_else(|| { + io::Error::new( + io::ErrorKind::PermissionDenied, + "socket inode is not registered for this sandbox", + ) + })?; + if entry.retained_preconnect.is_some() { + entry.validate_retained_identity()?; + } + Ok(entry) + } + + /// Mutable form of [`Self::resolve`]. + pub fn resolve_mut(&mut self, tid: u32, fd: RawFd) -> io::Result<&mut SocketEntry> { + let inode = proc_fd::socket_inode(tid, fd)?; + let entry = self.entries.get_mut(&inode).ok_or_else(|| { + io::Error::new( + io::ErrorKind::PermissionDenied, + "socket inode is not registered for this sandbox", + ) + })?; + if entry.retained_preconnect.is_some() { + entry.validate_retained_identity()?; + } + Ok(entry) + } + + /// Remove metadata after descendant-FD collection proves no installed + /// alias remains. + pub fn remove_inode(&mut self, inode: u64) -> bool { + self.entries.remove(&inode).is_some() + } +} + +fn socket_identity(fd: RawFd, listener_generation: u64) -> io::Result { + // SAFETY: `fd` remains open for this function; the borrow never escapes. + let borrowed = unsafe { BorrowedFd::borrow_raw(fd) }; + let stat = fstat(borrowed)?; + if stat.st_mode & libc::S_IFMT != libc::S_IFSOCK { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "registry source descriptor is not a socket", + )); + } + let mut cookie = 0_u64; + let mut length = + libc::socklen_t::try_from(size_of::()).expect("SO_COOKIE length fits socklen_t"); + // SAFETY: getsockopt writes at most the supplied u64 and socklen_t. + let result = unsafe { + libc::getsockopt( + fd, + libc::SOL_SOCKET, + libc::SO_COOKIE, + std::ptr::addr_of_mut!(cookie).cast(), + std::ptr::addr_of_mut!(length), + ) + }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + if usize::try_from(length).ok() != Some(size_of::()) || cookie == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "kernel returned an invalid SO_COOKIE", + )); + } + Ok(SocketIdentity { + listener_generation, + inode: stat.st_ino, + cookie, + }) +} + +#[cfg(test)] +mod tests { + use std::os::fd::FromRawFd; + + use super::*; + + fn tcp_socket() -> OwnedFd { + // SAFETY: socket returns one newly owned descriptor on success. + let fd = unsafe { + libc::socket( + libc::AF_INET, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_TCP, + ) + }; + assert!(fd >= 0, "socket: {}", io::Error::last_os_error()); + // SAFETY: successful socket returned one owned descriptor. + unsafe { OwnedFd::from_raw_fd(fd) } + } + + fn metadata() -> SocketMetadata { + SocketMetadata { + family: InetFamily::V4, + kind: InetKind::Tcp, + close_on_exec: true, + nonblocking: false, + creator_generation: 11, + } + } + + #[test] + fn tentative_entry_is_invisible_until_commit() { + let mut registry = SocketRegistry::new(7, 1).unwrap(); + let socket = tcp_socket(); + let fd = socket.as_raw_fd(); + let tentative = registry.stage(socket, metadata()).unwrap(); + assert!(registry.is_empty()); + assert_eq!( + registry + .resolve(std::process::id(), fd) + .expect_err("tentative socket must be invisible") + .kind(), + io::ErrorKind::PermissionDenied + ); + + let identity = registry.commit(tentative).unwrap(); + let entry = registry.resolve(std::process::id(), fd).unwrap(); + assert_eq!(entry.identity(), identity); + assert_eq!(entry.metadata(), metadata()); + assert_eq!(entry.state(), &SocketState::Created); + assert_eq!(registry.len(), 1); + + assert_eq!( + registry + .stage(tcp_socket(), metadata()) + .expect_err("quota must fail before injection") + .raw_os_error(), + Some(libc::EMFILE) + ); + } + + #[test] + fn dup_alias_resolves_to_same_open_file_description() { + let mut registry = SocketRegistry::new(9, 4).unwrap(); + let socket = tcp_socket(); + let original_fd = socket.as_raw_fd(); + // SAFETY: dup returns a new descriptor for the same open-file + // description or a negative error. + let alias_fd = unsafe { libc::dup(original_fd) }; + assert!(alias_fd >= 0, "dup: {}", io::Error::last_os_error()); + // SAFETY: successful dup returned one owned descriptor. + let alias = unsafe { OwnedFd::from_raw_fd(alias_fd) }; + + let tentative = registry.stage(socket, metadata()).unwrap(); + let identity = registry.commit(tentative).unwrap(); + assert_eq!( + registry + .resolve(std::process::id(), alias.as_raw_fd()) + .unwrap() + .identity(), + identity + ); + } + + #[test] + fn collection_reclaims_only_uninstalled_socket_metadata() { + let mut registry = SocketRegistry::new(12, 2).unwrap(); + let first = registry + .commit(registry.stage(tcp_socket(), metadata()).unwrap()) + .unwrap(); + let second = registry + .commit(registry.stage(tcp_socket(), metadata()).unwrap()) + .unwrap(); + assert!(registry.is_full()); + + registry.retain_installed(&std::collections::BTreeSet::from([first.inode])); + + assert_eq!(registry.len(), 1); + assert!(registry.remove_inode(first.inode)); + assert!(!registry.remove_inode(second.inode)); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/task_memory.rs b/crates/openshell-isolation-interface/src/linux/task_memory.rs new file mode 100644 index 0000000000..4a0422a006 --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/task_memory.rs @@ -0,0 +1,411 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Bounded, exact access to a notifying task's memory. +//! +//! Seccomp user-notification arguments contain addresses in the notifying +//! task. Callers must copy pointer-bearing inputs once into trusted memory and +//! must never treat a partial copy as valid. + +#![allow(unsafe_code)] + +use std::io; +use std::mem::size_of; +use std::os::fd::{AsRawFd, FromRawFd, OwnedFd}; +use std::os::unix::fs::FileExt as _; + +/// Maximum number of task-memory bytes copied by one operation. +pub const MAX_TASK_MEMORY_COPY: usize = 64 * 1024; + +/// Read exactly `destination.len()` bytes from `address` in `tid`. +/// +/// Empty and oversized requests, null addresses, and partial reads fail +/// closed. The caller must still revalidate the notification and task +/// generation after the copy. +pub fn read_exact(tid: u32, address: u64, destination: &mut [u8]) -> io::Result<()> { + validate_request(tid, address, destination.len())?; + let pid = libc::pid_t::try_from(tid) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "TID does not fit pid_t"))?; + let remote_address = usize::try_from(address).map_err(|_| { + io::Error::new( + io::ErrorKind::InvalidInput, + "remote address does not fit usize", + ) + })?; + let local = libc::iovec { + iov_base: destination.as_mut_ptr().cast(), + iov_len: destination.len(), + }; + let remote = libc::iovec { + iov_base: remote_address as *mut libc::c_void, + iov_len: destination.len(), + }; + + // SAFETY: the local iovec spans the caller-provided live buffer. The + // remote address is untrusted but bounded; the kernel validates it in the + // target process and returns EFAULT or a short count when unavailable. + let copied = retry_eintr(|| unsafe { + libc::process_vm_readv( + pid, + std::ptr::addr_of!(local), + 1, + std::ptr::addr_of!(remote), + 1, + 0, + ) + }); + match copied { + Ok(copied) => require_exact(copied, destination.len(), "task-memory read"), + Err(error) if syscall_profile_denied(&error) => { + read_exact_from_proc_mem(tid, address, destination) + } + Err(error) => Err(error), + } +} + +/// Write exactly all of `source` to `address` in `tid`. +/// +/// This is used only for syscall outputs such as `getpeername` and +/// `sendmmsg.msg_len`. Revalidate the notification, task generation, and +/// destination layout immediately before calling it. +pub fn write_exact(tid: u32, address: u64, source: &[u8]) -> io::Result<()> { + validate_request(tid, address, source.len())?; + let pid = libc::pid_t::try_from(tid) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "TID does not fit pid_t"))?; + let remote_address = usize::try_from(address).map_err(|_| { + io::Error::new( + io::ErrorKind::InvalidInput, + "remote address does not fit usize", + ) + })?; + let local = libc::iovec { + iov_base: source.as_ptr().cast_mut().cast(), + iov_len: source.len(), + }; + let remote = libc::iovec { + iov_base: remote_address as *mut libc::c_void, + iov_len: source.len(), + }; + + // SAFETY: the local iovec spans the caller-provided live buffer. The + // remote address is untrusted but bounded; the kernel validates that it is + // writable in the target process. + let copied = retry_eintr(|| unsafe { + libc::process_vm_writev( + pid, + std::ptr::addr_of!(local), + 1, + std::ptr::addr_of!(remote), + 1, + 0, + ) + }); + match copied { + Ok(copied) => require_exact(copied, source.len(), "task-memory write"), + Err(error) if syscall_profile_denied(&error) => { + write_exact_to_proc_mem(tid, address, source) + } + Err(error) => Err(error), + } +} + +fn syscall_profile_denied(error: &io::Error) -> bool { + matches!( + error.raw_os_error(), + Some(libc::EPERM | libc::EACCES | libc::ENOSYS) + ) +} + +fn read_exact_from_proc_mem(tid: u32, address: u64, destination: &mut [u8]) -> io::Result<()> { + let file = std::fs::File::open(format!("/proc/{tid}/mem"))?; + let copied = file.read_at(destination, address)?; + require_exact(copied, destination.len(), "proc task-memory read") +} + +fn write_exact_to_proc_mem(tid: u32, address: u64, source: &[u8]) -> io::Result<()> { + let file = std::fs::OpenOptions::new() + .write(true) + .open(format!("/proc/{tid}/mem"))?; + let copied = file.write_at(source, address)?; + require_exact(copied, source.len(), "proc task-memory write") +} + +/// Prove same-UID parent-to-child read and write access under the active Yama, +/// LSM, and outer seccomp posture. +/// +/// Call this only from a single-threaded probe process. The child executes +/// raw, allocation-free syscalls between `fork` and `_exit`. +pub fn probe_child_access() -> io::Result<()> { + const INITIAL: u64 = 0x1122_3344_5566_7788; + const REPLACEMENT: u64 = 0xaabb_ccdd_eeff_0011; + // SAFETY: mmap creates one private anonymous page owned by this process. + let mapping = unsafe { + libc::mmap( + std::ptr::null_mut(), + size_of::(), + libc::PROT_READ | libc::PROT_WRITE, + libc::MAP_PRIVATE | libc::MAP_ANONYMOUS, + -1, + 0, + ) + }; + if mapping == libc::MAP_FAILED { + return Err(io::Error::last_os_error()); + } + let mapping_address = mapping as u64; + // SAFETY: mapping spans at least one aligned u64-sized region. + unsafe { mapping.cast::().write(INITIAL) }; + + // SAFETY: eventfd returns independently owned descriptors on success. + let ready = unsafe { libc::eventfd(0, libc::EFD_CLOEXEC) }; + if ready < 0 { + // SAFETY: mapping is the live region returned above. + unsafe { libc::munmap(mapping, size_of::()) }; + return Err(io::Error::last_os_error()); + } + // SAFETY: successful eventfd returned one owned descriptor. + let ready = unsafe { OwnedFd::from_raw_fd(ready) }; + // SAFETY: eventfd returns independently owned descriptors on success. + let proceed = unsafe { libc::eventfd(0, libc::EFD_CLOEXEC) }; + if proceed < 0 { + // SAFETY: mapping is the live region returned above. + unsafe { libc::munmap(mapping, size_of::()) }; + return Err(io::Error::last_os_error()); + } + // SAFETY: successful eventfd returned one owned descriptor. + let proceed = unsafe { OwnedFd::from_raw_fd(proceed) }; + + // SAFETY: the caller promises this probe process is single-threaded. The + // child performs only raw syscalls and memory operations before `_exit`. + let child = unsafe { libc::fork() }; + if child < 0 { + // SAFETY: mapping is the live region returned above. + unsafe { libc::munmap(mapping, size_of::()) }; + return Err(io::Error::last_os_error()); + } + if child == 0 { + // The sandbox remains nondumpable, but an exec'd workload must be + // observable by its same-UID ancestor. This child contains no trusted + // parent address space secrets beyond this synthetic probe value. + // SAFETY: these calls use live inherited eventfds and scalar prctl + // arguments. No Rust cleanup runs in the child. + unsafe { + if libc::prctl(libc::PR_SET_DUMPABLE, 1, 0, 0, 0) < 0 + || write_eventfd(ready.as_raw_fd()).is_err() + || read_eventfd(proceed.as_raw_fd()).is_err() + || mapping.cast::().read() != REPLACEMENT + { + libc::_exit(1); + } + libc::_exit(0); + } + } + + let outcome = (|| { + read_eventfd(ready.as_raw_fd())?; + let mut observed = [0_u8; size_of::()]; + read_exact( + u32::try_from(child).map_err(|_| io::Error::other("child PID does not fit u32"))?, + mapping_address, + &mut observed, + )?; + if u64::from_ne_bytes(observed) != INITIAL { + return Err(io::Error::other( + "cross-child memory read returned wrong data", + )); + } + write_exact( + u32::try_from(child).map_err(|_| io::Error::other("child PID does not fit u32"))?, + mapping_address, + &REPLACEMENT.to_ne_bytes(), + )?; + write_eventfd(proceed.as_raw_fd())?; + let mut status = 0; + // SAFETY: child is a live direct child and status points to storage. + if unsafe { libc::waitpid(child, std::ptr::addr_of_mut!(status), 0) } != child { + return Err(io::Error::last_os_error()); + } + if !libc::WIFEXITED(status) || libc::WEXITSTATUS(status) != 0 { + return Err(io::Error::other("cross-child memory probe failed in child")); + } + Ok(()) + })(); + + if outcome.is_err() { + // SAFETY: a failed parent-side operation may leave this direct child + // blocked on eventfd. SIGKILL and waitpid guarantee cleanup. + unsafe { + libc::kill(child, libc::SIGKILL); + libc::waitpid(child, std::ptr::null_mut(), 0); + } + } + // SAFETY: mapping is the live region returned above and no child remains. + unsafe { libc::munmap(mapping, size_of::()) }; + outcome +} + +fn read_eventfd(fd: libc::c_int) -> io::Result<()> { + let mut value = 0_u64; + // SAFETY: eventfd reads exactly one u64 into live storage. + let result = unsafe { libc::read(fd, std::ptr::addr_of_mut!(value).cast(), size_of::()) }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + require_exact( + usize::try_from(result).map_err(|_| io::Error::other("eventfd read length invalid"))?, + size_of::(), + "eventfd read", + ) +} + +fn write_eventfd(fd: libc::c_int) -> io::Result<()> { + let value = 1_u64; + // SAFETY: eventfd reads exactly one u64 from live storage. + let result = unsafe { libc::write(fd, std::ptr::addr_of!(value).cast(), size_of::()) }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + require_exact( + usize::try_from(result).map_err(|_| io::Error::other("eventfd write length invalid"))?, + size_of::(), + "eventfd write", + ) +} +fn validate_request(tid: u32, address: u64, length: usize) -> io::Result<()> { + if tid == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "task-memory TID must be nonzero", + )); + } + if address == 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "task-memory address must be nonzero", + )); + } + if length == 0 || length > MAX_TASK_MEMORY_COPY { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("task-memory copy length must be between 1 and {MAX_TASK_MEMORY_COPY} bytes"), + )); + } + let start = usize::try_from(address) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "remote address is too large"))?; + start.checked_add(length - 1).ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidInput, + "task-memory address range overflows", + ) + })?; + Ok(()) +} + +fn retry_eintr(mut operation: impl FnMut() -> isize) -> io::Result { + loop { + let result = operation(); + if result >= 0 { + return usize::try_from(result) + .map_err(|_| io::Error::other("task-memory result does not fit usize")); + } + let error = io::Error::last_os_error(); + if error.kind() != io::ErrorKind::Interrupted { + return Err(error); + } + } +} + +fn require_exact(copied: usize, expected: usize, operation: &str) -> io::Result<()> { + if copied == expected { + Ok(()) + } else { + Err(io::Error::new( + io::ErrorKind::InvalidData, + format!("{operation} was partial: copied {copied} of {expected} bytes"), + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn reads_and_writes_exact_same_process_memory() { + let source = 0x1122_3344_5566_7788_u64; + let mut destination = 0_u64; + let mut bytes = [0_u8; size_of::()]; + + read_exact( + std::process::id(), + std::ptr::addr_of!(source) as u64, + &mut bytes, + ) + .expect("read source"); + assert_eq!(u64::from_ne_bytes(bytes), source); + + let replacement = 0xaabb_ccdd_eeff_0011_u64; + write_exact( + std::process::id(), + std::ptr::addr_of_mut!(destination) as u64, + &replacement.to_ne_bytes(), + ) + .expect("write destination"); + assert_eq!(destination, replacement); + } + + #[test] + fn proc_mem_fallback_reads_and_writes_exact_memory() { + let source = 0x0102_0304_0506_0708_u64; + let mut destination = 0_u64; + let mut bytes = [0_u8; size_of::()]; + read_exact_from_proc_mem( + std::process::id(), + std::ptr::addr_of!(source) as u64, + &mut bytes, + ) + .expect("read through proc mem"); + assert_eq!(u64::from_ne_bytes(bytes), source); + + write_exact_to_proc_mem( + std::process::id(), + std::ptr::addr_of_mut!(destination) as u64, + &source.to_ne_bytes(), + ) + .expect("write through proc mem"); + assert_eq!(destination, source); + } + + #[test] + fn rejects_invalid_ranges_before_syscall() { + let mut byte = [0_u8; 1]; + assert_eq!( + read_exact(0, 1, &mut byte).expect_err("zero TID").kind(), + io::ErrorKind::InvalidInput + ); + assert_eq!( + read_exact(std::process::id(), 0, &mut byte) + .expect_err("null address") + .kind(), + io::ErrorKind::InvalidInput + ); + assert_eq!( + read_exact(std::process::id(), 1, &mut []) + .expect_err("empty copy") + .kind(), + io::ErrorKind::InvalidInput + ); + assert_eq!( + validate_request(std::process::id(), 1, MAX_TASK_MEMORY_COPY + 1) + .expect_err("oversized copy") + .kind(), + io::ErrorKind::InvalidInput + ); + assert_eq!( + validate_request(std::process::id(), u64::MAX, 2) + .expect_err("overflowing range") + .kind(), + io::ErrorKind::InvalidInput + ); + } +} diff --git a/crates/openshell-isolation-interface/src/linux/workload_launcher.rs b/crates/openshell-isolation-interface/src/linux/workload_launcher.rs new file mode 100644 index 0000000000..8602d9df80 --- /dev/null +++ b/crates/openshell-isolation-interface/src/linux/workload_launcher.rs @@ -0,0 +1,191 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! One-listener launch thread for capability-free workload descendants. +//! +//! Seccomp filters are per-thread. This launcher installs the networking +//! listener without TSYNC, then serializes every fork/exec operation on that +//! thread. Children inherit the filter while the sandbox's broker and +//! lifecycle threads remain unfiltered. The listener moves to the caller over +//! an in-process channel; no descriptor handoff syscall or reusable exception +//! is needed. + +use std::io; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::mpsc; +use std::thread; + +use crate::linux::seccomp_notify::{NotificationListener, install_workload_listener}; + +type LaunchJob = Box; + +/// Serialized child-launch executor whose thread owns the inherited listener +/// filter. +#[derive(Clone)] +pub struct WorkloadLauncher { + jobs: mpsc::SyncSender, + alive: Arc, +} + +impl WorkloadLauncher { + /// Execute one prebuilt spawn operation on the filtered launcher thread. + /// + /// The closure must only perform audited launch work. It must not open an + /// INET socket itself: the launcher is trusted and deliberately has no + /// notification broker. + pub fn execute( + &self, + operation: impl FnOnce() -> T + Send + 'static, + ) -> io::Result { + if !self.alive.load(Ordering::Acquire) { + return Err(io::Error::new( + io::ErrorKind::BrokenPipe, + "workload launcher is not running", + )); + } + let (result_tx, result_rx) = mpsc::sync_channel(1); + self.jobs + .send(Box::new(move || { + let _ = result_tx.send(operation()); + })) + .map_err(|_| io::Error::new(io::ErrorKind::BrokenPipe, "workload launcher stopped"))?; + result_rx.recv().map_err(|_| { + io::Error::new( + io::ErrorKind::BrokenPipe, + "workload launcher dropped the spawn result", + ) + }) + } + + /// Whether the launch thread is still able to accept work. + #[must_use] + pub fn is_alive(&self) -> bool { + self.alive.load(Ordering::Acquire) + } +} + +/// Start the only workload launcher and return its listener to an unfiltered +/// sandbox thread. +pub fn start() -> io::Result<(WorkloadLauncher, NotificationListener)> { + let (jobs_tx, jobs_rx) = mpsc::sync_channel::(64); + let (ready_tx, ready_rx) = mpsc::sync_channel(1); + let alive = Arc::new(AtomicBool::new(true)); + let thread_alive = alive.clone(); + thread::Builder::new() + .name("openshell-workload-launcher".to_string()) + .spawn(move || { + match install_workload_listener() { + Ok(listener) => { + if ready_tx.send(Ok(listener)).is_err() { + thread_alive.store(false, Ordering::Release); + return; + } + } + Err(error) => { + let _ = ready_tx.send(Err(io::Error::new( + error.kind(), + format!("install workload listener: {error}"), + ))); + thread_alive.store(false, Ordering::Release); + return; + } + } + while let Ok(job) = jobs_rx.recv() { + job(); + } + thread_alive.store(false, Ordering::Release); + }) + .map_err(|error| io::Error::other(format!("start workload launcher thread: {error}")))?; + + let listener = ready_rx.recv().map_err(|_| { + io::Error::new( + io::ErrorKind::BrokenPipe, + "workload launcher exited before publishing its listener", + ) + })??; + Ok(( + WorkloadLauncher { + jobs: jobs_tx, + alive, + }, + listener, + )) +} + +#[cfg(test)] +#[allow(unsafe_code)] +mod tests { + use std::mem::size_of; + use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; + + use super::*; + + #[test] + fn one_listener_mediates_launcher_and_inherited_child() { + let (launcher, listener) = start().expect("start launcher"); + let executable = std::env::current_exe().expect("test executable"); + let mut child = launcher + .execute(move || { + let mut command = std::process::Command::new(executable); + command + .arg("--exact") + .arg("linux::workload_launcher::tests::inherited_listener_child") + .arg("--nocapture") + .env("OPENSHELL_WORKLOAD_LAUNCHER_CHILD", "1"); + command.spawn() + }) + .expect("launcher result") + .expect("spawn child"); + let notification = listener.receive().expect("receive child socket"); + assert_eq!(i64::from(notification.syscall), libc::SYS_socket); + assert!( + std::path::Path::new(&format!("/proc/{}/task/{}", child.id(), notification.tid)) + .exists() + ); + // SAFETY: eventfd returns one newly owned descriptor on success. + let eventfd = unsafe { libc::eventfd(7, libc::EFD_CLOEXEC) }; + assert!(eventfd >= 0, "eventfd: {}", io::Error::last_os_error()); + // SAFETY: successful eventfd returned one owned descriptor. + let eventfd = unsafe { OwnedFd::from_raw_fd(eventfd) }; + listener + .add_fd_and_send(notification.id, eventfd.as_raw_fd(), true) + .expect("inject child descriptor"); + assert!(child.wait().expect("wait child").success()); + assert!(launcher.is_alive()); + assert!(listener.as_raw_fd() >= 0); + } + + #[test] + fn inherited_listener_child() { + if std::env::var_os("OPENSHELL_WORKLOAD_LAUNCHER_CHILD").is_none() { + return; + } + // SAFETY: the inherited listener intercepts this scalar socket call + // and returns the descriptor injected by the parent test. + let descriptor = unsafe { + libc::socket( + libc::AF_INET, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_TCP, + ) + }; + assert!(descriptor >= 0, "socket: {}", io::Error::last_os_error()); + let mut value = 0_u64; + // SAFETY: the broker injected an eventfd and `value` is live storage. + let read = unsafe { + libc::read( + descriptor, + std::ptr::addr_of_mut!(value).cast(), + size_of::(), + ) + }; + // SAFETY: descriptor is owned by this process. + unsafe { libc::close(descriptor) }; + assert_eq!( + read, + isize::try_from(size_of::()).expect("u64 size fits") + ); + assert_eq!(value, 7); + } +} diff --git a/crates/openshell-isolation-interface/tests/backend_conformance.rs b/crates/openshell-isolation-interface/tests/backend_conformance.rs new file mode 100644 index 0000000000..73559b203f --- /dev/null +++ b/crates/openshell-isolation-interface/tests/backend_conformance.rs @@ -0,0 +1,907 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Conformance harness for the runtime-selectable contract. +//! +//! Two materially different mock backends (`Primary`, `Secondary`) with +//! distinct concrete state structs (each generic over a marker, so each kind +//! monomorphizes to its own types) prove the registry holds heterogeneous +//! backends behind `dyn` with no enum over concrete state, and that one driver +//! runs both unchanged. + +use std::collections::BTreeMap; +use std::marker::PhantomData; +use std::path::PathBuf; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; + +use async_trait::async_trait; +use openshell_isolation_interface::AgentSpec; +use openshell_isolation_interface::contract::*; +use tokio::sync::oneshot; + +// --------------------------------------------------------------------------- +// Marker kinds: two materially different backends. +// --------------------------------------------------------------------------- + +trait MockKind: Send + Sync + 'static { + const BACKEND_ID: &'static str; + /// Whether this backend can produce a binary digest (a heterogeneity axis: + /// one backend resolves a full identity, the other resolves path-only). + const HAS_DIGEST: bool; +} + +struct Primary; +impl MockKind for Primary { + const BACKEND_ID: &'static str = "mock-primary"; + const HAS_DIGEST: bool = true; +} + +struct Secondary; +impl MockKind for Secondary { + const BACKEND_ID: &'static str = "mock-secondary"; + const HAS_DIGEST: bool = false; +} + +// --------------------------------------------------------------------------- +// Runtime interfaces (shared across kinds where behavior is identical). +// --------------------------------------------------------------------------- + +struct MockProcess { + status: BoundaryExitStatus, + alive: AtomicBool, + signals: Mutex>, +} + +impl MockProcess { + fn new() -> Arc { + Arc::new(Self { + status: BoundaryExitStatus::Exited(0), + alive: AtomicBool::new(true), + signals: Mutex::new(Vec::new()), + }) + } +} + +#[async_trait] +impl BoundaryProcess for MockProcess { + async fn wait(&self) -> Result { + // Stable across repeated calls. + self.alive.store(false, Ordering::SeqCst); + Ok(self.status) + } + async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError> { + if !self.alive.load(Ordering::SeqCst) { + return Err(BackendError::Terminated("process has exited".to_string())); + } + self.signals.lock().unwrap().push(signal); + Ok(()) + } + async fn terminate(&self) -> Result<(), BackendError> { + self.alive + .swap(false, Ordering::SeqCst) + .then_some(()) + .ok_or_else(|| BackendError::Terminated("process has exited".to_string())) + } +} + +/// Mediation source: hands the mediation service a connection carrying its +/// per-connection identity-resolution result. +struct MockSource(PhantomData); + +#[async_trait] +impl NetworkMediationSource for MockSource { + async fn accept_tcp(&self) -> Result { + let (near, _far) = tokio::io::duplex(64); + let (decision, _decision_rx) = oneshot::channel(); + Ok(PendingTcpOpen { + stream: Box::new(near), + binary_identity: Ok(BinaryIdentity { + binary_path: PathBuf::from("/usr/bin/agent"), + binary_digest: K::HAS_DIGEST + .then(|| "00".repeat(32).parse().expect("valid digest")), + ancestors: vec![], + cmdline_paths: vec![], + }), + destination: "203.0.113.10:443".parse().unwrap(), + socket: NetworkSocketMetadata { + socket_cookie: 7, + nonblocking: false, + process_generation: 1, + }, + policy_generation: 1, + timing: MediationTiming::default(), + decision, + }) + } + + async fn accept_dns(&self) -> Result { + let (response, _response_rx) = oneshot::channel(); + Ok(PendingDnsQuery { + message: vec![0; 12], + transport: DnsTransport::Udp, + binary_identity: Err(ResolveError::Failed( + "mock DNS attribution unavailable".to_string(), + )), + timing: MediationTiming::default(), + response, + }) + } +} + +/// An source whose backend cannot attribute the connection: the connection is +/// still delivered, carrying `Err`, so the mediation service denies and audits +/// it. It never authorizes anything. +struct UnattributedSource; + +#[async_trait] +impl NetworkMediationSource for UnattributedSource { + async fn accept_tcp(&self) -> Result { + let (near, _far) = tokio::io::duplex(64); + let (decision, _decision_rx) = oneshot::channel(); + Ok(PendingTcpOpen { + stream: Box::new(near), + binary_identity: Err(ResolveError::Failed("hash unavailable".to_string())), + destination: "203.0.113.10:443".parse().unwrap(), + socket: NetworkSocketMetadata { + socket_cookie: 8, + nonblocking: false, + process_generation: 1, + }, + policy_generation: 1, + timing: MediationTiming::default(), + decision, + }) + } + + async fn accept_dns(&self) -> Result { + Err(BackendError::Unavailable( + "mock DNS mediation unavailable".to_string(), + )) + } +} + +struct MockExec; + +struct MockTerminal { + size: Mutex>, +} + +#[async_trait] +impl BoundaryTerminal for MockTerminal { + async fn resize(&self, cols: u16, rows: u16) -> Result<(), BackendError> { + *self.size.lock().unwrap() = Some((cols, rows)); + Ok(()) + } +} + +#[async_trait] +impl BoundaryExec for MockExec { + async fn exec(&self, spec: ExecSpec) -> Result { + let (_near, far) = tokio::io::duplex(64); + let (out_r, _out_w) = tokio::io::duplex(64); + let (err_r, _err_w) = tokio::io::duplex(64); + let stdin: BoundaryInput = Box::new(far); + let stderr: BoundaryOutput = Box::new(err_r); + let terminal: Arc = Arc::new(MockTerminal { + size: Mutex::new(None), + }); + Ok(ExecSession { + process: MockProcess::new(), + stdin: (!spec.pty).then_some(stdin), + stdout: Box::new(out_r), + stderr: (!spec.pty).then_some(stderr), + terminal: spec.pty.then_some(terminal), + }) + } +} + +struct MockLoopbackConnector; + +#[async_trait] +impl BoundaryLoopbackConnector for MockLoopbackConnector { + async fn connect(&self, _target: LoopbackTarget) -> Result { + let (near, far) = tokio::io::duplex(64); + tokio::spawn(async move { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + let mut far = far; + let mut buf = [0u8; 4]; + if far.read_exact(&mut buf).await.is_ok() { + let _ = far.write_all(&buf).await; + } + }); + Ok(Box::new(near)) + } +} + +// --------------------------------------------------------------------------- +// Boxed lifecycle states (distinct concrete struct per kind). +// --------------------------------------------------------------------------- + +struct MockBound { + source: Arc>, +} +struct MockReady { + _k: PhantomData, +} +struct MockRunning { + process: Arc, + exec: Arc, + loopback_connector: Arc, + _k: PhantomData, +} + +#[async_trait] +impl BoundBoundary for MockBound { + fn network_mediation_source(&self) -> Arc { + self.source.clone() + } + async fn confirm(self: Box) -> Result { + ConfirmedBoundary::try_new( + Box::new(MockReady:: { _k: PhantomData }), + confirmation_evidence(), + &workload_identity(), + ) + } +} + +#[async_trait] +impl ReadyBoundary for MockReady { + async fn start_agent(self: Box) -> Result, BackendError> { + Ok(Box::new(MockRunning:: { + process: MockProcess::new(), + exec: Arc::new(MockExec), + loopback_connector: Arc::new(MockLoopbackConnector), + _k: PhantomData, + })) + } +} + +#[async_trait] +impl RunningBoundary for MockRunning { + fn agent(&self) -> Arc { + self.process.clone() + } + fn exec(&self) -> Arc { + self.exec.clone() + } + fn loopback_connector(&self) -> Arc { + self.loopback_connector.clone() + } + async fn terminate(&self) -> Result<(), BackendError> { + self.process.terminate().await + } +} + +/// One backend per boundary resource: `attach` is atomic and never binds a +/// resource that is already bound to an active boundary, so a second attach +/// against the same mock resource is `Denied`. +struct MockBackend { + attached: AtomicBool, + _k: PhantomData, +} + +impl MockBackend { + fn new() -> Self { + Self { + attached: AtomicBool::new(false), + _k: PhantomData, + } + } +} + +#[async_trait] +impl IsolationBackend for MockBackend { + fn backend_name(&self) -> &'static str { + K::BACKEND_ID + } + async fn attach( + &self, + descriptor: VerifiedBackendDescriptor, + sandbox: SandboxContext, + ) -> Result, BackendError> { + assert_eq!(descriptor.backend_name(), K::BACKEND_ID); + assert!(!sandbox.sandbox_id.is_empty()); + if self.attached.swap(true, Ordering::SeqCst) { + return Err(BackendError::Denied( + "resource is already bound to an active boundary".to_string(), + )); + } + Ok(Box::new(MockBound:: { + source: Arc::new(MockSource(PhantomData)), + })) + } +} + +// --------------------------------------------------------------------------- +// Helpers. +// --------------------------------------------------------------------------- + +fn registry() -> BackendRegistry { + let mut reg = BackendRegistry::new(); + reg.register(Arc::new(MockBackend::::new())) + .expect("register primary"); + reg.register(Arc::new(MockBackend::::new())) + .expect("register secondary"); + reg +} + +fn descriptor(backend_name: &str) -> BackendDescriptor { + BackendDescriptor { + backend_name: backend_name.to_string(), + payload: vec![], + } +} + +fn sandbox_ctx() -> SandboxContext { + SandboxContext { + sandbox_id: "sb-1".to_string(), + session_id: "550e8400-e29b-41d4-a716-446655440000" + .parse() + .expect("valid session ID"), + policy: SandboxPolicy { + version: 1, + filesystem: openshell_core::policy::FilesystemPolicy::default(), + network: openshell_core::policy::NetworkPolicy::default(), + landlock: openshell_core::policy::LandlockPolicy::default(), + process: openshell_core::policy::ProcessPolicy::default(), + }, + agent: AgentSpec { + program: "/bin/true".to_string(), + args: vec![], + workdir: None, + timeout_secs: 0, + interactive: false, + }, + identity: workload_identity(), + } +} + +fn workload_identity() -> ResolvedWorkloadIdentity { + ResolvedWorkloadIdentity::new( + 1000, + 1000, + vec![1000], + "policy".to_string(), + "sha256:test".to_string(), + ) + .unwrap() +} + +fn confirmation_evidence() -> SandboxConfirmEvidence { + SandboxConfirmEvidence { + generation: "generation-1".to_string(), + identity: workload_identity(), + capabilities: CapabilityEvidence { + inheritable: 0, + permitted: 0, + effective: 0, + bounding: 0, + ambient: 0, + }, + no_new_privileges: true, + sandbox_dumpable: false, + child_dumpable: true, + core_limit_zero: true, + native_architecture: std::env::consts::ARCH.to_string(), + kernel_release: "test".to_string(), + seccomp: SeccompEvidence { + new_listener: true, + notification_round_trip: true, + id_validation: true, + addfd_send: true, + retained_socket_operation: true, + proc_fd_identity: true, + task_memory_read: true, + task_memory_write: true, + cancellation: true, + }, + landlock_abi: 3, + landlock_allow_deny: true, + udp_dns_round_trip: true, + tcp_dns_round_trip: true, + tcp_allow_round_trip: true, + tcp_deny_round_trip: true, + authenticated_supervisor: true, + session_id: SandboxSessionId::new(), + driver_fence: DriverFenceEvidence::Vm { + generation: "generation-1".to_string(), + network_device_count: 0, + }, + runtime_exit_terminates_workload: true, + resource_claims: BTreeMap::new(), + } +} + +#[test] +fn driver_fence_evidence_is_backend_specific_and_fail_closed() { + let docker = DriverFenceEvidence::Docker { + container_id: "sha256:container".to_string(), + network_mode: "none".to_string(), + unexpected_networks: Vec::new(), + }; + let kubernetes = DriverFenceEvidence::Kubernetes { + network_policy_uid: "policy-uid".to_string(), + network_policy_resource_version: "42".to_string(), + ingress_isolated: true, + egress_isolated: true, + egress_rule_count: 0, + }; + let vm = DriverFenceEvidence::Vm { + generation: "generation-1".to_string(), + network_device_count: 0, + }; + + assert!(docker.validate().is_ok()); + assert!(kubernetes.validate().is_ok()); + assert!(vm.validate().is_ok()); + + let drifted = DriverFenceEvidence::Docker { + container_id: "sha256:container".to_string(), + network_mode: "bridge".to_string(), + unexpected_networks: vec!["bridge".to_string()], + }; + assert!(drifted.validate().is_err()); +} + +/// The backend-independent supervisor sequence. Identical for every backend: +/// this is the proof that adding a backend needs no supervisor lifecycle change. +async fn drive( + reg: &BackendRegistry, + descriptor: BackendDescriptor, + admitted: &str, +) -> Result, BackendError> { + let (backend, verified) = reg.resolve(descriptor, admitted)?; + let bound = backend.attach(verified, sandbox_ctx()).await?; + // The mediation source is retained before consuming `Bound` and stays + // usable across the confirm/start transitions. + let _ingress = bound.network_mediation_source(); + assert_eq!(bound.host_gateway_ip(), None); + let confirmed = bound.confirm().await?; + confirmed.evidence().validate(&sandbox_ctx().identity)?; + confirmed.into_boundary().start_agent().await +} + +// --------------------------------------------------------------------------- +// Registry and descriptor. +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn registry_selects_correct_backend() { + let reg = registry(); + let (f, _v) = reg + .resolve(descriptor("mock-secondary"), "mock-secondary") + .expect("resolve"); + assert_eq!(f.backend_name(), "mock-secondary"); +} + +#[test] +fn registry_rejects_duplicate_registration() { + let mut reg = BackendRegistry::new(); + reg.register(Arc::new(MockBackend::::new())) + .expect("first"); + let err = reg + .register(Arc::new(MockBackend::::new())) + .expect_err("duplicate must fail"); + assert!(matches!(err, BackendError::Descriptor(_))); +} + +#[test] +fn registry_rejects_unknown_backend() { + let reg = registry(); + let err = reg + .resolve(descriptor("nope"), "nope") + .map(|_| ()) + .expect_err("unknown must fail"); + assert!(matches!(err, BackendError::NotRegistered(_))); +} + +#[test] +fn registry_rejects_descriptor_admission_mismatch_without_fallback() { + let reg = registry(); + // Descriptor names primary, admission says secondary: must fail, and must + // not silently fall back to either backend. + let err = reg + .resolve(descriptor("mock-primary"), "mock-secondary") + .map(|_| ()) + .expect_err("mismatch must fail"); + assert!(matches!(err, BackendError::Descriptor(_))); +} + +// --------------------------------------------------------------------------- +// Lifecycle: one driver, two heterogeneous backends, no consumer change. +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn one_driver_runs_both_backends() { + let reg = registry(); + // The exact same driver code runs a backend with distinct concrete state + // structs; the registry holds them behind `dyn`, no enum. + let primary = drive(®, descriptor("mock-primary"), "mock-primary") + .await + .expect("primary lifecycle"); + let secondary = drive(®, descriptor("mock-secondary"), "mock-secondary") + .await + .expect("secondary lifecycle"); + + // Both expose a usable agent process handle past start_agent. + assert_eq!( + primary.agent().wait().await.expect("wait"), + BoundaryExitStatus::Exited(0) + ); + assert_eq!( + secondary.agent().wait().await.expect("wait"), + BoundaryExitStatus::Exited(0) + ); +} + +#[test] +fn confirmation_constructor_rejects_incomplete_evidence() { + let mut evidence = confirmation_evidence(); + evidence.seccomp.cancellation = false; + let result = ConfirmedBoundary::try_new( + Box::new(MockReady:: { _k: PhantomData }), + evidence, + &workload_identity(), + ); + assert!(matches!(result, Err(BackendError::Confirm(_)))); +} + +#[test] +fn confirmation_constructor_rejects_another_workload_identity() { + let expected = ResolvedWorkloadIdentity::new( + 1001, + 1001, + vec![1001], + "policy".to_string(), + "sha256:test".to_string(), + ) + .unwrap(); + let result = ConfirmedBoundary::try_new( + Box::new(MockReady:: { _k: PhantomData }), + confirmation_evidence(), + &expected, + ); + assert!(matches!(result, Err(BackendError::Confirm(_)))); +} + +#[tokio::test] +async fn one_boundary_termination_does_not_change_another_boundary() { + let reg = registry(); + let primary = drive(®, descriptor("mock-primary"), "mock-primary") + .await + .expect("primary lifecycle"); + let secondary = drive(®, descriptor("mock-secondary"), "mock-secondary") + .await + .expect("secondary lifecycle"); + + primary + .agent() + .terminate() + .await + .expect("terminate primary"); + secondary + .agent() + .signal(BoundarySignal::Term) + .await + .expect("secondary remains active"); +} + +#[tokio::test] +async fn attach_never_binds_an_already_bound_resource() { + let reg = registry(); + // First attach binds the mock resource. + drive(®, descriptor("mock-primary"), "mock-primary") + .await + .expect("first lifecycle"); + // A second attach against the same active boundary must be denied, not + // silently create a second binding. + let err = drive(®, descriptor("mock-primary"), "mock-primary") + .await + .map(|_| ()) + .expect_err("second attach must fail"); + assert_eq!(err.kind(), BackendErrorKind::Denied); +} + +#[tokio::test] +async fn runtime_interfaces_survive_lifecycle_consumption() { + let reg = registry(); + let (backend, verified) = reg + .resolve(descriptor("mock-primary"), "mock-primary") + .expect("resolve"); + let bound = backend + .attach(verified, sandbox_ctx()) + .await + .expect("attach"); + + // Retain the source at Bound, then consume the bound state with confirm. + // The retained Arc must remain usable afterward. + let source = bound.network_mediation_source(); + let confirmed = bound.confirm().await.expect("confirm"); + let _running = confirmed + .into_boundary() + .start_agent() + .await + .expect("start"); + + let conn = source.accept_tcp().await.expect("accept after consumption"); + let identity = conn.binary_identity.expect("identity resolves"); + assert_eq!(identity.binary_path, PathBuf::from("/usr/bin/agent")); +} + +// --------------------------------------------------------------------------- +// Process and I/O. +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn agent_process_survives_and_wait_is_stable() { + let reg = registry(); + let running = drive(®, descriptor("mock-primary"), "mock-primary") + .await + .expect("lifecycle"); + let agent = running.agent(); + // Survives start_agent returning; wait is stable across repeated calls. + assert_eq!( + agent.wait().await.expect("wait 1"), + BoundaryExitStatus::Exited(0) + ); + assert_eq!( + agent.wait().await.expect("wait 2"), + BoundaryExitStatus::Exited(0) + ); + assert!(matches!( + agent.signal(BoundarySignal::Term).await, + Err(BackendError::Terminated(_)) + )); +} + +#[tokio::test] +async fn every_signal_reaches_the_backend_unchanged() { + let process = MockProcess::new(); + for signal in [ + BoundarySignal::Term, + BoundarySignal::Kill, + BoundarySignal::Int, + BoundarySignal::Hup, + ] { + process.signal(signal).await.expect("signal"); + } + assert_eq!( + *process.signals.lock().unwrap(), + vec![ + BoundarySignal::Term, + BoundarySignal::Kill, + BoundarySignal::Int, + BoundarySignal::Hup, + ] + ); +} + +#[tokio::test] +async fn normal_and_signaled_exit_are_distinct_and_stable() { + let signaled = MockProcess { + status: BoundaryExitStatus::Signaled(9), + alive: AtomicBool::new(false), + signals: Mutex::new(Vec::new()), + }; + assert_eq!( + signaled.wait().await.expect("first wait"), + BoundaryExitStatus::Signaled(9) + ); + assert_eq!( + signaled.wait().await.expect("second wait"), + BoundaryExitStatus::Signaled(9) + ); + assert_ne!( + signaled.wait().await.expect("third wait"), + BoundaryExitStatus::Exited(137) + ); +} + +#[tokio::test] +async fn exec_session_owns_its_process_and_streams() { + let reg = registry(); + let running = drive(®, descriptor("mock-primary"), "mock-primary") + .await + .expect("lifecycle"); + let session = running + .exec() + .exec(ExecSpec { + program: "/bin/sh".to_string(), + args: vec!["-c".to_string(), "true".to_string()], + env: vec![], + workdir: None, + pty: false, + }) + .await + .expect("exec"); + // The exec'd process survives `exec` returning, and stdout/stderr are distinct. + assert!(session.stderr.is_some()); + assert!(session.stdin.is_some()); + assert_eq!( + session.process.wait().await.expect("exec wait"), + BoundaryExitStatus::Exited(0) + ); +} + +#[tokio::test] +async fn pty_exec_merges_output_and_supports_resize() { + let session = MockExec + .exec(ExecSpec { + program: "/bin/sh".to_string(), + args: vec![], + env: vec![], + workdir: None, + pty: true, + }) + .await + .expect("pty exec"); + assert!(session.stdin.is_none()); + assert!(session.stderr.is_none()); + session + .terminal + .expect("terminal") + .resize(120, 40) + .await + .expect("resize"); +} + +#[tokio::test] +async fn port_forward_rejects_non_loopback() { + let target = LoopbackTarget::new("8.8.8.8".parse().unwrap(), 53); + assert!(target.is_err()); + let loopback = LoopbackTarget::new("127.0.0.1".parse().unwrap(), 8080).expect("loopback ok"); + assert_eq!(loopback.port(), 8080); +} + +#[tokio::test] +async fn validated_port_forward_stream_remains_usable() { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + let target = LoopbackTarget::new("127.0.0.1".parse().unwrap(), 8080).unwrap(); + let mut stream = MockLoopbackConnector + .connect(target) + .await + .expect("connect"); + stream.write_all(b"ping").await.expect("write"); + let mut response = [0_u8; 4]; + stream.read_exact(&mut response).await.expect("read"); + assert_eq!(&response, b"ping"); +} + +// --------------------------------------------------------------------------- +// Mediation and binary identity. +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn pending_network_open_carries_socket_bound_identity() { + let reg = registry(); + let (backend, verified) = reg + .resolve(descriptor("mock-primary"), "mock-primary") + .expect("resolve"); + let bound = backend + .attach(verified, sandbox_ctx()) + .await + .expect("attach"); + let conn = bound + .network_mediation_source() + .accept_tcp() + .await + .expect("accept"); + let identity = conn.binary_identity.expect("identity resolves"); + assert_eq!(identity.binary_path, PathBuf::from("/usr/bin/agent")); + // A missing digest is `None`, never an empty value. + assert_eq!( + identity.binary_digest.expect("digest").to_string(), + "00".repeat(32) + ); + assert_eq!(conn.destination, "203.0.113.10:443".parse().unwrap()); + assert_eq!(conn.socket.socket_cookie, 7); +} + +#[tokio::test] +async fn missing_digest_is_none_never_empty() { + // The secondary backend resolves path-only identity: the digest is `None`, + // so policy that requires a digest cannot authorize the connection. + let source = MockSource::(PhantomData); + let conn = source.accept_tcp().await.expect("accept"); + let identity = conn.binary_identity.expect("identity resolves"); + assert!(identity.binary_digest.is_none()); +} + +#[test] +fn sha256_digest_rejects_signed_hex_chunks() { + let signed = format!("+0{}", "00".repeat(31)); + assert!(signed.parse::().is_err()); + assert!("00".repeat(32).parse::().is_ok()); +} + +#[tokio::test] +async fn unresolved_identity_travels_with_the_pending_open_and_fails_closed() { + // Attribution failure does not tear down the source: the connection is + // delivered carrying `Err`, and the mediation service denies it. + let source = UnattributedSource; + let conn = source.accept_tcp().await.expect("accept"); + assert!(conn.binary_identity.is_err()); +} + +#[test] +fn workload_identity_rejects_root_and_normalizes_groups() { + assert!( + ResolvedWorkloadIdentity::new(0, 1000, vec![], "policy".into(), "digest".into()).is_err() + ); + let identity = ResolvedWorkloadIdentity::new( + 1000, + 1001, + vec![1003, 1001, 1002, 1003], + "policy".into(), + "digest".into(), + ) + .unwrap(); + assert_eq!(identity.supplementary_gids, vec![1002, 1003]); +} + +#[test] +fn confirmation_evidence_rejects_identity_or_posture_drift() { + let expected = workload_identity(); + let evidence = confirmation_evidence(); + evidence.validate(&expected).unwrap(); + + let mut drifted = confirmation_evidence(); + drifted.capabilities.effective = 1; + assert!(drifted.validate(&expected).is_err()); + + let mut unmanaged = confirmation_evidence(); + unmanaged.runtime_exit_terminates_workload = false; + assert!(unmanaged.validate(&expected).is_err()); + + let different = ResolvedWorkloadIdentity::new( + 1002, + 1000, + vec![1000], + "policy".into(), + "sha256:test".into(), + ) + .unwrap(); + assert!(evidence.validate(&different).is_err()); +} + +// --------------------------------------------------------------------------- +// Errors. +// --------------------------------------------------------------------------- + +#[test] +fn error_kinds_map_to_supervisor_status_classes() { + assert_eq!( + BackendError::Descriptor("x".into()).kind(), + BackendErrorKind::Invalid + ); + assert_eq!( + BackendError::NotRegistered("x".into()).kind(), + BackendErrorKind::Invalid + ); + assert_eq!( + BackendError::Denied("x".into()).kind(), + BackendErrorKind::Denied + ); + assert_eq!( + BackendError::Unavailable("x".into()).kind(), + BackendErrorKind::Unavailable + ); + assert_eq!( + BackendError::Unsupported("x".into()).kind(), + BackendErrorKind::Unsupported + ); + assert_eq!( + BackendError::Attach("x".into()).kind(), + BackendErrorKind::Failed + ); + assert_eq!( + BackendError::Confirm("x".into()).kind(), + BackendErrorKind::Failed + ); + assert_eq!( + BackendError::Terminated("x".into()).kind(), + BackendErrorKind::Terminated + ); +} diff --git a/crates/openshell-sandbox-backend/Cargo.toml b/crates/openshell-sandbox-backend/Cargo.toml new file mode 100644 index 0000000000..6d39fc37d5 --- /dev/null +++ b/crates/openshell-sandbox-backend/Cargo.toml @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-sandbox-backend" +description = "OpenShell Sandbox Protocol and its supervisor-side IsolationBackend implementation" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +openshell-core = { path = "../openshell-core", default-features = false, features = ["jwt"] } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +async-trait = "0.1" +hyper-util = { workspace = true } +prost = { workspace = true } +rcgen = { workspace = true } +rustls = { workspace = true } +rustls-pemfile = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +sha2 = { workspace = true } +socket2 = { workspace = true } +thiserror = { workspace = true } +tokio = { workspace = true } +tokio-rustls = { workspace = true } +tokio-stream = { workspace = true } +tonic = { workspace = true } +tonic-prost = { workspace = true } +tower = { workspace = true } +tracing = { workspace = true } +uuid = { workspace = true } + +[target.'cfg(unix)'.dependencies] +libc = "0.2" + +[target.'cfg(target_os = "linux")'.dependencies] +rustix = { workspace = true, features = ["fs", "process"] } + +[build-dependencies] +protoc-bin-vendored = { workspace = true } +tonic-prost-build = { workspace = true } + +[lints] +workspace = true diff --git a/crates/openshell-sandbox-backend/build.rs b/crates/openshell-sandbox-backend/build.rs new file mode 100644 index 0000000000..8be206f42a --- /dev/null +++ b/crates/openshell-sandbox-backend/build.rs @@ -0,0 +1,22 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +#![allow(unsafe_code)] + +use std::env; + +fn main() -> Result<(), Box> { + println!("cargo:rerun-if-changed=proto/openshell_sandbox.proto"); + + // SAFETY: Cargo build scripts run this setup before starting code generation. + unsafe { + env::set_var("PROTOC", protoc_bin_vendored::protoc_bin_path()?); + env::set_var("PROTOC_INCLUDE", protoc_bin_vendored::include_path()?); + } + + tonic_prost_build::configure() + .build_server(true) + .build_client(true) + .compile_protos(&["proto/openshell_sandbox.proto"], &["proto"])?; + Ok(()) +} diff --git a/crates/openshell-sandbox-backend/proto/openshell_sandbox.proto b/crates/openshell-sandbox-backend/proto/openshell_sandbox.proto new file mode 100644 index 0000000000..a86d911779 --- /dev/null +++ b/crates/openshell-sandbox-backend/proto/openshell_sandbox.proto @@ -0,0 +1,20 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +syntax = "proto3"; + +package openshell.sandbox.protocol.v1; + +// IsolationBoundary multiplexes the OpenShell Sandbox Protocol +// over one authenticated gRPC connection. Each Exchange call is an independent +// HTTP/2 stream; Mediate is the persistent DNS exchange channel. +service IsolationBoundary { + rpc Exchange(stream BoundaryChunk) returns (stream BoundaryChunk); + rpc Mediate(stream BoundaryChunk) returns (stream BoundaryChunk); +} + +// BoundaryChunk carries an ordered fragment of the existing versioned +// isolation protocol. Message boundaries are not semantically significant. +message BoundaryChunk { + bytes data = 1; +} diff --git a/crates/openshell-sandbox-backend/src/boundary_protocol.rs b/crates/openshell-sandbox-backend/src/boundary_protocol.rs new file mode 100644 index 0000000000..e619d68dc2 --- /dev/null +++ b/crates/openshell-sandbox-backend/src/boundary_protocol.rs @@ -0,0 +1,1386 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Versioned control protocol shared by every remote isolation boundary. +//! +//! Drivers choose and provision the transport, but they do not redefine the +//! process lifecycle, streaming, identity, or authentication messages. The +//! supervisor and sandbox exchange these length-delimited JSON frames inside +//! gRPC streams on one mutually authenticated connection. The driver chooses +//! the underlying private Unix socket, TCP connection, or virtio-vsock stream. + +use std::fmt; +use std::io; +use std::io::{Read, Write}; +use std::path::PathBuf; +use std::str::FromStr; + +use openshell_core::SandboxSessionId; +use openshell_core::policy::{ + FilesystemPolicy, LandlockCompatibility, LandlockPolicy, NetworkMode, NetworkPolicy, + ProcessPolicy, ProxyPolicy, SandboxPolicy, +}; +use openshell_isolation_interface::AgentSpec; +use openshell_isolation_interface::contract::Sha256Digest; +use openshell_isolation_interface::contract::{ + BackendDescriptor, BackendError, BinaryIdentity, BoundaryExitStatus, BoundarySignal, + DriverFenceEvidence, ExecSpec, ResolveError, SandboxConfirmEvidence, +}; +use rcgen::{CertificateParams, DnType, ExtendedKeyUsagePurpose, IsCa, KeyPair, KeyUsagePurpose}; +use serde::de::DeserializeOwned; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; + +pub const MAX_CONTROL_FRAME_BYTES: usize = 1024 * 1024; +pub const STREAM_STDIN: u8 = 0; +pub const STREAM_STDOUT: u8 = 1; +pub const STREAM_STDERR: u8 = 2; +pub const STREAM_EXIT: u8 = 3; +pub const STREAM_STDIN_CLOSED: u8 = 4; +/// Supervisor decision for a staged seccomp-mediated TCP open. +pub const STREAM_NETWORK_DECISION: u8 = 5; +pub const MAX_STREAM_FRAME_BYTES: usize = 64 * 1024; + +/// Ephemeral identity of the supervisor process that owns one sandbox runtime. +/// +/// The supervisor generates this value in memory and presents it on every +/// attach, including transport reconnects. The sandbox pins the first value it +/// accepts for its process lifetime, so a replacement supervisor cannot reuse +/// launch credentials to take over an existing runtime generation. +#[derive(Clone, Copy, PartialEq, Eq, Hash)] +pub struct SupervisorInstanceId(uuid::Uuid); + +impl SupervisorInstanceId { + /// Generate a fresh supervisor-process identity. + #[must_use] + pub fn new() -> Self { + Self(uuid::Uuid::new_v4()) + } +} + +impl Default for SupervisorInstanceId { + fn default() -> Self { + Self::new() + } +} + +impl fmt::Display for SupervisorInstanceId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + self.0.hyphenated().fmt(formatter) + } +} + +impl fmt::Debug for SupervisorInstanceId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("SupervisorInstanceId([REDACTED])") + } +} + +impl FromStr for SupervisorInstanceId { + type Err = SupervisorInstanceIdError; + + fn from_str(value: &str) -> Result { + let parsed = uuid::Uuid::parse_str(value).map_err(|_| SupervisorInstanceIdError)?; + if parsed.is_nil() || parsed.hyphenated().to_string() != value { + return Err(SupervisorInstanceIdError); + } + Ok(Self(parsed)) + } +} + +impl Serialize for SupervisorInstanceId { + fn serialize(&self, serializer: S) -> Result + where + S: serde::Serializer, + { + serializer.collect_str(self) + } +} + +impl<'de> Deserialize<'de> for SupervisorInstanceId { + fn deserialize(deserializer: D) -> Result + where + D: serde::Deserializer<'de>, + { + let value = String::deserialize(deserializer)?; + value.parse().map_err(|_| { + serde::de::Error::invalid_value( + serde::de::Unexpected::Str(&value), + &"a non-nil canonical lowercase UUID", + ) + }) + } +} + +/// A supervisor instance ID was not a canonical non-nil UUID. +#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)] +#[error("supervisor instance ID must be a non-nil canonical lowercase UUID")] +pub struct SupervisorInstanceIdError; + +/// Driver-selected byte-stream transport for the `OpenShell` Sandbox Protocol. +/// Authentication is configured separately and is identical for every variant. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "kebab-case", deny_unknown_fields)] +pub enum SandboxTransport { + Unix { + socket_path: PathBuf, + }, + Tcp { + /// Stable logical Kubernetes Service authority used for diagnostics. + authority: String, + /// Explicit connection candidates resolved by the compute driver. + addresses: Vec, + }, + Vsock { + guest_cid: u32, + port: u32, + }, +} + +/// Supervisor-side, generation-pinned TLS server authentication. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct SandboxTlsClientConfig { + pub server_name: String, + pub trust_anchor_pem: String, +} + +impl fmt::Debug for SandboxTlsClientConfig { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SandboxTlsClientConfig") + .field("server_name", &self.server_name) + .field("trust_anchor_pem", &"") + .finish() + } +} + +/// Sandbox-side TLS server files staged by the compute driver. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct SandboxTlsServerConfig { + pub certificate_chain_path: PathBuf, + pub private_key_path: PathBuf, +} + +/// Fresh, server-only TLS material for one sandbox session. +#[derive(Clone)] +pub struct SandboxTlsMaterial { + pub server_name: String, + pub trust_anchor_pem: String, + pub certificate_chain_pem: String, + pub private_key_pem: String, +} + +impl fmt::Debug for SandboxTlsMaterial { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SandboxTlsMaterial") + .field("server_name", &self.server_name) + .field("trust_anchor_pem", &"") + .field("certificate_chain_pem", &"") + .field("private_key_pem", &"") + .finish() + } +} + +/// Generate a generation-pinned TLS server identity for the Sandbox Protocol. +/// +/// The CA private key is local to this function and is discarded after the +/// server leaf is signed. The JWT, rather than certificate time, determines +/// caller authorization and renewal. +pub fn generate_sandbox_tls_material( + session_id: SandboxSessionId, +) -> Result { + let server_name = format!("sandbox.{session_id}.openshell.internal"); + let ca_key = KeyPair::generate_for(&rcgen::PKCS_ED25519) + .map_err(|error| BackendError::Descriptor(format!("generate sandbox CA key: {error}")))?; + let mut ca_params = CertificateParams::default(); + ca_params.not_before = rcgen::date_time_ymd(1975, 1, 1); + ca_params.not_after = rcgen::date_time_ymd(4096, 1, 1); + ca_params.is_ca = IsCa::Ca(rcgen::BasicConstraints::Unconstrained); + ca_params + .distinguished_name + .push(DnType::CommonName, "OpenShell sandbox session CA"); + ca_params.key_usages = vec![KeyUsagePurpose::KeyCertSign, KeyUsagePurpose::CrlSign]; + let ca = ca_params.self_signed(&ca_key).map_err(|error| { + BackendError::Descriptor(format!("generate sandbox CA certificate: {error}")) + })?; + + let sandbox_key = KeyPair::generate_for(&rcgen::PKCS_ED25519).map_err(|error| { + BackendError::Descriptor(format!("generate sandbox TLS server key: {error}")) + })?; + let mut sandbox_params = CertificateParams::new(vec![server_name.clone()]) + .map_err(|error| BackendError::Descriptor(format!("build sandbox certificate: {error}")))?; + sandbox_params.not_before = rcgen::date_time_ymd(1975, 1, 1); + sandbox_params.not_after = rcgen::date_time_ymd(4096, 1, 1); + sandbox_params + .distinguished_name + .push(DnType::CommonName, "OpenShell sandbox runtime"); + sandbox_params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ServerAuth]; + let sandbox = sandbox_params + .signed_by(&sandbox_key, &ca, &ca_key) + .map_err(|error| { + BackendError::Descriptor(format!("sign sandbox TLS server certificate: {error}")) + })?; + + Ok(SandboxTlsMaterial { + server_name, + trust_anchor_pem: ca.pem(), + certificate_chain_pem: sandbox.pem(), + private_key_pem: sandbox_key.serialize_pem(), + }) +} + +/// Boundary-side listener provisioned by a compute driver. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "kebab-case", deny_unknown_fields)] +pub enum BoundaryListener { + /// TLS over a private Unix socket shared with the host supervisor. + Unix { + socket_path: PathBuf, + tls: SandboxTlsServerConfig, + }, + /// TLS over TCP. An unspecified IP is valid for the sandbox bind. + TlsTcp { + address: std::net::SocketAddr, + tls: SandboxTlsServerConfig, + }, + /// TLS over guest `AF_VSOCK`. + Vsock { + control_port: u32, + tls: SandboxTlsServerConfig, + }, +} + +/// Public verification key staged in the sandbox's immutable auth bundle. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GatewayVerificationKey { + pub key_id: String, + pub public_key_pem: String, +} + +/// Protected runtime descriptor consumed by `openshell-supervisor`. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct SandboxRuntimeDescriptor { + /// Stable identity of the boundary, normally the sandbox ID. + pub boundary_id: String, + /// Immutable driver-owned workload generation. + pub generation: String, + /// Fresh gateway-issued identity for this exact launch. + pub session_id: SandboxSessionId, + /// Immutable numeric identity already applied to the sandbox workload. + pub workload_identity: openshell_isolation_interface::contract::ResolvedWorkloadIdentity, + /// Driver-provisioned byte-stream endpoint. + pub transport: SandboxTransport, + /// Per-generation pinned TLS server identity. + pub tls: SandboxTlsClientConfig, + /// Trusted dial target for well-known host-gateway aliases, when the + /// network supervisor cannot use the boundary's resolver view. + #[serde(default)] + pub host_gateway_ip: Option, + /// Driver-specific immutable resource coordinates bound at attach (for + /// example pod UID, VM generation, or container ID). + #[serde(default)] + pub resource_claims: std::collections::BTreeMap, + /// Concrete outer-fence evidence validated by the driver. + pub driver_fence: DriverFenceEvidence, +} + +impl fmt::Debug for SandboxRuntimeDescriptor { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("SandboxRuntimeDescriptor") + .field("boundary_id", &self.boundary_id) + .field("generation", &self.generation) + .field("session_id", &self.session_id) + .field("transport", &self.transport) + .field("tls", &self.tls) + .field("host_gateway_ip", &self.host_gateway_ip) + .field("resource_claims", &self.resource_claims) + .field("driver_fence", &self.driver_fence) + .finish() + } +} + +impl SandboxRuntimeDescriptor { + /// Encode this runtime configuration for the `openshell-sandbox` backend. + pub fn backend_descriptor(&self) -> Result { + let payload = serde_json::to_vec(self).map_err(|error| { + BackendError::Descriptor(format!("encode runtime descriptor: {error}")) + })?; + Ok(BackendDescriptor { + backend_name: crate::BACKEND_NAME.to_string(), + payload, + }) + } +} + +/// Protected bootstrap configuration consumed by `openshell-sandbox`. +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct BoundaryConfig { + /// Stable identity expected in every authenticated request. + pub boundary_id: String, + /// Immutable driver-owned workload generation. + pub generation: String, + /// Fresh gateway-issued identity for this exact launch. + pub session_id: SandboxSessionId, + /// Monotonic signed supervisor replacement within this generation. + pub session_rotation: openshell_core::jwt::SessionRotation, + /// Durable authorization epoch required in every Sandbox Protocol JWT. + pub auth_epoch: openshell_core::jwt::CredentialEpoch, + /// Gateway identity expected in Sandbox Protocol JWTs. + pub gateway_id: String, + /// Immutable current and staged-next gateway verification keys. + pub verification_keys: Vec, + /// Driver-provisioned listener. + pub listener: BoundaryListener, + /// Immutable coordinates the boundary requires from the control-side + /// runtime descriptor before accepting attachment. + #[serde(default)] + pub resource_claims: std::collections::BTreeMap, + /// Driver-provisioned, read-only runtime evidence for resource claims. + /// + /// Each entry maps a claim key to an absolute file whose trimmed contents + /// must equal the corresponding value in `resource_claims` before the + /// boundary opens its listener. Kubernetes uses this to bind a one-use + /// bootstrap bundle to the admitted workload Pod UID exposed by the + /// Downward API. Other drivers may leave the map empty. + #[serde(default)] + pub resource_claim_files: std::collections::BTreeMap, + /// Exact identity already applied by the runtime to the sandbox process. + pub workload_identity: openshell_isolation_interface::contract::ResolvedWorkloadIdentity, + /// Concrete outer-fence evidence validated by the driver. + pub driver_fence: DriverFenceEvidence, + /// Driver-resolved environment exposed only to workload processes. + #[serde(default)] + pub child_env: std::collections::HashMap, +} + +impl fmt::Debug for BoundaryConfig { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("BoundaryConfig") + .field("boundary_id", &self.boundary_id) + .field("generation", &self.generation) + .field("session_id", &self.session_id) + .field("session_rotation", &self.session_rotation) + .field("auth_epoch", &self.auth_epoch) + .field("gateway_id", &self.gateway_id) + .field( + "verification_key_ids", + &self + .verification_keys + .iter() + .map(|key| key.key_id.as_str()) + .collect::>(), + ) + .field("listener", &self.listener) + .field("resource_claims", &self.resource_claims) + .field("resource_claim_files", &self.resource_claim_files) + .field("workload_identity", &self.workload_identity) + .field("driver_fence", &self.driver_fence) + .field("child_env_keys", &self.child_env.keys().collect::>()) + .finish() + } +} + +impl BoundaryConfig { + /// Serialize the protected driver-owned boundary configuration. + pub fn encode(&self) -> Result, BackendError> { + serde_json::to_vec(self) + .map_err(|error| BackendError::Descriptor(format!("encode boundary config: {error}"))) + } +} + +/// Validate driver-specific immutable coordinates before a boundary binds them. +/// +/// Claim values are opaque to the common protocol, but empty or +/// whitespace-bearing identifiers cannot safely distinguish runtime objects. +pub fn validate_resource_claims( + claims: &std::collections::BTreeMap, +) -> Result<(), BackendError> { + for (key, value) in claims { + if key.is_empty() || key.chars().any(char::is_whitespace) { + return Err(BackendError::Descriptor( + "boundary resource-claim keys must be non-empty and contain no whitespace" + .to_string(), + )); + } + if value.is_empty() || value.chars().any(char::is_whitespace) { + return Err(BackendError::Descriptor(format!( + "boundary resource claim {key:?} must be non-empty and contain no whitespace" + ))); + } + } + Ok(()) +} + +pub async fn write_stream_frame( + writer: &mut (impl AsyncWrite + Unpin), + channel: u8, + payload: &[u8], +) -> io::Result<()> { + if payload.len() > MAX_STREAM_FRAME_BYTES { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "boundary stream frame exceeds limit", + )); + } + writer.write_u8(channel).await?; + writer + .write_u32(payload.len().try_into().map_err(|_| { + io::Error::new( + io::ErrorKind::InvalidData, + "boundary stream frame length overflow", + ) + })?) + .await?; + writer.write_all(payload).await?; + writer.flush().await +} + +pub async fn read_stream_frame( + reader: &mut (impl AsyncRead + Unpin), +) -> io::Result)>> { + let channel = match reader.read_u8().await { + Ok(channel) => channel, + Err(error) if error.kind() == io::ErrorKind::UnexpectedEof => return Ok(None), + Err(error) => return Err(error), + }; + let declared = reader.read_u32().await? as usize; + if declared > MAX_STREAM_FRAME_BYTES { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!("boundary stream frame is too large: {declared} bytes"), + )); + } + let mut payload = vec![0; declared]; + reader.read_exact(&mut payload).await?; + Ok(Some((channel, payload))) +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct RequestEnvelope { + /// Cryptographically random idempotency key scoped to one sandbox generation. + pub request_id: String, + /// SHA-256 of the canonically serialized request payload. + pub payload_digest: String, + pub request: Request, +} + +impl RequestEnvelope { + /// Build a request envelope with a fresh idempotency key and normalized + /// payload digest. + pub fn new(request: Request) -> Result { + let payload_digest = request_payload_digest(&request)?; + Ok(Self { + request_id: uuid::Uuid::new_v4().to_string(), + payload_digest, + request, + }) + } + + /// Verify that the request body still matches the immutable digest bound + /// to this idempotency key. + pub fn validate_payload_digest(&self) -> Result<(), FrameError> { + let actual = request_payload_digest(&self.request)?; + if actual == self.payload_digest { + Ok(()) + } else { + Err(FrameError::PayloadDigestMismatch) + } + } +} + +fn request_payload_digest(request: &Request) -> Result { + // Round-tripping through Value canonicalizes every JSON object by key. In + // particular, this makes HashMap-backed provider environments stable + // across process restarts and independently serialized retries. + let normalized = serde_json::to_value(request).map_err(FrameError::Serialize)?; + let payload = serde_json::to_vec(&normalized).map_err(FrameError::Serialize)?; + let digest = Sha256::digest(payload); + Ok(format!("{digest:x}")) +} + +impl fmt::Debug for RequestEnvelope { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("RequestEnvelope") + .field("request_id", &self.request_id) + .field("request", &self.request) + .finish() + } +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "operation", rename_all = "snake_case")] +pub enum Request { + Attach { + supervisor_instance_id: SupervisorInstanceId, + policy: Box, + resource_claims: std::collections::BTreeMap, + }, + Confirm, + StartAgent { + sandbox_id: String, + spec: AgentSpecWire, + policy: Box, + ca_cert: Option>, + ca_bundle: Option>, + provider_env_revision: u64, + provider_env: std::collections::HashMap, + }, + UpdateProviderEnvironment { + expected_revision: u64, + revision: u64, + provider_env: std::collections::HashMap, + }, + AttachProcess { + process_id: String, + }, + Wait { + process_id: String, + }, + Signal { + process_id: String, + signal: SignalWire, + }, + Terminate { + process_id: String, + }, + /// Permanently end this sandbox session and acknowledge that every owned + /// workload process has been terminated. + TerminateBoundary, + Exec { + spec: ExecSpecWire, + }, + ExecSignal { + process_id: String, + signal: SignalWire, + }, + Resize { + process_id: String, + cols: u16, + rows: u16, + }, + LoopbackConnect { + host: std::net::IpAddr, + port: u16, + }, + /// Upgrade one authenticated logical stream into the persistent DNS data + /// plane. + OpenMediation, + AcceptNetwork, +} + +impl Request { + /// Whether this control-path request changes generation-owned sandbox + /// state and therefore must be replayed from the idempotency ledger. + #[must_use] + pub const fn is_replayable_mutation(&self) -> bool { + matches!( + self, + Self::Attach { .. } + | Self::Confirm + | Self::StartAgent { .. } + | Self::UpdateProviderEnvironment { .. } + | Self::Exec { .. } + | Self::Signal { .. } + | Self::Terminate { .. } + | Self::TerminateBoundary + | Self::ExecSignal { .. } + | Self::Resize { .. } + ) + } +} + +impl fmt::Debug for Request { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Attach { + supervisor_instance_id: _, + policy: _, + resource_claims, + } => formatter + .debug_struct("Attach") + .field("policy", &"") + .field("resource_claims", resource_claims) + .finish(), + Self::Confirm => formatter.write_str("Confirm"), + Self::StartAgent { + sandbox_id, + spec, + policy: _, + ca_cert, + ca_bundle, + provider_env_revision, + provider_env, + } => formatter + .debug_struct("StartAgent") + .field("sandbox_id", sandbox_id) + .field("spec", spec) + .field("policy", &"") + .field("ca_cert_present", &ca_cert.is_some()) + .field("ca_bundle_present", &ca_bundle.is_some()) + .field("provider_env_revision", provider_env_revision) + .field( + "provider_env_keys", + &provider_env.keys().collect::>(), + ) + .finish(), + Self::UpdateProviderEnvironment { + expected_revision, + revision, + provider_env, + } => formatter + .debug_struct("UpdateProviderEnvironment") + .field("expected_revision", expected_revision) + .field("revision", revision) + .field( + "provider_env_keys", + &provider_env.keys().collect::>(), + ) + .finish(), + Self::Wait { process_id } => formatter + .debug_struct("Wait") + .field("process_id", process_id) + .finish(), + Self::AttachProcess { process_id } => formatter + .debug_struct("AttachProcess") + .field("process_id", process_id) + .finish(), + Self::Signal { process_id, signal } => formatter + .debug_struct("Signal") + .field("process_id", process_id) + .field("signal", signal) + .finish(), + Self::Terminate { process_id } => formatter + .debug_struct("Terminate") + .field("process_id", process_id) + .finish(), + Self::TerminateBoundary => formatter.write_str("TerminateBoundary"), + Self::Exec { spec } => formatter.debug_tuple("Exec").field(spec).finish(), + Self::ExecSignal { process_id, signal } => formatter + .debug_struct("ExecSignal") + .field("process_id", process_id) + .field("signal", signal) + .finish(), + Self::Resize { + process_id, + cols, + rows, + } => formatter + .debug_struct("Resize") + .field("process_id", process_id) + .field("cols", cols) + .field("rows", rows) + .finish(), + Self::LoopbackConnect { host, port } => formatter + .debug_struct("LoopbackConnect") + .field("host", host) + .field("port", port) + .finish(), + Self::OpenMediation => formatter.write_str("OpenMediation"), + Self::AcceptNetwork => formatter.write_str("AcceptNetwork"), + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ResponseEnvelope { + pub request_id: String, + pub response: Response, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "result", rename_all = "snake_case")] +pub enum Response { + Attached { + snapshot: SessionSnapshotWire, + }, + Confirmed { + /// Measured capability-free posture produced before workload launch. + evidence: Box, + }, + Started { + process_id: String, + provider_env_revision: u64, + }, + ProviderEnvironmentUpdated { + revision: u64, + }, + ProcessAttached { + terminal: bool, + }, + Exited { + status: ExitStatusWire, + }, + Signaled, + Terminated, + BoundaryTerminated, + ExecStarted { + process_id: String, + pty: bool, + }, + Resized, + PortConnected, + MediationReady, + NetworkConnected { + identity: BinaryIdentityWire, + destination: std::net::SocketAddr, + socket: openshell_isolation_interface::contract::NetworkSocketMetadata, + policy_generation: u64, + timing: MediationTimingWire, + }, + Error { + kind: BoundaryErrorKind, + message: String, + }, +} + +/// Sandbox-monotonic timing carried across the boundary protocol. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct MediationTimingWire { + pub notification_to_queue_us: u64, + pub queue_wait_us: u64, +} + +/// Boundary-owned process/session state returned on every supervisor attach. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct SessionSnapshotWire { + pub generation: String, + pub processes: Vec, +} + +/// Stable generation-scoped process state available to a replacement supervisor. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ProcessSnapshotWire { + pub process_id: String, + pub kind: ProcessKindWire, + pub terminal: bool, + pub status: Option, + pub retained_output: OutputWindowWire, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ProcessKindWire { + Main, + Exec, +} + +/// Sequence range retained by the sandbox output ring. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct OutputWindowWire { + pub first_sequence: u64, + pub next_sequence: u64, + pub truncated: bool, +} + +/// Completion of one sandbox-local DNS relay exchange. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "result", content = "value", rename_all = "snake_case")] +pub enum DnsQueryResultWire { + Response(Vec), + Error(String), +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum BoundaryErrorKind { + Invalid, + Denied, + Unavailable, + Terminated, + Process, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "result", rename_all = "snake_case", deny_unknown_fields)] +pub enum BinaryIdentityWire { + Resolved { + binary_path: PathBuf, + binary_digest: Option, + ancestors: Vec, + cmdline_paths: Vec, + }, + Failed { + message: String, + }, +} + +impl From> for BinaryIdentityWire { + fn from(identity: Result) -> Self { + match identity { + Ok(identity) => Self::Resolved { + binary_path: identity.binary_path, + binary_digest: identity.binary_digest, + ancestors: identity.ancestors, + cmdline_paths: identity.cmdline_paths, + }, + Err(error) => Self::Failed { + message: error.to_string(), + }, + } + } +} + +impl BinaryIdentityWire { + pub fn into_result(self) -> Result { + match self { + Self::Resolved { + binary_path, + binary_digest, + ancestors, + cmdline_paths, + } => Ok(BinaryIdentity { + binary_path, + binary_digest, + ancestors, + cmdline_paths, + }), + Self::Failed { message } => Err(ResolveError::Failed(message)), + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ExecSpecWire { + pub program: String, + pub args: Vec, + pub env: Vec<(String, String)>, + pub workdir: Option, + pub pty: bool, +} + +impl From for ExecSpecWire { + fn from(spec: ExecSpec) -> Self { + Self { + program: spec.program, + args: spec.args, + env: spec.env, + workdir: spec.workdir, + pty: spec.pty, + } + } +} + +impl From for ExecSpec { + fn from(spec: ExecSpecWire) -> Self { + Self { + program: spec.program, + args: spec.args, + env: spec.env, + workdir: spec.workdir, + pty: spec.pty, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct AgentSpecWire { + pub program: String, + pub args: Vec, + pub workdir: Option, + pub timeout_secs: u64, + pub interactive: bool, +} + +impl From for AgentSpecWire { + fn from(spec: AgentSpec) -> Self { + Self { + program: spec.program, + args: spec.args, + workdir: spec.workdir, + timeout_secs: spec.timeout_secs, + interactive: spec.interactive, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct SandboxPolicyWire { + pub version: u32, + pub read_only: Vec, + pub read_write: Vec, + pub include_workdir: bool, + pub network: NetworkModeWire, + pub proxy_addr: Option, + pub landlock: LandlockCompatibilityWire, + pub run_as_user: Option, + pub run_as_group: Option, +} + +impl From for SandboxPolicyWire { + fn from(policy: SandboxPolicy) -> Self { + // Exhaustively destructure the policy so adding a `SandboxPolicy` + // field is a compile error here instead of a silently dropped field + // across the host-to-guest trust boundary. + let SandboxPolicy { + version, + filesystem, + network, + landlock, + process, + } = policy; + let FilesystemPolicy { + read_only, + read_write, + include_workdir, + } = filesystem; + let NetworkPolicy { mode, proxy } = network; + let LandlockPolicy { compatibility } = landlock; + let ProcessPolicy { + run_as_user, + run_as_group, + } = process; + Self { + version, + read_only, + read_write, + include_workdir, + network: NetworkModeWire::from(mode), + proxy_addr: proxy.and_then(|proxy| proxy.http_addr), + landlock: LandlockCompatibilityWire::from(compatibility), + run_as_user, + run_as_group, + } + } +} + +impl From for SandboxPolicy { + fn from(policy: SandboxPolicyWire) -> Self { + let proxy = matches!(policy.network, NetworkModeWire::Proxy).then_some(ProxyPolicy { + http_addr: policy.proxy_addr, + }); + Self { + version: policy.version, + filesystem: FilesystemPolicy { + read_only: policy.read_only, + read_write: policy.read_write, + include_workdir: policy.include_workdir, + }, + network: NetworkPolicy { + mode: policy.network.into(), + proxy, + }, + landlock: LandlockPolicy { + compatibility: policy.landlock.into(), + }, + process: ProcessPolicy { + run_as_user: policy.run_as_user, + run_as_group: policy.run_as_group, + }, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum NetworkModeWire { + Block, + Proxy, + Allow, +} + +impl From for NetworkModeWire { + fn from(mode: NetworkMode) -> Self { + match mode { + NetworkMode::Block => Self::Block, + NetworkMode::Proxy => Self::Proxy, + NetworkMode::Allow => Self::Allow, + } + } +} + +impl From for NetworkMode { + fn from(mode: NetworkModeWire) -> Self { + match mode { + NetworkModeWire::Block => Self::Block, + NetworkModeWire::Proxy => Self::Proxy, + NetworkModeWire::Allow => Self::Allow, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum LandlockCompatibilityWire { + BestEffort, + HardRequirement, +} + +impl From for LandlockCompatibilityWire { + fn from(compatibility: LandlockCompatibility) -> Self { + match compatibility { + LandlockCompatibility::BestEffort => Self::BestEffort, + LandlockCompatibility::HardRequirement => Self::HardRequirement, + } + } +} + +impl From for LandlockCompatibility { + fn from(compatibility: LandlockCompatibilityWire) -> Self { + match compatibility { + LandlockCompatibilityWire::BestEffort => Self::BestEffort, + LandlockCompatibilityWire::HardRequirement => Self::HardRequirement, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum SignalWire { + Term, + Kill, + Int, + Hup, +} + +impl From for SignalWire { + fn from(signal: BoundarySignal) -> Self { + match signal { + BoundarySignal::Term => Self::Term, + BoundarySignal::Kill => Self::Kill, + BoundarySignal::Int => Self::Int, + BoundarySignal::Hup => Self::Hup, + } + } +} + +impl From for BoundarySignal { + fn from(signal: SignalWire) -> Self { + match signal { + SignalWire::Term => Self::Term, + SignalWire::Kill => Self::Kill, + SignalWire::Int => Self::Int, + SignalWire::Hup => Self::Hup, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", content = "value", rename_all = "snake_case")] +pub enum ExitStatusWire { + Exited(i32), + Signaled(i32), +} + +impl From for BoundaryExitStatus { + fn from(status: ExitStatusWire) -> Self { + match status { + ExitStatusWire::Exited(code) => Self::Exited(code), + ExitStatusWire::Signaled(signal) => Self::Signaled(signal), + } + } +} + +impl From for ExitStatusWire { + fn from(status: BoundaryExitStatus) -> Self { + match status { + BoundaryExitStatus::Exited(code) => Self::Exited(code), + BoundaryExitStatus::Signaled(signal) => Self::Signaled(signal), + } + } +} + +pub fn encode_frame(message: &T) -> Result, FrameError> { + let payload = serde_json::to_vec(message).map_err(FrameError::Serialize)?; + if payload.len() > MAX_CONTROL_FRAME_BYTES { + return Err(FrameError::TooLarge(payload.len())); + } + let length = u32::try_from(payload.len()).map_err(|_| FrameError::TooLarge(payload.len()))?; + let mut frame = Vec::with_capacity(4 + payload.len()); + frame.extend_from_slice(&length.to_be_bytes()); + frame.extend_from_slice(&payload); + Ok(frame) +} + +pub fn decode_frame(frame: &[u8]) -> Result { + let header: [u8; 4] = frame + .get(..4) + .ok_or(FrameError::Truncated)? + .try_into() + .map_err(|_| FrameError::Truncated)?; + let declared = u32::from_be_bytes(header) as usize; + if declared > MAX_CONTROL_FRAME_BYTES { + return Err(FrameError::TooLarge(declared)); + } + let payload = frame.get(4..).ok_or(FrameError::Truncated)?; + if payload.len() != declared { + return Err(FrameError::LengthMismatch { + declared, + actual: payload.len(), + }); + } + serde_json::from_slice(payload).map_err(FrameError::Deserialize) +} + +pub fn read_frame(reader: &mut impl Read) -> Result { + let mut header = [0_u8; 4]; + reader.read_exact(&mut header)?; + let declared = u32::from_be_bytes(header) as usize; + if declared > MAX_CONTROL_FRAME_BYTES { + return Err(FrameError::TooLarge(declared)); + } + let mut frame = Vec::with_capacity(4 + declared); + frame.extend_from_slice(&header); + frame.resize(4 + declared, 0); + reader.read_exact(&mut frame[4..])?; + decode_frame(&frame) +} + +pub async fn read_frame_async(reader: &mut R) -> Result +where + R: AsyncRead + Unpin, + T: DeserializeOwned, +{ + use tokio::io::AsyncReadExt as _; + + let mut header = [0_u8; 4]; + reader.read_exact(&mut header).await?; + let declared = u32::from_be_bytes(header) as usize; + if declared > MAX_CONTROL_FRAME_BYTES { + return Err(FrameError::TooLarge(declared)); + } + let mut frame = Vec::with_capacity(4 + declared); + frame.extend_from_slice(&header); + frame.resize(4 + declared, 0); + reader.read_exact(&mut frame[4..]).await?; + decode_frame(&frame) +} + +pub fn write_frame(writer: &mut impl Write, message: &T) -> Result<(), FrameError> { + let frame = encode_frame(message)?; + writer.write_all(&frame)?; + writer.flush()?; + Ok(()) +} + +#[derive(Debug, thiserror::Error)] +pub enum FrameError { + #[error("control frame is truncated")] + Truncated, + #[error("control frame is too large: {0} bytes")] + TooLarge(usize), + #[error("control frame declared {declared} bytes but contained {actual}")] + LengthMismatch { declared: usize, actual: usize }, + #[error("serialize control frame: {0}")] + Serialize(serde_json::Error), + #[error("deserialize control frame: {0}")] + Deserialize(serde_json::Error), + #[error("control request payload digest does not match its envelope")] + PayloadDigestMismatch, + #[error("read or write control frame: {0}")] + Io(#[from] io::Error), +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn binary_identity_wire_rejects_ambiguous_or_invalid_shapes() { + for encoded in [ + r#"{"result":"resolved","ancestors":[],"cmdline_paths":[]}"#, + r#"{"result":"resolved","binary_path":"/bin/tool","binary_digest":"invalid","ancestors":[],"cmdline_paths":[]}"#, + r#"{"result":"failed","message":"unavailable","binary_path":"/bin/tool"}"#, + ] { + assert!(serde_json::from_str::(encoded).is_err()); + } + let identity = BinaryIdentityWire::from(Ok(BinaryIdentity { + binary_path: PathBuf::from("/bin/tool"), + binary_digest: Some("a".repeat(64).parse().unwrap()), + ancestors: Vec::new(), + cmdline_paths: Vec::new(), + })); + let encoded = serde_json::to_vec(&identity).unwrap(); + assert_eq!( + serde_json::from_slice::(&encoded).unwrap(), + identity + ); + } + + #[test] + fn boundary_error_kind_rejects_unknown_values() { + assert!(serde_json::from_str::(r#""unkown""#).is_err()); + for kind in [ + BoundaryErrorKind::Invalid, + BoundaryErrorKind::Denied, + BoundaryErrorKind::Unavailable, + BoundaryErrorKind::Terminated, + BoundaryErrorKind::Process, + ] { + let encoded = serde_json::to_vec(&kind).unwrap(); + assert_eq!( + serde_json::from_slice::(&encoded).unwrap(), + kind + ); + } + } + + #[test] + fn supervisor_instance_id_round_trips_canonically_and_stays_redacted() { + let instance_id = SupervisorInstanceId::new(); + let encoded = serde_json::to_string(&instance_id).expect("encode supervisor instance ID"); + let decoded = serde_json::from_str::(&encoded) + .expect("decode supervisor instance ID"); + assert_eq!(decoded, instance_id); + assert_eq!( + format!("{instance_id:?}"), + "SupervisorInstanceId([REDACTED])" + ); + assert!( + serde_json::from_str::( + r#""00000000-0000-0000-0000-000000000000""# + ) + .is_err() + ); + } + + #[test] + fn request_round_trips_and_redacts_secrets() { + let request = RequestEnvelope { + request_id: "4e94636d-54f8-4d85-8e4e-58954fb5af0a".to_string(), + payload_digest: String::new(), + request: Request::StartAgent { + sandbox_id: "sandbox-1".to_string(), + spec: AgentSpecWire { + program: "/bin/true".to_string(), + args: Vec::new(), + workdir: Some("/sandbox".to_string()), + timeout_secs: 5, + interactive: false, + }, + policy: Box::new(SandboxPolicyWire::from(SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy::default(), + network: NetworkPolicy::default(), + landlock: LandlockPolicy::default(), + process: ProcessPolicy::default(), + })), + ca_cert: Some(b"test certificate".to_vec()), + ca_bundle: Some(b"test bundle".to_vec()), + provider_env_revision: 7, + provider_env: std::collections::HashMap::from([( + "OPENAI_API_KEY".to_string(), + "test credential".to_string(), + )]), + }, + }; + let request = RequestEnvelope { + payload_digest: request_payload_digest(&request.request).expect("request digest"), + ..request + }; + let frame = encode_frame(&request).expect("encode request"); + let decoded: RequestEnvelope = decode_frame(&frame).expect("decode request"); + assert_eq!(decoded, request); + let debug = format!("{request:?}"); + assert!(debug.contains("")); + assert!(!debug.contains("test credential")); + assert!(!debug.contains("test certificate")); + assert!(!debug.contains("test bundle")); + assert!(debug.contains("OPENAI_API_KEY")); + assert!(request.validate_payload_digest().is_ok()); + } + + #[test] + fn request_digest_is_stable_across_map_order_and_detects_mutation() { + let mut first = std::collections::HashMap::new(); + first.insert("B".to_string(), "2".to_string()); + first.insert("A".to_string(), "1".to_string()); + let mut second = std::collections::HashMap::new(); + second.insert("A".to_string(), "1".to_string()); + second.insert("B".to_string(), "2".to_string()); + let build = |provider_env| Request::UpdateProviderEnvironment { + expected_revision: 1, + revision: 2, + provider_env, + }; + assert_eq!( + request_payload_digest(&build(first)).expect("first digest"), + request_payload_digest(&build(second)).expect("second digest") + ); + + let mut envelope = RequestEnvelope::new(build(std::collections::HashMap::new())) + .expect("request envelope"); + envelope.request = Request::Terminate { + process_id: "different".to_string(), + }; + assert!(matches!( + envelope.validate_payload_digest(), + Err(FrameError::PayloadDigestMismatch) + )); + } + + #[test] + fn rejects_declared_oversize() { + let oversized = u32::try_from(MAX_CONTROL_FRAME_BYTES + 1).expect("test size fits u32"); + let mut frame = Vec::from(oversized.to_be_bytes()); + frame.extend_from_slice(b"{}"); + assert!(matches!( + decode_frame::(&frame), + Err(FrameError::TooLarge(_)) + )); + } + + #[test] + fn resource_claims_reject_empty_or_ambiguous_identities() { + assert!( + validate_resource_claims(&std::collections::BTreeMap::from([( + "kubernetes.pod_uid".to_string(), + String::new() + ),])) + .is_err() + ); + assert!( + validate_resource_claims(&std::collections::BTreeMap::from([( + "kubernetes.pod uid".to_string(), + "uid-1".to_string() + ),])) + .is_err() + ); + validate_resource_claims(&std::collections::BTreeMap::from([( + "kubernetes.pod_uid".to_string(), + "uid-1".to_string(), + )])) + .expect("opaque resource identity should be valid"); + } + + #[test] + fn sandbox_tls_identity_is_unique_per_session_and_redacted() { + let first_session = SandboxSessionId::new(); + let second_session = SandboxSessionId::new(); + let first = generate_sandbox_tls_material(first_session).expect("first TLS material"); + let second = generate_sandbox_tls_material(second_session).expect("second TLS material"); + + assert_eq!( + first.server_name, + format!("sandbox.{first_session}.openshell.internal") + ); + assert_ne!(first.server_name, second.server_name); + assert_ne!(first.trust_anchor_pem, second.trust_anchor_pem); + assert!(first.certificate_chain_pem.contains("BEGIN CERTIFICATE")); + assert!(first.private_key_pem.contains("BEGIN PRIVATE KEY")); + + let debug = format!("{first:?}"); + assert!(!debug.contains(&first.private_key_pem)); + assert!(!debug.contains(&first.certificate_chain_pem)); + } + + #[test] + fn sandbox_transport_does_not_embed_authentication_material() { + let transport = SandboxTransport::Tcp { + authority: "sandbox.default.svc.cluster.local".to_string(), + addresses: vec!["192.0.2.10:8443".parse().expect("test address")], + }; + let encoded = serde_json::to_vec(&transport).expect("encode transport"); + let decoded: SandboxTransport = serde_json::from_slice(&encoded).expect("decode transport"); + assert_eq!(decoded, transport); + } +} diff --git a/crates/openshell-sandbox-backend/src/lib.rs b/crates/openshell-sandbox-backend/src/lib.rs new file mode 100644 index 0000000000..49cbf1212e --- /dev/null +++ b/crates/openshell-sandbox-backend/src/lib.rs @@ -0,0 +1,43 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! `OpenShell` Sandbox Protocol and its concrete RFC 0012 backend. +//! +//! [`OpenShellRuntimeBackend`] is the supervisor-side implementation of the +//! generic `IsolationBackend` interface. The workload-side `openshell-sandbox` +//! runtime serves the same protocol using the generated server and shared wire +//! types in this crate. + +pub mod boundary_protocol; +pub mod mediation; +mod runtime; +pub mod sandbox_auth; + +pub use runtime::OpenShellRuntimeBackend; + +/// Stable isolation backend name implemented by `openshell-sandbox`. +pub const BACKEND_NAME: &str = "openshell-sandbox"; + +/// Memory-backed parent used for supervisor CA material. +pub const SUPERVISOR_CA_RUNTIME_ROOT: &str = "/run/openshell-supervisor-ca"; + +/// Workload-visible directory for the supervisor's public HTTPS interception +/// certificate and combined trust bundle. +/// +/// Keeping the material below the mount root lets runtimes that cannot assign +/// tmpfs ownership create this child as the unprivileged sandbox identity. +pub const SUPERVISOR_CA_RUNTIME_DIR: &str = "/run/openshell-supervisor-ca/material"; + +/// Generated gRPC transport envelope for the OpenShell Sandbox Protocol. +#[allow( + clippy::all, + clippy::pedantic, + clippy::nursery, + dead_code, + unused_imports, + unused_qualifications, + rust_2018_idioms +)] +pub mod proto { + tonic::include_proto!("openshell.sandbox.protocol.v1"); +} diff --git a/crates/openshell-sandbox-backend/src/mediation.rs b/crates/openshell-sandbox-backend/src/mediation.rs new file mode 100644 index 0000000000..9a728369f1 --- /dev/null +++ b/crates/openshell-sandbox-backend/src/mediation.rs @@ -0,0 +1,156 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Framing for the persistent sandbox-to-supervisor DNS data plane. +//! +//! TCP connections use independent streams on the authenticated HTTP/2 +//! transport so one busy connection cannot head-of-line block another. + +use std::io; + +use serde::de::DeserializeOwned; +use serde::{Deserialize, Serialize}; +use tokio::io::{AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _}; + +use crate::boundary_protocol::{BinaryIdentityWire, MediationTimingWire}; +use openshell_isolation_interface::contract::DnsTransport; + +const HEADER_BYTES: usize = 13; +const MAX_METADATA_BYTES: usize = 256 * 1024; +const MAX_FRAME_BYTES: usize = MAX_METADATA_BYTES; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[repr(u8)] +pub enum MediationFrameKind { + DnsQuery = 5, + DnsResponse = 6, +} + +impl TryFrom for MediationFrameKind { + type Error = io::Error; + + fn try_from(value: u8) -> Result { + match value { + 5 => Ok(Self::DnsQuery), + 6 => Ok(Self::DnsResponse), + _ => Err(io::Error::new( + io::ErrorKind::InvalidData, + format!("unknown mediation frame kind {value}"), + )), + } + } +} + +#[derive(Debug)] +pub struct MediationFrame { + pub kind: MediationFrameKind, + pub stream_id: u64, + pub payload: Vec, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct DnsQueryWire { + pub request: Vec, + pub transport: DnsTransport, + pub identity: BinaryIdentityWire, + pub timing: MediationTimingWire, +} + +pub fn encode_json(value: &T) -> io::Result> { + let payload = serde_json::to_vec(value) + .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?; + if payload.len() > MAX_METADATA_BYTES { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "mediation metadata frame exceeds limit", + )); + } + Ok(payload) +} + +pub fn decode_json(payload: &[u8]) -> io::Result { + serde_json::from_slice(payload) + .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error)) +} + +pub async fn write_frame( + writer: &mut W, + kind: MediationFrameKind, + stream_id: u64, + payload: &[u8], +) -> io::Result<()> { + if payload.len() > MAX_FRAME_BYTES.max(MAX_METADATA_BYTES) { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "mediation frame exceeds limit", + )); + } + let length = u32::try_from(payload.len()) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "mediation frame too large"))?; + let mut header = [0_u8; HEADER_BYTES]; + header[0] = kind as u8; + header[1..9].copy_from_slice(&stream_id.to_be_bytes()); + header[9..13].copy_from_slice(&length.to_be_bytes()); + writer.write_all(&header).await?; + writer.write_all(payload).await?; + writer.flush().await +} + +pub async fn read_frame( + reader: &mut R, +) -> io::Result> { + let mut header = [0_u8; HEADER_BYTES]; + if reader.read(&mut header[..1]).await? == 0 { + return Ok(None); + } + reader.read_exact(&mut header[1..]).await?; + let kind = MediationFrameKind::try_from(header[0])?; + let stream_id = u64::from_be_bytes( + header[1..9] + .try_into() + .map_err(|_| io::Error::other("invalid stream ID header"))?, + ); + let length = u32::from_be_bytes( + header[9..13] + .try_into() + .map_err(|_| io::Error::other("invalid payload length header"))?, + ) as usize; + if length > MAX_FRAME_BYTES.max(MAX_METADATA_BYTES) { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "mediation frame exceeds limit", + )); + } + let mut payload = vec![0_u8; length]; + reader.read_exact(&mut payload).await?; + Ok(Some(MediationFrame { + kind, + stream_id, + payload, + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn frame_round_trip_preserves_binary_payload() { + let (mut writer, mut reader) = tokio::io::duplex(128); + let send = tokio::spawn(async move { + write_frame( + &mut writer, + MediationFrameKind::DnsResponse, + 42, + &[0, 1, 2, 255], + ) + .await + .unwrap(); + }); + let frame = read_frame(&mut reader).await.unwrap().unwrap(); + assert_eq!(frame.kind, MediationFrameKind::DnsResponse); + assert_eq!(frame.stream_id, 42); + assert_eq!(frame.payload, vec![0, 1, 2, 255]); + send.await.unwrap(); + } +} diff --git a/crates/openshell-sandbox-backend/src/runtime.rs b/crates/openshell-sandbox-backend/src/runtime.rs new file mode 100644 index 0000000000..86964fc3c8 --- /dev/null +++ b/crates/openshell-sandbox-backend/src/runtime.rs @@ -0,0 +1,2827 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Host-side RFC 0012 backend for an already-provisioned remote boundary. + +#![allow(unsafe_code)] + +#[cfg(test)] +use std::collections::HashMap; +#[cfg(target_os = "linux")] +use std::mem::size_of; +#[cfg(target_os = "linux")] +use std::os::fd::{FromRawFd as _, IntoRawFd as _}; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::time::{Duration, Instant}; + +use crate::proto::{BoundaryChunk, isolation_boundary_client::IsolationBoundaryClient}; +use async_trait::async_trait; +use hyper_util::rt::TokioIo; +use openshell_isolation_interface::AgentSpec; +use openshell_isolation_interface::contract::{ + BackendError, BoundBoundary, BoundaryDuplexStream, BoundaryExec, BoundaryExitStatus, + BoundaryInput, BoundaryLoopbackConnector, BoundaryOutput, BoundaryProcess, BoundarySignal, + BoundaryTerminal, ConfirmedBoundary, ExecSession, ExecSpec, IsolationBackend, LoopbackTarget, + MediationTiming, NetworkMediationSource, PendingDnsQuery, PendingTcpOpen, ProcessAttachment, + ReadyBoundary, RunningBoundary, SandboxContext, TcpOpenDecision, TcpOpenDenial, + VerifiedBackendDescriptor, +}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +#[cfg(unix)] +use tokio::net::UnixStream; +use tokio_stream::wrappers::ReceiverStream; + +use crate::boundary_protocol::{ + AgentSpecWire, DnsQueryResultWire, ExecSpecWire, MAX_CONTROL_FRAME_BYTES, Request, + RequestEnvelope, Response, ResponseEnvelope, STREAM_EXIT, STREAM_STDERR, STREAM_STDIN, + STREAM_STDIN_CLOSED, STREAM_STDOUT, SandboxPolicyWire, SandboxRuntimeDescriptor, + SandboxTlsClientConfig, SandboxTransport, SignalWire, decode_frame, encode_frame, + read_stream_frame, validate_resource_claims, write_stream_frame, +}; +use crate::mediation::{self, DnsQueryWire, MediationFrame, MediationFrameKind}; + +const REQUEST_TIMEOUT: Duration = Duration::from_secs(30); +/// Initial attachment may include runtime image pulls and trusted bootstrap +/// work before the boundary begins listening. Keep this aligned with the +/// driver bootstrap grace period rather than the normal operation timeout. +const ATTACH_REQUEST_TIMEOUT: Duration = Duration::from_mins(5); +/// How long one control call keeps retrying boundary connect attempts. Boot-time +/// callers retry whole calls above this; past boot, exhausting this window +/// means the remote boundary (or its launcher) is gone rather than still starting. +const CONNECT_RETRY_TIMEOUT: Duration = Duration::from_secs(30); + +fn begin_recovery_window( + deadline: &mut Option, + failure_time: tokio::time::Instant, +) -> tokio::time::Instant { + *deadline.get_or_insert(failure_time + CONNECT_RETRY_TIMEOUT) +} + +/// Host-side `OpenShell` Sandbox Protocol implementation registered with the supervisor. +#[derive(Debug)] +pub struct OpenShellRuntimeBackend { + ca_file_paths: Arc>>, + provider_credentials: openshell_core::provider_credentials::ProviderCredentialState, + sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, +} + +impl OpenShellRuntimeBackend { + pub fn new( + ca_file_paths: Arc>>, + provider_credentials: openshell_core::provider_credentials::ProviderCredentialState, + sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, + ) -> Self { + Self { + ca_file_paths, + provider_credentials, + sandbox_bearer, + } + } +} + +#[async_trait] +impl IsolationBackend for OpenShellRuntimeBackend { + fn backend_name(&self) -> &str { + crate::BACKEND_NAME + } + + async fn attach( + &self, + descriptor: VerifiedBackendDescriptor, + sandbox: SandboxContext, + ) -> Result, BackendError> { + let runtime_descriptor: SandboxRuntimeDescriptor = + serde_json::from_slice(descriptor.payload()).map_err(|error| { + BackendError::Descriptor(format!("decode runtime descriptor: {error}")) + })?; + validate_runtime_descriptor(&runtime_descriptor, &sandbox)?; + let host_gateway_ip = runtime_descriptor.host_gateway_ip; + let resource_claims = runtime_descriptor.resource_claims.clone(); + let generation = runtime_descriptor.generation.clone(); + let session_id = runtime_descriptor.session_id; + let driver_fence = runtime_descriptor.driver_fence.clone(); + let client = Arc::new(BoundaryClient::new( + runtime_descriptor, + self.sandbox_bearer.clone(), + )); + let response = client + .call_idempotent(Request::Attach { + supervisor_instance_id: client.supervisor_instance_id, + policy: Box::new(SandboxPolicyWire::from(sandbox.policy.clone())), + resource_claims: resource_claims.clone(), + }) + .await?; + let Response::Attached { snapshot } = response else { + return Err(unexpected_response("attached", &response)); + }; + if snapshot.generation != generation { + return Err(BackendError::Confirm( + "sandbox session snapshot generation does not match runtime descriptor".to_string(), + )); + } + Ok(Box::new(RemoteBound { + client: client.clone(), + agent: sandbox.agent, + policy: sandbox.policy, + sandbox_id: sandbox.sandbox_id, + mediation: Arc::new(RemoteNetworkMediation { client }), + host_gateway_ip, + ca_file_paths: self.ca_file_paths.clone(), + provider_credentials: self.provider_credentials.clone(), + identity: sandbox.identity, + generation, + session_id, + resource_claims, + driver_fence, + })) + } +} + +fn validate_runtime_descriptor( + runtime_descriptor: &SandboxRuntimeDescriptor, + sandbox: &SandboxContext, +) -> Result<(), BackendError> { + if runtime_descriptor.boundary_id != sandbox.sandbox_id { + return Err(BackendError::Descriptor(format!( + "boundary {:?} does not match sandbox {:?}", + runtime_descriptor.boundary_id, sandbox.sandbox_id + ))); + } + if runtime_descriptor.generation.is_empty() { + return Err(BackendError::Descriptor( + "boundary generation must not be empty".to_string(), + )); + } + if runtime_descriptor.session_id != sandbox.session_id { + return Err(BackendError::Descriptor( + "runtime descriptor session ID does not match admitted sandbox session".to_string(), + )); + } + if runtime_descriptor.workload_identity != sandbox.identity { + return Err(BackendError::Descriptor( + "runtime descriptor workload identity does not match admitted sandbox identity" + .to_string(), + )); + } + validate_resource_claims(&runtime_descriptor.resource_claims)?; + runtime_descriptor.driver_fence.validate()?; + match &runtime_descriptor.transport { + SandboxTransport::Unix { socket_path } => { + validate_socket_path(socket_path)?; + } + SandboxTransport::Tcp { + authority, + addresses, + } => { + if authority.is_empty() || addresses.is_empty() { + return Err(BackendError::Descriptor( + "boundary TCP transport requires an authority and at least one address" + .to_string(), + )); + } + for address in addresses { + validate_tcp_address(*address)?; + } + } + SandboxTransport::Vsock { guest_cid, port } => { + if *guest_cid < 3 { + return Err(BackendError::Descriptor( + "boundary CID must be at least 3".to_string(), + )); + } + validate_control_port(*port)?; + } + } + validate_client_tls(&runtime_descriptor.tls)?; + Ok(()) +} + +fn validate_tcp_address(address: std::net::SocketAddr) -> Result<(), BackendError> { + if address.port() == 0 || address.ip().is_unspecified() { + Err(BackendError::Descriptor( + "boundary TCP address must have a concrete IP and nonzero port".to_string(), + )) + } else { + Ok(()) + } +} + +fn validate_client_tls(tls: &SandboxTlsClientConfig) -> Result<(), BackendError> { + rustls::pki_types::ServerName::try_from(tls.server_name.clone()).map_err(|error| { + BackendError::Descriptor(format!( + "boundary TLS server name {:?} is invalid: {error}", + tls.server_name + )) + })?; + tls_client_config(tls).map(|_| ()) +} + +fn tls_client_config(tls: &SandboxTlsClientConfig) -> Result { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let certificates = rustls_pemfile::certs(&mut tls.trust_anchor_pem.as_bytes()) + .collect::, _>>() + .map_err(|error| { + BackendError::Descriptor(format!("parse boundary TLS CA certificate: {error}")) + })?; + if certificates.is_empty() { + return Err(BackendError::Descriptor( + "boundary TLS CA certificate PEM contains no certificates".to_string(), + )); + } + let mut roots = rustls::RootCertStore::empty(); + for certificate in certificates { + roots.add(certificate).map_err(|error| { + BackendError::Descriptor(format!("load boundary TLS CA certificate: {error}")) + })?; + } + let mut config = + rustls::ClientConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .with_root_certificates(roots) + .with_no_client_auth(); + config.alpn_protocols = vec![b"h2".to_vec()]; + Ok(config) +} + +fn validate_socket_path(path: &std::path::Path) -> Result<(), BackendError> { + if path.is_absolute() { + Ok(()) + } else { + Err(BackendError::Descriptor( + "boundary control Unix socket path must be absolute".to_string(), + )) + } +} + +fn validate_control_port(port: u32) -> Result<(), BackendError> { + if port == 0 { + Err(BackendError::Descriptor( + "boundary control port must be nonzero".to_string(), + )) + } else { + Ok(()) + } +} + +struct RemoteBound { + client: Arc, + agent: AgentSpec, + policy: openshell_core::policy::SandboxPolicy, + sandbox_id: String, + mediation: Arc, + host_gateway_ip: Option, + ca_file_paths: Arc>>, + provider_credentials: openshell_core::provider_credentials::ProviderCredentialState, + identity: openshell_isolation_interface::contract::ResolvedWorkloadIdentity, + generation: String, + session_id: openshell_core::SandboxSessionId, + resource_claims: std::collections::BTreeMap, + driver_fence: openshell_isolation_interface::contract::DriverFenceEvidence, +} + +#[async_trait] +impl BoundBoundary for RemoteBound { + fn network_mediation_source(&self) -> Arc { + self.mediation.clone() + } + + fn host_gateway_ip(&self) -> Option { + self.host_gateway_ip + } + + async fn confirm(self: Box) -> Result { + let response = self.client.call_idempotent(Request::Confirm).await?; + let Response::Confirmed { evidence } = response else { + return Err(unexpected_response("confirmed_with_evidence", &response)); + }; + if evidence.generation != self.generation + || evidence.session_id != self.session_id + || evidence.resource_claims != self.resource_claims + || evidence.driver_fence != self.driver_fence + { + return Err(BackendError::Confirm( + "sandbox confirmation generation, session, resource claims, or driver fence do not match runtime descriptor" + .to_string(), + )); + } + self.client.start_credential_monitor(); + ConfirmedBoundary::try_new( + Box::new(RemoteReady { + client: self.client, + agent: self.agent, + policy: self.policy, + sandbox_id: self.sandbox_id, + ca_file_paths: self.ca_file_paths, + provider_credentials: self.provider_credentials, + }), + *evidence, + &self.identity, + ) + } +} + +struct RemoteReady { + client: Arc, + agent: AgentSpec, + policy: openshell_core::policy::SandboxPolicy, + sandbox_id: String, + ca_file_paths: Arc>>, + provider_credentials: openshell_core::provider_credentials::ProviderCredentialState, +} + +#[async_trait] +impl ReadyBoundary for RemoteReady { + async fn start_agent(self: Box) -> Result, BackendError> { + let ca_paths = self + .ca_file_paths + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone(); + let (ca_cert, ca_bundle) = if let Some((ca_cert, ca_bundle)) = ca_paths { + let ca_cert = tokio::fs::read(&ca_cert).await.map_err(|error| { + BackendError::Process(format!("read host proxy CA {}: {error}", ca_cert.display())) + })?; + let ca_bundle = tokio::fs::read(&ca_bundle).await.map_err(|error| { + BackendError::Process(format!( + "read host proxy CA bundle {}: {error}", + ca_bundle.display() + )) + })?; + (Some(ca_cert), Some(ca_bundle)) + } else { + (None, None) + }; + let (provider_env_revision, provider_env) = self + .provider_credentials + .child_env_snapshot_with_gcp_resolved() + .map_err(|error| { + BackendError::Process(format!("snapshot provider environment: {error}")) + })?; + let response = self + .client + .call_idempotent(Request::StartAgent { + sandbox_id: self.sandbox_id, + spec: AgentSpecWire::from(self.agent), + policy: Box::new(SandboxPolicyWire::from(self.policy)), + ca_cert, + ca_bundle, + provider_env_revision, + provider_env, + }) + .await?; + let Response::Started { + process_id, + provider_env_revision, + } = response + else { + return Err(unexpected_response("started", &response)); + }; + let process = Arc::new(RemoteProcess { + client: self.client.clone(), + process_id, + }); + Ok(Box::new(RemoteRunning { + process, + exec: Arc::new(RemoteExec { + client: self.client.clone(), + provider_credentials: self.provider_credentials, + boundary_revision: tokio::sync::Mutex::new(provider_env_revision), + }), + loopback_connector: Arc::new(RemoteLoopbackConnector { + client: self.client, + }), + })) + } +} + +struct RemoteRunning { + process: Arc, + exec: Arc, + loopback_connector: Arc, +} + +#[async_trait] +impl RunningBoundary for RemoteRunning { + fn agent(&self) -> Arc { + self.process.clone() + } + + fn exec(&self) -> Arc { + self.exec.clone() + } + + fn loopback_connector(&self) -> Arc { + self.loopback_connector.clone() + } + + async fn terminate(&self) -> Result<(), BackendError> { + let response = self + .process + .client + .call_idempotent(Request::TerminateBoundary) + .await?; + expect_response(response, "boundary_terminated") + } +} + +struct RemoteProcess { + client: Arc, + process_id: String, +} + +#[async_trait] +impl BoundaryProcess for RemoteProcess { + async fn attach(&self) -> Result { + open_process_attachment(self.client.clone(), self.process_id.clone()).await + } + + async fn wait(&self) -> Result { + let response = self + .client + .call_wait(Request::Wait { + process_id: self.process_id.clone(), + }) + .await + .map_err(|error| match error { + // A wait that can no longer reach the boundary leaf means the + // boundary is gone, not that a retry could still observe the + // exit status; report boundary loss per the contract. + BackendError::Unavailable(message) => { + BackendError::Terminated(format!("boundary lost during wait: {message}")) + } + error => error, + })?; + let Response::Exited { status } = response else { + return Err(unexpected_response("exited", &response)); + }; + Ok(status.into()) + } + + async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError> { + let response = self + .client + .call_idempotent(Request::Signal { + process_id: self.process_id.clone(), + signal: SignalWire::from(signal), + }) + .await?; + expect_response(response, "signaled") + } + + async fn terminate(&self) -> Result<(), BackendError> { + let response = self + .client + .call_idempotent(Request::Terminate { + process_id: self.process_id.clone(), + }) + .await?; + expect_response(response, "terminated") + } +} + +async fn open_process_attachment( + client: Arc, + process_id: String, +) -> Result { + let (stream, response) = client + .call_stream(Request::AttachProcess { + process_id: process_id.clone(), + }) + .await?; + let Response::ProcessAttached { + terminal: has_terminal, + } = response + else { + return Err(unexpected_response("process_attached", &response)); + }; + let (network_reader, network_writer) = tokio::io::split(stream); + let (stdin, stdin_pump) = tokio::io::duplex(64 * 1024); + let (stdout, stdout_pump) = tokio::io::duplex(64 * 1024); + let (stderr, stderr_pump) = tokio::io::duplex(64 * 1024); + tokio::spawn(pump_exec_input(stdin_pump, network_writer)); + tokio::spawn(pump_process_responses( + network_reader, + stdout_pump, + stderr_pump, + )); + let terminal: Option> = if has_terminal { + let terminal: Arc = Arc::new(RemoteTerminal { client, process_id }); + Some(terminal) + } else { + None + }; + let stderr: Option = if has_terminal { + None + } else { + let stderr: BoundaryOutput = Box::new(stderr); + Some(stderr) + }; + Ok(ProcessAttachment { + stdin: Box::new(stdin), + stdout: Box::new(stdout), + stderr, + terminal, + }) +} + +async fn pump_process_responses( + mut network: tokio::io::ReadHalf, + mut stdout: tokio::io::DuplexStream, + mut stderr: tokio::io::DuplexStream, +) { + loop { + match read_stream_frame(&mut network).await { + Ok(Some((STREAM_STDOUT, payload))) => { + if stdout.write_all(&payload).await.is_err() { + return; + } + } + Ok(Some((STREAM_STDERR, payload))) => { + if stderr.write_all(&payload).await.is_err() { + return; + } + } + Ok(Some((STREAM_EXIT, _)) | None) | Err(_) => return, + Ok(Some((_channel, _))) => return, + } + } +} + +struct RemoteExec { + client: Arc, + provider_credentials: openshell_core::provider_credentials::ProviderCredentialState, + boundary_revision: tokio::sync::Mutex, +} + +#[async_trait] +impl BoundaryExec for RemoteExec { + async fn exec(&self, spec: ExecSpec) -> Result { + let mut boundary_revision = self.boundary_revision.lock().await; + for _ in 0..3 { + let (revision, provider_env) = self + .provider_credentials + .child_env_snapshot_with_gcp_resolved() + .map_err(|error| { + BackendError::Process(format!("snapshot provider environment: {error}")) + })?; + let response = self + .client + .call_idempotent(Request::UpdateProviderEnvironment { + expected_revision: *boundary_revision, + revision, + provider_env, + }) + .await?; + let Response::ProviderEnvironmentUpdated { + revision: effective_revision, + } = response + else { + return Err(unexpected_response( + "provider_environment_updated", + &response, + )); + }; + *boundary_revision = effective_revision; + if effective_revision == revision { + return open_exec_session(self.client.clone(), spec).await; + } + } + Err(BackendError::Process( + "boundary provider environment changed concurrently during reconciliation".to_string(), + )) + } +} + +struct RemoteLoopbackConnector { + client: Arc, +} + +#[async_trait] +impl BoundaryLoopbackConnector for RemoteLoopbackConnector { + async fn connect(&self, target: LoopbackTarget) -> Result { + let (stream, response) = self + .client + .call_stream(Request::LoopbackConnect { + host: target.host(), + port: target.port(), + }) + .await?; + match response { + Response::PortConnected => Ok(stream), + response => Err(unexpected_response("port_connected", &response)), + } + } +} + +struct RemoteExecProcess { + client: Arc, + process_id: String, +} + +#[async_trait] +impl BoundaryProcess for RemoteExecProcess { + async fn attach(&self) -> Result { + open_process_attachment(self.client.clone(), self.process_id.clone()).await + } + + async fn wait(&self) -> Result { + RemoteProcess { + client: self.client.clone(), + process_id: self.process_id.clone(), + } + .wait() + .await + } + + async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError> { + expect_response( + self.client + .call_idempotent(Request::ExecSignal { + process_id: self.process_id.clone(), + signal: SignalWire::from(signal), + }) + .await?, + "signaled", + ) + } + + async fn terminate(&self) -> Result<(), BackendError> { + self.signal(BoundarySignal::Kill).await + } +} + +struct RemoteTerminal { + client: Arc, + process_id: String, +} + +#[async_trait] +impl BoundaryTerminal for RemoteTerminal { + async fn resize(&self, cols: u16, rows: u16) -> Result<(), BackendError> { + let response = self + .client + .call_idempotent(Request::Resize { + process_id: self.process_id.clone(), + cols, + rows, + }) + .await?; + if matches!(response, Response::Resized) { + Ok(()) + } else { + Err(unexpected_response("resized", &response)) + } + } +} + +async fn open_exec_session( + client: Arc, + spec: ExecSpec, +) -> Result { + let (stream, response) = client + .call_stream_idempotent(Request::Exec { + spec: ExecSpecWire::from(spec), + }) + .await?; + let Response::ExecStarted { process_id, pty } = response else { + return Err(unexpected_response("exec_started", &response)); + }; + let (network_reader, network_writer) = tokio::io::split(stream); + let (stdin, stdin_pump) = tokio::io::duplex(64 * 1024); + let (stdout, stdout_pump) = tokio::io::duplex(64 * 1024); + let (stderr, stderr_pump) = tokio::io::duplex(64 * 1024); + tokio::spawn(pump_exec_input(stdin_pump, network_writer)); + tokio::spawn(pump_process_responses( + network_reader, + stdout_pump, + stderr_pump, + )); + + let process: Arc = Arc::new(RemoteExecProcess { + client: client.clone(), + process_id: process_id.clone(), + }); + let terminal: Option> = if pty { + Some(Arc::new(RemoteTerminal { client, process_id })) + } else { + None + }; + let stdin: BoundaryInput = Box::new(stdin); + let stdout: BoundaryOutput = Box::new(stdout); + let stderr: Option = if pty { None } else { Some(Box::new(stderr)) }; + Ok(ExecSession { + process, + stdin: Some(stdin), + stdout, + stderr, + terminal, + }) +} + +async fn pump_exec_input( + mut input: tokio::io::DuplexStream, + mut network: tokio::io::WriteHalf, +) { + let mut buffer = vec![0; 16 * 1024]; + loop { + match input.read(&mut buffer).await { + Ok(0) => { + let _ = write_stream_frame(&mut network, STREAM_STDIN_CLOSED, &[]).await; + return; + } + Ok(read) => { + if write_stream_frame(&mut network, STREAM_STDIN, &buffer[..read]) + .await + .is_err() + { + return; + } + } + Err(_) => return, + } + } +} + +/// Pulls boundary proxy connections over independent HTTP/2 streams. +/// +/// DNS control messages share the compact persistent mediation +/// session, but TCP byte streams use HTTP/2's native multiplexing. Nesting all +/// TCP connections inside one application-level writer creates avoidable +/// head-of-line blocking during concurrent TLS handshakes. +struct RemoteNetworkMediation { + client: Arc, +} + +#[async_trait] +impl NetworkMediationSource for RemoteNetworkMediation { + async fn accept_tcp(&self) -> Result { + let (stream, response) = self.client.open_exchange(Request::AcceptNetwork).await?; + let Response::NetworkConnected { + identity, + destination, + socket, + policy_generation, + timing, + } = response + else { + return Err(unexpected_response("network_connected", &response)); + }; + let (decision, completion) = tokio::sync::oneshot::channel(); + let (proxy_stream, transport_stream) = tokio::io::duplex(64 * 1024); + tokio::spawn(complete_network_open(stream, transport_stream, completion)); + Ok(PendingTcpOpen { + stream: Box::new(proxy_stream), + binary_identity: identity.into_result(), + destination, + socket, + policy_generation, + timing: MediationTiming { + sandbox_notification_to_queue: Duration::from_micros( + timing.notification_to_queue_us, + ), + sandbox_queue_wait: Duration::from_micros(timing.queue_wait_us), + supervisor_received_at: Instant::now(), + }, + decision, + }) + } + + async fn accept_dns(&self) -> Result { + loop { + let session = self.client.mediation_session().await?; + match session.accept_dns().await { + Ok(query) => return Ok(query), + Err(BackendError::Unavailable(_)) if !session.is_healthy() => {} + Err(error) => return Err(error), + } + } + } +} + +async fn complete_network_open( + mut boundary: BoundaryDuplexStream, + mut transport: tokio::io::DuplexStream, + completion: tokio::sync::oneshot::Receiver, +) { + let decision = completion + .await + .unwrap_or(TcpOpenDecision::Denied(TcpOpenDenial::MediationUnavailable)); + let Ok(payload) = serde_json::to_vec(&decision) else { + return; + }; + if write_stream_frame( + &mut boundary, + crate::boundary_protocol::STREAM_NETWORK_DECISION, + &payload, + ) + .await + .is_err() + { + return; + } + if matches!(decision, TcpOpenDecision::RelayReady) { + let _ = tokio::io::copy_bidirectional(&mut boundary, &mut transport).await; + } +} + +const MEDIATION_EVENT_QUEUE: usize = 256; +struct OutboundMediationFrame { + kind: MediationFrameKind, + stream_id: u64, + payload: Vec, +} + +struct ClientMediationSession { + dns: tokio::sync::Mutex>, + healthy: Arc, +} + +impl ClientMediationSession { + fn start(stream: BoundaryDuplexStream) -> Arc { + let (dns_tx, dns_rx) = tokio::sync::mpsc::channel(MEDIATION_EVENT_QUEUE); + let healthy = Arc::new(AtomicBool::new(true)); + let session = Arc::new(Self { + dns: tokio::sync::Mutex::new(dns_rx), + healthy: healthy.clone(), + }); + tokio::spawn(async move { + if let Err(error) = run_client_mediation(stream, dns_tx).await { + tracing::debug!(%error, "persistent mediation session ended"); + } + healthy.store(false, Ordering::Release); + }); + session + } + + fn is_healthy(&self) -> bool { + self.healthy.load(Ordering::Acquire) + } + + async fn accept_dns(&self) -> Result { + self.dns.lock().await.recv().await.ok_or_else(|| { + BackendError::Unavailable("persistent DNS mediation session ended".to_string()) + }) + } +} + +async fn run_client_mediation( + stream: BoundaryDuplexStream, + dns_tx: tokio::sync::mpsc::Sender, +) -> std::io::Result<()> { + let (mut reader, mut writer) = tokio::io::split(stream); + let (outbound_tx, mut outbound_rx) = + tokio::sync::mpsc::channel::(MEDIATION_EVENT_QUEUE); + let writer_task = async { + while let Some(frame) = outbound_rx.recv().await { + mediation::write_frame(&mut writer, frame.kind, frame.stream_id, &frame.payload) + .await?; + } + Ok::<(), std::io::Error>(()) + }; + let reader_task = async { + while let Some(frame) = mediation::read_frame(&mut reader).await? { + dispatch_client_mediation_frame(frame, &dns_tx, &outbound_tx).await?; + } + Ok::<(), std::io::Error>(()) + }; + tokio::pin!(writer_task); + tokio::pin!(reader_task); + let result = tokio::select! { + result = &mut writer_task => result, + result = &mut reader_task => result, + }; + result +} + +async fn dispatch_client_mediation_frame( + frame: MediationFrame, + dns_tx: &tokio::sync::mpsc::Sender, + outbound: &tokio::sync::mpsc::Sender, +) -> std::io::Result<()> { + match frame.kind { + MediationFrameKind::DnsQuery => { + let query: DnsQueryWire = mediation::decode_json(&frame.payload)?; + let (response, completion) = + tokio::sync::oneshot::channel::, BackendError>>(); + let outbound = outbound.clone(); + tokio::spawn(async move { + let response = match completion.await { + Ok(Ok(response)) => DnsQueryResultWire::Response(response), + Ok(Err(error)) => DnsQueryResultWire::Error(error.to_string()), + Err(_) => DnsQueryResultWire::Error( + "supervisor dropped the mediated DNS query".to_string(), + ), + }; + if let Ok(payload) = mediation::encode_json(&response) { + let _ = outbound + .send(OutboundMediationFrame { + kind: MediationFrameKind::DnsResponse, + stream_id: frame.stream_id, + payload, + }) + .await; + } + }); + dns_tx + .send(PendingDnsQuery { + message: query.request, + transport: query.transport, + binary_identity: query.identity.into_result(), + timing: MediationTiming { + sandbox_notification_to_queue: Duration::from_micros( + query.timing.notification_to_queue_us, + ), + sandbox_queue_wait: Duration::from_micros(query.timing.queue_wait_us), + supervisor_received_at: Instant::now(), + }, + response, + }) + .await + .map_err(|_| { + std::io::Error::new( + std::io::ErrorKind::BrokenPipe, + "DNS mediation consumer stopped", + ) + })?; + } + MediationFrameKind::DnsResponse => { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidData, + format!( + "unexpected supervisor-bound mediation frame {:?}", + frame.kind + ), + )); + } + } + Ok(()) +} + +struct BoundaryClient { + runtime_descriptor: SandboxRuntimeDescriptor, + supervisor_instance_id: crate::boundary_protocol::SupervisorInstanceId, + sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, + grpc_channel: tokio::sync::Mutex>, + mediation: tokio::sync::Mutex>>, + mediation_open: tokio::sync::Mutex<()>, + attach_request: std::sync::Mutex>, + confirm_request: std::sync::Mutex>, + reconnect: tokio::sync::Mutex<()>, + next_connection_generation: AtomicU64, + credential_monitor_started: AtomicBool, +} + +#[derive(Clone)] +struct CachedGrpcChannel { + credential_epoch: openshell_core::jwt::CredentialEpoch, + generation: u64, + channel: tonic::transport::Channel, +} + +impl BoundaryClient { + fn new( + runtime_descriptor: SandboxRuntimeDescriptor, + sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, + ) -> Self { + Self { + runtime_descriptor, + supervisor_instance_id: crate::boundary_protocol::SupervisorInstanceId::new(), + sandbox_bearer, + grpc_channel: tokio::sync::Mutex::new(None), + mediation: tokio::sync::Mutex::new(None), + mediation_open: tokio::sync::Mutex::new(()), + attach_request: std::sync::Mutex::new(None), + confirm_request: std::sync::Mutex::new(None), + reconnect: tokio::sync::Mutex::new(()), + next_connection_generation: AtomicU64::new(1), + credential_monitor_started: AtomicBool::new(false), + } + } + + async fn call_idempotent(&self, request: Request) -> Result { + let remember_attach = matches!(request, Request::Attach { .. }); + let remember_confirm = matches!(request, Request::Confirm); + let timeout = if remember_attach { + ATTACH_REQUEST_TIMEOUT + } else { + REQUEST_TIMEOUT + }; + let envelope = Self::prepare_request(request)?; + tokio::time::timeout(timeout, async { + loop { + match self.exchange_envelope(&envelope).await { + Ok(response) => { + if remember_attach { + *self + .attach_request + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = + Some(envelope.clone()); + } + if remember_confirm { + *self + .confirm_request + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = + Some(envelope.clone()); + } + return Ok(response); + } + Err(BackendError::Unavailable(message)) + if is_transport_unavailable(&message) => + { + self.recover_after_unavailable().await?; + tokio::time::sleep(Duration::from_millis(25)).await; + } + Err(error) => return Err(error), + } + } + }) + .await + .map_err(|_| { + BackendError::Unavailable( + "boundary idempotent control request timed out while waiting for remote boundary boot".to_string(), + ) + })? + } + + async fn call_wait(&self, request: Request) -> Result { + let envelope = Self::prepare_request(request)?; + let mut recovery_deadline = None; + loop { + match self.exchange_envelope(&envelope).await { + Ok(response) => return Ok(response), + Err(BackendError::Unavailable(message)) if is_transport_unavailable(&message) => { + let deadline = + begin_recovery_window(&mut recovery_deadline, tokio::time::Instant::now()); + if tokio::time::Instant::now() >= deadline { + return Err(BackendError::Unavailable(message)); + } + self.recover_after_unavailable().await?; + tokio::time::sleep(Duration::from_millis(25)).await; + } + Err(error) => return Err(error), + } + } + } + + async fn call_stream( + &self, + request: Request, + ) -> Result<(BoundaryDuplexStream, Response), BackendError> { + let envelope = Self::prepare_request(request)?; + tokio::time::timeout(REQUEST_TIMEOUT, async { + loop { + match self.open_exchange_envelope(&envelope).await { + Ok(response) => return Ok(response), + Err(BackendError::Unavailable(message)) + if is_transport_unavailable(&message) => + { + self.recover_after_unavailable().await?; + tokio::time::sleep(Duration::from_millis(25)).await; + } + Err(error) => return Err(error), + } + } + }) + .await + .map_err(|_| BackendError::Unavailable("boundary stream request timed out".to_string()))? + } + + async fn call_stream_idempotent( + &self, + request: Request, + ) -> Result<(BoundaryDuplexStream, Response), BackendError> { + let envelope = Self::prepare_request(request)?; + tokio::time::timeout(REQUEST_TIMEOUT, async { + loop { + match self.open_exchange_envelope(&envelope).await { + Ok(response) => return Ok(response), + Err(BackendError::Unavailable(message)) + if is_transport_unavailable(&message) => + { + self.recover_after_unavailable().await?; + tokio::time::sleep(Duration::from_millis(25)).await; + } + Err(error) => return Err(error), + } + } + }) + .await + .map_err(|_| { + BackendError::Unavailable("boundary idempotent stream request timed out".to_string()) + })? + } + + #[cfg(test)] + async fn exchange(&self, request: Request) -> Result { + let (_, response) = self.open_exchange(request).await?; + Ok(response) + } + + fn prepare_request(request: Request) -> Result { + RequestEnvelope::new(request) + .map_err(|error| BackendError::Process(format!("encode control request: {error}"))) + } + + async fn open_exchange( + &self, + request: Request, + ) -> Result<(BoundaryDuplexStream, Response), BackendError> { + let envelope = Self::prepare_request(request)?; + self.open_exchange_envelope(&envelope).await + } + + async fn exchange_envelope( + &self, + envelope: &RequestEnvelope, + ) -> Result { + let (_, response) = self.open_exchange_envelope(envelope).await?; + Ok(response) + } + + async fn open_exchange_envelope( + &self, + envelope: &RequestEnvelope, + ) -> Result<(BoundaryDuplexStream, Response), BackendError> { + let request_id = envelope.request_id.clone(); + let mut stream = self.open_grpc_stream(GrpcStreamKind::Exchange).await?; + let frame = encode_frame(envelope) + .map_err(|error| BackendError::Process(format!("encode control request: {error}")))?; + stream.write_all(&frame).await.map_err(|error| { + BackendError::Unavailable(format!("write boundary control request: {error}")) + })?; + // `tokio-rustls` may retain part of a large plaintext frame in its + // internal TLS buffer. Flush before waiting for the response so the + // synchronous boundary reader can receive the complete request. + stream.flush().await.map_err(|error| { + BackendError::Unavailable(format!("flush boundary control request: {error}")) + })?; + let mut header = [0_u8; 4]; + stream.read_exact(&mut header).await.map_err(|error| { + BackendError::Unavailable(format!("read boundary control response header: {error}")) + })?; + let declared = u32::from_be_bytes(header) as usize; + if declared > MAX_CONTROL_FRAME_BYTES { + return Err(BackendError::Process(format!( + "boundary control response is too large: {declared} bytes" + ))); + } + let mut frame = Vec::with_capacity(4 + declared); + frame.extend_from_slice(&header); + frame.resize(4 + declared, 0); + stream.read_exact(&mut frame[4..]).await.map_err(|error| { + BackendError::Unavailable(format!("read boundary control response: {error}")) + })?; + let response: ResponseEnvelope = decode_frame(&frame) + .map_err(|error| BackendError::Process(format!("decode control response: {error}")))?; + if response.request_id != request_id { + return Err(BackendError::Process(format!( + "boundary response ID {} did not match request ID {request_id}", + response.request_id + ))); + } + let response = match response.response { + Response::Error { kind, message } => Err(guest_error(kind, message)), + response => Ok(response), + }?; + Ok((stream, response)) + } + + async fn open_grpc_stream( + &self, + kind: GrpcStreamKind, + ) -> Result { + self.ensure_current_credential_connection().await?; + let channel = self.grpc_channel().await?; + open_grpc_client_stream(channel, kind, &self.sandbox_bearer).await + } + + async fn ensure_current_credential_connection(&self) -> Result<(), BackendError> { + let credential_epoch = self.sandbox_bearer.credential_epoch().ok_or_else(|| { + BackendError::Unavailable("Sandbox Protocol credential unavailable".to_string()) + })?; + if self + .grpc_channel + .lock() + .await + .as_ref() + .is_none_or(|cached| cached.credential_epoch == credential_epoch) + { + return Ok(()); + } + + let _reconnect = self.reconnect.lock().await; + if self + .grpc_channel + .lock() + .await + .as_ref() + .is_some_and(|cached| cached.credential_epoch == credential_epoch) + { + return Ok(()); + } + let attach = self + .attach_request + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .ok_or_else(|| { + BackendError::Unavailable( + "cannot rotate Sandbox Protocol connection before attach".to_string(), + ) + })?; + let confirm = self + .confirm_request + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + .ok_or_else(|| { + BackendError::Unavailable( + "cannot rotate Sandbox Protocol connection before confirmation".to_string(), + ) + })?; + let channel = self.build_grpc_channel().await?; + self.exchange_on_channel(channel.clone(), &attach).await?; + self.exchange_on_channel(channel.clone(), &confirm).await?; + *self.grpc_channel.lock().await = Some(CachedGrpcChannel { + credential_epoch, + generation: self + .next_connection_generation + .fetch_add(1, Ordering::Relaxed), + channel, + }); + *self.mediation.lock().await = None; + Ok(()) + } + + /// Replace a failed physical transport and replay the authenticated + /// lifecycle needed to make the new HTTP/2 connection authoritative. + async fn recover_after_unavailable(&self) -> Result<(), BackendError> { + let observed_generation = self + .grpc_channel + .lock() + .await + .as_ref() + .map(|cached| cached.generation); + let _reconnect = self.reconnect.lock().await; + if self + .grpc_channel + .lock() + .await + .as_ref() + .map(|cached| cached.generation) + != observed_generation + { + return Ok(()); + } + + *self.grpc_channel.lock().await = None; + *self.mediation.lock().await = None; + let attach = self + .attach_request + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone(); + let Some(attach) = attach else { + return Ok(()); + }; + let confirm = self + .confirm_request + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone(); + let credential_epoch = self.sandbox_bearer.credential_epoch().ok_or_else(|| { + BackendError::Unavailable("Sandbox Protocol credential unavailable".to_string()) + })?; + let channel = self.build_grpc_channel().await?; + self.exchange_on_channel(channel.clone(), &attach).await?; + if let Some(confirm) = confirm { + self.exchange_on_channel(channel.clone(), &confirm).await?; + } + *self.grpc_channel.lock().await = Some(CachedGrpcChannel { + credential_epoch, + generation: self + .next_connection_generation + .fetch_add(1, Ordering::Relaxed), + channel, + }); + Ok(()) + } + + fn start_credential_monitor(self: &Arc) { + if self.credential_monitor_started.swap(true, Ordering::AcqRel) { + return; + } + let client = Arc::downgrade(self); + tokio::spawn(async move { + loop { + tokio::time::sleep(Duration::from_secs(1)).await; + let Some(client) = client.upgrade() else { + return; + }; + if let Err(error) = client.ensure_current_credential_connection().await { + tracing::warn!(%error, "failed to rotate Sandbox Protocol connection"); + } + } + }); + } + + async fn exchange_on_channel( + &self, + channel: tonic::transport::Channel, + envelope: &RequestEnvelope, + ) -> Result { + let request_id = envelope.request_id.clone(); + let mut stream = + open_grpc_client_stream(channel, GrpcStreamKind::Exchange, &self.sandbox_bearer) + .await?; + let frame = encode_frame(envelope) + .map_err(|error| BackendError::Process(format!("encode control request: {error}")))?; + stream.write_all(&frame).await.map_err(|error| { + BackendError::Unavailable(format!("write boundary control request: {error}")) + })?; + stream.flush().await.map_err(|error| { + BackendError::Unavailable(format!("flush boundary control request: {error}")) + })?; + let response = + crate::boundary_protocol::read_frame_async::<_, ResponseEnvelope>(&mut stream) + .await + .map_err(|error| { + BackendError::Unavailable(format!("read boundary control response: {error}")) + })?; + if response.request_id != request_id { + return Err(BackendError::Process( + "boundary response ID did not match request ID".to_string(), + )); + } + match response.response { + Response::Error { kind, message } => Err(guest_error(kind, message)), + response => Ok(response), + } + } + + async fn mediation_session(&self) -> Result, BackendError> { + if let Some(session) = self.healthy_mediation_session().await { + return Ok(session); + } + + // Serialize creation without holding the cached-session mutex. Opening + // a stream may rotate credentials or recover the physical connection; + // both paths clear the cache and must be free to acquire that mutex. + let _opening = self.mediation_open.lock().await; + if let Some(session) = self.healthy_mediation_session().await { + return Ok(session); + } + + // The boundary owns exclusive-lease retirement and bounds replacement + // waiting. Never multiply that deadline with message-matching retries. + let session = tokio::time::timeout(REQUEST_TIMEOUT, async { + loop { + match self.open_mediation_session().await { + Ok(session) => return Ok(session), + Err(BackendError::Unavailable(message)) + if is_transport_unavailable(&message) => + { + self.recover_after_unavailable().await?; + tokio::time::sleep(Duration::from_millis(25)).await; + } + Err(error) => return Err(error), + } + } + }) + .await + .map_err(|_| { + BackendError::Unavailable("boundary mediation attach timed out".to_string()) + })??; + *self.mediation.lock().await = Some(session.clone()); + Ok(session) + } + + async fn healthy_mediation_session(&self) -> Option> { + self.mediation + .lock() + .await + .as_ref() + .filter(|session| session.is_healthy()) + .cloned() + } + + async fn open_mediation_session(&self) -> Result, BackendError> { + let mut stream = self.open_grpc_stream(GrpcStreamKind::Mediate).await?; + let envelope = Self::prepare_request(Request::OpenMediation)?; + let request_id = envelope.request_id.clone(); + let frame = encode_frame(&envelope) + .map_err(|error| BackendError::Process(format!("encode mediation attach: {error}")))?; + stream.write_all(&frame).await.map_err(|error| { + BackendError::Unavailable(format!("write mediation attach: {error}")) + })?; + stream.flush().await.map_err(|error| { + BackendError::Unavailable(format!("flush mediation attach: {error}")) + })?; + let response = + crate::boundary_protocol::read_frame_async::<_, ResponseEnvelope>(&mut stream) + .await + .map_err(|error| { + BackendError::Unavailable(format!("read mediation attach: {error}")) + })?; + if response.request_id != request_id { + return Err(BackendError::Process( + "mediation attach response ID did not match request".to_string(), + )); + } + match response.response { + Response::MediationReady => {} + Response::Error { kind, message } => return Err(guest_error(kind, message)), + response => return Err(unexpected_response("mediation_ready", &response)), + } + Ok(ClientMediationSession::start(stream)) + } + + async fn grpc_channel(&self) -> Result { + let mut state = self.grpc_channel.lock().await; + if let Some(cached) = state.as_ref() { + return Ok(cached.channel.clone()); + } + let credential_epoch = self.sandbox_bearer.credential_epoch().ok_or_else(|| { + BackendError::Unavailable("Sandbox Protocol credential unavailable".to_string()) + })?; + let channel = self.build_grpc_channel().await?; + *state = Some(CachedGrpcChannel { + credential_epoch, + generation: self + .next_connection_generation + .fetch_add(1, Ordering::Relaxed), + channel: channel.clone(), + }); + Ok(channel) + } + + async fn build_grpc_channel(&self) -> Result { + let runtime_descriptor = self.runtime_descriptor.clone(); + let endpoint = + tonic::transport::Endpoint::from_static("http://boundary.openshell.internal") + .initial_stream_window_size(16 * 1024 * 1024) + .initial_connection_window_size(16 * 1024 * 1024) + .http2_keep_alive_interval(Duration::from_secs(10)) + .keep_alive_while_idle(true); + let channel = endpoint + .connect_with_connector(tower::service_fn(move |_: tonic::transport::Uri| { + let runtime_descriptor = runtime_descriptor.clone(); + async move { + connect_boundary_with_retry(&runtime_descriptor) + .await + .map(TokioIo::new) + .map_err(|error| std::io::Error::other(error.to_string())) + } + })) + .await + .map_err(|error| { + BackendError::Unavailable(format!("start boundary gRPC channel: {error}")) + })?; + Ok(channel) + } + + #[cfg(test)] + async fn connect_boundary_once(&self) -> Result { + connect_boundary_once(&self.runtime_descriptor).await + } +} + +async fn connect_boundary_with_retry( + runtime_descriptor: &SandboxRuntimeDescriptor, +) -> Result { + let deadline = tokio::time::Instant::now() + CONNECT_RETRY_TIMEOUT; + loop { + match connect_boundary_once(runtime_descriptor).await { + Ok(stream) => return Ok(stream), + Err(error) if tokio::time::Instant::now() >= deadline => return Err(error), + Err(_) => tokio::time::sleep(Duration::from_millis(25)).await, + } + } +} + +async fn connect_boundary_once( + runtime_descriptor: &SandboxRuntimeDescriptor, +) -> Result { + let stream: BoundaryDuplexStream = match &runtime_descriptor.transport { + #[cfg(unix)] + SandboxTransport::Unix { socket_path } => { + let stream = UnixStream::connect(socket_path).await.map_err(|error| { + BackendError::Unavailable(format!( + "connect to mapped boundary control socket {}: {error}", + socket_path.display() + )) + })?; + Box::new(stream) + } + #[cfg(not(unix))] + SandboxTransport::Unix { .. } => { + return Err(BackendError::Unavailable( + "Unix boundary transport requires a Unix host".to_string(), + )); + } + SandboxTransport::Tcp { + authority, + addresses, + } => { + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addresses) + .await + .map_err(|error| { + BackendError::Unavailable(format!( + "connect to boundary TLS endpoint {authority}: {error}" + )) + })?; + enable_boundary_tcp_keepalive(&stream); + Box::new(stream) + } + SandboxTransport::Vsock { guest_cid, port } => connect_host_vsock(*guest_cid, *port)?, + }; + let tls = &runtime_descriptor.tls; + let server_name = + rustls::pki_types::ServerName::try_from(tls.server_name.clone()).map_err(|error| { + BackendError::Descriptor(format!( + "boundary TLS server name {:?} is invalid: {error}", + tls.server_name + )) + })?; + let connector = tokio_rustls::TlsConnector::from(Arc::new(tls_client_config(tls)?)); + let stream = connector + .connect(server_name, stream) + .await + .map_err(|error| { + BackendError::Unavailable(format!("authenticate sandbox channel: {error}")) + })?; + Ok(Box::new(stream)) +} + +#[derive(Clone, Copy)] +enum GrpcStreamKind { + Exchange, + Mediate, +} + +async fn open_grpc_client_stream( + channel: tonic::transport::Channel, + kind: GrpcStreamKind, + sandbox_bearer: &openshell_core::jwt::SessionBearerTokenSlot, +) -> Result { + let (application, bridge) = tokio::io::duplex(256 * 1024); + let (reader, writer) = tokio::io::split(bridge); + let (outbound, outbound_rx) = tokio::sync::mpsc::channel::(64); + tokio::spawn(pump_to_grpc(reader, outbound)); + let mut client = IsolationBoundaryClient::new(channel) + .max_decoding_message_size(64 * 1024) + .max_encoding_message_size(64 * 1024); + let mut request = tonic::Request::new(ReceiverStream::new(outbound_rx)); + let authorization = sandbox_bearer.authorization_metadata().map_err(|error| { + BackendError::Unavailable(format!("Sandbox Protocol credential unavailable: {error}")) + })?; + request + .metadata_mut() + .insert("authorization", authorization); + let response = match kind { + GrpcStreamKind::Exchange => client.exchange(request).await, + GrpcStreamKind::Mediate => client.mediate(request).await, + } + .map_err(|error| match error.code() { + tonic::Code::Unauthenticated | tonic::Code::PermissionDenied => { + BackendError::Denied(format!("authenticate Sandbox Protocol stream: {error}")) + } + _ => BackendError::Unavailable(format!("open boundary gRPC stream: {error}")), + })?; + tokio::spawn(pump_from_grpc(response.into_inner(), writer)); + Ok(Box::new(application)) +} + +async fn pump_to_grpc(mut reader: R, sender: tokio::sync::mpsc::Sender) +where + R: tokio::io::AsyncRead + Unpin, +{ + let mut buffer = vec![0_u8; 16 * 1024]; + loop { + let read = match reader.read(&mut buffer).await { + Ok(read) => read, + Err(error) => { + tracing::debug!(%error, "boundary gRPC request reader ended"); + return; + } + }; + if read == 0 { + return; + } + if sender + .send(BoundaryChunk { + data: buffer[..read].to_vec(), + }) + .await + .is_err() + { + return; + } + } +} + +async fn pump_from_grpc(mut stream: tonic::Streaming, mut writer: W) +where + W: tokio::io::AsyncWrite + Unpin, +{ + loop { + match stream.message().await { + Ok(Some(chunk)) => { + if let Err(error) = writer.write_all(&chunk.data).await { + tracing::debug!(%error, "boundary gRPC response writer ended"); + return; + } + } + Ok(None) => { + let _ = writer.shutdown().await; + return; + } + Err(error) => { + tracing::debug!(%error, "boundary gRPC response stream ended"); + return; + } + } + } +} + +fn enable_boundary_tcp_keepalive(stream: &tokio::net::TcpStream) { + let keepalive = socket2::TcpKeepalive::new() + .with_time(Duration::from_secs(30)) + .with_interval(Duration::from_secs(10)); + let _ = socket2::SockRef::from(stream).set_tcp_keepalive(&keepalive); +} + +#[cfg(target_os = "linux")] +fn connect_host_vsock( + guest_cid: u32, + control_port: u32, +) -> Result { + let fd = unsafe { libc::socket(libc::AF_VSOCK, libc::SOCK_STREAM | libc::SOCK_CLOEXEC, 0) }; + if fd < 0 { + return Err(BackendError::Unavailable(format!( + "create host vsock: {}", + std::io::Error::last_os_error() + ))); + } + let fd = unsafe { std::os::fd::OwnedFd::from_raw_fd(fd) }; + let family = libc::sa_family_t::try_from(libc::AF_VSOCK).map_err(|error| { + BackendError::Unavailable(format!("convert host vsock address family: {error}")) + })?; + let address = libc::sockaddr_vm { + svm_family: family, + svm_reserved1: 0, + svm_port: control_port, + svm_cid: guest_cid, + svm_zero: [0; 4], + }; + let address_length = + libc::socklen_t::try_from(size_of::()).map_err(|error| { + BackendError::Unavailable(format!("convert host vsock address length: {error}")) + })?; + let result = unsafe { + libc::connect( + std::os::fd::AsRawFd::as_raw_fd(&fd), + (&raw const address).cast::(), + address_length, + ) + }; + if result != 0 { + return Err(BackendError::Unavailable(format!( + "connect host vsock CID {guest_cid} port {control_port}: {}", + std::io::Error::last_os_error() + ))); + } + let stream = unsafe { std::os::unix::net::UnixStream::from_raw_fd(fd.into_raw_fd()) }; + stream.set_nonblocking(true).map_err(|error| { + BackendError::Unavailable(format!("set host vsock nonblocking: {error}")) + })?; + let stream = UnixStream::from_std(stream).map_err(|error| { + BackendError::Unavailable(format!("register host vsock with Tokio: {error}")) + })?; + Ok(Box::new(stream)) +} + +#[cfg(not(target_os = "linux"))] +fn connect_host_vsock( + _guest_cid: u32, + _control_port: u32, +) -> Result { + Err(BackendError::Unavailable( + "host AF_VSOCK transport is supported only on Linux".to_string(), + )) +} + +fn expect_response(response: Response, expected: &str) -> Result<(), BackendError> { + let matches = matches!( + (&response, expected), + (Response::Attached { .. }, "attached") + | (Response::Confirmed { .. }, "confirmed") + | (Response::Signaled, "signaled") + | (Response::Terminated, "terminated") + | (Response::BoundaryTerminated, "boundary_terminated") + ); + if matches { + Ok(()) + } else { + Err(unexpected_response(expected, &response)) + } +} + +fn unexpected_response(expected: &str, response: &Response) -> BackendError { + BackendError::Process(format!( + "expected boundary response {expected:?}, received {response:?}" + )) +} + +fn guest_error(kind: crate::boundary_protocol::BoundaryErrorKind, message: String) -> BackendError { + use crate::boundary_protocol::BoundaryErrorKind; + let message = format!("boundary process leaf: {message}"); + match kind { + BoundaryErrorKind::Invalid => BackendError::Descriptor(message), + BoundaryErrorKind::Denied => BackendError::Denied(message), + BoundaryErrorKind::Unavailable => BackendError::Unavailable(message), + BoundaryErrorKind::Terminated => BackendError::Terminated(message), + BoundaryErrorKind::Process => BackendError::Process(message), + } +} + +fn is_transport_unavailable(message: &str) -> bool { + !message.starts_with("boundary process leaf:") +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + use std::pin::Pin; + use std::task::{Context, Poll}; + + use super::*; + use crate::boundary_protocol::{ExitStatusWire, generate_sandbox_tls_material}; + use crate::proto::{ + BoundaryChunk, + isolation_boundary_server::{IsolationBoundary, IsolationBoundaryServer}, + }; + use openshell_core::jwt::{SecretJwt, SessionBearerTokenSlot}; + use openshell_core::policy::{ + FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, SandboxPolicy, + }; + + fn test_driver_fence() -> openshell_isolation_interface::contract::DriverFenceEvidence { + openshell_isolation_interface::contract::DriverFenceEvidence::Vm { + generation: "test-generation".to_string(), + network_device_count: 0, + } + } + + #[tokio::test] + async fn boundary_tcp_connections_enable_keepalive() { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let connected = tokio::spawn(async move { tokio::net::TcpStream::connect(address).await }); + let (_server, _) = listener.accept().await.unwrap(); + let client = connected.await.unwrap().unwrap(); + + enable_boundary_tcp_keepalive(&client); + + assert!(socket2::SockRef::from(&client).keepalive().unwrap()); + } + + #[derive(Clone)] + struct TestGrpcBoundary { + wait_for_half_close: bool, + expected_token: String, + requests: Arc, + mediation_failures: Arc, + mediation_ready: bool, + } + + type TestGrpcStream = Pin< + Box> + Send + 'static>, + >; + + #[tonic::async_trait] + impl IsolationBoundary for TestGrpcBoundary { + type ExchangeStream = TestGrpcStream; + type MediateStream = TestGrpcStream; + + async fn exchange( + &self, + request: tonic::Request>, + ) -> Result, tonic::Status> { + let authorization = request + .metadata() + .get("authorization") + .and_then(|value| value.to_str().ok()); + if authorization != Some(format!("Bearer {}", self.expected_token).as_str()) { + return Err(tonic::Status::unauthenticated( + "Sandbox Protocol bearer token did not match", + )); + } + let mut inbound = request.into_inner(); + let wait_for_half_close = self.wait_for_half_close; + let requests = self.requests.clone(); + let mediation_ready = self.mediation_ready; + let (outbound, outbound_rx) = tokio::sync::mpsc::channel(1); + tokio::spawn(async move { + let mut frame = Vec::new(); + loop { + match inbound.message().await { + Ok(Some(chunk)) => { + frame.extend_from_slice(&chunk.data); + if !wait_for_half_close && complete_control_frame(&frame) { + break; + } + } + Ok(None) => break, + Err(error) => { + let _ = outbound.send(Err(error)).await; + return; + } + } + } + requests.fetch_add(1, Ordering::AcqRel); + let response = if complete_control_frame(&frame) { + let envelope: RequestEnvelope = match decode_frame(&frame) { + Ok(envelope) => envelope, + Err(error) => { + let _ = outbound + .send(Err(tonic::Status::invalid_argument(error.to_string()))) + .await; + return; + } + }; + match encode_frame(&ResponseEnvelope { + request_id: envelope.request_id, + response: match envelope.request { + Request::Attach { .. } => Response::Attached { + snapshot: crate::boundary_protocol::SessionSnapshotWire { + generation: "test-generation".to_string(), + processes: Vec::new(), + }, + }, + Request::Confirm => Response::Confirmed { + evidence: Box::new(test_confirmation_evidence()), + }, + Request::OpenMediation if mediation_ready => Response::MediationReady, + Request::OpenMediation => Response::Error { + kind: crate::boundary_protocol::BoundaryErrorKind::Denied, + message: "a mediation session is already active".to_string(), + }, + Request::Wait { .. } => Response::Exited { + status: ExitStatusWire::Exited(23), + }, + Request::Exec { .. } => Response::ExecStarted { + process_id: "test-generation:exec:1".to_string(), + pty: false, + }, + Request::AttachProcess { .. } => { + Response::ProcessAttached { terminal: false } + } + Request::Signal { .. } | Request::ExecSignal { .. } => { + Response::Signaled + } + Request::Terminate { .. } => Response::Terminated, + Request::TerminateBoundary => Response::BoundaryTerminated, + Request::UpdateProviderEnvironment { revision, .. } => { + Response::ProviderEnvironmentUpdated { revision } + } + Request::Resize { .. } => Response::Resized, + Request::LoopbackConnect { .. } => Response::PortConnected, + Request::StartAgent { + provider_env_revision, + .. + } => Response::Started { + process_id: "test-generation:main:0".to_string(), + provider_env_revision, + }, + Request::AcceptNetwork => Response::Error { + kind: crate::boundary_protocol::BoundaryErrorKind::Unavailable, + message: "no pending network request".to_string(), + }, + }, + }) { + Ok(response) => response, + Err(error) => { + let _ = outbound + .send(Err(tonic::Status::internal(error.to_string()))) + .await; + return; + } + } + } else { + b"complete response".to_vec() + }; + let _ = outbound.send(Ok(BoundaryChunk { data: response })).await; + }); + Ok(tonic::Response::new(Box::pin(ReceiverStream::new( + outbound_rx, + )))) + } + + async fn mediate( + &self, + request: tonic::Request>, + ) -> Result, tonic::Status> { + if self + .mediation_failures + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |remaining| { + remaining.checked_sub(1) + }) + .is_ok() + { + return Err(tonic::Status::unavailable( + "injected mediation transport failure", + )); + } + self.exchange(request).await + } + } + + fn complete_control_frame(frame: &[u8]) -> bool { + frame.len() >= 4 + && frame.len() + >= 4 + usize::try_from(u32::from_be_bytes( + frame[..4].try_into().expect("frame header"), + )) + .expect("frame length") + } + + #[tokio::test] + async fn mediation_denial_is_not_retried_or_cached_as_a_session() { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let service = TestGrpcBoundary { + wait_for_half_close: false, + expected_token: "a".repeat(32), + requests: requests.clone(), + mediation_failures: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + mediation_ready: false, + }; + let server = tokio::spawn(async move { + let (stream, _) = listener.accept().await.unwrap(); + tonic::transport::Server::builder() + .add_service(IsolationBoundaryServer::new(service)) + .serve_with_incoming(tokio_stream::iter([Ok::<_, std::io::Error>(stream)])) + .await + .unwrap(); + }); + let channel = tonic::transport::Endpoint::from_shared(format!("http://{address}")) + .unwrap() + .connect() + .await + .unwrap(); + let client = BoundaryClient::new( + tls_runtime_descriptor(address, test_certificate().client_tls), + test_bearer(&"a".repeat(32)), + ); + *client.grpc_channel.lock().await = Some(CachedGrpcChannel { + credential_epoch: openshell_core::jwt::CredentialEpoch::new(1).expect("test epoch"), + generation: 1, + channel, + }); + // A caller may try again later, but each call makes exactly one + // bounded attach attempt and preserves the server's typed denial. + for expected_requests in 1..=2 { + assert!(matches!( + client.mediation_session().await, + Err(BackendError::Denied(_)) + )); + assert_eq!(requests.load(Ordering::Acquire), expected_requests); + assert!(client.mediation.lock().await.is_none()); + } + server.abort(); + } + + #[tokio::test] + async fn recovery_replays_attach_and_confirm_on_a_new_physical_connection() { + let certificate = test_certificate(); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let accepted = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let server_requests = requests.clone(); + let server_accepted = accepted.clone(); + let server_config = certificate.server_config.clone(); + let server = tokio::spawn(async move { + for _ in 0..2 { + let (stream, _) = listener.accept().await.unwrap(); + let stream = tokio_rustls::TlsAcceptor::from(server_config.clone()) + .accept(stream) + .await + .unwrap(); + server_accepted.fetch_add(1, Ordering::AcqRel); + let service = TestGrpcBoundary { + wait_for_half_close: false, + expected_token: "a".repeat(32), + requests: server_requests.clone(), + mediation_failures: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + mediation_ready: false, + }; + tokio::spawn(async move { + tonic::transport::Server::builder() + .add_service(IsolationBoundaryServer::new(service)) + .serve_with_incoming(tokio_stream::iter([Ok::<_, std::io::Error>( + TestTlsIo(Box::new(stream)), + )])) + .await + .unwrap(); + }); + } + }); + let client = BoundaryClient::new( + tls_runtime_descriptor(address, certificate.client_tls), + test_bearer(&"a".repeat(32)), + ); + let attach = Request::Attach { + supervisor_instance_id: client.supervisor_instance_id, + policy: Box::new(SandboxPolicyWire::from(sandbox().policy)), + resource_claims: std::collections::BTreeMap::new(), + }; + assert!(matches!( + client.call_idempotent(attach).await.unwrap(), + Response::Attached { .. } + )); + assert!(matches!( + client.call_idempotent(Request::Confirm).await.unwrap(), + Response::Confirmed { .. } + )); + + client.recover_after_unavailable().await.unwrap(); + tokio::time::timeout(Duration::from_secs(1), server) + .await + .expect("replacement connection accepted") + .unwrap(); + assert_eq!(accepted.load(Ordering::Acquire), 2); + assert_eq!(requests.load(Ordering::Acquire), 4); + } + + #[test] + fn wait_recovery_window_begins_at_transport_failure() { + let wait_started = tokio::time::Instant::now(); + let failure_time = wait_started + CONNECT_RETRY_TIMEOUT + Duration::from_secs(5); + let mut deadline = None; + + assert_eq!( + begin_recovery_window(&mut deadline, failure_time), + failure_time + CONNECT_RETRY_TIMEOUT + ); + } + + #[tokio::test] + async fn mediation_transport_failure_recovers_without_deadlocking_the_cache() { + let certificate = test_certificate(); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let mediation_failures = Arc::new(std::sync::atomic::AtomicUsize::new(1)); + let server_config = certificate.server_config.clone(); + let server_requests = requests.clone(); + let server_failures = mediation_failures.clone(); + let server = tokio::spawn(async move { + for _ in 0..2 { + let (stream, _) = listener.accept().await.unwrap(); + let stream = tokio_rustls::TlsAcceptor::from(server_config.clone()) + .accept(stream) + .await + .unwrap(); + let service = TestGrpcBoundary { + wait_for_half_close: false, + expected_token: "a".repeat(32), + requests: server_requests.clone(), + mediation_failures: server_failures.clone(), + mediation_ready: true, + }; + tokio::spawn(async move { + tonic::transport::Server::builder() + .add_service(IsolationBoundaryServer::new(service)) + .serve_with_incoming(tokio_stream::iter([Ok::<_, std::io::Error>( + TestTlsIo(Box::new(stream)), + )])) + .await + .unwrap(); + }); + } + }); + let client = BoundaryClient::new( + tls_runtime_descriptor(address, certificate.client_tls), + test_bearer(&"a".repeat(32)), + ); + let attach = Request::Attach { + supervisor_instance_id: client.supervisor_instance_id, + policy: Box::new(SandboxPolicyWire::from(sandbox().policy)), + resource_claims: std::collections::BTreeMap::new(), + }; + assert!(matches!( + client.call_idempotent(attach).await.unwrap(), + Response::Attached { .. } + )); + assert!(matches!( + client.call_idempotent(Request::Confirm).await.unwrap(), + Response::Confirmed { .. } + )); + + tokio::time::timeout(Duration::from_secs(2), client.mediation_session()) + .await + .expect("mediation recovery must not deadlock") + .expect("mediation recovery must open a replacement session"); + tokio::time::timeout(Duration::from_secs(1), server) + .await + .expect("replacement physical connection must be accepted") + .unwrap(); + assert_eq!(mediation_failures.load(Ordering::Acquire), 0); + assert_eq!(requests.load(Ordering::Acquire), 5); + } + + #[tokio::test] + async fn grpc_stream_preserves_response_after_request_half_close() { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let service = TestGrpcBoundary { + wait_for_half_close: true, + expected_token: "a".repeat(32), + requests, + mediation_failures: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + mediation_ready: false, + }; + let server = tokio::spawn(async move { + let (stream, _) = listener.accept().await.unwrap(); + tonic::transport::Server::builder() + .add_service(IsolationBoundaryServer::new(service)) + .serve_with_incoming(tokio_stream::iter([Ok::<_, std::io::Error>(stream)])) + .await + .unwrap(); + }); + let channel = tonic::transport::Endpoint::from_shared(format!("http://{address}")) + .unwrap() + .connect() + .await + .unwrap(); + let mut stream = open_grpc_client_stream( + channel, + GrpcStreamKind::Exchange, + &test_bearer(&"a".repeat(32)), + ) + .await + .unwrap(); + stream.write_all(b"finite request").await.unwrap(); + stream.shutdown().await.unwrap(); + let mut response = [0_u8; 17]; + stream.read_exact(&mut response).await.unwrap(); + assert_eq!(&response, b"complete response"); + drop(stream); + server.abort(); + } + + #[tokio::test] + async fn persistent_dns_exchange_returns_supervisor_response() { + let (client_stream, mut server_stream) = tokio::io::duplex(4096); + let session = ClientMediationSession::start(Box::new(client_stream)); + let server = tokio::spawn(async move { + let query = DnsQueryWire { + request: vec![1, 2, 3], + transport: openshell_isolation_interface::contract::DnsTransport::Udp, + identity: crate::boundary_protocol::BinaryIdentityWire::Resolved { + binary_path: PathBuf::from("/usr/bin/dig"), + binary_digest: Some("a".repeat(64).parse().unwrap()), + ancestors: Vec::new(), + cmdline_paths: Vec::new(), + }, + timing: crate::boundary_protocol::MediationTimingWire::default(), + }; + mediation::write_frame( + &mut server_stream, + MediationFrameKind::DnsQuery, + 42, + &mediation::encode_json(&query).unwrap(), + ) + .await + .unwrap(); + let reply = mediation::read_frame(&mut server_stream) + .await + .unwrap() + .unwrap(); + assert_eq!(reply.kind, MediationFrameKind::DnsResponse); + assert_eq!(reply.stream_id, 42); + assert_eq!( + mediation::decode_json::(&reply.payload).unwrap(), + DnsQueryResultWire::Response(vec![4, 5, 6]) + ); + }); + let query = session.accept_dns().await.unwrap(); + assert_eq!(query.message, [1, 2, 3]); + assert_eq!( + query.binary_identity.unwrap().binary_path, + PathBuf::from("/usr/bin/dig") + ); + query.response.send(Ok(vec![4, 5, 6])).unwrap(); + server.await.unwrap(); + } + + struct TestCertificate { + client_tls: SandboxTlsClientConfig, + server_config: Arc, + } + + fn test_certificate() -> TestCertificate { + test_certificate_with_protocol_versions(&[&rustls::version::TLS13]) + } + + fn test_certificate_with_protocol_versions( + protocol_versions: &[&'static rustls::SupportedProtocolVersion], + ) -> TestCertificate { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let material = + generate_sandbox_tls_material(test_session_id()).expect("generate test material"); + let certificates = rustls_pemfile::certs(&mut material.certificate_chain_pem.as_bytes()) + .collect::, _>>() + .expect("parse server certificate"); + let private_key = rustls_pemfile::private_key(&mut material.private_key_pem.as_bytes()) + .expect("parse server private key") + .expect("server private key"); + let mut server_config = + rustls::ServerConfig::builder_with_protocol_versions(protocol_versions) + .with_no_client_auth() + .with_single_cert(certificates, private_key) + .expect("build test TLS server config"); + server_config.alpn_protocols = vec![b"h2".to_vec()]; + TestCertificate { + client_tls: SandboxTlsClientConfig { + server_name: material.server_name, + trust_anchor_pem: material.trust_anchor_pem, + }, + server_config: Arc::new(server_config), + } + } + + fn tls_runtime_descriptor( + address: std::net::SocketAddr, + tls: SandboxTlsClientConfig, + ) -> SandboxRuntimeDescriptor { + SandboxRuntimeDescriptor { + boundary_id: "sandbox-1".to_string(), + generation: "test-generation".to_string(), + session_id: test_session_id(), + workload_identity: sandbox().identity, + transport: SandboxTransport::Tcp { + authority: "sandbox.test".to_string(), + addresses: vec![address], + }, + tls, + host_gateway_ip: None, + resource_claims: std::collections::BTreeMap::new(), + driver_fence: test_driver_fence(), + } + } + + fn test_session_id() -> openshell_core::SandboxSessionId { + "550e8400-e29b-41d4-a716-446655440000" + .parse() + .expect("test session ID") + } + + fn test_bearer(token: &str) -> SessionBearerTokenSlot { + SessionBearerTokenSlot::new( + SecretJwt::parse(token).expect("test bearer"), + i64::MAX, + openshell_core::jwt::CredentialEpoch::new(1).expect("test epoch"), + ) + .expect("test bearer slot") + } + + async fn spawn_tls_boundary( + certificate: Arc, + expected_token: String, + ) -> (std::net::SocketAddr, tokio::task::JoinHandle<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { + let (stream, _) = listener.accept().await.unwrap(); + let Ok(stream) = tokio_rustls::TlsAcceptor::from(certificate) + .accept(stream) + .await + else { + return; + }; + serve_test_grpc(Box::new(stream), expected_token).await; + }); + (address, task) + } + + async fn serve_test_grpc(stream: BoundaryDuplexStream, expected_token: String) { + let service = TestGrpcBoundary { + wait_for_half_close: false, + expected_token, + requests: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + mediation_failures: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + mediation_ready: false, + }; + tonic::transport::Server::builder() + .add_service(IsolationBoundaryServer::new(service)) + .serve_with_incoming(tokio_stream::iter([Ok::<_, std::io::Error>(TestTlsIo( + stream, + ))])) + .await + .unwrap(); + } + + fn sandbox() -> SandboxContext { + SandboxContext { + sandbox_id: "sandbox-1".to_string(), + session_id: test_session_id(), + policy: SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy::default(), + network: NetworkPolicy::default(), + landlock: LandlockPolicy::default(), + process: ProcessPolicy::default(), + }, + agent: AgentSpec { + program: "/bin/true".to_string(), + args: Vec::new(), + workdir: Some("/sandbox".to_string()), + timeout_secs: 5, + interactive: false, + }, + identity: openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new( + 10_001, + 10_001, + Vec::new(), + "test".to_string(), + "sha256:test".to_string(), + ) + .expect("identity"), + } + } + + fn test_confirmation_evidence() + -> openshell_isolation_interface::contract::SandboxConfirmEvidence { + openshell_isolation_interface::contract::SandboxConfirmEvidence { + generation: "test-generation".to_string(), + identity: sandbox().identity, + capabilities: openshell_isolation_interface::contract::CapabilityEvidence { + inheritable: 0, + permitted: 0, + effective: 0, + bounding: 0, + ambient: 0, + }, + no_new_privileges: true, + sandbox_dumpable: false, + child_dumpable: true, + core_limit_zero: true, + native_architecture: std::env::consts::ARCH.to_string(), + kernel_release: "test".to_string(), + seccomp: openshell_isolation_interface::contract::SeccompEvidence { + new_listener: true, + notification_round_trip: true, + id_validation: true, + addfd_send: true, + retained_socket_operation: true, + proc_fd_identity: true, + task_memory_read: true, + task_memory_write: true, + cancellation: true, + }, + landlock_abi: 3, + landlock_allow_deny: true, + udp_dns_round_trip: true, + tcp_dns_round_trip: true, + tcp_allow_round_trip: true, + tcp_deny_round_trip: true, + authenticated_supervisor: true, + session_id: test_session_id(), + driver_fence: test_driver_fence(), + runtime_exit_terminates_workload: true, + resource_claims: std::collections::BTreeMap::new(), + } + } + + #[test] + fn runtime_descriptor_debug_redacts_trust_anchor() { + let certificate = test_certificate(); + let runtime_descriptor = SandboxRuntimeDescriptor { + boundary_id: "sandbox-1".to_string(), + generation: "test-generation".to_string(), + session_id: test_session_id(), + workload_identity: sandbox().identity, + transport: SandboxTransport::Unix { + socket_path: PathBuf::from("/tmp/vsock.sock"), + }, + tls: certificate.client_tls.clone(), + host_gateway_ip: Some(std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST)), + resource_claims: std::collections::BTreeMap::new(), + driver_fence: test_driver_fence(), + }; + let debug = format!("{runtime_descriptor:?}"); + assert!(debug.contains("")); + assert!(!debug.contains(&certificate.client_tls.trust_anchor_pem)); + } + + #[test] + fn runtime_descriptor_must_match_sandbox() { + let runtime_descriptor = SandboxRuntimeDescriptor { + boundary_id: "other".to_string(), + generation: "test-generation".to_string(), + session_id: test_session_id(), + workload_identity: sandbox().identity, + transport: SandboxTransport::Unix { + socket_path: PathBuf::from("/tmp/vsock.sock"), + }, + tls: test_certificate().client_tls, + host_gateway_ip: None, + resource_claims: std::collections::BTreeMap::new(), + driver_fence: test_driver_fence(), + }; + assert!(matches!( + validate_runtime_descriptor(&runtime_descriptor, &sandbox()), + Err(BackendError::Descriptor(_)) + )); + } + + #[test] + fn runtime_descriptor_rejects_an_unspecified_tcp_target() { + let runtime_descriptor = SandboxRuntimeDescriptor { + boundary_id: "sandbox-1".to_string(), + generation: "test-generation".to_string(), + session_id: test_session_id(), + workload_identity: sandbox().identity, + transport: SandboxTransport::Tcp { + authority: "sandbox.test".to_string(), + addresses: vec!["0.0.0.0:5500".parse().expect("valid address")], + }, + tls: test_certificate().client_tls, + host_gateway_ip: None, + resource_claims: std::collections::BTreeMap::new(), + driver_fence: test_driver_fence(), + }; + assert!(matches!( + validate_runtime_descriptor(&runtime_descriptor, &sandbox()), + Err(BackendError::Descriptor(_)) + )); + } + + #[test] + fn runtime_descriptor_accepts_a_concrete_tcp_target() { + let runtime_descriptor = SandboxRuntimeDescriptor { + boundary_id: "sandbox-1".to_string(), + generation: "test-generation".to_string(), + session_id: test_session_id(), + workload_identity: sandbox().identity, + transport: SandboxTransport::Tcp { + authority: "sandbox.test".to_string(), + addresses: vec!["10.42.0.7:5500".parse().expect("valid address")], + }, + tls: test_certificate().client_tls, + host_gateway_ip: None, + resource_claims: std::collections::BTreeMap::new(), + driver_fence: test_driver_fence(), + }; + validate_runtime_descriptor(&runtime_descriptor, &sandbox()) + .expect("TCP runtime descriptor should be valid"); + } + + #[test] + fn runtime_descriptor_rejects_invalid_tls_configuration() { + let runtime_descriptor = tls_runtime_descriptor( + "127.0.0.1:5500".parse().expect("valid address"), + SandboxTlsClientConfig { + server_name: "not a dns name!".to_string(), + trust_anchor_pem: "not a certificate".to_string(), + }, + ); + assert!(matches!( + validate_runtime_descriptor(&runtime_descriptor, &sandbox()), + Err(BackendError::Descriptor(_)) + )); + } + + #[tokio::test] + async fn tls_tcp_round_trip_verifies_server_certificate() { + let certificate = test_certificate(); + let (address, server) = spawn_tls_boundary(certificate.server_config, "a".repeat(32)).await; + let client = BoundaryClient::new( + tls_runtime_descriptor(address, certificate.client_tls), + test_bearer(&"a".repeat(32)), + ); + + assert_eq!( + client + .exchange(Request::Confirm) + .await + .expect("TLS request"), + Response::Confirmed { + evidence: Box::new(test_confirmation_evidence()), + } + ); + server.abort(); + } + + #[tokio::test] + async fn tls_tcp_rejects_tls12_only_server() { + let certificate = test_certificate_with_protocol_versions(&[&rustls::version::TLS12]); + let (address, server) = spawn_tls_boundary(certificate.server_config, "a".repeat(32)).await; + let client_config = tls_client_config(&certificate.client_tls).expect("client TLS config"); + let server_name = + rustls::pki_types::ServerName::try_from(certificate.client_tls.server_name.clone()) + .expect("server name"); + let stream = tokio::net::TcpStream::connect(address) + .await + .expect("connect to TLS test server"); + + assert!( + tokio_rustls::TlsConnector::from(Arc::new(client_config)) + .connect(server_name, stream) + .await + .is_err() + ); + server.await.expect("TLS test server task"); + } + + #[tokio::test] + async fn exec_wait_survives_output_loss_and_reattachment() { + let certificate = test_certificate(); + let (address, server) = spawn_tls_boundary(certificate.server_config, "a".repeat(32)).await; + let client = Arc::new(BoundaryClient::new( + tls_runtime_descriptor(address, certificate.client_tls), + test_bearer(&"a".repeat(32)), + )); + let session = open_exec_session( + client, + ExecSpec { + program: "/bin/true".to_string(), + args: Vec::new(), + env: Vec::new(), + workdir: None, + pty: false, + }, + ) + .await + .unwrap(); + // The test peer closes its I/O stream without an exit frame. Neither + // that loss nor a dropped reader can invalidate the process handle. + drop(session.stdin); + drop(session.stdout); + drop(session.stderr); + let attachment = session.process.attach().await.unwrap(); + drop(attachment); + for _ in 0..2 { + assert_eq!( + session.process.wait().await.unwrap(), + BoundaryExitStatus::Exited(23) + ); + } + server.abort(); + } + + struct TestTlsIo(BoundaryDuplexStream); + + impl tokio::io::AsyncRead for TestTlsIo { + fn poll_read( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + buffer: &mut tokio::io::ReadBuf<'_>, + ) -> Poll> { + Pin::new(&mut self.0).poll_read(context, buffer) + } + } + + impl tokio::io::AsyncWrite for TestTlsIo { + fn poll_write( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + buffer: &[u8], + ) -> Poll> { + Pin::new(&mut self.0).poll_write(context, buffer) + } + + fn poll_flush( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + ) -> Poll> { + Pin::new(&mut self.0).poll_flush(context) + } + + fn poll_shutdown( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + ) -> Poll> { + Pin::new(&mut self.0).poll_shutdown(context) + } + } + + impl tonic::transport::server::Connected for TestTlsIo { + type ConnectInfo = (); + + fn connect_info(&self) -> Self::ConnectInfo {} + } + + #[tokio::test] + async fn grpc_session_reuses_one_tls_connection_for_concurrent_requests() { + const REQUESTS: usize = 8; + let certificate = test_certificate(); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind gRPC test boundary"); + let address = listener.local_addr().expect("gRPC listener address"); + let server_config = certificate.server_config; + let accepted = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let accepted_by_server = accepted.clone(); + let handled = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let service = TestGrpcBoundary { + wait_for_half_close: false, + expected_token: "a".repeat(32), + requests: handled.clone(), + mediation_failures: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + mediation_ready: false, + }; + let server = tokio::spawn(async move { + loop { + let (stream, _) = listener.accept().await.expect("accept TLS session"); + accepted_by_server.fetch_add(1, Ordering::AcqRel); + let stream = tokio_rustls::TlsAcceptor::from(server_config.clone()) + .accept(stream) + .await + .expect("authenticate gRPC TLS session"); + let service = service.clone(); + tokio::spawn(async move { + tonic::transport::Server::builder() + .add_service(IsolationBoundaryServer::new(service)) + .serve_with_incoming(tokio_stream::iter([Ok::<_, std::io::Error>( + TestTlsIo(Box::new(stream)), + )])) + .await + .expect("serve test gRPC connection"); + }); + } + }); + let client = Arc::new(BoundaryClient::new( + SandboxRuntimeDescriptor { + boundary_id: "sandbox-1".to_string(), + generation: "test-generation".to_string(), + session_id: test_session_id(), + workload_identity: sandbox().identity, + transport: SandboxTransport::Tcp { + authority: "sandbox.test".to_string(), + addresses: vec![address], + }, + tls: certificate.client_tls, + host_gateway_ip: None, + resource_claims: std::collections::BTreeMap::new(), + driver_fence: test_driver_fence(), + }, + test_bearer(&"a".repeat(32)), + )); + let mut requests = Vec::new(); + for _ in 0..REQUESTS { + let client = client.clone(); + requests.push(tokio::spawn(async move { + let response = client + .exchange(Request::Confirm) + .await + .expect("gRPC confirm request"); + assert!(matches!(response, Response::Confirmed { .. })); + })); + } + for request in requests { + request.await.expect("gRPC client request"); + } + tokio::time::sleep(Duration::from_millis(50)).await; + assert_eq!(handled.load(Ordering::Acquire), REQUESTS); + assert_eq!(accepted.load(Ordering::Acquire), 1); + server.abort(); + } + + #[tokio::test] + async fn tls_tcp_flushes_large_control_requests_before_reading_response() { + let certificate = test_certificate(); + let (address, server) = spawn_tls_boundary(certificate.server_config, "a".repeat(32)).await; + let client = BoundaryClient::new( + tls_runtime_descriptor(address, certificate.client_tls), + test_bearer(&"a".repeat(32)), + ); + let context = sandbox(); + + assert!(matches!( + client + .exchange(Request::StartAgent { + sandbox_id: context.sandbox_id, + spec: AgentSpecWire::from(context.agent), + policy: Box::new(SandboxPolicyWire::from(context.policy)), + ca_cert: Some(vec![b'c'; 16 * 1024]), + ca_bundle: Some(vec![b'b'; 256 * 1024]), + provider_env_revision: 0, + provider_env: HashMap::new(), + }) + .await + .expect("large TLS request"), + Response::Started { .. } + )); + server.abort(); + } + + #[cfg(unix)] + #[tokio::test] + async fn tls_unix_flushes_large_control_requests_before_reading_response() { + let socket_path = std::env::temp_dir().join(format!( + "openshell-large-control-{}-{}.sock", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system time") + .as_nanos() + )); + let certificate = test_certificate(); + let server_config = certificate.server_config; + let listener = tokio::net::UnixListener::bind(&socket_path).expect("bind test socket"); + let server = tokio::spawn(async move { + let (stream, _) = listener.accept().await.unwrap(); + let stream = tokio_rustls::TlsAcceptor::from(server_config) + .accept(stream) + .await + .unwrap(); + serve_test_grpc(Box::new(stream), "a".repeat(32)).await; + }); + let context = sandbox(); + let client = BoundaryClient::new( + SandboxRuntimeDescriptor { + boundary_id: "sandbox-1".to_string(), + generation: "test-generation".to_string(), + session_id: test_session_id(), + workload_identity: context.identity.clone(), + transport: SandboxTransport::Unix { + socket_path: socket_path.clone(), + }, + tls: certificate.client_tls, + host_gateway_ip: None, + resource_claims: std::collections::BTreeMap::new(), + driver_fence: test_driver_fence(), + }, + test_bearer(&"a".repeat(32)), + ); + + assert!(matches!( + tokio::time::timeout( + Duration::from_secs(2), + client.exchange(Request::StartAgent { + sandbox_id: context.sandbox_id, + spec: AgentSpecWire::from(context.agent), + policy: Box::new(SandboxPolicyWire::from(context.policy)), + ca_cert: Some(vec![b'c'; 16 * 1024]), + ca_bundle: Some(vec![b'b'; 256 * 1024]), + provider_env_revision: 0, + provider_env: HashMap::new(), + }) + ) + .await + .expect("large Unix TLS request timed out") + .expect("large Unix TLS request"), + Response::Started { .. } + )); + server.abort(); + let _ = std::fs::remove_file(socket_path); + } + + #[tokio::test] + async fn tls_tcp_preserves_boundary_token_authentication() { + let certificate = test_certificate(); + let (address, server) = spawn_tls_boundary( + certificate.server_config, + "expected-token-expected-token-12".to_string(), + ) + .await; + let client = BoundaryClient::new( + tls_runtime_descriptor(address, certificate.client_tls), + test_bearer("incorrect-token-incorrect-token"), + ); + + assert!(matches!( + client.exchange(Request::Confirm).await, + Err(BackendError::Denied(_)) + )); + server.abort(); + } + + #[tokio::test] + async fn tls_tcp_rejects_an_untrusted_server_certificate() { + let presented = test_certificate(); + let trusted = test_certificate(); + let (address, server) = spawn_tls_boundary(presented.server_config, "a".repeat(32)).await; + let client = BoundaryClient::new( + tls_runtime_descriptor(address, trusted.client_tls), + test_bearer(&"a".repeat(32)), + ); + + assert!(matches!( + client.connect_boundary_once().await, + Err(BackendError::Unavailable(_)) + )); + // The server observes the client's fatal alert and may fail its accept; + // completing the task is sufficient for this rejection test. + let _ = server.await; + } +} diff --git a/crates/openshell-sandbox-backend/src/sandbox_auth.rs b/crates/openshell-sandbox-backend/src/sandbox_auth.rs new file mode 100644 index 0000000000..8cd3878283 --- /dev/null +++ b/crates/openshell-sandbox-backend/src/sandbox_auth.rs @@ -0,0 +1,553 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Transport-neutral authentication for `OpenShell` Sandbox Protocol connections. + +use std::sync::Mutex; + +use openshell_core::jwt::{ + AuthenticatedSandboxSession, CredentialEpoch, SandboxId, SessionJwtError, SessionJwtVerifier, +}; +use openshell_core::sandbox_generation::SandboxGenerationId; +use tonic::metadata::MetadataMap; +use uuid::Uuid; + +/// Server-local identity assigned after a byte stream completes TLS. +/// +/// It cannot be supplied by a compute driver or workload. Protocol handlers use +/// it to bind authenticated requests to the connection that performed attach. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct SandboxConnectionId(Uuid); + +impl SandboxConnectionId { + #[must_use] + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for SandboxConnectionId { + fn default() -> Self { + Self::new() + } +} + +/// Principal returned only after strict bearer validation and identity binding. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SandboxProtocolPrincipal { + connection_id: SandboxConnectionId, + session: AuthenticatedSandboxSession, +} + +impl SandboxProtocolPrincipal { + #[must_use] + pub const fn connection_id(&self) -> SandboxConnectionId { + self.connection_id + } + + #[must_use] + pub const fn session(&self) -> &AuthenticatedSandboxSession { + &self.session + } +} + +/// Validates Sandbox Protocol metadata without depending on its byte transport. +pub struct SandboxProtocolAuthenticator { + verifier: SessionJwtVerifier, + expected_sandbox_id: SandboxId, + expected_runtime_generation: SandboxGenerationId, + expected_auth_epoch: CredentialEpoch, +} + +impl std::fmt::Debug for SandboxProtocolAuthenticator { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("SandboxProtocolAuthenticator") + .field("verifier", &self.verifier) + .field("expected_sandbox_id", &self.expected_sandbox_id) + .field( + "expected_runtime_generation", + &self.expected_runtime_generation, + ) + .field("expected_auth_epoch", &self.expected_auth_epoch) + .finish() + } +} + +impl SandboxProtocolAuthenticator { + #[must_use] + pub const fn new( + verifier: SessionJwtVerifier, + expected_sandbox_id: SandboxId, + expected_runtime_generation: SandboxGenerationId, + expected_auth_epoch: CredentialEpoch, + ) -> Self { + Self { + verifier, + expected_sandbox_id, + expected_runtime_generation, + expected_auth_epoch, + } + } + + pub fn authenticate( + &self, + connection_id: SandboxConnectionId, + metadata: &MetadataMap, + ) -> Result { + let mut values = metadata.get_all("authorization").iter(); + let value = values.next().ok_or(SandboxAuthError::MissingBearer)?; + if values.next().is_some() { + return Err(SandboxAuthError::DuplicateBearer); + } + let value = value + .to_str() + .map_err(|_| SandboxAuthError::InvalidBearer)?; + let token = value + .strip_prefix("Bearer ") + .filter(|token| !token.is_empty() && !token.chars().any(char::is_whitespace)) + .ok_or(SandboxAuthError::InvalidBearer)?; + let session = self.verifier.verify(token)?; + if session.sandbox_id != self.expected_sandbox_id { + return Err(SandboxAuthError::WrongSandbox); + } + if session.runtime_generation != self.expected_runtime_generation { + return Err(SandboxAuthError::WrongRuntimeGeneration); + } + if session.auth_epoch != self.expected_auth_epoch { + return Err(SandboxAuthError::StaleCredentialEpoch); + } + Ok(SandboxProtocolPrincipal { + connection_id, + session, + }) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct ActiveConnection { + id: SandboxConnectionId, + epoch: CredentialEpoch, +} + +#[derive(Debug)] +struct ConnectionState { + active: Option, + pending: Option, + highest_epoch: Option, + supervisor_instance_id: Option, + terminal: bool, +} + +/// Enforces the one-active-connection and monotonically increasing epoch rules. +#[derive(Debug)] +pub struct SandboxConnectionRegistry { + state: Mutex, +} + +impl SandboxConnectionRegistry { + #[must_use] + pub fn new( + _session_id: openshell_core::SandboxSessionId, + _session_rotation: openshell_core::jwt::SessionRotation, + ) -> Self { + Self { + state: Mutex::new(ConnectionState { + active: None, + pending: None, + highest_epoch: None, + supervisor_instance_id: None, + terminal: false, + }), + } + } + + /// Stage a fully authenticated connection for confirmation. The returned + /// ID identifies an older unconfirmed candidate that may be closed. The + /// current active connection remains authoritative until [`Self::confirm`] + /// promotes this candidate. + pub fn attach( + &self, + principal: &SandboxProtocolPrincipal, + supervisor_instance_id: crate::boundary_protocol::SupervisorInstanceId, + ) -> Result, SandboxAuthError> { + let epoch = principal.session.auth_epoch; + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if state.terminal { + return Err(SandboxAuthError::TerminalSession); + } + match state.supervisor_instance_id { + Some(expected) if expected != supervisor_instance_id => { + return Err(SandboxAuthError::WrongSupervisorInstance); + } + Some(_) => {} + None => state.supervisor_instance_id = Some(supervisor_instance_id), + } + if let Some(active) = state.active { + if active.id == principal.connection_id && active.epoch == epoch { + return Ok(None); + } + if epoch <= active.epoch { + return Err(SandboxAuthError::StaleCredentialEpoch); + } + } + if state.highest_epoch.is_some_and(|highest| epoch < highest) { + return Err(SandboxAuthError::StaleCredentialEpoch); + } + if let Some(pending) = state.pending { + if pending.id == principal.connection_id && pending.epoch == epoch { + return Ok(None); + } + if epoch < pending.epoch { + return Err(SandboxAuthError::StaleCredentialEpoch); + } + } + let replaced = state.pending.map(|pending| pending.id); + state.highest_epoch = Some(epoch); + state.pending = Some(ActiveConnection { + id: principal.connection_id, + epoch, + }); + Ok(replaced) + } + + /// Promote an attached, confirmed candidate to the active connection. The + /// returned ID is the previously active connection, which may now be + /// closed without creating an unsupervised interval. + pub fn confirm( + &self, + principal: &SandboxProtocolPrincipal, + ) -> Result, SandboxAuthError> { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if state.terminal { + return Err(SandboxAuthError::TerminalSession); + } + if state + .active + .is_some_and(|active| active.id == principal.connection_id) + { + return Ok(None); + } + let pending = state + .pending + .filter(|pending| pending.id == principal.connection_id) + .ok_or(SandboxAuthError::ConnectionNotAttached)?; + let replaced = state.active.map(|active| active.id); + state.active = Some(pending); + state.pending = None; + Ok(replaced) + } + + /// Confirm may run on either the active connection (an idempotent replay) + /// or its staged replacement. Other operations require the active one. + pub fn require_attached( + &self, + principal: &SandboxProtocolPrincipal, + ) -> Result<(), SandboxAuthError> { + let state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if state.terminal { + return Err(SandboxAuthError::TerminalSession); + } + if state + .active + .is_some_and(|active| active.id == principal.connection_id) + || state + .pending + .is_some_and(|pending| pending.id == principal.connection_id) + { + Ok(()) + } else { + Err(SandboxAuthError::ConnectionNotAttached) + } + } + + pub fn require_active( + &self, + principal: &SandboxProtocolPrincipal, + ) -> Result<(), SandboxAuthError> { + let state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if state.terminal { + return Err(SandboxAuthError::TerminalSession); + } + if state + .active + .is_none_or(|active| active.id != principal.connection_id) + { + return Err(SandboxAuthError::ConnectionNotAttached); + } + Ok(()) + } + + /// Remove a physical connection. Returns `true` only when it was the + /// confirmed active connection and recovery must begin. + pub fn disconnect(&self, connection_id: SandboxConnectionId) -> bool { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let was_active = state + .active + .is_some_and(|active| active.id == connection_id); + if was_active { + state.active = None; + } + if state + .pending + .is_some_and(|pending| pending.id == connection_id) + { + state.pending = None; + } + was_active + } + + pub fn mark_terminal(&self) { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + state.terminal = true; + state.active = None; + state.pending = None; + } +} + +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum SandboxAuthError { + #[error("authorization metadata is missing")] + MissingBearer, + #[error("authorization metadata must occur exactly once")] + DuplicateBearer, + #[error("authorization metadata is not a valid bearer credential")] + InvalidBearer, + #[error("sandbox JWT validation failed: {0}")] + Jwt(#[from] SessionJwtError), + #[error("authenticated sandbox identity does not match this runtime")] + WrongSandbox, + #[error("authenticated runtime generation does not match this sandbox runtime")] + WrongRuntimeGeneration, + #[error("Sandbox Protocol credential epoch is stale or already active")] + StaleCredentialEpoch, + #[error("sandbox runtime is already bound to another supervisor process")] + WrongSupervisorInstance, + #[error("Sandbox Protocol connection has not completed attach")] + ConnectionNotAttached, + #[error("sandbox session is terminal")] + TerminalSession, +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use openshell_core::jwt::{ + DEFAULT_SESSION_TOKEN_TTL, JwtClock, SandboxRuntimeIdentity, SessionJwtIssuer, + SessionTokenProfile, SessionVerificationKey, + }; + use openshell_core::sandbox_generation::SandboxGenerationId; + use rcgen::{KeyPair, PKCS_ED25519}; + + use super::*; + + #[derive(Debug)] + struct FixedClock; + + impl JwtClock for FixedClock { + fn now_unix_seconds(&self) -> i64 { + 1_900_000_000 + } + } + + fn fixture( + epoch: u64, + ) -> ( + SandboxProtocolAuthenticator, + openshell_core::jwt::MintedSessionToken, + ) { + let key = KeyPair::generate_for(&PKCS_ED25519).expect("generate key"); + let public_key_pem = key.public_key_pem().into_bytes(); + let clock: Arc = Arc::new(FixedClock); + let sandbox_id = SandboxId::parse("sandbox-a").expect("sandbox ID"); + let issuer = SessionJwtIssuer::from_ed25519_pem( + key.serialize_pem().as_bytes(), + "current", + "test", + DEFAULT_SESSION_TOKEN_TTL, + clock.clone(), + ) + .expect("issuer"); + let verifier = SessionJwtVerifier::new( + "test", + SessionTokenProfile::Sandbox, + [SessionVerificationKey { + key_id: "current".to_string(), + public_key_pem, + }], + clock, + ) + .expect("verifier"); + let token = issuer + .mint_pair(&SandboxRuntimeIdentity { + sandbox_id: sandbox_id.clone(), + runtime_generation: SandboxGenerationId::parse("generation-1") + .expect("runtime generation"), + auth_epoch: CredentialEpoch::new(epoch).expect("epoch"), + }) + .expect("token pair") + .sandbox; + ( + SandboxProtocolAuthenticator::new( + verifier, + sandbox_id, + SandboxGenerationId::parse("generation-1").expect("runtime generation"), + CredentialEpoch::new(epoch).expect("epoch"), + ), + token, + ) + } + + fn registry_for(_principal: &SandboxProtocolPrincipal) -> SandboxConnectionRegistry { + SandboxConnectionRegistry::new( + openshell_core::SandboxSessionId::new(), + openshell_core::jwt::SessionRotation::new(1).expect("rotation"), + ) + } + + fn metadata(token: &str) -> MetadataMap { + let mut metadata = MetadataMap::new(); + metadata.insert( + "authorization", + format!("Bearer {token}").parse().expect("metadata value"), + ); + metadata + } + + #[test] + fn bearer_metadata_must_occur_exactly_once() { + let (authenticator, token) = fixture(1); + assert_eq!( + authenticator.authenticate(SandboxConnectionId::new(), &MetadataMap::new()), + Err(SandboxAuthError::MissingBearer) + ); + let mut duplicate = metadata(token.token.expose_secret()); + duplicate.append( + "authorization", + format!("Bearer {}", token.token.expose_secret()) + .parse() + .expect("metadata value"), + ); + assert_eq!( + authenticator.authenticate(SandboxConnectionId::new(), &duplicate), + Err(SandboxAuthError::DuplicateBearer) + ); + } + + #[test] + fn reconnect_requires_disconnect_and_terminal_is_final() { + let (first_authenticator, first_token) = fixture(1); + let first_id = SandboxConnectionId::new(); + let first = first_authenticator + .authenticate(first_id, &metadata(first_token.token.expose_secret())) + .expect("first principal"); + let registry = registry_for(&first); + let instance = crate::boundary_protocol::SupervisorInstanceId::new(); + assert_eq!(registry.attach(&first, instance), Ok(None)); + assert_eq!(registry.attach(&first, instance), Ok(None)); + assert_eq!(registry.confirm(&first), Ok(None)); + assert_eq!(registry.confirm(&first), Ok(None)); + + assert!(registry.disconnect(first_id)); + assert_eq!(registry.attach(&first, instance), Ok(None)); + assert_eq!(registry.confirm(&first), Ok(None)); + registry.mark_terminal(); + assert_eq!( + registry.require_active(&first), + Err(SandboxAuthError::TerminalSession) + ); + assert_eq!( + registry.attach(&first, instance), + Err(SandboxAuthError::TerminalSession) + ); + } + + #[test] + fn replacement_supervisor_cannot_resume_existing_runtime_generation() { + let (authenticator, token) = fixture(1); + let first_id = SandboxConnectionId::new(); + let first = authenticator + .authenticate(first_id, &metadata(token.token.expose_secret())) + .expect("first principal"); + let registry = registry_for(&first); + let first_instance = crate::boundary_protocol::SupervisorInstanceId::new(); + registry + .attach(&first, first_instance) + .expect("attach first supervisor"); + registry.confirm(&first).expect("confirm first supervisor"); + assert!(registry.disconnect(first_id)); + + let replacement_id = SandboxConnectionId::new(); + let replacement = authenticator + .authenticate(replacement_id, &metadata(token.token.expose_secret())) + .expect("replacement principal"); + assert_eq!( + registry.attach( + &replacement, + crate::boundary_protocol::SupervisorInstanceId::new(), + ), + Err(SandboxAuthError::WrongSupervisorInstance) + ); + assert_eq!(registry.attach(&replacement, first_instance), Ok(None)); + } + + #[test] + fn replacement_does_not_displace_active_connection_before_confirm() { + let (first_authenticator, first_token) = fixture(1); + let first_id = SandboxConnectionId::new(); + let first = first_authenticator + .authenticate(first_id, &metadata(first_token.token.expose_secret())) + .expect("first principal"); + let (replacement_authenticator, replacement_token) = fixture(2); + let replacement_id = SandboxConnectionId::new(); + let replacement = replacement_authenticator + .authenticate( + replacement_id, + &metadata(replacement_token.token.expose_secret()), + ) + .expect("replacement principal"); + let registry = registry_for(&first); + let instance = crate::boundary_protocol::SupervisorInstanceId::new(); + registry.attach(&first, instance).expect("attach first"); + registry.confirm(&first).expect("confirm first"); + + assert_eq!(registry.attach(&replacement, instance), Ok(None)); + registry + .require_active(&first) + .expect("first remains active"); + assert_eq!( + registry.require_active(&replacement), + Err(SandboxAuthError::ConnectionNotAttached) + ); + registry + .require_attached(&replacement) + .expect("replacement may confirm"); + assert_eq!(registry.confirm(&replacement), Ok(Some(first_id))); + assert_eq!( + registry.require_active(&first), + Err(SandboxAuthError::ConnectionNotAttached) + ); + registry + .require_active(&replacement) + .expect("replacement became active"); + } +} diff --git a/crates/openshell-sandbox/Cargo.toml b/crates/openshell-sandbox/Cargo.toml index 3463f03767..1283dd063b 100644 --- a/crates/openshell-sandbox/Cargo.toml +++ b/crates/openshell-sandbox/Cargo.toml @@ -14,22 +14,34 @@ repository.workspace = true name = "openshell-sandbox" path = "src/main.rs" +[[bin]] +name = "openshell-seccomp-perf" +path = "src/bin/seccomp-perf.rs" +required-features = ["perf-harness"] + +[features] +perf-harness = [] + [dependencies] openshell-core = { path = "../openshell-core", default-features = false } -openshell-extension-core = { path = "../openshell-extension-core" } +openshell-binary-identity = { path = "../openshell-binary-identity" } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } openshell-ocsf = { path = "../openshell-ocsf" } openshell-policy = { path = "../openshell-policy" } -openshell-supervisor-network = { path = "../openshell-supervisor-network", default-features = false } -openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } -openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" } -openshell-supervisor-process = { path = "../openshell-supervisor-process" } + +anyhow = { workspace = true } +async-trait = "0.1" +bytes = { workspace = true } +hex = "0.4" +ipnet = "2" +rand = "0.10" +sha2 = { workspace = true } # Async runtime tokio = { workspace = true } - -# gRPC (tonic::Status downcast in error mapping) -tonic = { workspace = true, features = ["channel", "tls-native-roots"] } -prost-types = { workspace = true } +tokio-stream = { workspace = true } +tonic = { workspace = true } # CLI clap = { workspace = true } @@ -37,49 +49,37 @@ clap = { workspace = true } # Error handling miette = { workspace = true } -# Unix ownership for Kubernetes sidecar init setup +# Unix identity and bootstrap ownership nix = { workspace = true } # TLS crypto provider install (main.rs) rustls = { workspace = true } +rustls-pemfile = { workspace = true } +tokio-rustls = { workspace = true } # Serialization (serde_json::json! for OCSF unmapped fields) +base64 = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } -prost = { workspace = true } # Logging tracing = { workspace = true } -uuid = { workspace = true } tracing-subscriber = { workspace = true } -tracing-appender = { workspace = true } -[features] -default = ["telemetry", "bundled-ca-roots"] -## Convenience alias: all defaults except bundled CA roots. Use -## `--no-default-features --features system-ca-roots` to build a supervisor -## that uses the platform trust store with telemetry intact. -system-ca-roots = ["telemetry"] -## Convenience alias: every default feature except `telemetry`. Build a -## telemetry-free supervisor with -## `--no-default-features --features defaults-without-telemetry` and stay -## correct as new default features are added. Cargo cannot subtract a single -## default feature, so this alias must be paired with `--no-default-features`; -## enabling it alongside `telemetry` is a compile error rather than a silent -## telemetry-on build. Kept in sync with `default` by -## `rust:verify:defaults-without-telemetry`. Do not pair it with -## `system-ca-roots`, which re-enables `telemetry`; a build with neither -## telemetry nor bundled CA roots is plain `--no-default-features`. -defaults-without-telemetry = ["bundled-ca-roots"] - -telemetry = ["openshell-core/telemetry"] -bundled-ca-roots = ["openshell-supervisor-network/bundled-ca-roots"] +[target.'cfg(unix)'.dependencies] +libc = "0.2" +rustix = { workspace = true } + +[target.'cfg(target_os = "linux")'.dependencies] +capctl = "0.2.4" +landlock = "0.4" +seccompiler = "0.5" +socket2 = { workspace = true } +tempfile = "3" [dev-dependencies] +rcgen = { workspace = true } tempfile = "3" -temp-env = "0.3" -tokio-tungstenite = { workspace = true } -futures = { workspace = true } [lints] workspace = true diff --git a/crates/openshell-sandbox/src/accept_interrupt.rs b/crates/openshell-sandbox/src/accept_interrupt.rs new file mode 100644 index 0000000000..86bbf39218 --- /dev/null +++ b/crates/openshell-sandbox/src/accept_interrupt.rs @@ -0,0 +1,297 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Cancellation for broker-owned blocking accepts without changing workload OFDs. +//! +//! SIGUSR2 is reserved by the sandbox binary. Its process-global disposition is +//! necessarily kernel state, not a global application context. All registration, +//! cancellation and thread ownership state belongs to one broker instance. + +#![allow(unsafe_code)] + +use std::collections::HashMap; +use std::io; +use std::marker::PhantomData; +use std::rc::Rc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Condvar, Mutex}; +use std::time::Duration; + +const INTERRUPT_SIGNAL: libc::c_int = libc::SIGUSR2; +const INTERRUPT_INTERVAL: Duration = Duration::from_millis(10); + +extern "C" fn interrupt_accept(_: libc::c_int) {} + +fn reserve_signal() -> io::Result<()> { + // SAFETY: both actions are initialized storage. The no-op handler is + // async-signal-safe and deliberately omits SA_RESTART so accept returns EINTR. + unsafe { + let mut previous: libc::sigaction = std::mem::zeroed(); + if libc::sigaction(INTERRUPT_SIGNAL, std::ptr::null(), &raw mut previous) < 0 { + return Err(io::Error::last_os_error()); + } + if previous.sa_sigaction != libc::SIG_DFL + && previous.sa_sigaction != interrupt_accept as *const () as usize + { + return Err(io::Error::other( + "sandbox SIGUSR2 is already reserved by another handler", + )); + } + let mut action: libc::sigaction = std::mem::zeroed(); + action.sa_sigaction = interrupt_accept as *const () as usize; + libc::sigemptyset(&raw mut action.sa_mask); + if libc::sigaction(INTERRUPT_SIGNAL, &raw const action, std::ptr::null_mut()) < 0 { + return Err(io::Error::last_os_error()); + } + } + Ok(()) +} + +#[derive(Default)] +struct State { + workers: Mutex>, + changed: Condvar, + stopped: AtomicBool, +} + +// musl represents pthread_t as an opaque pointer, unlike glibc's integer. It +// is only passed back to pthread_kill, never dereferenced by this module. +struct RegisteredThread(libc::pthread_t); + +// SAFETY: POSIX permits signaling a live pthread from another thread. The +// handle is accessed only under State::workers, and the owning worker removes +// its registration under that same mutex before returning. AcceptRegistration +// cannot move to another thread, so its Drop cannot outlive the owning worker. +unsafe impl Send for RegisteredThread {} + +pub struct AcceptMonitor { + state: Arc, + thread: Option>, +} + +impl AcceptMonitor { + pub(crate) fn start(valid: impl Fn(u64) -> bool + Send + 'static) -> io::Result { + reserve_signal()?; + let state = Arc::new(State::default()); + let worker_state = state.clone(); + let thread = std::thread::Builder::new() + .name("openshell-accept-cancellation".into()) + .spawn(move || monitor(&worker_state, valid))?; + Ok(Self { + state, + thread: Some(thread), + }) + } + + pub(crate) fn registrar(&self) -> AcceptRegistrar { + AcceptRegistrar(self.state.clone()) + } +} + +impl Drop for AcceptMonitor { + fn drop(&mut self) { + let workers = lock(&self.state.workers); + self.state.stopped.store(true, Ordering::Release); + self.state.changed.notify_all(); + drop(workers); + // The monitor keeps interrupting registered workers during shutdown. + // Registrations are removed before their threads can exit/reuse IDs. + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + } +} + +#[derive(Clone)] +pub struct AcceptRegistrar(Arc); + +impl AcceptRegistrar { + pub(crate) fn register(&self, notification_id: u64) -> io::Result { + // SAFETY: this changes only the current broker worker's signal mask. + // Workload launchers do not inherit this mask; exec resets the handler. + let thread = unsafe { + let mut mask: libc::sigset_t = std::mem::zeroed(); + libc::sigemptyset(&raw mut mask); + libc::sigaddset(&raw mut mask, INTERRUPT_SIGNAL); + let error = + libc::pthread_sigmask(libc::SIG_UNBLOCK, &raw const mask, std::ptr::null_mut()); + if error != 0 { + return Err(io::Error::from_raw_os_error(error)); + } + libc::pthread_self() + }; + let mut workers = lock(&self.0.workers); + if self.0.stopped.load(Ordering::Acquire) { + return Err(io::Error::from_raw_os_error(libc::ECANCELED)); + } + if workers.contains_key(¬ification_id) { + return Err(io::Error::other( + "duplicate accept notification registration", + )); + } + workers.insert(notification_id, RegisteredThread(thread)); + self.0.changed.notify_one(); + Ok(AcceptRegistration { + state: self.0.clone(), + notification_id, + owning_thread: PhantomData, + }) + } +} + +pub struct AcceptRegistration { + state: Arc, + notification_id: u64, + // Drop must run on the registering thread before its pthread_t can expire. + // No Rc is allocated; this marker makes the guard neither Send nor Sync. + owning_thread: PhantomData>, +} + +impl AcceptRegistration { + pub(crate) fn ensure_running(&self) -> io::Result<()> { + if self.state.stopped.load(Ordering::Acquire) { + Err(io::Error::from_raw_os_error(libc::ECANCELED)) + } else { + Ok(()) + } + } +} + +impl Drop for AcceptRegistration { + fn drop(&mut self) { + lock(&self.state.workers).remove(&self.notification_id); + self.state.changed.notify_one(); + } +} + +fn monitor(state: &State, valid: impl Fn(u64) -> bool) { + let mut workers = lock(&state.workers); + loop { + let stopped = state.stopped.load(Ordering::Acquire); + if stopped && workers.is_empty() { + return; + } + for (¬ification_id, thread) in &*workers { + if stopped || !valid(notification_id) { + // SAFETY: the registration lock pins this live pthread_t. + // Repeated interrupts close the check-to-accept race: a signal + // received before accept cannot leave a later accept stranded. + let _ = unsafe { libc::pthread_kill(thread.0, INTERRUPT_SIGNAL) }; + } + } + workers = if workers.is_empty() { + state + .changed + .wait(workers) + .unwrap_or_else(std::sync::PoisonError::into_inner) + } else { + state + .changed + .wait_timeout(workers, INTERRUPT_INTERVAL) + .unwrap_or_else(std::sync::PoisonError::into_inner) + .0 + }; + } +} + +fn lock(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { + mutex + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::net::{TcpListener, TcpStream}; + use std::os::fd::AsRawFd; + + #[test] + fn registrar_crosses_threads_but_registration_ends_before_worker_exit() { + fn assert_send_sync() {} + assert_send_sync::(); + + let monitor = AcceptMonitor::start(|_| true).unwrap(); + let registrar = monitor.registrar(); + std::thread::spawn(move || { + let registration = registrar.register(3).unwrap(); + assert!(registrar.register(3).is_err()); + assert!(lock(®istrar.0.workers).contains_key(&3)); + drop(registration); + assert!(lock(®istrar.0.workers).is_empty()); + }) + .join() + .unwrap(); + assert!(lock(&monitor.state.workers).is_empty()); + } + + #[test] + fn cancellation_interrupts_competing_accept_after_readiness_was_consumed() { + let valid = Arc::new(AtomicBool::new(true)); + let monitored = valid.clone(); + let monitor = AcceptMonitor::start(move |_| monitored.load(Ordering::Acquire)).unwrap(); + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let client = TcpStream::connect(listener.local_addr().unwrap()).unwrap(); + // Both contenders could observe this same readable listener. Consume + // its only connection before the second contender actually accepts. + let accepted = listener.accept().unwrap(); + let registrar = monitor.registrar(); + let (ready_tx, ready_rx) = std::sync::mpsc::channel(); + let (done_tx, done_rx) = std::sync::mpsc::channel(); + let worker = std::thread::spawn(move || { + let registration = registrar.register(1).unwrap(); + ready_tx.send(()).unwrap(); + // SAFETY: the listener is live and null address outputs are valid. + // Use the syscall directly: std::net retries EINTR internally. + let result = unsafe { + libc::accept4( + listener.as_raw_fd(), + std::ptr::null_mut(), + std::ptr::null_mut(), + libc::SOCK_CLOEXEC, + ) + }; + assert_eq!(result, -1); + let error = io::Error::last_os_error(); + assert_eq!(error.kind(), io::ErrorKind::Interrupted); + drop(registration); + done_tx.send(()).unwrap(); + }); + ready_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + valid.store(false, Ordering::Release); + done_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + worker.join().unwrap(); + drop((accepted, client, monitor)); + } + + #[test] + fn shutdown_interrupts_registered_accepts_and_reclaims_the_monitor() { + let monitor = AcceptMonitor::start(|_| true).unwrap(); + let registrar = monitor.registrar(); + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let (ready_tx, ready_rx) = std::sync::mpsc::channel(); + let (done_tx, done_rx) = std::sync::mpsc::channel(); + let worker = std::thread::spawn(move || { + let registration = registrar.register(2).unwrap(); + ready_tx.send(()).unwrap(); + // SAFETY: owned listener and optional null address outputs. + let result = unsafe { + libc::accept4( + listener.as_raw_fd(), + std::ptr::null_mut(), + std::ptr::null_mut(), + libc::SOCK_CLOEXEC, + ) + }; + assert_eq!(result, -1); + assert!(registration.ensure_running().is_err()); + drop(registration); + done_tx.send(()).unwrap(); + }); + ready_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + let shutdown = std::thread::spawn(move || drop(monitor)); + done_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + worker.join().unwrap(); + shutdown.join().unwrap(); + } +} diff --git a/crates/openshell-sandbox/src/bin/seccomp-perf.rs b/crates/openshell-sandbox/src/bin/seccomp-perf.rs new file mode 100644 index 0000000000..4114d9a7a9 --- /dev/null +++ b/crates/openshell-sandbox/src/bin/seccomp-perf.rs @@ -0,0 +1,113 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Microbenchmark entry point for the production seccomp network broker. + +#[cfg(target_os = "linux")] +use std::net::SocketAddr; + +#[cfg(target_os = "linux")] +use clap::builder::{PossibleValue, PossibleValuesParser}; +#[cfg(target_os = "linux")] +use clap::{Parser, Subcommand, ValueEnum as _}; +#[cfg(target_os = "linux")] +use openshell_sandbox::perf::{BenchmarkOptions, Layer, Protocol}; + +#[cfg(target_os = "linux")] +#[derive(Debug, Parser)] +#[command( + about = "Measure native and seccomp-filtered socket performance", + long_about = "Measure native and seccomp-filtered TCP socket performance." +)] +struct Cli { + /// Benchmark layer: native, filtered, or all. + #[arg(long, default_value = "all", value_parser = ["native", "filtered", "all"])] + layer: String, + /// Implemented protocol to benchmark, or all. + #[arg(long, default_value = "all", value_parser = protocol_value_parser())] + protocol: String, + #[arg(long, default_value_t = 10_000)] + iterations: u64, + #[arg(long, default_value_t = 1_000)] + warmup: u64, + #[arg(long, default_value_t = 1)] + concurrency: usize, + #[arg(long, default_value_t = 64)] + payload_bytes: usize, + #[command(subcommand)] + command: Option, +} + +#[cfg(target_os = "linux")] +fn protocol_value_parser() -> PossibleValuesParser { + let mut values = vec![PossibleValue::new("all")]; + values.extend( + Protocol::value_variants() + .iter() + .filter_map(clap::ValueEnum::to_possible_value), + ); + PossibleValuesParser::new(values) +} + +#[cfg(target_os = "linux")] +#[derive(Debug, Subcommand)] +enum Command { + #[command(hide = true)] + Worker { + #[arg(long)] + protocol: Protocol, + #[arg(long)] + target: SocketAddr, + #[arg(long)] + iterations: u64, + #[arg(long)] + warmup: u64, + #[arg(long)] + concurrency: usize, + #[arg(long)] + payload_bytes: usize, + }, +} + +#[cfg(target_os = "linux")] +fn main() -> anyhow::Result<()> { + let cli = Cli::parse(); + if let Some(Command::Worker { + protocol, + target, + iterations, + warmup, + concurrency, + payload_bytes, + }) = cli.command + { + let report = openshell_sandbox::perf::run_worker( + protocol, + target, + iterations, + warmup, + concurrency, + payload_bytes, + )?; + println!("{}", serde_json::to_string(&report)?); + return Ok(()); + } + + let options = BenchmarkOptions { + layers: Layer::selection(&cli.layer)?, + protocols: Protocol::selection(&cli.protocol)?, + iterations: cli.iterations, + warmup: cli.warmup, + concurrency: cli.concurrency, + payload_bytes: cli.payload_bytes, + }; + for report in openshell_sandbox::perf::run(options)? { + println!("{}", serde_json::to_string(&report)?); + } + Ok(()) +} + +#[cfg(not(target_os = "linux"))] +fn main() -> anyhow::Result<()> { + anyhow::bail!("the seccomp performance harness requires Linux") +} diff --git a/crates/openshell-sandbox/src/boundary_exec.rs b/crates/openshell-sandbox/src/boundary_exec.rs new file mode 100644 index 0000000000..57d9778a6f --- /dev/null +++ b/crates/openshell-sandbox/src/boundary_exec.rs @@ -0,0 +1,725 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Workload-side implementation of RFC 0012 sandbox exec. + +use std::collections::HashMap; +use std::os::fd::{AsRawFd, OwnedFd}; +use std::process::{Child, Command, Stdio}; +use std::sync::Arc; + +use async_trait::async_trait; +use nix::pty::{Winsize, openpty}; +use nix::sys::signal::{Signal, killpg}; +use nix::unistd::Pid; +use openshell_core::policy::SandboxPolicy; +use openshell_core::provider_credentials::ProviderCredentialState; +use openshell_isolation_interface::contract::{ + BackendError, BoundaryExec, BoundaryExitStatus, BoundaryInput, BoundaryOutput, BoundaryProcess, + BoundarySignal, BoundaryTerminal, ExecSession, ExecSpec, +}; + +/// The sandbox executor. Every spawn reuses the same admitted policy and +/// execution-environment controls while taking a fresh provider credential +/// snapshot. +#[derive(Clone)] +pub struct LocalBoundaryExec { + policy: SandboxPolicy, + base_workdir: Option, + ca_file_paths: Option>, + provider_credentials: ProviderCredentialState, + user_environment: HashMap, + runtime: Arc, + #[cfg(target_os = "linux")] + launcher: openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, +} + +impl LocalBoundaryExec { + /// Construct the executor owned by an active sandbox boundary. + #[must_use] + pub fn new( + policy: SandboxPolicy, + base_workdir: Option, + ca_file_paths: Option>, + provider_credentials: ProviderCredentialState, + user_environment: HashMap, + runtime: Arc, + #[cfg(target_os = "linux")] + launcher: openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, + ) -> Self { + Self { + policy, + base_workdir, + ca_file_paths, + provider_credentials, + user_environment, + runtime, + #[cfg(target_os = "linux")] + launcher, + } + } + + fn command(&self, spec: &ExecSpec) -> Result { + if spec.program.is_empty() { + return Err(BackendError::Process("exec program is empty".to_string())); + } + let mut command = Command::new(&spec.program); + command.args(&spec.args); + let effective_workdir = spec.workdir.as_deref().or(self.base_workdir.as_deref()); + let (session_user, session_home) = + crate::process::session_user_and_home(&self.policy, effective_workdir); + let path = std::env::var("PATH").unwrap_or_else(|_| "/usr/local/bin:/usr/bin:/bin".into()); + command + .env_clear() + .env(openshell_core::sandbox_env::SANDBOX, "1") + .env("HOME", session_home) + .env("USER", session_user) + .env("SHELL", "/bin/bash") + .env("PATH", path) + .env("TERM", if spec.pty { "xterm-256color" } else { "dumb" }); + for (key, value) in &self.user_environment { + if !key.starts_with("OPENSHELL_") { + command.env(key, value); + } + } + if let Some((ca_cert_path, combined_bundle_path)) = self.ca_file_paths.as_deref() { + for (key, value) in crate::child_env::tls_env_vars(ca_cert_path, combined_bundle_path) { + command.env(key, value); + } + } + for (key, value) in self.provider_credentials.child_env_with_gcp_resolved() { + if !crate::process::is_supervisor_only_env_var(&key) { + command.env(key, value); + } + } + crate::process::strip_proxy_env_std(&mut command); + for (key, value) in &spec.env { + if !key.starts_with("OPENSHELL_") { + command.env(key, value); + } + } + if let Some(workdir) = spec.workdir.as_deref().or(self.base_workdir.as_deref()) { + command.current_dir(workdir); + } + Ok(command) + } + + #[cfg(target_os = "linux")] + fn prepare_sandbox( + &self, + workdir: Option<&str>, + ) -> Result, BackendError> { + crate::sandbox::linux::log_sandbox_readiness(&self.policy, workdir); + let runtime_read_only = + crate::process::ca_runtime_read_only_paths(self.ca_file_paths.as_deref()); + crate::process::prepare_child_sandbox(&self.policy, workdir, &runtime_read_only) + .map_err(|error| BackendError::Process(error.to_string())) + } + + fn spawn_piped(&self, spec: &ExecSpec) -> Result { + self.runtime.ensure_active()?; + let mut command = self.command(spec)?; + command + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let effective_workdir = spec.workdir.as_deref().or(self.base_workdir.as_deref()); + #[cfg(target_os = "linux")] + let prepared = self.prepare_sandbox(effective_workdir)?; + #[cfg(target_os = "linux")] + let child_hardening = + openshell_isolation_interface::linux::child_seccomp::prepare(std::process::id()) + .map_err(|error| BackendError::Process(error.to_string()))?; + crate::pty::install_dedicated_process_group(&mut command); + crate::pty::install_pre_exec_no_pty( + &mut command, + self.policy.clone(), + effective_workdir.map(str::to_string), + #[cfg(target_os = "linux")] + prepared, + #[cfg(target_os = "linux")] + child_hardening, + ) + .map_err(|error| BackendError::Process(error.to_string()))?; + #[cfg(target_os = "linux")] + let mut child_registry = crate::managed_children::lock(); + #[cfg(target_os = "linux")] + let mut child = + crate::process::spawn_std_command_with_workload_launcher(&self.launcher, command) + .map_err(|error| BackendError::Process(error.to_string()))?; + #[cfg(not(target_os = "linux"))] + let mut child = command + .spawn() + .map_err(|error| BackendError::Process(error.to_string()))?; + let pid = child.id(); + let process_terminal = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let signal_lock = Arc::new(std::sync::Mutex::new(())); + if let Err(error) = + self.runtime + .register_process_group(pid, process_terminal.clone(), signal_lock.clone()) + { + let _ = killpg( + Pid::from_raw(i32::try_from(pid).unwrap_or(i32::MAX)), + Signal::SIGKILL, + ); + let _ = child.wait(); + return Err(error); + } + #[cfg(target_os = "linux")] + let managed_child = child_registry.register(pid); + #[cfg(target_os = "linux")] + drop(child_registry); + let stdin = child.stdin.take().map(|file| -> BoundaryInput { + let fd: OwnedFd = file.into(); + Box::new(tokio::fs::File::from_std(std::fs::File::from(fd))) + }); + let stdout = child + .stdout + .take() + .map(|file| -> BoundaryOutput { + let fd: OwnedFd = file.into(); + Box::new(tokio::fs::File::from_std(std::fs::File::from(fd))) + }) + .ok_or_else(|| BackendError::Process("exec stdout pipe missing".to_string()))?; + let stderr = child.stderr.take().map(|file| -> BoundaryOutput { + let fd: OwnedFd = file.into(); + Box::new(tokio::fs::File::from_std(std::fs::File::from(fd))) + }); + let process = Arc::new(LocalExecProcess::new( + child, + pid, + self.runtime.clone(), + process_terminal, + signal_lock, + #[cfg(target_os = "linux")] + managed_child, + )); + Ok(SpawnedExec { + session: Some(ExecSession { + process: process.clone(), + stdin, + stdout, + stderr, + terminal: None, + }), + process, + armed: true, + }) + } + + fn spawn_pty(&self, spec: &ExecSpec) -> Result { + self.runtime.ensure_active()?; + let winsize = Winsize { + ws_row: 24, + ws_col: 80, + ws_xpixel: 0, + ws_ypixel: 0, + }; + let pty = openpty(Some(&winsize), None) + .map_err(|error| BackendError::Process(error.to_string()))?; + let master = std::fs::File::from(pty.master); + let slave = std::fs::File::from(pty.slave); + let slave_fd = slave.as_raw_fd(); + let input = master + .try_clone() + .map_err(|error| BackendError::Process(error.to_string()))?; + let output = master + .try_clone() + .map_err(|error| BackendError::Process(error.to_string()))?; + let stdin = slave + .try_clone() + .map_err(|error| BackendError::Process(error.to_string()))?; + let stdout = slave + .try_clone() + .map_err(|error| BackendError::Process(error.to_string()))?; + let mut command = self.command(spec)?; + command.stdin(stdin).stdout(stdout).stderr(slave); + let effective_workdir = spec.workdir.as_deref().or(self.base_workdir.as_deref()); + #[cfg(target_os = "linux")] + let prepared = self.prepare_sandbox(effective_workdir)?; + #[cfg(target_os = "linux")] + let child_hardening = + openshell_isolation_interface::linux::child_seccomp::prepare(std::process::id()) + .map_err(|error| BackendError::Process(error.to_string()))?; + crate::pty::install_pre_exec( + &mut command, + self.policy.clone(), + effective_workdir.map(str::to_string), + slave_fd, + #[cfg(target_os = "linux")] + prepared, + #[cfg(target_os = "linux")] + child_hardening, + ) + .map_err(|error| BackendError::Process(error.to_string()))?; + #[cfg(target_os = "linux")] + let mut child_registry = crate::managed_children::lock(); + #[cfg(target_os = "linux")] + let mut child = + crate::process::spawn_std_command_with_workload_launcher(&self.launcher, command) + .map_err(|error| BackendError::Process(error.to_string()))?; + #[cfg(not(target_os = "linux"))] + let mut child = command + .spawn() + .map_err(|error| BackendError::Process(error.to_string()))?; + let pid = child.id(); + let process_terminal = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let signal_lock = Arc::new(std::sync::Mutex::new(())); + if let Err(error) = + self.runtime + .register_process_group(pid, process_terminal.clone(), signal_lock.clone()) + { + let _ = killpg( + Pid::from_raw(i32::try_from(pid).unwrap_or(i32::MAX)), + Signal::SIGKILL, + ); + let _ = child.wait(); + return Err(error); + } + #[cfg(target_os = "linux")] + let managed_child = child_registry.register(pid); + #[cfg(target_os = "linux")] + drop(child_registry); + let terminal: Arc = Arc::new(LocalTerminal { master }); + let process = Arc::new(LocalExecProcess::new( + child, + pid, + self.runtime.clone(), + process_terminal, + signal_lock, + #[cfg(target_os = "linux")] + managed_child, + )); + Ok(SpawnedExec { + session: Some(ExecSession { + process: process.clone(), + stdin: Some(Box::new(tokio::fs::File::from_std(input))), + stdout: Box::new(tokio::fs::File::from_std(output)), + stderr: None, + terminal: Some(terminal), + }), + process, + armed: true, + }) + } +} + +struct SpawnedExec { + session: Option, + process: Arc, + armed: bool, +} + +impl SpawnedExec { + fn into_session(mut self) -> ExecSession { + self.armed = false; + self.session.take().expect("spawned exec session") + } +} + +impl Drop for SpawnedExec { + fn drop(&mut self) { + if self.armed { + let _ = self.process.deliver(Signal::SIGKILL); + } + } +} + +#[async_trait] +impl BoundaryExec for LocalBoundaryExec { + async fn exec(&self, spec: ExecSpec) -> Result { + let executor = self.clone(); + let (send, receive) = tokio::sync::oneshot::channel(); + tokio::task::spawn_blocking(move || { + let result = if spec.pty { + executor.spawn_pty(&spec) + } else { + executor.spawn_piped(&spec) + }; + // If the caller cancelled, either send fails and drops the armed + // process guard here, or the queued guard is dropped with the + // receiver. Both paths terminate an unobservable exec process. + let _ = send.send(result); + }); + receive + .await + .map_err(|_| BackendError::Process("exec spawn task failed".to_string()))? + .map(SpawnedExec::into_session) + } +} + +struct LocalTerminal { + master: std::fs::File, +} + +#[async_trait] +impl BoundaryTerminal for LocalTerminal { + async fn resize(&self, cols: u16, rows: u16) -> Result<(), BackendError> { + crate::pty::set_winsize( + self.master.as_raw_fd(), + Winsize { + ws_row: rows.max(1), + ws_col: cols.max(1), + ws_xpixel: 0, + ws_ypixel: 0, + }, + ) + .map_err(|error| BackendError::Process(error.to_string())) + } +} + +struct LocalExecProcess { + pid: u32, + result: Arc>>>, + exited: Arc, + runtime: Arc, + terminal: Arc, + signal_lock: Arc>, +} + +impl LocalExecProcess { + fn new( + child: Child, + pid: u32, + runtime: Arc, + terminal: Arc, + signal_lock: Arc>, + #[cfg(target_os = "linux")] managed_child: Option, + ) -> Self { + let result = Arc::new(std::sync::Mutex::new(None)); + let exited = Arc::new(tokio::sync::Notify::new()); + let result_for_wait = result.clone(); + let exited_for_wait = exited.clone(); + let runtime_for_wait = runtime.clone(); + let terminal_for_wait = terminal.clone(); + let registration_terminal = terminal.clone(); + #[cfg(target_os = "linux")] + let signal_lock_for_wait = signal_lock.clone(); + tokio::spawn(async move { + let waited = tokio::task::spawn_blocking(move || { + let mut child = child; + #[cfg(target_os = "linux")] + { + let terminal_observed = crate::managed_children::wait_until_terminal(pid); + let _signal_guard = signal_lock_for_wait + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let result = child.wait(); + terminal_for_wait.store(true, std::sync::atomic::Ordering::Release); + if let Some(managed_child) = managed_child { + crate::managed_children::unregister(managed_child); + } + match (terminal_observed, result) { + (_, Ok(status)) => Ok(status), + (Err(observe_error), Err(wait_error)) => Err(std::io::Error::other( + format!( + "observe exec terminal state: {observe_error}; reap exec: {wait_error}" + ), + )), + (Ok(()), Err(wait_error)) => Err(wait_error), + } + } + #[cfg(not(target_os = "linux"))] + { + let result = child.wait(); + terminal_for_wait.store(true, std::sync::atomic::Ordering::Release); + result + } + }) + .await + .map_err(|error| error.to_string()) + .and_then(|status| status.map_err(|error| error.to_string())) + .map(|status| { + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + if let Some(signal) = status.signal() { + return BoundaryExitStatus::Signaled(signal); + } + } + BoundaryExitStatus::Exited(status.code().unwrap_or(1)) + }); + runtime_for_wait.unregister_process_group(pid, ®istration_terminal); + if let Ok(mut slot) = result_for_wait.lock() { + *slot = Some(waited); + } + exited_for_wait.notify_waiters(); + }); + Self { + pid, + result, + exited, + runtime, + terminal, + signal_lock, + } + } + + fn deliver(&self, signal: Signal) -> Result<(), BackendError> { + self.runtime.ensure_active()?; + let _signal_guard = self + .signal_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if self.terminal.load(std::sync::atomic::Ordering::Acquire) { + return Err(BackendError::Terminated("process has exited".to_string())); + } + let pid = i32::try_from(self.pid).unwrap_or(i32::MAX); + killpg(Pid::from_raw(pid), signal).map_err(|error| BackendError::Process(error.to_string())) + } +} + +#[async_trait] +impl BoundaryProcess for LocalExecProcess { + async fn wait(&self) -> Result { + loop { + let notified = self.exited.notified(); + let result = self + .result + .lock() + .map_err(|_| BackendError::Process("exec result lock poisoned".to_string()))? + .clone(); + if let Some(result) = result { + return result.map_err(BackendError::Process); + } + notified.await; + } + } + + async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError> { + self.deliver(match signal { + BoundarySignal::Term => Signal::SIGTERM, + BoundarySignal::Kill => Signal::SIGKILL, + BoundarySignal::Int => Signal::SIGINT, + BoundarySignal::Hup => Signal::SIGHUP, + }) + } + + async fn terminate(&self) -> Result<(), BackendError> { + self.deliver(Signal::SIGKILL) + } +} + +#[cfg(all(test, target_os = "linux"))] +mod tests { + use super::*; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + fn executor() -> LocalBoundaryExec { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start test workload launcher"); + std::thread::spawn(move || { + while let Ok(notification) = listener.receive() { + let _ = listener.respond_errno(notification.id, libc::EPERM); + } + }); + LocalBoundaryExec::new( + SandboxPolicy { + version: 1, + filesystem: openshell_core::policy::FilesystemPolicy::default(), + network: openshell_core::policy::NetworkPolicy::default(), + landlock: openshell_core::policy::LandlockPolicy::default(), + process: openshell_core::policy::ProcessPolicy::default(), + }, + None, + None, + ProviderCredentialState::from_environment( + 0, + HashMap::new(), + HashMap::new(), + HashMap::new(), + ), + HashMap::new(), + crate::boundary_io::BoundaryRuntimeState::new(), + launcher, + ) + } + + #[tokio::test] + async fn non_pty_exec_preserves_stdin_stdout_and_stderr() { + let mut session = executor() + .exec(ExecSpec { + program: "/bin/sh".to_string(), + args: vec![ + "-c".to_string(), + "read line; printf 'out:%s' \"$line\"; printf 'err:%s' \"$line\" >&2" + .to_string(), + ], + env: vec![], + workdir: None, + pty: false, + }) + .await + .expect("spawn exec"); + let mut stdin = session.stdin.take().expect("stdin"); + stdin.write_all(b"value\n").await.expect("write stdin"); + drop(stdin); + let mut stdout = String::new(); + let mut stderr = String::new(); + session + .stdout + .read_to_string(&mut stdout) + .await + .expect("read stdout"); + session + .stderr + .take() + .expect("stderr") + .read_to_string(&mut stderr) + .await + .expect("read stderr"); + assert_eq!( + session.process.wait().await.unwrap(), + BoundaryExitStatus::Exited(0) + ); + assert_eq!(stdout, "out:value"); + assert_eq!(stderr, "err:value"); + } + + #[tokio::test] + async fn exec_rejects_after_boundary_end() { + let executor = executor(); + executor.runtime.deactivate(); + let result = executor + .exec(ExecSpec { + program: "/bin/sh".to_string(), + args: vec!["-c".to_string(), "exit 0".to_string()], + env: vec![], + workdir: None, + pty: false, + }) + .await; + assert!(matches!(result, Err(BackendError::Terminated(_)))); + } + + #[tokio::test] + async fn failed_exec_leaves_boundary_active_without_registered_processes() { + let executor = executor(); + let runtime = executor.runtime.clone(); + let result = executor + .exec(ExecSpec { + program: "/definitely/missing/openshell-exec".to_string(), + args: vec![], + env: vec![], + workdir: None, + pty: false, + }) + .await; + assert!(matches!(result, Err(BackendError::Process(_)))); + runtime.ensure_active().expect("boundary remains active"); + assert_eq!(runtime.registered_process_group_count(), 0); + } + + #[tokio::test] + async fn cancelled_exec_does_not_leave_a_registered_process() { + let executor = executor(); + let runtime = executor.runtime.clone(); + let task = tokio::spawn(async move { + executor + .exec(ExecSpec { + program: "/bin/sleep".to_string(), + args: vec!["30".to_string()], + env: vec![], + workdir: None, + pty: false, + }) + .await + }); + tokio::task::yield_now().await; + task.abort(); + let _ = task.await; + + // Give the detached blocking setup time to reach its cancelled + // handoff, including the case where cancellation won before spawn. + tokio::time::sleep(std::time::Duration::from_millis(250)).await; + tokio::time::timeout(std::time::Duration::from_secs(2), async { + while runtime.registered_process_group_count() != 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("cancelled exec process must be terminated and reaped"); + runtime.ensure_active().expect("boundary remains active"); + } + + #[tokio::test] + async fn dropping_undelivered_exec_guard_terminates_process() { + let executor = executor(); + let runtime = executor.runtime.clone(); + let spawned = tokio::task::spawn_blocking(move || { + executor.spawn_piped(&ExecSpec { + program: "/bin/sleep".to_string(), + args: vec!["30".to_string()], + env: vec![], + workdir: None, + pty: false, + }) + }) + .await + .expect("spawn task") + .expect("spawn exec"); + assert_eq!(runtime.registered_process_group_count(), 1); + + // This is the post-send/pre-receive cancellation case: dropping the + // queued ownership guard must kill the process before it is observable. + drop(spawned); + tokio::time::timeout(std::time::Duration::from_secs(2), async { + while runtime.registered_process_group_count() != 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("undelivered exec process must be terminated and reaped"); + runtime.ensure_active().expect("boundary remains active"); + } + + #[tokio::test] + async fn completed_exec_removes_its_process_group_registration() { + let executor = executor(); + let runtime = executor.runtime.clone(); + let session = executor + .exec(ExecSpec { + program: "/bin/sh".to_string(), + args: vec!["-c".to_string(), "exit 0".to_string()], + env: vec![], + workdir: None, + pty: false, + }) + .await + .expect("spawn exec"); + assert_eq!( + session.process.wait().await.unwrap(), + BoundaryExitStatus::Exited(0) + ); + assert_eq!(runtime.registered_process_group_count(), 0); + } + + #[tokio::test] + async fn pty_exec_exposes_resize_and_stable_wait() { + let session = executor() + .exec(ExecSpec { + program: "/bin/sh".to_string(), + args: vec!["-c".to_string(), "exit 7".to_string()], + env: vec![], + workdir: None, + pty: true, + }) + .await + .expect("spawn pty exec"); + session + .terminal + .as_ref() + .expect("terminal") + .resize(120, 40) + .await + .expect("resize"); + assert_eq!( + session.process.wait().await.unwrap(), + BoundaryExitStatus::Exited(7) + ); + assert_eq!( + session.process.wait().await.unwrap(), + BoundaryExitStatus::Exited(7) + ); + } +} diff --git a/crates/openshell-sandbox/src/boundary_io.rs b/crates/openshell-sandbox/src/boundary_io.rs new file mode 100644 index 0000000000..d404739af0 --- /dev/null +++ b/crates/openshell-sandbox/src/boundary_io.rs @@ -0,0 +1,539 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Sandbox-local [`BoundaryLoopbackConnector`] implementation. + +use async_trait::async_trait; +use openshell_isolation_interface::contract::{ + BackendError, BoundaryDuplexStream, BoundaryLoopbackConnector, LoopbackTarget, +}; +use std::collections::HashMap; +use std::sync::atomic::{AtomicU8, Ordering}; +use std::sync::{Arc, Mutex}; + +const RUNTIME_ACTIVE: u8 = 0; +const RUNTIME_FROZEN: u8 = 1; +const RUNTIME_TERMINATED: u8 = 2; +const RUNTIME_ENFORCEMENT_LOST: u8 = 3; + +/// Shared liveness and child-process ownership for one active boundary. +pub struct BoundaryRuntimeState { + state: AtomicU8, + process_groups: Mutex>, + exclusive_pid_namespace: bool, +} + +impl BoundaryRuntimeState { + #[must_use] + pub fn new() -> Arc { + Arc::new(Self { + state: AtomicU8::new(RUNTIME_ACTIVE), + process_groups: Mutex::new(HashMap::new()), + exclusive_pid_namespace: false, + }) + } + + /// Construct state for a boundary that exclusively owns its PID namespace. + #[must_use] + pub fn new_exclusive_pid_namespace() -> Arc { + Arc::new(Self { + state: AtomicU8::new(RUNTIME_ACTIVE), + process_groups: Mutex::new(HashMap::new()), + exclusive_pid_namespace: true, + }) + } + + #[must_use] + pub const fn requires_dedicated_process_group(&self) -> bool { + self.exclusive_pid_namespace + } + + pub fn ensure_active(&self) -> Result<(), BackendError> { + match self.state.load(Ordering::Acquire) { + RUNTIME_ACTIVE => Ok(()), + RUNTIME_FROZEN => Err(BackendError::Unavailable( + "boundary is frozen while supervisor control recovers".to_string(), + )), + _ => Err(BackendError::Terminated("boundary has ended".to_string())), + } + } + + #[must_use] + pub fn is_active(&self) -> bool { + self.state.load(Ordering::Acquire) == RUNTIME_ACTIVE + } + + #[must_use] + pub fn enforcement_was_lost(&self) -> bool { + self.state.load(Ordering::Acquire) == RUNTIME_ENFORCEMENT_LOST + } + + pub fn register_process_group( + &self, + pid: u32, + terminal: Arc, + signal_lock: Arc>, + ) -> Result<(), BackendError> { + let mut groups = self + .process_groups + .lock() + .map_err(|_| BackendError::Process("boundary process registry poisoned".to_string()))?; + self.ensure_active()?; + groups.insert( + pid, + RegisteredProcessGroup { + pid, + terminal, + signal_lock, + }, + ); + Ok(()) + } + + pub fn unregister_process_group( + &self, + pid: u32, + terminal: &Arc, + ) { + if let Ok(mut groups) = self.process_groups.lock() + && groups + .get(&pid) + .is_some_and(|group| Arc::ptr_eq(&group.terminal, terminal)) + { + groups.remove(&pid); + } + } + + #[cfg(test)] + pub fn registered_process_group_count(&self) -> usize { + self.process_groups.lock().map_or(0, |groups| groups.len()) + } + + /// End the boundary and terminate every registered workload process group. + pub fn deactivate(&self) { + let previous = self.state.swap(RUNTIME_TERMINATED, Ordering::AcqRel); + if matches!(previous, RUNTIME_ACTIVE | RUNTIME_FROZEN) { + self.signal_registered_processes(nix::sys::signal::Signal::SIGCONT); + self.signal_registered_processes(nix::sys::signal::Signal::SIGKILL); + } + } + + /// Stop every owned workload process while the supervisor reconnects. + /// + /// New process and loopback operations fail while frozen. The registered + /// process groups include the canonical workload and every sandbox exec. + #[must_use] + pub fn freeze(&self) -> bool { + if self + .state + .compare_exchange( + RUNTIME_ACTIVE, + RUNTIME_FROZEN, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_err() + { + return false; + } + self.signal_registered_processes(nix::sys::signal::Signal::SIGSTOP); + true + } + + /// Resume a workload only after the replacement supervisor connection has + /// authenticated, attached, and reconfirmed the boundary. + #[must_use] + pub fn resume(&self) -> bool { + if self + .state + .compare_exchange( + RUNTIME_FROZEN, + RUNTIME_ACTIVE, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_err() + { + return false; + } + self.signal_registered_processes(nix::sys::signal::Signal::SIGCONT); + true + } + + /// Begin fail-closed termination after authenticated recovery times out. + /// Frozen tasks are continued before `SIGTERM` so they can run their + /// ordinary shutdown handlers. + #[must_use] + pub fn begin_enforcement_loss_termination(&self) -> bool { + if self + .state + .compare_exchange( + RUNTIME_FROZEN, + RUNTIME_ENFORCEMENT_LOST, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_err() + { + return false; + } + self.signal_registered_processes(nix::sys::signal::Signal::SIGCONT); + self.signal_registered_processes(nix::sys::signal::Signal::SIGTERM); + true + } + + /// Begin an authenticated, graceful boundary shutdown. + #[must_use] + pub fn begin_termination(&self) -> bool { + loop { + let state = self.state.load(Ordering::Acquire); + if !matches!(state, RUNTIME_ACTIVE | RUNTIME_FROZEN) { + return false; + } + if self + .state + .compare_exchange( + state, + RUNTIME_TERMINATED, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_ok() + { + self.signal_registered_processes(nix::sys::signal::Signal::SIGCONT); + self.signal_registered_processes(nix::sys::signal::Signal::SIGTERM); + return true; + } + } + } + + /// Force all remaining owned process groups to exit. + pub fn force_kill(&self) { + self.signal_registered_processes(nix::sys::signal::Signal::SIGCONT); + self.signal_registered_processes(nix::sys::signal::Signal::SIGKILL); + } + + #[must_use] + pub fn has_registered_processes(&self) -> bool { + self.process_groups + .lock() + .is_ok_and(|groups| !groups.is_empty()) + } + + /// End the boundary because required standing enforcement was lost. + /// + /// Returns `true` only to the caller that won the active-to-terminated + /// transition. A concurrent normal teardown cannot later be reclassified + /// as enforcement loss. + pub fn deactivate_for_enforcement_loss(&self) -> bool { + loop { + let state = self.state.load(Ordering::Acquire); + if !matches!(state, RUNTIME_ACTIVE | RUNTIME_FROZEN) { + return false; + } + if self + .state + .compare_exchange( + state, + RUNTIME_ENFORCEMENT_LOST, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_ok() + { + self.force_kill(); + return true; + } + } + } + + fn signal_registered_processes(&self, signal: nix::sys::signal::Signal) { + let groups = self + .process_groups + .lock() + .map(|groups| groups.values().cloned().collect::>()) + .unwrap_or_default(); + for group in &groups { + group.signal(signal); + } + #[cfg(target_os = "linux")] + { + let roots = groups.iter().map(|group| group.pid).collect::>(); + // A workload may create another process group or session. Once its + // registered roots are stopped they cannot fork again, so bounded + // repeated descendant scans close the signal-to-scan race without + // requiring ptrace or a capability. + let mut previous = Vec::new(); + for _ in 0..4 { + let owned = owned_process_ids(&roots, self.exclusive_pid_namespace); + for pid in &owned { + if roots.contains(pid) { + continue; + } + if let Ok(pid) = i32::try_from(*pid) { + let _ = nix::sys::signal::kill(nix::unistd::Pid::from_raw(pid), signal); + } + } + if owned == previous { + break; + } + previous = owned; + } + } + } +} + +#[cfg(target_os = "linux")] +fn owned_process_ids(roots: &[u32], exclusive_pid_namespace: bool) -> Vec { + let mut parents = HashMap::new(); + let Ok(entries) = std::fs::read_dir("/proc") else { + return roots.to_vec(); + }; + for entry in entries.flatten() { + let Some(pid) = entry + .file_name() + .to_str() + .and_then(|name| name.parse::().ok()) + else { + continue; + }; + let Ok(stat) = std::fs::read_to_string(entry.path().join("stat")) else { + continue; + }; + let Some(after_name) = stat.rsplit_once(") ").map(|(_, fields)| fields) else { + continue; + }; + let Some(parent) = after_name + .split_whitespace() + .nth(1) + .and_then(|field| field.parse::().ok()) + else { + continue; + }; + parents.insert(pid, parent); + } + + // When openshell-sandbox is PID 1, every other process in its exclusive + // namespace is workload-owned, including an orphan reparented during the + // scan. Outside that deployment shape, restrict the walk to registered + // roots so unit tests and development runs cannot affect sibling tasks. + if exclusive_pid_namespace && std::process::id() == 1 { + let mut owned = parents + .keys() + .copied() + .filter(|pid| *pid != 1) + .collect::>(); + owned.sort_unstable(); + return owned; + } + + let mut owned = roots.to_vec(); + loop { + let mut changed = false; + for (&pid, &parent) in &parents { + if !owned.contains(&pid) && owned.contains(&parent) { + owned.push(pid); + changed = true; + } + } + if !changed { + break; + } + } + owned.sort_unstable(); + owned.dedup(); + owned +} + +#[derive(Clone)] +struct RegisteredProcessGroup { + pid: u32, + terminal: Arc, + signal_lock: Arc>, +} + +impl RegisteredProcessGroup { + fn signal(&self, signal: nix::sys::signal::Signal) { + let _signal_guard = self + .signal_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if self.terminal.load(Ordering::Acquire) { + return; + } + if let Ok(pid) = i32::try_from(self.pid) { + let _ = nix::sys::signal::killpg(nix::unistd::Pid::from_raw(pid), signal); + } + } +} + +/// Loopback port-forward owned by the sandbox process. +pub struct LocalLoopbackConnector { + runtime: Option>, +} + +impl LocalLoopbackConnector { + #[must_use] + pub fn new(runtime: Option>) -> Self { + Self { runtime } + } +} + +#[async_trait] +impl BoundaryLoopbackConnector for LocalLoopbackConnector { + async fn connect(&self, target: LoopbackTarget) -> Result { + if let Some(runtime) = &self.runtime { + runtime.ensure_active()?; + } + let addr = std::net::SocketAddr::new(target.host(), target.port()); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(&[addr]) + .await + .map_err(|e| BackendError::Process(format!("port-forward connect to {addr}: {e}")))?; + if let Some(runtime) = &self.runtime { + runtime.ensure_active()?; + } + Ok(Box::new(stream)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::net::Ipv4Addr; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + /// Stands in for the SSH server's port-forward path: connect through the + /// interface, write, and read the echo. + #[tokio::test] + async fn loopback_connector_connects_and_round_trips() { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + let (mut sock, _) = listener.accept().await.unwrap(); + let mut buf = [0u8; 4]; + sock.read_exact(&mut buf).await.unwrap(); + sock.write_all(&buf).await.unwrap(); + }); + + let pf = LocalLoopbackConnector::new(None); + let target = + LoopbackTarget::new(Ipv4Addr::LOCALHOST.into(), addr.port()).expect("loopback target"); + let mut conn = pf.connect(target).await.expect("connect through interface"); + conn.write_all(b"ping").await.unwrap(); + let mut buf = [0u8; 4]; + conn.read_exact(&mut buf).await.unwrap(); + assert_eq!(&buf, b"ping"); + } + + /// Drive the port-forward interface through a generic `&dyn` consumer, proving a + /// kernel-separated backend (tunneling into a guest) would use the same call. + #[tokio::test] + async fn loopback_connector_is_driven_via_dyn() { + async fn forward_one(pf: &dyn BoundaryLoopbackConnector, target: LoopbackTarget) -> bool { + pf.connect(target).await.is_ok() + } + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + let _ = listener.accept().await; + }); + let pf = LocalLoopbackConnector::new(None); + let target = LoopbackTarget::new(Ipv4Addr::LOCALHOST.into(), addr.port()).unwrap(); + assert!(forward_one(&pf, target).await); + } + + #[tokio::test] + async fn loopback_connector_rejects_after_boundary_end() { + let runtime = BoundaryRuntimeState::new(); + let pf = LocalLoopbackConnector::new(Some(runtime.clone())); + runtime.deactivate(); + let target = LoopbackTarget::new(Ipv4Addr::LOCALHOST.into(), 1).unwrap(); + assert!(matches!( + pf.connect(target).await, + Err(BackendError::Terminated(_)) + )); + } + + #[tokio::test] + async fn failed_loopback_connector_keeps_boundary_active() { + let runtime = BoundaryRuntimeState::new(); + let pf = LocalLoopbackConnector::new(Some(runtime.clone())); + // Port zero is never a connectable TCP destination. Reserving an ephemeral + // port and dropping its listener races other parallel tests that may bind it. + let target = LoopbackTarget::new(Ipv4Addr::LOCALHOST.into(), 0).unwrap(); + assert!(matches!( + pf.connect(target).await, + Err(BackendError::Process(_)) + )); + runtime.ensure_active().expect("boundary remains active"); + } + + #[test] + fn stale_unregister_preserves_reused_process_group_registration() { + let runtime = BoundaryRuntimeState::new(); + let first_terminal = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let second_terminal = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let pid = 42; + runtime + .register_process_group(pid, first_terminal.clone(), Arc::new(Mutex::new(()))) + .expect("first registration"); + runtime + .register_process_group(pid, second_terminal.clone(), Arc::new(Mutex::new(()))) + .expect("replacement registration"); + + runtime.unregister_process_group(pid, &first_terminal); + assert_eq!(runtime.registered_process_group_count(), 1); + + runtime.unregister_process_group(pid, &second_terminal); + assert_eq!(runtime.registered_process_group_count(), 0); + } + + #[test] + fn canonical_process_completion_does_not_end_boundary_runtime() { + let runtime = BoundaryRuntimeState::new_exclusive_pid_namespace(); + let terminal = Arc::new(std::sync::atomic::AtomicBool::new(true)); + runtime + .register_process_group(42, terminal.clone(), Arc::new(Mutex::new(()))) + .expect("register canonical process"); + + runtime.unregister_process_group(42, &terminal); + + runtime + .ensure_active() + .expect("canonical completion must preserve exec and forwarding"); + assert_eq!(runtime.registered_process_group_count(), 0); + runtime.deactivate(); + assert!(matches!( + runtime.ensure_active(), + Err(BackendError::Terminated(_)) + )); + } + + #[test] + fn freeze_blocks_new_operations_until_explicit_resume() { + let runtime = BoundaryRuntimeState::new_exclusive_pid_namespace(); + + assert!(runtime.freeze()); + assert!(matches!( + runtime.ensure_active(), + Err(BackendError::Unavailable(_)) + )); + assert!(!runtime.freeze()); + assert!(runtime.resume()); + runtime.ensure_active().expect("runtime resumed"); + } + + #[test] + fn enforcement_loss_is_terminal_after_freeze() { + let runtime = BoundaryRuntimeState::new_exclusive_pid_namespace(); + + assert!(runtime.freeze()); + assert!(runtime.begin_enforcement_loss_termination()); + assert!(runtime.enforcement_was_lost()); + assert!(!runtime.resume()); + assert!(matches!( + runtime.ensure_active(), + Err(BackendError::Terminated(_)) + )); + } +} diff --git a/crates/openshell-sandbox/src/boundary_server.rs b/crates/openshell-sandbox/src/boundary_server.rs new file mode 100644 index 0000000000..571b83fe37 --- /dev/null +++ b/crates/openshell-sandbox/src/boundary_server.rs @@ -0,0 +1,4956 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared implementation of the capability-free `openshell-sandbox` runtime. +//! +//! This is transport and lifecycle glue, not another supervisor model. When +//! the control role authorizes `start_agent`, it invokes the existing process +//! supervisor inside the driver-provisioned boundary. + +#![allow(unsafe_code)] + +use std::path::Path; + +#[cfg(target_os = "linux")] +mod linux { + use std::fs::File; + use std::io::{self, Read, Write}; + use std::mem::size_of; + use std::os::fd::{AsRawFd, FromRawFd, IntoRawFd as _, OwnedFd}; + use std::os::unix::fs::{FileTypeExt as _, MetadataExt as _, PermissionsExt as _}; + use std::path::Path; + use std::pin::Pin; + use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicUsize, Ordering}; + use std::sync::{Arc, Condvar, Mutex}; + use std::task::{Context, Poll}; + use std::time::Duration; + + use crate::boundary_io::BoundaryRuntimeState; + use crate::delegated::{AgentSignaler, spawn_workload}; + use crate::identity::{DriverIdentity, resolve_process_identity}; + use crate::main_session::{MainOutput, MainSession}; + use crate::network_broker::NetworkBroker; + use crate::process::ProcessStatus; + use openshell_core::jwt::{ + SandboxId, SessionJwtVerifier, SessionTokenProfile, SessionVerificationKey, SystemJwtClock, + }; + use openshell_core::provider_credentials::ProviderCredentialState; + use openshell_isolation_interface::contract::{ + BoundaryExec, BoundaryLoopbackConnector, BoundaryProcess, BoundaryTerminal, + CapabilityEvidence, ExecSession, LoopbackTarget, ResolvedWorkloadIdentity, + SandboxConfirmEvidence, + }; + use openshell_sandbox_backend::mediation::{ + self, DnsQueryWire, MediationFrame, MediationFrameKind, + }; + #[cfg(test)] + use openshell_sandbox_backend::proto::isolation_boundary_client::IsolationBoundaryClient; + use openshell_sandbox_backend::proto::{ + BoundaryChunk, + isolation_boundary_server::{IsolationBoundary, IsolationBoundaryServer}, + }; + use openshell_sandbox_backend::sandbox_auth::{ + SandboxConnectionId, SandboxConnectionRegistry, SandboxProtocolAuthenticator, + SandboxProtocolPrincipal, + }; + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + use tokio_stream::wrappers::ReceiverStream; + + use openshell_sandbox_backend::boundary_protocol::{ + AgentSpecWire, BinaryIdentityWire, BoundaryConfig, BoundaryErrorKind, + BoundaryListener as BoundaryListenerConfig, DnsQueryResultWire, ExecSpecWire, + ExitStatusWire, MediationTimingWire, OutputWindowWire, ProcessKindWire, + ProcessSnapshotWire, Request, RequestEnvelope, Response, ResponseEnvelope, STREAM_EXIT, + STREAM_NETWORK_DECISION, STREAM_STDERR, STREAM_STDIN, STREAM_STDIN_CLOSED, STREAM_STDOUT, + SandboxPolicyWire, SessionSnapshotWire, SignalWire, encode_frame, read_frame, + read_stream_frame, validate_resource_claims, write_frame, write_stream_frame, + }; + + const CONTROL_IO_TIMEOUT: Duration = Duration::from_secs(30); + const CONTROL_HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(5); + const CONTROL_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(5); + const CONTROL_KEEPALIVE_TIMEOUT: Duration = Duration::from_secs(10); + const MEDIATION_REPLACEMENT_TIMEOUT: Duration = Duration::from_secs(20); + const AUTHENTICATED_RECONNECT_TIMEOUT: Duration = Duration::from_secs(30); + const ENFORCEMENT_LOSS_TERMINATION_GRACE: Duration = + Duration::from_secs(openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS as u64); + const FORCE_KILL_REAP_TIMEOUT: Duration = Duration::from_secs(2); + const MAX_PENDING_HANDSHAKES: usize = 32; + const MAX_CONTROL_CONNECTIONS: usize = 128; + const MAX_REPLAY_LEDGER_ENTRIES: usize = 4096; + const MAX_RETAINED_EXEC_PROCESSES: usize = 64; + + fn duration_micros(duration: Duration) -> u64 { + u64::try_from(duration.as_micros()).unwrap_or(u64::MAX) + } + + struct ControlConnectionSlot(Arc); + + impl Drop for ControlConnectionSlot { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::AcqRel); + } + } + + fn acquire_control_connection_slot(active: &Arc) -> Option { + active + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { + (current < MAX_CONTROL_CONNECTIONS).then_some(current + 1) + }) + .ok() + .map(|_| ControlConnectionSlot(active.clone())) + } + static BOUNDARY_TERMINATION_REQUESTED: AtomicBool = AtomicBool::new(false); + + extern "C" fn request_boundary_termination(_signal: libc::c_int) { + BOUNDARY_TERMINATION_REQUESTED.store(true, Ordering::Release); + } + + pub fn run_boundary( + config_path: &Path, + qualification: crate::RuntimeQualification, + ) -> Result<(), String> { + install_boundary_signal_handlers()?; + make_boundary_nondumpable()?; + disable_core_dumps()?; + let bytes = std::fs::read(config_path) + .map_err(|error| format!("read boundary config {}: {error}", config_path.display()))?; + let config: BoundaryConfig = serde_json::from_slice(&bytes).map_err(|error| { + format!("decode boundary config {}: {error}", config_path.display()) + })?; + validate_config(&config)?; + validate_runtime_resource_claims(&config)?; + validate_running_identity(&config.workload_identity)?; + std::fs::remove_file(config_path).map_err(|error| { + format!("consume boundary config {}: {error}", config_path.display()) + })?; + let child_env = serde_json::to_string(&config.child_env) + .map_err(|error| format!("encode boundary workload environment: {error}"))?; + // This runs before the Tokio runtime or control threads exist. The process + // supervisor consumes the serialized map and applies values only to + // workload children. + unsafe { + std::env::set_var(openshell_core::sandbox_env::USER_ENVIRONMENT, child_env); + } + crate::sandbox::apply_supervisor_startup_hardening() + .map_err(|error| format!("install sandbox process prelude: {error}"))?; + if nix::unistd::getpid().as_raw() == 1 { + crate::managed_children::start_orphan_reaper() + .map_err(|error| format!("start sandbox orphan reaper: {error}"))?; + } + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .map_err(|error| format!("start sandbox workload launcher: {error}"))?; + let protected_control_port = match &config.listener { + BoundaryListenerConfig::TlsTcp { address, .. } => Some(address.port()), + BoundaryListenerConfig::Unix { .. } | BoundaryListenerConfig::Vsock { .. } => None, + }; + let network_broker = NetworkBroker::start(listener, protected_control_port) + .map_err(|error| format!("start sandbox network broker: {error}"))?; + let process_runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .map_err(|error| format!("create boundary process runtime: {error}"))?; + let runtime = Arc::new(BoundaryRuntime::new( + config.clone(), + process_runtime.handle().clone(), + network_broker, + launcher, + qualification, + )?); + serve(&config.listener, runtime) + } + + fn make_boundary_nondumpable() -> Result<(), String> { + // SAFETY: PR_SET_DUMPABLE accepts one scalar flag. The sandbox keeps + // bootstrap and protected-channel keys in memory after this point. + if unsafe { libc::prctl(libc::PR_SET_DUMPABLE, 0, 0, 0, 0) } == 0 { + Ok(()) + } else { + Err(format!( + "make sandbox process nondumpable: {}", + io::Error::last_os_error() + )) + } + } + + fn disable_core_dumps() -> Result<(), String> { + let limit = libc::rlimit { + rlim_cur: 0, + rlim_max: 0, + }; + // SAFETY: `limit` is a valid immutable rlimit value. + if unsafe { libc::setrlimit(libc::RLIMIT_CORE, &raw const limit) } == 0 { + Ok(()) + } else { + Err(format!( + "disable sandbox core dumps: {}", + io::Error::last_os_error() + )) + } + } + + fn install_boundary_signal_handlers() -> Result<(), String> { + BOUNDARY_TERMINATION_REQUESTED.store(false, Ordering::Release); + let action = nix::sys::signal::SigAction::new( + nix::sys::signal::SigHandler::Handler(request_boundary_termination), + nix::sys::signal::SaFlags::empty(), + nix::sys::signal::SigSet::empty(), + ); + for signal in [ + nix::sys::signal::Signal::SIGTERM, + nix::sys::signal::Signal::SIGINT, + ] { + // SAFETY: the installed handler only performs a lock-free atomic + // store, which is async-signal-safe, and remains valid for the + // lifetime of the boundary process. + unsafe { nix::sys::signal::sigaction(signal, &action) } + .map_err(|error| format!("install boundary {signal:?} handler: {error}"))?; + } + Ok(()) + } + + fn validate_config(config: &BoundaryConfig) -> Result<(), String> { + if config.boundary_id.is_empty() { + return Err("boundary ID must not be empty".to_string()); + } + if config.generation.is_empty() { + return Err("boundary generation must not be empty".to_string()); + } + if config.gateway_id.is_empty() { + return Err("gateway ID must not be empty".to_string()); + } + if config.verification_keys.is_empty() { + return Err("at least one gateway verification key is required".to_string()); + } + validate_resource_claims(&config.resource_claims).map_err(|error| error.to_string())?; + config + .driver_fence + .validate() + .map_err(|error| error.to_string())?; + for (claim, path) in &config.resource_claim_files { + if !config.resource_claims.contains_key(claim) { + return Err(format!( + "runtime resource-claim file refers to unknown claim {claim}" + )); + } + if !path.is_absolute() { + return Err(format!( + "runtime resource-claim file for {claim} must be absolute" + )); + } + } + match &config.listener { + BoundaryListenerConfig::Unix { socket_path, tls } + if !socket_path.is_absolute() || !tls_paths_are_absolute(tls) => + { + return Err("boundary Unix socket path must be absolute".to_string()); + } + BoundaryListenerConfig::TlsTcp { address, tls } + if address.port() == 0 || !tls_paths_are_absolute(tls) => + { + return Err( + "boundary TLS listener requires a nonzero port and absolute certificate paths" + .to_string(), + ); + } + BoundaryListenerConfig::Vsock { + control_port: 0, .. + } => { + return Err("boundary control port must be nonzero".to_string()); + } + BoundaryListenerConfig::Unix { .. } + | BoundaryListenerConfig::TlsTcp { .. } + | BoundaryListenerConfig::Vsock { .. } => {} + } + if config.workload_identity.uid == 0 || config.workload_identity.gid == 0 { + return Err("sandbox workload UID and GID must be nonzero".to_string()); + } + Ok(()) + } + + fn tls_paths_are_absolute( + tls: &openshell_sandbox_backend::boundary_protocol::SandboxTlsServerConfig, + ) -> bool { + tls.certificate_chain_path.is_absolute() && tls.private_key_path.is_absolute() + } + + fn validate_runtime_resource_claims(config: &BoundaryConfig) -> Result<(), String> { + for (claim, path) in &config.resource_claim_files { + let expected = config + .resource_claims + .get(claim) + .ok_or_else(|| format!("resource claim file has no expected value: {claim}"))?; + let observed = std::fs::read_to_string(path).map_err(|error| { + format!( + "read runtime resource claim {claim} from {}: {error}", + path.display() + ) + })?; + if observed.trim() != expected { + return Err(format!( + "runtime resource claim {claim} does not match the admitted resource" + )); + } + } + Ok(()) + } + + fn normalized_supplementary_groups(mut groups: Vec, primary_gid: u32) -> Vec { + groups.retain(|gid| *gid != primary_gid); + groups.sort_unstable(); + groups.dedup(); + groups + } + + #[allow(clippy::similar_names)] + fn validate_running_identity(expected: &ResolvedWorkloadIdentity) -> Result<(), String> { + let mut real_uid = 0; + let mut effective_uid = 0; + let mut saved_uid = 0; + let mut real_gid = 0; + let mut effective_gid = 0; + let mut saved_gid = 0; + // SAFETY: all pointers refer to live scalar output storage. + if unsafe { + libc::getresuid( + &raw mut real_uid, + &raw mut effective_uid, + &raw mut saved_uid, + ) + } != 0 + || unsafe { + libc::getresgid( + &raw mut real_gid, + &raw mut effective_gid, + &raw mut saved_gid, + ) + } != 0 + { + return Err(format!( + "measure sandbox identity: {}", + io::Error::last_os_error() + )); + } + if [real_uid, effective_uid, saved_uid] + .iter() + .any(|uid| *uid != expected.uid) + || [real_gid, effective_gid, saved_gid] + .iter() + .any(|gid| *gid != expected.gid) + { + return Err(format!( + "sandbox identity does not match resolved workload {}:{}", + expected.uid, expected.gid + )); + } + // SAFETY: a null buffer with size zero queries the group count. + let count = unsafe { libc::getgroups(0, std::ptr::null_mut()) }; + if count < 0 { + return Err(format!( + "measure sandbox supplementary groups: {}", + io::Error::last_os_error() + )); + } + let mut groups = vec![0_u32; usize::try_from(count).unwrap_or(0)]; + if count > 0 { + // SAFETY: groups has capacity for exactly `count` gid_t values. + if unsafe { libc::getgroups(count, groups.as_mut_ptr()) } != count { + return Err(format!( + "read sandbox supplementary groups: {}", + io::Error::last_os_error() + )); + } + } + let groups = normalized_supplementary_groups(groups, expected.gid); + if groups != expected.supplementary_gids { + return Err(format!( + "sandbox supplementary groups {groups:?} do not match resolved workload {:?}", + expected.supplementary_gids + )); + } + Ok(()) + } + + fn serve(config: &BoundaryListenerConfig, runtime: Arc) -> Result<(), String> { + let listener = ControlListener::bind(config) + .map_err(|error| format!("bind boundary control listener: {error}"))?; + let active_connections = Arc::new(AtomicUsize::new(0)); + let pending_handshakes = Arc::new(tokio::sync::Semaphore::new(MAX_PENDING_HANDSHAKES)); + tracing::info!(?config, "Boundary control listener ready"); + loop { + if BOUNDARY_TERMINATION_REQUESTED.load(Ordering::Acquire) { + runtime.shutdown(); + return Ok(()); + } + match listener.accept() { + Ok(stream) => { + // Unauthenticated sockets use only a bounded async TLS + // task, never an OS thread or an authenticated session slot. + let Ok(pending) = pending_handshakes.clone().try_acquire_owned() else { + continue; + }; + let active_connections = active_connections.clone(); + let runtime = runtime.clone(); + runtime.process_runtime.spawn({ + let runtime = runtime.clone(); + async move { + if let Err(error) = serve_control_connection( + stream, + runtime, + pending, + active_connections, + ) + .await + { + tracing::debug!(%error, "Boundary control connection ended"); + } + } + }); + } + Err(error) if error.kind() == io::ErrorKind::Interrupted => {} + Err(error) if error.kind() == io::ErrorKind::PermissionDenied => {} + Err(error) if error.kind() == io::ErrorKind::WouldBlock => { + std::thread::sleep(Duration::from_millis(10)); + } + Err(error) => return Err(format!("accept boundary control connection: {error}")), + } + } + } + + async fn serve_control_connection( + stream: ControlStream, + runtime: Arc, + pending: tokio::sync::OwnedSemaphorePermit, + active_connections: Arc, + ) -> Result<(), String> { + let stream = stream + .establish_async(&runtime.process_runtime) + .await + .map_err(|error| format!("authenticate boundary transport: {error}"))?; + drop(pending); + let Some(_slot) = acquire_control_connection_slot(&active_connections) else { + return Err("authenticated control connection limit reached".to_string()); + }; + serve_grpc(stream.into_tokio()?, runtime, SandboxConnectionId::new()).await + } + + async fn serve_grpc( + stream: openshell_isolation_interface::contract::BoundaryDuplexStream, + runtime: Arc, + connection_id: SandboxConnectionId, + ) -> Result<(), String> { + let (connection_shutdown, connection_closed) = tokio::sync::watch::channel(()); + runtime.register_connection(connection_id, connection_shutdown.clone()); + let connection_expiry = Arc::new(ConnectionExpiry::new(connection_shutdown.clone())); + let incoming = tokio_stream::StreamExt::chain( + tokio_stream::iter([Ok::<_, io::Error>(GrpcServerIo { + stream, + _connection_alive: connection_shutdown.clone(), + _disconnect: TransportDisconnectGuard { + runtime: Arc::downgrade(&runtime), + connection_id, + }, + })]), + tokio_stream::pending(), + ); + let mut shutdown = connection_closed.clone(); + let result = tonic::transport::Server::builder() + .http2_keepalive_interval(Some(CONTROL_KEEPALIVE_INTERVAL)) + .http2_keepalive_timeout(Some(CONTROL_KEEPALIVE_TIMEOUT)) + .max_concurrent_streams( + u32::try_from(MAX_CONTROL_CONNECTIONS) + .map_err(|error| format!("invalid control connection limit: {error}"))?, + ) + .initial_stream_window_size(16 * 1024 * 1024) + .initial_connection_window_size(16 * 1024 * 1024) + .add_service( + IsolationBoundaryServer::new(GrpcBoundaryService { + runtime: runtime.clone(), + connection_id, + connection_expiry, + connection_closed, + }) + .max_decoding_message_size(64 * 1024) + .max_encoding_message_size(64 * 1024), + ) + .serve_with_incoming_shutdown(incoming, async move { + let _ = shutdown.changed().await; + }) + .await; + runtime.transport_disconnected(connection_id); + result.map_err(|error| format!("serve boundary gRPC connection: {error}")) + } + + struct GrpcServerIo { + stream: openshell_isolation_interface::contract::BoundaryDuplexStream, + // Dropping the actual HTTP/2 transport stops all detached stream + // bridges, including on keepalive failure or task cancellation. + _connection_alive: tokio::sync::watch::Sender<()>, + _disconnect: TransportDisconnectGuard, + } + + struct TransportDisconnectGuard { + runtime: std::sync::Weak, + connection_id: SandboxConnectionId, + } + + impl Drop for TransportDisconnectGuard { + fn drop(&mut self) { + if let Some(runtime) = self.runtime.upgrade() { + runtime.transport_disconnected(self.connection_id); + } + } + } + + impl tokio::io::AsyncRead for GrpcServerIo { + fn poll_read( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + buffer: &mut tokio::io::ReadBuf<'_>, + ) -> Poll> { + Pin::new(&mut self.stream).poll_read(context, buffer) + } + } + + impl tokio::io::AsyncWrite for GrpcServerIo { + fn poll_write( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + buffer: &[u8], + ) -> Poll> { + Pin::new(&mut self.stream).poll_write(context, buffer) + } + + fn poll_flush(mut self: Pin<&mut Self>, context: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.stream).poll_flush(context) + } + + fn poll_shutdown( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + ) -> Poll> { + Pin::new(&mut self.stream).poll_shutdown(context) + } + } + + impl tonic::transport::server::Connected for GrpcServerIo { + type ConnectInfo = (); + + fn connect_info(&self) -> Self::ConnectInfo {} + } + + #[derive(Clone)] + struct GrpcBoundaryService { + runtime: Arc, + connection_id: SandboxConnectionId, + connection_expiry: Arc, + connection_closed: tokio::sync::watch::Receiver<()>, + } + + struct ConnectionExpiry { + deadline: tokio::sync::watch::Sender>, + worker: tokio::task::AbortHandle, + } + + impl ConnectionExpiry { + fn new(connection_shutdown: tokio::sync::watch::Sender<()>) -> Self { + let (deadline, deadline_updates) = tokio::sync::watch::channel(None); + let worker = tokio::spawn(run_connection_expiry(deadline_updates, connection_shutdown)) + .abort_handle(); + Self { deadline, worker } + } + + fn update(&self, expires_at: i64) { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |duration| duration.as_secs()); + let expires_at = u64::try_from(expires_at).unwrap_or_default(); + self.update_deadline( + tokio::time::Instant::now() + Duration::from_secs(expires_at.saturating_sub(now)), + ); + } + + fn update_deadline(&self, deadline: tokio::time::Instant) { + let _ = self.deadline.send_if_modified(|current| { + if *current == Some(deadline) { + false + } else { + *current = Some(deadline); + true + } + }); + } + } + + impl Drop for ConnectionExpiry { + fn drop(&mut self) { + self.worker.abort(); + } + } + + async fn run_connection_expiry( + mut deadline_updates: tokio::sync::watch::Receiver>, + connection_shutdown: tokio::sync::watch::Sender<()>, + ) { + loop { + let deadline = *deadline_updates.borrow_and_update(); + let Some(deadline) = deadline else { + if deadline_updates.changed().await.is_err() { + return; + } + continue; + }; + tokio::select! { + () = tokio::time::sleep_until(deadline) => { + let _ = connection_shutdown.send(()); + return; + } + result = deadline_updates.changed() => { + if result.is_err() { + return; + } + } + } + } + } + + type GrpcResponseStream = ReceiverStream>; + + #[tonic::async_trait] + impl IsolationBoundary for GrpcBoundaryService { + type ExchangeStream = GrpcResponseStream; + type MediateStream = GrpcResponseStream; + + async fn exchange( + &self, + request: tonic::Request>, + ) -> Result, tonic::Status> { + let principal = self + .runtime + .authenticate_request(self.connection_id, request.metadata())?; + self.connection_expiry + .update(principal.session().expires_at); + let (stream, response) = + bridge_grpc_server_stream(request.into_inner(), self.connection_closed.clone()); + let runtime = self.runtime.clone(); + tokio::task::spawn_blocking(move || { + let stream = ControlStream::Grpc { + stream, + runtime: runtime.process_runtime.clone(), + }; + if let Err(error) = serve_one(stream, &runtime, &principal) { + tracing::warn!(%error, "Boundary gRPC exchange failed"); + } + }); + Ok(tonic::Response::new(response)) + } + + async fn mediate( + &self, + request: tonic::Request>, + ) -> Result, tonic::Status> { + let principal = self + .runtime + .authenticate_request(self.connection_id, request.metadata())?; + self.connection_expiry + .update(principal.session().expires_at); + let (stream, response) = + bridge_grpc_server_stream(request.into_inner(), self.connection_closed.clone()); + let runtime = self.runtime.clone(); + tokio::spawn(async move { + if let Err(error) = serve_persistent_mediation(stream, runtime, principal).await { + tracing::warn!(%error, "Persistent boundary mediation ended"); + } + }); + Ok(tonic::Response::new(response)) + } + } + + fn bridge_grpc_server_stream( + mut inbound: tonic::Streaming, + connection_closed: tokio::sync::watch::Receiver<()>, + ) -> (tokio::io::DuplexStream, GrpcResponseStream) { + let (application, bridge) = tokio::io::duplex(256 * 1024); + let (mut reader, mut writer) = tokio::io::split(bridge); + let (outbound, outbound_rx) = + tokio::sync::mpsc::channel::>(64); + let mut inbound_closed = connection_closed.clone(); + tokio::spawn(async move { + tokio::select! { + _ = inbound_closed.changed() => {}, + () = async { + loop { + match inbound.message().await { + Ok(Some(chunk)) => { + if writer.write_all(&chunk.data).await.is_err() { + return; + } + } + Ok(None) => { + let _ = writer.shutdown().await; + return; + } + Err(error) => { + tracing::debug!(%error, "Boundary gRPC request stream ended"); + return; + } + } + } + } => {}, + } + }); + let mut outbound_closed = connection_closed; + tokio::spawn(async move { + tokio::select! { + _ = outbound_closed.changed() => {}, + () = async { + let mut buffer = vec![0_u8; 16 * 1024]; + loop { + let read = match reader.read(&mut buffer).await { + Ok(read) => read, + Err(error) => { + tracing::debug!(%error, "Boundary gRPC response reader ended"); + return; + } + }; + if read == 0 { + return; + } + if outbound + .send(Ok(BoundaryChunk { + data: buffer[..read].to_vec(), + })) + .await + .is_err() + { + return; + } + } + } => {}, + } + }); + (application, ReceiverStream::new(outbound_rx)) + } + + const MEDIATION_EVENT_QUEUE: usize = 256; + const MEDIATION_ROUTE_QUEUE: usize = 64; + + struct BoundaryOutboundFrame { + kind: MediationFrameKind, + stream_id: u64, + payload: Vec, + } + + type BoundaryMediationRoutes = Arc< + tokio::sync::Mutex< + std::collections::HashMap>, + >, + >; + + async fn serve_persistent_mediation( + mut stream: tokio::io::DuplexStream, + runtime: Arc, + principal: SandboxProtocolPrincipal, + ) -> Result<(), String> { + let request: RequestEnvelope = tokio::time::timeout( + CONTROL_IO_TIMEOUT, + openshell_sandbox_backend::boundary_protocol::read_frame_async(&mut stream), + ) + .await + .map_err(|_| "mediation attach timed out".to_string())? + .map_err(|error| format!("read mediation attach: {error}"))?; + let request_id = request.request_id.clone(); + runtime.authorize_request(&principal, &request.request)?; + if !matches!(request.request, Request::OpenMediation) { + return Err("persistent mediation stream omitted OpenMediation".to_string()); + } + let mut response = runtime.dispatch(request); + let lease = if matches!(response, Response::MediationReady) { + tokio::time::timeout( + MEDIATION_REPLACEMENT_TIMEOUT, + runtime.mediation_active.lock(), + ) + .await + .map_or_else( + |_| { + response = guest_error( + BoundaryErrorKind::Denied, + "a mediation session is already active", + ); + None + }, + Some, + ) + } else { + None + }; + let response_frame = encode_frame(&ResponseEnvelope { + request_id, + response, + }) + .map_err(|error| format!("encode mediation attach response: {error}"))?; + stream + .write_all(&response_frame) + .await + .map_err(|error| format!("write mediation attach response: {error}"))?; + stream + .flush() + .await + .map_err(|error| format!("flush mediation attach response: {error}"))?; + let Some(_lease) = lease else { + return Ok(()); + }; + let broker = runtime.network_accept_context()?; + run_boundary_mediation(stream, runtime.clone(), broker).await + } + + async fn run_boundary_mediation( + stream: tokio::io::DuplexStream, + runtime: Arc, + broker: NetworkBroker, + ) -> Result<(), String> { + let (mut reader, mut writer) = tokio::io::split(stream); + let (outbound_tx, mut outbound_rx) = + tokio::sync::mpsc::channel::(MEDIATION_EVENT_QUEUE); + let routes: BoundaryMediationRoutes = + Arc::new(tokio::sync::Mutex::new(std::collections::HashMap::new())); + let writer_task = async { + while let Some(frame) = outbound_rx.recv().await { + mediation::write_frame(&mut writer, frame.kind, frame.stream_id, &frame.payload) + .await + .map_err(|error| format!("write persistent mediation frame: {error}"))?; + } + Ok::<(), String>(()) + }; + let reader_routes = routes.clone(); + let reader_task = async { + while let Some(frame) = mediation::read_frame(&mut reader) + .await + .map_err(|error| format!("read persistent mediation frame: {error}"))? + { + let route = reader_routes.lock().await.get(&frame.stream_id).cloned(); + if let Some(route) = route { + let _ = route.send(frame).await; + } + } + Ok::<(), String>(()) + }; + let accept_task = + run_boundary_accepts(runtime, broker, outbound_tx.clone(), routes.clone()); + tokio::pin!(writer_task); + tokio::pin!(reader_task); + tokio::pin!(accept_task); + let result = tokio::select! { + result = &mut writer_task => result, + result = &mut reader_task => result, + result = &mut accept_task => result, + }; + routes.lock().await.clear(); + result + } + + async fn run_boundary_accepts( + runtime: Arc, + broker: NetworkBroker, + outbound: tokio::sync::mpsc::Sender, + routes: BoundaryMediationRoutes, + ) -> Result<(), String> { + loop { + let pending = broker + .accept_dns() + .await + .map_err(|error| format!("accept sandbox DNS query: {error}"))?; + let stream_id = runtime + .next_mediation_stream_id + .fetch_add(1, Ordering::Relaxed); + let (route_tx, route_rx) = tokio::sync::mpsc::channel(MEDIATION_ROUTE_QUEUE); + routes.lock().await.insert(stream_id, route_tx); + tokio::spawn(run_boundary_dns_stream( + stream_id, + pending, + route_rx, + outbound.clone(), + routes.clone(), + )); + } + } + + async fn run_boundary_dns_stream( + stream_id: u64, + pending: crate::network_broker::PendingDnsQuery, + mut inbound: tokio::sync::mpsc::Receiver, + outbound: tokio::sync::mpsc::Sender, + routes: BoundaryMediationRoutes, + ) { + let query = DnsQueryWire { + request: pending.request.clone(), + transport: pending.transport, + identity: BinaryIdentityWire::from(pending.identity.clone()), + timing: MediationTimingWire { + notification_to_queue_us: duration_micros(pending.notification_to_queue), + queue_wait_us: duration_micros(pending.queued_at.elapsed()), + }, + }; + let Ok(payload) = mediation::encode_json(&query) else { + routes.lock().await.remove(&stream_id); + return; + }; + if outbound + .send(BoundaryOutboundFrame { + kind: MediationFrameKind::DnsQuery, + stream_id, + payload, + }) + .await + .is_err() + { + routes.lock().await.remove(&stream_id); + return; + } + let result = match inbound.recv().await { + Some(MediationFrame { + kind: MediationFrameKind::DnsResponse, + payload, + .. + }) => mediation::decode_json::(&payload) + .map_err(io::Error::other) + .and_then(|response| match response { + DnsQueryResultWire::Response(response) => Ok(response), + DnsQueryResultWire::Error(error) => Err(io::Error::other(error)), + }), + _ => Err(io::Error::new( + io::ErrorKind::BrokenPipe, + "mediation session ended before DNS response", + )), + }; + let _ = pending.complete(result); + routes.lock().await.remove(&stream_id); + } + + fn serve_one( + mut stream: ControlStream, + runtime: &BoundaryRuntime, + principal: &SandboxProtocolPrincipal, + ) -> Result<(), String> { + stream + .set_timeout(CONTROL_IO_TIMEOUT) + .map_err(|error| format!("set control timeout: {error}"))?; + let request: RequestEnvelope = + read_frame(&mut stream).map_err(|error| format!("read control frame: {error}"))?; + runtime.authorize_request(principal, &request.request)?; + if request.validate_payload_digest().is_err() { + let response = ResponseEnvelope { + request_id: request.request_id, + response: guest_error( + BoundaryErrorKind::Denied, + "control request payload digest mismatch", + ), + }; + return write_frame(&mut stream, &response) + .map_err(|error| format!("write control frame: {error}")); + } + match request.request.clone() { + Request::TerminateBoundary => { + let response = runtime + .process_runtime + .block_on(runtime.terminate_boundary()); + return write_frame( + &mut stream, + &ResponseEnvelope { + request_id: request.request_id, + response, + }, + ) + .map_err(|error| format!("write boundary termination response: {error}")); + } + Request::Exec { spec } => { + let started = + match runtime.start_exec(&request.request_id, &request.payload_digest, spec) { + Ok(started) => started, + Err(response) => { + return write_frame( + &mut stream, + &ResponseEnvelope { + request_id: request.request_id, + response, + }, + ) + .map_err(|error| format!("write exec error response: {error}")); + } + }; + if let Err(error) = write_frame( + &mut stream, + &ResponseEnvelope { + request_id: request.request_id, + response: Response::ExecStarted { + process_id: started.process_id.clone(), + pty: started.terminal, + }, + }, + ) { + return Err(format!("write exec start response: {error}")); + } + return runtime.stream_process(stream, started.attachment); + } + Request::AttachProcess { process_id } => { + let (attachment, terminal) = match runtime.attach_process(&process_id) { + Ok(attachment) => attachment, + Err(response) => { + return write_frame( + &mut stream, + &ResponseEnvelope { + request_id: request.request_id, + response, + }, + ) + .map_err(|error| format!("write process attachment error: {error}")); + } + }; + write_frame( + &mut stream, + &ResponseEnvelope { + request_id: request.request_id, + response: Response::ProcessAttached { terminal }, + }, + ) + .map_err(|error| format!("write process attachment response: {error}"))?; + return runtime.stream_process(stream, attachment); + } + Request::LoopbackConnect { host, port } => { + let target = match LoopbackTarget::new(host, port) + .map_err(|error| format!("validate port-forward target: {error}")) + .and_then(|target| { + runtime + .connect_port(target) + .map_err(|error| format!("connect boundary loopback port: {error}")) + }) { + Ok(target) => target, + Err(error) => { + write_frame( + &mut stream, + &ResponseEnvelope { + request_id: request.request_id, + response: guest_error(BoundaryErrorKind::Process, error), + }, + ) + .map_err(|error| format!("write port-forward error response: {error}"))?; + return Ok(()); + } + }; + let mut target = target; + write_frame( + &mut stream, + &ResponseEnvelope { + request_id: request.request_id, + response: Response::PortConnected, + }, + ) + .map_err(|error| format!("write port-forward response: {error}"))?; + runtime.process_runtime.block_on(async move { + let mut stream = stream.into_tokio()?; + tokio::io::copy_bidirectional(&mut stream, &mut target) + .await + .map_err(|error| format!("bridge boundary loopback stream: {error}")) + })?; + return Ok(()); + } + Request::AcceptNetwork => { + let broker = runtime.network_accept_context()?; + let request_id = request.request_id; + runtime.process_runtime.block_on(async move { + let mut stream = stream.into_tokio()?; + let mut disconnect_probe = [0_u8; 1]; + let pending = tokio::select! { + biased; + read = stream.read(&mut disconnect_probe) => { + match read { + Ok(0) => return Ok(()), + Ok(_) => return Err("control sent data before network mediation response".to_string()), + Err(error) => return Err(format!("watch network mediation control stream: {error}")), + } + } + pending = broker.accept() => pending + .map_err(|error| format!("accept sandbox network open: {error}"))?, + }; + let response = encode_frame(&ResponseEnvelope { + request_id, + response: Response::NetworkConnected { + identity: BinaryIdentityWire::from(pending.identity.clone()), + destination: pending.destination, + socket: pending.socket, + policy_generation: 0, + timing: MediationTimingWire { + notification_to_queue_us: duration_micros( + pending.notification_to_queue, + ), + queue_wait_us: duration_micros(pending.queued_at.elapsed()), + }, + }, + }) + .map_err(|error| format!("encode network mediation response: {error}"))?; + stream + .write_all(&response) + .await + .map_err(|error| format!("write network mediation response: {error}"))?; + let Some((channel, payload)) = read_stream_frame(&mut stream) + .await + .map_err(|error| format!("read network-open decision: {error}"))? + else { + return Err("control disconnected before network-open decision".to_string()); + }; + if channel != STREAM_NETWORK_DECISION { + return Err(format!( + "unexpected network-open decision channel {channel}" + )); + } + let decision = serde_json::from_slice(&payload) + .map_err(|error| format!("decode network-open decision: {error}"))?; + let Some(target) = pending + .complete(decision) + .await + .map_err(|error| format!("complete sandbox network open: {error}"))? + else { + return Ok(()); + }; + target + .set_nonblocking(true) + .map_err(|error| format!("set sandbox relay nonblocking: {error}"))?; + let mut target = tokio::net::TcpStream::from_std(target) + .map_err(|error| format!("register sandbox relay: {error}"))?; + openshell_core::net::set_tcp_nodelay_best_effort(&target); + tokio::io::copy_bidirectional(&mut stream, &mut target) + .await + .map(|_| ()) + .map_err(|error| format!("bridge sandbox network stream: {error}")) + })?; + return Ok(()); + } + _ => {} + } + let supervisor_instance_id = match &request.request { + Request::Attach { + supervisor_instance_id, + .. + } => Some(*supervisor_instance_id), + _ => None, + }; + let is_attach = supervisor_instance_id.is_some(); + let is_confirm = matches!(&request.request, Request::Confirm); + let response = ResponseEnvelope { + request_id: request.request_id.clone(), + response: runtime.dispatch(request), + }; + if is_attach && matches!(&response.response, Response::Attached { .. }) { + let supervisor_instance_id = supervisor_instance_id + .ok_or_else(|| "attach request lost supervisor instance identity".to_string())?; + runtime.commit_attach(principal, supervisor_instance_id)?; + } + if is_confirm && matches!(&response.response, Response::Confirmed { .. }) { + runtime.commit_confirm(principal)?; + } + write_frame(&mut stream, &response) + .map_err(|error| format!("write control frame: {error}"))?; + Ok(()) + } + + struct BoundaryRuntime { + config: BoundaryConfig, + authenticator: SandboxProtocolAuthenticator, + connections: SandboxConnectionRegistry, + connection_shutdowns: + Mutex>>, + process_runtime: tokio::runtime::Handle, + state: Mutex, + supervisor_connection: Mutex, + next_recovery_id: AtomicU64, + /// The wire policy bound at first attach, so an idempotent attach retry + /// carrying a different policy is denied instead of silently keeping + /// the first policy. + attached_policy: Mutex>, + /// The complete launch request accepted by the boundary. A replacement + /// control process may replay it after reconnecting, but may not change + /// any launch input or start a second workload. + started_agent: Mutex>, + next_exec_id: AtomicU64, + mediation_active: tokio::sync::Mutex<()>, + next_mediation_stream_id: AtomicU64, + exec_handles: Mutex>, + /// Never evicted within a boundary generation. Reclaiming process I/O + /// must not make an old command executable again. At capacity, reject + /// new commands instead of silently weakening at-most-once execution. + exec_requests: Mutex>, + replay_ledger: Mutex, + network_broker: NetworkBroker, + workload_launcher: + openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, + qualification: crate::RuntimeQualification, + } + + #[derive(Clone)] + struct ExecHandle { + request_id: String, + payload_digest: String, + process: Arc, + terminal: Option>, + session: Arc, + attached: Arc, + status: Arc>>, + } + + #[allow(clippy::result_large_err)] + fn reserve_exec_request( + requests: &mut std::collections::HashSet, + request_id: &str, + ) -> Result<(), Response> { + if requests.contains(request_id) { + return Err(guest_error( + BoundaryErrorKind::Denied, + "exec request has expired; it cannot be executed again", + )); + } + if requests.len() >= MAX_REPLAY_LEDGER_ENTRIES { + return Err(guest_error( + BoundaryErrorKind::Unavailable, + "boundary generation exec request limit reached", + )); + } + requests.insert(request_id.to_owned()); + Ok(()) + } + + struct StartedExec { + process_id: String, + terminal: bool, + attachment: MainAttachment, + } + + #[derive(Clone)] + struct ReplayRecord { + payload_digest: String, + response: Response, + } + + #[derive(Default)] + struct ReplayLedger { + entries: std::collections::HashMap, + order: std::collections::VecDeque, + } + + impl ReplayLedger { + fn get(&self, request_id: &str) -> Option<&ReplayRecord> { + self.entries.get(request_id) + } + + fn insert(&mut self, request_id: String, record: ReplayRecord) { + if let Some(existing) = self.entries.get_mut(&request_id) { + *existing = record; + return; + } + while self.entries.len() >= MAX_REPLAY_LEDGER_ENTRIES { + let Some(oldest) = self.order.pop_front() else { + break; + }; + self.entries.remove(&oldest); + } + self.order.push_back(request_id.clone()); + self.entries.insert(request_id, record); + } + } + + #[derive(Clone, PartialEq, Eq)] + struct StartedAgent { + sandbox_id: String, + spec: AgentSpecWire, + policy: SandboxPolicyWire, + ca_cert: Option>, + ca_bundle: Option>, + provider_env_revision: u64, + provider_env: std::collections::HashMap, + } + + impl StartedAgent { + /// Provider environment is mutable runtime state. A replacement + /// control must replay every immutable launch input exactly, then + /// reconcile the current provider snapshot through the CAS update. + fn matches_replay(&self, other: &Self) -> bool { + self.sandbox_id == other.sandbox_id + && self.spec == other.spec + && self.policy == other.policy + && self.ca_cert == other.ca_cert + && self.ca_bundle == other.ca_bundle + } + } + + struct MainAttachment { + session: Arc, + attached: Arc, + status: AttachmentStatus, + } + + enum AttachmentStatus { + Main(Arc), + Exec(Arc>>), + } + + impl MainAttachment { + fn exit_status(&self, fallback_code: i32) -> ExitStatusWire { + match &self.status { + AttachmentStatus::Main(process) => process + .exit_status() + .unwrap_or(ExitStatusWire::Exited(fallback_code)), + AttachmentStatus::Exec(status) => { + (*lock(status)).unwrap_or(ExitStatusWire::Exited(fallback_code)) + } + } + } + } + + impl Drop for MainAttachment { + fn drop(&mut self) { + self.attached.store(false, Ordering::Release); + } + } + + #[allow(clippy::result_large_err)] + fn acquire_exec_attachment(handle: &ExecHandle) -> Result { + acquire_attachment( + handle.session.clone(), + handle.attached.clone(), + AttachmentStatus::Exec(handle.status.clone()), + ) + } + + #[allow(clippy::result_large_err)] + fn acquire_attachment( + session: Arc, + attached: Arc, + status: AttachmentStatus, + ) -> Result { + if attached + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + return Err(guest_error( + BoundaryErrorKind::Denied, + "process already has a control attachment", + )); + } + Ok(MainAttachment { + session, + attached, + status, + }) + } + + enum RuntimeState { + AwaitingAttach, + Bound(PreparedBoundary), + Ready(PreparedBoundary), + Running(Arc), + } + + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum SupervisorConnectionState { + AwaitingConfirmation, + Connected(SandboxConnectionId), + Frozen { recovery_id: u64 }, + Terminating, + Terminal, + } + + #[derive(Clone)] + struct PreparedBoundary { + network_broker: NetworkBroker, + } + + impl BoundaryRuntime { + fn new( + config: BoundaryConfig, + process_runtime: tokio::runtime::Handle, + network_broker: NetworkBroker, + workload_launcher: openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, + qualification: crate::RuntimeQualification, + ) -> Result { + let sandbox_id = SandboxId::parse(config.boundary_id.clone()) + .map_err(|error| format!("validate sandbox ID: {error}"))?; + let runtime_generation = + openshell_core::sandbox_generation::SandboxGenerationId::parse( + config.generation.clone(), + ) + .map_err(|error| format!("validate sandbox runtime generation: {error}"))?; + let verifier = SessionJwtVerifier::new( + &config.gateway_id, + SessionTokenProfile::Sandbox, + config + .verification_keys + .iter() + .map(|key| SessionVerificationKey { + key_id: key.key_id.clone(), + public_key_pem: key.public_key_pem.as_bytes().to_vec(), + }), + Arc::new(SystemJwtClock), + ) + .map_err(|error| format!("configure Sandbox Protocol JWT verifier: {error}"))?; + Ok(Self { + authenticator: SandboxProtocolAuthenticator::new( + verifier, + sandbox_id, + runtime_generation, + config.auth_epoch, + ), + connections: SandboxConnectionRegistry::new( + config.session_id, + config.session_rotation, + ), + connection_shutdowns: Mutex::new(std::collections::HashMap::new()), + config, + process_runtime, + state: Mutex::new(RuntimeState::AwaitingAttach), + supervisor_connection: Mutex::new(SupervisorConnectionState::AwaitingConfirmation), + next_recovery_id: AtomicU64::new(1), + attached_policy: Mutex::new(None), + started_agent: Mutex::new(None), + next_exec_id: AtomicU64::new(1), + mediation_active: tokio::sync::Mutex::new(()), + next_mediation_stream_id: AtomicU64::new(1), + exec_handles: Mutex::new(std::collections::HashMap::new()), + exec_requests: Mutex::new(std::collections::HashSet::new()), + replay_ledger: Mutex::new(ReplayLedger::default()), + network_broker, + workload_launcher, + qualification, + }) + } + + fn authenticate_request( + &self, + connection_id: SandboxConnectionId, + metadata: &tonic::metadata::MetadataMap, + ) -> Result { + self.authenticator + .authenticate(connection_id, metadata) + .map_err(|error| tonic::Status::unauthenticated(error.to_string())) + } + + fn authorize_request( + &self, + principal: &SandboxProtocolPrincipal, + request: &Request, + ) -> Result<(), String> { + if matches!(request, Request::Attach { .. }) { + return Ok(()); + } + if matches!(request, Request::Confirm) { + self.connections + .require_attached(principal) + .map_err(|error| error.to_string()) + } else { + self.connections + .require_active(principal) + .map_err(|error| error.to_string()) + } + } + + fn commit_attach( + &self, + principal: &SandboxProtocolPrincipal, + supervisor_instance_id: openshell_sandbox_backend::boundary_protocol::SupervisorInstanceId, + ) -> Result<(), String> { + if let Some(replaced) = self + .connections + .attach(principal, supervisor_instance_id) + .map_err(|error| error.to_string())? + { + self.close_connection(replaced); + } + Ok(()) + } + + fn commit_confirm(&self, principal: &SandboxProtocolPrincipal) -> Result<(), String> { + let replaced = self + .connections + .confirm(principal) + .map_err(|error| error.to_string())?; + let process = { + let state = lock(&self.state); + match &*state { + RuntimeState::Running(process) => Some(process.clone()), + RuntimeState::AwaitingAttach + | RuntimeState::Bound(_) + | RuntimeState::Ready(_) => None, + } + }; + { + let mut connection = lock(&self.supervisor_connection); + if matches!( + *connection, + SupervisorConnectionState::Terminating | SupervisorConnectionState::Terminal + ) { + self.connections.mark_terminal(); + return Err("sandbox session is terminating".to_string()); + } + if matches!(*connection, SupervisorConnectionState::Frozen { .. }) + && let Some(process) = process + { + if !process.boundary_runtime.resume() { + return Err("frozen workload could not be resumed".to_string()); + } + tracing::info!( + connection_id = ?principal.connection_id(), + "Sandbox Protocol connection recovered; workload resumed" + ); + } + *connection = SupervisorConnectionState::Connected(principal.connection_id()); + } + if let Some(replaced) = replaced { + self.close_connection(replaced); + } + Ok(()) + } + + fn register_connection( + &self, + connection_id: SandboxConnectionId, + shutdown: tokio::sync::watch::Sender<()>, + ) { + lock(&self.connection_shutdowns).insert(connection_id, shutdown); + } + + fn close_connection(&self, connection_id: SandboxConnectionId) { + let shutdown = lock(&self.connection_shutdowns).remove(&connection_id); + if let Some(shutdown) = shutdown { + let _ = shutdown.send(()); + } + } + + fn transport_disconnected(self: &Arc, connection_id: SandboxConnectionId) { + let shutdown = lock(&self.connection_shutdowns).remove(&connection_id); + if let Some(shutdown) = shutdown { + let _ = shutdown.send(()); + } + if !self.connections.disconnect(connection_id) { + return; + } + + let recovery_id = self.next_recovery_id.fetch_add(1, Ordering::Relaxed); + let process = { + let state = lock(&self.state); + match &*state { + RuntimeState::Running(process) => Some(process.clone()), + RuntimeState::AwaitingAttach + | RuntimeState::Bound(_) + | RuntimeState::Ready(_) => None, + } + }; + { + let mut connection = lock(&self.supervisor_connection); + if !matches!( + *connection, + SupervisorConnectionState::Connected(active) if active == connection_id + ) { + return; + } + if let Some(process) = &process { + let _ = process.boundary_runtime.freeze(); + } + *connection = SupervisorConnectionState::Frozen { recovery_id }; + } + tracing::warn!( + recovery_id, + "Sandbox Protocol connection lost; workload frozen pending authenticated recovery" + ); + openshell_ocsf::ocsf_emit!( + openshell_ocsf::DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(openshell_ocsf::ActivityId::Open) + .severity(openshell_ocsf::SeverityId::Medium) + .confidence(openshell_ocsf::ConfidenceId::High) + .is_alert(true) + .finding_info(openshell_ocsf::FindingInfo::new( + "sandbox-supervisor-connection-lost", + "Sandbox Supervisor Connection Lost", + )) + .message("Sandbox Protocol connection lost; workload frozen") + .build() + ); + let runtime = Arc::downgrade(self); + self.process_runtime.spawn(async move { + tokio::time::sleep(AUTHENTICATED_RECONNECT_TIMEOUT).await; + if let Some(runtime) = runtime.upgrade() { + runtime.expire_recovery(recovery_id).await; + } + }); + } + + async fn expire_recovery(&self, recovery_id: u64) { + let process = { + let mut connection = lock(&self.supervisor_connection); + if *connection != (SupervisorConnectionState::Frozen { recovery_id }) { + return; + } + *connection = SupervisorConnectionState::Terminating; + let state = lock(&self.state); + match &*state { + RuntimeState::Running(process) => Some(process.clone()), + RuntimeState::AwaitingAttach + | RuntimeState::Bound(_) + | RuntimeState::Ready(_) => None, + } + }; + self.connections.mark_terminal(); + tracing::error!( + recovery_id, + "Sandbox Protocol recovery deadline expired; terminating workload" + ); + openshell_ocsf::ocsf_emit!( + openshell_ocsf::DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(openshell_ocsf::ActivityId::Open) + .severity(openshell_ocsf::SeverityId::High) + .confidence(openshell_ocsf::ConfidenceId::High) + .is_alert(true) + .finding_info(openshell_ocsf::FindingInfo::new( + "sandbox-supervisor-recovery-expired", + "Sandbox Supervisor Recovery Expired", + )) + .message("Supervisor recovery expired; terminating sandbox workload") + .build() + ); + if let Some(process) = process + && let Err(error) = Self::terminate_process_tree(&process, true).await + { + tracing::error!(%error, "sandbox workload did not terminate after recovery loss"); + return; + } + *lock(&self.supervisor_connection) = SupervisorConnectionState::Terminal; + } + + async fn terminate_boundary(&self) -> Response { + { + let mut connection = lock(&self.supervisor_connection); + if *connection == SupervisorConnectionState::Terminal { + return Response::BoundaryTerminated; + } + *connection = SupervisorConnectionState::Terminating; + } + // Revocation happens before process shutdown so no concurrent or + // replacement connection can race the terminal transition. + self.connections.mark_terminal(); + let process = { + let state = lock(&self.state); + match &*state { + RuntimeState::Running(process) => Some(process.clone()), + RuntimeState::AwaitingAttach + | RuntimeState::Bound(_) + | RuntimeState::Ready(_) => None, + } + }; + if let Some(process) = process + && let Err(error) = Self::terminate_process_tree(&process, false).await + { + return guest_error(BoundaryErrorKind::Process, error); + } + *lock(&self.supervisor_connection) = SupervisorConnectionState::Terminal; + Response::BoundaryTerminated + } + + async fn terminate_process_tree( + process: &ManagedProcess, + enforcement_was_lost: bool, + ) -> Result<(), String> { + if enforcement_was_lost { + let _ = process + .boundary_runtime + .begin_enforcement_loss_termination(); + } else { + let _ = process.boundary_runtime.begin_termination(); + } + if Self::wait_for_process_tree_exit(process, ENFORCEMENT_LOSS_TERMINATION_GRACE).await { + return Ok(()); + } + + process.boundary_runtime.force_kill(); + if Self::wait_for_process_tree_exit(process, FORCE_KILL_REAP_TIMEOUT).await { + Ok(()) + } else { + Err("owned workload processes remain after forced termination".to_string()) + } + } + + async fn wait_for_process_tree_exit(process: &ManagedProcess, timeout: Duration) -> bool { + let deadline = tokio::time::Instant::now() + timeout; + while process.boundary_runtime.has_registered_processes() + && tokio::time::Instant::now() < deadline + { + tokio::time::sleep(Duration::from_millis(25)).await; + } + !process.boundary_runtime.has_registered_processes() + } + + fn shutdown(&self) { + let process = { + let state = lock(&self.state); + match &*state { + RuntimeState::Running(process) => Some(process.clone()), + RuntimeState::AwaitingAttach + | RuntimeState::Bound(_) + | RuntimeState::Ready(_) => None, + } + }; + if let Some(process) = process { + process.boundary_runtime.deactivate(); + } + } + + fn dispatch(&self, envelope: RequestEnvelope) -> Response { + if envelope.validate_payload_digest().is_err() { + return guest_error( + BoundaryErrorKind::Denied, + "control request payload digest mismatch", + ); + } + let replayable = envelope.request.is_replayable_mutation(); + let mut replay_ledger = replayable.then(|| lock(&self.replay_ledger)); + if let Some(record) = replay_ledger + .as_ref() + .and_then(|ledger| ledger.get(&envelope.request_id)) + { + return if record.payload_digest == envelope.payload_digest { + record.response.clone() + } else { + guest_error( + BoundaryErrorKind::Denied, + "control request ID was reused with a different payload", + ) + }; + } + let request_id = envelope.request_id; + let payload_digest = envelope.payload_digest; + let response = match envelope.request { + Request::Attach { + supervisor_instance_id: _, + policy, + resource_claims, + } => { + if resource_claims == self.config.resource_claims { + self.attach(*policy) + } else { + guest_error( + BoundaryErrorKind::Denied, + "backend resource claims do not match the boundary configuration", + ) + } + } + Request::Confirm => self.confirm(), + Request::StartAgent { + sandbox_id, + spec, + policy, + ca_cert, + ca_bundle, + provider_env_revision, + provider_env, + } => self.start_agent( + sandbox_id, + spec, + *policy, + ca_cert, + ca_bundle, + provider_env_revision, + provider_env, + ), + Request::UpdateProviderEnvironment { + expected_revision, + revision, + provider_env, + } => self.update_provider_environment(expected_revision, revision, provider_env), + Request::Wait { process_id } => self.wait(&process_id), + Request::Signal { process_id, signal } => self.signal(&process_id, signal), + Request::Terminate { process_id } => self.terminate(&process_id), + Request::ExecSignal { process_id, signal } => self.signal_exec(&process_id, signal), + Request::Resize { + process_id, + cols, + rows, + } => self.resize_process(&process_id, cols, rows), + Request::OpenMediation => self.network_accept_context().map_or_else( + |error| guest_error(BoundaryErrorKind::Unavailable, error), + |_| Response::MediationReady, + ), + Request::Exec { .. } + | Request::TerminateBoundary + | Request::AttachProcess { .. } + | Request::LoopbackConnect { .. } + | Request::AcceptNetwork => guest_error( + BoundaryErrorKind::Invalid, + "streaming request used on control path", + ), + }; + if let Some(ledger) = replay_ledger.as_mut() { + ledger.insert( + request_id, + ReplayRecord { + payload_digest, + response: response.clone(), + }, + ); + } + response + } + + #[allow( + clippy::result_large_err, + reason = "protocol errors are returned directly as complete response frames" + )] + fn start_exec( + &self, + request_id: &str, + payload_digest: &str, + spec: ExecSpecWire, + ) -> Result { + let executor = { + let state = lock(&self.state); + let RuntimeState::Running(process) = &*state else { + return Err(guest_error( + BoundaryErrorKind::Invalid, + "agent process has not been started", + )); + }; + process.boundary_exec() + }; + let mut handles = lock(&self.exec_handles); + if let Some((process_id, handle)) = handles + .iter() + .find(|(_, handle)| handle.request_id == request_id) + { + if handle.payload_digest != payload_digest { + return Err(guest_error( + BoundaryErrorKind::Denied, + "exec request ID was reused with a different payload", + )); + } + if handle.attached.load(Ordering::Acquire) { + return Err(guest_error( + BoundaryErrorKind::Unavailable, + "prior exec attachment is still being released", + )); + } + return Ok(StartedExec { + process_id: process_id.clone(), + terminal: handle.terminal.is_some(), + attachment: acquire_exec_attachment(handle)?, + }); + } + if handles.len() >= MAX_RETAINED_EXEC_PROCESSES { + let exited = handles + .iter() + .find(|(_, handle)| { + lock(&handle.status).is_some() && !handle.attached.load(Ordering::Acquire) + }) + .map(|(process_id, _)| process_id.clone()); + if let Some(process_id) = exited { + handles.remove(&process_id); + } else { + return Err(guest_error( + BoundaryErrorKind::Unavailable, + "retained exec process limit reached", + )); + } + } + { + let mut requests = lock(&self.exec_requests); + reserve_exec_request(&mut requests, request_id)?; + } + let session = self + .process_runtime + .block_on(executor.exec(spec.into())) + .map_err(|error| guest_error(BoundaryErrorKind::Process, error.to_string()))?; + let process_id = format!( + "{}:exec:{}", + self.config.generation, + self.next_exec_id.fetch_add(1, Ordering::Relaxed) + ); + let ExecSession { + process, + stdin, + stdout, + stderr, + terminal, + } = session; + let Some(stdin) = stdin else { + return Err(guest_error( + BoundaryErrorKind::Process, + "exec process stdin pipe is unavailable", + )); + }; + let retained = { + let _runtime = self.process_runtime.enter(); + MainSession::from_boundary( + openshell_isolation_interface::contract::ProcessAttachment { + stdin, + stdout, + stderr, + terminal: terminal.clone(), + }, + process.clone(), + ) + }; + let status = Arc::new(Mutex::new(None)); + let wait_process = process.clone(); + let wait_session = retained.clone(); + let wait_status = status.clone(); + self.process_runtime.spawn(async move { + if let Ok(exit_status) = wait_process.wait().await { + *lock(&wait_status) = Some(ExitStatusWire::from(exit_status)); + let exit_code = match exit_status { + openshell_isolation_interface::contract::BoundaryExitStatus::Exited( + code, + ) => code, + openshell_isolation_interface::contract::BoundaryExitStatus::Signaled( + signal, + ) => 128 + signal, + }; + let _ = wait_session.finish_remote(exit_code, false).await; + } + }); + let handle = ExecHandle { + request_id: request_id.to_string(), + payload_digest: payload_digest.to_string(), + process, + terminal, + session: retained, + attached: Arc::new(AtomicBool::new(false)), + status, + }; + let terminal = handle.terminal.is_some(); + let attachment = acquire_exec_attachment(&handle)?; + handles.insert(process_id.clone(), handle); + Ok(StartedExec { + process_id, + terminal, + attachment, + }) + } + + fn signal_exec(&self, process_id: &str, signal: SignalWire) -> Response { + let process = lock(&self.exec_handles) + .get(process_id) + .map(|handle| handle.process.clone()); + let Some(process) = process else { + return guest_error(BoundaryErrorKind::Invalid, "unknown exec process ID"); + }; + match self.process_runtime.block_on(process.signal(signal.into())) { + Ok(()) => Response::Signaled, + Err(error) => guest_error(BoundaryErrorKind::Process, error.to_string()), + } + } + + fn resize_process(&self, process_id: &str, cols: u16, rows: u16) -> Response { + if let Ok(process) = self.running_process(process_id) { + let session = process.main_session(); + if !session.terminal() { + return guest_error( + BoundaryErrorKind::Invalid, + "agent process has no terminal", + ); + } + self.process_runtime.block_on(session.resize( + u32::from(cols), + u32::from(rows), + 0, + 0, + )); + return Response::Resized; + } + let terminal = lock(&self.exec_handles) + .get(process_id) + .and_then(|handle| handle.terminal.clone()); + let Some(terminal) = terminal else { + return guest_error(BoundaryErrorKind::Invalid, "exec process has no terminal"); + }; + match self.process_runtime.block_on(terminal.resize(cols, rows)) { + Ok(()) => Response::Resized, + Err(error) => guest_error(BoundaryErrorKind::Process, error.to_string()), + } + } + + fn connect_port( + &self, + target: LoopbackTarget, + ) -> Result { + let loopback_connector = { + let state = lock(&self.state); + let RuntimeState::Running(process) = &*state else { + return Err("agent process has not been started".to_string()); + }; + process.loopback_connector() + }; + self.process_runtime + .block_on(loopback_connector.connect(target)) + .map_err(|error| error.to_string()) + } + + fn network_accept_context(&self) -> Result { + self.network_broker + .confirm_healthy() + .map_err(|error| format!("sandbox network broker unavailable: {error}"))?; + Ok(self.network_broker.clone()) + } + + #[allow( + clippy::result_large_err, + reason = "protocol errors are returned directly as complete response frames" + )] + fn attach_process(&self, process_id: &str) -> Result<(MainAttachment, bool), Response> { + if let Ok(process) = self.running_process(process_id) { + let session = process.main_session(); + let terminal = session.terminal(); + let attachment = acquire_attachment( + session, + process.attached.clone(), + AttachmentStatus::Main(process), + )?; + return Ok((attachment, terminal)); + } + let handles = lock(&self.exec_handles); + let handle = handles + .get(process_id) + .ok_or_else(|| guest_error(BoundaryErrorKind::Invalid, "unknown process ID"))?; + Ok((acquire_exec_attachment(handle)?, handle.terminal.is_some())) + } + + fn stream_process( + &self, + stream: ControlStream, + attachment: MainAttachment, + ) -> Result<(), String> { + self.process_runtime.block_on(async move { + let stream = stream.into_tokio()?; + bridge_main_stream(stream, attachment).await + }) + } + + fn attach(&self, policy: SandboxPolicyWire) -> Response { + let mut state = lock(&self.state); + let accepted = match &*state { + RuntimeState::AwaitingAttach => { + let prepared = match PreparedBoundary::establish(self.network_broker.clone()) { + Ok(prepared) => prepared, + Err(error) => return guest_error(BoundaryErrorKind::Process, error), + }; + *lock(&self.attached_policy) = Some(policy); + *state = RuntimeState::Bound(prepared); + true + } + RuntimeState::Bound(_) | RuntimeState::Ready(_) | RuntimeState::Running(_) => { + // Idempotent retry of the same attach; a different policy + // must not be silently coalesced onto the bound boundary. + lock(&self.attached_policy).as_ref() == Some(&policy) + } + }; + drop(state); + if accepted { + Response::Attached { + snapshot: self.session_snapshot(), + } + } else { + guest_error( + BoundaryErrorKind::Denied, + "attach policy does not match the bound boundary", + ) + } + } + + fn session_snapshot(&self) -> SessionSnapshotWire { + let process = { + let state = lock(&self.state); + match &*state { + RuntimeState::Running(process) => Some(process.clone()), + RuntimeState::AwaitingAttach + | RuntimeState::Bound(_) + | RuntimeState::Ready(_) => None, + } + }; + let mut processes = process + .into_iter() + .map(|process| { + let (first_sequence, next_sequence, truncated) = + process.main_session().output_window(); + ProcessSnapshotWire { + process_id: process.process_id(), + kind: ProcessKindWire::Main, + terminal: process.main_session().terminal(), + status: process.exit_status(), + retained_output: OutputWindowWire { + first_sequence, + next_sequence, + truncated, + }, + } + }) + .collect::>(); + processes.extend(lock(&self.exec_handles).iter().map(|(process_id, handle)| { + let (first_sequence, next_sequence, truncated) = handle.session.output_window(); + ProcessSnapshotWire { + process_id: process_id.clone(), + kind: ProcessKindWire::Exec, + terminal: handle.terminal.is_some(), + status: *lock(&handle.status), + retained_output: OutputWindowWire { + first_sequence, + next_sequence, + truncated, + }, + } + })); + processes.sort_by(|left, right| left.process_id.cmp(&right.process_id)); + SessionSnapshotWire { + generation: self.config.generation.clone(), + processes, + } + } + + fn confirm(&self) -> Response { + let mut state = lock(&self.state); + match &*state { + RuntimeState::Bound(prepared) => { + if let Err(error) = prepared.confirm(&self.process_runtime) { + return guest_error(BoundaryErrorKind::Process, error); + } + let evidence = match self.measure_confirmation_evidence() { + Ok(evidence) => evidence, + Err(error) => return guest_error(BoundaryErrorKind::Process, error), + }; + *state = RuntimeState::Ready(prepared.clone()); + Response::Confirmed { + evidence: Box::new(evidence), + } + } + RuntimeState::Ready(_) | RuntimeState::Running(_) => { + self.measure_confirmation_evidence().map_or_else( + |error| guest_error(BoundaryErrorKind::Process, error), + |evidence| Response::Confirmed { + evidence: Box::new(evidence), + }, + ) + } + RuntimeState::AwaitingAttach => guest_error( + BoundaryErrorKind::Invalid, + "boundary must be attached before confirm", + ), + } + } + + fn measure_confirmation_evidence(&self) -> Result { + validate_running_identity(&self.config.workload_identity)?; + self.network_broker + .confirm_healthy() + .map_err(|error| format!("verify sandbox network broker: {error}"))?; + if !self.workload_launcher.is_alive() { + return Err("sandbox workload launcher is not running".to_string()); + } + let status = std::fs::read_to_string("/proc/self/status") + .map_err(|error| format!("read sandbox process status: {error}"))?; + let capabilities = CapabilityEvidence { + inheritable: parse_status_hex(&status, "CapInh")?, + permitted: parse_status_hex(&status, "CapPrm")?, + effective: parse_status_hex(&status, "CapEff")?, + bounding: parse_status_hex(&status, "CapBnd")?, + ambient: parse_status_hex(&status, "CapAmb")?, + }; + let no_new_privileges = parse_status_decimal(&status, "NoNewPrivs")? == 1; + // SAFETY: PR_GET_DUMPABLE reads one scalar process property. + let sandbox_dumpable = unsafe { libc::prctl(libc::PR_GET_DUMPABLE, 0, 0, 0, 0) } != 0; + let mut core_limit = std::mem::MaybeUninit::::uninit(); + // SAFETY: getrlimit initializes the supplied output value on success. + if unsafe { libc::getrlimit(libc::RLIMIT_CORE, core_limit.as_mut_ptr()) } != 0 { + return Err(format!( + "read sandbox core limit: {}", + io::Error::last_os_error() + )); + } + // SAFETY: successful getrlimit initialized the value. + let core_limit = unsafe { core_limit.assume_init() }; + let (native_architecture, kernel_release) = uname_values()?; + Ok(SandboxConfirmEvidence { + generation: self.config.generation.clone(), + identity: self.config.workload_identity.clone(), + capabilities, + no_new_privileges, + sandbox_dumpable, + child_dumpable: true, + core_limit_zero: core_limit.rlim_cur == 0 && core_limit.rlim_max == 0, + native_architecture, + kernel_release, + seccomp: self.qualification.seccomp, + landlock_abi: self.qualification.landlock_abi, + landlock_allow_deny: self.qualification.landlock_allow_deny, + udp_dns_round_trip: self.qualification.udp_dns_round_trip, + tcp_dns_round_trip: self.qualification.tcp_dns_round_trip, + tcp_allow_round_trip: self.qualification.tcp_allow_round_trip, + tcp_deny_round_trip: self.qualification.tcp_deny_round_trip, + authenticated_supervisor: true, + session_id: self.config.session_id, + driver_fence: self.config.driver_fence.clone(), + runtime_exit_terminates_workload: true, + resource_claims: self.config.resource_claims.clone(), + }) + } + + #[allow(clippy::too_many_arguments)] + fn start_agent( + &self, + sandbox_id: String, + spec: AgentSpecWire, + policy: SandboxPolicyWire, + ca_cert: Option>, + ca_bundle: Option>, + provider_env_revision: u64, + provider_env: std::collections::HashMap, + ) -> Response { + let spec = match resolve_agent_spec(spec) { + Ok(spec) => spec, + Err(error) => return guest_error(BoundaryErrorKind::Process, error), + }; + let mut state = lock(&self.state); + let requested = StartedAgent { + sandbox_id, + spec: spec.clone(), + policy: policy.clone(), + ca_cert: ca_cert.clone(), + ca_bundle: ca_bundle.clone(), + provider_env_revision, + provider_env: provider_env.clone(), + }; + if let RuntimeState::Running(process) = &*state { + return if lock(&self.started_agent) + .as_ref() + .is_some_and(|accepted| accepted.matches_replay(&requested)) + { + Response::Started { + process_id: process.process_id(), + provider_env_revision: process.provider_credentials.snapshot().revision, + } + } else { + guest_error( + BoundaryErrorKind::Denied, + "start_agent inputs do not match the running boundary", + ) + }; + } + let RuntimeState::Ready(prepared) = &*state else { + return guest_error( + BoundaryErrorKind::Invalid, + "boundary must be confirmed before start_agent", + ); + }; + let ca_file_paths = match install_ca_material(ca_cert, ca_bundle) { + Ok(paths) => paths, + Err(error) => return guest_error(BoundaryErrorKind::Process, error), + }; + let mut policy = policy.into(); + let driver_identity = DriverIdentity::Resolved { + uid: self.config.workload_identity.uid, + gid: self.config.workload_identity.gid, + }; + if let Err(error) = resolve_process_identity(&mut policy, &driver_identity) { + return guest_error(BoundaryErrorKind::Process, error.to_string()); + } + let launch = ManagedProcessLaunch { + process_id: format!("{}:main:0", self.config.generation), + spec, + policy, + provider_env_revision, + provider_env, + ca_file_paths, + }; + let process = match ManagedProcess::spawn( + &self.process_runtime, + &self.workload_launcher, + launch, + prepared.clone(), + ) { + Ok(process) => Arc::new(process), + Err(error) => return guest_error(BoundaryErrorKind::Process, error), + }; + let process_id = process.process_id(); + *lock(&self.started_agent) = Some(requested); + *state = RuntimeState::Running(process); + Response::Started { + process_id, + provider_env_revision, + } + } + + fn update_provider_environment( + &self, + expected_revision: u64, + revision: u64, + provider_env: std::collections::HashMap, + ) -> Response { + let process = { + let state = lock(&self.state); + let RuntimeState::Running(process) = &*state else { + return guest_error( + BoundaryErrorKind::Invalid, + "agent process must be running before provider environment updates", + ); + }; + process.clone() + }; + let revision = match process + .provider_credentials + .compare_and_install_child_env_snapshot(expected_revision, revision, provider_env) + { + Ok(revision) => revision, + Err(error) => return guest_error(BoundaryErrorKind::Process, error.to_string()), + }; + Response::ProviderEnvironmentUpdated { revision } + } + + fn wait(&self, process_id: &str) -> Response { + let exec = lock(&self.exec_handles) + .get(process_id) + .map(|handle| handle.process.clone()); + if let Some(process) = exec { + // Wait independently of the output attachment. Retain the + // process, not the registry lock, while its exit is pending. + return match self.process_runtime.block_on(process.wait()) { + Ok(status) => Response::Exited { + status: status.into(), + }, + Err(error) => guest_error(BoundaryErrorKind::Process, error.to_string()), + }; + } + let process = match self.running_process(process_id) { + Ok(process) => process, + Err(response) => return response, + }; + match process.wait() { + Ok(status) => Response::Exited { status }, + Err(error) => guest_error(BoundaryErrorKind::Process, error), + } + } + + fn signal(&self, process_id: &str, signal: SignalWire) -> Response { + let process = match self.running_process(process_id) { + Ok(process) => process, + Err(response) => return response, + }; + match process.signal(signal) { + Ok(()) => Response::Signaled, + Err(error) => guest_error(BoundaryErrorKind::Terminated, error), + } + } + + fn terminate(&self, process_id: &str) -> Response { + let process = match self.running_process(process_id) { + Ok(process) => process, + Err(response) => return response, + }; + match process.signal(SignalWire::Kill) { + Ok(()) => Response::Terminated, + Err(_) if process.has_exited() => Response::Terminated, + Err(error) => guest_error(BoundaryErrorKind::Process, error), + } + } + + #[allow( + clippy::result_large_err, + reason = "protocol errors are returned directly as complete response frames" + )] + fn running_process(&self, process_id: &str) -> Result, Response> { + let state = lock(&self.state); + let RuntimeState::Running(process) = &*state else { + return Err(guest_error( + BoundaryErrorKind::Invalid, + "agent process has not been started", + )); + }; + if process.process_id() != process_id { + return Err(guest_error( + BoundaryErrorKind::Invalid, + "unknown process ID", + )); + } + Ok(process.clone()) + } + } + + fn parse_status_hex(status: &str, name: &str) -> Result { + let value = status + .lines() + .find_map(|line| { + line.strip_prefix(name) + .and_then(|value| value.strip_prefix(':')) + }) + .map(str::trim) + .ok_or_else(|| format!("sandbox process status omitted {name}"))?; + u64::from_str_radix(value, 16) + .map_err(|error| format!("parse sandbox process status {name}: {error}")) + } + + fn parse_status_decimal(status: &str, name: &str) -> Result { + let value = status + .lines() + .find_map(|line| { + line.strip_prefix(name) + .and_then(|value| value.strip_prefix(':')) + }) + .map(str::trim) + .ok_or_else(|| format!("sandbox process status omitted {name}"))?; + value + .parse::() + .map_err(|error| format!("parse sandbox process status {name}: {error}")) + } + + fn uname_values() -> Result<(String, String), String> { + let mut value = std::mem::MaybeUninit::::zeroed(); + // SAFETY: uname initializes the supplied utsname value on success. + if unsafe { libc::uname(value.as_mut_ptr()) } != 0 { + return Err(format!( + "measure sandbox kernel: {}", + io::Error::last_os_error() + )); + } + // SAFETY: successful uname initialized every fixed-size C string. + let value = unsafe { value.assume_init() }; + Ok((c_char_array(&value.machine), c_char_array(&value.release))) + } + + fn c_char_array(value: &[libc::c_char]) -> String { + let length = value + .iter() + .position(|byte| *byte == 0) + .unwrap_or(value.len()); + let bytes = value[..length] + .iter() + .map(|byte| byte.to_ne_bytes()[0]) + .collect::>(); + String::from_utf8_lossy(&bytes).into_owned() + } + + impl PreparedBoundary { + fn establish(network_broker: NetworkBroker) -> Result { + network_broker + .confirm_healthy() + .map_err(|error| format!("verify sandbox network broker: {error}"))?; + Ok(Self { network_broker }) + } + + fn confirm(&self, _runtime: &tokio::runtime::Handle) -> Result<(), String> { + self.network_broker + .confirm_healthy() + .map_err(|error| format!("verify sandbox network broker: {error}")) + } + } + + fn install_ca_material( + ca_cert: Option>, + ca_bundle: Option>, + ) -> Result, String> { + let (ca_cert, ca_bundle) = match (ca_cert, ca_bundle) { + (Some(ca_cert), Some(ca_bundle)) => (ca_cert, ca_bundle), + (None, None) => return Ok(None), + _ => { + return Err( + "supervisor CA certificate and bundle must be supplied together".to_string(), + ); + } + }; + install_ca_material_at( + Path::new(openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_DIR), + &ca_cert, + &ca_bundle, + ) + } + + fn install_ca_material_at( + directory: &Path, + ca_cert: &[u8], + ca_bundle: &[u8], + ) -> Result, String> { + use std::io::Write as _; + use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _}; + + let parent = directory + .parent() + .ok_or_else(|| "supervisor CA directory has no parent".to_string())?; + for path in [parent, directory] { + match std::fs::symlink_metadata(path) { + Ok(metadata) if metadata.file_type().is_symlink() => { + return Err(format!( + "supervisor CA directory component is a symlink: {}", + path.display() + )); + } + Ok(metadata) if !metadata.is_dir() => { + return Err(format!( + "supervisor CA directory component is not a directory: {}", + path.display() + )); + } + Ok(_) => {} + Err(error) if error.kind() == io::ErrorKind::NotFound => { + std::fs::create_dir(path).map_err(|error| { + format!("create supervisor CA directory {}: {error}", path.display()) + })?; + } + Err(error) => { + return Err(format!( + "inspect supervisor CA directory {}: {error}", + path.display() + )); + } + } + let current_mode = std::fs::metadata(path) + .map_err(|error| { + format!( + "inspect supervisor CA directory permissions {}: {error}", + path.display() + ) + })? + .permissions() + .mode(); + if path == directory { + std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o755)).map_err( + |error| { + format!( + "set supervisor CA directory permissions {}: {error}", + path.display() + ) + }, + )?; + } else if current_mode & 0o111 != 0o111 { + return Err(format!( + "supervisor CA parent is not traversable by workload identities: {}", + path.display() + )); + } + } + let ca_path = directory.join("ca.crt"); + let bundle_path = directory.join("ca-bundle.crt"); + for (path, contents, label) in [ + (&ca_path, ca_cert, "supervisor CA"), + (&bundle_path, ca_bundle, "supervisor CA bundle"), + ] { + let temporary = path.with_extension("tmp"); + if let Ok(metadata) = std::fs::symlink_metadata(&temporary) { + if !metadata.is_file() || metadata.file_type().is_symlink() { + return Err(format!( + "refusing unsafe temporary {label} path: {}", + temporary.display() + )); + } + std::fs::remove_file(&temporary) + .map_err(|error| format!("remove stale temporary {label}: {error}"))?; + } + let mut file = std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o444) + .custom_flags(libc::O_NOFOLLOW) + .open(&temporary) + .map_err(|error| format!("create temporary {label}: {error}"))?; + if let Err(error) = file + .write_all(contents) + .and_then(|()| file.sync_all()) + .and_then(|()| file.set_permissions(std::fs::Permissions::from_mode(0o444))) + .and_then(|()| std::fs::rename(&temporary, path)) + { + let _ = std::fs::remove_file(&temporary); + return Err(format!("install {label}: {error}")); + } + } + Ok(Some((ca_path, bundle_path))) + } + + type ProcessExit = Result; + type SharedProcessExit = Arc<(Mutex>, Condvar)>; + + struct ManagedProcess { + process_id: String, + signaler: AgentSignaler, + exit: SharedProcessExit, + boundary_exec: Arc, + loopback_connector: Arc, + main_session: Arc, + attached: Arc, + boundary_runtime: Arc, + provider_credentials: ProviderCredentialState, + } + + struct ManagedProcessLaunch { + process_id: String, + spec: AgentSpecWire, + policy: openshell_core::policy::SandboxPolicy, + provider_env_revision: u64, + provider_env: std::collections::HashMap, + ca_file_paths: Option<(std::path::PathBuf, std::path::PathBuf)>, + } + + fn resolve_agent_spec(mut spec: AgentSpecWire) -> Result { + if !spec.program.is_empty() { + return Ok(spec); + } + if !spec.args.is_empty() { + return Err("default agent command cannot include arguments".to_string()); + } + let shell = openshell_core::shell::detect_login_shell(); + if !openshell_core::shell::is_executable(&shell) { + return Err(format!( + "sandbox image does not provide an executable login shell at {shell}" + )); + } + spec.program = shell; + spec.args = vec!["-l".to_string()]; + Ok(spec) + } + + impl ManagedProcess { + fn spawn( + runtime: &tokio::runtime::Handle, + launcher: &openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, + launch: ManagedProcessLaunch, + _prepared: PreparedBoundary, + ) -> Result { + let ManagedProcessLaunch { + process_id, + spec, + policy, + provider_env_revision, + provider_env, + ca_file_paths, + } = launch; + debug_assert!(!spec.program.is_empty()); + let boundary_runtime = BoundaryRuntimeState::new_exclusive_pid_namespace(); + let entrypoint_pid = Arc::new(AtomicU32::new(0)); + let provider_credentials = ProviderCredentialState::from_child_env_snapshot( + provider_env_revision, + provider_env.clone(), + ); + let mut spawned = runtime + .block_on(spawn_workload( + launcher, + &spec.program, + &spec.args, + spec.workdir.as_deref(), + spec.timeout_secs, + spec.interactive, + &policy, + entrypoint_pid, + provider_credentials.clone(), + provider_env, + ca_file_paths, + Some(boundary_runtime.clone()), + )) + .map_err(|error| format!("start process supervisor leaf: {error:?}"))?; + let signaler = spawned.signaler(); + let boundary_exec = spawned.boundary_exec(); + let loopback_connector = spawned.loopback_connector(); + let main_session = spawned.main_session(); + let exit = Arc::new((Mutex::new(None), Condvar::new())); + let reaper_exit = exit.clone(); + runtime.spawn(async move { + let result = spawned + .wait() + .await + .map(process_status) + .map_err(|error| format!("wait for process supervisor leaf: {error}")); + let (state, changed) = &*reaper_exit; + *lock(state) = Some(result); + changed.notify_all(); + }); + Ok(Self { + process_id, + signaler, + exit, + boundary_exec, + loopback_connector, + main_session, + attached: Arc::new(AtomicBool::new(false)), + boundary_runtime, + provider_credentials, + }) + } + + fn process_id(&self) -> String { + self.process_id.clone() + } + + fn wait(&self) -> ProcessExit { + let (state, changed) = &*self.exit; + let mut exit = lock(state); + loop { + if let Some(result) = exit.as_ref() { + return result.clone(); + } + exit = changed + .wait(exit) + .unwrap_or_else(std::sync::PoisonError::into_inner); + } + } + + fn signal(&self, signal: SignalWire) -> Result<(), String> { + if self.has_exited() { + return Err("agent process has already exited".to_string()); + } + let result = match signal { + SignalWire::Term => self.signaler.term(), + SignalWire::Kill => self.signaler.kill(), + SignalWire::Int => self.signaler.interrupt(), + SignalWire::Hup => self.signaler.hangup(), + }; + result.map_err(|error| format!("signal process supervisor group: {error}")) + } + + fn has_exited(&self) -> bool { + let (state, _) = &*self.exit; + lock(state).is_some() + } + + fn exit_status(&self) -> Option { + let (state, _) = &*self.exit; + lock(state).as_ref().and_then(|result| result.clone().ok()) + } + + fn boundary_exec(&self) -> Arc { + self.boundary_exec.clone() + } + + fn loopback_connector(&self) -> Arc { + self.loopback_connector.clone() + } + + fn main_session(&self) -> Arc { + self.main_session.clone() + } + } + + impl Drop for ManagedProcess { + fn drop(&mut self) { + self.boundary_runtime.deactivate(); + } + } + + async fn bridge_main_stream( + stream: openshell_isolation_interface::contract::BoundaryDuplexStream, + attachment: MainAttachment, + ) -> Result<(), String> { + let session = attachment.session.clone(); + let (mut reader, writer) = tokio::io::split(stream); + let writer = Arc::new(tokio::sync::Mutex::new(writer)); + let input = session.acquire_input_if_open().map_err(str::to_string)?; + let owner = input.as_ref().map(|(owner, _)| *owner); + let mut output = session.subscribe(); + let input_session = session.clone(); + let mut input_task = tokio::spawn(async move { + let mut input = input.map(|(_, input)| input); + while let Some((channel, payload)) = read_stream_frame(&mut reader).await? { + match channel { + STREAM_STDIN => { + let Some(input) = input.as_ref() else { + return Err(io::Error::new( + io::ErrorKind::BrokenPipe, + "main process stdin already closed", + )); + }; + input.send(payload).await.map_err(|_| { + io::Error::new(io::ErrorKind::BrokenPipe, "main process stdin closed") + })?; + } + // Keep reading after stdin closes so transport EOF still + // releases this control process's attachment lease. + STREAM_STDIN_CLOSED => { + input.take(); + if let Some(owner) = owner { + input_session.close_input(owner).await; + } + } + _ => { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "unexpected host-to-boundary main stream channel", + )); + } + } + } + Ok::<(), io::Error>(()) + }); + let result = loop { + let output_message = tokio::select! { + input_result = &mut input_task => { + break match input_result { + Ok(Ok(())) => Ok(()), + Ok(Err(error)) => Err(format!("read main process attachment: {error}")), + Err(error) => Err(format!("join main process input stream: {error}")), + }; + } + output_message = output.recv() => output_message, + }; + let (channel, payload) = match output_message { + Ok(MainOutput::Stdout(payload)) => (STREAM_STDOUT, payload.to_vec()), + Ok(MainOutput::Stderr(payload)) => (STREAM_STDERR, payload.to_vec()), + Ok(MainOutput::Exit(code)) => { + let status = serde_json::to_vec(&attachment.exit_status(code)) + .map_err(|error| format!("encode main process exit: {error}"))?; + break write_stream_frame(&mut *writer.lock().await, STREAM_EXIT, &status) + .await + .map_err(|error| format!("write main process exit: {error}")); + } + Err(error) => { + tracing::warn!( + skipped_chunks = error.skipped, + "main process attachment resumed after dropping retained output" + ); + continue; + } + }; + if let Err(error) = + write_stream_frame(&mut *writer.lock().await, channel, &payload).await + { + break Err(format!("write main process output: {error}")); + } + }; + input_task.abort(); + if let Some(owner) = owner { + session.release_input(owner); + } + result + } + + fn lock(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { + mutex + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + } + + fn process_status(status: ProcessStatus) -> ExitStatusWire { + status.signal().map_or_else( + || ExitStatusWire::Exited(status.code()), + ExitStatusWire::Signaled, + ) + } + + fn guest_error(kind: BoundaryErrorKind, message: impl Into) -> Response { + Response::Error { + kind, + message: message.into(), + } + } + + enum ControlListener { + Vsock { + listener: OwnedFd, + server_config: Arc, + }, + Unix { + listener: std::os::unix::net::UnixListener, + server_config: Arc, + }, + Tcp { + listener: std::net::TcpListener, + server_config: Arc, + }, + } + + impl ControlListener { + fn bind(config: &BoundaryListenerConfig) -> io::Result { + match config { + BoundaryListenerConfig::Vsock { control_port, tls } => { + let listener = Self::bind_vsock(*control_port)?; + let server_config = Arc::new(load_tls_server_config(tls)?); + Ok(Self::Vsock { + listener, + server_config, + }) + } + BoundaryListenerConfig::Unix { socket_path, tls } => { + remove_owned_stale_control_socket(socket_path)?; + let listener = std::os::unix::net::UnixListener::bind(socket_path)?; + // Mutual TLS makes a same-UID pathname replacement a + // detectable denial of service rather than impersonation. + std::fs::set_permissions(socket_path, std::fs::Permissions::from_mode(0o666))?; + listener.set_nonblocking(true)?; + let server_config = Arc::new(load_tls_server_config(tls)?); + Ok(Self::Unix { + listener, + server_config, + }) + } + BoundaryListenerConfig::TlsTcp { address, tls } => { + let listener = std::net::TcpListener::bind(address)?; + listener.set_nonblocking(true)?; + let server_config = Arc::new(load_tls_server_config(tls)?); + Ok(Self::Tcp { + listener, + server_config, + }) + } + } + } + + fn bind_vsock(port: u32) -> io::Result { + let family = libc::sa_family_t::try_from(libc::AF_VSOCK).map_err(|_| { + io::Error::new(io::ErrorKind::InvalidInput, "AF_VSOCK exceeds sa_family_t") + })?; + let address_length = libc::socklen_t::try_from(size_of::()) + .map_err(|_| { + io::Error::new(io::ErrorKind::InvalidInput, "sockaddr_vm exceeds socklen_t") + })?; + let raw_fd = unsafe { + libc::socket( + libc::AF_VSOCK, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK, + 0, + ) + }; + if raw_fd < 0 { + return Err(io::Error::last_os_error()); + } + let fd = unsafe { OwnedFd::from_raw_fd(raw_fd) }; + let address = libc::sockaddr_vm { + svm_family: family, + svm_reserved1: 0, + svm_port: port, + svm_cid: libc::VMADDR_CID_ANY, + svm_zero: [0; 4], + }; + let result = unsafe { + libc::bind( + fd.as_raw_fd(), + (&raw const address).cast::(), + address_length, + ) + }; + if result < 0 { + return Err(io::Error::last_os_error()); + } + if unsafe { libc::listen(fd.as_raw_fd(), 16) } < 0 { + return Err(io::Error::last_os_error()); + } + Ok(fd) + } + + #[cfg(test)] + fn tcp_local_addr(&self) -> io::Result { + match self { + Self::Tcp { listener, .. } => listener.local_addr(), + Self::Unix { .. } | Self::Vsock { .. } => Err(io::Error::new( + io::ErrorKind::InvalidInput, + "control listener is not TCP", + )), + } + } + + fn accept(&self) -> io::Result { + match self { + Self::Vsock { + listener, + server_config, + } => { + let raw_fd = unsafe { + libc::accept4( + listener.as_raw_fd(), + std::ptr::null_mut(), + std::ptr::null_mut(), + libc::SOCK_CLOEXEC, + ) + }; + if raw_fd < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(ControlStream::PendingTls { + stream: PlainControlStream::Vsock(unsafe { File::from_raw_fd(raw_fd) }), + server_config: server_config.clone(), + }) + } + } + Self::Unix { + listener, + server_config, + } => { + let (stream, _) = listener.accept()?; + reject_workload_unix_peer(&stream)?; + Ok(ControlStream::PendingTls { + stream: PlainControlStream::Unix(stream), + server_config: server_config.clone(), + }) + } + Self::Tcp { + listener, + server_config, + } => { + let (stream, _) = listener.accept()?; + if let Err(error) = stream.set_nodelay(true) { + tracing::debug!(%error, "Failed to set boundary TCP_NODELAY"); + } + Ok(ControlStream::PendingTls { + stream: PlainControlStream::Tcp(stream), + server_config: server_config.clone(), + }) + } + } + } + } + + fn reject_workload_unix_peer(stream: &std::os::unix::net::UnixStream) -> io::Result<()> { + let mut credentials = libc::ucred { + pid: 0, + uid: 0, + gid: 0, + }; + let mut length = + libc::socklen_t::try_from(size_of::()).map_err(io::Error::other)?; + // SAFETY: both output pointers reference initialized storage of the + // declared length, and stream owns the connected Unix descriptor. + if unsafe { + libc::getsockopt( + stream.as_raw_fd(), + libc::SOL_SOCKET, + libc::SO_PEERCRED, + (&raw mut credentials).cast(), + &raw mut length, + ) + } != 0 + { + return Err(io::Error::last_os_error()); + } + let peer = u32::try_from(credentials.pid) + .map_err(|_| io::Error::from_raw_os_error(libc::EACCES))?; + // Linux reports PID zero for a peer outside our PID namespace. Such a + // peer still must authenticate with the per-sandbox mTLS certificate. + if peer != 0 + && peer != std::process::id() + && is_process_descendant(peer, std::process::id()) + .map_err(|_| io::Error::from_raw_os_error(libc::EACCES))? + { + return Err(io::Error::from_raw_os_error(libc::EACCES)); + } + Ok(()) + } + + fn is_process_descendant(mut process: u32, ancestor: u32) -> io::Result { + // Drivers run the sandbox as workload PID 1, so orphaned descendants + // reparent to it and cannot escape this check by double-forking. + // Read kernel-owned ancestry, never workload-supplied paths or UIDs. + // If a peer exits during inspection, fail closed for that connection. + for _ in 0..1024 { + if process == ancestor { + return Ok(true); + } + if process == 0 { + return Ok(false); + } + let stat = std::fs::read_to_string(format!("/proc/{process}/stat"))?; + let parent = stat + .rsplit_once(')') + .and_then(|(_, fields)| fields.split_whitespace().nth(1)) + .ok_or_else(|| { + io::Error::new(io::ErrorKind::InvalidData, "missing peer process parent") + })? + .parse::() + .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?; + if parent == process { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "cyclic peer process ancestry", + )); + } + process = parent; + } + Err(io::Error::from_raw_os_error(libc::EACCES)) + } + + fn remove_owned_stale_control_socket(socket_path: &Path) -> io::Result<()> { + let metadata = match std::fs::symlink_metadata(socket_path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(()), + Err(error) => return Err(error), + }; + if !metadata.file_type().is_socket() { + return Err(io::Error::new( + io::ErrorKind::AlreadyExists, + format!( + "refusing to replace non-socket boundary control path {}", + socket_path.display() + ), + )); + } + // The private channel directory is driver-provisioned. Requiring the + // stale inode to have been created by this exact sandbox identity + // prevents a replacement run from unlinking another principal's path. + if metadata.uid() != unsafe { libc::geteuid() } { + return Err(io::Error::new( + io::ErrorKind::PermissionDenied, + format!( + "refusing to replace boundary control socket {} owned by UID {}", + socket_path.display(), + metadata.uid() + ), + )); + } + std::fs::remove_file(socket_path) + } + + fn load_tls_server_config( + tls: &openshell_sandbox_backend::boundary_protocol::SandboxTlsServerConfig, + ) -> io::Result { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let certificate_bytes = std::fs::read(&tls.certificate_chain_path)?; + let certificates = rustls_pemfile::certs(&mut certificate_bytes.as_slice()) + .collect::, _>>()?; + if certificates.is_empty() { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "boundary TLS certificate chain contains no certificates", + )); + } + let private_key_bytes = std::fs::read(&tls.private_key_path)?; + let private_key = rustls_pemfile::private_key(&mut private_key_bytes.as_slice())? + .ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidInput, + "boundary TLS private-key file contains no private key", + ) + })?; + let mut config = + rustls::ServerConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .with_no_client_auth() + .with_single_cert(certificates, private_key) + .map_err(|error| io::Error::new(io::ErrorKind::InvalidInput, error))?; + config.alpn_protocols = vec![b"h2".to_vec()]; + for path in [&tls.certificate_chain_path, &tls.private_key_path] { + std::fs::remove_file(path)?; + } + Ok(config) + } + + enum PlainControlStream { + Vsock(File), + Unix(std::os::unix::net::UnixStream), + Tcp(std::net::TcpStream), + } + + impl PlainControlStream { + fn into_tokio( + self, + ) -> io::Result { + match self { + Self::Vsock(file) => { + let stream = + unsafe { std::os::unix::net::UnixStream::from_raw_fd(file.into_raw_fd()) }; + stream.set_nonblocking(true)?; + Ok(Box::new(tokio::net::UnixStream::from_std(stream)?)) + } + Self::Unix(stream) => { + stream.set_nonblocking(true)?; + Ok(Box::new(tokio::net::UnixStream::from_std(stream)?)) + } + Self::Tcp(stream) => { + stream.set_nonblocking(true)?; + let stream = tokio::net::TcpStream::from_std(stream)?; + openshell_core::net::set_tcp_nodelay_best_effort(&stream); + Ok(Box::new(stream)) + } + } + } + } + + enum ControlStream { + PendingTls { + stream: PlainControlStream, + server_config: Arc, + }, + Tls { + stream: Box< + tokio_rustls::server::TlsStream< + openshell_isolation_interface::contract::BoundaryDuplexStream, + >, + >, + runtime: tokio::runtime::Handle, + }, + Grpc { + stream: tokio::io::DuplexStream, + runtime: tokio::runtime::Handle, + }, + #[cfg(test)] + TestUnix(std::os::unix::net::UnixStream), + } + + impl ControlStream { + #[cfg(test)] + fn establish(self, runtime: &tokio::runtime::Handle) -> io::Result { + runtime.block_on(self.establish_async(runtime)) + } + + async fn establish_async(self, runtime: &tokio::runtime::Handle) -> io::Result { + let Self::PendingTls { + stream, + server_config, + } = self + else { + return Ok(self); + }; + let stream = { + let _guard = runtime.enter(); + stream.into_tokio()? + }; + let acceptor = tokio_rustls::TlsAcceptor::from(server_config); + let stream = { + tokio::time::timeout(CONTROL_HANDSHAKE_TIMEOUT, acceptor.accept(stream)) + .await + .map_err(|_| { + io::Error::new(io::ErrorKind::TimedOut, "boundary TLS handshake timed out") + })? + .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error)) + }?; + Ok(Self::Tls { + stream: Box::new(stream), + runtime: runtime.clone(), + }) + } + + fn set_timeout(&self, timeout: Duration) -> io::Result<()> { + let _ = timeout; + match self { + Self::Tls { .. } | Self::Grpc { .. } => Ok(()), + Self::PendingTls { .. } => Err(io::Error::new( + io::ErrorKind::NotConnected, + "boundary TLS stream has not completed its handshake", + )), + #[cfg(test)] + Self::TestUnix(stream) => { + stream.set_read_timeout(Some(timeout))?; + stream.set_write_timeout(Some(timeout)) + } + } + } + + fn into_tokio( + self, + ) -> Result { + match self { + Self::Tls { stream, .. } => Ok(stream), + Self::PendingTls { .. } => { + Err("boundary TLS stream has not completed its handshake".to_string()) + } + Self::Grpc { stream, .. } => Ok(Box::new(stream)), + #[cfg(test)] + Self::TestUnix(stream) => { + stream + .set_nonblocking(true) + .map_err(|error| format!("set test Unix stream nonblocking: {error}"))?; + Ok(Box::new(tokio::net::UnixStream::from_std(stream).map_err( + |error| format!("register test Unix stream with Tokio: {error}"), + )?)) + } + } + } + } + + impl Read for ControlStream { + fn read(&mut self, buffer: &mut [u8]) -> io::Result { + match self { + Self::Tls { stream, runtime } => runtime.block_on(async { + tokio::time::timeout(CONTROL_IO_TIMEOUT, stream.read(buffer)) + .await + .map_err(|_| { + io::Error::new(io::ErrorKind::TimedOut, "boundary TLS read timed out") + })? + }), + Self::PendingTls { .. } => Err(io::Error::new( + io::ErrorKind::NotConnected, + "boundary TLS stream has not completed its handshake", + )), + Self::Grpc { stream, runtime } => runtime.block_on(async { + tokio::time::timeout(CONTROL_IO_TIMEOUT, stream.read(buffer)) + .await + .map_err(|_| { + io::Error::new(io::ErrorKind::TimedOut, "gRPC boundary read timed out") + })? + }), + #[cfg(test)] + Self::TestUnix(stream) => stream.read(buffer), + } + } + } + + impl Write for ControlStream { + fn write(&mut self, buffer: &[u8]) -> io::Result { + match self { + Self::Tls { stream, runtime } => runtime.block_on(async { + tokio::time::timeout(CONTROL_IO_TIMEOUT, stream.write(buffer)) + .await + .map_err(|_| { + io::Error::new(io::ErrorKind::TimedOut, "boundary TLS write timed out") + })? + }), + Self::PendingTls { .. } => Err(io::Error::new( + io::ErrorKind::NotConnected, + "boundary TLS stream has not completed its handshake", + )), + Self::Grpc { stream, runtime } => runtime.block_on(async { + tokio::time::timeout(CONTROL_IO_TIMEOUT, stream.write(buffer)) + .await + .map_err(|_| { + io::Error::new(io::ErrorKind::TimedOut, "gRPC boundary write timed out") + })? + }), + #[cfg(test)] + Self::TestUnix(stream) => stream.write(buffer), + } + } + + fn flush(&mut self) -> io::Result<()> { + match self { + Self::Tls { stream, runtime } => runtime.block_on(async { + tokio::time::timeout(CONTROL_IO_TIMEOUT, stream.flush()) + .await + .map_err(|_| { + io::Error::new(io::ErrorKind::TimedOut, "boundary TLS flush timed out") + })? + }), + Self::PendingTls { .. } => Err(io::Error::new( + io::ErrorKind::NotConnected, + "boundary TLS stream has not completed its handshake", + )), + Self::Grpc { stream, runtime } => runtime.block_on(async { + tokio::time::timeout(CONTROL_IO_TIMEOUT, stream.flush()) + .await + .map_err(|_| { + io::Error::new(io::ErrorKind::TimedOut, "gRPC boundary flush timed out") + })? + }), + #[cfg(test)] + Self::TestUnix(stream) => stream.flush(), + } + } + } + + #[cfg(test)] + mod tests { + use super::*; + use openshell_core::jwt::{ + CredentialEpoch, DEFAULT_SESSION_TOKEN_TTL, SandboxRuntimeIdentity, SessionJwtIssuer, + }; + use openshell_sandbox_backend::boundary_protocol::{ + GatewayVerificationKey, SandboxTlsClientConfig, SandboxTlsServerConfig, + generate_sandbox_tls_material, + }; + use rcgen::{KeyPair, PKCS_ED25519}; + + #[test] + fn exec_tombstones_outlive_retained_handles_and_fail_closed_at_capacity() { + let mut requests = std::collections::HashSet::new(); + reserve_exec_request(&mut requests, "first").unwrap(); + // Process/I/O retention is deliberately not consulted by this + // ledger: dropping all handles cannot make this ID executable. + assert!(reserve_exec_request(&mut requests, "first").is_err()); + for index in 1..MAX_REPLAY_LEDGER_ENTRIES { + reserve_exec_request(&mut requests, &format!("request-{index}")).unwrap(); + } + assert!(reserve_exec_request(&mut requests, "overflow").is_err()); + assert!(requests.contains("first")); + assert_eq!(requests.len(), MAX_REPLAY_LEDGER_ENTRIES); + } + + #[test] + fn replay_ledger_evicts_oldest_records_without_disabling_control() { + let mut ledger = ReplayLedger::default(); + for index in 0..=MAX_REPLAY_LEDGER_ENTRIES { + ledger.insert( + format!("request-{index}"), + ReplayRecord { + payload_digest: format!("digest-{index}"), + response: Response::Signaled, + }, + ); + } + assert!(ledger.get("request-0").is_none()); + assert!( + ledger + .get(&format!("request-{MAX_REPLAY_LEDGER_ENTRIES}")) + .is_some() + ); + assert_eq!(ledger.entries.len(), MAX_REPLAY_LEDGER_ENTRIES); + } + + #[test] + fn scratch_agent_command_resolves_inside_the_workload_filesystem() { + let resolved = resolve_agent_spec(AgentSpecWire { + program: String::new(), + args: Vec::new(), + workdir: Some("/sandbox".to_string()), + timeout_secs: 0, + interactive: true, + }) + .expect("resolve scratch command"); + + assert!(openshell_core::shell::is_executable(&resolved.program)); + assert_eq!(resolved.args, vec!["-l".to_string()]); + assert_eq!(resolved.workdir.as_deref(), Some("/sandbox")); + assert!(resolved.interactive); + } + + fn test_session_id() -> openshell_core::SandboxSessionId { + "550e8400-e29b-41d4-a716-446655440000" + .parse() + .expect("test session ID") + } + + fn test_supervisor_instance_id() + -> openshell_sandbox_backend::boundary_protocol::SupervisorInstanceId { + openshell_sandbox_backend::boundary_protocol::SupervisorInstanceId::new() + } + + fn test_verification_key() -> GatewayVerificationKey { + let key = KeyPair::generate_for(&PKCS_ED25519).expect("generate gateway key"); + GatewayVerificationKey { + key_id: "test-key".to_string(), + public_key_pem: key.public_key_pem(), + } + } + + fn test_auth_material(sandbox_id: &str) -> (GatewayVerificationKey, String) { + let key = KeyPair::generate_for(&PKCS_ED25519).expect("generate gateway key"); + let verification_key = GatewayVerificationKey { + key_id: "test-key".to_string(), + public_key_pem: key.public_key_pem(), + }; + let issuer = SessionJwtIssuer::from_ed25519_pem( + key.serialize_pem().as_bytes(), + "test-key", + "test-gateway", + DEFAULT_SESSION_TOKEN_TTL, + Arc::new(SystemJwtClock), + ) + .expect("test session issuer"); + let token = issuer + .mint_pair(&SandboxRuntimeIdentity { + sandbox_id: SandboxId::parse(sandbox_id).expect("test sandbox ID"), + runtime_generation: + openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-1", + ) + .expect("runtime generation"), + auth_epoch: CredentialEpoch::new(1).expect("test auth epoch"), + }) + .expect("test token pair") + .sandbox + .token + .expose_secret() + .to_string(); + (verification_key, token) + } + + fn bearer_request(message: T, token: &str) -> tonic::Request { + let mut request = tonic::Request::new(message); + request.metadata_mut().insert( + "authorization", + format!("Bearer {token}") + .parse() + .expect("test authorization metadata"), + ); + request + } + + fn placeholder_server_tls() -> SandboxTlsServerConfig { + SandboxTlsServerConfig { + certificate_chain_path: Path::new("/tmp/openshell-sandbox.crt").to_path_buf(), + private_key_path: Path::new("/tmp/openshell-sandbox.key").to_path_buf(), + } + } + + fn stage_test_tls( + directory: &Path, + prefix: &str, + ) -> (SandboxTlsServerConfig, SandboxTlsClientConfig) { + let material = + generate_sandbox_tls_material(test_session_id()).expect("generate test TLS"); + let certificate_chain_path = directory.join(format!("{prefix}-sandbox.crt")); + let private_key_path = directory.join(format!("{prefix}-sandbox.key")); + std::fs::write(&certificate_chain_path, material.certificate_chain_pem) + .expect("write sandbox certificate"); + std::fs::write(&private_key_path, material.private_key_pem).expect("write sandbox key"); + ( + SandboxTlsServerConfig { + certificate_chain_path, + private_key_path, + }, + SandboxTlsClientConfig { + server_name: material.server_name, + trust_anchor_pem: material.trust_anchor_pem, + }, + ) + } + + fn test_client_config(tls: &SandboxTlsClientConfig) -> rustls::ClientConfig { + let mut roots = rustls::RootCertStore::empty(); + for certificate in rustls_pemfile::certs(&mut tls.trust_anchor_pem.as_bytes()) { + roots + .add(certificate.expect("parse test CA")) + .expect("add test CA"); + } + let mut config = rustls::ClientConfig::builder() + .with_root_certificates(roots) + .with_no_client_auth(); + config.alpn_protocols = vec![b"h2".to_vec()]; + config + } + + #[test] + fn boundary_config_debug_redacts_verification_material() { + let verification_key = test_verification_key(); + let config = BoundaryConfig { + boundary_id: "sandbox-1".to_string(), + generation: "generation-1".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![verification_key.clone()], + listener: BoundaryListenerConfig::Vsock { + control_port: 5500, + tls: placeholder_server_tls(), + }, + resource_claims: std::collections::BTreeMap::new(), + resource_claim_files: std::collections::BTreeMap::new(), + workload_identity: test_workload_identity(), + driver_fence: test_driver_fence(), + child_env: std::collections::HashMap::new(), + }; + let debug = format!("{config:?}"); + assert!(debug.contains("test-key")); + assert!(!debug.contains(&verification_key.public_key_pem)); + } + + #[test] + fn installed_supervisor_ca_is_readable_by_a_non_root_workload_identity() { + use std::os::unix::fs::PermissionsExt as _; + + let root = tempfile::tempdir().expect("temporary CA root"); + std::fs::set_permissions(root.path(), std::fs::Permissions::from_mode(0o755)).unwrap(); + let directory = root.path().join("openshell-supervisor-ca"); + let (ca_path, bundle_path) = install_ca_material_at( + &directory, + b"public test certificate", + b"public test bundle", + ) + .expect("install supervisor CA") + .expect("CA paths"); + + assert_eq!( + std::fs::metadata(directory.parent().unwrap()) + .unwrap() + .permissions() + .mode() + & 0o111, + 0o111, + "non-root workload identities must be able to traverse the full path" + ); + assert_eq!( + std::fs::metadata(&directory).unwrap().permissions().mode() & 0o777, + 0o755, + "non-root workload identities must be able to traverse the CA directory" + ); + for (path, expected) in [ + (&ca_path, b"public test certificate".as_slice()), + (&bundle_path, b"public test bundle".as_slice()), + ] { + assert_eq!( + std::fs::metadata(path).unwrap().permissions().mode() & 0o777, + 0o444, + "public CA material must be readable by the workload" + ); + assert_eq!(std::fs::read(path).unwrap(), expected); + } + } + + #[test] + fn supervisor_ca_install_rejects_a_symlinked_directory() { + let root = tempfile::tempdir().expect("temporary CA root"); + let target = root.path().join("target"); + std::fs::create_dir(&target).unwrap(); + let parent = root.path(); + std::os::unix::fs::symlink(&target, parent.join("openshell-supervisor-ca")).unwrap(); + + let error = install_ca_material_at( + &parent.join("openshell-supervisor-ca"), + b"certificate", + b"bundle", + ) + .expect_err("symlinked CA directory must fail closed"); + assert!(error.contains("symlink"), "unexpected error: {error}"); + } + + #[test] + fn supplementary_group_measurement_excludes_the_primary_group() { + assert_eq!( + normalized_supplementary_groups(vec![1002, 1001, 1000, 1001], 1000), + vec![1001, 1002] + ); + } + + #[test] + fn control_connection_slots_bound_authenticated_sessions() { + let active = Arc::new(AtomicUsize::new(MAX_CONTROL_CONNECTIONS - 1)); + let slot = acquire_control_connection_slot(&active).expect("last available slot"); + assert!(acquire_control_connection_slot(&active).is_none()); + drop(slot); + assert_eq!(active.load(Ordering::Acquire), MAX_CONTROL_CONNECTIONS - 1); + } + + #[test] + fn unix_control_rejects_workload_descendants_before_admission() { + const CHILD_SOCKET: &str = "OPENSHELL_TEST_CONTROL_PEER_SOCKET"; + if let Some(path) = std::env::var_os(CHILD_SOCKET) { + let mut stream = std::os::unix::net::UnixStream::connect(path).unwrap(); + stream + .set_read_timeout(Some(Duration::from_secs(5))) + .unwrap(); + assert_eq!(stream.read(&mut [0_u8; 1]).unwrap(), 0); + return; + } + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("control.sock"); + let listener = std::os::unix::net::UnixListener::bind(&path).unwrap(); + let mut child = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", "boundary_server::linux::tests::unix_control_rejects_workload_descendants_before_admission", "--nocapture"]) + .env(CHILD_SOCKET, &path).spawn().unwrap(); + let (stream, _) = listener.accept().unwrap(); + assert_eq!( + reject_workload_unix_peer(&stream) + .unwrap_err() + .raw_os_error(), + Some(libc::EACCES) + ); + drop(stream); + assert!(child.wait().unwrap().success()); + assert!(!is_process_descendant(std::process::id(), child.id()).unwrap()); + // Trusted same-process connections and external ancestors remain + // eligible for mTLS; we do not equate same UID with workload trust. + let client = std::os::unix::net::UnixStream::connect(&path).unwrap(); + let (stream, _) = listener.accept().unwrap(); + reject_workload_unix_peer(&stream).unwrap(); + drop(client); + } + + fn availability_test_runtime() -> (Arc, String) { + let (broker, launcher) = test_network_broker(); + let (verification_key, token) = test_auth_material("availability"); + let runtime = Arc::new( + BoundaryRuntime::new( + BoundaryConfig { + boundary_id: "availability".to_string(), + generation: "generation-1".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![verification_key], + listener: BoundaryListenerConfig::TlsTcp { + address: "127.0.0.1:5500".parse().unwrap(), + tls: placeholder_server_tls(), + }, + resource_claims: std::collections::BTreeMap::new(), + resource_claim_files: std::collections::BTreeMap::new(), + workload_identity: test_workload_identity(), + driver_fence: test_driver_fence(), + child_env: std::collections::HashMap::new(), + }, + tokio::runtime::Handle::current(), + broker, + launcher, + test_runtime_qualification(), + ) + .expect("test boundary runtime"), + ); + (runtime, token) + } + + #[tokio::test] + async fn connection_expiry_uses_one_updateable_deadline() { + let (shutdown, mut closed) = tokio::sync::watch::channel(()); + let expiry = ConnectionExpiry::new(shutdown); + expiry.update_deadline(tokio::time::Instant::now() + Duration::from_millis(20)); + expiry.update_deadline(tokio::time::Instant::now() + Duration::from_millis(200)); + + assert!( + tokio::time::timeout(Duration::from_millis(80), closed.changed()) + .await + .is_err(), + "replacing the deadline must cancel the earlier expiry" + ); + tokio::time::timeout(Duration::from_millis(250), closed.changed()) + .await + .expect("updated connection deadline must fire") + .expect("expiry worker must keep the shutdown channel open"); + } + + #[tokio::test(flavor = "multi_thread")] + async fn idle_uds_and_tcp_handshakes_do_not_consume_authenticated_slots() { + let directory = tempfile::tempdir().unwrap(); + let (tls, _) = stage_test_tls(directory.path(), "pending"); + let server_config = Arc::new(load_tls_server_config(&tls).unwrap()); + let (runtime, _) = availability_test_runtime(); + let active = Arc::new(AtomicUsize::new(0)); + let pending = Arc::new(tokio::sync::Semaphore::new(MAX_PENDING_HANDSHAKES)); + let mut clients: Vec> = Vec::new(); + let mut tasks = tokio::task::JoinSet::new(); + for index in 0..MAX_PENDING_HANDSHAKES { + let stream = if index % 2 == 0 { + let (server, client) = std::os::unix::net::UnixStream::pair().unwrap(); + clients.push(Box::new(client)); + PlainControlStream::Unix(server) + } else { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + clients.push(Box::new( + std::net::TcpStream::connect(listener.local_addr().unwrap()).unwrap(), + )); + PlainControlStream::Tcp(listener.accept().unwrap().0) + }; + let permit = pending.clone().try_acquire_owned().unwrap(); + tasks.spawn(serve_control_connection( + ControlStream::PendingTls { + stream, + server_config: server_config.clone(), + }, + runtime.clone(), + permit, + active.clone(), + )); + } + assert!(pending.clone().try_acquire_owned().is_err()); + assert_eq!(active.load(Ordering::Acquire), 0); + tokio::time::timeout(CONTROL_HANDSHAKE_TIMEOUT + Duration::from_secs(2), async { + while let Some(result) = tasks.join_next().await { + assert!(result.unwrap().unwrap_err().contains("timed out")); + } + }) + .await + .unwrap(); + assert_eq!(pending.available_permits(), MAX_PENDING_HANDSHAKES); + assert_eq!(active.load(Ordering::Acquire), 0); + drop(clients); + } + + #[tokio::test(flavor = "multi_thread")] + async fn disconnected_session_reconfirms_before_becoming_active() { + let (runtime, token) = availability_test_runtime(); + let supervisor_instance_id = test_supervisor_instance_id(); + let first_id = SandboxConnectionId::new(); + let first = runtime + .authenticate_request(first_id, bearer_request((), &token).metadata()) + .expect("first principal"); + runtime + .commit_attach(&first, supervisor_instance_id) + .expect("attach first"); + runtime.commit_confirm(&first).expect("confirm first"); + assert_eq!( + *lock(&runtime.supervisor_connection), + SupervisorConnectionState::Connected(first_id) + ); + + runtime.transport_disconnected(first_id); + let connection_state = *lock(&runtime.supervisor_connection); + let recovery_id = match connection_state { + SupervisorConnectionState::Frozen { recovery_id } => recovery_id, + state => panic!("expected frozen connection, got {state:?}"), + }; + let replacement_id = SandboxConnectionId::new(); + let replacement = runtime + .authenticate_request(replacement_id, bearer_request((), &token).metadata()) + .expect("replacement principal"); + assert!( + runtime + .commit_attach(&replacement, test_supervisor_instance_id()) + .is_err(), + "a replacement supervisor process must not claim the generation" + ); + runtime + .commit_attach(&replacement, supervisor_instance_id) + .expect("reattach replacement"); + assert_eq!( + runtime.connections.require_active(&replacement), + Err(openshell_sandbox_backend::sandbox_auth::SandboxAuthError::ConnectionNotAttached) + ); + runtime + .commit_confirm(&replacement) + .expect("reconfirm replacement"); + assert_eq!( + *lock(&runtime.supervisor_connection), + SupervisorConnectionState::Connected(replacement_id) + ); + + runtime.expire_recovery(recovery_id).await; + assert_eq!( + *lock(&runtime.supervisor_connection), + SupervisorConnectionState::Connected(replacement_id), + "stale recovery deadline must not terminate a reconfirmed session" + ); + } + + #[tokio::test(flavor = "multi_thread")] + async fn expired_recovery_makes_the_session_terminal() { + let (runtime, token) = availability_test_runtime(); + let supervisor_instance_id = test_supervisor_instance_id(); + let connection_id = SandboxConnectionId::new(); + let principal = runtime + .authenticate_request(connection_id, bearer_request((), &token).metadata()) + .expect("test principal"); + runtime + .commit_attach(&principal, supervisor_instance_id) + .expect("attach"); + runtime.commit_confirm(&principal).expect("confirm"); + runtime.transport_disconnected(connection_id); + let connection_state = *lock(&runtime.supervisor_connection); + let recovery_id = match connection_state { + SupervisorConnectionState::Frozen { recovery_id } => recovery_id, + state => panic!("expected frozen connection, got {state:?}"), + }; + + runtime.expire_recovery(recovery_id).await; + assert_eq!( + *lock(&runtime.supervisor_connection), + SupervisorConnectionState::Terminal + ); + let replacement_id = SandboxConnectionId::new(); + let replacement = runtime + .authenticate_request(replacement_id, bearer_request((), &token).metadata()) + .expect("replacement principal"); + assert!( + runtime + .commit_attach(&replacement, supervisor_instance_id) + .is_err() + ); + } + + #[tokio::test(flavor = "multi_thread")] + async fn explicit_shutdown_acknowledges_terminal_session_state() { + let (runtime, token) = availability_test_runtime(); + let supervisor_instance_id = test_supervisor_instance_id(); + let connection_id = SandboxConnectionId::new(); + let principal = runtime + .authenticate_request(connection_id, bearer_request((), &token).metadata()) + .expect("test principal"); + runtime + .commit_attach(&principal, supervisor_instance_id) + .expect("attach"); + runtime.commit_confirm(&principal).expect("confirm"); + + assert_eq!( + runtime.terminate_boundary().await, + Response::BoundaryTerminated + ); + assert_eq!( + *lock(&runtime.supervisor_connection), + SupervisorConnectionState::Terminal + ); + assert_eq!( + runtime.connections.require_active(&principal), + Err(openshell_sandbox_backend::sandbox_auth::SandboxAuthError::TerminalSession) + ); + } + + #[tokio::test(flavor = "multi_thread")] + async fn grpc_blackhole_expires_connection_and_releases_mediation_lease() { + let (runtime, token) = availability_test_runtime(); + let supervisor_instance_id = test_supervisor_instance_id(); + let connection_id = SandboxConnectionId::new(); + let principal = runtime + .authenticate_request(connection_id, bearer_request((), &token).metadata()) + .expect("test principal"); + runtime + .connections + .attach(&principal, supervisor_instance_id) + .expect("attach test connection"); + runtime + .connections + .confirm(&principal) + .expect("confirm test connection"); + *lock(&runtime.supervisor_connection) = + SupervisorConnectionState::Connected(connection_id); + let server_listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let server_address = server_listener.local_addr().unwrap(); + let server_runtime = runtime.clone(); + let server = tokio::spawn(async move { + let (stream, _) = server_listener.accept().await.unwrap(); + serve_grpc(Box::new(stream), server_runtime, connection_id).await + }); + let proxy_listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let proxy_address = proxy_listener.local_addr().unwrap(); + let (blackhole, stop_forwarding) = tokio::sync::oneshot::channel::<()>(); + let proxy = tokio::spawn(async move { + let (mut downstream, _) = proxy_listener.accept().await.unwrap(); + let mut upstream = tokio::net::TcpStream::connect(server_address) + .await + .unwrap(); + tokio::select! { + _ = stop_forwarding => {}, + _ = tokio::io::copy_bidirectional(&mut upstream, &mut downstream) => panic!("proxy closed before blackhole"), + } + // Keep both sockets open without forwarding PING or ACK: this + // models a silently dropped Kubernetes TCP path, not FIN/RST. + std::future::pending::<()>().await; + drop((upstream, downstream)); + }); + let channel = + tonic::transport::Endpoint::from_shared(format!("http://{proxy_address}")) + .unwrap() + .connect() + .await + .unwrap(); + let (sender, receiver) = tokio::sync::mpsc::channel(4); + let request = RequestEnvelope::new(Request::OpenMediation).unwrap(); + sender + .send(BoundaryChunk { + data: encode_frame(&request).unwrap(), + }) + .await + .unwrap(); + let mut response = IsolationBoundaryClient::new(channel) + .mediate(bearer_request(ReceiverStream::new(receiver), &token)) + .await + .unwrap() + .into_inner(); + assert!(response.message().await.unwrap().is_some()); + tokio::time::sleep(CONTROL_KEEPALIVE_INTERVAL + Duration::from_secs(1)).await; + assert!( + !server.is_finished(), + "healthy idle session must survive keepalive" + ); + assert!(runtime.mediation_active.try_lock().is_err()); + blackhole.send(()).unwrap(); + tokio::time::timeout( + CONTROL_KEEPALIVE_INTERVAL + CONTROL_KEEPALIVE_TIMEOUT + Duration::from_secs(3), + server, + ) + .await + .expect("blackholed HTTP/2 connection must expire") + .unwrap() + .unwrap(); + tokio::time::timeout(Duration::from_secs(1), async { + while runtime.mediation_active.try_lock().is_err() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("connection teardown must stop all bridges and release lease"); + let principal = runtime + .authenticate_request( + SandboxConnectionId::new(), + bearer_request((), &token).metadata(), + ) + .expect("test principal"); + runtime + .connections + .attach(&principal, supervisor_instance_id) + .expect("attach test connection"); + runtime + .connections + .confirm(&principal) + .expect("confirm test connection"); + let (mut replacement, task) = request_test_mediation(runtime, principal).await; + let ready: ResponseEnvelope = + openshell_sandbox_backend::boundary_protocol::read_frame_async(&mut replacement) + .await + .unwrap(); + assert!(matches!(ready.response, Response::MediationReady)); + drop(replacement); + task.await.unwrap().unwrap(); + drop(sender); + proxy.abort(); + } + + async fn request_test_mediation( + runtime: Arc, + principal: SandboxProtocolPrincipal, + ) -> ( + tokio::io::DuplexStream, + tokio::task::JoinHandle>, + ) { + let (mut client, server) = tokio::io::duplex(4096); + let task = tokio::spawn(serve_persistent_mediation(server, runtime, principal)); + let envelope = RequestEnvelope::new(Request::OpenMediation).unwrap(); + client + .write_all(&encode_frame(&envelope).unwrap()) + .await + .unwrap(); + (client, task) + } + + #[tokio::test(flavor = "multi_thread")] + async fn mediation_replacement_waits_for_lease_and_rejects_bad_authentication() { + let (runtime, token) = availability_test_runtime(); + let connection_id = SandboxConnectionId::new(); + let principal = runtime + .authenticate_request(connection_id, bearer_request((), &token).metadata()) + .expect("test principal"); + runtime + .connections + .attach(&principal, test_supervisor_instance_id()) + .expect("attach test connection"); + runtime + .connections + .confirm(&principal) + .expect("confirm test connection"); + let (mut first, first_task) = + request_test_mediation(runtime.clone(), principal.clone()).await; + let ready: ResponseEnvelope = + openshell_sandbox_backend::boundary_protocol::read_frame_async(&mut first) + .await + .unwrap(); + assert!(matches!(ready.response, Response::MediationReady)); + let (mut denied, denied_task) = + request_test_mediation(runtime.clone(), principal.clone()).await; + let response: ResponseEnvelope = + openshell_sandbox_backend::boundary_protocol::read_frame_async(&mut denied) + .await + .unwrap(); + assert!(matches!( + response.response, + Response::Error { + kind: BoundaryErrorKind::Denied, + .. + } + )); + denied_task.await.unwrap().unwrap(); + let (mut replacement, replacement_task) = + request_test_mediation(runtime.clone(), principal).await; + assert!( + tokio::time::timeout(Duration::from_millis(50), replacement.read_u8()) + .await + .is_err(), + "a live lease cannot be preempted" + ); + drop(first); + first_task.await.unwrap().unwrap(); + let ready: ResponseEnvelope = tokio::time::timeout( + Duration::from_secs(1), + openshell_sandbox_backend::boundary_protocol::read_frame_async(&mut replacement), + ) + .await + .unwrap() + .unwrap(); + assert!(matches!(ready.response, Response::MediationReady)); + assert!(runtime.mediation_active.try_lock().is_err()); + drop(replacement); + replacement_task.await.unwrap().unwrap(); + assert!(runtime.mediation_active.try_lock().is_ok()); + } + + fn test_workload_identity() -> ResolvedWorkloadIdentity { + let mut supplementary_gids = nix::unistd::getgroups() + .unwrap() + .into_iter() + .map(nix::unistd::Gid::as_raw) + .collect::>(); + supplementary_gids.sort_unstable(); + supplementary_gids.dedup(); + ResolvedWorkloadIdentity::new( + nix::unistd::geteuid().as_raw(), + nix::unistd::getegid().as_raw(), + supplementary_gids, + "test".to_string(), + "a".repeat(64), + ) + .unwrap() + } + + fn test_driver_fence() -> openshell_isolation_interface::contract::DriverFenceEvidence { + openshell_isolation_interface::contract::DriverFenceEvidence::Vm { + generation: "generation-1".to_string(), + network_device_count: 0, + } + } + + fn test_runtime_qualification() -> crate::RuntimeQualification { + crate::RuntimeQualification { + seccomp: openshell_isolation_interface::contract::SeccompEvidence { + new_listener: true, + notification_round_trip: true, + id_validation: true, + addfd_send: true, + retained_socket_operation: true, + proc_fd_identity: true, + task_memory_read: true, + task_memory_write: true, + cancellation: true, + }, + landlock_abi: 6, + landlock_allow_deny: true, + udp_dns_round_trip: true, + tcp_dns_round_trip: true, + tcp_allow_round_trip: true, + tcp_deny_round_trip: true, + } + } + + fn test_network_broker() -> ( + NetworkBroker, + openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, + ) { + let (launcher, listener) = + openshell_isolation_interface::linux::workload_launcher::start() + .expect("start test listener"); + ( + NetworkBroker::start_for_test(listener).expect("start test network broker"), + launcher, + ) + } + + #[test] + fn unix_listener_allows_authenticated_cross_uid_control() { + use std::os::unix::fs::PermissionsExt as _; + + let directory = tempfile::tempdir().expect("temporary directory"); + let socket_path = directory.path().join("control.sock"); + let (tls, _) = stage_test_tls(directory.path(), "initial"); + let _listener = ControlListener::bind(&BoundaryListenerConfig::Unix { + socket_path: socket_path.clone(), + tls, + }) + .expect("bind Unix listener"); + let mode = socket_path + .metadata() + .expect("socket metadata") + .permissions() + .mode(); + assert_eq!(mode & 0o777, 0o666); + } + + #[test] + fn unix_listener_replaces_only_an_owned_stale_socket() { + let directory = tempfile::tempdir().expect("temporary directory"); + let socket_path = directory.path().join("control.sock"); + let (initial_tls, _) = stage_test_tls(directory.path(), "initial"); + drop( + ControlListener::bind(&BoundaryListenerConfig::Unix { + socket_path: socket_path.clone(), + tls: initial_tls, + }) + .expect("bind initial Unix listener"), + ); + let (replacement_tls, _) = stage_test_tls(directory.path(), "replacement"); + let replacement = ControlListener::bind(&BoundaryListenerConfig::Unix { + socket_path: socket_path.clone(), + tls: replacement_tls, + }) + .expect("replace owned stale Unix listener"); + + drop(replacement); + std::fs::remove_file(&socket_path).expect("remove stale socket"); + std::fs::write(&socket_path, b"not a socket").expect("write collision"); + let (collision_tls, _) = stage_test_tls(directory.path(), "collision"); + let error = ControlListener::bind(&BoundaryListenerConfig::Unix { + socket_path, + tls: collision_tls, + }) + .err() + .expect("regular-file collision must fail"); + assert_eq!(error.kind(), io::ErrorKind::AlreadyExists); + } + + #[test] + fn exact_workload_identity_is_required() { + let config = BoundaryConfig { + boundary_id: "sandbox-1".to_string(), + generation: "generation-1".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![test_verification_key()], + listener: BoundaryListenerConfig::Vsock { + control_port: 5500, + tls: placeholder_server_tls(), + }, + resource_claims: std::collections::BTreeMap::new(), + resource_claim_files: std::collections::BTreeMap::new(), + workload_identity: test_workload_identity(), + driver_fence: test_driver_fence(), + child_env: std::collections::HashMap::new(), + }; + + validate_config(&config).unwrap(); + validate_running_identity(&config.workload_identity).unwrap(); + } + + #[test] + fn runtime_resource_claim_file_must_match_admitted_claim() { + let directory = tempfile::tempdir().expect("temporary directory"); + let pod_uid_path = directory.path().join("pod-uid"); + std::fs::write(&pod_uid_path, "pod-uid-a\n").expect("write runtime claim"); + let mut config = BoundaryConfig { + boundary_id: "sandbox-1".to_string(), + generation: "generation-1".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![test_verification_key()], + listener: BoundaryListenerConfig::Vsock { + control_port: 5500, + tls: placeholder_server_tls(), + }, + resource_claims: std::collections::BTreeMap::from([( + "kubernetes.pod_uid".to_string(), + "pod-uid-a".to_string(), + )]), + resource_claim_files: std::collections::BTreeMap::from([( + "kubernetes.pod_uid".to_string(), + pod_uid_path, + )]), + workload_identity: test_workload_identity(), + driver_fence: test_driver_fence(), + child_env: std::collections::HashMap::new(), + }; + + validate_config(&config).expect("valid runtime claim configuration"); + validate_runtime_resource_claims(&config).expect("matching runtime claim"); + + config.resource_claims.insert( + "kubernetes.pod_uid".to_string(), + "replacement-pod-uid".to_string(), + ); + assert!(validate_runtime_resource_claims(&config).is_err()); + } + + #[tokio::test(flavor = "multi_thread")] + async fn grpc_server_dispatches_authenticated_logical_streams() { + let (workload_launcher, listener) = + openshell_isolation_interface::linux::workload_launcher::start() + .expect("start multiplexed test listener"); + let network_broker = + NetworkBroker::start_for_test(listener).expect("start multiplexed test broker"); + let (verification_key, token) = test_auth_material("sandbox-multiplexed"); + let boundary = Arc::new( + BoundaryRuntime::new( + BoundaryConfig { + boundary_id: "sandbox-multiplexed".to_string(), + generation: "generation-1".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![verification_key], + listener: BoundaryListenerConfig::TlsTcp { + address: "127.0.0.1:5500".parse().expect("control address"), + tls: placeholder_server_tls(), + }, + resource_claims: std::collections::BTreeMap::new(), + resource_claim_files: std::collections::BTreeMap::new(), + workload_identity: test_workload_identity(), + driver_fence: test_driver_fence(), + child_env: std::collections::HashMap::new(), + }, + tokio::runtime::Handle::current(), + network_broker, + workload_launcher, + test_runtime_qualification(), + ) + .expect("test boundary runtime"), + ); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind gRPC test listener"); + let address = listener.local_addr().expect("gRPC test address"); + let server = tokio::spawn(async move { + let (stream, _) = listener.accept().await.expect("accept gRPC client"); + serve_grpc(Box::new(stream), boundary, SandboxConnectionId::new()).await + }); + let channel = tonic::transport::Endpoint::from_shared(format!("http://{address}")) + .expect("valid gRPC endpoint") + .connect() + .await + .expect("connect gRPC client"); + let policy = SandboxPolicyWire::from(openshell_core::policy::SandboxPolicy { + version: 1, + filesystem: openshell_core::policy::FilesystemPolicy::default(), + network: openshell_core::policy::NetworkPolicy::default(), + landlock: openshell_core::policy::LandlockPolicy::default(), + process: openshell_core::policy::ProcessPolicy::default(), + }); + let request = RequestEnvelope::new(Request::Attach { + supervisor_instance_id: test_supervisor_instance_id(), + policy: Box::new(policy), + resource_claims: std::collections::BTreeMap::new(), + }) + .expect("encode attach request"); + let request_stream = tokio_stream::iter([BoundaryChunk { + data: encode_frame(&request).expect("encode logical request"), + }]); + let mut body = IsolationBoundaryClient::new(channel) + .exchange(bearer_request(request_stream, &token)) + .await + .expect("exchange logical request") + .into_inner(); + let mut frame = Vec::new(); + while let Some(chunk) = body.message().await.expect("read gRPC response") { + frame.extend_from_slice(&chunk.data); + } + let response: ResponseEnvelope = + openshell_sandbox_backend::boundary_protocol::decode_frame(&frame) + .expect("decode logical response"); + assert!(matches!(response.response, Response::Attached { .. })); + server.abort(); + } + + #[test] + fn tls_listener_preserves_session_when_control_switches_to_async_streaming() { + let directory = tempfile::tempdir().expect("temporary directory"); + let (server_tls, client_tls) = stage_test_tls(directory.path(), "stream"); + let listener = ControlListener::bind(&BoundaryListenerConfig::TlsTcp { + address: "127.0.0.1:0".parse().expect("valid address"), + tls: server_tls, + }) + .expect("bind TLS listener"); + let address = listener.tcp_local_addr().expect("TLS listener address"); + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("test runtime"); + let server_runtime = runtime.handle().clone(); + let server = std::thread::spawn(move || { + let mut stream = loop { + match listener.accept() { + Ok(stream) => break stream, + Err(error) if error.kind() == io::ErrorKind::WouldBlock => { + std::thread::yield_now(); + } + Err(error) => panic!("accept TLS stream: {error}"), + } + } + .establish(&server_runtime) + .expect("establish TLS stream"); + let mut first = [0_u8; 4]; + Read::read_exact(&mut stream, &mut first).expect("read blocking TLS phase"); + assert_eq!(&first, b"sync"); + Write::write_all(&mut stream, b"ack1").expect("write blocking TLS phase"); + server_runtime.block_on(async move { + let mut stream = stream.into_tokio().expect("convert negotiated TLS stream"); + let mut second = [0_u8; 5]; + stream + .read_exact(&mut second) + .await + .expect("read async TLS phase"); + assert_eq!(&second, b"async"); + stream + .write_all(b"ack2") + .await + .expect("write async TLS phase"); + }); + }); + + runtime.block_on(async { + let client_config = test_client_config(&client_tls); + let stream = tokio::net::TcpStream::connect(address) + .await + .expect("connect TLS listener"); + let server_name = rustls::pki_types::ServerName::try_from(client_tls.server_name) + .expect("valid server name"); + let mut stream = tokio_rustls::TlsConnector::from(Arc::new(client_config)) + .connect(server_name, stream) + .await + .expect("verify TLS listener"); + stream.write_all(b"sync").await.expect("write first phase"); + let mut first_ack = [0_u8; 4]; + stream + .read_exact(&mut first_ack) + .await + .expect("read first acknowledgement"); + assert_eq!(&first_ack, b"ack1"); + stream + .write_all(b"async") + .await + .expect("write second phase"); + let mut second_ack = [0_u8; 4]; + stream + .read_exact(&mut second_ack) + .await + .expect("read second acknowledgement"); + assert_eq!(&second_ack, b"ack2"); + }); + server.join().expect("TLS boundary server thread"); + } + + #[test] + fn control_restart_replays_running_lifecycle_exactly_once() { + const CHILD_MARKER: &str = "OPENSHELL_TEST_BOUNDARY_RECONNECT_CHILD"; + if std::env::var_os(CHILD_MARKER).is_none() { + let status = std::process::Command::new( + std::env::current_exe().expect("current test executable"), + ) + .args([ + "--exact", + "boundary_server::linux::tests::control_restart_replays_running_lifecycle_exactly_once", + "--nocapture", + ]) + .env(CHILD_MARKER, "1") + .status() + .expect("run isolated reconnect test"); + assert!(status.success(), "isolated reconnect test failed"); + return; + } + + let process_runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("test process runtime"); + let (network_broker, workload_launcher) = test_network_broker(); + let boundary = Arc::new( + BoundaryRuntime::new( + BoundaryConfig { + boundary_id: "sandbox-reconnect".to_string(), + generation: "generation-reconnect".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![test_verification_key()], + listener: BoundaryListenerConfig::TlsTcp { + address: "127.0.0.1:5500".parse().expect("control address"), + tls: placeholder_server_tls(), + }, + resource_claims: std::collections::BTreeMap::new(), + resource_claim_files: std::collections::BTreeMap::new(), + workload_identity: test_workload_identity(), + driver_fence: test_driver_fence(), + child_env: std::collections::HashMap::new(), + }, + process_runtime.handle().clone(), + network_broker, + workload_launcher, + test_runtime_qualification(), + ) + .expect("test boundary runtime"), + ); + let policy = SandboxPolicyWire::from(openshell_core::policy::SandboxPolicy { + version: 1, + filesystem: openshell_core::policy::FilesystemPolicy::default(), + network: openshell_core::policy::NetworkPolicy::default(), + landlock: openshell_core::policy::LandlockPolicy::default(), + process: openshell_core::policy::ProcessPolicy::default(), + }); + let spec = AgentSpecWire { + program: "/bin/sleep".to_string(), + args: vec!["30".to_string()], + workdir: None, + timeout_secs: 60, + interactive: false, + }; + + assert!(matches!( + boundary.attach(policy.clone()), + Response::Attached { .. } + )); + assert!(matches!(boundary.confirm(), Response::Confirmed { .. })); + let start = || { + boundary.start_agent( + "sandbox-reconnect".to_string(), + spec.clone(), + policy.clone(), + None, + None, + 0, + std::collections::HashMap::new(), + ) + }; + let Response::Started { + process_id, + provider_env_revision: 0, + } = start() + else { + panic!("initial start did not succeed"); + }; + + let update = RequestEnvelope::new(Request::UpdateProviderEnvironment { + expected_revision: 0, + revision: 7, + provider_env: std::collections::HashMap::from([( + "REPLAY_TEST".to_string(), + "set-once".to_string(), + )]), + }) + .expect("build replayed update"); + assert_eq!( + boundary.dispatch(update.clone()), + Response::ProviderEnvironmentUpdated { revision: 7 } + ); + assert_eq!( + boundary.dispatch(update.clone()), + Response::ProviderEnvironmentUpdated { revision: 7 }, + "the same request ID and payload must replay its recorded response" + ); + let mut changed = RequestEnvelope::new(Request::Terminate { + process_id: process_id.clone(), + }) + .expect("build changed request"); + changed.request_id = update.request_id; + assert!(matches!( + boundary.dispatch(changed), + Response::Error { kind, .. } if kind == BoundaryErrorKind::Denied + )); + + let (first_attachment, _) = boundary + .attach_process(&process_id) + .expect("initial main-process attachment"); + assert!(boundary.attach_process(&process_id).is_err()); + let (boundary_stream, control_stream) = + std::os::unix::net::UnixStream::pair().expect("main attachment socket pair"); + let stream_boundary = boundary.clone(); + let stream_thread = std::thread::spawn(move || { + stream_boundary + .stream_process(ControlStream::TestUnix(boundary_stream), first_attachment) + }); + drop(control_stream); + stream_thread + .join() + .expect("join disconnected main attachment") + .expect("transport EOF cleanly ends main attachment"); + let (replacement_attachment, _) = boundary + .attach_process(&process_id) + .expect("replacement main-process attachment after disconnect"); + drop(replacement_attachment); + + assert!(matches!( + boundary.attach(policy.clone()), + Response::Attached { .. } + )); + assert!(matches!(boundary.confirm(), Response::Confirmed { .. })); + assert_eq!( + start(), + Response::Started { + process_id: process_id.clone(), + provider_env_revision: 7, + } + ); + + let mut changed_policy = policy.clone(); + changed_policy.version += 1; + assert!(matches!( + boundary.attach(changed_policy.clone()), + Response::Error { kind, .. } if kind == BoundaryErrorKind::Denied + )); + assert!(matches!( + boundary.start_agent( + "sandbox-reconnect".to_string(), + spec, + changed_policy, + None, + None, + 0, + std::collections::HashMap::new(), + ), + Response::Error { kind, .. } if kind == BoundaryErrorKind::Denied + )); + + let exec_spec = ExecSpecWire { + program: "/bin/sh".to_string(), + args: vec!["-c".to_string(), "printf reconnected".to_string()], + env: Vec::new(), + workdir: None, + pty: false, + }; + let exec_request = RequestEnvelope::new(Request::Exec { + spec: exec_spec.clone(), + }) + .expect("build exec request"); + let exec = boundary + .start_exec( + &exec_request.request_id, + &exec_request.payload_digest, + exec_spec, + ) + .expect("exec after reconnect"); + let exec_id = exec.process_id.clone(); + let mut output = String::new(); + let mut cursor = exec.attachment.session.subscribe(); + process_runtime.block_on(async { + loop { + match cursor.recv().await.expect("retained exec output") { + MainOutput::Stdout(bytes) => { + output.push_str(std::str::from_utf8(&bytes).expect("UTF-8 output")); + } + MainOutput::Stderr(_) => {} + MainOutput::Exit(code) => { + assert_eq!(code, 0); + break; + } + } + } + }); + assert_eq!(output, "reconnected"); + drop(exec); + for _ in 0..2 { + assert_eq!( + boundary.wait(&exec_id), + Response::Exited { + status: ExitStatusWire::Exited(0), + }, + "exec status must remain available after its output attachment closes" + ); + } + let Response::Attached { snapshot } = boundary.attach(policy.clone()) else { + panic!("reconnect attach did not return a session snapshot"); + }; + assert_eq!(snapshot.generation, "generation-reconnect"); + assert!(snapshot.processes.iter().any(|process| { + process.process_id == process_id && process.kind == ProcessKindWire::Main + })); + assert!(snapshot.processes.iter().any(|process| { + process.process_id == exec_id + && process.kind == ProcessKindWire::Exec + && process.status == Some(ExitStatusWire::Exited(0)) + && process.retained_output.next_sequence > 0 + })); + assert_eq!(boundary.terminate(&process_id), Response::Terminated); + } + + #[test] + fn canonical_exit_preserves_pending_network_accept_and_exec() { + const CHILD_MARKER: &str = "OPENSHELL_TEST_RETAINED_BOUNDARY_CHILD"; + if std::env::var_os(CHILD_MARKER).is_none() { + let status = std::process::Command::new( + std::env::current_exe().expect("current test executable"), + ) + .args([ + "--exact", + "boundary_server::linux::tests::canonical_exit_preserves_pending_network_accept_and_exec", + "--nocapture", + ]) + .env(CHILD_MARKER, "1") + .status() + .expect("run isolated retained-boundary test"); + assert!(status.success(), "isolated retained-boundary test failed"); + return; + } + + let process_runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("test process runtime"); + let policy = openshell_core::policy::SandboxPolicy { + version: 1, + filesystem: openshell_core::policy::FilesystemPolicy::default(), + network: openshell_core::policy::NetworkPolicy { + mode: openshell_core::policy::NetworkMode::Proxy, + proxy: Some(openshell_core::policy::ProxyPolicy { + http_addr: Some("127.0.0.1:3128".parse().expect("proxy address")), + }), + }, + landlock: openshell_core::policy::LandlockPolicy::default(), + process: openshell_core::policy::ProcessPolicy::default(), + }; + let (network_broker, workload_launcher) = test_network_broker(); + let prepared = PreparedBoundary { + network_broker: network_broker.clone(), + }; + let agent_spec = AgentSpecWire { + program: "/bin/true".to_string(), + args: Vec::new(), + workdir: None, + timeout_secs: 5, + interactive: false, + }; + let wire_policy = SandboxPolicyWire::from(policy.clone()); + let process = Arc::new( + ManagedProcess::spawn( + process_runtime.handle(), + &workload_launcher, + ManagedProcessLaunch { + process_id: "generation-retained:main:0".to_string(), + spec: agent_spec.clone(), + policy, + provider_env_revision: 0, + provider_env: std::collections::HashMap::new(), + ca_file_paths: None, + }, + prepared, + ) + .expect("spawn canonical process"), + ); + let boundary = Arc::new( + BoundaryRuntime::new( + BoundaryConfig { + boundary_id: "sandbox-retained".to_string(), + generation: "generation-retained".to_string(), + session_id: test_session_id(), + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .expect("session rotation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + gateway_id: "test-gateway".to_string(), + verification_keys: vec![test_verification_key()], + listener: BoundaryListenerConfig::TlsTcp { + address: "127.0.0.1:5500".parse().expect("control address"), + tls: placeholder_server_tls(), + }, + resource_claims: std::collections::BTreeMap::new(), + resource_claim_files: std::collections::BTreeMap::new(), + workload_identity: test_workload_identity(), + driver_fence: test_driver_fence(), + child_env: std::collections::HashMap::new(), + }, + process_runtime.handle().clone(), + network_broker, + workload_launcher, + test_runtime_qualification(), + ) + .expect("test boundary runtime"), + ); + *lock(&boundary.state) = RuntimeState::Running(process.clone()); + *lock(&boundary.attached_policy) = Some(wire_policy.clone()); + *lock(&boundary.started_agent) = Some(StartedAgent { + sandbox_id: "sandbox-retained".to_string(), + spec: agent_spec.clone(), + policy: wire_policy.clone(), + ca_cert: None, + ca_bundle: None, + provider_env_revision: 0, + provider_env: std::collections::HashMap::new(), + }); + + // A replacement control process replays the durable lifecycle and + // receives the original process rather than spawning another one. + assert!(matches!( + boundary.attach(wire_policy.clone()), + Response::Attached { .. } + )); + assert!(matches!(boundary.confirm(), Response::Confirmed { .. })); + assert_eq!( + boundary.start_agent( + "sandbox-retained".to_string(), + agent_spec.clone(), + wire_policy.clone(), + None, + None, + 0, + std::collections::HashMap::new(), + ), + Response::Started { + process_id: process.process_id(), + provider_env_revision: 0, + } + ); + + assert_eq!( + boundary.update_provider_environment( + 0, + 2, + std::collections::HashMap::from([( + "ROTATED_TOKEN".to_string(), + "refreshed".to_string(), + )]), + ), + Response::ProviderEnvironmentUpdated { revision: 2 } + ); + assert_eq!( + boundary.update_provider_environment( + 0, + 1, + std::collections::HashMap::from([( + "ROTATED_TOKEN".to_string(), + "stale".to_string(), + )]), + ), + Response::ProviderEnvironmentUpdated { revision: 2 } + ); + assert_eq!( + boundary.update_provider_environment(2, 1, std::collections::HashMap::new()), + Response::ProviderEnvironmentUpdated { revision: 1 }, + "a numerically smaller opaque revision must revoke the environment" + ); + assert_eq!( + boundary.update_provider_environment( + 2, + 3, + std::collections::HashMap::from([( + "ROTATED_TOKEN".to_string(), + "out-of-order".to_string(), + )]), + ), + Response::ProviderEnvironmentUpdated { revision: 1 }, + "a stale expected revision must not overwrite current state" + ); + assert_eq!( + boundary.update_provider_environment(1, 1, std::collections::HashMap::new()), + Response::ProviderEnvironmentUpdated { revision: 1 }, + "a duplicate update must be idempotent" + ); + + assert!(matches!( + boundary.attach(wire_policy.clone()), + Response::Attached { .. } + )); + assert!(matches!(boundary.confirm(), Response::Confirmed { .. })); + assert_eq!( + boundary.start_agent( + "sandbox-retained".to_string(), + agent_spec, + wire_policy, + None, + None, + 99, + std::collections::HashMap::from([( + "ROTATED_TOKEN".to_string(), + "replacement-control-snapshot".to_string(), + )]), + ), + Response::Started { + process_id: process.process_id(), + provider_env_revision: 1, + }, + "a replacement control must resume from the boundary's current revision" + ); + + let sleep_spec = ExecSpecWire { + program: "/bin/sleep".to_string(), + args: vec!["30".to_string()], + env: Vec::new(), + workdir: None, + pty: false, + }; + let sleep_request = RequestEnvelope::new(Request::Exec { + spec: sleep_spec.clone(), + }) + .expect("build retained exec request"); + let started = boundary + .start_exec( + &sleep_request.request_id, + &sleep_request.payload_digest, + sleep_spec.clone(), + ) + .expect("start exec whose response is disconnected"); + let retained_id = started.process_id.clone(); + drop(started); + let replayed = boundary + .start_exec( + &sleep_request.request_id, + &sleep_request.payload_digest, + sleep_spec.clone(), + ) + .expect("reattach exec after response loss"); + assert_eq!(replayed.process_id, retained_id); + assert_eq!(lock(&boundary.exec_handles).len(), 1); + drop(replayed); + assert_eq!( + boundary.signal_exec(&retained_id, SignalWire::Kill), + Response::Signaled + ); + for _ in 0..2 { + assert_eq!( + boundary.wait(&retained_id), + Response::Exited { + status: ExitStatusWire::Signaled(libc::SIGKILL), + }, + "independent waits must preserve a retained exec's signal status" + ); + } + lock(&boundary.exec_handles).remove(&retained_id); + assert!( + boundary + .start_exec( + &sleep_request.request_id, + &sleep_request.payload_digest, + sleep_spec, + ) + .is_err(), + "an evicted exec request must never start a second process" + ); + + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while !process.has_exited() && std::time::Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(10)); + } + assert!(process.has_exited(), "canonical process did not exit"); + assert_eq!( + boundary.wait(&process.process_id()), + Response::Exited { + status: ExitStatusWire::Exited(0), + } + ); + for exit_code in [0, 7] { + let spec = ExecSpecWire { + program: "/bin/sh".to_string(), + args: vec!["-c".to_string(), format!("exit {exit_code}")], + env: Vec::new(), + workdir: None, + pty: false, + }; + let request = RequestEnvelope::new(Request::Exec { spec: spec.clone() }) + .expect("build exec status request"); + let exec = boundary + .start_exec(&request.request_id, &request.payload_digest, spec) + .expect("start exec after canonical exit"); + for _ in 0..2 { + assert_eq!( + boundary.wait(&exec.process_id), + Response::Exited { + status: ExitStatusWire::Exited(exit_code), + }, + "wait must work independently of attachment consumption and main exit" + ); + } + } + assert!(matches!( + boundary.wait("generation-retained:exec:unknown"), + Response::Error { + kind: BoundaryErrorKind::Invalid, + .. + } + )); + + let mut session = process_runtime + .block_on( + process.boundary_exec().exec( + ExecSpecWire { + program: "/bin/sh".to_string(), + args: vec![ + "-c".to_string(), + "if [ -z \"${ROTATED_TOKEN+x}\" ]; then printf revoked; else printf 'unexpected:%s' \"$ROTATED_TOKEN\"; fi" + .to_string(), + ], + env: Vec::new(), + workdir: None, + pty: false, + } + .into(), + ), + ) + .expect("exec after canonical exit"); + let mut output = String::new(); + process_runtime + .block_on(session.stdout.read_to_string(&mut output)) + .expect("read retained exec output"); + assert_eq!( + output, "revoked", + "exec after canonical exit must use the latest reconciled provider snapshot" + ); + assert!(matches!( + process_runtime.block_on(session.process.wait()), + Ok(openshell_isolation_interface::contract::BoundaryExitStatus::Exited(0)) + )); + } + } +} + +#[cfg(target_os = "linux")] +pub fn run_boundary( + config_path: &Path, + qualification: crate::RuntimeQualification, +) -> Result<(), String> { + linux::run_boundary(config_path, qualification) +} + +#[cfg(not(target_os = "linux"))] +pub fn run_boundary( + _config_path: &Path, + _qualification: crate::RuntimeQualification, +) -> Result<(), String> { + Err("boundary mode is supported only on Linux".to_string()) +} diff --git a/crates/openshell-supervisor-process/src/child_env.rs b/crates/openshell-sandbox/src/child_env.rs similarity index 59% rename from crates/openshell-supervisor-process/src/child_env.rs rename to crates/openshell-sandbox/src/child_env.rs index 32eecbee35..50549a7439 100644 --- a/crates/openshell-supervisor-process/src/child_env.rs +++ b/crates/openshell-sandbox/src/child_env.rs @@ -3,24 +3,6 @@ use std::path::Path; -const LOCAL_NO_PROXY: &str = "127.0.0.1,localhost,::1"; - -pub fn proxy_env_vars(proxy_url: &str) -> [(&'static str, String); 9] { - [ - ("ALL_PROXY", proxy_url.to_owned()), - ("HTTP_PROXY", proxy_url.to_owned()), - ("HTTPS_PROXY", proxy_url.to_owned()), - ("NO_PROXY", LOCAL_NO_PROXY.to_owned()), - ("http_proxy", proxy_url.to_owned()), - ("https_proxy", proxy_url.to_owned()), - ("no_proxy", LOCAL_NO_PROXY.to_owned()), - ("grpc_proxy", proxy_url.to_owned()), - // Node.js only honors HTTP(S)_PROXY for built-in fetch/http clients when - // proxy support is explicitly enabled at process startup. - ("NODE_USE_ENV_PROXY", "1".to_owned()), - ] -} - pub fn tls_env_vars( ca_cert_path: &Path, combined_bundle_path: &Path, @@ -45,26 +27,6 @@ mod tests { use std::process::Command; use std::process::Stdio; - #[test] - fn apply_proxy_env_includes_node_proxy_opt_in_and_local_bypass() { - let mut cmd = Command::new("/usr/bin/env"); - cmd.stdin(Stdio::null()) - .stdout(Stdio::piped()) - .stderr(Stdio::null()); - - for (key, value) in proxy_env_vars("http://10.200.0.1:3128") { - cmd.env(key, value); - } - - let output = cmd.output().expect("spawn env"); - let stdout = String::from_utf8(output.stdout).expect("utf8"); - - assert!(stdout.contains("HTTP_PROXY=http://10.200.0.1:3128")); - assert!(stdout.contains("NO_PROXY=127.0.0.1,localhost,::1")); - assert!(stdout.contains("NODE_USE_ENV_PROXY=1")); - assert!(stdout.contains("no_proxy=127.0.0.1,localhost,::1")); - } - #[test] fn apply_tls_env_sets_node_and_bundle_paths() { let mut cmd = Command::new("/usr/bin/env"); diff --git a/crates/openshell-sandbox/src/delegated.rs b/crates/openshell-sandbox/src/delegated.rs new file mode 100644 index 0000000000..6fe65c4b72 --- /dev/null +++ b/crates/openshell-sandbox/src/delegated.rs @@ -0,0 +1,238 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Process and access-plane assembly for the capability-free sandbox boundary. + +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; +use std::time::Duration; + +#[cfg(target_os = "linux")] +use miette::WrapErr as _; +use miette::{IntoDiagnostic as _, Result}; +use openshell_core::policy::SandboxPolicy; +use openshell_core::provider_credentials::ProviderCredentialState; +use openshell_isolation_interface::contract::{BoundaryExec, BoundaryLoopbackConnector}; +use openshell_ocsf::{ + ActionId, ActivityId, DispositionId, LaunchTypeId, Process as OcsfProcess, + ProcessActivityBuilder, SeverityId, StatusId, ocsf_emit, +}; + +use crate::process::{ProcessHandle, ProcessStatus, ResolvedWorkspace}; + +fn ocsf_ctx() -> &'static openshell_ocsf::EventContext { + openshell_ocsf::ctx::ctx() +} + +/// Spawn the admitted workload without placing the gateway or policy authority +/// inside its boundary. +#[allow(clippy::too_many_arguments, clippy::implicit_hasher)] +pub async fn spawn_workload( + launcher: &openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, + program: &str, + args: &[String], + workdir: Option<&str>, + timeout_secs: u64, + interactive: bool, + policy: &SandboxPolicy, + entrypoint_pid: Arc, + provider_credentials: ProviderCredentialState, + provider_env: std::collections::HashMap, + ca_file_paths: Option<(std::path::PathBuf, std::path::PathBuf)>, + boundary_runtime: Option>, +) -> Result { + // Driver-selected workspaces are the sandbox identity's home. This keeps + // canonical and later exec processes consistent for image WorkingDir and + // the managed /sandbox fallback without consulting privileged account + // setup inside the capability-free boundary. + let workspace = ResolvedWorkspace::new(workdir.map(str::to_string), true); + + #[cfg(target_os = "linux")] + { + let mode = if std::env::var_os("OPENSHELL_REQUIRE_RUNTIME_PID_LIMIT").is_some() { + crate::process::RuntimePidLimitMode::Require + } else { + crate::process::RuntimePidLimitMode::Warn + }; + crate::process::check_runtime_pid_limit(mode).wrap_err("check runtime PID limit")?; + } + + let boundary_runtime = boundary_runtime + .unwrap_or_else(crate::boundary_io::BoundaryRuntimeState::new_exclusive_pid_namespace); + let mut user_environment: std::collections::HashMap = + std::env::var(openshell_core::sandbox_env::USER_ENVIRONMENT) + .ok() + .and_then(|json| serde_json::from_str(&json).ok()) + .unwrap_or_default(); + user_environment.retain(|key, _value| !crate::process::is_proxy_env_var(key)); + let loopback_connector: Arc = Arc::new( + crate::boundary_io::LocalLoopbackConnector::new(Some(boundary_runtime.clone())), + ); + let boundary_exec: Arc = + Arc::new(crate::boundary_exec::LocalBoundaryExec::new( + policy.clone(), + workspace.owned_root(), + ca_file_paths.clone().map(Arc::new), + provider_credentials, + user_environment, + boundary_runtime.clone(), + launcher.clone(), + )); + + #[cfg(target_os = "linux")] + let mut handle = ProcessHandle::spawn( + launcher, + program, + args, + &workspace, + interactive, + policy, + ca_file_paths.as_ref(), + &provider_env, + ) + .wrap_err("spawn delegated workload process")?; + #[cfg(not(target_os = "linux"))] + let mut handle = ProcessHandle::spawn( + program, + args, + &workspace, + interactive, + policy, + ca_file_paths.as_ref(), + &provider_env, + )?; + + entrypoint_pid.store(handle.pid(), Ordering::Release); + let main_session = crate::main_session::MainSession::new(handle.take_io(), handle.pid()); + let (terminal, signal_lock) = handle.signaling_state(); + boundary_runtime + .register_process_group(handle.pid(), terminal.clone(), signal_lock.clone()) + .map_err(|error| miette::miette!(error.to_string()))?; + + ocsf_emit!( + ProcessActivityBuilder::new(ocsf_ctx()) + .activity(ActivityId::Open) + .action(ActionId::Allowed) + .disposition(DispositionId::Allowed) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .launch_type(LaunchTypeId::Spawn) + .process(OcsfProcess::new(program, i64::from(handle.pid()))) + .message(format!("Process started: pid={}", handle.pid())) + .build() + ); + + Ok(SpawnedAgent { + handle, + timeout_secs, + terminal, + signal_lock, + main_session, + boundary_exec, + loopback_connector, + boundary_runtime, + }) +} + +/// Owned workload process and its live boundary capabilities. +pub struct SpawnedAgent { + handle: ProcessHandle, + timeout_secs: u64, + terminal: Arc, + signal_lock: Arc>, + main_session: Arc, + boundary_exec: Arc, + loopback_connector: Arc, + boundary_runtime: Arc, +} + +impl SpawnedAgent { + #[must_use] + pub fn signaler(&self) -> AgentSignaler { + AgentSignaler { + pid: self.handle.pid(), + terminal: self.terminal.clone(), + signal_lock: self.signal_lock.clone(), + } + } + + #[must_use] + pub fn boundary_exec(&self) -> Arc { + self.boundary_exec.clone() + } + + #[must_use] + pub fn loopback_connector(&self) -> Arc { + self.loopback_connector.clone() + } + + /// Retained canonical-process I/O owned by the boundary. + #[must_use] + pub fn main_session(&self) -> Arc { + self.main_session.clone() + } + + /// Wait for the canonical process to exit, enforcing its admitted + /// wall-clock timeout. Completion does not end the boundary: exec and + /// loopback forwarding remain available until the boundary owner tears + /// down the retained runtime. + pub async fn wait(&mut self) -> Result { + let signaler = self.signaler(); + let status = if self.timeout_secs == 0 { + self.handle.wait().await.into_diagnostic()? + } else if let Ok(status) = + tokio::time::timeout(Duration::from_secs(self.timeout_secs), self.handle.wait()).await + { + status.into_diagnostic()? + } else { + let _ = signaler.term(); + tokio::time::sleep(Duration::from_millis(100)).await; + let _ = signaler.kill(); + self.handle.wait().await.into_diagnostic()? + }; + self.boundary_runtime + .unregister_process_group(self.handle.pid(), &self.terminal); + let _ = self.main_session.finish(status.code(), false).await; + self.main_session.mark_terminal_reported(); + Ok(status) + } +} + +/// Lock-free process-group signal handle used while another task owns `wait`. +#[derive(Clone)] +pub struct AgentSignaler { + pid: u32, + terminal: Arc, + signal_lock: Arc>, +} + +#[cfg(unix)] +impl AgentSignaler { + fn deliver(&self, signal: nix::sys::signal::Signal) -> Result<()> { + let _guard = self + .signal_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if self.terminal.load(Ordering::Acquire) { + return Err(miette::miette!("agent has exited")); + } + let pid = i32::try_from(self.pid).unwrap_or(i32::MAX); + nix::sys::signal::killpg(nix::unistd::Pid::from_raw(pid), signal).into_diagnostic() + } + + pub fn term(&self) -> Result<()> { + self.deliver(nix::sys::signal::Signal::SIGTERM) + } + + pub fn kill(&self) -> Result<()> { + self.deliver(nix::sys::signal::Signal::SIGKILL) + } + + pub fn interrupt(&self) -> Result<()> { + self.deliver(nix::sys::signal::Signal::SIGINT) + } + + pub fn hangup(&self) -> Result<()> { + self.deliver(nix::sys::signal::Signal::SIGHUP) + } +} diff --git a/crates/openshell-sandbox/src/google_cloud_metadata.rs b/crates/openshell-sandbox/src/google_cloud_metadata.rs deleted file mode 100644 index 9e1e179872..0000000000 --- a/crates/openshell-sandbox/src/google_cloud_metadata.rs +++ /dev/null @@ -1,536 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! GCE metadata server emulator for sandbox credential injection. -//! -//! Implements a subset of the GCE instance metadata API so that GCP client -//! libraries (Go, Python, Node.js) can obtain `OAuth2` tokens natively inside -//! sandboxes. Tokens are served from the existing `ProviderCredentialState` -//! store — no separate refresh mechanism is needed. -//! -//! The emulator runs as a loopback HTTP server inside the sandbox network -//! namespace (see [`metadata_server`](crate::metadata_server)). GCP SDKs -//! discover it via the `GCE_METADATA_HOST` environment variable, which is -//! set to the loopback address by `child_env_with_gcp_resolved()`. - -use miette::{IntoDiagnostic, Result}; -use openshell_core::provider_credentials::ProviderCredentialState; -use openshell_core::secrets; -use openshell_ocsf::{ActivityId, HttpActivityBuilder, SeverityId, StatusId, ocsf_emit}; -use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt}; - -type MetadataResponse = (u16, &'static str, String); - -const PATH_SERVICE_ACCOUNTS: &str = "/computeMetadata/v1/instance/service-accounts"; -const PATH_SERVICE_ACCOUNT_DEFAULT: &str = "/computeMetadata/v1/instance/service-accounts/default"; -const PATH_TOKEN: &str = "/computeMetadata/v1/instance/service-accounts/default/token"; -const PATH_EMAIL: &str = "/computeMetadata/v1/instance/service-accounts/default/email"; -const PATH_SCOPES: &str = "/computeMetadata/v1/instance/service-accounts/default/scopes"; -const PATH_ALIASES: &str = "/computeMetadata/v1/instance/service-accounts/default/aliases"; -const PATH_PROJECT_ID: &str = "/computeMetadata/v1/project/project-id"; - -const ENV_GCP_PROJECT_ID: &str = openshell_core::google_cloud::PROJECT_ID_ENV_VARS[0]; -const ENV_GCP_SERVICE_ACCOUNT_EMAIL: &str = - openshell_core::google_cloud::SERVICE_ACCOUNT_EMAIL_ENV_VARS[0]; - -const METADATA_FLAVOR_HEADER: &str = "metadata-flavor"; -const METADATA_FLAVOR_VALUE: &str = "Google"; -const X_FORWARDED_FOR_HEADER: &str = "x-forwarded-for"; - -#[derive(Debug, Clone)] -pub struct MetadataContext { - credentials: ProviderCredentialState, -} - -impl MetadataContext { - pub fn new(credentials: ProviderCredentialState) -> Self { - Self { credentials } - } -} - -impl crate::metadata_server::MetadataHandler for MetadataContext { - async fn handle( - &self, - method: &str, - path: &str, - request: &[u8], - stream: &mut S, - ) -> Result<()> { - handle_forward_request(self, method, path, request, stream).await - } -} - -async fn handle_forward_request( - ctx: &MetadataContext, - method: &str, - path: &str, - initial_request: &[u8], - client: &mut S, -) -> Result<()> -where - S: AsyncRead + AsyncWrite + Unpin, -{ - let headers = parse_request_headers(initial_request); - let (status, content_type, body) = route_request(ctx, method, path, &headers); - write_metadata_response(client, status, content_type, &body).await -} - -fn route_request( - ctx: &MetadataContext, - method: &str, - path: &str, - headers: &[(String, String)], -) -> MetadataResponse { - if method != "GET" { - emit_metadata_event( - ActivityId::Refuse, - SeverityId::Low, - StatusId::Failure, - &format!("metadata: unsupported method {method}"), - ); - return (405, "text/html", "Method Not Allowed".to_string()); - } - - if let Err(resp) = validate_metadata_headers(headers) { - emit_metadata_event( - ActivityId::Refuse, - SeverityId::Medium, - StatusId::Failure, - &format!("metadata: header validation failed for {path}"), - ); - return resp; - } - - let (route, query) = path.split_once('?').map_or((path, ""), |(r, q)| (r, q)); - let route = route.strip_suffix('/').unwrap_or(route); - let recursive = query.split('&').any(|p| p == "recursive=true"); - - match route { - PATH_TOKEN => handle_token(ctx), - PATH_EMAIL => handle_env(ctx, ENV_GCP_SERVICE_ACCOUNT_EMAIL), - PATH_PROJECT_ID => handle_env(ctx, ENV_GCP_PROJECT_ID), - PATH_ALIASES => (200, "text/plain", "default\n".to_string()), - PATH_SCOPES => ( - 200, - "text/plain", - "https://www.googleapis.com/auth/cloud-platform".to_string(), - ), - PATH_SERVICE_ACCOUNT_DEFAULT => { - if recursive { - handle_service_account_recursive(ctx) - } else { - ( - 200, - "text/plain", - "aliases\nemail\nscopes\ntoken\n".to_string(), - ) - } - } - PATH_SERVICE_ACCOUNTS => (200, "text/plain", "default/\n".to_string()), - "" | "/" | "/computeMetadata" | "/computeMetadata/v1" => { - (200, "text/plain", "computeMetadata/\n".to_string()) - } - "/computeMetadata/v1/instance" => (200, "text/plain", "service-accounts/\n".to_string()), - _ => { - emit_metadata_event( - ActivityId::Refuse, - SeverityId::Low, - StatusId::Failure, - &format!("metadata: unknown path {route}"), - ); - ( - 404, - "application/json", - serde_json::json!({"error": "not_found"}).to_string(), - ) - } - } -} - -fn handle_token(ctx: &MetadataContext) -> MetadataResponse { - let Some((placeholder, expires_in)) = ctx.credentials.gcp_token_response() else { - let has_resolver = ctx.credentials.resolver().is_some(); - let (msg, error_key) = if has_resolver { - ( - "metadata: no GCP access token available or expired", - "token_unavailable", - ) - } else { - ( - "metadata: token request but no credentials configured", - "credentials_unavailable", - ) - }; - emit_metadata_event(ActivityId::Fail, SeverityId::Medium, StatusId::Failure, msg); - return ( - 503, - "application/json", - serde_json::json!({"error": error_key}).to_string(), - ); - }; - - emit_metadata_event( - ActivityId::Open, - SeverityId::Informational, - StatusId::Success, - "metadata: token placeholder served", - ); - - let body = serde_json::json!({ - "access_token": placeholder, - "expires_in": expires_in, - "token_type": "Bearer" - }); - (200, "application/json", body.to_string()) -} - -fn handle_service_account_recursive(ctx: &MetadataContext) -> MetadataResponse { - let resolver = ctx.credentials.resolver(); - let email = resolver - .as_ref() - .and_then(|r| { - let p = secrets::placeholder_for_env_key(ENV_GCP_SERVICE_ACCOUNT_EMAIL); - r.resolve_placeholder(&p).map(str::to_string) - }) - .unwrap_or_default(); - - let scopes = "https://www.googleapis.com/auth/cloud-platform"; - - let body = serde_json::json!({ - "aliases": ["default"], - "email": email, - "scopes": [scopes], - }); - (200, "application/json", body.to_string()) -} - -/// Serve a non-secret config value (project ID, SA email) as plain text. -/// -/// Unlike `handle_token` which serves placeholders, this resolves to the real -/// value. This matches real GCE metadata server behavior and is safe because -/// these values are non-secret configuration (project IDs, email addresses). -fn handle_env(ctx: &MetadataContext, env_key: &str) -> MetadataResponse { - let Some(resolver) = ctx.credentials.resolver() else { - emit_metadata_event( - ActivityId::Fail, - SeverityId::Medium, - StatusId::Failure, - &format!("metadata: {env_key} request but no credentials configured"), - ); - return (503, "text/plain", String::new()); - }; - - let placeholder = secrets::placeholder_for_env_key(env_key); - resolver.resolve_placeholder(&placeholder).map_or_else( - || { - emit_metadata_event( - ActivityId::Fail, - SeverityId::Low, - StatusId::Failure, - &format!("metadata: {env_key} not configured"), - ); - ( - 404, - "application/json", - serde_json::json!({"error": "not_found"}).to_string(), - ) - }, - |value| (200, "text/plain", value.to_string()), - ) -} - -fn validate_metadata_headers(headers: &[(String, String)]) -> Result<(), MetadataResponse> { - if headers - .iter() - .any(|(name, _)| name.eq_ignore_ascii_case(X_FORWARDED_FOR_HEADER)) - { - return Err((403, "text/html", "Forbidden".to_string())); - } - - let has_flavor = headers.iter().any(|(name, value)| { - name.eq_ignore_ascii_case(METADATA_FLAVOR_HEADER) - && value.trim().eq_ignore_ascii_case(METADATA_FLAVOR_VALUE) - }); - if !has_flavor { - return Err((403, "text/html", "Forbidden".to_string())); - } - - Ok(()) -} - -fn parse_request_headers(raw: &[u8]) -> Vec<(String, String)> { - let request = String::from_utf8_lossy(raw); - let mut headers = Vec::new(); - for line in request.split("\r\n").skip(1) { - if line.is_empty() { - break; - } - if let Some((name, value)) = line.split_once(':') { - headers.push((name.trim().to_string(), value.trim().to_string())); - } - } - headers -} - -fn status_text(status: u16) -> &'static str { - match status { - 403 => "Forbidden", - 404 => "Not Found", - 405 => "Method Not Allowed", - 503 => "Service Unavailable", - _ => "OK", - } -} - -async fn write_metadata_response( - client: &mut S, - status: u16, - content_type: &str, - body: &str, -) -> Result<()> -where - S: AsyncWrite + Unpin, -{ - let response = format!( - "HTTP/1.1 {status} {}\r\nContent-Type: {content_type}\r\nMetadata-Flavor: Google\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", - status_text(status), - body.len(), - ); - client - .write_all(response.as_bytes()) - .await - .into_diagnostic()?; - client.flush().await.into_diagnostic()?; - Ok(()) -} - -fn emit_metadata_event( - activity: ActivityId, - severity: SeverityId, - status: StatusId, - message: &str, -) { - let event = HttpActivityBuilder::new(crate::ocsf_ctx()) - .activity(activity) - .severity(severity) - .status(status) - .message(message.to_string()) - .build(); - ocsf_emit!(event); -} - -#[cfg(test)] -mod tests { - use super::*; - use std::collections::HashMap; - - fn make_context(env: HashMap) -> MetadataContext { - let state = - ProviderCredentialState::from_environment(0, env, HashMap::new(), HashMap::new()); - MetadataContext::new(state) - } - - fn make_context_with_expiry( - env: HashMap, - expires: HashMap, - ) -> MetadataContext { - let state = ProviderCredentialState::from_environment(0, env, expires, HashMap::new()); - MetadataContext::new(state) - } - - fn flavor_headers() -> Vec<(String, String)> { - vec![("Metadata-Flavor".to_string(), "Google".to_string())] - } - - #[test] - fn token_returns_placeholder_not_real_value() { - let ctx = make_context(HashMap::from([( - "GCP_ADC_ACCESS_TOKEN".to_string(), - "ya29.test-token".to_string(), - )])); - let (status, ct, body) = route_request(&ctx, "GET", PATH_TOKEN, &flavor_headers()); - assert_eq!(status, 200); - assert_eq!(ct, "application/json"); - let json: serde_json::Value = serde_json::from_str(&body).unwrap(); - let token = json["access_token"].as_str().unwrap(); - assert!( - token.starts_with("openshell:resolve:env:"), - "token should be a placeholder, got: {token}" - ); - assert!(!token.contains("ya29"), "real token must not be served"); - assert_eq!(json["token_type"], "Bearer"); - assert!(json["expires_in"].is_number()); - } - - #[test] - fn token_expires_in_computed_from_credential_expiry() { - let now_ms = openshell_core::time::now_ms(); - let expires_at = now_ms + 1_800_000; // 30 minutes from now - let ctx = make_context_with_expiry( - HashMap::from([("GCP_ADC_ACCESS_TOKEN".to_string(), "ya29.tok".to_string())]), - HashMap::from([("GCP_ADC_ACCESS_TOKEN".to_string(), expires_at)]), - ); - let (status, _, body) = route_request(&ctx, "GET", PATH_TOKEN, &flavor_headers()); - assert_eq!(status, 200); - let json: serde_json::Value = serde_json::from_str(&body).unwrap(); - let expires_in = json["expires_in"].as_i64().unwrap(); - assert!( - expires_in > 1700 && expires_in <= 1800, - "expires_in={expires_in}" - ); - } - - #[test] - fn token_no_expiry_defaults_to_3600() { - let ctx = make_context(HashMap::from([( - "GCP_ADC_ACCESS_TOKEN".to_string(), - "ya29.tok".to_string(), - )])); - let (_, _, body) = route_request(&ctx, "GET", PATH_TOKEN, &flavor_headers()); - let json: serde_json::Value = serde_json::from_str(&body).unwrap(); - assert_eq!(json["expires_in"], 3600); - } - - #[test] - fn missing_metadata_flavor_header_403() { - let ctx = make_context(HashMap::new()); - let (status, _, _) = route_request(&ctx, "GET", PATH_TOKEN, &[]); - assert_eq!(status, 403); - } - - #[test] - fn x_forwarded_for_header_403() { - let ctx = make_context(HashMap::new()); - let headers = vec![ - ("Metadata-Flavor".to_string(), "Google".to_string()), - ("X-Forwarded-For".to_string(), "10.0.0.1".to_string()), - ]; - let (status, _, _) = route_request(&ctx, "GET", PATH_TOKEN, &headers); - assert_eq!(status, 403); - } - - #[test] - fn unknown_path_404() { - let ctx = make_context(HashMap::new()); - let (status, _, _) = route_request( - &ctx, - "GET", - "/computeMetadata/v1/unknown", - &flavor_headers(), - ); - assert_eq!(status, 404); - } - - #[test] - fn no_credentials_503() { - let ctx = make_context(HashMap::new()); - let (status, _, _) = route_request(&ctx, "GET", PATH_TOKEN, &flavor_headers()); - assert_eq!(status, 503); - } - - #[test] - fn post_method_405() { - let ctx = make_context(HashMap::new()); - let (status, _, _) = route_request(&ctx, "POST", PATH_TOKEN, &flavor_headers()); - assert_eq!(status, 405); - } - - #[test] - fn project_id_served_as_plain_text() { - let ctx = make_context(HashMap::from([( - "GCP_PROJECT_ID".to_string(), - "my-project-123".to_string(), - )])); - let (status, ct, body) = route_request(&ctx, "GET", PATH_PROJECT_ID, &flavor_headers()); - assert_eq!(status, 200); - assert_eq!(ct, "text/plain"); - assert_eq!(body, "my-project-123"); - } - - #[test] - fn email_served_as_plain_text() { - let ctx = make_context(HashMap::from([( - "GCP_SERVICE_ACCOUNT_EMAIL".to_string(), - "sa@project.iam.gserviceaccount.com".to_string(), - )])); - let (status, ct, body) = route_request(&ctx, "GET", PATH_EMAIL, &flavor_headers()); - assert_eq!(status, 200); - assert_eq!(ct, "text/plain"); - assert_eq!(body, "sa@project.iam.gserviceaccount.com"); - } - - #[test] - fn scopes_returns_cloud_platform() { - let ctx = make_context(HashMap::new()); - let (status, _, body) = route_request(&ctx, "GET", PATH_SCOPES, &flavor_headers()); - assert_eq!(status, 200); - assert_eq!(body, "https://www.googleapis.com/auth/cloud-platform"); - } - - #[test] - fn query_parameters_ignored_for_routing() { - let ctx = make_context(HashMap::from([( - "GCP_ADC_ACCESS_TOKEN".to_string(), - "ya29.tok".to_string(), - )])); - let path = format!("{PATH_TOKEN}?scopes=cloud-platform"); - let (status, _, _) = route_request(&ctx, "GET", &path, &flavor_headers()); - assert_eq!(status, 200); - } - - #[test] - fn metadata_flavor_case_insensitive() { - let ctx = make_context(HashMap::from([( - "GCP_ADC_ACCESS_TOKEN".to_string(), - "ya29.tok".to_string(), - )])); - let headers = vec![("metadata-FLAVOR".to_string(), "google".to_string())]; - let (status, _, _) = route_request(&ctx, "GET", PATH_TOKEN, &headers); - assert_eq!(status, 200); - } - - #[test] - fn missing_env_var_returns_404() { - let ctx = make_context(HashMap::from([( - "GCP_ADC_ACCESS_TOKEN".to_string(), - "ya29.tok".to_string(), - )])); - // project-id not set - let (status, _, _) = route_request(&ctx, "GET", PATH_PROJECT_ID, &flavor_headers()); - assert_eq!(status, 404); - } - - #[test] - fn trailing_slash_handled_for_service_account_default() { - let ctx = make_context(HashMap::from([( - "GCP_ADC_ACCESS_TOKEN".to_string(), - "ya29.tok".to_string(), - )])); - let with_slash = route_request( - &ctx, - "GET", - "/computeMetadata/v1/instance/service-accounts/default/", - &flavor_headers(), - ); - let without_slash = route_request( - &ctx, - "GET", - "/computeMetadata/v1/instance/service-accounts/default", - &flavor_headers(), - ); - assert_eq!(with_slash.0, 200); - assert_eq!(without_slash.0, 200); - assert_eq!(with_slash.2, without_slash.2); - } - - #[test] - fn parse_request_headers_extracts_correctly() { - let raw = b"GET /path HTTP/1.1\r\nHost: example.com\r\nMetadata-Flavor: Google\r\n\r\n"; - let headers = parse_request_headers(raw); - assert_eq!(headers.len(), 2); - assert_eq!(headers[0].0, "Host"); - assert_eq!(headers[0].1, "example.com"); - assert_eq!(headers[1].0, "Metadata-Flavor"); - assert_eq!(headers[1].1, "Google"); - } -} diff --git a/crates/openshell-supervisor-process/src/identity.rs b/crates/openshell-sandbox/src/identity.rs similarity index 100% rename from crates/openshell-supervisor-process/src/identity.rs rename to crates/openshell-sandbox/src/identity.rs diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index b5b9358ac0..4928fe6f40 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -1,6306 +1,67 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! `OpenShell` Sandbox library. -//! -//! This crate provides process sandboxing and monitoring capabilities. +//! Capability-free in-workload sandbox boundary. -// `defaults-without-telemetry` is an alias for the default feature set minus -// `telemetry`, not a switch that turns telemetry off. Cargo cannot subtract a -// default feature, so adding it on top of the defaults would otherwise produce -// a telemetry-on build that reads as telemetry-free. Fail the build instead. -#[cfg(all(feature = "telemetry", feature = "defaults-without-telemetry"))] -compile_error!( - "features `telemetry` and `defaults-without-telemetry` are mutually exclusive; \ - build a telemetry-free supervisor with `--no-default-features --features defaults-without-telemetry`" -); - -mod activity_aggregator; -mod denial_aggregator; -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -mod google_cloud_metadata; -mod mechanistic_mapper; -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -mod metadata_server; -mod sidecar_control; - -use miette::{IntoDiagnostic, Result, WrapErr}; -use std::future::Future; -use std::pin::Pin; -use std::sync::Arc; #[cfg(target_os = "linux")] -use std::sync::atomic::Ordering; -use std::sync::atomic::{AtomicBool, AtomicU32}; -use std::time::Duration; -use tracing::{debug, info, warn}; - -use openshell_core::PolicyValidationFailureMode; - -use openshell_ocsf::{ - ActionId, ActivityId, AppLifecycleBuilder, ConfidenceId, ConfigStateChangeBuilder, - DetectionFindingBuilder, DispositionId, EventContext, FindingInfo, OcsfEvent, SeverityId, - StateId, StatusId, ocsf_emit, -}; - -// --------------------------------------------------------------------------- -// OCSF Context -// --------------------------------------------------------------------------- -// -// The following log sites intentionally remain as plain `tracing` macros -// and are NOT migrated to OCSF builders: -// -// - DEBUG/TRACE events (zombie reaping, ip commands, gRPC connects, PTY state) -// - Transient "about to do X" events where the result is logged separately -// (e.g., "Fetching sandbox policy via gRPC", "Creating OPA engine from proto") -// - Internal SSH channel warnings (unknown channel, PTY resize failures) -// - Denial flush telemetry (the individual denials are already OCSF events) -// - Status reporting failures (sync to gateway, non-actionable) -// - Route refresh interval validation warnings -// -// These are operational plumbing that don't represent security decisions, -// policy changes, or observable sandbox behavior worth structuring. -// --------------------------------------------------------------------------- - -/// Re-export the process-wide OCSF sandbox context getter. -/// -/// The singleton lives in `openshell-ocsf` so both supervisor leaves can -/// reach it without depending on `openshell-sandbox`. Initialised once during -/// `run_sandbox()` startup via `openshell_ocsf::ctx::set_ctx`. -pub(crate) use openshell_ocsf::ctx::ctx as ocsf_ctx; - -use openshell_core::denial::DenialEvent; -use openshell_core::policy::{NetworkMode, NetworkPolicy, ProxyPolicy, SandboxPolicy}; -use openshell_core::proposals::AgentProposals; -use openshell_core::provider_credentials::ProviderCredentialState; -use openshell_supervisor_network::opa::OpaEngine; -use openshell_supervisor_network::proxy::ProxyHandle; -use openshell_supervisor_process::process::ProcessEnforcementMode; -pub use openshell_supervisor_process::process::{ProcessHandle, ProcessStatus}; -use openshell_supervisor_process::skills; -use tokio::sync::mpsc::UnboundedSender; -#[cfg(any(test, target_os = "linux"))] -use tokio::time::timeout; - -const SIDECAR_NETWORK_ENFORCEMENT_MODE: &str = "sidecar-nftables"; -const SIDECAR_TLS_DIR: &str = openshell_core::container_paths::SIDECAR_TLS_DIR; -const SIDECAR_CA_CERT: &str = "openshell-ca.pem"; -const SIDECAR_CA_BUNDLE: &str = "ca-bundle.pem"; - -#[cfg(any(test, target_os = "linux"))] -fn has_network_runtime_capability(capabilities: Option<&str>, required: &str) -> bool { - capabilities.is_some_and(|capabilities| { - capabilities - .split(',') - .any(|capability| capability.trim() == required) - }) -} -const SIDECAR_PROCESS_PROXY_ADDR: &str = "127.0.0.1:3128"; -const SIDECAR_READY_TIMEOUT_SECS: u64 = 120; - -/// Run a command in the sandbox. -/// -/// # Errors -/// -/// Returns an error if the command fails to start or encounters a fatal error. -#[allow( - clippy::too_many_arguments, - clippy::implicit_hasher, - clippy::similar_names, - clippy::fn_params_excessive_bools -)] -pub async fn run_sandbox( - command: Vec, - workdir: Option, - timeout_secs: u64, - interactive: bool, - await_main_process_attachment: bool, - sandbox_id: Option, - sandbox: Option, - openshell_endpoint: Option, - policy_rules: Option, - policy_data: Option, - ssh_socket_path: Option, - _health_check: bool, - _health_port: u16, - ocsf_enabled: Arc, - ocsf_schema_version: Arc>, - network_enabled: bool, - process_enabled: bool, - upstream_proxy_args: openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs, -) -> Result { - let (program, args) = command - .split_first() - .ok_or_else(|| miette::miette!("No command specified"))?; - - // Initialize the process-wide OCSF context early so that events emitted - // during policy loading (filesystem config, validation) have a context. - // Proxy IP/port use defaults here; they are only significant for network - // events which happen after the netns is created. - { - let hostname = std::fs::read_to_string("/etc/hostname").map_or_else( - |_| "openshell-sandbox".to_string(), - |s| s.trim().to_string(), - ); - - if !openshell_ocsf::ctx::set_ctx(EventContext { - sandbox_id: sandbox_id.clone().unwrap_or_default(), - sandbox_name: sandbox.as_deref().unwrap_or_default().to_string(), - container_image: std::env::var("OPENSHELL_CONTAINER_IMAGE").unwrap_or_default(), - hostname, - product_version: openshell_core::VERSION.to_string(), - proxy_ip: std::net::IpAddr::from([127, 0, 0, 1]), - proxy_port: 3128, - }) { - debug!("OCSF context already initialized, keeping existing"); - } - } - - let sidecar_network_enforcement = sidecar_network_enforcement_enabled(); - let process_enforcement_mode = process_enforcement_mode(); - let process_uses_sidecar_control = - process_enabled && !network_enabled && sidecar_network_enforcement; - let mut process_control_connection = None; - let sidecar_bootstrap = if process_uses_sidecar_control { - let socket = sidecar_control_socket().ok_or_else(|| { - miette::miette!( - "{} is required for process-only sidecar topology", - openshell_core::sandbox_env::SIDECAR_CONTROL_SOCKET - ) - })?; - let (bootstrap, connection) = sidecar_control::connect_process_client( - &socket, - Duration::from_secs(SIDECAR_READY_TIMEOUT_SECS), - ) - .await?; - process_control_connection = Some(connection); - Some(bootstrap) - } else { - None - }; - - // Extension credentials are owned by this supervisor and shared by every - // gateway connection it opens, so the middleware registry's bearer slots - // and the policy poll loop that rotates them stay the same objects. - let extension_credentials = openshell_extension_core::ExtensionCredentialStore::new(); - - // Load policy and initialize OPA engine - let openshell_endpoint_for_proxy = openshell_endpoint.clone(); - let sandbox_name_for_agg = sandbox.clone(); - let ( - mut policy, - opa_engine, - retained_proto, - middleware_registry_status, - loaded_policy_origin, - initial_agent_proposals_enabled, - initial_extension_authentication_enabled, - ) = if let Some(bootstrap) = sidecar_bootstrap.as_ref() { - let (policy, opa_engine, retained_proto, loaded_policy_origin) = - load_policy_from_sidecar_bootstrap(bootstrap)?; - ( - policy, - opa_engine, - retained_proto, - MiddlewareRegistryStatus::Synchronized, - loaded_policy_origin, - bootstrap.agent_proposals_enabled, - false, - ) - } else { - load_policy( - sandbox_id.clone(), - sandbox, - openshell_endpoint.clone(), - policy_rules, - policy_data, - &extension_credentials, - ) - .await? - }; - - // Normalize the active driver's identity contract once, while both the - // policy and launched image filesystem are available. Kubernetes and - // OpenShift retain their authoritative numeric pair; Docker fills only - // omitted policy fields from OCI Config.User. - #[cfg(unix)] - let (resolved_process_identity, workspace) = { - let driver_identity = openshell_supervisor_process::identity::DriverIdentity::from_env()?; - let use_workdir_as_home = matches!( - &driver_identity, - openshell_supervisor_process::identity::DriverIdentity::OciUser { .. } - ); - let resolved = openshell_supervisor_process::identity::resolve_process_identity( - &mut policy, - &driver_identity, - )?; - ( - resolved, - openshell_supervisor_process::process::ResolvedWorkspace::new( - workdir.clone(), - use_workdir_as_home, - ), - ) - }; - #[cfg(not(unix))] - let (resolved_process_identity, workspace) = ( - openshell_supervisor_process::process::ResolvedProcessIdentity::default(), - openshell_supervisor_process::process::ResolvedWorkspace::new(workdir.clone(), false), - ); - - #[cfg_attr(not(target_os = "linux"), allow(unused_mut))] - let (provider_credentials, mut provider_env) = if let Some(bootstrap) = - sidecar_bootstrap.as_ref() - { - let provider_credentials = ProviderCredentialState::from_child_env_snapshot( - bootstrap.provider_env_revision, - bootstrap.provider_child_env.clone(), - ); - (provider_credentials, bootstrap.provider_child_env.clone()) - } else { - // Fetch provider environment variables from the server. - // This is done after loading the policy so the sandbox can still start - // even if provider env fetch fails (graceful degradation). - let ( - provider_env_revision, - provider_env, - provider_credential_expires_at_ms, - dynamic_credentials, - static_credential_bindings, - non_secret_environment_keys, - ) = if let (Some(id), Some(endpoint)) = (&sandbox_id, &openshell_endpoint) { - match openshell_core::grpc_client::fetch_provider_environment(endpoint, id).await { - Ok(result) => { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .message(format!( - "Fetched provider environment [env_count:{}]", - result.environment.len() - )) - .build() - ); - ( - result.provider_env_revision, - result.environment, - result.credential_expires_at_ms, - result.dynamic_credentials, - result.static_credential_bindings, - result.non_secret_environment_keys, - ) - } - Err(e) => { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::High) - .status(StatusId::Failure) - .state(StateId::Disabled, "fail_closed") - .message(format!( - "Failed to fetch provider environment; no provider credentials are active: {e}" - )) - .build() - ); - ( - 0, - std::collections::HashMap::new(), - std::collections::HashMap::new(), - std::collections::HashMap::new(), - std::collections::HashMap::new(), - Vec::new(), - ) - } - } - } else { - ( - 0, - std::collections::HashMap::new(), - std::collections::HashMap::new(), - std::collections::HashMap::new(), - std::collections::HashMap::new(), - Vec::new(), - ) - }; - - let dynamic_credentials_fallback = dynamic_credentials.clone(); - let provider_credentials = match ProviderCredentialState::from_bound_environment( - provider_env_revision, - provider_env, - provider_credential_expires_at_ms, - dynamic_credentials, - static_credential_bindings, - non_secret_environment_keys, - ) { - Ok(credentials) => credentials, - Err(error) => { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::High) - .status(StatusId::Failure) - .state(StateId::Disabled, "fail_closed") - .message(format!( - "Rejected provider environment bindings; static provider credentials were revoked; fetched dynamic token grants remain active: {error}" - )) - .build() - ); - ProviderCredentialState::from_environment( - provider_env_revision, - std::collections::HashMap::new(), - std::collections::HashMap::new(), - dynamic_credentials_fallback, - ) - } - }; - let provider_env = provider_credentials.child_env_with_gcp_resolved(); - (provider_credentials, provider_env) - }; - - if credential_gating_unavailable( - &loaded_policy_origin, - provider_credentials.resolver().is_some(), - network_enabled, - ) { - report_credential_gating_unavailable(); - } - - // Canonical-process overrides are deliberately applied only to the main - // child. Keep the provider snapshot pristine because Kubernetes forwards - // it to the process sidecar for later exec/editor/SFTP children. - - // Shared agent-proposals feature flag. Seed from the same initial settings - // snapshot that produced the policy so networking and process setup agree - // before the poll loop starts reconciling later changes. - let agent_proposals = AgentProposals::new(initial_agent_proposals_enabled); - - let process_control_writer = process_control_connection - .as_ref() - .map(|connection| connection.writer.clone()); - let process_exit_ack = Arc::new(tokio::sync::Mutex::new(None)); - let initial_provider_env_generation = sidecar_bootstrap - .as_ref() - .map_or(0, |bootstrap| bootstrap.provider_env_generation); - let mut process_control_closed = None; - if let Some(connection) = process_control_connection { - process_control_closed = Some(connection.closed); - spawn_sidecar_control_update_watcher( - connection.updates, - provider_credentials.clone(), - agent_proposals.clone(), - Arc::clone(&process_exit_ack), - initial_provider_env_generation, - ); - } - - // Shared PID: set after process spawn so the proxy can look up - // the entrypoint process's /proc/net/tcp for identity binding. - let entrypoint_pid = Arc::new(AtomicU32::new(0)); - - // Create the workload's network namespace. It is shared infrastructure: - // the proxy binds to its host-side veth IP, the bypass monitor reads - // /dev/kmsg from inside it, and the workload child / SSH sessions enter - // it via setns(). The RAII handle lives in this frame for the duration - // of the sandbox. - #[cfg(target_os = "linux")] - let netns = if network_enabled && !sidecar_network_enforcement { - openshell_supervisor_process::netns::create_netns_for_proxy(&policy)? - } else { - None - }; - - #[cfg(target_os = "linux")] - let transparent_tcp_requested = opa_engine - .as_ref() - .map(|engine| engine.policy_dns_eligibility_snapshot()) - .transpose()? - .is_some_and(|snapshot| !snapshot.endpoints.is_empty()); - #[cfg(target_os = "linux")] - let runtime_capabilities = - std::env::var(openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES).ok(); - #[cfg(target_os = "linux")] - let transparent_tcp_capable = has_network_runtime_capability( - runtime_capabilities.as_deref(), - openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY, - ); - #[cfg(not(target_os = "linux"))] - let transparent_tcp_capable = false; - #[cfg(target_os = "linux")] - let transparent_runtime = if transparent_tcp_requested { - if !transparent_tcp_capable { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Disabled, "unsupported_runtime") - .message( - "Policy DNS and transparent TCP unavailable: runtime capability is missing" - ) - .build() - ); - return Err(miette::miette!( - "policy contains protocol: tcp endpoints, but the selected runtime does not advertise policy DNS and transparent TCP support" - )); - } - if sidecar_network_enforcement { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Disabled, "unsupported_topology") - .message("Policy DNS and transparent TCP unavailable: sidecar topology is unsupported") - .build() - ); - return Err(miette::miette!( - "policy DNS and transparent TCP are not yet supported by the sidecar topology" - )); - } - let namespace = netns.as_ref().ok_or_else(|| { - miette::miette!("policy DNS and transparent TCP require a workload network namespace") - })?; - let listeners = namespace - .bind_transparent_tcp_listeners() - .await - .into_diagnostic() - .wrap_err("failed to bind transparent TCP listeners")?; - let (dns_udp, dns_tcp) = namespace - .bind_policy_dns_sockets() - .await - .into_diagnostic() - .wrap_err("failed to bind policy DNS listeners")?; - let proxy_port = policy - .network - .proxy - .as_ref() - .and_then(|proxy| proxy.http_addr) - .map_or(3128, |address| address.port()); - let runtime = openshell_supervisor_network::run::TransparentRuntimeSetup::new( - listeners, - dns_udp, - dns_tcp, - sandbox_id.as_deref(), - )?; - let (ipv4_cidr, ipv6_cidr) = runtime.synthetic_cidrs(); - namespace.install_transparent_tcp_rules(proxy_port, &ipv4_cidr, &ipv6_cidr)?; - Some(runtime) - } else { - None - }; - #[cfg(target_os = "linux")] - let transparent_tcp_substrate_ready = transparent_runtime.is_some(); - #[cfg(not(target_os = "linux"))] - let transparent_tcp_substrate_ready = false; - // The denial channel is owned by the orchestrator: the proxy (in the - // networking leaf) and the bypass monitor (in the process leaf) both - // produce DenialEvents that the denial aggregator (orchestrator-side) - // consumes via the matching receiver. Both leaves are pure producers; - // the orchestrator owns the consumer task spawned below. - let (denial_tx, denial_rx, bypass_denial_tx): ( - Option>, - _, - Option>, - ) = if sandbox_id.is_some() { - let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); - let bypass_tx = tx.clone(); - (Some(tx), Some(rx), Some(bypass_tx)) - } else { - (None, None, None) - }; - #[cfg(not(target_os = "linux"))] - drop(bypass_denial_tx); - - // Anonymous activity channel: same orchestrator-owned pattern as the - // denial channel. The proxy and the bypass monitor both emit per-event - // activity records; the orchestrator-side aggregator drains, sanitizes, - // and flushes anonymous summaries to the gateway. - let (activity_tx, activity_rx, bypass_activity_tx) = if sandbox_id.is_some() { - let (tx, rx) = - tokio::sync::mpsc::channel(openshell_core::activity::ACTIVITY_EVENT_QUEUE_CAPACITY); - let bypass_tx = tx.clone(); - (Some(tx), Some(rx), Some(bypass_tx)) - } else { - (None, None, None) - }; - #[cfg(not(target_os = "linux"))] - drop(bypass_activity_tx); - - // Workspace watch: the policy poll loop learns the workspace from - // GetSandboxConfig and broadcasts it. Flush tasks and the policy.local - // API read the current value so proposals target the correct workspace. - let (workspace_tx, workspace_rx) = tokio::sync::watch::channel(String::new()); - - let mut networking = if network_enabled { - #[cfg(target_os = "linux")] - let proxy_bind_ip = netns - .as_ref() - .map(openshell_supervisor_process::netns::NetworkNamespace::host_ip); - #[cfg(not(target_os = "linux"))] - let proxy_bind_ip: Option = None; - - Some( - openshell_supervisor_network::run::run_networking( - &policy, - proxy_bind_ip, - opa_engine.as_ref(), - retained_proto.as_ref(), - entrypoint_pid.clone(), - process_enabled, - &provider_credentials, - sandbox_id.as_deref(), - sandbox_name_for_agg.as_deref(), - openshell_endpoint_for_proxy.as_deref(), - denial_tx, - activity_tx, - agent_proposals.clone(), - workspace_rx.clone(), - &upstream_proxy_args, - #[cfg(target_os = "linux")] - transparent_runtime, - ) - .await?, - ) - } else { - None - }; - - #[cfg(target_os = "linux")] - let sidecar_control_server = if network_enabled && sidecar_network_enforcement { - if !matches!(policy.network.mode, NetworkMode::Proxy) { - return Err(miette::miette!( - "sidecar network enforcement requires proxy network mode" - )); - } - let socket = sidecar_control_socket().ok_or_else(|| { - miette::miette!( - "{} is required for sidecar topology", - openshell_core::sandbox_env::SIDECAR_CONTROL_SOCKET - ) - })?; - let proto = retained_proto.as_ref().ok_or_else(|| { - miette::miette!( - "sidecar topology requires gateway policy data for the process supervisor" - ) - })?; - let ca_paths = networking.as_ref().and_then(|n| n.ca_file_paths.clone()); - Some(sidecar_control::spawn_server( - &socket, - sidecar_control::BootstrapData { - policy_proto: proto.clone(), - provider_env_revision: provider_credentials.snapshot().revision, - provider_env_generation: 0, - provider_child_env: provider_env.clone(), - agent_proposals_enabled: agent_proposals.enabled(), - proxy_ca_cert_path: ca_paths.as_ref().map(|paths| paths.0.clone()), - proxy_ca_bundle_path: ca_paths.as_ref().map(|paths| paths.1.clone()), - }, - sidecar_expected_peer()?, - )?) - } else { - None - }; - #[cfg(not(target_os = "linux"))] - let sidecar_control_server: Option = None; - - let sidecar_control_publisher = sidecar_control_server - .as_ref() - .map(sidecar_control::ServerHandle::publisher); - - #[cfg(target_os = "linux")] - let mut sidecar_control_task = None; - - #[cfg(target_os = "linux")] - if network_enabled - && sidecar_network_enforcement - && let Some(server) = sidecar_control_server - { - let trusted_ssh_socket_path = ssh_socket_path.clone().ok_or_else(|| { - miette::miette!( - "{} is required for sidecar network topology", - openshell_core::sandbox_env::SSH_SOCKET_PATH - ) - })?; - let (entrypoint_rx, connection_task) = server.into_runtime_parts(); - sidecar_control_task = Some(connection_task); - spawn_sidecar_entrypoint_handler( - entrypoint_rx, - SidecarEntrypointHandler { - entrypoint_pid: entrypoint_pid.clone(), - opa_engine: opa_engine.clone(), - retained_proto: retained_proto.clone(), - openshell_endpoint: openshell_endpoint.clone(), - sandbox_id: sandbox_id.clone(), - trusted_ssh_socket_path: std::path::PathBuf::from(trusted_ssh_socket_path), - control_publisher: sidecar_control_publisher.clone(), - }, - ); - } - - #[cfg(not(target_os = "linux"))] - if network_enabled && sidecar_network_enforcement { - return Err(miette::miette!( - "sidecar network enforcement is only supported on Linux" - )); - } - - // Spawn the denial-aggregator flush task. The aggregator drains denial - // events from the proxy + bypass monitor, batches them, and ships - // summaries to the gateway via `SubmitPolicyAnalysis`. - if let (Some(rx), Some(endpoint)) = (denial_rx, openshell_endpoint_for_proxy.as_deref()) { - // SubmitPolicyAnalysis resolves by sandbox *name*, not UUID — fall - // back to the ID when the name isn't set. - let agg_name = sandbox_name_for_agg - .clone() - .or_else(|| sandbox_id.clone()) - .unwrap_or_default(); - let agg_endpoint = endpoint.to_string(); - let flush_interval_secs: u64 = std::env::var("OPENSHELL_DENIAL_FLUSH_INTERVAL_SECS") - .ok() - .and_then(|v| v.parse().ok()) - .unwrap_or(10); - - let aggregator = denial_aggregator::DenialAggregator::new(rx, flush_interval_secs); - let denial_workspace_gate = workspace_rx.clone(); - let denial_workspace_rx = workspace_rx.clone(); - - tokio::spawn(async move { - aggregator - .run( - |summaries| { - let endpoint = agg_endpoint.clone(); - let sandbox_name = agg_name.clone(); - let workspace = denial_workspace_rx.borrow().clone(); - async move { - if let Err(e) = flush_proposals_to_gateway( - &endpoint, - &sandbox_name, - &workspace, - summaries, - ) - .await - { - warn!(error = %e, "Failed to flush denial summaries to gateway"); - } - } - }, - move || !denial_workspace_gate.borrow().is_empty(), - ) - .await; - }); - } - - // Spawn the activity-aggregator flush task. The aggregator drains - // anonymous activity events from the proxy, sanitizes deny groups, - // and ships periodic summaries to the gateway. - if let (Some(rx), Some(endpoint)) = (activity_rx, openshell_endpoint_for_proxy.as_deref()) { - let agg_name = sandbox_name_for_agg - .clone() - .or_else(|| sandbox_id.clone()) - .unwrap_or_default(); - let agg_endpoint = endpoint.to_string(); - let flush_interval_secs = activity_aggregator::activity_flush_interval_secs_from_env( - std::env::var("OPENSHELL_ACTIVITY_FLUSH_INTERVAL_SECS") - .ok() - .as_deref(), - ); - - let aggregator = activity_aggregator::ActivityAggregator::new(rx, flush_interval_secs); - let activity_workspace_gate = workspace_rx.clone(); - let activity_workspace_rx = workspace_rx.clone(); - - tokio::spawn(async move { - aggregator - .run( - move |summary| { - let endpoint = agg_endpoint.clone(); - let sandbox_name = agg_name.clone(); - let workspace = activity_workspace_rx.borrow().clone(); - async move { - if let Err(e) = flush_activity_to_gateway( - &endpoint, - &sandbox_name, - &workspace, - summary, - ) - .await - { - warn!(error = %e, "Failed to flush activity summary to gateway"); - } - } - }, - move || !activity_workspace_gate.borrow().is_empty(), - ) - .await; - }); - } - - // Spawn background policy poll task (gRPC mode only). - if !process_uses_sidecar_control - && let (Some(id), Some(endpoint), Some(engine)) = ( - sandbox_id.as_deref(), - openshell_endpoint.as_deref(), - opa_engine.as_ref(), - ) - { - let poll_id = id.to_string(); - let poll_endpoint = endpoint.to_string(); - let poll_engine = engine.clone(); - let poll_ocsf_enabled = ocsf_enabled.clone(); - let poll_ocsf_schema_version = ocsf_schema_version.clone(); - let poll_pid = entrypoint_pid.clone(); - let poll_provider_credentials = provider_credentials.clone(); - let poll_policy_local = networking.as_ref().map(|n| n.policy_local_ctx.clone()); - let poll_interval_secs: u64 = std::env::var("OPENSHELL_POLICY_POLL_INTERVAL_SECS") - .ok() - .and_then(|v| v.parse().ok()) - .unwrap_or(10); - let poll_ctx = PolicyPollLoopContext { - endpoint: poll_endpoint, - sandbox_id: poll_id, - opa_engine: poll_engine, - loaded_policy_origin, - entrypoint_pid: poll_pid, - interval_secs: poll_interval_secs, - ocsf_enabled: poll_ocsf_enabled, - ocsf_schema_version: poll_ocsf_schema_version, - provider_credentials: poll_provider_credentials, - policy_local_ctx: poll_policy_local, - agent_proposals: agent_proposals.clone(), - middleware_registry_status, - sidecar_control_publisher: sidecar_control_publisher.clone(), - workspace_tx, - extension_credentials: extension_credentials.clone(), - extension_authentication_enabled: initial_extension_authentication_enabled, - middleware_connector: default_middleware_connector(), - transparent_tcp: TransparentTcpReloadState { - capable: transparent_tcp_capable, - substrate_ready: transparent_tcp_substrate_ready, - }, - }; - - tokio::spawn(async move { - if let Err(e) = run_policy_poll_loop(poll_ctx).await { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .message(format!("Policy poll loop exited with error: {e}")) - .build() - ); - } - }); - } - - // Start GCE metadata loopback server inside the network namespace so - // Go's cloud.google.com/go/compute/metadata (which bypasses HTTP_PROXY) - // can reach it via direct TCP. Must start before the process leaf so SSH - // sessions also see corrected env vars on bind failure. - #[cfg(target_os = "linux")] - if let Some(ns) = netns.as_ref() - && provider_credentials - .snapshot() - .child_env - .contains_key("GCE_METADATA_HOST") - { - let ctx = google_cloud_metadata::MetadataContext::new(provider_credentials.clone()); - let (ready_tx, ready_rx) = tokio::sync::oneshot::channel(); - match ns - .bind_tcp_in_netns(openshell_core::google_cloud::METADATA_LOOPBACK_ADDR) - .await - { - Ok(listener) => { - tokio::spawn(metadata_server::run(listener, ctx, ready_tx)); - if let Ok(Ok(addr)) = timeout(Duration::from_secs(5), ready_rx).await { - info!(addr = %addr, "GCE metadata loopback server ready"); - } else { - warn!("GCE metadata server failed to become ready, removing metadata env vars"); - provider_env.remove("GCE_METADATA_HOST"); - provider_env.remove("GCE_METADATA_IP"); - provider_env.remove("METADATA_SERVER_DETECTION"); - provider_credentials.remove_env_key("GCE_METADATA_HOST"); - } - } - Err(e) => { - warn!(error = %e, "GCE metadata server bind failed, Go SDK may not discover credentials"); - provider_env.remove("GCE_METADATA_HOST"); - provider_env.remove("GCE_METADATA_IP"); - provider_env.remove("METADATA_SERVER_DETECTION"); - provider_credentials.remove_env_key("GCE_METADATA_HOST"); - } - } - } - - let process_policy = process_policy_for_topology(&policy, sidecar_network_enforcement)?; - let main_env = provider_env.clone(); - let sidecar_bootstrap_ca_file_paths = sidecar_bootstrap.as_ref().and_then(|bootstrap| { - bootstrap - .proxy_ca_cert_path - .clone() - .zip(bootstrap.proxy_ca_bundle_path.clone()) - }); - - let proxy_exited: Pin + Send>> = if let Some(rx) = networking - .as_mut() - .and_then(|n| n.proxy.as_mut()) - .and_then(ProxyHandle::take_exit_receiver) - { - Box::pin(async { - let _ = rx.await; - }) - } else { - Box::pin(std::future::pending()) - }; - tokio::pin!(proxy_exited); - - let exit_code = if process_enabled { - let ca_file_paths = networking - .as_ref() - .and_then(|n| n.ca_file_paths.clone()) - .or_else(|| { - if sidecar_network_enforcement { - sidecar_bootstrap_ca_file_paths - .clone() - .or_else(sidecar_ca_file_paths) - } else { - None - } - }); - - let (ssh_exit_tx, ssh_exit_rx) = if ssh_socket_path.is_some() { - let (tx, rx) = tokio::sync::oneshot::channel::<()>(); - (Some(tx), Some(rx)) - } else { - (None, None) - }; - let ssh_exited: Pin + Send>> = if let Some(rx) = ssh_exit_rx { - Box::pin(async { - let _ = rx.await; - }) - } else { - Box::pin(std::future::pending()) - }; - tokio::pin!(ssh_exited); - - let entrypoint_started_tx = - if process_uses_sidecar_control && let Some(writer) = process_control_writer.clone() { - let (tx, rx) = tokio::sync::oneshot::channel(); - tokio::spawn(async move { - match rx.await { - Ok((pid, instance_id)) => { - if let Err(err) = - sidecar_control::send_entrypoint_started(&writer, pid, instance_id) - .await - { - warn!(error = %err, "Failed to send sidecar entrypoint event"); - } - } - Err(_closed) => { - debug!("Entrypoint exited before sidecar entrypoint event was sent"); - } - } - }); - Some(tx) - } else { - None - }; - let sidecar_exit_tx = if process_uses_sidecar_control - && let Some(writer) = process_control_writer.clone() - { - let exit_ack = Arc::clone(&process_exit_ack); - let (tx, mut rx) = tokio::sync::mpsc::channel::< - openshell_supervisor_process::run::SidecarExitReport, - >(1); - tokio::spawn(async move { - while let Some(report) = rx.recv().await { - match report { - openshell_supervisor_process::run::SidecarExitReport::Exited { - instance_id, - exit_code, - ack, - } => { - let (durable_tx, durable_rx) = tokio::sync::oneshot::channel(); - *exit_ack.lock().await = Some((instance_id.clone(), durable_tx)); - let result = match sidecar_control::send_main_process_exited( - &writer, - instance_id, - exit_code, - ) - .await - { - Ok(()) => durable_rx.await.map_err(|_| { - "sidecar durable exit acknowledgement closed".to_string() - }), - Err(error) => Err(error.to_string()), - }; - let _ = ack.send(result); - } - openshell_supervisor_process::run::SidecarExitReport::Finalized { - instance_id, - ack, - } => { - let result = - sidecar_control::send_main_process_finalized(&writer, instance_id) - .await - .map_err(|error| error.to_string()); - let _ = ack.send(result); - } - } - } - }); - Some(tx) - } else { - None - }; - - let process = openshell_supervisor_process::run::run_process( - program, - args, - workspace, - timeout_secs, - interactive, - await_main_process_attachment, - sandbox_id.as_deref(), - openshell_endpoint.as_deref(), - ssh_socket_path, - sidecar_network_enforcement, - ssh_exit_tx, - &process_policy, - resolved_process_identity, - process_enforcement_mode, - entrypoint_pid, - entrypoint_started_tx, - sidecar_exit_tx, - provider_credentials, - main_env, - ca_file_paths, - agent_proposals.clone(), - #[cfg(target_os = "linux")] - netns.as_ref(), - #[cfg(target_os = "linux")] - bypass_denial_tx, - #[cfg(target_os = "linux")] - bypass_activity_tx, - ); - - if let Some(control_closed) = process_control_closed.as_mut() { - tokio::select! { - result = process => result?, - _ = control_closed => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "Authoritative network-sidecar control channel closed; terminating process container" - ) - .build() - ); - return Err(miette::miette!( - "authoritative network-sidecar control channel closed" - )); - } - () = &mut proxy_exited => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "Proxy accept loop exited unexpectedly; terminating sandbox" - ) - .build() - ); - return Err(miette::miette!( - "proxy accept loop exited unexpectedly" - )); - } - () = &mut ssh_exited => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "SSH accept loop exited unexpectedly; terminating sandbox" - ) - .build() - ); - return Err(miette::miette!( - "SSH accept loop exited unexpectedly" - )); - } - } - } else { - tokio::select! { - result = process => result?, - () = &mut proxy_exited => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "Proxy accept loop exited unexpectedly; terminating sandbox" - ) - .build() - ); - return Err(miette::miette!( - "proxy accept loop exited unexpectedly" - )); - } - () = &mut ssh_exited => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "SSH accept loop exited unexpectedly; terminating sandbox" - ) - .build() - ); - return Err(miette::miette!( - "SSH accept loop exited unexpectedly" - )); - } - } - } - } else { - // Network-only sidecar mode: keep the proxy and its background - // tasks alive (held via the `networking` value) until shutdown. If the - // sole authenticated process-supervisor control connection closes, - // exit non-zero so Kubernetes restarts the network sidecar and creates - // a fresh one-client bootstrap listener for the restarted agent. - #[cfg(target_os = "linux")] - if let Some(control_task) = sidecar_control_task { - tokio::select! { - () = wait_for_shutdown_signal() => 0, - result = control_task => { - warn!(?result, "Authoritative sidecar control channel exited; restarting sidecar"); - 1 - } - () = &mut proxy_exited => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "Proxy accept loop exited unexpectedly; terminating sandbox" - ) - .build() - ); - return Err(miette::miette!( - "proxy accept loop exited unexpectedly" - )); - } - } - } else { - tokio::select! { - () = wait_for_shutdown_signal() => 0, - () = &mut proxy_exited => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "Proxy accept loop exited unexpectedly; terminating sandbox" - ) - .build() - ); - return Err(miette::miette!( - "proxy accept loop exited unexpectedly" - )); - } - } - } - #[cfg(not(target_os = "linux"))] - { - tokio::select! { - () = wait_for_shutdown_signal() => 0, - () = &mut proxy_exited => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message( - "Proxy accept loop exited unexpectedly; terminating sandbox" - ) - .build() - ); - return Err(miette::miette!( - "proxy accept loop exited unexpectedly" - )); - } - } - } - }; - - // Drop networking explicitly so the proxy + bypass monitor RAII - // handles tear down before we return. - drop(networking); - - Ok(exit_code) -} - -/// Wait for SIGINT or SIGTERM. Used in network-only mode where there is -/// no entrypoint child whose lifetime drives the supervisor's exit. -async fn wait_for_shutdown_signal() { - #[cfg(unix)] - { - use tokio::signal::unix::{SignalKind, signal}; - let mut sigterm = match signal(SignalKind::terminate()) { - Ok(s) => s, - Err(e) => { - tracing::warn!( - error = %e, - "Failed to install SIGTERM handler; waiting on SIGINT only" - ); - let _ = tokio::signal::ctrl_c().await; - return; - } - }; - tokio::select! { - _ = tokio::signal::ctrl_c() => { - info!("Received SIGINT, shutting down network-only supervisor"); - } - _ = sigterm.recv() => { - info!("Received SIGTERM, shutting down network-only supervisor"); - } - } - } - #[cfg(not(unix))] - { - let _ = tokio::signal::ctrl_c().await; - info!("Received Ctrl-C, shutting down network-only supervisor"); - } -} - -fn sidecar_network_enforcement_enabled() -> bool { - std::env::var(openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE) - .is_ok_and(|value| value == SIDECAR_NETWORK_ENFORCEMENT_MODE) -} - -fn process_enforcement_mode() -> ProcessEnforcementMode { - match std::env::var(openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY) - .ok() - .as_deref() - { - Some("sidecar") => ProcessEnforcementMode::NetworkOnly, - _ => ProcessEnforcementMode::Full, - } -} - -fn sidecar_control_socket() -> Option { - std::env::var(openshell_core::sandbox_env::SIDECAR_CONTROL_SOCKET) - .ok() - .filter(|path| !path.is_empty()) - .map(std::path::PathBuf::from) -} - -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -fn sidecar_expected_peer() -> Result { - fn required_numeric_env(name: &str) -> Result { - let value = std::env::var(name) - .into_diagnostic() - .wrap_err_with(|| format!("{name} is required for sidecar control authentication"))?; - value.parse::().into_diagnostic().wrap_err_with(|| { - format!("{name} must be a numeric ID for sidecar control authentication") - }) - } - - Ok(sidecar_control::ExpectedPeer { - uid: required_numeric_env(openshell_core::sandbox_env::SANDBOX_UID)?, - gid: required_numeric_env(openshell_core::sandbox_env::SANDBOX_GID)?, - }) -} - -type LoadedPolicyBundle = ( - SandboxPolicy, - Option>, - Option, - LoadedPolicyOrigin, -); - -type MainProcessExitAckWaiter = - Arc)>>>; - -fn load_policy_from_sidecar_bootstrap( - bootstrap: &sidecar_control::BootstrapData, -) -> Result { - let proto = bootstrap.policy_proto.clone(); - let opa_engine = Some(Arc::new(OpaEngine::from_proto(&proto)?)); - let policy = SandboxPolicy::try_from(proto.clone())?; - info!("Loaded sidecar policy from control socket bootstrap"); - Ok(( - policy, - opa_engine, - Some(proto), - LoadedPolicyOrigin::Gateway { - revision: None, - has_last_valid_policy: true, - }, - )) -} - -fn spawn_sidecar_control_update_watcher( - mut updates: tokio::sync::mpsc::UnboundedReceiver, - provider_credentials: ProviderCredentialState, - agent_proposals: AgentProposals, - exit_ack: MainProcessExitAckWaiter, - mut provider_env_generation: u64, -) -> tokio::task::JoinHandle<()> { - tokio::spawn(async move { - while let Some(update) = updates.recv().await { - match update { - sidecar_control::ControlUpdate::ProviderEnv { - revision, - generation, - provider_child_env, - } => { - if generation <= provider_env_generation { - continue; - } - let env_count = provider_credentials - .install_child_env_snapshot(revision, provider_child_env); - provider_env_generation = generation; - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("provider_env_revision", serde_json::json!(revision)) - .unmapped("provider_env_generation", serde_json::json!(generation)) - .message(format!( - "Sidecar provider environment refreshed [revision:{revision} env_count:{env_count}]" - )) - .build() - ); - } - sidecar_control::ControlUpdate::Policy { - policy_proto, - policy_hash, - config_revision, - } => { - debug!( - version = policy_proto.version, - policy_hash, - config_revision, - "Received sidecar policy update for process supervisor" - ); - } - sidecar_control::ControlUpdate::AgentProposals { - enabled, - config_revision, - } => { - apply_agent_proposals_enabled( - &agent_proposals, - enabled, - "sidecar control", - Some(config_revision), - None, - skills::install_static_skills, - ); - } - sidecar_control::ControlUpdate::MainProcessExitAck { instance_id } => { - let mut waiter = exit_ack.lock().await; - if waiter - .as_ref() - .is_some_and(|(expected, _)| expected == &instance_id) - && let Some((_, ack)) = waiter.take() - { - let _ = ack.send(()); - } - } - } - } - }) -} - +mod accept_interrupt; +pub mod boundary_exec; +pub mod boundary_io; +mod boundary_server; +pub mod child_env; #[cfg(target_os = "linux")] -struct SidecarEntrypointHandler { - entrypoint_pid: Arc, - opa_engine: Option>, - retained_proto: Option, - openshell_endpoint: Option, - sandbox_id: Option, - trusted_ssh_socket_path: std::path::PathBuf, - control_publisher: Option, -} - +pub(crate) mod delegated; +#[cfg(unix)] +pub mod identity; #[cfg(target_os = "linux")] -fn spawn_sidecar_entrypoint_handler( - mut entrypoint_rx: tokio::sync::mpsc::Receiver, - handler: SidecarEntrypointHandler, -) { - tokio::spawn(async move { - let SidecarEntrypointHandler { - entrypoint_pid, - opa_engine, - retained_proto, - openshell_endpoint, - sandbox_id, - trusted_ssh_socket_path, - control_publisher, - } = handler; - let mut session_started = false; - let mut session_task: Option> = None; - let mut trusted_supervisor_pid = None; - let terminating = Arc::new(AtomicBool::new(false)); - while let Some(started) = entrypoint_rx.recv().await { - if started.finalized { - if let (Some(endpoint), Some(id)) = - (openshell_endpoint.as_ref(), sandbox_id.as_ref()) - { - let mut delay = Duration::from_millis(250); - loop { - match openshell_supervisor_process::supervisor_session::finalize_main_process_exit( - endpoint, - id, - &started.instance_id, - ) - .await - { - Ok(()) => break, - Err(error) => { - warn!(%error, "sidecar main-process finalization failed; retrying"); - tokio::time::sleep(delay).await; - delay = (delay * 2).min(Duration::from_secs(2)); - } - } - } - } - terminating.store(true, Ordering::Release); - if let Some(task) = session_task.take() { - task.abort(); - } - break; - } - if let Some(exit_code) = started.exit_code { - if let (Some(endpoint), Some(id)) = - (openshell_endpoint.as_ref(), sandbox_id.as_ref()) - { - let mut delay = Duration::from_millis(250); - loop { - match openshell_supervisor_process::supervisor_session::report_main_process_exit( - endpoint, - id, - &started.instance_id, - exit_code, - ) - .await - { - Ok(()) => break, - Err(error) => { - warn!(%error, "sidecar main-process exit report failed; retrying"); - tokio::time::sleep(delay).await; - delay = (delay * 2).min(Duration::from_secs(2)); - } - } - } - if let Some(publisher) = control_publisher.as_ref() { - publisher.publish_main_process_exit_ack(started.instance_id.clone()); - } - } - continue; - } - entrypoint_pid.store(started.pid, Ordering::Release); - if started.start_session { - info!( - pid = started.pid, - ssh_socket = %trusted_ssh_socket_path.display(), - "Sidecar process supervisor reported entrypoint start" - ); - } else { - trusted_supervisor_pid = Some(started.pid); - info!( - pid = started.pid, - "Sidecar process supervisor reported initial process anchor" - ); - } - - if let (Some(engine), Some(proto)) = (opa_engine.as_ref(), retained_proto.as_ref()) { - match engine.reload_from_proto_with_pid(proto, started.pid) { - Ok(()) => info!( - pid = started.pid, - "Policy binary symlink resolution complete for sidecar process anchor" - ), - Err(err) => warn!( - error = %err, - pid = started.pid, - "Failed to rebuild OPA engine with sidecar process anchor PID" - ), - } - } - - if started.start_session - && !session_started - && let (Some(endpoint), Some(id)) = - (openshell_endpoint.as_ref(), sandbox_id.as_ref()) - { - let Some(supervisor_pid) = trusted_supervisor_pid else { - warn!( - pid = started.pid, - "Ignoring sidecar entrypoint event before authenticated supervisor anchor" - ); - continue; - }; - session_task = Some(openshell_supervisor_process::supervisor_session::spawn( - endpoint.clone(), - id.clone(), - trusted_ssh_socket_path.clone(), - None, - Some(supervisor_pid), - Arc::clone(&terminating), - started.instance_id.clone(), - )); - session_started = true; - info!("sidecar supervisor session task spawned"); - } - } - terminating.store(true, Ordering::Release); - }); -} - -fn sidecar_ca_file_paths() -> Option<(std::path::PathBuf, std::path::PathBuf)> { - let tls_dir = std::env::var(openshell_core::sandbox_env::PROXY_TLS_DIR) - .unwrap_or_else(|_| SIDECAR_TLS_DIR.to_string()); - let cert = std::path::Path::new(&tls_dir).join(SIDECAR_CA_CERT); - let bundle = std::path::Path::new(&tls_dir).join(SIDECAR_CA_BUNDLE); - (cert.exists() && bundle.exists()).then_some((cert, bundle)) -} - -fn process_policy_for_topology( - policy: &SandboxPolicy, - sidecar_network_enforcement: bool, -) -> Result { - let mut process_policy = policy.clone(); - if sidecar_network_enforcement && matches!(process_policy.network.mode, NetworkMode::Proxy) { - let proxy = process_policy - .network - .proxy - .get_or_insert(ProxyPolicy { http_addr: None }); - if proxy.http_addr.is_none() { - proxy.http_addr = Some(SIDECAR_PROCESS_PROXY_ADDR.parse().into_diagnostic()?); - } - } - Ok(process_policy) -} - -/// Flush aggregated denial summaries to the gateway via `SubmitPolicyAnalysis`. -async fn flush_proposals_to_gateway( - endpoint: &str, - sandbox_name: &str, - workspace: &str, - summaries: Vec, -) -> Result<()> { - use openshell_core::grpc_client::CachedOpenShellClient; - use openshell_core::proto::{DenialSummary, L7RequestSample}; - - let client = CachedOpenShellClient::connect(endpoint).await?; - client.set_workspace(workspace.to_string()); - - let proto_summaries: Vec = summaries - .into_iter() - .map(|s| DenialSummary { - sandbox_id: String::new(), - host: s.host, - port: u32::from(s.port), - binary: s.binary, - ancestors: s.ancestors, - deny_reason: s.deny_reason, - first_seen_ms: s.first_seen_ms, - last_seen_ms: s.last_seen_ms, - count: s.count, - suppressed_count: 0, - total_count: s.count, - sample_cmdlines: s.sample_cmdlines, - binary_sha256: String::new(), - persistent: false, - denial_stage: s.denial_stage, - l7_request_samples: s - .l7_samples - .into_iter() - .map(|l| L7RequestSample { - method: l.method, - path: l.path, - decision: "deny".to_string(), - count: l.count, - }) - .collect(), - l7_inspection_active: false, - }) - .collect(); - - // Run the mechanistic mapper sandbox-side to generate proposals. - // The gateway is a thin persistence + validation layer — it never - // generates proposals itself. - let proposals = mechanistic_mapper::generate_proposals(&proto_summaries); - - info!( - sandbox_name = %sandbox_name, - summaries = proto_summaries.len(), - proposals = proposals.len(), - "Flushed denial analysis to gateway" - ); - - client - .submit_policy_analysis( - sandbox_name, - proto_summaries, - proposals, - Vec::new(), - "mechanistic", - ) - .await?; - - Ok(()) -} - -/// Flush an anonymous activity summary to the gateway via `SubmitPolicyAnalysis`. -async fn flush_activity_to_gateway( - endpoint: &str, - sandbox_name: &str, - workspace: &str, - summary: activity_aggregator::FlushableActivitySummary, -) -> Result<()> { - use openshell_core::grpc_client::CachedOpenShellClient; - use openshell_core::proto::{DenialGroupCount, NetworkActivitySummary}; - - let client = CachedOpenShellClient::connect(endpoint).await?; - client.set_workspace(workspace.to_string()); - - let proto_summary = NetworkActivitySummary { - network_activity_count: summary.network_activity_count, - denied_action_count: summary.denied_action_count, - denials_by_group: summary - .denials_by_group - .into_iter() - .map(|(group, count)| DenialGroupCount { - deny_group: group, - denied_count: count, - }) - .collect(), - }; - - info!( - sandbox_name = %sandbox_name, - network_activity_count = proto_summary.network_activity_count, - denied_action_count = proto_summary.denied_action_count, - "Flushed activity summary to gateway" - ); - - client - .submit_policy_analysis( - sandbox_name, - Vec::new(), - Vec::new(), - vec![proto_summary], - "activity", - ) - .await?; - - Ok(()) -} - -// ============================================================================ -// Baseline filesystem path enrichment -// ============================================================================ - -/// Minimum read-only paths required for a proxy-mode sandbox child process to -/// function: dynamic linker, shared libraries, DNS resolution, CA certs, -/// Python venv, openshell logs, process info, and random bytes. -/// -/// `/proc` and `/dev/urandom` are included here for the same reasons they -/// appear in `restrictive_default_policy()`: virtually every process needs -/// them. Before the Landlock per-path fix (#677) these were effectively free -/// because a missing path silently disabled the entire ruleset; now they must -/// be explicit. -const PROXY_BASELINE_READ_ONLY: &[&str] = &[ - "/usr", - "/lib", - "/etc", - "/app", - "/var/log", - "/proc", - "/dev/urandom", -]; - -/// Minimum read-write paths required for a proxy-mode sandbox child process. -/// The active workspace is granted separately through `include_workdir`. -const PROXY_BASELINE_READ_WRITE: &[&str] = &["/tmp"]; - -/// GPU read-only paths. -/// -/// `/run/nvidia-persistenced`: NVML tries to connect to the persistenced -/// socket at init time. If the directory exists but Landlock denies traversal -/// (EACCES vs ECONNREFUSED), NVML returns `NVML_ERROR_INSUFFICIENT_PERMISSIONS` -/// even though the daemon is optional. Only read/traversal access is needed. -/// -/// `/usr/lib/wsl`: On WSL2, CDI bind-mounts GPU libraries (libdxcore.so, -/// libcuda.so.1.1, etc.) into paths under `/usr/lib/wsl/`. Although `/usr` -/// is already in `PROXY_BASELINE_READ_ONLY`, individual file bind-mounts may -/// not be covered by the parent-directory Landlock rule when the mount crosses -/// a filesystem boundary. Listing `/usr/lib/wsl` explicitly ensures traversal -/// is permitted regardless of Landlock's cross-mount behaviour. -const GPU_BASELINE_READ_ONLY: &[&str] = &[ - "/run/nvidia-persistenced", - "/usr/lib/wsl", // WSL2: CDI-injected GPU library directory -]; - -/// GPU read-write paths (static). -/// -/// `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools`, -/// `/dev/nvidia-modeset`: control and UVM devices injected by CDI on native -/// Linux. Landlock restricts `open(2)` on device files even when DAC allows -/// it; these need read-write because NVML/CUDA opens them with `O_RDWR`. -/// These devices do not exist on WSL2 and will be skipped by the existence -/// check in `enrich_proto_baseline_paths()`. -/// -/// `/dev/dxg`: On WSL2, NVIDIA GPUs are exposed through the DXG kernel driver -/// (DirectX Graphics) rather than the native nvidia* devices. CDI injects -/// `/dev/dxg` as the sole GPU device node; it does not exist on native Linux -/// and will be skipped there by the existence check. -/// -/// `/proc`: CUDA writes to `/proc//task//comm` during `cuInit()` -/// to set thread names. Without write access, `cuInit()` returns error 304. -/// Must use `/proc` (not `/proc/self/task`) because Landlock rules bind to -/// inodes and child processes have different procfs inodes than the parent. -/// -/// Per-GPU device files (`/dev/nvidia0`, …) are enumerated at runtime by -/// `enumerate_gpu_device_nodes()` since the count varies. -const GPU_BASELINE_READ_WRITE: &[&str] = &[ - "/dev/nvidiactl", - "/dev/nvidia-uvm", - "/dev/nvidia-uvm-tools", - "/dev/nvidia-modeset", - "/dev/dxg", // WSL2: DXG device (GPU via DirectX kernel driver, injected by CDI) - "/proc", -]; - -/// Returns true if GPU devices are present in the container. -/// -/// Checks both the native Linux NVIDIA control device (`/dev/nvidiactl`) and -/// the WSL2 DXG device (`/dev/dxg`). CDI injects exactly one of these -/// depending on the host kernel; the other will not exist. -fn has_gpu_devices() -> bool { - std::path::Path::new("/dev/nvidiactl").exists() || std::path::Path::new("/dev/dxg").exists() -} - -/// Enumerate per-GPU device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …). -fn enumerate_gpu_device_nodes() -> Vec { - let mut paths = Vec::new(); - if let Ok(entries) = std::fs::read_dir("/dev") { - for entry in entries.flatten() { - let name = entry.file_name(); - let name = name.to_string_lossy(); - if let Some(suffix) = name.strip_prefix("nvidia") { - if suffix.is_empty() || !suffix.chars().all(|c| c.is_ascii_digit()) { - continue; - } - paths.push(entry.path().to_string_lossy().into_owned()); - } - } - } - paths -} - -fn push_unique(paths: &mut Vec, path: String) { - if !paths.iter().any(|p| p == &path) { - paths.push(path); - } -} - -fn collect_baseline_enrichment_paths( - include_proxy: bool, - include_gpu: bool, - gpu_device_nodes: Vec, -) -> (Vec, Vec) { - let mut ro = Vec::new(); - let mut rw = Vec::new(); - - if include_proxy { - for &path in PROXY_BASELINE_READ_ONLY { - push_unique(&mut ro, path.to_string()); - } - for &path in PROXY_BASELINE_READ_WRITE { - push_unique(&mut rw, path.to_string()); - } - } - - if include_gpu { - for &path in GPU_BASELINE_READ_ONLY { - push_unique(&mut ro, path.to_string()); - } - for &path in GPU_BASELINE_READ_WRITE { - push_unique(&mut rw, path.to_string()); - } - for path in gpu_device_nodes { - push_unique(&mut rw, path); - } - } - - // A path promoted to read_write (e.g. /proc for GPU) should not also - // appear in read_only — Landlock handles the overlap correctly but the - // duplicate is confusing when inspecting the effective policy. - ro.retain(|p| !rw.contains(p)); - - (ro, rw) -} - -fn active_baseline_enrichment_paths(include_proxy: bool) -> (Vec, Vec) { - let include_gpu = has_gpu_devices(); - let gpu_device_nodes = if include_gpu { - enumerate_gpu_device_nodes() - } else { - Vec::new() - }; - collect_baseline_enrichment_paths(include_proxy, include_gpu, gpu_device_nodes) -} - -/// Collect all active baseline paths for tests and diagnostics. -/// Returns `(read_only, read_write)` as owned `String` vecs. -#[cfg(test)] -fn baseline_enrichment_paths() -> (Vec, Vec) { - active_baseline_enrichment_paths(true) -} - -fn enrich_proto_baseline_paths_with( - proto: &mut openshell_core::proto::SandboxPolicy, - ro: &[String], - rw: &[String], - path_exists: F, -) -> bool -where - F: Fn(&str) -> bool, -{ - if ro.is_empty() && rw.is_empty() { - return false; - } - - let fs = proto - .filesystem - .get_or_insert_with(|| openshell_core::proto::FilesystemPolicy { - include_workdir: true, - ..Default::default() - }); - - let mut modified = false; - for path in ro { - if !fs.read_only.iter().any(|p| p == path) && !fs.read_write.iter().any(|p| p == path) { - if !path_exists(path) { - debug!( - path, - "Baseline read-only path does not exist, skipping enrichment" - ); - continue; - } - fs.read_only.push(path.clone()); - modified = true; - } - } - for path in rw { - if fs.read_write.iter().any(|p| p == path) { - continue; - } - if !path_exists(path) { - debug!( - path, - "Baseline read-write path does not exist, skipping enrichment" - ); - continue; - } - if fs.read_only.iter().any(|p| p == path) { - if path == "/proc" { - info!( - path, - "Promoting /proc from read-only to read-write for GPU runtime compatibility" - ); - fs.read_only.retain(|p| p != path); - fs.read_write.push(path.clone()); - modified = true; - } - continue; - } - fs.read_write.push(path.clone()); - modified = true; - } - - modified -} - -/// Ensure a proto `SandboxPolicy` includes the baseline filesystem paths -/// required by proxy-mode sandboxes and GPU runtimes. Paths are only added if -/// missing; user-specified paths are never removed. -/// -/// Returns `true` if the policy was modified (caller may want to sync back). -fn enrich_proto_baseline_paths(proto: &mut openshell_core::proto::SandboxPolicy) -> bool { - let (ro, rw) = active_baseline_enrichment_paths(!proto.network_policies.is_empty()); - - // Baseline paths are system-injected, not user-specified. Skip paths - // that do not exist in this container image to avoid noisy warnings from - // Landlock and, more critically, to prevent a single missing baseline - // path from abandoning the entire Landlock ruleset under best-effort - // mode (see issue #664). - let modified = enrich_proto_baseline_paths_with(proto, &ro, &rw, |path| { - std::path::Path::new(path).exists() - }); - - if modified { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "enriched") - .message("Enriched policy with baseline filesystem paths for proxy mode") - .build() - ); - } - - modified -} - -fn strip_proto_provider_policy_entries(proto: &mut openshell_core::proto::SandboxPolicy) -> bool { - openshell_policy::strip_provider_rule_names(proto) -} - -fn proto_sync_payload_for_enriched_policy( - proto: &openshell_core::proto::SandboxPolicy, - enriched: bool, -) -> Option { - if !enriched { - return None; - } - - let mut sync_policy = proto.clone(); - strip_proto_provider_policy_entries(&mut sync_policy); - Some(sync_policy) -} - -/// Ensure a `SandboxPolicy` (Rust type) includes the baseline filesystem -/// paths required by proxy-mode sandboxes and GPU runtimes. Used for the -/// local-file code path where no proto is available. -fn enrich_sandbox_baseline_paths(policy: &mut SandboxPolicy) { - let (ro, rw) = - active_baseline_enrichment_paths(matches!(policy.network.mode, NetworkMode::Proxy)); - if ro.is_empty() && rw.is_empty() { - return; - } - - let mut modified = false; - for path in &ro { - let p = std::path::PathBuf::from(path); - if !policy.filesystem.read_only.contains(&p) && !policy.filesystem.read_write.contains(&p) { - if !p.exists() { - debug!( - path, - "Baseline read-only path does not exist, skipping enrichment" - ); - continue; - } - policy.filesystem.read_only.push(p); - modified = true; - } - } - for path in &rw { - let p = std::path::PathBuf::from(path); - if policy.filesystem.read_only.contains(&p) || policy.filesystem.read_write.contains(&p) { - continue; - } - if !p.exists() { - debug!( - path, - "Baseline read-write path does not exist, skipping enrichment" - ); - continue; - } - policy.filesystem.read_write.push(p); - modified = true; - } - - if modified { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "enriched") - .message("Enriched policy with baseline filesystem paths for proxy mode") - .build() - ); - } -} - -#[cfg(test)] +pub mod main_session; +pub mod managed_children; +#[cfg(target_os = "linux")] +mod network_broker; +#[cfg(all(target_os = "linux", feature = "perf-harness"))] +pub mod perf; +#[cfg(unix)] +pub mod process; +mod pty; +pub mod sandbox; + +/// Results of actively qualifying the admitted workload runtime before the +/// sandbox consumes protected bootstrap material. +#[cfg(target_os = "linux")] +#[derive(Clone, Copy, Debug)] #[allow( - clippy::needless_raw_string_hashes, - clippy::iter_on_single_items, - clippy::similar_names, - clippy::manual_string_new, - clippy::doc_markdown, - reason = "Test code: test fixtures often use idiomatic forms not flagged in production." + clippy::struct_excessive_bools, + reason = "qualification preserves independently exercised security results" )] -mod baseline_tests { - use super::*; - use openshell_core::policy::{FilesystemPolicy, LandlockPolicy, ProcessPolicy}; - use std::path::PathBuf; - - #[test] - fn proc_not_in_both_read_only_and_read_write_when_gpu_present() { - // When GPU devices are present, /proc is promoted to read_write - // (CUDA needs to write /proc//task//comm). It should - // NOT also appear in read_only. - if !has_gpu_devices() { - // Can't test GPU dedup without GPU devices; skip silently. - return; - } - let (ro, rw) = baseline_enrichment_paths(); - assert!( - rw.contains(&"/proc".to_string()), - "/proc should be in read_write when GPU is present" - ); - assert!( - !ro.contains(&"/proc".to_string()), - "/proc should NOT be in read_only when it is already in read_write" - ); - } - - #[test] - fn proc_in_read_only_without_gpu() { - if has_gpu_devices() { - // On a GPU host we can't test the non-GPU path; skip silently. - return; - } - let (ro, _rw) = baseline_enrichment_paths(); - assert!( - ro.contains(&"/proc".to_string()), - "/proc should be in read_only when GPU is not present" - ); - } - - #[test] - fn baseline_read_write_does_not_hardcode_sandbox() { - let (_ro, rw) = baseline_enrichment_paths(); - assert!(rw.contains(&"/tmp".to_string())); - assert!(!rw.contains(&"/sandbox".to_string())); - } - - #[test] - fn enumerate_gpu_device_nodes_skips_bare_nvidia() { - // "nvidia" (without a trailing digit) is a valid /dev entry on some - // systems but is not a per-GPU device node. The enumerator must - // not match it. - let nodes = enumerate_gpu_device_nodes(); - assert!( - !nodes.contains(&"/dev/nvidia".to_string()), - "bare /dev/nvidia should not be enumerated: {nodes:?}" - ); - } - - #[test] - fn no_duplicate_paths_in_baseline() { - let (ro, rw) = baseline_enrichment_paths(); - // No path should appear in both lists. - for path in &ro { - assert!( - !rw.contains(path), - "path {path} appears in both read_only and read_write" - ); - } - } - - #[test] - fn proto_enrichment_preserves_explicit_read_only_for_baseline_read_write_paths() { - let mut policy = openshell_policy::restrictive_default_policy(); - policy.filesystem = Some(openshell_core::proto::FilesystemPolicy { - read_only: vec!["/tmp".to_string()], - read_write: vec![], - include_workdir: false, - }); - policy.network_policies.insert( - "test".into(), - openshell_core::proto::NetworkPolicyRule { - name: "test-rule".into(), - endpoints: vec![openshell_core::proto::NetworkEndpoint { - host: "example.com".into(), - port: 443, - ..Default::default() - }], - ..Default::default() - }, - ); - - enrich_proto_baseline_paths(&mut policy); - - let filesystem = policy.filesystem.expect("filesystem policy"); - assert!( - filesystem.read_only.contains(&"/tmp".to_string()), - "explicit read_only baseline path should be preserved" - ); - assert!( - !filesystem.read_write.contains(&"/tmp".to_string()), - "baseline enrichment must not promote explicit read_only /tmp to read_write" - ); - } - - #[test] - fn proto_strip_provider_policy_entries_removes_only_reserved_entries() { - let mut policy = openshell_policy::restrictive_default_policy(); - policy.network_policies.insert( - "_provider_work_github".to_string(), - openshell_core::proto::NetworkPolicyRule { - name: "_provider_work_github".to_string(), - ..Default::default() - }, - ); - policy.network_policies.insert( - "sandbox_only".to_string(), - openshell_core::proto::NetworkPolicyRule { - name: "sandbox_only".to_string(), - ..Default::default() - }, - ); - - assert!(strip_proto_provider_policy_entries(&mut policy)); - assert!( - !policy - .network_policies - .contains_key("_provider_work_github") - ); - assert!(policy.network_policies.contains_key("sandbox_only")); - assert!(!strip_proto_provider_policy_entries(&mut policy)); - } - - #[test] - fn proto_sync_payload_not_created_for_provider_entries_without_enrichment() { - let mut runtime_policy = openshell_policy::restrictive_default_policy(); - runtime_policy.network_policies.insert( - "_provider_work_github".to_string(), - openshell_core::proto::NetworkPolicyRule { - name: "_provider_work_github".to_string(), - ..Default::default() - }, - ); - - assert!(proto_sync_payload_for_enriched_policy(&runtime_policy, false).is_none()); - assert!( - runtime_policy - .network_policies - .contains_key("_provider_work_github"), - "provider-derived rules alone must not trigger sync or mutate runtime policy" - ); - } - - #[test] - fn proto_sync_payload_for_enrichment_strips_provider_entries_without_mutating_runtime_policy() { - let mut runtime_policy = openshell_policy::restrictive_default_policy(); - runtime_policy.network_policies.insert( - "_provider_work_github".to_string(), - openshell_core::proto::NetworkPolicyRule { - name: "_provider_work_github".to_string(), - ..Default::default() - }, - ); - runtime_policy.network_policies.insert( - "sandbox_only".to_string(), - openshell_core::proto::NetworkPolicyRule { - name: "sandbox_only".to_string(), - ..Default::default() - }, - ); - - let sync_policy = proto_sync_payload_for_enriched_policy(&runtime_policy, true) - .expect("enrichment should create a sync payload"); - - assert!( - runtime_policy - .network_policies - .contains_key("_provider_work_github"), - "runtime policy must retain provider-derived rules for OPA input" - ); - assert!( - !sync_policy - .network_policies - .contains_key("_provider_work_github") - ); - assert!(sync_policy.network_policies.contains_key("sandbox_only")); - } - - #[test] - fn proto_gpu_enrichment_promotes_proc_without_network_policy() { - let mut policy = openshell_policy::restrictive_default_policy(); - assert!( - policy.network_policies.is_empty(), - "regression setup must exercise the no-network default path" - ); - let (ro, rw) = - collect_baseline_enrichment_paths(false, true, vec!["/dev/nvidia0".to_string()]); - - let enriched = enrich_proto_baseline_paths_with(&mut policy, &ro, &rw, |path| { - matches!(path, "/proc" | "/dev/nvidia0") - }); - - let filesystem = policy.filesystem.expect("filesystem policy"); - assert!( - enriched, - "GPU enrichment should not require network policies" - ); - assert!( - filesystem.read_write.contains(&"/dev/nvidia0".to_string()), - "GPU enrichment should add enumerated device nodes without network policies" - ); - assert!( - !filesystem.read_only.contains(&"/proc".to_string()), - "GPU enrichment should remove /proc from read_only" - ); - assert!( - filesystem.read_write.contains(&"/proc".to_string()), - "GPU enrichment should promote /proc to read_write" - ); - } - - #[test] - fn gpu_baseline_read_write_contains_dxg() { - // /dev/dxg must be present so WSL2 sandboxes get the Landlock - // read-write rule for the CDI-injected DXG device. The existence - // check in enrich_proto_baseline_paths() skips it on native Linux. - assert!( - GPU_BASELINE_READ_WRITE.contains(&"/dev/dxg"), - "/dev/dxg must be in GPU_BASELINE_READ_WRITE for WSL2 support" - ); - } - - #[test] - fn local_enrichment_preserves_explicit_read_only_for_baseline_read_write_paths() { - let mut policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy { - read_only: vec![PathBuf::from("/tmp")], - read_write: vec![], - include_workdir: false, - }, - network: NetworkPolicy { - mode: NetworkMode::Proxy, - proxy: Some(ProxyPolicy { http_addr: None }), - }, - landlock: LandlockPolicy::default(), - process: ProcessPolicy::default(), - }; - - enrich_sandbox_baseline_paths(&mut policy); - - assert!( - policy.filesystem.read_only.contains(&PathBuf::from("/tmp")), - "explicit read_only baseline path should be preserved" - ); - assert!( - !policy - .filesystem - .read_write - .contains(&PathBuf::from("/tmp")), - "baseline enrichment must not promote explicit read_only /tmp to read_write" - ); - } - - #[test] - fn gpu_baseline_read_only_contains_usr_lib_wsl() { - // /usr/lib/wsl must be present so CDI-injected WSL2 GPU library - // bind-mounts are accessible under Landlock. Skipped on native Linux. - assert!( - GPU_BASELINE_READ_ONLY.contains(&"/usr/lib/wsl"), - "/usr/lib/wsl must be in GPU_BASELINE_READ_ONLY for WSL2 CDI library paths" - ); - } - - #[test] - fn has_gpu_devices_reflects_dxg_or_nvidiactl() { - // Verify the OR logic: result must match the manual disjunction of - // the two path checks. Passes in all environments. - let nvidiactl = std::path::Path::new("/dev/nvidiactl").exists(); - let dxg = std::path::Path::new("/dev/dxg").exists(); - assert_eq!( - has_gpu_devices(), - nvidiactl || dxg, - "has_gpu_devices() should be true iff /dev/nvidiactl or /dev/dxg exists" - ); - } -} - -/// Returns `true` if the error is transient and worth retrying. -/// -/// Walks the `miette::Report` error chain looking for a `tonic::Status`. If -/// found, only the gRPC codes that represent transient failures are retryable. -/// If no `tonic::Status` is present (e.g. a raw connection error), assume the -/// failure is transient. -fn is_retryable_error(err: &miette::Report) -> bool { - let mut source: Option<&dyn std::error::Error> = Some(err.as_ref()); - while let Some(e) = source { - if let Some(status) = e.downcast_ref::() { - return matches!( - status.code(), - tonic::Code::Unavailable - | tonic::Code::DeadlineExceeded - | tonic::Code::ResourceExhausted - | tonic::Code::Aborted - | tonic::Code::Internal - | tonic::Code::Unknown - ); - } - source = e.source(); - } - true -} - -/// Retry a gRPC operation with exponential backoff (capped at 4 s). -/// -/// Non-transient gRPC errors (e.g. `NOT_FOUND`, `INVALID_ARGUMENT`, -/// `PERMISSION_DENIED`) are returned immediately without retrying. -async fn grpc_retry(op_name: &str, f: F) -> Result -where - F: Fn() -> Fut, - Fut: Future>, -{ - let mut last_err = None; - for attempt in 1..=5u32 { - match f().await { - Ok(val) => return Ok(val), - Err(e) => { - if !is_retryable_error(&e) { - return Err(e); - } - if attempt < 5 { - warn!( - attempt, - max_attempts = 5, - error = %e, - "{op_name} failed, retrying" - ); - let backoff = Duration::from_secs((1u64 << (attempt - 1)).min(4)); - tokio::time::sleep(backoff).await; - } - last_err = Some(e); - } - } - } - Err(miette::miette!( - "{op_name} failed after 5 attempts: {}", - last_err.expect("loop executed at least once") - )) -} - -/// Load sandbox policy from local files or gRPC. -/// -/// Priority: -/// 1. If `policy_rules` and `policy_data` are provided, load OPA engine from local files -/// 2. If `sandbox_id` and `openshell_endpoint` are provided, fetch via gRPC -/// 3. If the server returns no policy, discover from disk or use restrictive default -/// 4. Otherwise, return an error -/// -/// Returns the policy, the OPA engine, and (for gRPC mode) the original proto -/// policy. The proto is retained so the OPA engine can be rebuilt with symlink -/// resolution after the container entrypoint starts. -async fn load_policy( - sandbox_id: Option, - sandbox: Option, - openshell_endpoint: Option, - policy_rules: Option, - policy_data: Option, - extension_credentials: &openshell_extension_core::ExtensionCredentialStore, -) -> Result<( - SandboxPolicy, - Option>, - Option, - MiddlewareRegistryStatus, - LoadedPolicyOrigin, - bool, - bool, -)> { - // File mode: load OPA engine from rego rules + YAML data (dev override) - if let (Some(policy_file), Some(data_file)) = (&policy_rules, &policy_data) { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "loading") - .unmapped("policy_rules", serde_json::json!(policy_file)) - .unmapped("policy_data", serde_json::json!(data_file)) - .message(format!( - "Loading OPA policy engine from local files [rules:{policy_file} data:{data_file}]" - )) - .build()); - let validate_middleware_config = |implementation: &str, config: &prost_types::Struct| { - openshell_supervisor_middleware_builtins::validate_config(implementation, config) - .map_err(|error| error.to_string()) - }; - let engine = OpaEngine::from_files_with_middleware_config( - std::path::Path::new(policy_file), - std::path::Path::new(data_file), - Some(&validate_middleware_config), - )?; - let middleware_registry = - openshell_supervisor_middleware::MiddlewareRegistry::connect_services( - openshell_supervisor_middleware_builtins::services(), - Vec::new(), - ) - .await?; - engine.replace_middleware_registry(middleware_registry)?; - let config = engine.query_sandbox_config()?; - let mut policy = SandboxPolicy { - version: 1, - filesystem: config.filesystem, - network: NetworkPolicy { - mode: NetworkMode::Proxy, - proxy: Some(ProxyPolicy { http_addr: None }), - }, - landlock: config.landlock, - process: config.process, - }; - enrich_sandbox_baseline_paths(&mut policy); - // File mode has no operator-registered middleware to connect. - return Ok(( - policy, - Some(Arc::new(engine)), - None, - MiddlewareRegistryStatus::Synchronized, - LoadedPolicyOrigin::LocalOverride, - false, - false, - )); - } - - // gRPC mode: fetch typed proto policy, construct OPA engine from baked rules + proto data - if let (Some(id), Some(endpoint)) = (&sandbox_id, &openshell_endpoint) { - info!( - sandbox_id = %id, - endpoint = %endpoint, - "Fetching sandbox policy via gRPC" - ); - let mut snapshot = grpc_retry("Policy fetch", || { - openshell_core::grpc_client::fetch_settings_snapshot(endpoint, id) - }) - .await?; - - let mut proto_policy = if let Some(p) = snapshot.policy.clone() { - p - } else { - // No policy configured on the server. Discover from disk or - // fall back to the restrictive default, then sync to the - // gateway so it becomes the authoritative baseline. - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "discovery") - .message("Server returned no policy; attempting local discovery") - .build() - ); - let mut discovered = discover_policy_from_disk_or_default(); - // Enrich before syncing so the gateway baseline includes - // baseline paths from the start. - enrich_proto_baseline_paths(&mut discovered); - strip_proto_provider_policy_entries(&mut discovered); - let sandbox = sandbox.as_deref().ok_or_else(|| { - miette::miette!( - "Cannot sync discovered policy: sandbox not available.\n\ - Set OPENSHELL_SANDBOX or --sandbox to enable policy sync." - ) - })?; - - // Sync and re-fetch over a single connection to avoid extra - // TLS handshakes. - let ws = snapshot.workspace.clone(); - snapshot = grpc_retry("Policy discovery sync", || { - openshell_core::grpc_client::sync_policy_and_fetch_snapshot( - endpoint, - id, - sandbox, - &discovered, - &ws, - ) - }) - .await?; - snapshot.policy.clone().ok_or_else(|| { - miette::miette!("Server still returned no policy after sync — this is a bug") - })? - }; - - // True only while `snapshot` describes the exact policy that will be - // constructed below. If enrichment cannot be synced and re-fetched, - // the policy remains enforceable but cannot be acknowledged by - // inferred structural equality. - let mut policy_bound_to_snapshot = true; - - // Ensure baseline filesystem paths are present for proxy-mode - // sandboxes. If the policy was enriched, sync the updated version - // back to the gateway so users can see the effective policy. - let enriched = enrich_proto_baseline_paths(&mut proto_policy); - let sync_policy = proto_sync_payload_for_enriched_policy(&proto_policy, enriched); - if let Some(sync_policy) = sync_policy { - if let Some(sandbox_name) = sandbox.as_deref() { - match openshell_core::grpc_client::sync_policy_and_fetch_snapshot( - endpoint, - id, - sandbox_name, - &sync_policy, - &snapshot.workspace, - ) - .await - { - Ok(canonical) => { - if let Some(policy) = canonical.policy.clone() { - proto_policy = policy; - snapshot = canonical; - } else { - policy_bound_to_snapshot = false; - warn!( - "Gateway returned no policy after enrichment sync; initial revision will be reconciled" - ); - } - } - Err(e) => { - policy_bound_to_snapshot = false; - warn!( - error = %e, - "Failed to sync enriched policy back to gateway; initial revision will be reconciled" - ); - } - } - } else { - policy_bound_to_snapshot = false; - } - } - - let mut loaded_policy_revision = - policy_bound_to_snapshot.then(|| LoadedPolicyRevision::from_snapshot(&snapshot)); - - // Build OPA engine from baked-in rules + typed proto data. - // In cluster mode, proxy networking is always enabled so OPA is - // always required for allow/deny decisions. - // The initial load uses pid=0 (no symlink resolution) because the - // container hasn't started yet. After the entrypoint spawns, the - // engine is rebuilt with the real PID for symlink resolution. - info!("Creating OPA engine from proto policy data"); - let mut has_last_valid_policy = true; - let engine = match OpaEngine::from_proto(&proto_policy) { - Ok(engine) => Arc::new(engine), - Err(e) => { - report_initial_policy_failure(endpoint, id, loaded_policy_revision.as_ref(), &e) - .await; - let validation_error = e.to_string(); - let candidate_version = snapshot.version; - let candidate_hash = snapshot.policy_hash.clone(); - // There is no in-memory last-known-good generation during - // startup, so both configured modes necessarily fail closed. - // Load the restrictive default atomically and keep the - // rejected revision unacknowledged for poll reconciliation. - has_last_valid_policy = false; - proto_policy = openshell_policy::restrictive_default_policy(); - let engine = Arc::new(OpaEngine::from_proto(&proto_policy)?); - let disposition = apply_policy_validation_failure( - &engine, - snapshot.policy_validation_failure_mode, - has_last_valid_policy, - candidate_version, - &validation_error, - )?; - emit_policy_validation_failure( - &disposition, - candidate_version, - &candidate_hash, - &validation_error, - ); - loaded_policy_revision = None; - engine - } - }; - - // Install the in-process catalog before any external connection can - // fail. A newly started sandbox must always be able to resolve built-in - // bindings, even while operator-run services are unavailable. - install_builtin_middleware_registry(&engine).await?; - - // Connect operator-registered middleware services. A connect/describe - // failure keeps the built-in registry active so each request's - // `on_error` policy governs matched traffic. The policy poll loop - // retries the install without waiting for a config change. - let middleware_services = snapshot.supervisor_middleware_services.clone(); - let middleware_registry_status = if middleware_services.is_empty() { - MiddlewareRegistryStatus::Synchronized - } else if let Err(error) = grpc_retry("Middleware connect", || { - let middleware_services = middleware_services.clone(); - let extension_credentials = extension_credentials.clone(); - let extension_authentication_enabled = snapshot.extension_authentication_enabled; - async move { - let credentials = if extension_authentication_enabled { - // Share the supervisor's store so the slots installed here - // are the ones the policy poll loop later rotates in place. - openshell_core::grpc_client::CachedOpenShellClient::connect_with_credentials( - endpoint, - extension_credentials, - ) - .await? - .refresh_extension_credentials(&middleware_services) - .await? - } else { - std::collections::HashMap::new() - }; - connect_middleware_registry( - &middleware_services, - &MiddlewareAuthentication { - credentials, - enabled: extension_authentication_enabled, - }, - ) - .await - } - }) - .await - .and_then(|registry| engine.replace_middleware_registry(registry)) - { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Other, "degraded") - .unmapped( - "supervisor_middleware_service_count", - serde_json::json!(middleware_services.len()) - ) - .message(format!( - "Supervisor middleware connect failed at startup; continuing with built-in middleware only, per-request on_error governs matched requests [error:{error}]" - )) - .build() - ); - MiddlewareRegistryStatus::NeedsReconciliation - } else { - MiddlewareRegistryStatus::Synchronized - }; - let opa_engine = Some(engine); - - let policy = match SandboxPolicy::try_from(proto_policy.clone()) { - Ok(policy) => policy, - Err(e) => { - report_initial_policy_failure(endpoint, id, loaded_policy_revision.as_ref(), &e) - .await; - return Err(e); - } - }; - return Ok(( - policy, - opa_engine, - Some(proto_policy), - middleware_registry_status, - LoadedPolicyOrigin::Gateway { - revision: loaded_policy_revision, - has_last_valid_policy, - }, - agent_proposals_enabled_from_settings(&snapshot.settings), - snapshot.extension_authentication_enabled, - )); - } - - // No policy source available - Err(miette::miette!( - "Sandbox policy required. Provide one of:\n\ - - --policy-rules and --policy-data (or OPENSHELL_POLICY_RULES and OPENSHELL_POLICY_DATA env vars)\n\ - - --sandbox-id and --openshell-endpoint (or OPENSHELL_SANDBOX_ID and OPENSHELL_ENDPOINT env vars)" - )) -} - -/// Try to discover a sandbox policy from the well-known disk path, falling -/// back to the legacy path, then to the hardcoded restrictive default. -fn discover_policy_from_disk_or_default() -> openshell_core::proto::SandboxPolicy { - let primary = std::path::Path::new(openshell_policy::CONTAINER_POLICY_PATH); - if primary.exists() { - return discover_policy_from_path(primary); - } - let legacy = std::path::Path::new(openshell_policy::LEGACY_CONTAINER_POLICY_PATH); - if legacy.exists() { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped( - "legacy_path", - serde_json::json!(legacy.display().to_string()) - ) - .unmapped("new_path", serde_json::json!(primary.display().to_string())) - .message(format!( - "Policy found at legacy path; consider moving [legacy_path:{} new_path:{}]", - legacy.display(), - primary.display() - )) - .build() - ); - return discover_policy_from_path(legacy); - } - discover_policy_from_path(primary) -} - -/// Try to read a sandbox policy YAML from `path`, falling back to the -/// hardcoded restrictive default if the file is missing or invalid. -fn discover_policy_from_path(path: &std::path::Path) -> openshell_core::proto::SandboxPolicy { - use openshell_policy::{ - parse_sandbox_policy, restrictive_default_policy, validate_sandbox_policy, - }; - - let Ok(yaml) = std::fs::read_to_string(path) else { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "default") - .message(format!( - "No policy file on disk, using restrictive default [path:{}]", - path.display() - )) - .build() - ); - return restrictive_default_policy(); - }; - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .message(format!( - "Loaded sandbox policy from container disk [path:{}]", - path.display() - )) - .build() - ); - match parse_sandbox_policy(&yaml) { - Ok(policy) => { - // Validate the disk-loaded policy for safety. - if let Err(violations) = validate_sandbox_policy(&policy) { - let messages: Vec = violations.iter().map(ToString::to_string).collect(); - ocsf_emit!(DetectionFindingBuilder::new(ocsf_ctx()) - .activity(ActivityId::Open) - .severity(SeverityId::Medium) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .finding_info( - FindingInfo::new( - "unsafe-disk-policy", - "Unsafe Disk Policy Content", - ) - .with_desc(&format!( - "Disk policy at {} contains unsafe content: {}", - path.display(), - messages.join("; "), - )), - ) - .message(format!( - "Disk policy contains unsafe content, using restrictive default [path:{}]", - path.display() - )) - .build()); - return restrictive_default_policy(); - } - policy - } - Err(e) => { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Other, "fallback") - .message(format!( - "Failed to parse disk policy, using restrictive default [path:{} error:{e}]", - path.display() - )) - .build()); - restrictive_default_policy() - } - } -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum MiddlewareRegistryStatus { - Synchronized, - NeedsReconciliation, -} - -#[derive(Debug)] -enum GatewayRuntimeReloadError { - PolicyValidation(miette::Report), - TransparentTcpPrerequisite(miette::Report), - MiddlewareRegistry(miette::Report), -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum GatewayRuntimeFailureClass { - PolicyValidation, - TransparentTcpPrerequisite, - MiddlewareRegistry, -} - -impl GatewayRuntimeReloadError { - fn class(&self) -> GatewayRuntimeFailureClass { - match self { - Self::PolicyValidation(_) => GatewayRuntimeFailureClass::PolicyValidation, - Self::TransparentTcpPrerequisite(_) => { - GatewayRuntimeFailureClass::TransparentTcpPrerequisite - } - Self::MiddlewareRegistry(_) => GatewayRuntimeFailureClass::MiddlewareRegistry, - } - } -} - -#[derive(Debug, PartialEq, Eq)] -struct FailedRuntimeRevision { - config_revision: u64, - policy_hash: String, - failure_class: GatewayRuntimeFailureClass, -} - -impl FailedRuntimeRevision { - fn new(config_revision: u64, policy_hash: &str, failure: &GatewayRuntimeReloadError) -> Self { - Self { - config_revision, - policy_hash: policy_hash.to_string(), - failure_class: failure.class(), - } - } -} - -struct MiddlewareReloadContext<'a> { - desired_services: &'a [openshell_core::proto::SupervisorMiddlewareService], - authentication: &'a MiddlewareAuthentication, - registry_changed: bool, - connector: &'a MiddlewareConnector, -} - -async fn reload_gateway_policy_runtime( - engine: &OpaEngine, - policy: Option<&openshell_core::proto::SandboxPolicy>, - entrypoint_pid: u32, - middleware: MiddlewareReloadContext<'_>, - transparent_tcp: TransparentTcpReloadState, -) -> std::result::Result<(), GatewayRuntimeReloadError> { - if let Some(policy) = policy - && policy_contains_explicit_tcp(policy) - { - if !transparent_tcp.capable { - return Err(GatewayRuntimeReloadError::TransparentTcpPrerequisite( - miette::miette!( - "candidate policy introduces protocol: tcp, but the runtime does not advertise transparent TCP support; previous policy remains active" - ), - )); - } - if !transparent_tcp.substrate_ready { - return Err(GatewayRuntimeReloadError::TransparentTcpPrerequisite( - miette::miette!( - "candidate policy introduces protocol: tcp, but this sandbox started without the transparent TCP substrate; recreate the sandbox to enable TCP; previous policy remains active" - ), - )); - } - } - match policy { - Some(policy) if middleware.registry_changed => { - let registry = (middleware.connector)( - middleware.desired_services.to_vec(), - middleware.authentication.clone(), - ) - .await - .map_err(GatewayRuntimeReloadError::MiddlewareRegistry)?; - engine - .reload_policy_and_middleware_from_proto_with_pid(policy, entrypoint_pid, registry) - .map_err(GatewayRuntimeReloadError::PolicyValidation) - } - // Policy-only change: the installed registry already matches the - // delivered service set, so swap the engine alone. This must not - // require middleware reachability. - Some(policy) => engine - .reload_from_proto_with_pid(policy, entrypoint_pid) - .map_err(GatewayRuntimeReloadError::PolicyValidation), - None => Err(GatewayRuntimeReloadError::PolicyValidation( - miette::miette!("runtime reload requires a policy payload but none was returned"), - )), - } -} - -fn policy_contains_explicit_tcp(policy: &openshell_core::proto::SandboxPolicy) -> bool { - policy.network_policies.values().any(|rule| { - rule.endpoints - .iter() - .any(|endpoint| endpoint.protocol.eq_ignore_ascii_case("tcp")) - }) -} - -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -struct TransparentTcpReloadState { - capable: bool, - substrate_ready: bool, +pub struct RuntimeQualification { + pub seccomp: openshell_isolation_interface::contract::SeccompEvidence, + pub landlock_abi: u32, + pub landlock_allow_deny: bool, + pub udp_dns_round_trip: bool, + pub tcp_dns_round_trip: bool, + pub tcp_allow_round_trip: bool, + pub tcp_deny_round_trip: bool, } -/// True when the installed middleware registry no longer matches the desired -/// service set and must be rebuilt (reconnecting every delivered service). +/// Placeholder used when compiling the package on a non-Linux host. /// -/// A policy-only change never requires a rebuild: middleware configs were -/// validated at gateway admission and the installed registry's manifests -/// already cover the unchanged service set, so requiring the services to be -/// reachable would only let a middleware outage block the policy update. -fn middleware_registry_needs_rebuild( - registry_status: MiddlewareRegistryStatus, - current_services: &[openshell_core::proto::SupervisorMiddlewareService], - desired_services: &[openshell_core::proto::SupervisorMiddlewareService], -) -> bool { - registry_status == MiddlewareRegistryStatus::NeedsReconciliation - || current_services != desired_services -} - -fn gateway_policy_runtime_needs_reconciliation( - reloads_gateway_policy: bool, - current_policy_hash: &str, - desired_policy_hash: &str, - current_services: &[openshell_core::proto::SupervisorMiddlewareService], - desired_services: &[openshell_core::proto::SupervisorMiddlewareService], - registry_status: MiddlewareRegistryStatus, -) -> bool { - reloads_gateway_policy - && (current_policy_hash != desired_policy_hash - || middleware_registry_needs_rebuild( - registry_status, - current_services, - desired_services, - )) -} - -/// Identity returned with the exact policy snapshot used to construct OPA. -#[derive(Clone, Debug, PartialEq, Eq)] -struct LoadedPolicyRevision { - version: u32, - policy_hash: String, - config_revision: u64, - policy_source: openshell_core::proto::PolicySource, -} +/// The sandbox binary rejects execution on those hosts before constructing a +/// qualification, but retaining the type keeps the library API portable for +/// workspace-wide checks. +#[cfg(not(target_os = "linux"))] +#[derive(Clone, Copy, Debug)] +pub struct RuntimeQualification; -/// Identifies where the policy currently loaded into OPA came from. +/// Run the authenticated boundary-local sandbox. /// -/// A missing gateway revision means the policy was loaded from the gateway but -/// could not be bound to an authoritative snapshot (for example, enrichment -/// sync failed). That state must reconcile on the first successful poll. A -/// local-file override is different: gateway policy revisions are observed for -/// settings/provider refreshes but must never replace the explicit local OPA -/// policy. -#[derive(Clone, Debug, PartialEq, Eq)] -enum LoadedPolicyOrigin { - LocalOverride, - Gateway { - revision: Option, - has_last_valid_policy: bool, - }, -} - -impl LoadedPolicyOrigin { - fn allows_gateway_policy_reload(&self) -> bool { - matches!(self, Self::Gateway { .. }) - } - - fn has_last_valid_policy(&self) -> bool { - match self { - Self::LocalOverride => true, - Self::Gateway { - has_last_valid_policy, - .. - } => *has_last_valid_policy, - } - } -} - -impl LoadedPolicyRevision { - fn from_snapshot(snapshot: &openshell_core::grpc_client::SettingsPollResult) -> Self { - Self { - version: snapshot.version, - policy_hash: snapshot.policy_hash.clone(), - config_revision: snapshot.config_revision, - policy_source: snapshot.policy_source, - } - } -} - -/// A sandbox-scoped policy revision that was constructed successfully at -/// startup and must be acknowledged to the gateway exactly once. -#[derive(Clone, Debug, PartialEq, Eq)] -struct InitialPolicyAck { - version: u32, - policy_hash: String, - config_revision: u64, -} - -#[derive(Clone, Debug, PartialEq, Eq)] -struct PolicyStatusUpdate { - version: u32, - loaded: bool, - error: String, - success_event: Option, -} - -#[derive(Clone, Debug, PartialEq, Eq)] -enum PolicyStatusSuccessEvent { - InitialAcknowledgement { policy_hash: String }, - UnchangedAcknowledgement { policy_hash: String }, -} - -impl PolicyStatusUpdate { - fn initial_loaded(ack: &InitialPolicyAck) -> Self { - Self { - version: ack.version, - loaded: true, - error: String::new(), - success_event: Some(PolicyStatusSuccessEvent::InitialAcknowledgement { - policy_hash: ack.policy_hash.clone(), - }), - } - } - - fn loaded(version: u32) -> Self { - Self { - version, - loaded: true, - error: String::new(), - success_event: None, - } - } - - fn unchanged_loaded(version: u32, policy_hash: String) -> Self { - Self { - version, - loaded: true, - error: String::new(), - success_event: Some(PolicyStatusSuccessEvent::UnchangedAcknowledgement { policy_hash }), - } - } - - fn failed(version: u32, error: String) -> Self { - Self { - version, - loaded: false, - error, - success_event: None, - } - } -} - -#[derive(Clone, Debug, PartialEq, Eq)] -enum InitialPollDisposition { - Acknowledge(InitialPolicyAck), - Reconcile, - TrackOnly, -} - -/// Determine whether the initially loaded policy corresponds to an -/// authoritative sandbox-scoped revision that must be acknowledged. +/// # Errors /// -/// Returns `Some` only for sandbox-sourced revisions (version > 0) whose -/// captured gateway identity matches the current version and hash. Global -/// policies, local-file development policies, version zero, and changed -/// identities yield `None`, so those paths never emit a sandbox-revision -/// acknowledgement. -fn initial_policy_ack_candidate( - loaded: Option<&LoadedPolicyRevision>, - canonical: &openshell_core::grpc_client::SettingsPollResult, -) -> Option { - let loaded = loaded?; - if loaded.policy_source != openshell_core::proto::PolicySource::Sandbox - || canonical.policy_source != openshell_core::proto::PolicySource::Sandbox - { - return None; - } - if loaded.version == 0 || canonical.version == 0 { - return None; - } - if loaded.version != canonical.version - || loaded.policy_hash != canonical.policy_hash - || canonical.config_revision < loaded.config_revision - { - return None; - } - Some(InitialPolicyAck { - version: loaded.version, - policy_hash: loaded.policy_hash.clone(), - config_revision: canonical.config_revision, - }) -} - -fn initial_poll_disposition( - origin: &LoadedPolicyOrigin, - canonical: &openshell_core::grpc_client::SettingsPollResult, -) -> InitialPollDisposition { - match origin { - LoadedPolicyOrigin::LocalOverride => InitialPollDisposition::TrackOnly, - LoadedPolicyOrigin::Gateway { revision, .. } => { - initial_policy_ack_candidate(revision.as_ref(), canonical).map_or( - InitialPollDisposition::Reconcile, - InitialPollDisposition::Acknowledge, - ) - } - } -} - -fn unchanged_policy_revision_candidate( - reloads_gateway_policy: bool, - recovering_rejected_policy: bool, - current_policy_version: u32, - current_policy_hash: &str, - result: &openshell_core::grpc_client::SettingsPollResult, -) -> Option { - (reloads_gateway_policy - && !recovering_rejected_policy - && !current_policy_hash.is_empty() - && result.policy_source == openshell_core::proto::PolicySource::Sandbox - && result.version > current_policy_version - && result.policy_hash == current_policy_hash) - .then_some(result.version) -} - -fn unchanged_policy_revision_ready_to_ack( - candidate: Option, - policy_runtime_changed: bool, - policy_runtime_reconciled: bool, -) -> Option { - candidate.filter(|_| !policy_runtime_changed || policy_runtime_reconciled) -} - -/// Whether the credential-provenance gates cannot apply to the loaded policy. -/// -/// The gateway derives `provider_credentialed` and deliberately keeps it out of -/// the policy YAML schema, so a local-file policy never carries it and never -/// will: gateway revisions are observed for settings and providers but must not -/// replace the local OPA policy. Provider credentials still arrive from the -/// gateway on that path, so the raw-tunnel and WebSocket binary-frame refusals -/// have nothing to match on. The request-body backstop is unaffected because it -/// keys off the secret resolver rather than endpoint provenance. -fn credential_gating_unavailable( - origin: &LoadedPolicyOrigin, - has_resolver: bool, - network_enabled: bool, -) -> bool { - network_enabled && has_resolver && matches!(origin, LoadedPolicyOrigin::LocalOverride) -} - -/// Report that credential provenance is unavailable for the loaded policy. -/// -/// Carries no credential name, host, or value: the finding states which -/// controls are inactive, nothing about what they would have protected. -fn report_credential_gating_unavailable() { - ocsf_emit!( - DetectionFindingBuilder::new(ocsf_ctx()) - .activity(ActivityId::Open) - .severity(SeverityId::High) - .confidence(ConfidenceId::High) - .is_alert(true) - .finding_info( - FindingInfo::new( - "credential-gating-unavailable", - "Credential Provenance Unavailable", - ) - .with_desc( - "Provider credentials are injected, but the loaded policy comes from local \ - files and carries no gateway-derived credential provenance. Uninspected \ - credentialed tunnels and WebSocket binary frames are not refused. Load \ - policy from the gateway to enable these controls." - ), - ) - .evidence_pairs(&[ - ("policy_source", "local-override"), - ("uninspected_connect_gate", "inactive"), - ("websocket_binary_gate", "inactive"), - ("request_body_backstop", "active"), - ]) - .remediation( - "Remove the local policy override so the gateway-delivered effective policy \ - applies, or detach provider credentials from this sandbox." - ) - .message( - "Credential provenance unavailable for local-file policy; uninspected credential gates inactive" - ) - .build() - ); -} - -/// Deliver policy status updates independently from policy reconciliation. -/// -/// The channel is FIFO, so a delayed older status can never arrive after a -/// newer status and move the gateway's active version backward. Delivery uses -/// the existing bounded retry, but failures never delay policy enforcement. -#[tonic::async_trait] -trait PolicyGatewayClient: Clone + Send + Sync + 'static { - async fn poll_settings( - &self, - sandbox_id: &str, - ) -> Result; - - async fn report_policy_status( - &self, - sandbox_id: &str, - version: u32, - loaded: bool, - error: &str, - ) -> Result<()>; - - async fn refresh_installed_extension_credentials(&self) -> Result<()> { - Ok(()) - } - - async fn extension_credentials_for( - &self, - _services: &[openshell_core::proto::SupervisorMiddlewareService], - ) -> Result> { - Ok(std::collections::HashMap::new()) - } - - fn workspace(&self) -> String; -} - -#[tonic::async_trait] -impl PolicyGatewayClient for openshell_core::grpc_client::CachedOpenShellClient { - async fn poll_settings( - &self, - sandbox_id: &str, - ) -> Result { - self.poll_settings(sandbox_id).await - } - - async fn report_policy_status( - &self, - sandbox_id: &str, - version: u32, - loaded: bool, - error: &str, - ) -> Result<()> { - self.report_policy_status(sandbox_id, version, loaded, error) - .await - } - - async fn refresh_installed_extension_credentials(&self) -> Result<()> { - self.refresh_installed_extension_credentials().await - } - - async fn extension_credentials_for( - &self, - services: &[openshell_core::proto::SupervisorMiddlewareService], - ) -> Result> { - self.extension_credentials_for(services).await - } - - fn workspace(&self) -> String { - self.workspace() - } -} - -async fn run_policy_status_reporter( - client: C, - sandbox_id: String, - mut updates: tokio::sync::mpsc::UnboundedReceiver, -) { - 'updates: while let Some(update) = updates.recv().await { - let operation = if matches!( - update.success_event, - Some(PolicyStatusSuccessEvent::InitialAcknowledgement { .. }) - ) { - "Initial policy acknowledgement" - } else { - "Policy status report" - }; - let mut attempt = 1_u32; - loop { - let sandbox_id = sandbox_id.clone(); - let error = update.error.clone(); - let client = client.clone(); - match client - .report_policy_status(&sandbox_id, update.version, update.loaded, &error) - .await - { - Ok(()) => break, - Err(error) if is_retryable_error(&error) => { - let backoff = Duration::from_secs(1_u64 << attempt.saturating_sub(1).min(5)); - warn!( - %error, - attempt, - version = update.version, - loaded = update.loaded, - retry_in_secs = backoff.as_secs(), - "{operation} failed transiently; retaining ordered update" - ); - tokio::time::sleep(backoff).await; - attempt = attempt.saturating_add(1); - } - Err(error) => { - warn!( - %error, - version = update.version, - loaded = update.loaded, - "Discarding terminal policy status update" - ); - continue 'updates; - } - } - } - - if let Some(event) = update.success_event { - let (policy_hash, message) = match event { - PolicyStatusSuccessEvent::InitialAcknowledgement { policy_hash } => ( - policy_hash, - format!( - "Acknowledged initial policy revision as loaded [version:{}]", - update.version - ), - ), - PolicyStatusSuccessEvent::UnchangedAcknowledgement { policy_hash } => ( - policy_hash, - format!( - "Acknowledged unchanged policy revision as loaded [version:{}]", - update.version - ), - ), - }; - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("version", serde_json::json!(update.version)) - .unmapped("policy_hash", serde_json::json!(policy_hash)) - .message(message) - .build() - ); - } - } -} - -fn enqueue_policy_status(sender: &UnboundedSender, update: PolicyStatusUpdate) { - let version = update.version; - if let Err(error) = sender.send(update) { - warn!( - %error, - version, - "Policy status reporter unavailable during shutdown" - ); - } -} - -/// Best-effort `FAILED` acknowledgement when initial policy construction or -/// conversion fails. -/// -/// Uses the revision identity captured with the policy that failed to build, -/// and preserves the original construction error as the reported message. A -/// delivery failure here is swallowed so it can never mask that error. -async fn report_initial_policy_failure( - endpoint: &str, - sandbox_id: &str, - revision: Option<&LoadedPolicyRevision>, - error: &miette::Report, -) { - let Some(revision) = revision.filter(|revision| { - revision.version > 0 - && revision.policy_source == openshell_core::proto::PolicySource::Sandbox - }) else { - return; - }; - let client = match openshell_core::grpc_client::CachedOpenShellClient::connect(endpoint).await { - Ok(client) => client, - Err(e) => { - warn!(error = %e, "Failed to connect to report initial policy failure"); - return; - } - }; - let message = error.to_string(); - if let Err(e) = grpc_retry("Initial policy failure report", || { - let client = client.clone(); - let message = message.clone(); - async move { - client - .report_policy_status(sandbox_id, revision.version, false, &message) - .await - } - }) - .await - { - warn!(error = %e, version = revision.version, "Failed to report initial policy failure"); - } -} - -/// Background loop that polls the server for policy updates. -/// -/// When a new version is detected, attempts to reload the OPA engine via -/// `reload_from_proto_with_pid()`. Reports load success/failure back to the -/// server. On failure, the previous engine is untouched (LKG behavior). -/// -/// When the entrypoint PID is available, policy reloads include symlink -/// resolution for binary paths via the container filesystem. -struct PolicyPollLoopContext { - endpoint: String, - sandbox_id: String, - opa_engine: Arc, - /// Source of the policy currently loaded into OPA. This distinguishes an - /// explicit local-file override from an unbound gateway revision so the - /// former is never replaced by policy polling. - loaded_policy_origin: LoadedPolicyOrigin, - entrypoint_pid: Arc, - interval_secs: u64, - ocsf_enabled: Arc, - ocsf_schema_version: Arc>, - provider_credentials: ProviderCredentialState, - policy_local_ctx: Option>, - agent_proposals: AgentProposals, - middleware_registry_status: MiddlewareRegistryStatus, - sidecar_control_publisher: Option, - workspace_tx: tokio::sync::watch::Sender, - extension_credentials: openshell_extension_core::ExtensionCredentialStore, - extension_authentication_enabled: bool, - middleware_connector: MiddlewareConnector, - /// Immutable driver capability and startup substrate state. - transparent_tcp: TransparentTcpReloadState, -} - -type MiddlewareConnector = Arc< - dyn Fn( - Vec, - MiddlewareAuthentication, - ) -> Pin< - Box< - dyn std::future::Future< - Output = Result, - > + Send, - >, - > + Send - + Sync, ->; - -#[derive(Clone, Default)] -struct MiddlewareAuthentication { - credentials: std::collections::HashMap, - enabled: bool, -} - -fn default_middleware_connector() -> MiddlewareConnector { - Arc::new(|services, authentication| { - Box::pin(async move { connect_middleware_registry(&services, &authentication).await }) - }) -} - -async fn connect_middleware_registry( - services: &[openshell_core::proto::SupervisorMiddlewareService], - authentication: &MiddlewareAuthentication, -) -> Result { - if authentication.enabled { - openshell_supervisor_middleware::MiddlewareRegistry::connect_services_authenticated( - openshell_supervisor_middleware_builtins::services(), - services.to_vec(), - &authentication.credentials, - ) - .await - } else { - openshell_supervisor_middleware::MiddlewareRegistry::connect_services( - openshell_supervisor_middleware_builtins::services(), - services.to_vec(), - ) - .await - } -} - -async fn install_builtin_middleware_registry(opa_engine: &OpaEngine) -> Result<()> { - let registry = openshell_supervisor_middleware::MiddlewareRegistry::connect_services( - openshell_supervisor_middleware_builtins::services(), - Vec::new(), - ) - .await?; - opa_engine.replace_middleware_registry(registry) -} - -/// Wait the configured poll interval, but never past the point at which an -/// installed extension credential must be rotated. -fn next_poll_delay( - store: &openshell_extension_core::ExtensionCredentialStore, - interval: Duration, -) -> Duration { - let now_ms = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_or(0, |elapsed| { - i64::try_from(elapsed.as_millis()).unwrap_or(i64::MAX) - }); - store.next_refresh_delay(interval, now_ms) -} - -/// Drop credentials for services no longer in the installed registry. -/// -/// Call only after a registry swap succeeds, so a failed candidate cannot -/// invalidate the last-known-good clients. -fn retain_extension_credentials( - store: &openshell_extension_core::ExtensionCredentialStore, - installed: &[openshell_core::proto::SupervisorMiddlewareService], - extension_authentication_enabled: bool, -) { - let retained = if extension_authentication_enabled { - installed - .iter() - .map(|service| service.name.as_str()) - .collect() - } else { - std::collections::HashSet::default() - }; - store.retain(&retained); -} - -struct MiddlewareRegistryReconciliation<'a> { - desired_services: &'a [openshell_core::proto::SupervisorMiddlewareService], - authentication: MiddlewareAuthentication, - registry_changed: bool, - extension_credentials: &'a openshell_extension_core::ExtensionCredentialStore, - current_services: &'a mut Vec, - status: &'a mut MiddlewareRegistryStatus, -} - -async fn reconcile_middleware_registry( - opa_engine: &OpaEngine, - middleware_connector: &MiddlewareConnector, - reconciliation: MiddlewareRegistryReconciliation<'_>, -) { - if !reconciliation.registry_changed { - return; - } - - match middleware_connector( - reconciliation.desired_services.to_vec(), - reconciliation.authentication.clone(), - ) - .await - .and_then(|registry| opa_engine.replace_middleware_registry(registry)) - { - Ok(()) => { - retain_extension_credentials( - reconciliation.extension_credentials, - reconciliation.desired_services, - reconciliation.authentication.enabled, - ); - reconciliation.current_services.clear(); - reconciliation - .current_services - .extend_from_slice(reconciliation.desired_services); - *reconciliation.status = MiddlewareRegistryStatus::Synchronized; - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped( - "supervisor_middleware_service_count", - serde_json::json!(reconciliation.current_services.len()) - ) - .message(format!( - "Supervisor middleware registry reloaded [service_count:{}]", - reconciliation.current_services.len() - )) - .build() - ); - } - Err(error) => { - // Emit only on the transition into the failed state to avoid - // repeating the same finding on every poll during an outage. - if *reconciliation.status == MiddlewareRegistryStatus::Synchronized { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Other, "failed") - .message(format!( - "Supervisor middleware registry reload failed, keeping last-known-good registry [error:{error}]" - )) - .build() - ); - } - *reconciliation.status = MiddlewareRegistryStatus::NeedsReconciliation; - } - } -} - -#[derive(Debug, PartialEq, Eq)] -struct PolicyValidationFailureDisposition { - configured_mode: PolicyValidationFailureMode, - mode: PolicyValidationFailureMode, - previous_policy_active: bool, - active_generation: u64, -} - -struct RejectedPolicyGeneration { - version: u32, - policy_hash: String, - validation_error: String, - configured_mode: PolicyValidationFailureMode, -} - -enum GatewayRuntimeFailureDisposition { - PolicyRejected { - error: String, - disposition: PolicyValidationFailureDisposition, - }, - MiddlewareUnavailable { - error: String, - }, - TransparentTcpExpansionRejected { - error: String, - active_generation: u64, - }, -} - -fn apply_gateway_runtime_reload_failure( - engine: &OpaEngine, - failure: GatewayRuntimeReloadError, - configured_mode: PolicyValidationFailureMode, - has_last_valid_policy: bool, - version: u32, -) -> Result { - match failure { - GatewayRuntimeReloadError::PolicyValidation(error) => { - let error = error.to_string(); - let disposition = apply_policy_validation_failure( - engine, - configured_mode, - has_last_valid_policy, - version, - &error, - )?; - Ok(GatewayRuntimeFailureDisposition::PolicyRejected { error, disposition }) - } - GatewayRuntimeReloadError::TransparentTcpPrerequisite(error) => Ok( - GatewayRuntimeFailureDisposition::TransparentTcpExpansionRejected { - error: error.to_string(), - active_generation: engine.current_generation(), - }, - ), - GatewayRuntimeReloadError::MiddlewareRegistry(error) => { - Ok(GatewayRuntimeFailureDisposition::MiddlewareUnavailable { - error: error.to_string(), - }) - } - } -} - -fn emit_transparent_tcp_expansion_rejection( - version: u32, - policy_hash: &str, - active_generation: u64, - error: &str, -) { - let message = format!( - "Transparent TCP policy expansion rejected; previous policy IS active [version:{version} active_generation:{active_generation} error:{error}]" - ); - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::High) - .status(StatusId::Failure) - .state(StateId::Enabled, "retained_previous_policy") - .unmapped("candidate_version", serde_json::json!(version)) - .unmapped("candidate_policy_hash", serde_json::json!(policy_hash)) - .unmapped("previous_policy_active", serde_json::json!(true)) - .unmapped("active_generation", serde_json::json!(active_generation)) - .unmapped("validation_error", serde_json::json!(error)) - .message(message) - .build() - ); -} - -fn apply_policy_validation_failure( - engine: &OpaEngine, - configured_mode: PolicyValidationFailureMode, - has_last_valid_policy: bool, - version: u32, - error: &str, -) -> Result { - let mode = if has_last_valid_policy { - configured_mode - } else { - PolicyValidationFailureMode::FailClosed - }; - match mode { - PolicyValidationFailureMode::FailClosed => { - let reason = format!( - "policy validation failed; fail-closed quarantine is active; candidate version {version} rejected: {error}" - ); - let active_generation = engine.enter_fail_closed(reason)?; - Ok(PolicyValidationFailureDisposition { - configured_mode, - mode, - previous_policy_active: false, - active_generation, - }) - } - PolicyValidationFailureMode::RetainLastValid => { - let active_generation = engine.exit_fail_closed()?; - Ok(PolicyValidationFailureDisposition { - configured_mode, - mode, - previous_policy_active: true, - active_generation, - }) - } - } -} - -fn policy_validation_failure_events( - disposition: &PolicyValidationFailureDisposition, - version: u32, - policy_hash: &str, - error: &str, -) -> [OcsfEvent; 2] { - let previous_policy_state = if disposition.previous_policy_active { - "IS active" - } else { - "IS NOT active" - }; - let state = if disposition.previous_policy_active { - (StateId::Enabled, "retained_last_valid") - } else { - (StateId::Disabled, "fail_closed") - }; - let message = format!( - "Policy validation failed; configured_mode={} effective_mode={}; previous policy {previous_policy_state} [version:{version} active_generation:{} error:{error}]", - disposition.configured_mode.as_str(), - disposition.mode.as_str(), - disposition.active_generation, - ); - let finding_uid = format!("policy-validation-failed-{version}"); - let version_string = version.to_string(); - let config = ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::High) - .status(StatusId::Failure) - .state(state.0, state.1) - .unmapped("candidate_version", serde_json::json!(version)) - .unmapped("candidate_policy_hash", serde_json::json!(policy_hash)) - .unmapped( - "validation_failure_mode", - serde_json::json!(disposition.mode.as_str()), - ) - .unmapped( - "configured_validation_failure_mode", - serde_json::json!(disposition.configured_mode.as_str()), - ) - .unmapped( - "previous_policy_active", - serde_json::json!(disposition.previous_policy_active), - ) - .unmapped( - "active_generation", - serde_json::json!(disposition.active_generation), - ) - .unmapped("validation_error", serde_json::json!(error)) - .message(message.clone()) - .build(); - let finding = DetectionFindingBuilder::new(ocsf_ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::High) - .is_alert(true) - .finding_info( - FindingInfo::new(&finding_uid, "Invalid policy generation rejected").with_desc(error), - ) - .evidence_pairs(&[ - ("candidate_version", &version_string), - ("candidate_policy_hash", policy_hash), - ("validation_failure_mode", disposition.mode.as_str()), - ( - "configured_validation_failure_mode", - disposition.configured_mode.as_str(), - ), - ( - "previous_policy_active", - if disposition.previous_policy_active { - "true" - } else { - "false" - }, - ), - ]) - .remediation("Submit a valid, unambiguous policy generation") - .message(message) - .build(); - [config, finding] -} - -fn emit_policy_validation_failure( - disposition: &PolicyValidationFailureDisposition, - version: u32, - policy_hash: &str, - error: &str, -) { - for event in policy_validation_failure_events(disposition, version, policy_hash, error) { - ocsf_emit!(event); - } -} - -async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> { - let client = openshell_core::grpc_client::CachedOpenShellClient::connect_with_credentials( - &ctx.endpoint, - ctx.extension_credentials.clone(), - ) - .await?; - run_policy_poll_loop_with_client(ctx, client).await -} - -async fn run_policy_poll_loop_with_client( - ctx: PolicyPollLoopContext, - client: C, -) -> Result<()> { - use openshell_core::proto::PolicySource; - use std::sync::atomic::Ordering; - - let (status_sender, status_receiver) = tokio::sync::mpsc::unbounded_channel(); - tokio::spawn(run_policy_status_reporter( - client.clone(), - ctx.sandbox_id.clone(), - status_receiver, - )); - - let mut current_config_revision: u64 = 0; - let mut current_provider_env_revision: u64 = ctx.provider_credentials.snapshot().revision; - let mut current_policy_version: u32 = 0; - let mut current_policy_hash = String::new(); - let mut current_middleware_services = Vec::new(); - let mut current_extension_authentication_enabled = ctx.extension_authentication_enabled; - let mut middleware_registry_status = ctx.middleware_registry_status; - let mut current_settings: std::collections::HashMap< - String, - openshell_core::proto::EffectiveSetting, - > = std::collections::HashMap::new(); - let reloads_gateway_policy = ctx.loaded_policy_origin.allows_gateway_policy_reload(); - let mut last_failed_runtime_revision: Option = None; - let mut rejected_policy_generation: Option = None; - let mut has_last_valid_policy = ctx.loaded_policy_origin.has_last_valid_policy(); - - // A first poll that does not match the policy already loaded into OPA must - // pass through the normal reconciliation path immediately. It must never - // seed the applied-state trackers before OPA actually loads it. - let mut pending_result = None; - - // Initialize revision from the first poll and acknowledge the initial - // policy revision the supervisor actually loaded. A mismatched result is - // reconciled below instead of being recorded as already applied. - match client.poll_settings(&ctx.sandbox_id).await { - Ok(result) => { - let _ = ctx.workspace_tx.send(client.workspace()); - match initial_poll_disposition(&ctx.loaded_policy_origin, &result) { - InitialPollDisposition::Acknowledge(candidate) => { - apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); - apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &result.settings); - apply_agent_proposals_enabled( - &ctx.agent_proposals, - agent_proposals_enabled_from_settings(&result.settings), - "initial settings poll", - Some(candidate.config_revision), - ctx.sidecar_control_publisher.as_ref(), - skills::install_static_skills, - ); - current_config_revision = candidate.config_revision; - current_policy_version = candidate.version; - current_policy_hash.clone_from(&candidate.policy_hash); - current_middleware_services = result.supervisor_middleware_services; - current_extension_authentication_enabled = - result.extension_authentication_enabled; - current_settings = result.settings; - enqueue_policy_status( - &status_sender, - PolicyStatusUpdate::initial_loaded(&candidate), - ); - debug!( - config_revision = current_config_revision, - "Settings poll: initial policy matches loaded revision" - ); - } - InitialPollDisposition::Reconcile => pending_result = Some(result), - InitialPollDisposition::TrackOnly => { - apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); - apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &result.settings); - apply_agent_proposals_enabled( - &ctx.agent_proposals, - agent_proposals_enabled_from_settings(&result.settings), - "initial settings poll", - Some(result.config_revision), - ctx.sidecar_control_publisher.as_ref(), - skills::install_static_skills, - ); - current_config_revision = result.config_revision; - current_policy_hash = result.policy_hash.clone(); - current_middleware_services = result.supervisor_middleware_services; - current_extension_authentication_enabled = - result.extension_authentication_enabled; - current_settings = result.settings; - debug!( - config_revision = current_config_revision, - "Settings poll: tracking gateway config while preserving local policy override" - ); - } - } - } - Err(e) => { - warn!(error = %e, "Settings poll: failed to fetch initial version, will retry"); - } - } - - let interval = Duration::from_secs(ctx.interval_secs); - loop { - let result = if let Some(result) = pending_result.take() { - result - } else { - tokio::time::sleep(next_poll_delay(&ctx.extension_credentials, interval)).await; - match client.poll_settings(&ctx.sandbox_id).await { - Ok(result) => { - let _ = ctx.workspace_tx.send(client.workspace()); - result - } - Err(e) => { - debug!(error = %e, "Settings poll: server unreachable, will retry"); - if current_extension_authentication_enabled - && let Err(refresh_error) = - client.refresh_installed_extension_credentials().await - { - warn!( - error = %refresh_error, - "Settings poll: extension credential refresh failed while configuration was unavailable" - ); - } - continue; - } - } - }; - - // Reuse installed per-service credentials, rotating only when one is - // missing or due. Rotation happens on the existing gateway channel and - // updates slots in place, so it is independent of config revision and - // registry equality. - let middleware_credentials = if result.extension_authentication_enabled { - match client - .extension_credentials_for(&result.supervisor_middleware_services) - .await - { - Ok(credentials) => credentials, - Err(error) => { - warn!(error = %error, "Settings poll: extension credential refresh failed"); - std::collections::HashMap::new() - } - } - } else { - std::collections::HashMap::new() - }; - - let config_changed = result.config_revision != current_config_revision; - let provider_env_changed = result.provider_env_revision != current_provider_env_revision; - let policy_changed = result.policy_hash != current_policy_hash; - let extension_authentication_changed = - current_extension_authentication_enabled != result.extension_authentication_enabled; - let middleware_registry_changed = extension_authentication_changed - || middleware_registry_needs_rebuild( - middleware_registry_status, - ¤t_middleware_services, - &result.supervisor_middleware_services, - ); - // A valid candidate may intentionally restore byte-for-byte policy - // content that was active before a rejected update. Its hash then - // equals `current_policy_hash`, but the runtime is still quarantined - // and must reload (or it would remain deny-all indefinitely). - let recovering_rejected_policy = reloads_gateway_policy - && rejected_policy_generation - .as_ref() - .is_some_and(|rejected| rejected.policy_hash != result.policy_hash); - let policy_runtime_changed = recovering_rejected_policy - || extension_authentication_changed - || gateway_policy_runtime_needs_reconciliation( - reloads_gateway_policy, - ¤t_policy_hash, - &result.policy_hash, - ¤t_middleware_services, - &result.supervisor_middleware_services, - middleware_registry_status, - ); - // Recovery already has its own acknowledgement path below. Giving it - // precedence here prevents a restored last-known-good policy from - // also being acknowledged as an ordinary same-hash revision. - let unchanged_policy_revision = unchanged_policy_revision_candidate( - reloads_gateway_policy, - recovering_rejected_policy, - current_policy_version, - ¤t_policy_hash, - &result, - ); - let mut policy_runtime_reconciled = false; - - // A local policy override is not coupled to the gateway policy - // snapshot, so its service registry can still be reconciled alone. - // Gateway policy snapshots, however, must install policy and registry - // as one generation below. - if !reloads_gateway_policy { - reconcile_middleware_registry( - &ctx.opa_engine, - &ctx.middleware_connector, - MiddlewareRegistryReconciliation { - desired_services: &result.supervisor_middleware_services, - authentication: MiddlewareAuthentication { - credentials: middleware_credentials.clone(), - enabled: result.extension_authentication_enabled, - }, - registry_changed: middleware_registry_changed, - extension_credentials: &ctx.extension_credentials, - current_services: &mut current_middleware_services, - status: &mut middleware_registry_status, - }, - ) - .await; - if middleware_registry_status == MiddlewareRegistryStatus::Synchronized { - current_extension_authentication_enabled = result.extension_authentication_enabled; - } - } - - if !config_changed - && !provider_env_changed - && !policy_runtime_changed - && unchanged_policy_revision.is_none() - { - continue; - } - - if config_changed || provider_env_changed { - // Log which settings changed. - log_setting_changes(¤t_settings, &result.settings); - - // A posture change after a rejected update takes effect immediately. - // The compiled last-known-good engine remains available beneath a - // fail-closed quarantine, so an explicit retain_last_valid selection - // can reactivate it without accepting any part of the invalid policy. - if !policy_changed && let Some(rejected) = rejected_policy_generation.as_mut() { - let mode = result.policy_validation_failure_mode; - if mode != rejected.configured_mode { - let disposition = apply_policy_validation_failure( - &ctx.opa_engine, - mode, - has_last_valid_policy, - rejected.version, - &rejected.validation_error, - )?; - emit_policy_validation_failure( - &disposition, - rejected.version, - &rejected.policy_hash, - &rejected.validation_error, - ); - rejected.configured_mode = mode; - } - } - - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "detected") - .unmapped("old_config_revision", serde_json::json!(current_config_revision)) - .unmapped("new_config_revision", serde_json::json!(result.config_revision)) - .unmapped("policy_changed", serde_json::json!(policy_changed)) - .unmapped("provider_env_changed", serde_json::json!(provider_env_changed)) - .message(format!( - "Settings poll: config change detected [old_revision:{current_config_revision} new_revision:{} policy_changed:{policy_changed} provider_env_changed:{provider_env_changed}]", - result.config_revision - )) - .build()); - } - - if provider_env_changed { - match openshell_core::grpc_client::fetch_provider_environment( - &ctx.endpoint, - &ctx.sandbox_id, - ) - .await - { - Ok(env_result) => { - let provider_env_revision = env_result.provider_env_revision; - let install_result = ctx.provider_credentials.install_bound_environment( - provider_env_revision, - env_result.environment, - env_result.credential_expires_at_ms, - env_result.dynamic_credentials, - env_result.static_credential_bindings, - env_result.non_secret_environment_keys, - ); - if let Err(error) = install_result { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::High) - .status(StatusId::Failure) - .state(StateId::Disabled, "fail_closed") - .message(format!( - "Rejected provider environment refresh; static provider credentials were revoked; fetched dynamic token grants remain active: {error}" - )) - .build() - ); - } else { - let child_env = ctx.provider_credentials.child_env_with_gcp_resolved(); - let env_count = child_env.len(); - if let Some(publisher) = ctx.sidecar_control_publisher.as_ref() { - publisher - .publish_provider_env(provider_env_revision, child_env.clone()); - } - current_provider_env_revision = provider_env_revision; - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped( - "provider_env_revision", - serde_json::json!(provider_env_revision) - ) - .message(format!( - "Provider environment refreshed [revision:{provider_env_revision} env_count:{env_count}]" - )) - .build() - ); - } - } - Err(e) => { - ctx.provider_credentials - .revoke_static_provider_environment(result.provider_env_revision); - warn!( - error = %e, - provider_env_revision = result.provider_env_revision, - "Settings poll: failed to refresh provider environment; static provider credentials were revoked; previous dynamic token grants remain active" - ); - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::High) - .status(StatusId::Failure) - .state(StateId::Disabled, "fail_closed") - .message( - "Provider environment refresh failed; static provider credentials were revoked; previous dynamic token grants remain active" - ) - .build() - ); - } - } - } - - if policy_runtime_changed { - let pid = ctx.entrypoint_pid.load(Ordering::Acquire); - let runtime_result = reload_gateway_policy_runtime( - &ctx.opa_engine, - result.policy.as_ref(), - pid, - MiddlewareReloadContext { - desired_services: &result.supervisor_middleware_services, - authentication: &MiddlewareAuthentication { - credentials: middleware_credentials.clone(), - enabled: result.extension_authentication_enabled, - }, - registry_changed: middleware_registry_changed, - connector: &ctx.middleware_connector, - }, - ctx.transparent_tcp, - ) - .await; - - match runtime_result { - Ok(()) => { - policy_runtime_reconciled = true; - let policy = result - .policy - .as_ref() - .expect("successful runtime reload requires a policy payload"); - has_last_valid_policy = true; - rejected_policy_generation = None; - if policy_changed { - if let Some(policy_local_ctx) = ctx.policy_local_ctx.as_ref() { - policy_local_ctx.set_current_policy(policy.clone()).await; - } - if let Some(publisher) = ctx.sidecar_control_publisher.as_ref() { - publisher.publish_policy( - policy.clone(), - result.policy_hash.clone(), - result.config_revision, - ); - } - if result.global_policy_version > 0 { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("policy_hash", serde_json::json!(&result.policy_hash)) - .unmapped("global_version", serde_json::json!(result.global_policy_version)) - .message(format!( - "Policy reloaded successfully (global) [policy_hash:{} global_version:{}]", - result.policy_hash, - result.global_policy_version - )) - .build()); - } else { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("policy_hash", serde_json::json!(&result.policy_hash)) - .message(format!( - "Policy reloaded successfully [policy_hash:{}]", - result.policy_hash - )) - .build() - ); - } - if result.version > 0 && result.policy_source == PolicySource::Sandbox { - enqueue_policy_status( - &status_sender, - PolicyStatusUpdate::loaded(result.version), - ); - current_policy_version = result.version; - } - } else if recovering_rejected_policy - && result.version > 0 - && result.policy_source == PolicySource::Sandbox - { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("policy_hash", serde_json::json!(&result.policy_hash)) - .message(format!( - "Policy reloaded successfully and fail-closed quarantine cleared [policy_hash:{}]", - result.policy_hash - )) - .build() - ); - enqueue_policy_status( - &status_sender, - PolicyStatusUpdate::loaded(result.version), - ); - current_policy_version = result.version; - } - - if middleware_registry_changed { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped( - "supervisor_middleware_service_count", - serde_json::json!(result.supervisor_middleware_services.len()) - ) - .message(format!( - "Supervisor policy runtime reloaded atomically [service_count:{}]", - result.supervisor_middleware_services.len() - )) - .build()); - } - - current_policy_hash.clone_from(&result.policy_hash); - current_middleware_services.clone_from(&result.supervisor_middleware_services); - current_extension_authentication_enabled = - result.extension_authentication_enabled; - retain_extension_credentials( - &ctx.extension_credentials, - &result.supervisor_middleware_services, - result.extension_authentication_enabled, - ); - middleware_registry_status = MiddlewareRegistryStatus::Synchronized; - last_failed_runtime_revision = None; - } - Err(failure) => { - let failed_revision = FailedRuntimeRevision::new( - result.config_revision, - &result.policy_hash, - &failure, - ); - if last_failed_runtime_revision.as_ref() != Some(&failed_revision) { - let failure_mode = result.policy_validation_failure_mode; - match apply_gateway_runtime_reload_failure( - &ctx.opa_engine, - failure, - failure_mode, - has_last_valid_policy, - result.version, - )? { - GatewayRuntimeFailureDisposition::PolicyRejected { - error, - disposition, - } => { - emit_policy_validation_failure( - &disposition, - result.version, - &result.policy_hash, - &error, - ); - rejected_policy_generation = Some(RejectedPolicyGeneration { - version: result.version, - policy_hash: result.policy_hash.clone(), - validation_error: error.clone(), - configured_mode: failure_mode, - }); - if policy_changed - && result.version > 0 - && result.policy_source == PolicySource::Sandbox - { - enqueue_policy_status( - &status_sender, - PolicyStatusUpdate::failed(result.version, error), - ); - } - } - GatewayRuntimeFailureDisposition::MiddlewareUnavailable { error } => { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Other, "failed") - .unmapped("version", serde_json::json!(result.version)) - .unmapped("error", serde_json::json!(&error)) - .unmapped("previous_policy_active", serde_json::json!(true)) - .message(format!( - "Supervisor middleware registry unavailable, keeping last-known-good policy runtime active [version:{} error:{error}]", - result.version - )) - .build()); - } - GatewayRuntimeFailureDisposition::TransparentTcpExpansionRejected { - error, - active_generation, - } => { - emit_transparent_tcp_expansion_rejection( - result.version, - &result.policy_hash, - active_generation, - &error, - ); - if policy_changed - && result.version > 0 - && result.policy_source == PolicySource::Sandbox - { - enqueue_policy_status( - &status_sender, - PolicyStatusUpdate::failed(result.version, error), - ); - } - } - } - } - last_failed_runtime_revision = Some(failed_revision); - // Nothing was installed, so the registry status still - // describes the live registry. The retry is driven by the - // persisting hash/service-set mismatch (or an existing - // NeedsReconciliation), not by degrading the status here. - } - } - } - - if let Some(version) = unchanged_policy_revision_ready_to_ack( - unchanged_policy_revision, - policy_runtime_changed, - policy_runtime_reconciled, - ) { - enqueue_policy_status( - &status_sender, - PolicyStatusUpdate::unchanged_loaded(version, result.policy_hash.clone()), - ); - current_policy_version = version; - } - - // Apply OCSF JSON toggle from the `ocsf_json_enabled` setting. - apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); - apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &result.settings); - - // Apply the agent-proposals feature toggle. On a false→true transition - // we lazily install the skill so a sandbox that started with the flag - // off picks up the surface without a recreate. We never uninstall on - // a true→false transition: stale skill content on disk is harmless - // because route_request and agent_next_steps both gate on the live - // shared flag, so the agent that reads the skill will see 404s and an - // empty `next_steps` array regardless. - apply_agent_proposals_enabled( - &ctx.agent_proposals, - agent_proposals_enabled_from_settings(&result.settings), - "settings poll", - Some(result.config_revision), - ctx.sidecar_control_publisher.as_ref(), - skills::install_static_skills, - ); - - current_config_revision = result.config_revision; - if !reloads_gateway_policy { - current_policy_hash = result.policy_hash; - } - current_settings = result.settings; - } -} - -fn apply_ocsf_json_setting( - enabled: &AtomicBool, - settings: &std::collections::HashMap, -) { - use std::sync::atomic::Ordering; - - let new_ocsf = extract_bool_setting(settings, "ocsf_json_enabled").unwrap_or(false); - let prev_ocsf = enabled.swap(new_ocsf, Ordering::Relaxed); - if new_ocsf != prev_ocsf { - info!(ocsf_json_enabled = new_ocsf, "OCSF JSONL logging toggled"); - } -} - -/// Extract a bool value from an effective setting, if present. -fn extract_bool_setting( - settings: &std::collections::HashMap, - key: &str, -) -> Option { - use openshell_core::proto::setting_value; - settings - .get(key) - .and_then(|es| es.value.as_ref()) - .and_then(|sv| sv.value.as_ref()) - .and_then(|v| match v { - setting_value::Value::BoolValue(b) => Some(*b), - _ => None, - }) -} - -fn apply_ocsf_schema_version_setting( - version: &std::sync::Mutex, - settings: &std::collections::HashMap, -) { - let new_version = extract_string_setting(settings, "ocsf_schema_version").unwrap_or_default(); - if let Ok(mut current) = version.lock() - && *current != new_version - { - info!( - ocsf_schema_version = %new_version, - "OCSF schema version target changed" - ); - *current = new_version; - } -} - -fn extract_string_setting( - settings: &std::collections::HashMap, - key: &str, -) -> Option { - use openshell_core::proto::setting_value; - settings - .get(key) - .and_then(|es| es.value.as_ref()) - .and_then(|sv| sv.value.as_ref()) - .and_then(|v| match v { - setting_value::Value::StringValue(s) => Some(s.clone()), - _ => None, - }) -} - -fn agent_proposals_enabled_from_settings( - settings: &std::collections::HashMap, -) -> bool { - extract_bool_setting( - settings, - openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY, - ) - .unwrap_or(false) -} - -fn apply_agent_proposals_enabled( - agent_proposals: &AgentProposals, - enabled: bool, - source: &'static str, - config_revision: Option, - sidecar_control_publisher: Option<&sidecar_control::Publisher>, - install_static_skills: impl FnOnce() -> Result, -) { - let previously_enabled = agent_proposals.swap_enabled(enabled); - if enabled == previously_enabled { - return; - } - - info!( - agent_policy_proposals_enabled = enabled, - source, config_revision, "agent-driven policy proposals toggled" - ); - - if let (Some(publisher), Some(config_revision)) = (sidecar_control_publisher, config_revision) { - publisher.publish_agent_proposals(enabled, config_revision); - } - - if enabled && !previously_enabled { - match install_static_skills() { - Ok(installed) => info!( - path = %installed.policy_advisor.display(), - "Installed sandbox agent skill on toggle-on" - ), - Err(error) => warn!( - error = %error, - "Failed to install sandbox agent skill on toggle-on" - ), - } - } -} - -/// Log individual setting changes between two snapshots. -fn log_setting_changes( - old: &std::collections::HashMap, - new: &std::collections::HashMap, -) { - for (key, new_es) in new { - let new_val = format_setting_value(new_es); - match old.get(key) { - Some(old_es) => { - let old_val = format_setting_value(old_es); - if old_val != new_val { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "updated") - .unmapped("key", serde_json::json!(key)) - .unmapped("old", serde_json::json!(old_val.clone())) - .unmapped("new", serde_json::json!(new_val.clone())) - .message(format!( - "Setting changed [key:{key} old:{old_val} new:{new_val}]" - )) - .build() - ); - } - } - None => { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "enabled") - .unmapped("key", serde_json::json!(key)) - .unmapped("value", serde_json::json!(new_val.clone())) - .message(format!("Setting added [key:{key} value:{new_val}]")) - .build() - ); - } - } - } - for key in old.keys() { - if !new.contains_key(key) { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Disabled, "disabled") - .unmapped("key", serde_json::json!(key)) - .message(format!("Setting removed [key:{key}]")) - .build() - ); - } - } -} - -/// Format an `EffectiveSetting` value for log display. -fn format_setting_value(es: &openshell_core::proto::EffectiveSetting) -> String { - use openshell_core::proto::setting_value; - match es.value.as_ref().and_then(|sv| sv.value.as_ref()) { - None => "".to_string(), - Some(setting_value::Value::StringValue(v)) => v.clone(), - Some(setting_value::Value::BoolValue(v)) => v.to_string(), - Some(setting_value::Value::IntValue(v)) => v.to_string(), - Some(setting_value::Value::BytesValue(_)) => "".to_string(), - } -} - -#[cfg(test)] -#[allow( - clippy::needless_raw_string_hashes, - clippy::iter_on_single_items, - clippy::similar_names, - clippy::manual_string_new, - clippy::doc_markdown, - reason = "Test code: test fixtures often use idiomatic forms not flagged in production." -)] -mod tests { - use super::*; - - #[test] - fn transparent_tcp_capability_requires_exact_driver_marker() { - let required = openshell_core::sandbox_env::POLICY_DNS_TRANSPARENT_TCP_CAPABILITY; - assert!(!has_network_runtime_capability(None, required)); - assert!(!has_network_runtime_capability(Some(""), required)); - assert!(!has_network_runtime_capability( - Some("policy-dns-transparent-tcp-extra"), - required - )); - assert!(has_network_runtime_capability( - Some("other, policy-dns-transparent-tcp"), - required - )); - } - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkMode, NetworkPolicy, ProcessPolicy, ProxyPolicy, - }; - use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; - - fn proxy_policy(http_addr: Option) -> SandboxPolicy { - SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy { - mode: NetworkMode::Proxy, - proxy: Some(ProxyPolicy { http_addr }), - }, - landlock: LandlockPolicy::default(), - process: ProcessPolicy::default(), - } - } - - fn effective_bool(value: bool) -> openshell_core::proto::EffectiveSetting { - openshell_core::proto::EffectiveSetting { - value: Some(openshell_core::proto::SettingValue { - value: Some(openshell_core::proto::setting_value::Value::BoolValue( - value, - )), - }), - scope: openshell_core::proto::SettingScope::Global.into(), - } - } - - #[test] - fn sidecar_process_policy_sets_loopback_proxy_addr() { - let policy = proxy_policy(None); - - let process_policy = process_policy_for_topology(&policy, true).unwrap(); - - let http_addr = process_policy - .network - .proxy - .and_then(|proxy| proxy.http_addr) - .expect("sidecar process policy should set proxy address"); - assert_eq!(http_addr.to_string(), SIDECAR_PROCESS_PROXY_ADDR); - assert!( - policy - .network - .proxy - .as_ref() - .expect("original policy should keep proxy config") - .http_addr - .is_none(), - "process policy normalization must not mutate the network policy" - ); - } - - #[test] - fn non_sidecar_process_policy_preserves_proxy_addr() { - let policy = proxy_policy(None); - - let process_policy = process_policy_for_topology(&policy, false).unwrap(); - - assert!( - process_policy - .network - .proxy - .and_then(|proxy| proxy.http_addr) - .is_none() - ); - } - - #[tokio::test] - async fn sidecar_control_provider_env_update_orders_by_generation() { - let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); - let provider_credentials = ProviderCredentialState::from_child_env_snapshot( - u64::MAX, - std::collections::HashMap::from([("TOKEN".to_string(), "old".to_string())]), - ); - let agent_proposals = AgentProposals::new(true); - let handle = spawn_sidecar_control_update_watcher( - rx, - provider_credentials.clone(), - agent_proposals.clone(), - Arc::new(tokio::sync::Mutex::new(None)), - 10, - ); - - tx.send(sidecar_control::ControlUpdate::ProviderEnv { - revision: 1, - generation: 11, - provider_child_env: std::collections::HashMap::from([( - "TOKEN".to_string(), - "new".to_string(), - )]), - }) - .unwrap(); - - timeout(Duration::from_secs(1), async { - loop { - if provider_credentials.snapshot().revision == 1 { - break; - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .unwrap(); - let snapshot = provider_credentials.snapshot(); - assert_eq!(snapshot.revision, 1); - assert_eq!( - snapshot.child_env.get("TOKEN").map(String::as_str), - Some("new") - ); - - tx.send(sidecar_control::ControlUpdate::ProviderEnv { - revision: 2, - generation: 11, - provider_child_env: std::collections::HashMap::from([( - "TOKEN".to_string(), - "duplicate-generation".to_string(), - )]), - }) - .unwrap(); - tx.send(sidecar_control::ControlUpdate::AgentProposals { - enabled: false, - config_revision: 1, - }) - .unwrap(); - timeout(Duration::from_secs(1), async { - while agent_proposals.enabled() { - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .unwrap(); - assert_eq!( - provider_credentials - .snapshot() - .child_env - .get("TOKEN") - .map(String::as_str), - Some("new") - ); - - tx.send(sidecar_control::ControlUpdate::ProviderEnv { - revision: 2, - generation: 12, - provider_child_env: std::collections::HashMap::from([( - "TOKEN".to_string(), - "newest".to_string(), - )]), - }) - .unwrap(); - timeout(Duration::from_secs(1), async { - loop { - if provider_credentials.snapshot().revision == 2 { - break; - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .unwrap(); - - tx.send(sidecar_control::ControlUpdate::ProviderEnv { - revision: u64::MAX, - generation: 11, - provider_child_env: std::collections::HashMap::from([( - "TOKEN".to_string(), - "stale".to_string(), - )]), - }) - .unwrap(); - tx.send(sidecar_control::ControlUpdate::AgentProposals { - enabled: true, - config_revision: 2, - }) - .unwrap(); - timeout(Duration::from_secs(1), async { - while !agent_proposals.enabled() { - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .unwrap(); - let snapshot = provider_credentials.snapshot(); - assert_eq!(snapshot.revision, 2); - assert_eq!( - snapshot.child_env.get("TOKEN").map(String::as_str), - Some("newest") - ); - handle.abort(); - } - - #[tokio::test] - async fn sidecar_control_agent_proposals_update_flips_shared_state() { - let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); - let provider_credentials = - ProviderCredentialState::from_child_env_snapshot(0, std::collections::HashMap::new()); - let agent_proposals = AgentProposals::new(true); - let handle = spawn_sidecar_control_update_watcher( - rx, - provider_credentials, - agent_proposals.clone(), - Arc::new(tokio::sync::Mutex::new(None)), - 0, - ); - - tx.send(sidecar_control::ControlUpdate::AgentProposals { - enabled: false, - config_revision: 5, - }) - .unwrap(); - - timeout(Duration::from_secs(1), async { - loop { - if !agent_proposals.enabled() { - break; - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .unwrap(); - handle.abort(); - } - - #[test] - fn apply_agent_proposals_enabled_installs_only_on_false_to_true() { - let agent_proposals = AgentProposals::default(); - let installs = AtomicUsize::new(0); - - apply_agent_proposals_enabled(&agent_proposals, true, "test", Some(1), None, || { - installs.fetch_add(1, Ordering::Relaxed); - Ok(skills::InstalledSkills { - policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), - policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), - agents: None, - }) - }); - assert!(agent_proposals.enabled()); - assert_eq!(installs.load(Ordering::Relaxed), 1); - - apply_agent_proposals_enabled(&agent_proposals, true, "test", Some(2), None, || { - installs.fetch_add(1, Ordering::Relaxed); - Ok(skills::InstalledSkills { - policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), - policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), - agents: None, - }) - }); - assert_eq!(installs.load(Ordering::Relaxed), 1); - - apply_agent_proposals_enabled(&agent_proposals, false, "test", Some(3), None, || { - installs.fetch_add(1, Ordering::Relaxed); - Ok(skills::InstalledSkills { - policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), - policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), - agents: None, - }) - }); - assert!(!agent_proposals.enabled()); - assert_eq!(installs.load(Ordering::Relaxed), 1); - } - - #[test] - fn apply_ocsf_json_setting_enables_from_initial_settings_snapshot() { - let enabled = AtomicBool::new(false); - let mut settings = std::collections::HashMap::new(); - settings.insert("ocsf_json_enabled".to_string(), effective_bool(true)); - - apply_ocsf_json_setting(&enabled, &settings); - - assert!(enabled.load(Ordering::Relaxed)); - } - - #[test] - fn apply_ocsf_json_setting_disables_when_setting_is_unset() { - let enabled = AtomicBool::new(true); - let settings = std::collections::HashMap::new(); - - apply_ocsf_json_setting(&enabled, &settings); - - assert!(!enabled.load(Ordering::Relaxed)); - } - - #[test] - fn agent_proposals_setting_enables_from_initial_settings_snapshot() { - let mut settings = std::collections::HashMap::new(); - settings.insert( - openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY.to_string(), - effective_bool(true), - ); - - assert!(agent_proposals_enabled_from_settings(&settings)); - } - - #[test] - fn agent_proposals_setting_defaults_false_when_unset() { - let settings = std::collections::HashMap::new(); - - assert!(!agent_proposals_enabled_from_settings(&settings)); - } - - // ---- Policy disk discovery tests ---- - - #[test] - fn discover_policy_from_nonexistent_path_returns_restrictive_default() { - let path = std::path::Path::new("/nonexistent/policy.yaml"); - let policy = discover_policy_from_path(path); - // Restrictive default has no network policies. - assert!(policy.network_policies.is_empty()); - // It keeps filesystem restrictions while leaving identity to the - // active compute driver. - assert!(policy.filesystem.is_some()); - assert!(policy.process.is_none()); - } - - #[test] - fn discover_policy_from_valid_yaml_file() { - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("policy.yaml"); - std::fs::write( - &path, - r#" -version: 1 -filesystem_policy: - include_workdir: false - read_only: - - /usr - read_write: - - /tmp -network_policies: - test: - name: test - endpoints: - - { host: example.com, port: 443 } - binaries: - - { path: /usr/bin/curl } -"#, - ) - .unwrap(); - - let policy = discover_policy_from_path(&path); - assert_eq!(policy.network_policies.len(), 1); - assert!(policy.network_policies.contains_key("test")); - let fs = policy.filesystem.unwrap(); - assert!(!fs.include_workdir); - } - - #[test] - fn discover_policy_from_invalid_yaml_returns_restrictive_default() { - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("policy.yaml"); - std::fs::write(&path, "this is not valid yaml: [[[").unwrap(); - - let policy = discover_policy_from_path(&path); - // Falls back to restrictive default. - assert!(policy.network_policies.is_empty()); - assert!(policy.filesystem.is_some()); - } - - #[test] - fn discover_policy_from_unsafe_yaml_falls_back_to_default() { - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("policy.yaml"); - std::fs::write( - &path, - r#" -version: 1 -process: - run_as_user: root - run_as_group: root -filesystem_policy: - include_workdir: true - read_only: - - /usr - read_write: - - /tmp -"#, - ) - .unwrap(); - - let policy = discover_policy_from_path(&path); - // Falls back to restrictive default because of root user. - assert!(policy.process.is_none()); - } - - #[test] - fn discover_policy_restrictive_default_blocks_network() { - // Cluster sandboxes keep proxy mode enabled so egress is evaluated by - // the network policy engine. - let proto = openshell_policy::restrictive_default_policy(); - let local_policy = SandboxPolicy::try_from(proto).expect("conversion should succeed"); - assert!(matches!(local_policy.network.mode, NetworkMode::Proxy)); - } - - // ---- Initial policy acknowledgement tests ---- - - fn proto_policy_fixture() -> openshell_core::proto::SandboxPolicy { - openshell_policy::restrictive_default_policy() - } - - fn proto_tcp_policy_fixture() -> openshell_core::proto::SandboxPolicy { - openshell_policy::parse_sandbox_policy( - r#" -version: 1 -network_policies: - redis: - name: redis - endpoints: - - host: redis.example.com - port: 6379 - protocol: tcp - binaries: - - path: /usr/bin/redis-cli -"#, - ) - .expect("parse TCP policy") - } - - fn settings_poll_result( - policy: Option, - version: u32, - source: openshell_core::proto::PolicySource, - ) -> openshell_core::grpc_client::SettingsPollResult { - openshell_core::grpc_client::SettingsPollResult { - policy, - version, - policy_hash: format!("hash-v{version}"), - config_revision: u64::from(version) * 100, - policy_source: source, - settings: std::collections::HashMap::new(), - global_policy_version: 0, - provider_env_revision: 0, - supervisor_middleware_services: Vec::new(), - workspace: String::new(), - policy_validation_failure_mode: PolicyValidationFailureMode::default(), - extension_authentication_enabled: false, - } - } - - #[derive(Clone)] - struct ScriptedPolicyGateway { - polls: Arc< - tokio::sync::Mutex< - tokio::sync::mpsc::UnboundedReceiver< - openshell_core::grpc_client::SettingsPollResult, - >, - >, - >, - reports: UnboundedSender<(u32, bool, String)>, - } - - #[tonic::async_trait] - impl PolicyGatewayClient for ScriptedPolicyGateway { - async fn poll_settings( - &self, - _sandbox_id: &str, - ) -> Result { - self.polls - .lock() - .await - .recv() - .await - .ok_or_else(|| miette::miette!("scripted policy poll channel closed")) - } - - async fn report_policy_status( - &self, - _sandbox_id: &str, - version: u32, - loaded: bool, - error: &str, - ) -> Result<()> { - self.reports - .send((version, loaded, error.to_string())) - .map_err(|_| miette::miette!("scripted policy report channel closed")) - } - - fn workspace(&self) -> String { - "test-workspace".to_string() - } - } - - #[derive(Clone)] - struct CredentialRejectingPolicyGateway { - inner: ScriptedPolicyGateway, - credential_requests: Arc, - } - - #[tonic::async_trait] - impl PolicyGatewayClient for CredentialRejectingPolicyGateway { - async fn poll_settings( - &self, - sandbox_id: &str, - ) -> Result { - self.inner.poll_settings(sandbox_id).await - } - - async fn report_policy_status( - &self, - sandbox_id: &str, - version: u32, - loaded: bool, - error: &str, - ) -> Result<()> { - self.inner - .report_policy_status(sandbox_id, version, loaded, error) - .await - } - - async fn extension_credentials_for( - &self, - _services: &[openshell_core::proto::SupervisorMiddlewareService], - ) -> Result> - { - self.credential_requests.fetch_add(1, Ordering::SeqCst); - Err(miette::miette!( - "gateway extension authentication is unavailable" - )) - } - - fn workspace(&self) -> String { - self.inner.workspace() - } - } - - fn scripted_policy_gateway() -> ( - ScriptedPolicyGateway, - UnboundedSender, - tokio::sync::mpsc::UnboundedReceiver<(u32, bool, String)>, - ) { - let (poll_tx, poll_rx) = tokio::sync::mpsc::unbounded_channel(); - let (report_tx, report_rx) = tokio::sync::mpsc::unbounded_channel(); - ( - ScriptedPolicyGateway { - polls: Arc::new(tokio::sync::Mutex::new(poll_rx)), - reports: report_tx, - }, - poll_tx, - report_rx, - ) - } - - fn policy_poll_test_context( - opa_engine: Arc, - loaded_policy_origin: LoadedPolicyOrigin, - middleware_connector: MiddlewareConnector, - ) -> PolicyPollLoopContext { - let (workspace_tx, _workspace_rx) = tokio::sync::watch::channel(String::new()); - PolicyPollLoopContext { - endpoint: String::new(), - sandbox_id: "sandbox-test".to_string(), - opa_engine, - loaded_policy_origin, - entrypoint_pid: Arc::new(AtomicU32::new(0)), - interval_secs: 0, - ocsf_enabled: Arc::new(AtomicBool::new(false)), - ocsf_schema_version: Arc::new(std::sync::Mutex::new(String::new())), - provider_credentials: ProviderCredentialState::from_child_env_snapshot( - 0, - std::collections::HashMap::new(), - ), - policy_local_ctx: None, - agent_proposals: AgentProposals::default(), - middleware_registry_status: MiddlewareRegistryStatus::Synchronized, - sidecar_control_publisher: None, - workspace_tx, - extension_credentials: openshell_extension_core::ExtensionCredentialStore::new(), - extension_authentication_enabled: false, - middleware_connector, - transparent_tcp: TransparentTcpReloadState::default(), - } - } - - async fn expect_policy_report( - reports: &mut tokio::sync::mpsc::UnboundedReceiver<(u32, bool, String)>, - version: u32, - ) { - let report = timeout(Duration::from_secs(1), reports.recv()) - .await - .expect("policy report timed out") - .expect("policy reporter stopped"); - assert_eq!(report, (version, true, String::new())); - } - - async fn expect_no_policy_report( - reports: &mut tokio::sync::mpsc::UnboundedReceiver<(u32, bool, String)>, - ) { - assert!( - timeout(Duration::from_millis(50), reports.recv()) - .await - .is_err(), - "unexpected policy status report" - ); - } - - #[tokio::test] - async fn same_hash_poll_revision_is_acknowledged_once_without_opa_reload() { - let mut v1 = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - v1.policy_hash = "same-policy".to_string(); - let mut v2 = v1.clone(); - v2.version = 2; - v2.config_revision = 200; - - let engine = - Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); - let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); - let ctx = policy_poll_test_context( - engine.clone(), - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_revision), - has_last_valid_policy: true, - }, - default_middleware_connector(), - ); - let (client, polls, mut reports) = scripted_policy_gateway(); - polls.send(v1).unwrap(); - - let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); - expect_policy_report(&mut reports, 1).await; - - polls.send(v2.clone()).unwrap(); - expect_policy_report(&mut reports, 2).await; - polls.send(v2).unwrap(); - expect_no_policy_report(&mut reports).await; - - assert_eq!( - engine.current_generation(), - 0, - "same-hash acknowledgement must not reload OPA" - ); - handle.abort(); - } - - #[tokio::test] - async fn poll_rejects_first_tcp_expansion_and_reports_previous_policy_active() { - let v1 = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - let v2 = settings_poll_result( - Some(proto_tcp_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Sandbox, - ); - let engine = - Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); - let active_generation = engine.current_generation(); - let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); - let mut ctx = policy_poll_test_context( - engine.clone(), - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_revision), - has_last_valid_policy: true, - }, - default_middleware_connector(), - ); - ctx.transparent_tcp = TransparentTcpReloadState { - capable: true, - substrate_ready: false, - }; - let (client, polls, mut reports) = scripted_policy_gateway(); - polls.send(v1).unwrap(); - - let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); - expect_policy_report(&mut reports, 1).await; - polls.send(v2).unwrap(); - let report = timeout(Duration::from_secs(1), reports.recv()) - .await - .expect("TCP rejection report timed out") - .expect("policy reporter stopped"); - - assert_eq!(report.0, 2); - assert!(!report.1); - assert!(report.2.contains("recreate the sandbox"), "{}", report.2); - assert!(report.2.contains("previous policy remains active")); - assert_eq!(engine.current_generation(), active_generation); - assert!(engine.fail_closed_reason().is_none()); - handle.abort(); - } - - #[tokio::test] - async fn same_hash_ack_waits_for_failed_middleware_reconciliation_and_retries_once() { - let mut v1 = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - v1.policy_hash = "same-policy".to_string(); - let mut v2 = v1.clone(); - v2.version = 2; - v2.config_revision = 200; - v2.supervisor_middleware_services = - vec![openshell_core::proto::SupervisorMiddlewareService { - name: "scripted-guard".to_string(), - grpc_endpoint: "http://scripted.invalid".to_string(), - ..Default::default() - }]; - - let connector_attempts = Arc::new(AtomicUsize::new(0)); - let (attempt_tx, mut attempt_rx) = tokio::sync::mpsc::unbounded_channel(); - let middleware_connector: MiddlewareConnector = { - let connector_attempts = connector_attempts.clone(); - Arc::new(move |_services, _authentication| { - let attempt = connector_attempts.fetch_add(1, Ordering::SeqCst) + 1; - attempt_tx.send(attempt).unwrap(); - Box::pin(async move { - if attempt == 1 { - Err(miette::miette!("scripted middleware connection failure")) - } else { - connect_middleware_registry(&[], &MiddlewareAuthentication::default()).await - } - }) - }) - }; - - let engine = - Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); - let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); - let ctx = policy_poll_test_context( - engine.clone(), - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_revision), - has_last_valid_policy: true, - }, - middleware_connector, - ); - let (client, polls, mut reports) = scripted_policy_gateway(); - polls.send(v1).unwrap(); - - let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); - expect_policy_report(&mut reports, 1).await; - - polls.send(v2.clone()).unwrap(); - assert_eq!( - timeout(Duration::from_secs(1), attempt_rx.recv()) - .await - .unwrap(), - Some(1) - ); - expect_no_policy_report(&mut reports).await; - assert_eq!(engine.current_generation(), 0); - - polls.send(v2.clone()).unwrap(); - assert_eq!( - timeout(Duration::from_secs(1), attempt_rx.recv()) - .await - .unwrap(), - Some(2) - ); - expect_policy_report(&mut reports, 2).await; - assert_eq!(engine.current_generation(), 1); - - polls.send(v2).unwrap(); - expect_no_policy_report(&mut reports).await; - assert_eq!(connector_attempts.load(Ordering::SeqCst), 2); - handle.abort(); - } - - #[tokio::test] - async fn no_signer_capability_uses_legacy_middleware_connector_without_credentials() { - let mut v1 = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - v1.policy_hash = "same-policy".to_string(); - let mut v2 = v1.clone(); - v2.version = 2; - v2.config_revision = 200; - v2.supervisor_middleware_services = - vec![openshell_core::proto::SupervisorMiddlewareService { - name: "legacy-guard".to_string(), - grpc_endpoint: "http://legacy.invalid".to_string(), - ..Default::default() - }]; - assert!(!v2.extension_authentication_enabled); - - let (inner, polls, mut reports) = scripted_policy_gateway(); - let credential_requests = Arc::new(AtomicUsize::new(0)); - let client = CredentialRejectingPolicyGateway { - inner, - credential_requests: credential_requests.clone(), - }; - let (connector_tx, mut connector_rx) = tokio::sync::mpsc::unbounded_channel(); - let connector: MiddlewareConnector = Arc::new(move |_services, authentication| { - connector_tx - .send((authentication.credentials.len(), authentication.enabled)) - .unwrap(); - Box::pin(async move { - connect_middleware_registry(&[], &MiddlewareAuthentication::default()).await - }) - }); - let engine = - Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); - let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); - let ctx = policy_poll_test_context( - engine, - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_revision), - has_last_valid_policy: true, - }, - connector, - ); - - polls.send(v1).unwrap(); - let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); - expect_policy_report(&mut reports, 1).await; - polls.send(v2).unwrap(); - assert_eq!( - timeout(Duration::from_secs(1), connector_rx.recv()) - .await - .unwrap(), - Some((0, false)) - ); - expect_policy_report(&mut reports, 2).await; - assert_eq!(credential_requests.load(Ordering::SeqCst), 0); - handle.abort(); - } - - #[tokio::test] - async fn enabled_extension_authentication_keeps_credential_failure_fail_closed() { - let mut v1 = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - v1.policy_hash = "same-policy".to_string(); - let mut v2 = v1.clone(); - v2.version = 2; - v2.config_revision = 200; - v2.extension_authentication_enabled = true; - v2.supervisor_middleware_services = - vec![openshell_core::proto::SupervisorMiddlewareService { - name: "authenticated-guard".to_string(), - grpc_endpoint: "https://guard.invalid".to_string(), - ..Default::default() - }]; - - let (inner, polls, mut reports) = scripted_policy_gateway(); - let credential_requests = Arc::new(AtomicUsize::new(0)); - let client = CredentialRejectingPolicyGateway { - inner, - credential_requests: credential_requests.clone(), - }; - let (connector_tx, mut connector_rx) = tokio::sync::mpsc::unbounded_channel(); - let connector: MiddlewareConnector = Arc::new(move |_services, authentication| { - connector_tx - .send((authentication.credentials.len(), authentication.enabled)) - .unwrap(); - Box::pin(async move { - if authentication.enabled && authentication.credentials.is_empty() { - Err(miette::miette!( - "missing authenticated middleware credential" - )) - } else { - connect_middleware_registry(&[], &authentication).await - } - }) - }); - let engine = - Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); - let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); - let ctx = policy_poll_test_context( - engine, - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_revision), - has_last_valid_policy: true, - }, - connector, - ); - - polls.send(v1).unwrap(); - let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); - expect_policy_report(&mut reports, 1).await; - polls.send(v2).unwrap(); - assert_eq!( - timeout(Duration::from_secs(1), connector_rx.recv()) - .await - .unwrap(), - Some((0, true)) - ); - expect_no_policy_report(&mut reports).await; - assert_eq!(credential_requests.load(Ordering::SeqCst), 1); - handle.abort(); - } - - async fn assert_poll_does_not_use_same_hash_acknowledgement( - initial: openshell_core::grpc_client::SettingsPollResult, - next: openshell_core::grpc_client::SettingsPollResult, - origin: LoadedPolicyOrigin, - initial_report: Option, - ) { - let engine = - Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); - let ctx = policy_poll_test_context(engine.clone(), origin, default_middleware_connector()); - let (client, polls, mut reports) = scripted_policy_gateway(); - polls.send(initial).unwrap(); - let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); - - if let Some(version) = initial_report { - expect_policy_report(&mut reports, version).await; - } else { - expect_no_policy_report(&mut reports).await; - } - - polls.send(next).unwrap(); - expect_no_policy_report(&mut reports).await; - assert_eq!( - engine.current_generation(), - 0, - "negative same-hash scope must not reload OPA" - ); - handle.abort(); - } - - #[tokio::test] - async fn same_hash_ack_poll_loop_rejects_local_global_empty_equal_and_older_scopes() { - let mut sandbox_v1 = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - sandbox_v1.policy_hash = "same-policy".to_string(); - let loaded_v1 = LoadedPolicyRevision::from_snapshot(&sandbox_v1); - let mut sandbox_v2 = sandbox_v1.clone(); - sandbox_v2.version = 2; - sandbox_v2.config_revision = 200; - - assert_poll_does_not_use_same_hash_acknowledgement( - sandbox_v1.clone(), - sandbox_v2.clone(), - LoadedPolicyOrigin::LocalOverride, - None, - ) - .await; - - let mut global_v2 = sandbox_v2.clone(); - global_v2.policy_source = openshell_core::proto::PolicySource::Global; - assert_poll_does_not_use_same_hash_acknowledgement( - sandbox_v1.clone(), - global_v2, - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_v1.clone()), - has_last_valid_policy: true, - }, - Some(1), - ) - .await; - - let mut empty_v1 = sandbox_v1.clone(); - empty_v1.policy_hash.clear(); - let empty_loaded = LoadedPolicyRevision::from_snapshot(&empty_v1); - let mut empty_v2 = sandbox_v2.clone(); - empty_v2.policy_hash.clear(); - assert_poll_does_not_use_same_hash_acknowledgement( - empty_v1, - empty_v2, - LoadedPolicyOrigin::Gateway { - revision: Some(empty_loaded), - has_last_valid_policy: true, - }, - Some(1), - ) - .await; - - assert_poll_does_not_use_same_hash_acknowledgement( - sandbox_v1.clone(), - sandbox_v1.clone(), - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_v1.clone()), - has_last_valid_policy: true, - }, - Some(1), - ) - .await; - - let loaded_v2 = LoadedPolicyRevision::from_snapshot(&sandbox_v2); - assert_poll_does_not_use_same_hash_acknowledgement( - sandbox_v2, - sandbox_v1, - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_v2), - has_last_valid_policy: true, - }, - Some(2), - ) - .await; - } - - #[tokio::test] - async fn changed_hash_poll_uses_normal_opa_reload_and_status_path() { - let v1 = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - let v2 = settings_poll_result( - Some(proto_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Sandbox, - ); - let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); - let engine = - Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); - let ctx = policy_poll_test_context( - engine.clone(), - LoadedPolicyOrigin::Gateway { - revision: Some(loaded_revision), - has_last_valid_policy: true, - }, - default_middleware_connector(), - ); - let (client, polls, mut reports) = scripted_policy_gateway(); - polls.send(v1).unwrap(); - let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); - - expect_policy_report(&mut reports, 1).await; - polls.send(v2).unwrap(); - expect_policy_report(&mut reports, 2).await; - assert_eq!( - engine.current_generation(), - 1, - "changed policy content must still reload OPA" - ); - handle.abort(); - } - - #[tokio::test] - async fn failed_external_startup_registry_build_preserves_installed_builtins() { - let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); - install_builtin_middleware_registry(&engine) - .await - .expect("install built-in middleware registry"); - let builtins_generation = engine.current_generation(); - assert_eq!(builtins_generation, 1); - - let invalid_external = openshell_core::proto::SupervisorMiddlewareService { - name: "unavailable-guard".into(), - grpc_endpoint: "http://127.0.0.1:1".into(), - max_payload_bytes: 1024, - ..Default::default() - }; - connect_middleware_registry(&[invalid_external], &MiddlewareAuthentication::default()) - .await - .expect_err("unavailable external service must not replace built-ins"); - - assert_eq!(engine.current_generation(), builtins_generation); - } - - #[tokio::test] - async fn unavailable_middleware_reload_keeps_last_known_good_runtime_active() { - let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); - install_builtin_middleware_registry(&engine) - .await - .expect("install built-in middleware registry"); - let active_generation = engine.current_generation(); - let unavailable_service = openshell_core::proto::SupervisorMiddlewareService { - name: "unavailable-guard".into(), - grpc_endpoint: "http://127.0.0.1:1".into(), - max_payload_bytes: 1024, - ..Default::default() - }; - - let failure = reload_gateway_policy_runtime( - &engine, - Some(&proto_policy_fixture()), - 0, - MiddlewareReloadContext { - desired_services: &[unavailable_service], - authentication: &MiddlewareAuthentication::default(), - registry_changed: true, - connector: &default_middleware_connector(), - }, - TransparentTcpReloadState::default(), - ) - .await - .expect_err("unavailable middleware must fail candidate preparation"); - let disposition = apply_gateway_runtime_reload_failure( - &engine, - failure, - PolicyValidationFailureMode::FailClosed, - true, - 2, - ) - .expect("middleware failure handling must succeed"); - - assert!(matches!( - disposition, - GatewayRuntimeFailureDisposition::MiddlewareUnavailable { .. } - )); - assert_eq!(engine.current_generation(), active_generation); - assert!(engine.fail_closed_reason().is_none()); - } - - #[tokio::test] - async fn tcp_policy_reload_without_startup_substrate_is_rejected_and_keeps_previous_policy() { - let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); - let active_generation = engine.current_generation(); - - let failure = reload_gateway_policy_runtime( - &engine, - Some(&proto_tcp_policy_fixture()), - 0, - MiddlewareReloadContext { - desired_services: &[], - authentication: &MiddlewareAuthentication::default(), - registry_changed: false, - connector: &default_middleware_connector(), - }, - TransparentTcpReloadState { - capable: true, - substrate_ready: false, - }, - ) - .await - .expect_err("TCP expansion must require startup substrate"); - let disposition = apply_gateway_runtime_reload_failure( - &engine, - failure, - PolicyValidationFailureMode::FailClosed, - true, - 2, - ) - .expect("runtime prerequisite failure handling must succeed"); - - assert!(matches!( - disposition, - GatewayRuntimeFailureDisposition::TransparentTcpExpansionRejected { - active_generation: generation, - .. - } if generation == active_generation - )); - assert_eq!(engine.current_generation(), active_generation); - assert!(engine.fail_closed_reason().is_none()); - } - - #[tokio::test] - async fn tcp_policy_reload_on_unsupported_runtime_is_rejected() { - let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); - - let failure = reload_gateway_policy_runtime( - &engine, - Some(&proto_tcp_policy_fixture()), - 0, - MiddlewareReloadContext { - desired_services: &[], - authentication: &MiddlewareAuthentication::default(), - registry_changed: false, - connector: &default_middleware_connector(), - }, - TransparentTcpReloadState::default(), - ) - .await - .expect_err("unsupported runtime must reject TCP expansion"); - - assert!(matches!( - failure, - GatewayRuntimeReloadError::TransparentTcpPrerequisite(_) - )); - assert_eq!(engine.current_generation(), 0); - } - - #[test] - fn policy_rejection_after_middleware_outage_is_not_deduplicated() { - let engine = OpaEngine::from_strings( - include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), - "network_policies: {}\n", - ) - .unwrap(); - let middleware_failure = GatewayRuntimeReloadError::MiddlewareRegistry(miette::miette!( - "middleware service unavailable" - )); - let first_failure = FailedRuntimeRevision::new(42, "sha256:candidate", &middleware_failure); - let middleware_disposition = apply_gateway_runtime_reload_failure( - &engine, - middleware_failure, - PolicyValidationFailureMode::FailClosed, - true, - 7, - ) - .unwrap(); - - assert!(matches!( - middleware_disposition, - GatewayRuntimeFailureDisposition::MiddlewareUnavailable { .. } - )); - assert!(engine.fail_closed_reason().is_none()); - - let policy_failure = GatewayRuntimeReloadError::PolicyValidation(miette::miette!( - "conflicting endpoint metadata" - )); - let second_failure = FailedRuntimeRevision::new(42, "sha256:candidate", &policy_failure); - assert_ne!( - first_failure, second_failure, - "a changed failure class for the same candidate must be handled" - ); - - let policy_disposition = apply_gateway_runtime_reload_failure( - &engine, - policy_failure, - PolicyValidationFailureMode::FailClosed, - true, - 7, - ) - .unwrap(); - assert!(matches!( - policy_disposition, - GatewayRuntimeFailureDisposition::PolicyRejected { .. } - )); - assert!(engine.fail_closed_reason().is_some()); - } - - #[test] - fn failed_gateway_runtime_snapshot_is_retried_without_revision_change() { - let services = Vec::new(); - - assert!(gateway_policy_runtime_needs_reconciliation( - true, - "hash-v1", - "hash-v1", - &services, - &services, - MiddlewareRegistryStatus::NeedsReconciliation, - )); - assert!(!gateway_policy_runtime_needs_reconciliation( - true, - "hash-v1", - "hash-v1", - &services, - &services, - MiddlewareRegistryStatus::Synchronized, - )); - } - - #[test] - fn gateway_runtime_reconciliation_tracks_policy_and_service_changes() { - let no_services = Vec::new(); - let desired_services = vec![openshell_core::proto::SupervisorMiddlewareService { - name: "guard".into(), - ..Default::default() - }]; - - assert!(gateway_policy_runtime_needs_reconciliation( - true, - "hash-v1", - "hash-v2", - &no_services, - &no_services, - MiddlewareRegistryStatus::Synchronized, - )); - assert!(gateway_policy_runtime_needs_reconciliation( - true, - "hash-v1", - "hash-v1", - &no_services, - &desired_services, - MiddlewareRegistryStatus::Synchronized, - )); - assert!(!gateway_policy_runtime_needs_reconciliation( - false, - "local-policy", - "hash-v2", - &no_services, - &desired_services, - MiddlewareRegistryStatus::NeedsReconciliation, - )); - } - - #[test] - fn policy_only_change_does_not_rebuild_middleware_registry() { - let services = vec![openshell_core::proto::SupervisorMiddlewareService { - name: "guard".into(), - ..Default::default() - }]; - - // The runtime must reconcile, but the registry (and therefore - // middleware reachability) is not part of that reconciliation. - assert!(gateway_policy_runtime_needs_reconciliation( - true, - "hash-v1", - "hash-v2", - &services, - &services, - MiddlewareRegistryStatus::Synchronized, - )); - assert!(!middleware_registry_needs_rebuild( - MiddlewareRegistryStatus::Synchronized, - &services, - &services, - )); - } - - #[test] - fn registry_rebuild_requires_service_set_change_or_degraded_registry() { - let no_services = Vec::new(); - let desired_services = vec![openshell_core::proto::SupervisorMiddlewareService { - name: "guard".into(), - ..Default::default() - }]; - - assert!(middleware_registry_needs_rebuild( - MiddlewareRegistryStatus::Synchronized, - &no_services, - &desired_services, - )); - assert!(middleware_registry_needs_rebuild( - MiddlewareRegistryStatus::NeedsReconciliation, - &desired_services, - &desired_services, - )); - assert!(!middleware_registry_needs_rebuild( - MiddlewareRegistryStatus::Synchronized, - &desired_services, - &desired_services, - )); - } - - #[test] - fn initial_ack_candidate_matches_sandbox_revision() { - let canonical = settings_poll_result( - Some(proto_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Sandbox, - ); - let loaded = LoadedPolicyRevision::from_snapshot(&canonical); - - let ack = initial_policy_ack_candidate(Some(&loaded), &canonical) - .expect("sandbox-sourced matching revision should be acknowledged"); - - assert_eq!(ack.version, 2); - assert_eq!(ack.policy_hash, "hash-v2"); - assert_eq!(ack.config_revision, 200); - } - - #[test] - fn initial_ack_candidate_ignores_global_policy() { - let canonical = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Global, - ); - let loaded = LoadedPolicyRevision::from_snapshot(&canonical); - - assert!(initial_policy_ack_candidate(Some(&loaded), &canonical).is_none()); - } - - #[test] - fn initial_ack_candidate_ignores_version_zero() { - let canonical = settings_poll_result( - Some(proto_policy_fixture()), - 0, - openshell_core::proto::PolicySource::Sandbox, - ); - let loaded = LoadedPolicyRevision::from_snapshot(&canonical); - - assert!(initial_policy_ack_candidate(Some(&loaded), &canonical).is_none()); - } - - #[test] - fn initial_ack_candidate_ignores_local_file_mode() { - // Local-file mode retains no proto policy, so there is nothing to - // acknowledge to the gateway. - let canonical = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - - assert!(initial_policy_ack_candidate(None, &canonical).is_none()); - } - - #[test] - fn initial_ack_candidate_rejects_mismatched_identity() { - let loaded_snapshot = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - let loaded = LoadedPolicyRevision::from_snapshot(&loaded_snapshot); - let canonical = settings_poll_result( - Some(proto_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Sandbox, - ); - - assert!(initial_policy_ack_candidate(Some(&loaded), &canonical).is_none()); - } - - #[test] - fn initial_poll_reconciles_provider_composition_that_was_not_loaded() { - let loaded_snapshot = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - let loaded = LoadedPolicyRevision::from_snapshot(&loaded_snapshot); - let mut newer = proto_policy_fixture(); - newer.network_policies.insert( - "_provider_work_github".to_string(), - openshell_core::proto::NetworkPolicyRule::default(), - ); - let canonical = - settings_poll_result(Some(newer), 1, openshell_core::proto::PolicySource::Sandbox); - let canonical = openshell_core::grpc_client::SettingsPollResult { - policy_hash: "hash-provider-change".to_string(), - config_revision: loaded.config_revision + 1, - ..canonical - }; - - assert_eq!( - initial_poll_disposition( - &LoadedPolicyOrigin::Gateway { - revision: Some(loaded), - has_last_valid_policy: true, - }, - &canonical, - ), - InitialPollDisposition::Reconcile - ); - } - - #[test] - fn initial_poll_tracks_local_override_without_reconciliation() { - let canonical = settings_poll_result( - Some(proto_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Sandbox, - ); - - assert_eq!( - initial_poll_disposition(&LoadedPolicyOrigin::LocalOverride, &canonical), - InitialPollDisposition::TrackOnly - ); - assert!(!LoadedPolicyOrigin::LocalOverride.allows_gateway_policy_reload()); - } - - #[test] - fn initial_poll_reconciles_unbound_gateway_policy() { - let canonical = settings_poll_result( - Some(proto_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Sandbox, - ); - let origin = LoadedPolicyOrigin::Gateway { - revision: None, - has_last_valid_policy: true, - }; - - assert_eq!( - initial_poll_disposition(&origin, &canonical), - InitialPollDisposition::Reconcile - ); - assert!(origin.allows_gateway_policy_reload()); - } - - #[test] - fn unchanged_sandbox_policy_revision_candidate_is_strictly_scoped() { - let sandbox_result = openshell_core::grpc_client::SettingsPollResult { - policy_hash: "same-policy".to_string(), - ..settings_poll_result( - Some(proto_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Sandbox, - ) - }; - - assert_eq!( - unchanged_policy_revision_candidate(true, false, 1, "same-policy", &sandbox_result), - Some(2) - ); - assert_eq!( - unchanged_policy_revision_candidate(true, false, 2, "same-policy", &sandbox_result), - None - ); - assert_eq!( - unchanged_policy_revision_candidate( - true, - false, - 1, - "different-policy", - &sandbox_result, - ), - None - ); - assert_eq!( - unchanged_policy_revision_candidate(false, false, 1, "same-policy", &sandbox_result), - None - ); - assert_eq!( - unchanged_policy_revision_candidate(true, false, 1, "", &sandbox_result), - None - ); - assert_eq!( - unchanged_policy_revision_candidate(true, true, 1, "same-policy", &sandbox_result), - None - ); - - let global_result = openshell_core::grpc_client::SettingsPollResult { - policy_hash: "same-policy".to_string(), - ..settings_poll_result( - Some(proto_policy_fixture()), - 2, - openshell_core::proto::PolicySource::Global, - ) - }; - assert_eq!( - unchanged_policy_revision_candidate(true, false, 1, "same-policy", &global_result), - None - ); - } - - #[test] - fn unchanged_policy_revision_waits_for_required_runtime_reconciliation() { - assert_eq!( - unchanged_policy_revision_ready_to_ack(Some(2), false, false), - Some(2), - "a same-hash revision needs no OPA reload" - ); - assert_eq!( - unchanged_policy_revision_ready_to_ack(Some(2), true, false), - None, - "failed runtime reconciliation must keep the revision pending" - ); - assert_eq!( - unchanged_policy_revision_ready_to_ack(Some(2), true, true), - Some(2), - "successful runtime reconciliation permits acknowledgement" - ); - assert_eq!( - unchanged_policy_revision_ready_to_ack(None, false, true), - None, - "runtime success cannot manufacture a revision candidate" - ); - } - - #[test] - fn credential_gating_unavailable_for_local_override_with_credentials() { - assert!(credential_gating_unavailable( - &LoadedPolicyOrigin::LocalOverride, - true, - true - )); - } - - #[test] - fn credential_gating_available_without_local_override_or_credentials() { - // A gateway policy is stamped with provenance, so the gates apply. - assert!(!credential_gating_unavailable( - &LoadedPolicyOrigin::Gateway { - revision: None, - has_last_valid_policy: true, - }, - true, - true - )); - // No provider credentials means there is nothing to leak. - assert!(!credential_gating_unavailable( - &LoadedPolicyOrigin::LocalOverride, - false, - true - )); - // Without networking the proxy never evaluates endpoint provenance. - assert!(!credential_gating_unavailable( - &LoadedPolicyOrigin::LocalOverride, - true, - false - )); - } - - #[test] - fn policy_status_outbox_preserves_all_revision_order() { - let (sender, mut receiver) = tokio::sync::mpsc::unbounded_channel(); - for version in 1..=128 { - enqueue_policy_status(&sender, PolicyStatusUpdate::loaded(version)); - } - - for version in 1..=128 { - assert_eq!( - receiver.try_recv().unwrap(), - PolicyStatusUpdate::loaded(version) - ); - } - } - - #[test] - fn settings_snapshot_carries_workspace_for_policy_sync() { - let mut snapshot = settings_poll_result( - Some(proto_policy_fixture()), - 1, - openshell_core::proto::PolicySource::Sandbox, - ); - snapshot.workspace = "beta".to_string(); - - let revision = LoadedPolicyRevision::from_snapshot(&snapshot); - assert_eq!(revision.version, 1); - assert_eq!( - snapshot.workspace, "beta", - "workspace must survive the snapshot so sync_policy_and_fetch_snapshot receives it" - ); - } - #[test] - fn fail_closed_validation_failure_deactivates_previous_generation() { - let engine = OpaEngine::from_strings( - include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), - "network_policies: {}\n", - ) - .unwrap(); - let previous_generation = engine.current_generation(); - - let disposition = apply_policy_validation_failure( - &engine, - PolicyValidationFailureMode::FailClosed, - true, - 7, - "conflicting tls metadata", - ) - .unwrap(); - - assert!(!disposition.previous_policy_active); - assert!(disposition.active_generation > previous_generation); - assert!( - engine - .fail_closed_reason() - .expect("quarantine reason") - .contains("candidate version 7 rejected") - ); - } - - #[test] - fn retain_validation_failure_keeps_previous_generation_active() { - let engine = OpaEngine::from_strings( - include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), - "network_policies: {}\n", - ) - .unwrap(); - let previous_generation = engine.current_generation(); - - let quarantined = apply_policy_validation_failure( - &engine, - PolicyValidationFailureMode::FailClosed, - true, - 6, - "conflicting tls metadata", - ) - .unwrap(); - assert!(!quarantined.previous_policy_active); - - let disposition = apply_policy_validation_failure( - &engine, - PolicyValidationFailureMode::RetainLastValid, - true, - 7, - "conflicting tls metadata", - ) - .unwrap(); - - assert!(disposition.previous_policy_active); - assert!(disposition.active_generation > quarantined.active_generation); - assert!(disposition.active_generation > previous_generation); - assert!(engine.fail_closed_reason().is_none()); - } - - #[test] - fn retain_validation_failure_without_last_valid_policy_stays_fail_closed() { - let engine = OpaEngine::from_strings( - include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), - "network_policies: {}\n", - ) - .unwrap(); - - let disposition = apply_policy_validation_failure( - &engine, - PolicyValidationFailureMode::RetainLastValid, - false, - 1, - "conflicting tls metadata", - ) - .unwrap(); - - assert_eq!( - disposition.configured_mode, - PolicyValidationFailureMode::RetainLastValid - ); - assert_eq!(disposition.mode, PolicyValidationFailureMode::FailClosed); - assert!(!disposition.previous_policy_active); - assert!(engine.fail_closed_reason().is_some()); - - let [config, _] = policy_validation_failure_events( - &disposition, - 1, - "sha256:test", - "conflicting tls metadata", - ); - let config = config.to_json().unwrap(); - assert_eq!(config["unmapped"]["validation_failure_mode"], "fail_closed"); - assert_eq!( - config["unmapped"]["configured_validation_failure_mode"], - "retain_last_valid" - ); - assert!( - config["message"] - .as_str() - .unwrap() - .contains("previous policy IS NOT active") - ); - } - - #[test] - fn validation_failure_ocsf_states_whether_previous_policy_is_active() { - let fail_closed = PolicyValidationFailureDisposition { - configured_mode: PolicyValidationFailureMode::FailClosed, - mode: PolicyValidationFailureMode::FailClosed, - previous_policy_active: false, - active_generation: 9, - }; - let [config, finding] = policy_validation_failure_events( - &fail_closed, - 8, - "sha256:test", - "conflicting tls metadata", - ); - let config = config.to_json().unwrap(); - assert_eq!(config["class_uid"], 5019); - assert_eq!(config["status"], "Failure"); - assert_eq!(config["unmapped"]["validation_failure_mode"], "fail_closed"); - assert_eq!( - config["unmapped"]["configured_validation_failure_mode"], - "fail_closed" - ); - assert_eq!(config["unmapped"]["previous_policy_active"], false); - assert_eq!( - config["unmapped"]["validation_error"], - "conflicting tls metadata" - ); - assert!( - config["message"] - .as_str() - .unwrap() - .contains("previous policy IS NOT active") - ); - assert!( - config["message"] - .as_str() - .unwrap() - .contains("error:conflicting tls metadata") - ); - - let finding = finding.to_json().unwrap(); - assert_eq!(finding["class_uid"], 2004); - assert_eq!(finding["action"], "Denied"); - assert_eq!(finding["disposition"], "Blocked"); - - let retained = PolicyValidationFailureDisposition { - configured_mode: PolicyValidationFailureMode::RetainLastValid, - mode: PolicyValidationFailureMode::RetainLastValid, - previous_policy_active: true, - active_generation: 4, - }; - let [config, _] = policy_validation_failure_events( - &retained, - 8, - "sha256:test", - "conflicting tls metadata", - ); - let config = config.to_json().unwrap(); - assert_eq!(config["unmapped"]["previous_policy_active"], true); - assert!( - config["message"] - .as_str() - .unwrap() - .contains("previous policy IS active") - ); - } +/// Returns an error when the protected bootstrap is invalid or the boundary +/// listener cannot be established. +pub fn run( + config_path: &std::path::Path, + qualification: RuntimeQualification, +) -> miette::Result<()> { + boundary_server::run_boundary(config_path, qualification) + .map_err(|error| miette::miette!(error)) } diff --git a/crates/openshell-sandbox/src/main.rs b/crates/openshell-sandbox/src/main.rs index 440676ad14..24fa65ab16 100644 --- a/crates/openshell-sandbox/src/main.rs +++ b/crates/openshell-sandbox/src/main.rs @@ -1,282 +1,1561 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! `OpenShell` Sandbox - process sandbox and monitor. +//! `OpenShell` capability-free in-workload sandbox boundary. +#[cfg(target_os = "linux")] +use std::mem::size_of; use std::path::Path; -use std::sync::Arc; -use std::sync::atomic::AtomicBool; use clap::Parser; use miette::{IntoDiagnostic, Result}; -use openshell_ocsf::{OcsfJsonlLayer, OcsfShorthandLayer}; -use tracing::{error, info, warn}; +#[cfg(target_os = "linux")] +use openshell_ocsf::OcsfShorthandLayer; +#[cfg(target_os = "linux")] use tracing_subscriber::EnvFilter; -use tracing_subscriber::filter::LevelFilter; +#[cfg(target_os = "linux")] use tracing_subscriber::{Layer, layer::SubscriberExt, util::SubscriberInitExt}; -use openshell_sandbox::run_sandbox; - -/// Subcommand name used to self-copy the supervisor binary into a shared volume. +/// Subcommand name used to self-copy the sandbox binary into a shared volume. /// /// Init containers invoke the binary directly instead of relying on `sh`/`cp` /// to copy the binary out. Invoking the binary itself with this argument /// performs the copy in pure Rust. const COPY_SELF_SUBCOMMAND: &str = "copy-self"; +const BOOTSTRAP_SUBCOMMAND: &str = "bootstrap"; +const SEED_WORKSPACE_SUBCOMMAND: &str = "seed-workspace"; +#[cfg(any(target_os = "linux", test))] +const KUBERNETES_BOOTSTRAP_SECRET_FILES: [&str; 3] = ["boundary.json", "tls.crt", "tls.key"]; +#[cfg(target_os = "linux")] +const BOOTSTRAP_INPUT_ROOT: &str = "/.openshell/bootstrap-input"; +#[cfg(target_os = "linux")] +const SANDBOX_RUNTIME_ROOT: &str = "/.openshell/runtime"; +#[cfg(target_os = "linux")] +const SANDBOX_STATE_ROOT: &str = "/.openshell/state"; -/// Subcommand for one-shot debug RPCs from inside a sandbox container. -/// -/// Reads the same token sources as the supervisor (env, file, K8s SA -/// bootstrap) and issues a single gRPC call against the gateway. Useful -/// for end-to-end verification: e.g. `docker exec` into a sandbox, then -/// run `openshell-sandbox debug-rpc get-sandbox-config --sandbox-id ` -/// to confirm the cross-sandbox IDOR guard fires. -const DEBUG_RPC_SUBCOMMAND: &str = "debug-rpc"; const VALIDATE_WORKSPACE_SUBCOMMAND: &str = "validate-workspace"; - -/// Default `--mode` value: run both supervisor leaves in a single binary. -const DEFAULT_MODE: &str = "network,process"; -const SIDECAR_STATE_DIR: &str = openshell_core::container_paths::SIDECAR_RUN_ROOT; -const SIDECAR_TLS_DIR: &str = openshell_core::container_paths::SIDECAR_TLS_DIR; +const CAPABILITY_PROBE_SUBCOMMAND: &str = "capability-probe"; +const CAPABILITY_PROBE_LAUNCH_SUBCOMMAND: &str = "capability-probe-launch"; +const CAPABILITY_SOCKET_CHILD_SUBCOMMAND: &str = "capability-socket-child"; +const CAPABILITY_LANDLOCK_CHILD_SUBCOMMAND: &str = "capability-landlock-child"; +const CAPABILITY_FREE_LAUNCH_SUBCOMMAND: &str = "launch-capability-free"; #[cfg(target_os = "linux")] -const CLIENT_TLS_DIR: &str = openshell_core::container_paths::CLIENT_TLS_DIR; +const PROBE_DENIED_TCP_PEER: &str = "203.0.113.1:9"; #[cfg(target_os = "linux")] -const SIDECAR_CLIENT_TLS_SUBDIR: &str = "client"; +const PROBE_SOCKADDR_IN_LEN: usize = size_of::(); #[cfg(target_os = "linux")] -const CLIENT_TLS_FILES: [&str; 3] = ["ca.crt", "tls.crt", "tls.key"]; +const LINUX_SIGNAL_LIMIT: i32 = 65; + +#[derive(Parser, Debug)] +#[command(name = "openshell-sandbox")] +#[command(version = openshell_core::VERSION)] +#[command(about = "OpenShell in-workload isolation boundary")] +struct BoundaryArgs { + /// Protected one-use bootstrap configuration staged by the driver. + #[arg(long)] + bootstrap: std::path::PathBuf, + + /// Log level (trace, debug, info, warn, error). + #[arg(long, default_value = "warn", env = openshell_core::sandbox_env::LOG_LEVEL)] + log_level: String, +} + +/// Internal one-shot command used by trusted driver bootstrap to validate an +/// image-provided workdir as the final sandbox identity. +#[derive(Parser, Debug)] +#[command(name = "validate-workspace", hide = true)] +struct ValidateWorkspaceArgs { + #[arg(long)] + workdir: String, + #[arg(long)] + expected_uid: u32, + #[arg(long)] + expected_gid: u32, +} + #[cfg(target_os = "linux")] -const SIDECAR_STATE_DIR_MODE: u32 = 0o2775; +fn validate_workspace(args: &[String]) -> Result<()> { + let args = ValidateWorkspaceArgs::try_parse_from( + std::iter::once(VALIDATE_WORKSPACE_SUBCOMMAND.to_string()).chain(args.iter().cloned()), + ) + .into_diagnostic()?; + let actual = ( + nix::unistd::geteuid().as_raw(), + nix::unistd::getegid().as_raw(), + ); + if actual != (args.expected_uid, args.expected_gid) { + return Err(miette::miette!( + "workspace validator privilege drop failed: expected {}:{}, got {}:{}", + args.expected_uid, + args.expected_gid, + actual.0, + actual.1 + )); + } + openshell_sandbox::process::validate_oci_workspace_as_effective_identity(Path::new( + &args.workdir, + )) +} + +#[cfg(not(target_os = "linux"))] +fn validate_workspace(_args: &[String]) -> Result<()> { + Err(miette::miette!( + "workspace validation is only supported on Unix" + )) +} + +/// Run the active Phase 0 probe inside the exact workload runtime profile. #[cfg(target_os = "linux")] -const SIDECAR_TLS_DIR_MODE: u32 = 0o755; +#[allow(unsafe_code)] +fn run_capability_probe() -> Result<()> { + let (qualification, report) = qualify_runtime()?; + debug_assert!(qualification.seccomp.notification_round_trip); + println!("{}", serde_json::to_string(&report).into_diagnostic()?); + Ok(()) +} + +/// Actively qualify every kernel primitive used by the capability-free +/// sandbox. Callers decide whether to emit the resulting diagnostic report. #[cfg(target_os = "linux")] -const SIDECAR_TLS_STAGING_DIR_MODE: u32 = 0o700; +#[derive(serde::Serialize)] +#[allow( + clippy::struct_excessive_bools, + reason = "diagnostic report records independent active probes" +)] +struct QualificationReport { + qualified: bool, + uid: u32, + gid: u32, + supplementary_groups: Vec, + capabilities_zero: bool, + no_new_privileges: bool, + sandbox_dumpable: bool, + child_dumpable: bool, + child_core_limit_zero: bool, + same_uid_self_protection: bool, + landlock_abi: u32, + landlock_allow_deny: bool, + seccomp_notification: bool, + seccomp_addfd_send: bool, + task_memory_copy: bool, + connected_send_fast_path: bool, + socket_virtualization: bool, + dns_relay_bind: bool, + udp_dns_round_trip: bool, + tcp_dns_round_trip: bool, + tcp_allow_round_trip: bool, + tcp_deny_round_trip: bool, + wait_killable_recv: bool, +} + #[cfg(target_os = "linux")] -const SIDECAR_CLIENT_TLS_DIR_MODE: u32 = 0o750; +#[allow(unsafe_code)] +fn qualify_runtime() -> Result<(openshell_sandbox::RuntimeQualification, QualificationReport)> { + use miette::Context as _; + + let uid = nix::unistd::geteuid().as_raw(); + let gid = nix::unistd::getegid().as_raw(); + if uid == 0 || gid == 0 { + return Err(miette::miette!( + "capability-free sandbox probe requires non-root UID and GID, got {uid}:{gid}" + )); + } + let status = std::fs::read_to_string("/proc/self/status") + .into_diagnostic() + .wrap_err("read /proc/self/status")?; + for field in ["CapInh", "CapPrm", "CapEff", "CapBnd", "CapAmb"] { + let value = proc_status_hex(&status, field)?; + if value != 0 { + return Err(miette::miette!( + "capability-free sandbox probe found {field}=0x{value:x}" + )); + } + } + // SAFETY: PR_GET_NO_NEW_PRIVS reads one scalar process property. + let no_new_privileges = unsafe { libc::prctl(libc::PR_GET_NO_NEW_PRIVS, 0, 0, 0, 0) }; + if no_new_privileges != 1 { + return Err(miette::miette!( + "capability-free sandbox probe requires no_new_privs=1" + )); + } + + // The trusted sandbox must be nondumpable before it handles bootstrap or + // channel secrets. Perform the parent-to-child observation probe after + // tightening the parent; the synthetic child explicitly becomes dumpable. + // SAFETY: PR_SET_DUMPABLE accepts one scalar and only tightens this process. + if unsafe { libc::prctl(libc::PR_SET_DUMPABLE, 0, 0, 0, 0) } < 0 { + return Err(miette::miette!( + "set sandbox probe nondumpable: {}", + std::io::Error::last_os_error() + )); + } + openshell_isolation_interface::linux::task_memory::probe_child_access() + .into_diagnostic() + .wrap_err("same-UID task-memory probe")?; + probe_landlock_allow_deny().wrap_err("Landlock allow/deny probe")?; + let notification = + openshell_isolation_interface::linux::seccomp_notify::probe_notification_api() + .into_diagnostic() + .wrap_err("seccomp notification probe")?; + probe_socket_virtualization().wrap_err("socket virtualization probe")?; + probe_dns_relay_bind().wrap_err("DNS relay bind probe")?; + let landlock_abi = openshell_isolation_interface::linux::landlock::abi_version() + .into_diagnostic() + .wrap_err("Landlock ABI probe")?; + if landlock_abi < 3 { + return Err(miette::miette!( + "sandbox self-protection requires Landlock ABI v3 or newer (including truncation), found v{landlock_abi}" + )); + } + + let groups = nix::unistd::getgroups() + .into_diagnostic()? + .into_iter() + .map(nix::unistd::Gid::as_raw) + .collect::>(); + let report = QualificationReport { + qualified: true, + uid, + gid, + supplementary_groups: groups, + capabilities_zero: true, + no_new_privileges: true, + sandbox_dumpable: false, + child_dumpable: true, + child_core_limit_zero: true, + same_uid_self_protection: true, + landlock_abi, + landlock_allow_deny: true, + seccomp_notification: notification.notification_round_trip(), + seccomp_addfd_send: notification.addfd_send(), + task_memory_copy: notification.task_memory_copy(), + connected_send_fast_path: notification.connected_send_fast_path(), + socket_virtualization: true, + dns_relay_bind: true, + udp_dns_round_trip: true, + tcp_dns_round_trip: true, + tcp_allow_round_trip: true, + tcp_deny_round_trip: true, + wait_killable_recv: notification.wait_killable_recv, + }; + let qualification = openshell_sandbox::RuntimeQualification { + seccomp: openshell_isolation_interface::contract::SeccompEvidence { + new_listener: notification.notification_round_trip(), + notification_round_trip: notification.notification_round_trip(), + id_validation: notification.notification_round_trip(), + addfd_send: notification.addfd_send(), + retained_socket_operation: true, + proc_fd_identity: true, + task_memory_read: notification.task_memory_copy(), + task_memory_write: notification.task_memory_copy(), + cancellation: notification.wait_killable_recv, + }, + landlock_abi, + landlock_allow_deny: true, + udp_dns_round_trip: true, + tcp_dns_round_trip: true, + tcp_allow_round_trip: true, + tcp_deny_round_trip: true, + }; + Ok((qualification, report)) +} + +#[cfg(target_os = "linux")] +fn probe_dns_relay_bind() -> Result<()> { + use std::net::{TcpListener, UdpSocket}; + + let unprivileged_port_start = + std::fs::read_to_string("/proc/sys/net/ipv4/ip_unprivileged_port_start") + .into_diagnostic()? + .trim() + .parse::() + .into_diagnostic()?; + if unprivileged_port_start != 0 { + return Err(miette::miette!( + "DNS relay requires net.ipv4.ip_unprivileged_port_start=0, got {unprivileged_port_start}" + )); + } + let tcp = TcpListener::bind("127.0.0.53:53").into_diagnostic()?; + let udp = UdpSocket::bind("127.0.0.53:53").into_diagnostic()?; + drop((tcp, udp)); + Ok(()) +} + +/// Prove that the exact unprivileged runtime can install a hard Landlock +/// allow-list which admits one path and rejects an adjacent path. Landlock is +/// irreversible, so the restriction is exercised in a fresh trusted child. #[cfg(target_os = "linux")] -const SIDECAR_CLIENT_TLS_FILE_MODE: u32 = 0o400; +fn probe_landlock_allow_deny() -> Result<()> { + use miette::Context as _; + + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .into_diagnostic()? + .as_nanos(); + let root = std::env::temp_dir().join(format!( + "openshell-landlock-probe-{}-{nonce}", + std::process::id() + )); + let allowed = root.join("allowed"); + let denied = root.join("denied"); + std::fs::create_dir(&root) + .into_diagnostic() + .wrap_err("create Landlock probe root")?; + let probe_result = (|| -> Result<()> { + std::fs::create_dir(&allowed).into_diagnostic()?; + std::fs::create_dir(&denied).into_diagnostic()?; + std::fs::write(allowed.join("sentinel"), b"allowed").into_diagnostic()?; + std::fs::write(denied.join("sentinel"), b"denied").into_diagnostic()?; + let status = std::process::Command::new(std::env::current_exe().into_diagnostic()?) + .arg(CAPABILITY_LANDLOCK_CHILD_SUBCOMMAND) + .arg(&allowed) + .arg(&denied) + .env_clear() + .stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::inherit()) + .status() + .into_diagnostic() + .wrap_err("run Landlock probe child")?; + if !status.success() { + return Err(miette::miette!( + "Landlock probe child exited with status {status}" + )); + } + Ok(()) + })(); + let cleanup_result = std::fs::remove_dir_all(&root).into_diagnostic(); + probe_result?; + cleanup_result.wrap_err("remove Landlock probe root") +} + +#[cfg(target_os = "linux")] +fn run_capability_landlock_child(args: &[String]) -> Result<()> { + use openshell_core::policy::{ + FilesystemPolicy, LandlockCompatibility, LandlockPolicy, NetworkPolicy, ProcessPolicy, + SandboxPolicy, + }; -/// Which supervisor leaves are enabled in this process. + let [allowed, denied] = args else { + return Err(miette::miette!( + "usage: {CAPABILITY_LANDLOCK_CHILD_SUBCOMMAND} " + )); + }; + let allowed = Path::new(allowed); + let denied = Path::new(denied); + let policy = SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy { + read_only: vec![allowed.to_path_buf()], + read_write: Vec::new(), + include_workdir: false, + }, + network: NetworkPolicy::default(), + landlock: LandlockPolicy { + compatibility: LandlockCompatibility::HardRequirement, + }, + process: ProcessPolicy::default(), + }; + let prepared = openshell_sandbox::sandbox::linux::prepare_capability_free(&policy, None)?; + openshell_sandbox::sandbox::linux::enforce(prepared)?; + if std::fs::read(allowed.join("sentinel")).into_diagnostic()? != b"allowed" { + return Err(miette::miette!("Landlock probe allowed-path mismatch")); + } + match std::fs::read(denied.join("sentinel")) { + Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied => Ok(()), + Err(error) => Err(error).into_diagnostic(), + Ok(_) => Err(miette::miette!( + "Landlock probe unexpectedly read the denied path" + )), + } +} + +#[cfg(not(target_os = "linux"))] +fn run_capability_landlock_child(_args: &[String]) -> Result<()> { + Err(miette::miette!( + "Landlock qualification is supported only on Linux" + )) +} + +/// Exercise the production listener inheritance and socket-time ADDFD shape. /// -/// Parsed from a comma-separated `--mode` value, e.g. `network`, -/// `process`, or `network,process`. `network-init` is a one-shot setup mode -/// used by the Kubernetes sidecar topology and cannot be combined with other -/// mode components. At least one must be set. -#[derive(Clone, Copy, Debug)] -struct Mode { - network: bool, - process: bool, - network_init: bool, -} - -impl std::str::FromStr for Mode { - type Err = String; - - fn from_str(s: &str) -> Result { - let mut mode = Self { - network: false, - process: false, - network_init: false, - }; - for part in s.split(',').map(str::trim).filter(|p| !p.is_empty()) { - match part { - "network" => mode.network = true, - "process" => mode.process = true, - "network-init" => mode.network_init = true, - other => { - return Err(format!( - "unknown mode component '{other}' (expected 'network', 'process', or 'network-init')" - )); +/// One dedicated launcher thread installs the non-TSYNC listener, moves the +/// listener FD to this unfiltered broker through an in-process channel, then +/// execs the child. The child proves that the injected open-file description +/// survives dup and epoll registration before connect and that the broker can +/// return the original peer rather than the local relay endpoint. +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn probe_socket_virtualization() -> Result<()> { + use std::io::{Read as _, Write as _}; + use std::net::{Ipv4Addr, SocketAddr, TcpListener, UdpSocket}; + use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; + use std::os::unix::process::CommandExt as _; + use std::sync::mpsc; + + use miette::Context as _; + use openshell_isolation_interface::linux::seccomp_notify::NotificationListener; + use openshell_isolation_interface::linux::socket_registry::{ + InetFamily, InetKind, SocketMetadata, SocketRegistry, SocketState, + }; + + let relay = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)) + .into_diagnostic() + .wrap_err("bind socket probe relay")?; + let original_peer = relay.local_addr().into_diagnostic()?; + let relay_thread = std::thread::Builder::new() + .name("openshell-probe-relay".to_string()) + .spawn(move || -> std::io::Result<()> { + let (mut stream, _) = relay.accept()?; + stream.set_nodelay(true)?; + let mut request = [0_u8; 4]; + stream.read_exact(&mut request)?; + if &request != b"ping" { + return Err(std::io::Error::other("socket probe payload mismatch")); + } + stream.write_all(b"pong") + }) + .into_diagnostic()?; + let dns_relay_addr = "127.0.0.53:53" + .parse::() + .expect("fixed DNS relay address is valid"); + let dns_relay = UdpSocket::bind(dns_relay_addr) + .into_diagnostic() + .wrap_err("bind socket probe DNS relay")?; + let dns_thread = std::thread::Builder::new() + .name("openshell-probe-dns".to_string()) + .spawn(move || -> std::io::Result<()> { + let mut query = [0_u8; 512]; + let (length, peer) = dns_relay.recv_from(&mut query)?; + let response = build_probe_dns_response(&query[..length])?; + let sent = dns_relay.send_to(&response, peer)?; + if sent != response.len() { + return Err(std::io::Error::new( + std::io::ErrorKind::WriteZero, + "short DNS probe response", + )); + } + Ok(()) + }) + .into_diagnostic()?; + let dns_tcp_relay = TcpListener::bind(dns_relay_addr) + .into_diagnostic() + .wrap_err("bind socket probe TCP DNS relay")?; + let dns_tcp_thread = std::thread::Builder::new() + .name("openshell-probe-dns-tcp".to_string()) + .spawn(move || -> std::io::Result<()> { + let (mut stream, _) = dns_tcp_relay.accept()?; + let mut length = [0_u8; 2]; + stream.read_exact(&mut length)?; + let length = usize::from(u16::from_be_bytes(length)); + if length == 0 || length > 512 { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidData, + "invalid TCP DNS probe length", + )); + } + let mut query = vec![0_u8; length]; + stream.read_exact(&mut query)?; + let response = build_probe_dns_response(&query)?; + stream.write_all( + &u16::try_from(response.len()) + .expect("probe DNS response fits u16") + .to_be_bytes(), + )?; + stream.write_all(&response) + }) + .into_diagnostic()?; + + let executable = std::env::current_exe() + .into_diagnostic() + .wrap_err("resolve socket probe executable")?; + let sandbox_tgid = std::process::id(); + let mut child_hardening = + openshell_isolation_interface::linux::child_seccomp::prepare(sandbox_tgid) + .into_diagnostic() + .wrap_err("prepare socket probe child hardening")?; + let (listener_tx, listener_rx) = mpsc::sync_channel::>(1); + let (child_tx, child_rx) = mpsc::sync_channel::>(1); + let launcher = std::thread::Builder::new() + .name("openshell-probe-launcher".to_string()) + .spawn(move || { + if let Err(error) = block_launcher_signals() { + let _ = listener_tx.send(Err(error)); + return; + } + let listener = + openshell_isolation_interface::linux::seccomp_notify::install_listener(&[ + libc::SYS_socket, + libc::SYS_connect, + libc::SYS_getpeername, + libc::SYS_sendto, + ]); + let Ok(listener) = listener else { + let _ = listener_tx.send(listener); + return; + }; + if listener_tx.send(Ok(listener)).is_err() { + return; + } + let mut command = std::process::Command::new(executable); + command + .arg(CAPABILITY_SOCKET_CHILD_SUBCOMMAND) + .arg(original_peer.to_string()) + .arg(sandbox_tgid.to_string()) + .env_clear() + .stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::inherit()); + // SAFETY: the hook uses only raw signal/process syscalls and the + // prebuilt, allocation-free seccomp installation path. + unsafe { + command.pre_exec(move || { + if libc::setpgid(0, 0) < 0 { + return Err(std::io::Error::last_os_error()); + } + set_child_core_limit()?; + reset_child_signal_dispositions()?; + child_hardening.install()?; + install_child_signal_mask() + }); + } + let child = command.spawn(); + let _ = child_tx.send(child); + }) + .into_diagnostic()?; + + let listener = listener_rx + .recv() + .into_diagnostic() + .wrap_err("socket probe launcher stopped before listener handoff")? + .into_diagnostic() + .wrap_err("install socket probe listener")?; + let mut child = child_rx + .recv() + .into_diagnostic() + .wrap_err("socket probe launcher stopped before child spawn")? + .into_diagnostic() + .wrap_err("spawn socket probe child")?; + let mut registry = SocketRegistry::new(1, 8).into_diagnostic()?; + let mut observed_tcp_sockets = 0_u8; + let mut observed_dns_socket = false; + let mut observed_connect = false; + let mut observed_dns_tcp_connect = false; + let mut observed_denied_connect = false; + let mut observed_peer = false; + let mut observed_dns_send = false; + + while !(observed_tcp_sockets == 3 + && observed_dns_socket + && observed_connect + && observed_dns_tcp_connect + && observed_denied_connect + && observed_peer + && observed_dns_send) + { + let notification = listener + .receive() + .into_diagnostic() + .wrap_err("receive socket probe notification")?; + match i64::from(notification.syscall) { + libc::SYS_socket => { + if notification.args[0] != u64::try_from(libc::AF_INET).unwrap() { + listener + .respond_errno(notification.id, libc::EPROTONOSUPPORT) + .into_diagnostic()?; + return Err(miette::miette!("unexpected socket probe request")); } + let requested_type = i32::try_from(notification.args[1]) + .map_err(|_| miette::miette!("socket type does not fit i32"))?; + let base_type = requested_type & !(libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK); + let protocol = i32::try_from(notification.args[2]) + .map_err(|_| miette::miette!("socket protocol does not fit i32"))?; + let (kind, canonical_protocol) = match (base_type, protocol) { + (libc::SOCK_STREAM, 0 | libc::IPPROTO_TCP) if observed_tcp_sockets < 3 => { + observed_tcp_sockets += 1; + (InetKind::Tcp, libc::IPPROTO_TCP) + } + (libc::SOCK_DGRAM, 0 | libc::IPPROTO_UDP) if !observed_dns_socket => { + observed_dns_socket = true; + (InetKind::DnsUdp, libc::IPPROTO_UDP) + } + _ => { + listener + .respond_errno(notification.id, libc::EPROTONOSUPPORT) + .into_diagnostic()?; + return Err(miette::miette!("unexpected socket probe request")); + } + }; + // SAFETY: scalar validated AF_INET/TCP arguments return one + // newly owned descriptor on success. + let source = + unsafe { libc::socket(libc::AF_INET, requested_type, canonical_protocol) }; + if source < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + // SAFETY: successful socket returned one newly owned FD. + let source = unsafe { OwnedFd::from_raw_fd(source) }; + let close_on_exec = requested_type & libc::SOCK_CLOEXEC != 0; + let tentative = registry + .stage( + source, + SocketMetadata { + family: InetFamily::V4, + kind, + close_on_exec, + nonblocking: requested_type & libc::SOCK_NONBLOCK != 0, + creator_generation: 1, + }, + ) + .into_diagnostic()?; + listener + .add_fd_and_send(notification.id, tentative.source_fd(), close_on_exec) + .into_diagnostic()?; + registry.commit(tentative).into_diagnostic()?; + } + libc::SYS_connect => { + let fd = i32::try_from(notification.args[0]) + .map_err(|_| miette::miette!("connect FD does not fit i32"))?; + let destination = read_probe_sockaddr( + notification.tid, + notification.args[1], + notification.args[2], + )?; + let denied_peer = PROBE_DENIED_TCP_PEER + .parse::() + .expect("fixed denied peer is valid"); + if destination == denied_peer && !observed_denied_connect { + listener + .respond_errno(notification.id, libc::EACCES) + .into_diagnostic()?; + observed_denied_connect = true; + continue; + } + if destination != original_peer && destination != dns_relay_addr { + listener + .respond_errno(notification.id, libc::EACCES) + .into_diagnostic()?; + return Err(miette::miette!("unexpected socket probe destination")); + } + if (destination == original_peer && observed_connect) + || (destination == dns_relay_addr && observed_dns_tcp_connect) + { + listener + .respond_errno(notification.id, libc::EALREADY) + .into_diagnostic()?; + return Err(miette::miette!("duplicate socket probe connect")); + } + let entry = registry + .resolve_mut(notification.tid, fd) + .into_diagnostic()?; + entry.validate_retained_identity().into_diagnostic()?; + let (sockaddr, length) = encode_probe_sockaddr(destination)?; + // SAFETY: the retained FD is the registered injected socket; + // `sockaddr` is live for the declared IPv4 length. + let connected = unsafe { + libc::connect( + entry.retained_preconnect().into_diagnostic()?.as_raw_fd(), + sockaddr.as_ptr().cast(), + length, + ) + }; + if connected != 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + if destination == dns_relay_addr { + entry.set_state(SocketState::DnsTcp { + relay: dns_relay_addr, + }); + observed_dns_tcp_connect = true; + } else { + entry.set_state(SocketState::Connected { original_peer }); + observed_connect = true; + } + entry.release_preconnect(); + listener + .respond_value(notification.id, 0) + .into_diagnostic()?; + } + libc::SYS_getpeername => { + let fd = i32::try_from(notification.args[0]) + .map_err(|_| miette::miette!("peer FD does not fit i32"))?; + let entry = registry.resolve(notification.tid, fd).into_diagnostic()?; + let SocketState::Connected { original_peer } = entry.state() else { + listener + .respond_errno(notification.id, libc::ENOTCONN) + .into_diagnostic()?; + return Err(miette::miette!("peer query preceded mediated connect")); + }; + write_probe_sockaddr( + notification.tid, + notification.args[1], + notification.args[2], + *original_peer, + )?; + listener + .respond_value(notification.id, 0) + .into_diagnostic()?; + observed_peer = true; + } + libc::SYS_sendto => { + let fd = i32::try_from(notification.args[0]) + .map_err(|_| miette::miette!("sendto FD does not fit i32"))?; + let length = usize::try_from(notification.args[2]) + .map_err(|_| miette::miette!("DNS payload length does not fit usize"))?; + if length == 0 || length > 512 || notification.args[3] != 0 { + listener + .respond_errno(notification.id, libc::EMSGSIZE) + .into_diagnostic()?; + return Err(miette::miette!("unexpected DNS probe payload shape")); + } + let destination = read_probe_sockaddr( + notification.tid, + notification.args[4], + notification.args[5], + )?; + if observed_dns_send || destination != dns_relay_addr { + listener + .respond_errno(notification.id, libc::EACCES) + .into_diagnostic()?; + return Err(miette::miette!("unexpected DNS probe destination")); + } + let mut payload = vec![0_u8; length]; + openshell_isolation_interface::linux::task_memory::read_exact( + notification.tid, + notification.args[1], + &mut payload, + ) + .into_diagnostic()?; + validate_probe_dns_query(&payload)?; + let entry = registry + .resolve_mut(notification.tid, fd) + .into_diagnostic()?; + if entry.metadata().kind != InetKind::DnsUdp + || !matches!(entry.state(), SocketState::Created) + { + listener + .respond_errno(notification.id, libc::EACCES) + .into_diagnostic()?; + return Err(miette::miette!("DNS probe socket is not eligible")); + } + let (sockaddr, sockaddr_length) = encode_probe_sockaddr(destination)?; + let retained = entry.retained_preconnect().into_diagnostic()?; + // SAFETY: the retained source is the exact injected OFD and + // both copied buffers remain live for their declared lengths. + if unsafe { + libc::connect( + retained.as_raw_fd(), + sockaddr.as_ptr().cast(), + sockaddr_length, + ) + } != 0 + { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + let sent = unsafe { + libc::send( + retained.as_raw_fd(), + payload.as_ptr().cast(), + payload.len(), + libc::MSG_NOSIGNAL, + ) + }; + if sent != isize::try_from(payload.len()).expect("DNS payload fits isize") { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + entry.set_state(SocketState::DnsUdp { + relay: dns_relay_addr, + }); + entry.release_preconnect(); + listener + .respond_value( + notification.id, + i64::try_from(length).expect("length fits i64"), + ) + .into_diagnostic()?; + observed_dns_send = true; + } + _ => { + listener + .respond_errno(notification.id, libc::EPERM) + .into_diagnostic()?; + return Err(miette::miette!("unexpected socket probe syscall")); } } - if mode.network_init && (mode.network || mode.process) { - return Err("--mode=network-init cannot be combined with other components".into()); + } + + let status = child + .wait() + .into_diagnostic() + .wrap_err("wait for socket probe child")?; + launcher + .join() + .map_err(|_| miette::miette!("socket probe launcher panicked"))?; + relay_thread + .join() + .map_err(|_| miette::miette!("socket probe relay panicked"))? + .into_diagnostic()?; + dns_thread + .join() + .map_err(|_| miette::miette!("socket probe DNS relay panicked"))? + .into_diagnostic()?; + dns_tcp_thread + .join() + .map_err(|_| miette::miette!("socket probe TCP DNS relay panicked"))? + .into_diagnostic()?; + if !status.success() { + return Err(miette::miette!( + "socket probe child exited with status {status}" + )); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn validate_probe_dns_query(query: &[u8]) -> Result<()> { + const EXPECTED_QUESTION: &[u8] = b"\x05probe\x09openshell\x04test\x00\x00\x01\x00\x01"; + if query.len() != 12 + EXPECTED_QUESTION.len() + || query[2] & 0x80 != 0 + || query[4..6] != [0, 1] + || &query[12..] != EXPECTED_QUESTION + { + return Err(miette::miette!("DNS probe query is malformed")); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn build_probe_dns_response(query: &[u8]) -> std::io::Result> { + validate_probe_dns_query(query).map_err(std::io::Error::other)?; + let mut response = query.to_vec(); + response[2..4].copy_from_slice(&[0x81, 0x80]); + response[6..8].copy_from_slice(&[0, 1]); + response.extend_from_slice(&[0xc0, 0x0c, 0, 1, 0, 1, 0, 0, 0, 30, 0, 4, 203, 0, 113, 7]); + Ok(response) +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn block_launcher_signals() -> std::io::Result<()> { + let mut signals = std::mem::MaybeUninit::::uninit(); + // SAFETY: `signals` points to writable sigset storage and pthread_sigmask + // copies it during the call. + if unsafe { libc::sigfillset(signals.as_mut_ptr()) } < 0 { + return Err(std::io::Error::last_os_error()); + } + // SAFETY: sigfillset initialized the value above. + let signals = unsafe { signals.assume_init() }; + // SAFETY: changing the mask affects only the dedicated launcher thread. + let result = + unsafe { libc::pthread_sigmask(libc::SIG_BLOCK, &raw const signals, std::ptr::null_mut()) }; + if result != 0 { + return Err(std::io::Error::from_raw_os_error(result)); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +unsafe fn set_child_core_limit() -> std::io::Result<()> { + let limit = libc::rlimit { + rlim_cur: 0, + rlim_max: 0, + }; + // SAFETY: `limit` is a live fixed-size rlimit and this child-only update + // permanently disables core dumps before any untrusted instruction. + if unsafe { libc::setrlimit(libc::RLIMIT_CORE, &raw const limit) } < 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +unsafe fn reset_child_signal_dispositions() -> std::io::Result<()> { + // SAFETY: an all-zero sigaction is a valid base before the explicit + // default handler and empty mask are installed below. + let mut action = unsafe { std::mem::zeroed::() }; + action.sa_sigaction = libc::SIG_DFL; + // SAFETY: action.sa_mask points to writable sigset storage. + if unsafe { libc::sigemptyset(&raw mut action.sa_mask) } < 0 { + return Err(std::io::Error::last_os_error()); + } + for signal in 1..LINUX_SIGNAL_LIMIT { + if signal == libc::SIGKILL || signal == libc::SIGSTOP { + continue; } - if !mode.network && !mode.process && !mode.network_init { - return Err( - "--mode must enable at least one of: network, process, network-init".into(), - ); + // SAFETY: action contains the default disposition and the null output + // pointer requests no previous action. + if unsafe { libc::sigaction(signal, &raw const action, std::ptr::null_mut()) } < 0 { + let error = std::io::Error::last_os_error(); + // glibc reserves two real-time signals for its threading runtime; + // Linux rejects sigaction for those numbers with EINVAL. + if error.raw_os_error() != Some(libc::EINVAL) { + return Err(error); + } } - Ok(mode) } + Ok(()) } -/// `OpenShell` Sandbox - process isolation and monitoring. -// CLI flags are naturally boolean switches; grouping them into structs would -// only obscure the clap definition. -#[allow(clippy::struct_excessive_bools)] -#[derive(Parser, Debug)] -#[command(name = "openshell-sandbox")] -#[command(version = openshell_core::VERSION)] -#[command(about = "Process sandbox and monitor", long_about = None)] -struct Args { - /// Command to execute in the sandbox. - /// Defaults to a login shell if neither this nor the driver specification is - /// provided: `/bin/bash -l` when available, otherwise a shell detected in the - /// sandbox image (e.g. `/bin/sh` on Alpine). - #[arg(trailing_var_arg = true)] - command: Vec, - - /// Working directory for the sandboxed process. - #[arg(long, short)] - workdir: Option, - - /// Timeout in seconds (0 = no timeout). - #[arg(long, short, default_value = "0")] - timeout: u64, - - /// Run in interactive mode (inherit process group for terminal control). - #[arg(long, short = 'i')] - interactive: bool, - - /// Sandbox ID for fetching policy via gRPC from `OpenShell` server. - /// Requires --openshell-endpoint to be set. - #[arg(long, env = openshell_core::sandbox_env::SANDBOX_ID)] - sandbox_id: Option, - - /// Sandbox (used for policy sync when the sandbox discovers policy - /// from disk or falls back to the restrictive default). - #[arg(long, env = openshell_core::sandbox_env::SANDBOX)] - sandbox: Option, - - /// `OpenShell` server gRPC endpoint for fetching policy. - /// Required when using --sandbox-id. - #[arg(long, env = openshell_core::sandbox_env::ENDPOINT)] - openshell_endpoint: Option, - - /// Path to Rego policy file for OPA-based network access control. - /// Requires --policy-data to also be set. - #[arg(long, env = "OPENSHELL_POLICY_RULES")] - policy_rules: Option, - - /// Path to YAML data file containing network policies and sandbox config. - /// Requires --policy-rules to also be set. - #[arg(long, env = "OPENSHELL_POLICY_DATA")] - policy_data: Option, +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn install_child_signal_mask() -> std::io::Result<()> { + let mut signals = std::mem::MaybeUninit::::uninit(); + // SAFETY: `signals` points to writable sigset storage. + if unsafe { libc::sigemptyset(signals.as_mut_ptr()) } < 0 { + return Err(std::io::Error::last_os_error()); + } + // SAFETY: sigemptyset initialized the value above. + let signals = unsafe { signals.assume_init() }; + // SAFETY: this installs the declared empty target mask immediately before + // exec, after copied sandbox dispositions have been reset. + let result = unsafe { + libc::pthread_sigmask(libc::SIG_SETMASK, &raw const signals, std::ptr::null_mut()) + }; + if result != 0 { + return Err(std::io::Error::from_raw_os_error(result)); + } + Ok(()) +} - /// Log level (trace, debug, info, warn, error). - #[arg(long, default_value = "warn", env = openshell_core::sandbox_env::LOG_LEVEL)] - log_level: String, +#[cfg(target_os = "linux")] +fn read_probe_sockaddr(tid: u32, address: u64, length: u64) -> Result { + let length = + usize::try_from(length).map_err(|_| miette::miette!("sockaddr length too large"))?; + if length != PROBE_SOCKADDR_IN_LEN { + return Err(miette::miette!("socket probe requires an IPv4 sockaddr")); + } + let mut bytes = vec![0_u8; length]; + openshell_isolation_interface::linux::task_memory::read_exact(tid, address, &mut bytes) + .into_diagnostic()?; + decode_probe_sockaddr(&bytes) +} + +#[cfg(target_os = "linux")] +fn encode_probe_sockaddr( + address: std::net::SocketAddr, +) -> Result<([u8; PROBE_SOCKADDR_IN_LEN], libc::socklen_t)> { + let std::net::SocketAddr::V4(address) = address else { + return Err(miette::miette!("socket probe requires IPv4")); + }; + let mut bytes = [0_u8; PROBE_SOCKADDR_IN_LEN]; + bytes[0..2].copy_from_slice( + &libc::sa_family_t::try_from(libc::AF_INET) + .expect("AF_INET fits sa_family_t") + .to_ne_bytes(), + ); + bytes[2..4].copy_from_slice(&address.port().to_be_bytes()); + bytes[4..8].copy_from_slice(&address.ip().octets()); + Ok(( + bytes, + libc::socklen_t::try_from(PROBE_SOCKADDR_IN_LEN) + .expect("sockaddr_in length fits socklen_t"), + )) +} - /// Unix socket the embedded SSH daemon binds. On Linux, a value beginning - /// with `@` selects an abstract socket in the network namespace. - /// The supervisor bridges `RelayStream` traffic from the gateway onto - /// this socket; nothing else should connect to it. - #[arg(long, env = openshell_core::sandbox_env::SSH_SOCKET_PATH)] - ssh_socket_path: Option, +#[cfg(target_os = "linux")] +fn decode_probe_sockaddr(bytes: &[u8]) -> Result { + if bytes.len() != PROBE_SOCKADDR_IN_LEN { + return Err(miette::miette!("socket probe requires an IPv4 sockaddr")); + } + let family = libc::sa_family_t::from_ne_bytes([bytes[0], bytes[1]]); + if i32::from(family) != libc::AF_INET { + return Err(miette::miette!("socket probe sockaddr is not IPv4")); + } + Ok(std::net::SocketAddr::V4(std::net::SocketAddrV4::new( + std::net::Ipv4Addr::new(bytes[4], bytes[5], bytes[6], bytes[7]), + u16::from_be_bytes([bytes[2], bytes[3]]), + ))) +} - /// Enable health check endpoint. - #[arg(long)] - health_check: bool, - - /// Port for health check endpoint. - #[arg(long, default_value = "8080")] - health_port: u16, - - /// Which supervisor components to run. Comma-separated list of - /// "network" and/or "process". Defaults to both (single-binary - /// topology). Use --mode=network for a network-only sidecar, or - /// --mode=process for a process-only supervisor when network - /// enforcement runs in another pod. Use --mode=network-init only in - /// the Kubernetes init container that prepares sidecar nftables. - #[arg(long, default_value = DEFAULT_MODE)] - mode: Mode, - - /// UID that the long-running Kubernetes network sidecar will run as. - /// `--mode=network-init` installs nftables rules that exempt this UID. - #[arg(long, env = "OPENSHELL_PROXY_UID", default_value_t = 1337)] - proxy_uid: u32, - - /// GID assigned to shared sidecar state directories. Defaults to - /// `--proxy-uid` when omitted. - #[arg(long, env = "OPENSHELL_PROXY_GID")] - proxy_gid: Option, - - /// Shared state directory between the network init container and sidecar. - #[arg(long, env = "OPENSHELL_SIDECAR_STATE_DIR", default_value = SIDECAR_STATE_DIR)] - sidecar_state_dir: String, - - /// Shared TLS work directory between the network init container and sidecar. - #[arg(long, env = "OPENSHELL_PROXY_TLS_DIR", default_value = SIDECAR_TLS_DIR)] - sidecar_tls_dir: String, - - // Corporate upstream proxy. Operator-owned egress boundary: accepted - // only as command-line arguments (no `env =`), because the driver - // controls the supervisor's argv while a sandbox image could bake - // matching `ENV` values. - /// Corporate forward proxy URL (`http://host:port`) for upstream TLS egress. - #[arg(long)] - upstream_proxy: Option, +#[cfg(target_os = "linux")] +fn write_probe_sockaddr( + tid: u32, + address: u64, + length_address: u64, + peer: std::net::SocketAddr, +) -> Result<()> { + use std::mem::size_of; + + let (sockaddr, sockaddr_length) = encode_probe_sockaddr(peer)?; + let mut requested_length = [0_u8; size_of::()]; + openshell_isolation_interface::linux::task_memory::read_exact( + tid, + length_address, + &mut requested_length, + ) + .into_diagnostic()?; + let requested_length = libc::socklen_t::from_ne_bytes(requested_length); + if requested_length < sockaddr_length { + return Err(miette::miette!("peer sockaddr buffer is too small")); + } + openshell_isolation_interface::linux::task_memory::write_exact(tid, address, &sockaddr) + .into_diagnostic()?; + openshell_isolation_interface::linux::task_memory::write_exact( + tid, + length_address, + &sockaddr_length.to_ne_bytes(), + ) + .into_diagnostic()?; + Ok(()) +} - /// Comma-separated `NO_PROXY` list for the corporate proxy. - #[arg(long)] - upstream_no_proxy: Option, +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn run_capability_socket_child(args: &[String]) -> Result<()> { + use std::io::Read as _; + use std::net::SocketAddr; + use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; + + let [expected_peer, sandbox_tgid] = args else { + return Err(miette::miette!( + "usage: {CAPABILITY_SOCKET_CHILD_SUBCOMMAND} " + )); + }; + let expected_peer = expected_peer + .parse::() + .into_diagnostic() + .map_err(|error| miette::miette!("parse socket probe peer: {error}"))?; + let sandbox_tgid = sandbox_tgid + .parse::() + .into_diagnostic() + .map_err(|error| miette::miette!("parse sandbox TGID: {error}"))?; + let (sockaddr, sockaddr_length) = encode_probe_sockaddr(expected_peer)?; + + // SAFETY: this call is intentionally intercepted and completed with one + // newly injected socket descriptor. + let socket = unsafe { + libc::socket( + libc::AF_INET, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_TCP, + ) + }; + if socket < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + // SAFETY: successful socket returned one newly owned descriptor. + let socket = unsafe { OwnedFd::from_raw_fd(socket) }; + // SAFETY: dup creates an alias of the same open-file description. + let alias = unsafe { libc::dup(socket.as_raw_fd()) }; + if alias < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + // SAFETY: successful dup returned one newly owned descriptor. + let alias = unsafe { OwnedFd::from_raw_fd(alias) }; + + // SAFETY: epoll_create1 returns one owned descriptor; epoll_ctl consumes + // only the live event value for this call. + let epoll = unsafe { libc::epoll_create1(libc::EPOLL_CLOEXEC) }; + if epoll < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + // SAFETY: successful epoll_create1 returned one newly owned descriptor. + let epoll = unsafe { OwnedFd::from_raw_fd(epoll) }; + let mut event = libc::epoll_event { + events: u32::try_from(libc::EPOLLIN | libc::EPOLLOUT).expect("epoll flags fit u32"), + u64: 1, + }; + // SAFETY: descriptors and event pointer are live for this call. + if unsafe { + libc::epoll_ctl( + epoll.as_raw_fd(), + libc::EPOLL_CTL_ADD, + socket.as_raw_fd(), + std::ptr::addr_of_mut!(event), + ) + } < 0 + { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + + // SAFETY: the sockaddr is live and the alias references the mediated OFD. + if unsafe { libc::connect(alias.as_raw_fd(), sockaddr.as_ptr().cast(), sockaddr_length) } != 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + + let mut observed_peer = [0_u8; PROBE_SOCKADDR_IN_LEN]; + let mut observed_length = + libc::socklen_t::try_from(PROBE_SOCKADDR_IN_LEN).expect("sockaddr length fits socklen_t"); + // SAFETY: the output objects are live for the full declared length. + if unsafe { + libc::getpeername( + socket.as_raw_fd(), + observed_peer.as_mut_ptr().cast(), + std::ptr::addr_of_mut!(observed_length), + ) + } != 0 + { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + let observed = decode_probe_sockaddr(&observed_peer)?; + if observed != expected_peer { + return Err(miette::miette!( + "socket probe peer mismatch: expected {expected_peer}, got {observed}" + )); + } + + let request = b"ping"; + // SAFETY: null destination on a connected socket follows the cBPF fast + // path and reads only the live request buffer. + let sent = unsafe { + libc::sendto( + alias.as_raw_fd(), + request.as_ptr().cast(), + request.len(), + libc::MSG_NOSIGNAL, + std::ptr::null(), + 0, + ) + }; + if sent != isize::try_from(request.len()).expect("request length fits isize") { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + let mut ready = libc::epoll_event { events: 0, u64: 0 }; + // SAFETY: event points to storage for one returned event. + if unsafe { libc::epoll_wait(epoll.as_raw_fd(), std::ptr::addr_of_mut!(ready), 1, 5_000) } <= 0 + { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + let mut stream = std::net::TcpStream::from(socket); + let mut response = [0_u8; 4]; + stream.read_exact(&mut response).into_diagnostic()?; + if &response != b"pong" { + return Err(miette::miette!("socket probe response mismatch")); + } + probe_dns_socket_round_trip()?; + probe_tcp_dns_socket_round_trip()?; + probe_tcp_denial()?; + probe_child_self_protection(sandbox_tgid, alias.as_raw_fd())?; + Ok(()) +} + +#[cfg(not(target_os = "linux"))] +fn run_capability_socket_child(_args: &[String]) -> Result<()> { + Err(miette::miette!( + "socket qualification is supported only on Linux" + )) +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn probe_dns_socket_round_trip() -> Result<()> { + use std::net::SocketAddr; + use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; + + const DNS_QUERY: &[u8] = + b"\x12\x34\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x05probe\x09openshell\x04test\x00\x00\x01\x00\x01"; + let relay = "127.0.0.53:53" + .parse::() + .expect("fixed DNS relay address is valid"); + let (sockaddr, sockaddr_length) = encode_probe_sockaddr(relay)?; + // SAFETY: the syscall is intercepted and completed with a DNS-only + // registered socket descriptor. + let socket = unsafe { + libc::socket( + libc::AF_INET, + libc::SOCK_DGRAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_UDP, + ) + }; + if socket < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + // SAFETY: successful socket returned one newly owned descriptor. + let socket = unsafe { OwnedFd::from_raw_fd(socket) }; + // SAFETY: the destination and query buffers are live for the call. The + // broker copies and emulates this send before replying to the notification. + let sent = unsafe { + libc::sendto( + socket.as_raw_fd(), + DNS_QUERY.as_ptr().cast(), + DNS_QUERY.len(), + 0, + sockaddr.as_ptr().cast(), + sockaddr_length, + ) + }; + if sent != isize::try_from(DNS_QUERY.len()).expect("DNS query length fits isize") { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + + let mut response = [0_u8; 512]; + let mut source = [0_u8; PROBE_SOCKADDR_IN_LEN]; + let mut source_length = + libc::socklen_t::try_from(source.len()).expect("sockaddr length fits socklen_t"); + // SAFETY: all output buffers are live for their declared lengths. + let received = unsafe { + libc::recvfrom( + socket.as_raw_fd(), + response.as_mut_ptr().cast(), + response.len(), + 0, + source.as_mut_ptr().cast(), + std::ptr::addr_of_mut!(source_length), + ) + }; + if received < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + let source = decode_probe_sockaddr(&source)?; + if source != relay { + return Err(miette::miette!( + "DNS response source mismatch: expected {relay}, got {source}" + )); + } + let received = usize::try_from(received).expect("positive recv length fits usize"); + if received < 16 + || response[0..2] != DNS_QUERY[0..2] + || response[2] & 0x80 == 0 + || response[received - 4..received] != [203, 0, 113, 7] + { + return Err(miette::miette!("DNS probe response is malformed")); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn probe_tcp_dns_socket_round_trip() -> Result<()> { + use std::io::{Read as _, Write as _}; + + const DNS_QUERY: &[u8] = + b"\x56\x78\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x05probe\x09openshell\x04test\x00\x00\x01\x00\x01"; + let mut stream = std::net::TcpStream::connect("127.0.0.53:53").into_diagnostic()?; + stream.set_nodelay(true).into_diagnostic()?; + stream + .write_all( + &u16::try_from(DNS_QUERY.len()) + .expect("probe DNS query fits u16") + .to_be_bytes(), + ) + .into_diagnostic()?; + stream.write_all(DNS_QUERY).into_diagnostic()?; + let mut length = [0_u8; 2]; + stream.read_exact(&mut length).into_diagnostic()?; + let length = usize::from(u16::from_be_bytes(length)); + if length == 0 || length > 512 { + return Err(miette::miette!("TCP DNS probe response length is invalid")); + } + let mut response = vec![0_u8; length]; + stream.read_exact(&mut response).into_diagnostic()?; + if length < 16 + || response[0..2] != DNS_QUERY[0..2] + || response[2] & 0x80 == 0 + || response[length - 4..] != [203, 0, 113, 7] + { + return Err(miette::miette!("TCP DNS probe response is malformed")); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn probe_tcp_denial() -> Result<()> { + use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; + + let peer = PROBE_DENIED_TCP_PEER + .parse::() + .expect("fixed denied peer is valid"); + let (sockaddr, length) = encode_probe_sockaddr(peer)?; + // SAFETY: socket creation is intercepted and returns one injected FD. + let socket = unsafe { + libc::socket( + libc::AF_INET, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_TCP, + ) + }; + if socket < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + // SAFETY: successful socket returned one newly owned descriptor. + let socket = unsafe { OwnedFd::from_raw_fd(socket) }; + // SAFETY: both the injected FD and encoded sockaddr are live. + let result = unsafe { libc::connect(socket.as_raw_fd(), sockaddr.as_ptr().cast(), length) }; + require_probe_errno( + isize::try_from(result).expect("connect result fits isize"), + libc::EACCES, + "denied TCP connect", + ) +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn probe_child_self_protection(sandbox_tgid: libc::pid_t, socket: libc::c_int) -> Result<()> { + // SAFETY: PR_GET_DUMPABLE reads one scalar property. Normal exec of the + // trusted child image must make it observable to the same-UID sandbox. + if unsafe { libc::prctl(libc::PR_GET_DUMPABLE, 0, 0, 0, 0) } != 1 { + return Err(miette::miette!("workload child is not dumpable after exec")); + } + let mut core_limit = libc::rlimit { + rlim_cur: libc::rlim_t::MAX, + rlim_max: libc::rlim_t::MAX, + }; + // SAFETY: `core_limit` is writable storage for the current limit. + if unsafe { libc::getrlimit(libc::RLIMIT_CORE, &raw mut core_limit) } < 0 { + return Err(std::io::Error::last_os_error()).into_diagnostic(); + } + if core_limit.rlim_cur != 0 || core_limit.rlim_max != 0 { + return Err(miette::miette!("workload child core limit is not zero")); + } + let mut local = 0_u8; + let remote = 0_u8; + let local_iov = libc::iovec { + iov_base: std::ptr::addr_of_mut!(local).cast(), + iov_len: 1, + }; + let remote_iov = libc::iovec { + iov_base: std::ptr::addr_of!(remote).cast_mut().cast(), + iov_len: 1, + }; + // SAFETY: live one-byte iovecs are supplied. The child filter must reject + // the operation before the kernel inspects the remote pointer. + let read = unsafe { + libc::process_vm_readv( + sandbox_tgid, + &raw const local_iov, + 1, + &raw const remote_iov, + 1, + 0, + ) + }; + require_probe_errno(read, libc::EPERM, "process_vm_readv sandbox")?; + // SAFETY: signal zero would only probe process existence if the filter did + // not reject the trusted sandbox target. + require_probe_errno( + isize::try_from(unsafe { libc::kill(sandbox_tgid, 0) }).expect("kill result fits isize"), + libc::EPERM, + "kill sandbox", + )?; + // SAFETY: scalar syscall arguments request a read-only resource query; + // the child filter rejects non-self targets. + require_probe_errno( + isize::try_from(unsafe { + libc::syscall(libc::SYS_prlimit64, sandbox_tgid, libc::RLIMIT_CORE, 0, 0) + }) + .expect("prlimit result fits isize"), + libc::EPERM, + "prlimit sandbox", + )?; + require_probe_errno( + isize::try_from(unsafe { libc::kill(-sandbox_tgid, 0) }).expect("kill result fits isize"), + libc::EPERM, + "process-group signal", + )?; + require_probe_errno( + isize::try_from(unsafe { libc::fcntl(socket, libc::F_SETOWN, sandbox_tgid) }) + .expect("fcntl result fits isize"), + libc::EPERM, + "fcntl F_SETOWN", + )?; + let mut owner = sandbox_tgid; + require_probe_errno( + isize::try_from(unsafe { + libc::syscall(libc::SYS_ioctl, socket, 0x8901_u32, &raw mut owner) + }) + .expect("ioctl result fits isize"), + libc::EPERM, + "ioctl FIOSETOWN", + )?; + Ok(()) +} + +#[cfg(target_os = "linux")] +fn require_probe_errno(result: isize, expected: i32, operation: &str) -> Result<()> { + if result == -1 && std::io::Error::last_os_error().raw_os_error() == Some(expected) { + Ok(()) + } else { + Err(miette::miette!( + "{operation} was not rejected with errno {expected}" + )) + } +} + +/// Exercise the trusted VM bootstrap transition before running the Phase 0 +/// capability probe. This command is intentionally hidden: it exists so the +/// VM conformance lane can prove that a privileged guest init can hand off to +/// a non-root, capability-free sandbox without relying on a shell utility. +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] +fn enter_capability_free_identity(uid: u32, gid: u32) -> Result<()> { + use miette::{Context as _, IntoDiagnostic as _}; + + #[repr(C)] + struct CapabilityHeader { + version: u32, + pid: i32, + } + #[repr(C)] + #[derive(Clone, Copy)] + struct CapabilityData { + effective: u32, + permitted: u32, + inheritable: u32, + } + + if uid == 0 || gid == 0 { + return Err(miette::miette!( + "capability-free launch requires non-root UID and GID" + )); + } + if nix::unistd::geteuid().as_raw() != 0 { + return Err(miette::miette!("capability-free launch must start as root")); + } + + let cap_last_cap = std::fs::read_to_string("/proc/sys/kernel/cap_last_cap") + .into_diagnostic() + .wrap_err("read cap_last_cap")? + .trim() + .parse::() + .into_diagnostic() + .wrap_err("parse cap_last_cap")?; + for capability in 0..=cap_last_cap { + // SAFETY: PR_CAPBSET_DROP only removes one capability from the current + // process' bounding set. The loop runs while guest init still has the + // authority required to perform the transition. + if unsafe { libc::prctl(libc::PR_CAPBSET_DROP, capability, 0, 0, 0) } < 0 { + return Err(miette::miette!( + "drop capability {capability} from bounding set: {}", + std::io::Error::last_os_error() + )); + } + } + + // SAFETY: the process is single-threaded at this pre-clap bootstrap path; + // the null pointer is valid for a zero-length supplementary group list. + if unsafe { libc::setgroups(0, std::ptr::null()) } < 0 { + return Err(miette::miette!( + "clear supplementary groups: {}", + std::io::Error::last_os_error() + )); + } + // SAFETY: scalar credential transition to the operator-selected guest + // identity. All saved IDs are changed so the process cannot regain root. + if unsafe { libc::setresgid(gid, gid, gid) } < 0 { + return Err(miette::miette!( + "set guest GID {gid}: {}", + std::io::Error::last_os_error() + )); + } + // SAFETY: see setresgid above. + if unsafe { libc::setresuid(uid, uid, uid) } < 0 { + return Err(miette::miette!( + "set guest UID {uid}: {}", + std::io::Error::last_os_error() + )); + } - /// Path to the root-only file holding corporate proxy credentials (`user:pass`). - #[arg(long)] - upstream_proxy_auth_file: Option, + let mut header = CapabilityHeader { + version: 0x2008_0522, + pid: 0, + }; + let data = [CapabilityData { + effective: 0, + permitted: 0, + inheritable: 0, + }; 2]; + // SAFETY: capset reads the fixed-size header and two zeroed V3 data words. + if unsafe { libc::syscall(libc::SYS_capset, &raw mut header, data.as_ptr()) } < 0 { + return Err(miette::miette!( + "clear process capability sets: {}", + std::io::Error::last_os_error() + )); + } + // SAFETY: clears any ambient capabilities, then permanently forbids + // privilege gain across the following exec. + if unsafe { + libc::prctl( + libc::PR_CAP_AMBIENT, + libc::PR_CAP_AMBIENT_CLEAR_ALL, + 0, + 0, + 0, + ) + } < 0 + { + return Err(miette::miette!( + "clear ambient capabilities: {}", + std::io::Error::last_os_error() + )); + } + // SAFETY: PR_SET_NO_NEW_PRIVS is a one-way process hardening transition. + if unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) } < 0 { + return Err(miette::miette!( + "set no_new_privs: {}", + std::io::Error::last_os_error() + )); + } - /// Acknowledge that proxy credentials travel as cleartext Basic auth over - /// the plain-TCP connection to the `http://` proxy. - #[arg(long)] - upstream_proxy_auth_allow_insecure: bool, + Ok(()) +} - /// Send the destination hostname in CONNECT instead of a validated IP - /// (for proxies whose ACLs filter on hostnames). - #[arg(long)] - upstream_proxy_connect_by_hostname: bool, +#[cfg(target_os = "linux")] +fn launch_capability_probe(args: &[String]) -> Result<()> { + use miette::{Context as _, IntoDiagnostic as _}; - /// Path to a PEM CA bundle trusted for the corporate proxy: the TLS - /// handshake with an `https://` proxy and, for TLS-intercepting proxies, - /// re-signed upstream certificates and the sandbox trust bundle. - #[arg(long)] - upstream_proxy_ca_bundle: Option, + let [uid, gid] = args else { + return Err(miette::miette!( + "usage: openshell-sandbox {CAPABILITY_PROBE_LAUNCH_SUBCOMMAND} " + )); + }; + let uid = uid.parse::().into_diagnostic().wrap_err("parse UID")?; + let gid = gid.parse::().into_diagnostic().wrap_err("parse GID")?; + enter_capability_free_identity(uid, gid)?; + run_capability_probe() } -/// Internal one-shot command used by the privileged supervisor to validate an -/// image-provided workdir as the final sandbox identity. -#[derive(Parser, Debug)] -#[command(name = "validate-workspace", hide = true)] -struct ValidateWorkspaceArgs { - #[arg(long)] - workdir: String, - #[arg(long)] - expected_uid: u32, - #[arg(long)] - expected_gid: u32, +#[cfg(not(target_os = "linux"))] +fn launch_capability_probe(_args: &[String]) -> Result<()> { + Err(miette::miette!( + "capability probe launch is supported only on Linux" + )) } #[cfg(target_os = "linux")] -fn validate_workspace(args: &[String]) -> Result<()> { - let args = ValidateWorkspaceArgs::try_parse_from( - std::iter::once(VALIDATE_WORKSPACE_SUBCOMMAND.to_string()).chain(args.iter().cloned()), - ) - .into_diagnostic()?; - let actual = ( - nix::unistd::geteuid().as_raw(), - nix::unistd::getegid().as_raw(), - ); - if actual != (args.expected_uid, args.expected_gid) { +fn launch_capability_free(args: &[String]) -> Result<()> { + use miette::{Context as _, IntoDiagnostic as _}; + + let [uid, gid, bootstrap] = args else { return Err(miette::miette!( - "workspace validator privilege drop failed: expected {}:{}, got {}:{}", - args.expected_uid, - args.expected_gid, - actual.0, - actual.1 + "usage: openshell-sandbox {CAPABILITY_FREE_LAUNCH_SUBCOMMAND} " )); - } - openshell_supervisor_process::process::validate_oci_workspace_as_effective_identity(Path::new( - &args.workdir, + }; + let uid = uid.parse::().into_diagnostic().wrap_err("parse UID")?; + let gid = gid.parse::().into_diagnostic().wrap_err("parse GID")?; + enter_capability_free_identity(uid, gid)?; + let log_level = std::env::var(openshell_core::sandbox_env::LOG_LEVEL) + .unwrap_or_else(|_| "warn".to_string()); + run_boundary(Path::new(bootstrap), &log_level) +} + +#[cfg(not(target_os = "linux"))] +fn launch_capability_free(_args: &[String]) -> Result<()> { + Err(miette::miette!( + "capability-free launch is only supported on Linux" )) } #[cfg(not(target_os = "linux"))] -fn validate_workspace(_args: &[String]) -> Result<()> { +fn run_capability_probe() -> Result<()> { Err(miette::miette!( - "workspace validation is only supported on Unix" + "capability-free sandbox probe is supported only on Linux" )) } +#[cfg(target_os = "linux")] +fn proc_status_hex(status: &str, field: &str) -> Result { + let value = status + .lines() + .find_map(|line| line.strip_prefix(&format!("{field}:"))) + .map(str::trim) + .ok_or_else(|| miette::miette!("/proc/self/status is missing {field}"))?; + u64::from_str_radix(value, 16) + .map_err(|error| miette::miette!("invalid {field} value {value:?}: {error}")) +} + /// Copy the running executable to `dest`, creating parent directories as /// needed and ensuring the result is executable (mode `0755`). /// @@ -317,440 +1596,397 @@ fn copy_self(dest: &str) -> Result<()> { Ok(()) } +/// Stage the immutable Kubernetes bootstrap Secret into private writable +/// memory-backed volumes. The projected Secret remains mounted only in this +/// trusted init container; the long-lived sandbox consumes and unlinks the +/// staged configuration before it starts workload code. #[cfg(target_os = "linux")] -fn prepare_sidecar_directory(path: &Path, uid: u32, gid: u32, mode: u32) -> Result<()> { - use miette::Context as _; - use nix::unistd::{Gid, Uid, chown}; - use std::os::unix::fs::PermissionsExt; - - std::fs::create_dir_all(path) - .into_diagnostic() - .wrap_err_with(|| format!("failed to create sidecar directory {}", path.display()))?; - let mut perms = std::fs::metadata(path).into_diagnostic()?.permissions(); - perms.set_mode(mode); - std::fs::set_permissions(path, perms) - .into_diagnostic() - .wrap_err_with(|| format!("failed to chmod sidecar directory {}", path.display()))?; - chown(path, Some(Uid::from_raw(uid)), Some(Gid::from_raw(gid))) - .into_diagnostic() - .wrap_err_with(|| { - format!( - "failed to chown sidecar directory {} to {uid}:{gid}", - path.display() - ) - })?; - Ok(()) +fn stage_kubernetes_bootstrap() -> Result<()> { + stage_kubernetes_bootstrap_at( + Path::new(BOOTSTRAP_INPUT_ROOT), + Path::new(SANDBOX_RUNTIME_ROOT), + Path::new(SANDBOX_STATE_ROOT), + ) } +/// Stage protected state without performing a duplicate runtime probe. The +/// long-lived sandbox actively qualifies its own exact admitted profile before +/// consuming this material. #[cfg(target_os = "linux")] -fn prepare_sidecar_directory_for_current_user(path: &Path, mode: u32) -> Result<()> { - use miette::Context as _; - use nix::unistd::{Gid, Uid, chown}; - use std::os::unix::fs::PermissionsExt; +fn run_kubernetes_bootstrap() -> Result<()> { + stage_kubernetes_bootstrap() +} - let uid = Uid::current(); - let gid = Gid::current(); - std::fs::create_dir_all(path) - .into_diagnostic() - .wrap_err_with(|| format!("failed to create sidecar directory {}", path.display()))?; - chown(path, Some(uid), Some(gid)) - .into_diagnostic() - .wrap_err_with(|| { - format!( - "failed to chown sidecar directory {} to {}:{}", - path.display(), - uid.as_raw(), - gid.as_raw() - ) - })?; - let mut perms = std::fs::metadata(path).into_diagnostic()?.permissions(); - perms.set_mode(mode); - std::fs::set_permissions(path, perms) - .into_diagnostic() - .wrap_err_with(|| format!("failed to chmod sidecar directory {}", path.display()))?; +#[cfg(not(target_os = "linux"))] +fn run_kubernetes_bootstrap() -> Result<()> { + Err(miette::miette!( + "Kubernetes sandbox bootstrap requires Linux" + )) +} + +#[cfg(any(target_os = "linux", test))] +fn stage_kubernetes_bootstrap_at(source: &Path, runtime: &Path, state: &Path) -> Result<()> { + use std::fs::{self, OpenOptions}; + use std::os::unix::fs::PermissionsExt as _; + + let uid = nix::unistd::geteuid().as_raw(); + let gid = nix::unistd::getegid().as_raw(); + if uid == 0 || gid == 0 { + return Err(miette::miette!( + "Kubernetes bootstrap staging requires a non-root UID and GID, got {uid}:{gid}" + )); + } + + fs::create_dir_all(runtime).into_diagnostic()?; + fs::create_dir_all(state).into_diagnostic()?; + + let nonce = format!( + "{}.{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .into_diagnostic()? + .as_nanos() + ); + let runtime_tmp = runtime.join(format!(".openshell-sandbox.{nonce}")); + let runtime_final = runtime.join("openshell-sandbox"); + let executable = std::env::current_exe().into_diagnostic()?; + copy_regular_file(&executable, &runtime_tmp, 0o500)?; + fs::rename(&runtime_tmp, &runtime_final).into_diagnostic()?; + + let bundle_tmp = state.join(format!(".bootstrap.{nonce}")); + let bundle_final = state.join("bootstrap"); + fs::create_dir(&bundle_tmp).into_diagnostic()?; + fs::set_permissions(&bundle_tmp, fs::Permissions::from_mode(0o700)).into_diagnostic()?; + for name in KUBERNETES_BOOTSTRAP_SECRET_FILES { + copy_projected_secret_file(source, name, &bundle_tmp.join(name), 0o600)?; + } + fs::rename(&bundle_tmp, &bundle_final).into_diagnostic()?; + + // Flush the two directory entries before the init container exits. Both + // targets are tmpfs in production, but keeping the staging operation + // durable also makes the helper safe in local conformance tests. + OpenOptions::new() + .read(true) + .open(runtime) + .into_diagnostic()? + .sync_all() + .into_diagnostic()?; + OpenOptions::new() + .read(true) + .open(state) + .into_diagnostic()? + .sync_all() + .into_diagnostic()?; Ok(()) } -#[cfg(target_os = "linux")] -fn copy_sidecar_client_tls_if_present( - source_dir: &Path, - sidecar_tls_dir: &Path, - uid: u32, - gid: u32, +#[cfg(any(target_os = "linux", test))] +fn copy_projected_secret_file( + source_root: &Path, + name: &str, + destination: &Path, + mode: u32, ) -> Result<()> { - use miette::Context as _; - use nix::unistd::{Gid, Uid, chown}; - use std::os::unix::fs::PermissionsExt; + let canonical_root = std::fs::canonicalize(source_root).into_diagnostic()?; + let canonical_source = std::fs::canonicalize(source_root.join(name)).into_diagnostic()?; + if !canonical_source.starts_with(&canonical_root) { + return Err(miette::miette!( + "projected bootstrap input escapes its mounted Secret: {}", + source_root.join(name).display() + )); + } + copy_regular_file(&canonical_source, destination, mode) +} - if !source_dir.exists() { - return Ok(()); +#[cfg(any(target_os = "linux", test))] +fn copy_regular_file(source: &Path, destination: &Path, mode: u32) -> Result<()> { + use std::fs::{self, OpenOptions}; + use std::io::{Read as _, Write as _}; + use std::os::unix::fs::OpenOptionsExt as _; + + let metadata = fs::symlink_metadata(source).into_diagnostic()?; + if !metadata.file_type().is_file() || metadata.len() == 0 { + return Err(miette::miette!( + "bootstrap input must be a non-empty regular file: {}", + source.display() + )); + } + let mut input = OpenOptions::new() + .read(true) + .open(source) + .into_diagnostic()?; + let mut output = OpenOptions::new() + .write(true) + .create_new(true) + .mode(mode) + .open(destination) + .into_diagnostic()?; + let mut buffer = [0_u8; 16 * 1024]; + loop { + let length = input.read(&mut buffer).into_diagnostic()?; + if length == 0 { + break; + } + output.write_all(&buffer[..length]).into_diagnostic()?; } + output.sync_all().into_diagnostic()?; + Ok(()) +} - let dest_dir = sidecar_tls_dir.join(SIDECAR_CLIENT_TLS_SUBDIR); - prepare_sidecar_directory_for_current_user(&dest_dir, SIDECAR_TLS_STAGING_DIR_MODE)?; - for file_name in CLIENT_TLS_FILES { - let source = source_dir.join(file_name); - if !source.exists() { +/// Seed the persistent workspace from the agent image as the final workload +/// identity. This replaces the former root shell/tar init container. +fn seed_kubernetes_workspace() -> Result<()> { + seed_kubernetes_workspace_at(Path::new("/sandbox"), Path::new("/mnt/openshell-workspace")) +} + +fn copy_workspace_tree(source: &Path, destination: &Path) -> Result<()> { + use std::fs::{self, OpenOptions}; + use std::io::{Read as _, Write as _}; + use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _, symlink}; + + for entry in fs::read_dir(source).into_diagnostic()? { + let entry = entry.into_diagnostic()?; + let source_path = entry.path(); + let destination_path = destination.join(entry.file_name()); + let metadata = fs::symlink_metadata(&source_path).into_diagnostic()?; + if metadata.file_type().is_symlink() { + symlink( + fs::read_link(&source_path).into_diagnostic()?, + &destination_path, + ) + .into_diagnostic()?; + } else if metadata.is_dir() { + fs::create_dir(&destination_path).into_diagnostic()?; + fs::set_permissions(&destination_path, fs::Permissions::from_mode(0o700)) + .into_diagnostic()?; + copy_workspace_tree(&source_path, &destination_path)?; + } else if metadata.is_file() { + let mut input = OpenOptions::new() + .read(true) + .open(&source_path) + .into_diagnostic()?; + let mode = 0o600 | (metadata.permissions().mode() & 0o100); + let mut output = OpenOptions::new() + .write(true) + .create_new(true) + .mode(mode) + .open(&destination_path) + .into_diagnostic()?; + let mut buffer = [0_u8; 16 * 1024]; + loop { + let length = input.read(&mut buffer).into_diagnostic()?; + if length == 0 { + break; + } + output.write_all(&buffer[..length]).into_diagnostic()?; + } + output.sync_all().into_diagnostic()?; + } else { return Err(miette::miette!( - "client TLS source file is missing: {}", - source.display() + "workspace seed contains unsupported file type: {}", + source_path.display() )); } - let dest = dest_dir.join(file_name); - if dest.exists() { - std::fs::remove_file(&dest) - .into_diagnostic() - .wrap_err_with(|| { - format!("failed to remove stale client TLS file {}", dest.display()) - })?; - } - std::fs::copy(&source, &dest) - .into_diagnostic() - .wrap_err_with(|| { - format!( - "failed to copy client TLS file {} to {}", - source.display(), - dest.display() - ) - })?; - let mut perms = std::fs::metadata(&dest).into_diagnostic()?.permissions(); - perms.set_mode(SIDECAR_CLIENT_TLS_FILE_MODE); - std::fs::set_permissions(&dest, perms) - .into_diagnostic() - .wrap_err_with(|| { - format!("failed to chmod copied client TLS file {}", dest.display()) - })?; - chown(&dest, Some(Uid::from_raw(uid)), Some(Gid::from_raw(gid))) - .into_diagnostic() - .wrap_err_with(|| { - format!( - "failed to chown copied client TLS file {} to {uid}:{gid}", - dest.display() - ) - })?; } - - prepare_sidecar_directory(&dest_dir, uid, gid, SIDECAR_CLIENT_TLS_DIR_MODE)?; - Ok(()) } -#[cfg(target_os = "linux")] -fn run_network_init( - proxy_user_id: u32, - proxy_primary_group_id: u32, - sidecar_state_dir: &str, - sidecar_tls_dir: &str, -) -> Result<()> { - validate_network_init_ids(proxy_user_id, proxy_primary_group_id)?; - - let sidecar_state_dir = Path::new(sidecar_state_dir); - let sidecar_tls_dir = Path::new(sidecar_tls_dir); - prepare_sidecar_directory( - sidecar_state_dir, - proxy_user_id, - proxy_primary_group_id, - SIDECAR_STATE_DIR_MODE, - )?; - // The init container runs as uid 0 with CAP_DAC_OVERRIDE dropped. Keep the - // TLS work directory owned by the init user until the client cert copy is - // complete, then hand it to the long-running proxy UID. - prepare_sidecar_directory_for_current_user(sidecar_tls_dir, SIDECAR_TLS_DIR_MODE)?; - copy_sidecar_client_tls_if_present( - Path::new(CLIENT_TLS_DIR), - sidecar_tls_dir, - proxy_user_id, - proxy_primary_group_id, - )?; - prepare_sidecar_directory( - sidecar_tls_dir, - proxy_user_id, - proxy_primary_group_id, - SIDECAR_TLS_DIR_MODE, - )?; - openshell_supervisor_process::netns::install_sidecar_bypass_rules(proxy_user_id) -} +fn seed_kubernetes_workspace_at(source: &Path, destination: &Path) -> Result<()> { + use std::fs::{self, OpenOptions}; + use std::io::Write as _; + use std::os::unix::fs::OpenOptionsExt as _; -#[cfg(target_os = "linux")] -fn validate_network_init_ids(proxy_user_id: u32, proxy_primary_group_id: u32) -> Result<()> { - if proxy_user_id != 0 - && !(openshell_policy::MIN_SANDBOX_PROXY_UID..=openshell_policy::MAX_SANDBOX_UID) - .contains(&proxy_user_id) - { - return Err(miette::miette!( - "--proxy-uid must be 0 or in range [{}, {}]", - openshell_policy::MIN_SANDBOX_PROXY_UID, - openshell_policy::MAX_SANDBOX_UID, - )); + let sentinel = destination.join(".openshell-initialized"); + if sentinel.try_exists().into_diagnostic()? { + return Ok(()); } - if !(openshell_policy::MIN_SANDBOX_UID..=openshell_policy::MAX_SANDBOX_UID) - .contains(&proxy_primary_group_id) - { + let destination_metadata = fs::symlink_metadata(destination).into_diagnostic()?; + if !destination_metadata.is_dir() || destination_metadata.file_type().is_symlink() { return Err(miette::miette!( - "--proxy-gid must be in range [{}, {}]", - openshell_policy::MIN_SANDBOX_UID, - openshell_policy::MAX_SANDBOX_UID, + "workspace target must be a real directory: {}", + destination.display() )); } + + if source.try_exists().into_diagnostic()? { + let metadata = fs::symlink_metadata(source).into_diagnostic()?; + if !metadata.is_dir() || metadata.file_type().is_symlink() { + return Err(miette::miette!( + "image workspace must be a real directory: {}", + source.display() + )); + } + copy_workspace_tree(source, destination)?; + } + let mut file = OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o600) + .open(&sentinel) + .into_diagnostic()?; + file.write_all(b"initialized\n").into_diagnostic()?; + file.sync_all().into_diagnostic()?; + OpenOptions::new() + .read(true) + .open(destination) + .into_diagnostic()? + .sync_all() + .into_diagnostic()?; Ok(()) } +#[cfg(target_os = "linux")] +fn run_boundary(bootstrap: &Path, log_level: &str) -> Result<()> { + let console_filter = + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(log_level)); + let _ = tracing_subscriber::registry() + .with( + OcsfShorthandLayer::new(std::io::stderr()) + .with_non_ocsf(true) + .with_filter(console_filter), + ) + .try_init(); + let (qualification, _) = qualify_runtime()?; + openshell_sandbox::run(bootstrap, qualification) +} + #[cfg(not(target_os = "linux"))] -fn run_network_init( - _proxy_uid: u32, - _proxy_gid: u32, - _sidecar_state_dir: &str, - _sidecar_tls_dir: &str, -) -> Result<()> { - Err(miette::miette!( - "--mode=network-init is only supported on Linux" - )) +fn run_boundary(_bootstrap: &Path, _log_level: &str) -> Result<()> { + Err(miette::miette!("openshell-sandbox requires Linux")) } fn main() -> Result<()> { - // Handle `copy-self ` before clap so it works without any of the - // sandbox flags. Kubernetes init containers invoke this path to seed an - // emptyDir volume that the agent container then executes from. - let raw_args: Vec = std::env::args().collect(); + let raw_args = std::env::args().collect::>(); if raw_args.get(1).map(String::as_str) == Some(COPY_SELF_SUBCOMMAND) { let dest = raw_args.get(2).ok_or_else(|| { miette::miette!("usage: openshell-sandbox {COPY_SELF_SUBCOMMAND} ") })?; return copy_self(dest); } - - // Handle `debug-rpc [args]` before clap. Uses a small - // dedicated runtime so we don't pay the supervisor's full startup cost. - if raw_args.get(1).map(String::as_str) == Some(DEBUG_RPC_SUBCOMMAND) { - let runtime = tokio::runtime::Builder::new_current_thread() - .enable_all() - .build() - .into_diagnostic()?; - return runtime.block_on(async move { - let exit = openshell_supervisor_process::debug_rpc::run(&raw_args[2..]).await?; - std::process::exit(exit); - }); + if raw_args.get(1).map(String::as_str) == Some(BOOTSTRAP_SUBCOMMAND) { + if raw_args.len() != 2 { + return Err(miette::miette!( + "usage: openshell-sandbox {BOOTSTRAP_SUBCOMMAND}" + )); + } + return run_kubernetes_bootstrap(); + } + if raw_args.get(1).map(String::as_str) == Some(SEED_WORKSPACE_SUBCOMMAND) { + if raw_args.len() != 2 { + return Err(miette::miette!( + "usage: openshell-sandbox {SEED_WORKSPACE_SUBCOMMAND}" + )); + } + return seed_kubernetes_workspace(); } if raw_args.get(1).map(String::as_str) == Some(VALIDATE_WORKSPACE_SUBCOMMAND) { return validate_workspace(&raw_args[2..]); } - - let args = Args::parse(); - - if args.mode.network_init { - let proxy_gid = args.proxy_gid.unwrap_or(args.proxy_uid); - return run_network_init( - args.proxy_uid, - proxy_gid, - &args.sidecar_state_dir, - &args.sidecar_tls_dir, - ); + if raw_args.get(1).map(String::as_str) == Some(CAPABILITY_PROBE_SUBCOMMAND) { + return run_capability_probe(); + } + if raw_args.get(1).map(String::as_str) == Some(CAPABILITY_PROBE_LAUNCH_SUBCOMMAND) { + return launch_capability_probe(&raw_args[2..]); + } + if raw_args.get(1).map(String::as_str) == Some(CAPABILITY_SOCKET_CHILD_SUBCOMMAND) { + return run_capability_socket_child(&raw_args[2..]); + } + if raw_args.get(1).map(String::as_str) == Some(CAPABILITY_LANDLOCK_CHILD_SUBCOMMAND) { + return run_capability_landlock_child(&raw_args[2..]); + } + if raw_args.get(1).map(String::as_str) == Some(CAPABILITY_FREE_LAUNCH_SUBCOMMAND) { + return launch_capability_free(&raw_args[2..]); } - // Try to open a rolling log file; fall back to stderr-only logging if it fails - // (e.g., /var/log is not writable in custom workload images). - // Rotates daily, keeps the 3 most recent files to bound disk usage. - let file_logging = tracing_appender::rolling::RollingFileAppender::builder() - .rotation(tracing_appender::rolling::Rotation::DAILY) - .filename_prefix("openshell") - .filename_suffix("log") - .max_log_files(3) - .build("/var/log") - .ok() - .map(|roller| { - let (writer, guard) = tracing_appender::non_blocking(roller); - (writer, guard) - }); - - let console_filter = - EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)); + let args = BoundaryArgs::parse(); + run_boundary(&args.bootstrap, &args.log_level) +} - let runtime = tokio::runtime::Builder::new_multi_thread() - .enable_all() - .build() - .into_diagnostic()?; +#[cfg(test)] +mod tests { + use super::*; + use std::os::unix::fs::PermissionsExt; - let result = runtime.block_on(async move { - // Set up optional log push layer (gRPC mode only). - let log_push_state = if let (Some(sandbox_id), Some(endpoint)) = - (&args.sandbox_id, &args.openshell_endpoint) - { - let (tx, handle) = openshell_supervisor_process::log_push::spawn_log_push_task( - endpoint.clone(), - sandbox_id.clone(), + #[test] + fn kubernetes_bootstrap_stages_private_memory_bundle() { + if nix::unistd::geteuid().is_root() || nix::unistd::getegid().as_raw() == 0 { + return; + } + let root = tempfile::tempdir().unwrap(); + let source = root.path().join("input"); + let runtime = root.path().join("runtime"); + let state = root.path().join("state"); + std::fs::create_dir(&source).unwrap(); + let revision = source.join("..2026_09_04_00_00_00"); + std::fs::create_dir(&revision).unwrap(); + for name in KUBERNETES_BOOTSTRAP_SECRET_FILES { + std::fs::write(revision.join(name), format!("contents-{name}")).unwrap(); + std::os::unix::fs::symlink(format!("..data/{name}"), source.join(name)).unwrap(); + } + std::os::unix::fs::symlink(revision.file_name().unwrap(), source.join("..data")).unwrap(); + + stage_kubernetes_bootstrap_at(&source, &runtime, &state).unwrap(); + + assert!(runtime.join("openshell-sandbox").is_file()); + assert_eq!( + std::fs::metadata(runtime.join("openshell-sandbox")) + .unwrap() + .permissions() + .mode() + & 0o777, + 0o500 + ); + for name in KUBERNETES_BOOTSTRAP_SECRET_FILES { + let staged = state.join("bootstrap").join(name); + assert_eq!( + std::fs::read(&staged).unwrap(), + std::fs::read(source.join(name)).unwrap() + ); + assert_eq!( + std::fs::metadata(staged).unwrap().permissions().mode() & 0o777, + 0o600 ); - let layer = - openshell_supervisor_process::log_push::LogPushLayer::new(sandbox_id.clone(), tx); - Some((layer, handle)) - } else { - None - }; - let push_layer = log_push_state.as_ref().map(|(layer, _)| layer.clone()); - let _log_push_handle = log_push_state.map(|(_, handle)| handle); - - // Shared flag: the sandbox poll loop toggles this when the - // `ocsf_json_enabled` setting changes. The JSONL layer checks it - // on each event and short-circuits when false. - let ocsf_enabled = Arc::new(AtomicBool::new(false)); - let ocsf_schema_version = Arc::new(std::sync::Mutex::new(String::new())); - - // Keep guards alive for the entire process. When a guard is dropped the - // non-blocking writer flushes remaining logs. - let (_file_guard, _jsonl_guard) = if let Some((file_writer, file_guard)) = file_logging { - let file_filter = EnvFilter::new("info"); - - // OCSF JSONL file: rolling appender matching the main log file - // (daily rotation, 3 files max). Created eagerly but gated by the - // enabled flag — no JSONL is written until ocsf_json_enabled is set. - let jsonl_logging = tracing_appender::rolling::RollingFileAppender::builder() - .rotation(tracing_appender::rolling::Rotation::DAILY) - .filename_prefix("openshell-ocsf") - .filename_suffix("log") - .max_log_files(3) - .build("/var/log") - .ok() - .map(|roller| { - let (writer, guard) = tracing_appender::non_blocking(roller); - let layer = OcsfJsonlLayer::new(writer) - .with_enabled_flag(ocsf_enabled.clone()) - .with_target_version(ocsf_schema_version.clone()); - (layer, guard) - }); - let (jsonl_layer, jsonl_guard) = match jsonl_logging { - Some((layer, guard)) => (Some(layer), Some(guard)), - None => (None, None), - }; - - tracing_subscriber::registry() - .with( - OcsfShorthandLayer::new(std::io::stderr()) - .with_non_ocsf(true) - .with_filter(console_filter), - ) - .with( - OcsfShorthandLayer::new(file_writer) - .with_non_ocsf(true) - .with_filter(file_filter), - ) - .with(jsonl_layer.with_filter(LevelFilter::INFO)) - .with(push_layer.clone()) - .init(); - (Some(file_guard), jsonl_guard) - } else { - tracing_subscriber::registry() - .with( - OcsfShorthandLayer::new(std::io::stderr()) - .with_non_ocsf(true) - .with_filter(console_filter), - ) - .with(push_layer) - .init(); - // Log the warning after the subscriber is initialized - warn!("Could not open /var/log for log rotation; using stderr-only logging"); - (None, None) - }; - - // Resolve an exact canonical process. Explicit offline/test argv wins; - // drivers otherwise provide a versioned JSON transport so argument - // boundaries are never reconstructed with shell parsing. - let workdir = args.workdir.clone(); - let (command, interactive, await_main_process_attachment) = if !args.command.is_empty() { - (args.command, args.interactive, false) - } else if let Ok(json) = std::env::var(openshell_core::sandbox_env::MAIN_PROCESS_SPEC) { - let config = openshell_core::sandbox_env::MainProcessConfig::decode(&json) - .map_err(|error| miette::miette!("{error}"))?; - ( - config.command, - config.tty, - config.await_main_process_attachment, - ) - } else { - let config = openshell_core::sandbox_env::MainProcessConfig::scratch(); - ( - config.command, - config.tty, - config.await_main_process_attachment, - ) - }; - - // An omitted command (the gateway leaves the default empty rather than - // baking a shell it cannot verify) is resolved to a login shell here, in - // the supervisor, so it matches the sandbox image: bash when present, - // otherwise /bin/sh (e.g. Alpine). An explicit command is used verbatim. - let command = resolve_default_command(command); - - info!(command = ?command, "Starting sandbox"); - // Note: "Starting sandbox" stays as plain info!() since the OCSF context - // is not yet initialized at this point (run_sandbox hasn't been called). - // The shorthand layer will render it in fallback format. - - let upstream_proxy_args = openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs { - https_proxy: args.upstream_proxy, - no_proxy: args.upstream_no_proxy, - proxy_auth_file: args.upstream_proxy_auth_file, - proxy_auth_allow_insecure: args.upstream_proxy_auth_allow_insecure, - proxy_connect_by_hostname: args.upstream_proxy_connect_by_hostname, - proxy_ca_bundle: args.upstream_proxy_ca_bundle, - }; - - run_sandbox( - command, - workdir, - args.timeout, - interactive, - await_main_process_attachment, - args.sandbox_id, - args.sandbox, - args.openshell_endpoint, - args.policy_rules, - args.policy_data, - args.ssh_socket_path, - args.health_check, - args.health_port, - ocsf_enabled, - ocsf_schema_version, - args.mode.network, - args.mode.process, - upstream_proxy_args, - ) - .await - }); - - let exit_code = match result { - Ok(exit_code) => exit_code, - Err(error) - if error - .to_string() - .contains("image workspace validation failed") => - { - error!(%error, "Image workspace validation failed"); - eprintln!("{error:?}"); - openshell_core::driver_utils::SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED } - Err(error) => return Err(error), - }; + } - std::process::exit(exit_code); -} + #[test] + fn kubernetes_workspace_seed_preserves_files_and_symlinks_without_root() { + let root = tempfile::tempdir().unwrap(); + let source = root.path().join("source"); + let destination = root.path().join("destination"); + std::fs::create_dir_all(source.join("bin")).unwrap(); + std::fs::create_dir(&destination).unwrap(); + std::fs::write(source.join("README"), b"workspace").unwrap(); + std::fs::write(source.join("bin/tool"), b"tool").unwrap(); + let mut executable = std::fs::metadata(source.join("bin/tool")) + .unwrap() + .permissions(); + executable.set_mode(0o755); + std::fs::set_permissions(source.join("bin/tool"), executable).unwrap(); + std::os::unix::fs::symlink("README", source.join("latest")).unwrap(); -/// Resolve an omitted canonical command to a login shell that exists in this -/// sandbox image. Empty means "use the default": the gateway leaves an omitted -/// command empty rather than persisting a shell it cannot verify, so the -/// supervisor picks one here against the real sandbox filesystem (bash when -/// present, otherwise `/bin/sh`). An explicit command is returned unchanged. -fn resolve_default_command(command: Vec) -> Vec { - if !command.is_empty() { - return command; - } - let shell = openshell_core::shell::detect_login_shell(); - info!(shell = %shell, "no command specified; resolved default login shell"); - vec![shell, "-l".to_string()] -} + seed_kubernetes_workspace_at(&source, &destination).unwrap(); + seed_kubernetes_workspace_at(&source, &destination).unwrap(); -#[cfg(test)] -mod tests { - use super::*; - use std::os::unix::fs::PermissionsExt; + assert_eq!( + std::fs::read(destination.join("README")).unwrap(), + b"workspace" + ); + assert_eq!( + std::fs::read_link(destination.join("latest")).unwrap(), + Path::new("README") + ); + assert_ne!( + std::fs::metadata(destination.join("bin/tool")) + .unwrap() + .permissions() + .mode() + & 0o100, + 0 + ); + assert!(destination.join(".openshell-initialized").is_file()); + } #[cfg(target_os = "linux")] #[test] @@ -829,56 +2065,4 @@ mod tests { let final_path = dest_dir.join("openshell-sandbox"); assert!(final_path.exists(), "binary should land inside dest dir"); } - - #[test] - fn mode_parses_network_init_standalone() { - let mode = "network-init".parse::().unwrap(); - assert!(mode.network_init); - assert!(!mode.network); - assert!(!mode.process); - } - - #[test] - fn mode_rejects_combined_network_init() { - let err = "network-init,network".parse::().unwrap_err(); - assert!(err.contains("cannot be combined")); - } - - #[test] - fn mode_rejects_empty_value() { - let err = "".parse::().unwrap_err(); - assert!(err.contains("at least one")); - } - - #[cfg(target_os = "linux")] - #[test] - fn sidecar_tls_modes_preserve_proxy_owned_parent_and_private_client_dir() { - assert_eq!(SIDECAR_TLS_DIR_MODE, 0o755); - assert_eq!(SIDECAR_TLS_STAGING_DIR_MODE, 0o700); - assert_eq!(SIDECAR_CLIENT_TLS_DIR_MODE, 0o750); - assert_eq!(SIDECAR_CLIENT_TLS_FILE_MODE, 0o400); - } - - #[cfg(target_os = "linux")] - #[test] - fn network_init_accepts_root_proxy_uid_for_binary_aware_sidecar() { - validate_network_init_ids(0, 30).unwrap(); - } - - #[cfg(target_os = "linux")] - #[test] - fn network_init_still_rejects_low_non_root_proxy_uid_and_root_gid() { - let uid_err = - validate_network_init_ids(999, openshell_policy::MIN_SANDBOX_UID).unwrap_err(); - assert!(uid_err.to_string().contains("--proxy-uid")); - - let gid_err = validate_network_init_ids(0, 0).unwrap_err(); - assert!(gid_err.to_string().contains("--proxy-gid")); - } - - #[cfg(target_os = "linux")] - #[test] - fn network_init_accepts_non_root_system_proxy_group() { - validate_network_init_ids(openshell_policy::MIN_SANDBOX_PROXY_UID, 30).unwrap(); - } } diff --git a/crates/openshell-sandbox/src/main_session.rs b/crates/openshell-sandbox/src/main_session.rs new file mode 100644 index 0000000000..898cfeaed3 --- /dev/null +++ b/crates/openshell-sandbox/src/main_session.rs @@ -0,0 +1,1067 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Retained I/O multiplexer for the canonical sandbox process. + +use std::collections::VecDeque; +use std::io::{Read, Write}; +use std::os::fd::AsRawFd; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; + +use bytes::Bytes; +use nix::fcntl::{FcntlArg, OFlag, fcntl}; +use nix::pty::Winsize; +use tokio::io::unix::AsyncFd; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::sync::Notify; +use tokio::sync::watch; + +use openshell_isolation_interface::contract::{ + BoundaryProcess, BoundarySignal, BoundaryTerminal, ProcessAttachment, +}; + +use crate::process::ProcessIo; + +const OUTPUT_BUFFER_BYTES: usize = 1024 * 1024; + +#[derive(Clone, Debug)] +pub enum MainOutput { + Stdout(Bytes), + Stderr(Bytes), + Exit(i32), +} + +impl MainOutput { + fn len(&self) -> usize { + match self { + Self::Stdout(data) | Self::Stderr(data) => data.len(), + Self::Exit(_) => 0, + } + } +} + +#[derive(Clone, Debug)] +struct SequencedOutput { + sequence: u64, + event: MainOutput, +} + +#[derive(Debug)] +struct OutputLogState { + events: VecDeque, + retained_bytes: usize, + next_sequence: u64, +} + +#[derive(Debug)] +struct OutputLog { + state: Mutex, + version: watch::Sender, + terminal_reported: AtomicBool, + terminal_reported_notify: Notify, +} + +impl OutputLog { + fn new() -> Arc { + let (version, _) = watch::channel(0); + Arc::new(Self { + state: Mutex::new(OutputLogState { + events: VecDeque::new(), + retained_bytes: 0, + next_sequence: 0, + }), + version, + terminal_reported: AtomicBool::new(false), + terminal_reported_notify: Notify::new(), + }) + } + + fn publish(&self, event: MainOutput) { + let version = { + let mut state = self.state.lock().expect("main output log lock poisoned"); + let sequence = state.next_sequence; + state.next_sequence = state + .next_sequence + .checked_add(1) + .expect("main output sequence exhausted"); + state.retained_bytes += event.len(); + state.events.push_back(SequencedOutput { sequence, event }); + while state.retained_bytes > OUTPUT_BUFFER_BYTES { + let Some(removed) = state.events.pop_front() else { + break; + }; + state.retained_bytes = state.retained_bytes.saturating_sub(removed.event.len()); + } + state.next_sequence + }; + self.version.send_replace(version); + } + + fn subscribe(self: &Arc) -> MainOutputCursor { + let version = self.version.subscribe(); + let state = self.state.lock().expect("main output log lock poisoned"); + let next_sequence = state + .events + .front() + .map_or(state.next_sequence, |retained| retained.sequence); + drop(state); + MainOutputCursor { + output: Arc::clone(self), + next_sequence, + version, + } + } +} + +#[derive(Debug)] +struct TerminalAttachmentState { + active: usize, + process_finished: bool, + expectation: AttachmentExpectation, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum AttachmentExpectation { + None, + Pending, + Satisfied, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct MainOutputLagged { + pub skipped: u64, +} + +pub struct MainOutputCursor { + output: Arc, + next_sequence: u64, + version: watch::Receiver, +} + +impl MainOutputCursor { + pub async fn recv(&mut self) -> Result { + loop { + let next = { + let state = self + .output + .state + .lock() + .expect("main output log lock poisoned"); + let oldest = state + .events + .front() + .map_or(state.next_sequence, |event| event.sequence); + if self.next_sequence < oldest { + let skipped = oldest - self.next_sequence; + self.next_sequence = oldest; + return Err(MainOutputLagged { skipped }); + } + if self.next_sequence >= state.next_sequence { + None + } else { + let offset = usize::try_from(self.next_sequence - oldest) + .expect("main output cursor offset exceeds usize"); + let event = state + .events + .get(offset) + .expect("main output cursor references retained event") + .event + .clone(); + self.next_sequence += 1; + Some(event) + } + }; + if let Some(event) = next { + return Ok(event); + } + // The log owns a sender for the cursor lifetime, so closure is not + // expected. A changed version means there is another event to read. + let _ = self.version.changed().await; + } + } +} + +enum MainInput { + Data(Vec), + Close, +} + +#[derive(Clone)] +pub(crate) struct MainInputSender { + sender: tokio::sync::mpsc::Sender, +} + +impl MainInputSender { + pub(crate) async fn send(&self, data: Vec) -> Result<(), &'static str> { + self.sender + .send(MainInput::Data(data)) + .await + .map_err(|_| "canonical process stdin closed") + } + + async fn close(&self) { + let _ = self.sender.send(MainInput::Close).await; + } +} + +pub struct MainSession { + pid: u32, + terminal: bool, + input: MainInputSender, + output: Arc, + input_owner: Mutex>, + input_closed: AtomicBool, + next_owner: AtomicU64, + pty_master: Option>, + boundary_process: Option>, + boundary_terminal: Option>, + readers_remaining: AtomicUsize, + readers_done: Notify, + finished: AtomicBool, + terminal_attachments: Mutex, + terminal_attachments_done: Notify, +} + +impl MainSession { + const REMOTE_OUTPUT_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2); + #[cfg(test)] + pub fn inert() -> Arc { + let (input, _input_rx) = tokio::sync::mpsc::channel(64); + Arc::new(Self { + pid: 1, + terminal: false, + input: MainInputSender { sender: input }, + output: OutputLog::new(), + input_owner: Mutex::new(None), + input_closed: AtomicBool::new(false), + next_owner: AtomicU64::new(1), + pty_master: None, + boundary_process: None, + boundary_terminal: None, + readers_remaining: AtomicUsize::new(0), + readers_done: Notify::new(), + finished: AtomicBool::new(false), + terminal_attachments: Mutex::new(TerminalAttachmentState { + active: 0, + process_finished: false, + expectation: AttachmentExpectation::None, + }), + terminal_attachments_done: Notify::new(), + }) + } + + #[cfg(test)] + pub fn terminal_for_test() -> (Arc, std::fs::File) { + let pty = nix::pty::openpty(None, None).expect("open test PTY"); + let slave = std::fs::File::from(pty.slave); + ( + Self::new(ProcessIo::Pty(std::fs::File::from(pty.master)), 1), + slave, + ) + } + + #[cfg(test)] + #[allow(unsafe_code)] + pub fn terminal_size_for_test(&self) -> (u16, u16) { + let master = self.pty_master.as_ref().expect("terminal PTY master"); + let mut winsize: libc::winsize = unsafe { std::mem::zeroed() }; + let result = unsafe { libc::ioctl(master.as_raw_fd(), libc::TIOCGWINSZ, &mut winsize) }; + assert_eq!(result, 0, "read terminal dimensions"); + (winsize.ws_col, winsize.ws_row) + } + + #[must_use] + pub fn new(io: ProcessIo, pid: u32) -> Arc { + let terminal = matches!(io, ProcessIo::Pty(_)); + let (input, input_rx) = tokio::sync::mpsc::channel(64); + let pty_master = match &io { + ProcessIo::Pty(master) => { + set_nonblocking(master).expect("set canonical PTY master nonblocking"); + master.try_clone().ok().map(Arc::new) + } + ProcessIo::Pipes { .. } => None, + }; + let session = Arc::new(Self { + pid, + terminal, + input: MainInputSender { sender: input }, + output: OutputLog::new(), + input_owner: Mutex::new(None), + input_closed: AtomicBool::new(false), + next_owner: AtomicU64::new(1), + pty_master, + boundary_process: None, + boundary_terminal: None, + readers_remaining: AtomicUsize::new(if terminal { 1 } else { 2 }), + readers_done: Notify::new(), + finished: AtomicBool::new(false), + terminal_attachments: Mutex::new(TerminalAttachmentState { + active: 0, + process_finished: false, + expectation: AttachmentExpectation::None, + }), + terminal_attachments_done: Notify::new(), + }); + Self::start_io(&session, io, input_rx); + session + } + + /// Build the control-side multiplexer around a boundary-owned admitted + /// process. Process lifecycle and PTY operations remain delegated to the + /// boundary process handle. + #[must_use] + pub fn from_boundary( + attachment: ProcessAttachment, + process: Arc, + ) -> Arc { + let ProcessAttachment { + stdin, + stdout, + stderr, + terminal, + } = attachment; + let terminal_mode = terminal.is_some(); + let (input, mut input_rx) = tokio::sync::mpsc::channel(64); + let session = Arc::new(Self { + pid: 0, + terminal: terminal_mode, + input: MainInputSender { sender: input }, + output: OutputLog::new(), + input_owner: Mutex::new(None), + input_closed: AtomicBool::new(false), + next_owner: AtomicU64::new(1), + pty_master: None, + boundary_process: Some(process), + boundary_terminal: terminal, + readers_remaining: AtomicUsize::new(if terminal_mode { 1 } else { 2 }), + readers_done: Notify::new(), + finished: AtomicBool::new(false), + terminal_attachments: Mutex::new(TerminalAttachmentState { + active: 0, + process_finished: false, + expectation: AttachmentExpectation::None, + }), + terminal_attachments_done: Notify::new(), + }); + let stdout_session = Arc::clone(&session); + tokio::spawn(async move { + let mut stdout = stdout; + let mut buffer = [0u8; 4096]; + loop { + match stdout.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(read) => stdout_session + .publish(MainOutput::Stdout(Bytes::copy_from_slice(&buffer[..read]))), + } + } + stdout_session.reader_finished(); + }); + if let Some(mut stderr) = stderr { + let stderr_session = Arc::clone(&session); + tokio::spawn(async move { + let mut buffer = [0u8; 4096]; + loop { + match stderr.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(read) => stderr_session + .publish(MainOutput::Stderr(Bytes::copy_from_slice(&buffer[..read]))), + } + } + stderr_session.reader_finished(); + }); + } + tokio::spawn(async move { + let mut stdin = stdin; + while let Some(input) = input_rx.recv().await { + match input { + MainInput::Data(data) => { + if stdin.write_all(&data).await.is_err() { + break; + } + let _ = stdin.flush().await; + } + MainInput::Close => break, + } + } + }); + session + } + + fn start_io( + this: &Arc, + io: ProcessIo, + mut input_rx: tokio::sync::mpsc::Receiver, + ) { + match io { + ProcessIo::Pty(master) => { + let master = Arc::new(AsyncFd::new(master).expect("register canonical PTY master")); + let reader = Arc::clone(&master); + let output = Arc::clone(this); + tokio::spawn(async move { + let mut buffer = [0u8; 4096]; + loop { + let Ok(mut ready) = reader.readable().await else { + break; + }; + match ready.try_io(|inner| { + let mut file = inner.get_ref(); + file.read(&mut buffer) + }) { + Ok(Ok(0) | Err(_)) => break, + Ok(Ok(read)) => output.publish(MainOutput::Stdout( + Bytes::copy_from_slice(&buffer[..read]), + )), + Err(_would_block) => {} + } + } + output.reader_finished(); + }); + tokio::spawn(async move { + while let Some(input) = input_rx.recv().await { + let MainInput::Data(data) = input else { + return; + }; + let mut remaining = data.as_slice(); + while !remaining.is_empty() { + let Ok(mut ready) = master.writable().await else { + return; + }; + match ready.try_io(|inner| { + let mut file = inner.get_ref(); + file.write(remaining) + }) { + Ok(Ok(0) | Err(_)) => return, + Ok(Ok(written)) => remaining = &remaining[written..], + Err(_would_block) => {} + } + } + } + }); + } + ProcessIo::Pipes { + mut stdin, + mut stdout, + mut stderr, + } => { + let stdout_session = Arc::clone(this); + tokio::spawn(async move { + let mut buffer = [0u8; 4096]; + loop { + match stdout.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(read) => { + stdout_session.publish(MainOutput::Stdout(Bytes::copy_from_slice( + &buffer[..read], + ))); + } + } + } + stdout_session.reader_finished(); + }); + let stderr_session = Arc::clone(this); + tokio::spawn(async move { + let mut buffer = [0u8; 4096]; + loop { + match stderr.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(read) => { + stderr_session.publish(MainOutput::Stderr(Bytes::copy_from_slice( + &buffer[..read], + ))); + } + } + } + stderr_session.reader_finished(); + }); + tokio::spawn(async move { + while let Some(input) = input_rx.recv().await { + match input { + MainInput::Data(data) => { + if stdin.write_all(&data).await.is_err() { + break; + } + let _ = stdin.flush().await; + } + MainInput::Close => break, + } + } + }); + } + } + } + + fn publish(&self, event: MainOutput) { + self.output.publish(event); + } + + fn reader_finished(&self) { + if self.readers_remaining.fetch_sub(1, Ordering::AcqRel) == 1 { + self.readers_done.notify_waiters(); + } + } + + /// Publish the terminal event and retain the transport only when a real + /// foreground attachment exists or the creating client declared one. + /// + /// Returns whether terminal delivery must complete before shutdown. + pub async fn finish(&self, exit_code: i32, attachment_expected: bool) -> bool { + self.wait_for_output_readers().await; + self.complete_finish(exit_code, attachment_expected) + } + + /// Finish a remotely owned process without allowing descendants that keep + /// inherited output descriptors open to block terminal publication forever. + pub async fn finish_remote(&self, exit_code: i32, attachment_expected: bool) -> bool { + self.finish_remote_with_timeout( + exit_code, + attachment_expected, + Self::REMOTE_OUTPUT_DRAIN_TIMEOUT, + ) + .await + } + + async fn finish_remote_with_timeout( + &self, + exit_code: i32, + attachment_expected: bool, + timeout: std::time::Duration, + ) -> bool { + let _ = tokio::time::timeout(timeout, self.wait_for_output_readers()).await; + self.complete_finish(exit_code, attachment_expected) + } + + async fn wait_for_output_readers(&self) { + let notified = self.readers_done.notified(); + if self.readers_remaining.load(Ordering::Acquire) != 0 { + notified.await; + } + } + + fn complete_finish(&self, exit_code: i32, attachment_expected: bool) -> bool { + let delivery_pending = { + let mut state = self + .terminal_attachments + .lock() + .expect("terminal attachment lock poisoned"); + state.process_finished = true; + state.expectation = if attachment_expected { + if state.active == 0 && state.expectation != AttachmentExpectation::Satisfied { + AttachmentExpectation::Pending + } else { + AttachmentExpectation::Satisfied + } + } else { + AttachmentExpectation::None + }; + attachment_expected || state.active != 0 + }; + self.finished.store(true, Ordering::Release); + self.publish(MainOutput::Exit(exit_code)); + delivery_pending + } + + pub fn subscribe(&self) -> MainOutputCursor { + self.output.subscribe() + } + + /// Return the bounded output sequence range currently retained for a + /// replacement supervisor. A nonzero first sequence is an explicit + /// truncation watermark rather than silent data loss. + #[must_use] + pub fn output_window(&self) -> (u64, u64, bool) { + let state = self + .output + .state + .lock() + .expect("main output log lock poisoned"); + let first_sequence = state + .events + .front() + .map_or(state.next_sequence, |event| event.sequence); + (first_sequence, state.next_sequence, first_sequence != 0) + } + + /// Wait until the gateway durably acknowledges the main-process result. + pub async fn wait_for_terminal_reported(&self) { + let notified = self.output.terminal_reported_notify.notified(); + if self.output.terminal_reported.load(Ordering::Acquire) { + return; + } + notified.await; + } + + /// Release attached clients to receive their SSH exit status after the + /// durable sandbox phase and exit code have been recorded. + pub fn mark_terminal_reported(&self) { + self.output.terminal_reported.store(true, Ordering::Release); + self.output.terminal_reported_notify.notify_waiters(); + } + + /// Register a foreground main attachment while the process is live. + pub fn begin_terminal_attachment(&self) -> Result<(), &'static str> { + let mut state = self + .terminal_attachments + .lock() + .expect("terminal attachment lock poisoned"); + if state.process_finished && state.expectation != AttachmentExpectation::Pending { + return Err("canonical main process already finished"); + } + state.active = state + .active + .checked_add(1) + .expect("terminal attachment count exhausted"); + state.expectation = AttachmentExpectation::Satisfied; + self.terminal_attachments_done.notify_waiters(); + Ok(()) + } + + /// Release a foreground main attachment after its SSH channel closes. + pub fn end_terminal_attachment(&self) { + let completed = { + let mut state = self + .terminal_attachments + .lock() + .expect("terminal attachment lock poisoned"); + debug_assert!(state.active != 0, "terminal attachment count underflow"); + if state.active == 0 { + return; + } + state.active -= 1; + state.active == 0 + }; + if completed { + self.terminal_attachments_done.notify_waiters(); + } + } + + /// Wait for the declared foreground attachment to start, then for every + /// accepted attachment to close naturally. + pub async fn wait_for_terminal_attachments(&self) { + loop { + let notified = self.terminal_attachments_done.notified(); + let complete = { + let state = self + .terminal_attachments + .lock() + .expect("terminal attachment lock poisoned"); + state.active == 0 && state.expectation != AttachmentExpectation::Pending + }; + if complete { + return; + } + notified.await; + } + } + + pub(crate) fn acquire_input(&self) -> Result<(u64, MainInputSender), &'static str> { + if self.input_closed.load(Ordering::Acquire) { + return Err("canonical process stdin closed"); + } + let mut owner = self.input_owner.lock().expect("main input lock poisoned"); + if owner.is_some() { + return Err("canonical main process already has an input owner"); + } + let id = self.next_owner.fetch_add(1, Ordering::Relaxed); + *owner = Some(id); + Ok((id, self.input.clone())) + } + + /// Acquire canonical input when it remains open. A replacement control may + /// still attach output after a prior control intentionally closed stdin. + pub(crate) fn acquire_input_if_open( + &self, + ) -> Result, &'static str> { + match self.acquire_input() { + Ok(input) => Ok(Some(input)), + Err(_) if self.input_closed.load(Ordering::Acquire) => Ok(None), + Err(error) => Err(error), + } + } + + pub(crate) fn release_input(&self, id: u64) { + let mut owner = self.input_owner.lock().expect("main input lock poisoned"); + if *owner == Some(id) { + *owner = None; + } + } + + pub(crate) async fn close_input(&self, id: u64) { + let owns_input = { + let mut owner = self.input_owner.lock().expect("main input lock poisoned"); + if *owner == Some(id) { + *owner = None; + true + } else { + false + } + }; + if owns_input && !self.input_closed.swap(true, Ordering::AcqRel) { + self.input.close().await; + } + } + + pub async fn resize(&self, columns: u32, rows: u32, pixel_width: u32, pixel_height: u32) { + if let Some(terminal) = self.boundary_terminal.as_ref() { + let _ = terminal + .resize( + u16::try_from(columns.max(1)).unwrap_or(u16::MAX), + u16::try_from(rows.max(1)).unwrap_or(u16::MAX), + ) + .await; + return; + } + let Some(master) = self.pty_master.as_ref() else { + return; + }; + let winsize = Winsize { + ws_row: u16::try_from(rows.max(1)).unwrap_or(u16::MAX), + ws_col: u16::try_from(columns.max(1)).unwrap_or(u16::MAX), + ws_xpixel: u16::try_from(pixel_width).unwrap_or(u16::MAX), + ws_ypixel: u16::try_from(pixel_height).unwrap_or(u16::MAX), + }; + #[allow(unsafe_code)] + unsafe { + libc::ioctl(master.as_raw_fd(), libc::TIOCSWINSZ, &winsize); + } + } + + pub async fn signal_group(&self, signal: nix::sys::signal::Signal) -> Result<(), String> { + if let Some(process) = self.boundary_process.as_ref() { + let signal = match signal { + nix::sys::signal::Signal::SIGHUP => BoundarySignal::Hup, + nix::sys::signal::Signal::SIGINT => BoundarySignal::Int, + nix::sys::signal::Signal::SIGKILL => BoundarySignal::Kill, + nix::sys::signal::Signal::SIGTERM => BoundarySignal::Term, + other => return Err(format!("boundary signal {other:?} is unsupported")), + }; + return process + .signal(signal) + .await + .map_err(|error| error.to_string()); + } + let pid = i32::try_from(self.pid).unwrap_or(i32::MAX); + nix::sys::signal::kill(nix::unistd::Pid::from_raw(-pid), signal) + .map_err(|error| error.to_string()) + } + + #[must_use] + pub const fn terminal(&self) -> bool { + self.terminal + } + + #[must_use] + pub fn finished(&self) -> bool { + self.finished.load(Ordering::Acquire) + } +} + +fn set_nonblocking(file: &std::fs::File) -> Result<(), nix::errno::Errno> { + let flags = fcntl(file.as_raw_fd(), FcntlArg::F_GETFL)?; + let flags = OFlag::from_bits_truncate(flags); + fcntl( + file.as_raw_fd(), + FcntlArg::F_SETFL(flags | OFlag::O_NONBLOCK), + )?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use openshell_isolation_interface::contract::{ + BackendError, BoundaryExitStatus, BoundaryInput, BoundaryOutput, + }; + + struct TestBoundaryProcess { + signals: Mutex>, + } + + #[async_trait::async_trait] + impl BoundaryProcess for TestBoundaryProcess { + async fn wait(&self) -> Result { + Ok(BoundaryExitStatus::Exited(0)) + } + + async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError> { + self.signals.lock().unwrap().push(signal); + Ok(()) + } + + async fn terminate(&self) -> Result<(), BackendError> { + Ok(()) + } + } + + struct TestBoundaryTerminal { + size: Mutex>, + } + + #[async_trait::async_trait] + impl BoundaryTerminal for TestBoundaryTerminal { + async fn resize(&self, cols: u16, rows: u16) -> Result<(), BackendError> { + *self.size.lock().unwrap() = Some((cols, rows)); + Ok(()) + } + } + + #[tokio::test] + async fn boundary_attachment_drives_main_io_signal_and_terminal() { + let (stdin, mut stdin_peer) = tokio::io::duplex(1024); + let (stdout, mut stdout_peer) = tokio::io::duplex(1024); + let process = Arc::new(TestBoundaryProcess { + signals: Mutex::new(Vec::new()), + }); + let terminal = Arc::new(TestBoundaryTerminal { + size: Mutex::new(None), + }); + let stdin: BoundaryInput = Box::new(stdin); + let stdout: BoundaryOutput = Box::new(stdout); + let attachment = ProcessAttachment { + stdin, + stdout, + stderr: None, + terminal: Some(terminal.clone()), + }; + let session = MainSession::from_boundary(attachment, process.clone()); + let mut output = session.subscribe(); + + stdout_peer.write_all(b"ready\n").await.unwrap(); + assert!(matches!( + output.recv().await.unwrap(), + MainOutput::Stdout(data) if data == b"ready\n"[..] + )); + + let (owner, input) = session.acquire_input().unwrap(); + input.send(b"hello\n".to_vec()).await.unwrap(); + let mut received = [0_u8; 6]; + stdin_peer.read_exact(&mut received).await.unwrap(); + assert_eq!(&received, b"hello\n"); + session.close_input(owner).await; + assert_eq!( + tokio::time::timeout( + std::time::Duration::from_secs(1), + stdin_peer.read(&mut received) + ) + .await + .expect("boundary stdin close timed out") + .expect("read boundary stdin EOF"), + 0 + ); + assert!(session.acquire_input().is_err()); + assert!(session.acquire_input_if_open().unwrap().is_none()); + + session.resize(120, 40, 0, 0).await; + assert_eq!(*terminal.size.lock().unwrap(), Some((120, 40))); + session + .signal_group(nix::sys::signal::Signal::SIGINT) + .await + .unwrap(); + assert_eq!(*process.signals.lock().unwrap(), vec![BoundarySignal::Int]); + } + + #[test] + fn input_lease_has_one_owner_and_can_be_reacquired() { + let session = MainSession::inert(); + let (first, _) = session.acquire_input().expect("first owner"); + assert!(session.acquire_input().is_err()); + + session.release_input(first); + let (second, _) = session.acquire_input().expect("replacement owner"); + assert_ne!(first, second); + } + + #[tokio::test] + async fn subscribers_receive_replay_then_live_output() { + let session = MainSession::inert(); + session.publish(MainOutput::Stdout(Bytes::from_static(b"before"))); + + let mut output = session.subscribe(); + assert!(matches!( + output.recv().await.expect("replayed output"), + MainOutput::Stdout(data) if data == b"before"[..] + )); + + session.publish(MainOutput::Stderr(Bytes::from_static(b"after"))); + assert!(matches!( + output.recv().await.expect("live output"), + MainOutput::Stderr(data) if data == b"after"[..] + )); + } + + #[tokio::test] + async fn finish_without_attachment_does_not_defer_shutdown() { + let session = MainSession::inert(); + assert!(!session.finish(0, false).await); + assert!(session.finished()); + assert!(session.begin_terminal_attachment().is_err()); + } + + #[tokio::test] + async fn terminal_report_acknowledgement_is_independent_from_delivery() { + let session = MainSession::inert(); + + assert!( + tokio::time::timeout( + std::time::Duration::from_millis(10), + session.wait_for_terminal_reported(), + ) + .await + .is_err(), + "draining output must not imply durable gateway persistence" + ); + + session.mark_terminal_reported(); + tokio::time::timeout( + std::time::Duration::from_secs(1), + session.wait_for_terminal_reported(), + ) + .await + .expect("durable report acknowledgement should wake waiter"); + } + + #[tokio::test] + async fn finish_waits_for_an_active_attachment_to_close_naturally() { + let session = MainSession::inert(); + session + .begin_terminal_attachment() + .expect("begin terminal attachment"); + assert!(session.finish(0, false).await); + + assert!( + tokio::time::timeout( + std::time::Duration::from_millis(10), + session.wait_for_terminal_attachments(), + ) + .await + .is_err(), + "an active attachment must keep terminal delivery open" + ); + + session.end_terminal_attachment(); + tokio::time::timeout( + std::time::Duration::from_secs(1), + session.wait_for_terminal_attachments(), + ) + .await + .expect("closing the attachment should wake the waiter"); + } + + #[tokio::test] + async fn remote_finish_bounds_output_drain_before_publishing_exit() { + let mut session = MainSession::inert(); + Arc::get_mut(&mut session) + .expect("sole test session reference") + .readers_remaining = AtomicUsize::new(1); + let mut output = session.subscribe(); + + session + .finish_remote_with_timeout(19, false, std::time::Duration::from_millis(10)) + .await; + + assert!(matches!( + output + .recv() + .await + .expect("terminal status after bounded drain"), + MainOutput::Exit(19) + )); + } + + #[tokio::test] + async fn declared_attachment_waits_for_connection_then_natural_close() { + let session = MainSession::inert(); + assert!(session.finish(0, true).await); + + assert!( + tokio::time::timeout( + std::time::Duration::from_millis(10), + session.wait_for_terminal_attachments(), + ) + .await + .is_err(), + "declared attachment must connect before delivery is complete" + ); + + session + .begin_terminal_attachment() + .expect("declared post-exit attachment"); + session.end_terminal_attachment(); + tokio::time::timeout( + std::time::Duration::from_secs(1), + session.wait_for_terminal_attachments(), + ) + .await + .expect("natural attachment close should complete delivery"); + } + + #[tokio::test] + async fn exit_is_retained_in_the_output_log() { + let session = MainSession::inert(); + let _ = session.finish(0, false).await; + + let mut output = session.subscribe(); + assert!(matches!( + output.recv().await.expect("replayed exit"), + MainOutput::Exit(0) + )); + } + + #[tokio::test] + async fn slow_subscriber_reports_evicted_events_then_resumes() { + let session = MainSession::inert(); + let mut output = session.subscribe(); + let chunk = Bytes::from(vec![0; 4096]); + for _ in 0..=(OUTPUT_BUFFER_BYTES / chunk.len()) { + session.publish(MainOutput::Stdout(chunk.clone())); + } + + let lag = output.recv().await.expect_err("oldest event was evicted"); + assert_eq!(lag.skipped, 1); + assert!(matches!( + output.recv().await.expect("resume at oldest retained event"), + MainOutput::Stdout(data) if data.len() == chunk.len() + )); + } + + #[tokio::test] + async fn terminal_pump_reads_output_and_writes_input() { + let (session, mut slave) = MainSession::terminal_for_test(); + set_nonblocking(&slave).expect("set test PTY slave nonblocking"); + let mut output = session.subscribe(); + + slave + .write_all(b"process output") + .expect("write PTY output"); + let event = tokio::time::timeout(std::time::Duration::from_secs(1), output.recv()) + .await + .expect("PTY output timed out") + .expect("PTY output was retained"); + assert!(matches!( + event, + MainOutput::Stdout(data) if data == b"process output"[..] + )); + + let (owner, input) = session.acquire_input().expect("acquire PTY input"); + input + .send(b"client input\n".to_vec()) + .await + .expect("queue PTY input"); + let mut received = [0; 64]; + let read = tokio::time::timeout(std::time::Duration::from_secs(1), async { + loop { + match slave.read(&mut received) { + Ok(read) => break read, + Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => { + tokio::task::yield_now().await; + } + Err(error) => panic!("read PTY input: {error}"), + } + } + }) + .await + .expect("PTY input timed out"); + assert_eq!(&received[..read], b"client input\n"); + session.release_input(owner); + } +} diff --git a/crates/openshell-sandbox/src/managed_children.rs b/crates/openshell-sandbox/src/managed_children.rs new file mode 100644 index 0000000000..28b02bd946 --- /dev/null +++ b/crates/openshell-sandbox/src/managed_children.rs @@ -0,0 +1,197 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Process-wide tracker for sandbox-managed child PIDs. +//! +//! The supervisor spawns several long-lived children (the entrypoint, SSH +//! sessions). Each registers its PID here on spawn and removes it on exit so +//! the orchestrator's `SIGCHLD` reaper can distinguish supervised processes +//! from incidental zombies. + +#![cfg(target_os = "linux")] + +use std::collections::{HashMap, HashSet}; +use std::io; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{LazyLock, Mutex, MutexGuard}; +use std::time::Duration; + +static MANAGED_CHILDREN: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); +static NEXT_GENERATION: AtomicU64 = AtomicU64::new(1); + +/// Identity of one registry entry. The generation prevents an old waiter from +/// removing a newer child that reused the same numeric PID after reap. +#[derive(Clone, Copy)] +pub struct ManagedChild { + pid: i32, + generation: u64, +} + +/// Exclusive access to the managed-child registry. +/// +/// A process spawner holds this guard from immediately before `spawn` or +/// `fork` until the returned PID is registered. The orphan reaper holds the +/// same guard while deciding whether to reap an exited child. This closes the +/// otherwise unavoidable window in which a fast-exiting managed child exists +/// but its PID has not yet been published. +pub struct RegistryGuard(MutexGuard<'static, HashMap>); + +impl RegistryGuard { + /// Add a newly spawned managed child. + pub fn register(&mut self, pid: u32) -> Option { + let Ok(pid) = i32::try_from(pid) else { + return None; + }; + if pid <= 0 { + return None; + } + let generation = NEXT_GENERATION.fetch_add(1, Ordering::Relaxed); + self.0.insert(pid, generation); + Some(ManagedChild { pid, generation }) + } + + /// Return whether the PID belongs to an explicit waiter. + #[must_use] + pub fn contains(&self, pid: i32) -> bool { + self.0.contains_key(&pid) + } +} + +/// Lock the registry for an atomic spawn-and-register or inspect-and-reap +/// operation. +pub fn lock() -> RegistryGuard { + RegistryGuard( + MANAGED_CHILDREN + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner), + ) +} + +/// Register a child and return the generation-bearing removal token. +pub fn register(pid: u32) -> Option { + lock().register(pid) +} + +/// Remove exactly this supervised-child registration. A newer registration +/// for a reused PID is preserved. +pub fn unregister(child: ManagedChild) { + if let Ok(mut children) = MANAGED_CHILDREN.lock() + && children.get(&child.pid) == Some(&child.generation) + { + children.remove(&child.pid); + } +} + +/// Return `true` if `pid` is currently in the supervised-child set. +#[must_use] +pub fn is_managed(pid: i32) -> bool { + lock().contains(pid) +} + +/// Wait until a managed child is terminal without reaping it. +/// +/// Keeping the child as a zombie prevents PID/process-group reuse until the +/// owner publishes terminal state and performs the final wait. +pub fn wait_until_terminal(pid: u32) -> io::Result<()> { + use nix::sys::wait::{Id, WaitPidFlag, waitid}; + let pid = i32::try_from(pid) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "PID out of range"))?; + waitid( + Id::Pid(nix::unistd::Pid::from_raw(pid)), + WaitPidFlag::WEXITED | WaitPidFlag::WNOWAIT, + ) + .map(|_| ()) + .map_err(io::Error::other) +} + +/// Start the background reaper used when `openshell-sandbox` owns PID 1. +/// +/// Explicitly managed children remain owned by their normal waiters. Only +/// unregistered children adopted from the workload process tree are reaped. +pub fn start_orphan_reaper() -> io::Result<()> { + std::thread::Builder::new() + .name("openshell-orphan-reaper".to_string()) + .spawn(|| { + loop { + if let Err(error) = reap_unmanaged_children_once() { + tracing::debug!(%error, "orphan reaper scan failed"); + } + std::thread::sleep(Duration::from_millis(50)); + } + }) + .map(|_| ()) +} + +fn reap_unmanaged_children_once() -> io::Result { + use nix::sys::wait::{WaitPidFlag, WaitStatus, waitpid}; + + let children = direct_child_pids()?; + let registry = lock(); + let mut reaped = 0; + for pid in children { + if registry.contains(pid) { + continue; + } + match waitpid(nix::unistd::Pid::from_raw(pid), Some(WaitPidFlag::WNOHANG)) { + Ok(WaitStatus::StillAlive) + | Err(nix::errno::Errno::ECHILD | nix::errno::Errno::ESRCH) => {} + Ok(_) => reaped += 1, + Err(error) => return Err(io::Error::other(error)), + } + } + Ok(reaped) +} + +fn direct_child_pids() -> io::Result> { + let mut children = HashSet::new(); + for task in std::fs::read_dir("/proc/self/task")? { + let task = task?; + let path = task.path().join("children"); + let contents = match std::fs::read_to_string(path) { + Ok(contents) => contents, + Err(error) if error.kind() == io::ErrorKind::NotFound => continue, + Err(error) => return Err(error), + }; + children.extend( + contents + .split_ascii_whitespace() + .filter_map(|value| value.parse::().ok()) + .filter(|pid| *pid > 0), + ); + } + Ok(children) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn stale_unregister_preserves_reused_pid_registration() { + let pid = i32::MAX as u32; + let first = lock().register(pid).expect("first registration"); + let second = lock().register(pid).expect("replacement registration"); + + unregister(first); + assert!(is_managed(i32::try_from(pid).expect("test pid"))); + + unregister(second); + assert!(!is_managed(i32::try_from(pid).expect("test pid"))); + } + + #[test] + fn child_pid_parser_observes_a_live_child() { + let mut child = std::process::Command::new("sleep") + .arg("30") + .spawn() + .expect("spawn child"); + assert!( + direct_child_pids() + .expect("read direct children") + .contains(&i32::try_from(child.id()).expect("child PID")) + ); + child.kill().expect("kill child"); + child.wait().expect("wait for child"); + } +} diff --git a/crates/openshell-sandbox/src/metadata_server.rs b/crates/openshell-sandbox/src/metadata_server.rs deleted file mode 100644 index dcfe3e439a..0000000000 --- a/crates/openshell-sandbox/src/metadata_server.rs +++ /dev/null @@ -1,231 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Loopback HTTP server for cloud metadata emulators. -//! -//! Binds a TCP listener inside the sandbox network namespace so that -//! cloud SDKs that bypass `HTTP_PROXY` (e.g. Go's -//! `cloud.google.com/go/compute/metadata`) can reach the emulator via -//! direct TCP. -//! -//! The server is generic over [`MetadataHandler`] — any cloud provider -//! that needs an instance metadata emulator can implement the trait. - -use miette::Result; -use openshell_core::net::set_tcp_nodelay_best_effort; -use std::future::Future; -use std::net::SocketAddr; -use std::sync::Arc; -use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; -use tokio::net::TcpListener; -use tokio::sync::{Semaphore, oneshot}; -use tracing::{debug, warn}; - -const MAX_REQUEST_BYTES: usize = 4096; -const MAX_CONCURRENT_CONNECTIONS: usize = 32; - -/// Handler for cloud metadata HTTP requests. -/// -/// Implementors receive the parsed HTTP method, path, raw request bytes, -/// and a bidirectional stream to write the response. The handler owns the -/// response format (status, headers, body) — the server only does TCP -/// accept and HTTP request-line parsing. -pub trait MetadataHandler: Send + Sync + 'static { - fn handle( - &self, - method: &str, - path: &str, - request: &[u8], - stream: &mut S, - ) -> impl Future> + Send; -} - -/// Bind a TCP listener inside the sandbox network namespace. -/// -/// Run the metadata server accept loop. -/// -/// Signals `ready_tx` with the bound address before entering the loop. -/// Returns when the listener encounters a fatal error or the runtime shuts down. -pub async fn run( - listener: TcpListener, - handler: H, - ready_tx: oneshot::Sender, -) { - let local_addr = match listener.local_addr() { - Ok(addr) => addr, - Err(e) => { - warn!("metadata server failed to get local address: {e}"); - return; - } - }; - - let _ = ready_tx.send(local_addr); - - let handler = Arc::new(handler); - let semaphore = Arc::new(Semaphore::new(MAX_CONCURRENT_CONNECTIONS)); - - loop { - let Ok(permit) = semaphore.clone().acquire_owned().await else { - break; - }; - - match listener.accept().await { - Ok((stream, _addr)) => { - // Small-request IMDS-style endpoint an agent polls for - // credentials/identity — disable Nagle to avoid delayed-ACK stalls. - set_tcp_nodelay_best_effort(&stream); - let handler = handler.clone(); - tokio::spawn(async move { - if let Err(e) = handle_connection(handler.as_ref(), stream).await { - debug!("metadata server connection error: {e}"); - } - drop(permit); - }); - } - Err(e) => { - warn!("metadata server accept error: {e}"); - tokio::time::sleep(std::time::Duration::from_millis(50)).await; - } - } - } -} - -const READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); - -async fn handle_connection( - handler: &H, - mut stream: tokio::net::TcpStream, -) -> Result<()> { - let mut buf = vec![0u8; MAX_REQUEST_BYTES]; - let mut used = 0; - let deadline = tokio::time::sleep(READ_TIMEOUT); - tokio::pin!(deadline); - loop { - tokio::select! { - result = stream.read(&mut buf[used..]) => { - let n = result.map_err(|e| miette::miette!("{e}"))?; - if n == 0 { - return Ok(()); - } - used += n; - if buf[..used].windows(4).any(|w| w == b"\r\n\r\n") { - break; - } - if used >= buf.len() { - let _ = stream - .write_all(b"HTTP/1.1 413 Request Entity Too Large\r\nContent-Length: 0\r\n\r\n") - .await; - return Ok(()); - } - } - () = &mut deadline => { - return Ok(()); - } - } - } - let request = String::from_utf8_lossy(&buf[..used]); - let request_line = request.split("\r\n").next().unwrap_or(""); - let mut parts = request_line.split_whitespace(); - let method = parts.next().unwrap_or(""); - let path = parts.next().unwrap_or("/"); - - tokio::time::timeout( - READ_TIMEOUT, - handler.handle(method, path, &buf[..used], &mut stream), - ) - .await - .unwrap_or_else(|_| { - debug!(method, path, "metadata handler timed out"); - Ok(()) - }) -} - -#[cfg(test)] -mod tests { - use super::*; - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - use tokio::sync::mpsc; - - struct RecordingHandler { - requests: mpsc::UnboundedSender<(String, String)>, - } - - impl MetadataHandler for RecordingHandler { - async fn handle( - &self, - method: &str, - path: &str, - _request: &[u8], - stream: &mut S, - ) -> Result<()> { - self.requests - .send((method.to_string(), path.to_string())) - .unwrap(); - stream - .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok") - .await - .map_err(|error| miette::miette!("{error}"))?; - Ok(()) - } - } - - async fn connection_pair() -> (tokio::net::TcpStream, tokio::net::TcpStream) { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let client = tokio::net::TcpStream::connect(listener.local_addr().unwrap()) - .await - .unwrap(); - let (server, _) = listener.accept().await.unwrap(); - (client, server) - } - - #[tokio::test] - async fn metadata_loopback_dispatches_method_path_and_response() { - let (requests_tx, mut requests_rx) = mpsc::unbounded_channel(); - let handler = RecordingHandler { - requests: requests_tx, - }; - let (mut client, server) = connection_pair().await; - let server_task = tokio::spawn(async move { handle_connection(&handler, server).await }); - - client - .write_all(b"GET /computeMetadata/v1/instance HTTP/1.1\r\nHost: metadata\r\n\r\n") - .await - .unwrap(); - let mut response = Vec::new(); - client.read_to_end(&mut response).await.unwrap(); - server_task.await.unwrap().unwrap(); - - assert_eq!( - requests_rx.try_recv().unwrap(), - ( - "GET".to_string(), - "/computeMetadata/v1/instance".to_string() - ) - ); - assert_eq!(response, b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok"); - } - - #[tokio::test] - async fn metadata_loopback_rejects_oversized_headers_before_handler() { - let (requests_tx, mut requests_rx) = mpsc::unbounded_channel(); - let handler = RecordingHandler { - requests: requests_tx, - }; - let (mut client, server) = connection_pair().await; - let server_task = tokio::spawn(async move { handle_connection(&handler, server).await }); - - client - .write_all(&vec![b'x'; MAX_REQUEST_BYTES]) - .await - .unwrap(); - let mut response = Vec::new(); - client.read_to_end(&mut response).await.unwrap(); - server_task.await.unwrap().unwrap(); - - assert_eq!( - response, - b"HTTP/1.1 413 Request Entity Too Large\r\nContent-Length: 0\r\n\r\n" - ); - assert!(requests_rx.try_recv().is_err()); - } -} diff --git a/crates/openshell-sandbox/src/network_broker.rs b/crates/openshell-sandbox/src/network_broker.rs new file mode 100644 index 0000000000..6b01cc0967 --- /dev/null +++ b/crates/openshell-sandbox/src/network_broker.rs @@ -0,0 +1,2357 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Seccomp-notification broker owned by the in-workload sandbox. + +#![allow(unsafe_code)] + +use std::collections::HashMap; +use std::io; +use std::mem::size_of; +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr, TcpListener, TcpStream, UdpSocket}; +use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd, RawFd}; +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use openshell_binary_identity::ProcfsIdentityResolver; +use openshell_isolation_interface::contract::{ + BinaryIdentity, DnsTransport, NetworkSocketMetadata, ResolveError, TcpOpenDecision, + TcpOpenDenial, +}; +use openshell_isolation_interface::linux::seccomp_notify::{Notification, NotificationListener}; +use openshell_isolation_interface::linux::socket_registry::{ + InetFamily, InetKind, SocketIdentity, SocketMetadata, SocketRegistry, SocketState, +}; +use openshell_isolation_interface::linux::task_memory; +use tokio::sync::{mpsc, oneshot}; + +const SOCKET_CAPACITY: usize = 4_096; +const OPEN_QUEUE_CAPACITY: usize = 256; +const ACCEPT_WORKER_CAPACITY: usize = 64; +const DNS_QUEUE_CAPACITY: usize = 256; +const DNS_WORKER_CAPACITY: usize = 256; +const DNS_QUERY_TIMEOUT: Duration = Duration::from_secs(10); +const ACCEPT_POLL_INTERVAL: Duration = Duration::from_millis(250); +const DNS_RELAY_ADDRESS: SocketAddr = SocketAddr::V4(std::net::SocketAddrV4::new( + Ipv4Addr::new(127, 0, 0, 53), + 53, +)); +const RELAY_CONNECT_TIMEOUT: Duration = Duration::from_secs(5); +const NETWORK_DECISION_TIMEOUT: Duration = Duration::from_secs(30); + +#[derive(Debug)] +struct PendingOpenSlot(Arc); + +impl Drop for PendingOpenSlot { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::AcqRel); + } +} + +#[derive(Debug)] +struct PendingDnsSlot(Arc); + +impl Drop for PendingDnsSlot { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::AcqRel); + } +} + +fn acquire_pending_dns_slot(active: &Arc) -> io::Result { + active + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { + (current < DNS_WORKER_CAPACITY).then_some(current + 1) + }) + .map(|_| PendingDnsSlot(Arc::clone(active))) + .map_err(|_| io::Error::from_raw_os_error(libc::EAGAIN)) +} + +struct PendingAcceptSlot { + active: Arc, +} + +impl Drop for PendingAcceptSlot { + fn drop(&mut self) { + self.active.fetch_sub(1, Ordering::AcqRel); + } +} + +fn acquire_pending_accept_slot(active: &Arc) -> io::Result { + active + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { + (current < ACCEPT_WORKER_CAPACITY).then_some(current + 1) + }) + .map_err(|_| io::Error::from_raw_os_error(libc::EAGAIN))?; + Ok(PendingAcceptSlot { + active: Arc::clone(active), + }) +} + +fn acquire_pending_open_slot(active: &Arc) -> io::Result { + active + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { + (current < OPEN_QUEUE_CAPACITY).then_some(current + 1) + }) + .map(|_| PendingOpenSlot(Arc::clone(active))) + .map_err(|_| io::Error::from_raw_os_error(libc::EAGAIN)) +} + +/// One external TCP open blocked in `connect(2)` until the supervisor decides. +pub struct PendingTcpOpen { + pub(crate) destination: SocketAddr, + pub(crate) identity: Result, + pub(crate) socket: NetworkSocketMetadata, + pub(crate) notification_to_queue: Duration, + pub(crate) queued_at: Instant, + decision: std::sync::mpsc::SyncSender, + relay: oneshot::Receiver>, +} + +impl PendingTcpOpen { + pub(crate) async fn complete(self, decision: TcpOpenDecision) -> io::Result> { + self.decision + .send(decision) + .map_err(|_| io::Error::new(io::ErrorKind::BrokenPipe, "network broker stopped"))?; + if matches!(decision, TcpOpenDecision::Denied(_)) { + return Ok(None); + } + self.relay + .await + .map_err(|_| { + io::Error::new( + io::ErrorKind::BrokenPipe, + "network relay setup was cancelled", + ) + })? + .map(Some) + } +} + +/// One DNS exchange received by the exact sandbox-local resolver endpoint. +pub struct PendingDnsQuery { + pub(crate) request: Vec, + pub(crate) transport: DnsTransport, + pub(crate) identity: Result, + pub(crate) notification_to_queue: Duration, + pub(crate) queued_at: Instant, + response: std::sync::mpsc::SyncSender>>, +} + +impl PendingDnsQuery { + pub(crate) fn complete(self, response: io::Result>) -> io::Result<()> { + self.response + .send(response) + .map_err(|_| io::Error::new(io::ErrorKind::BrokenPipe, "DNS relay stopped")) + } +} + +#[derive(Clone)] +struct DnsRelay { + address: SocketAddr, + udp_admissions: Arc>>, + tcp_admissions: Arc>>, +} + +fn dns_sender_identity() -> Result { + // Native writes can come from an inheriting process or after execve. A + // connect-time identity (or a later procfs holder scan) cannot identify + // the sender of an already queued query. Never assert that it can. + Err(ResolveError::Failed( + "DNS sender identity is unavailable for kernel-driven socket writes".to_string(), + )) +} + +fn register_dns_socket( + admissions: &Mutex>, + peer: SocketAddr, + identity: SocketIdentity, +) -> io::Result<()> { + let mut admissions = lock(admissions); + if admissions.len() >= SOCKET_CAPACITY && !admissions.contains_key(&peer) { + let installed = + openshell_isolation_interface::linux::proc_fd::installed_socket_inodes_excluding( + std::process::id(), + )?; + admissions.retain(|_, socket| installed.contains(&socket.inode)); + if admissions.len() >= SOCKET_CAPACITY { + return Err(io::Error::from_raw_os_error(libc::EMFILE)); + } + } + admissions.insert(peer, identity); + Ok(()) +} + +#[derive(Clone)] +struct NotificationQueues { + protected_control_port: Option, + accept_registrar: crate::accept_interrupt::AcceptRegistrar, + identity_resolver: ProcfsIdentityResolver, + pending: mpsc::Sender, + dns_relay: DnsRelay, + active_opens: Arc, + active_accepts: Arc, + decision_timeout: Duration, +} + +/// Live broker handle retained by the sandbox boundary. +#[derive(Clone)] +pub struct NetworkBroker { + _accept_monitor: Arc, + pending: Arc>>, + pending_dns: Arc>>, + dns_address: SocketAddr, + healthy: Arc, +} + +impl NetworkBroker { + pub(crate) fn start( + listener: NotificationListener, + protected_control_port: Option, + ) -> io::Result { + Self::start_with_dns_address(listener, DNS_RELAY_ADDRESS, protected_control_port) + } + + #[cfg(any(test, feature = "perf-harness"))] + pub(crate) fn start_for_test(listener: NotificationListener) -> io::Result { + Self::start_with_dns_address( + listener, + SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0), + None, + ) + } + + fn start_with_dns_address( + listener: NotificationListener, + dns_address: SocketAddr, + protected_control_port: Option, + ) -> io::Result { + Self::start_with_decision_timeout( + listener, + dns_address, + protected_control_port, + NETWORK_DECISION_TIMEOUT, + ) + } + + fn start_with_decision_timeout( + listener: NotificationListener, + dns_address: SocketAddr, + protected_control_port: Option, + decision_timeout: Duration, + ) -> io::Result { + let listener = Arc::new(listener); + let monitor_listener = listener.clone(); + let accept_monitor = Arc::new(crate::accept_interrupt::AcceptMonitor::start(move |id| { + monitor_listener.validate_id(id).is_ok() + })?); + let (pending_tx, pending_rx) = mpsc::channel(OPEN_QUEUE_CAPACITY); + let (pending_dns_tx, pending_dns_rx) = mpsc::channel(DNS_QUEUE_CAPACITY); + let registry = Arc::new(Mutex::new(SocketRegistry::new(1, SOCKET_CAPACITY)?)); + let active_opens = Arc::new(AtomicUsize::new(0)); + let active_accepts = Arc::new(AtomicUsize::new(0)); + let dns_relay = start_dns_relay(dns_address, pending_dns_tx)?; + let dns_address = dns_relay.address; + let queues = NotificationQueues { + protected_control_port, + accept_registrar: accept_monitor.registrar(), + identity_resolver: ProcfsIdentityResolver::for_pid_namespace(), + pending: pending_tx, + dns_relay, + active_opens, + active_accepts, + decision_timeout, + }; + let healthy = Arc::new(AtomicBool::new(true)); + let broker_healthy = healthy.clone(); + std::thread::Builder::new() + .name("openshell-network-broker".to_string()) + .spawn(move || { + while broker_healthy.load(Ordering::Acquire) { + let notification = match listener.receive() { + Ok(notification) => notification, + Err(error) if error.kind() == io::ErrorKind::Interrupted => continue, + Err(error) => { + tracing::error!(%error, "sandbox network broker listener failed"); + broker_healthy.store(false, Ordering::Release); + break; + } + }; + if let Err(error) = dispatch_notification( + Arc::clone(®istry), + Arc::clone(&listener), + notification, + queues.clone(), + ) { + tracing::warn!( + tid = notification.tid, + syscall = notification.syscall, + %error, + "sandbox network notification denied (tid={}, syscall={}): {error}", + notification.tid, + notification.syscall + ); + let _ = listener.respond_errno(notification.id, error_to_errno(&error)); + } + } + }) + .map_err(|error| io::Error::other(format!("start network broker: {error}")))?; + Ok(Self { + _accept_monitor: accept_monitor, + pending: Arc::new(tokio::sync::Mutex::new(pending_rx)), + pending_dns: Arc::new(tokio::sync::Mutex::new(pending_dns_rx)), + dns_address, + healthy, + }) + } + + pub(crate) async fn accept(&self) -> io::Result { + self.pending + .lock() + .await + .recv() + .await + .ok_or_else(|| io::Error::new(io::ErrorKind::BrokenPipe, "network broker queue closed")) + } + + pub(crate) async fn accept_dns(&self) -> io::Result { + self.pending_dns + .lock() + .await + .recv() + .await + .ok_or_else(|| io::Error::new(io::ErrorKind::BrokenPipe, "DNS broker queue closed")) + } + + #[cfg(test)] + pub(crate) fn dns_address(&self) -> SocketAddr { + self.dns_address + } + + pub(crate) fn confirm_healthy(&self) -> io::Result<()> { + if self.healthy.load(Ordering::Acquire) && self.dns_address.port() != 0 { + Ok(()) + } else { + Err(io::Error::new( + io::ErrorKind::BrokenPipe, + "network broker is not running", + )) + } + } +} + +fn start_dns_relay( + address: SocketAddr, + pending: mpsc::Sender, +) -> io::Result { + let (udp, tcp, address) = bind_dns_relay_sockets(address)?; + let udp_admissions = Arc::new(Mutex::new(HashMap::new())); + let tcp_admissions = Arc::new(Mutex::new(HashMap::new())); + let active_workers = Arc::new(AtomicUsize::new(0)); + let relay = DnsRelay { + address, + udp_admissions: Arc::clone(&udp_admissions), + tcp_admissions: Arc::clone(&tcp_admissions), + }; + + let udp_active_workers = Arc::clone(&active_workers); + let udp_pending = pending.clone(); + std::thread::Builder::new() + .name("openshell-dns-udp".to_string()) + .spawn(move || { + let mut request = vec![0_u8; u16::MAX as usize]; + while let Ok((length, peer)) = udp.recv_from(&mut request) { + if !lock(&udp_admissions).contains_key(&peer) { + tracing::warn!(%peer, "dropping DNS datagram from unregistered socket"); + continue; + } + let Ok(worker_slot) = acquire_pending_dns_slot(&udp_active_workers) else { + tracing::warn!(%peer, "dropping DNS datagram because the worker quota is full"); + continue; + }; + let (response_tx, response_rx) = std::sync::mpsc::sync_channel(1); + let query = PendingDnsQuery { + request: request[..length].to_vec(), + transport: DnsTransport::Udp, + identity: dns_sender_identity(), + notification_to_queue: Duration::ZERO, + queued_at: Instant::now(), + response: response_tx, + }; + if pending_try_send(&udp_pending, query).is_err() { + continue; + } + let Ok(udp_response) = udp.try_clone() else { + continue; + }; + let _ = std::thread::Builder::new() + .name("openshell-dns-udp-query".to_string()) + .spawn(move || { + let _worker_slot = worker_slot; + if let Ok(Ok(response)) = response_rx.recv_timeout(DNS_QUERY_TIMEOUT) { + let _ = udp_response.send_to(&response, peer); + } + }); + } + }) + .map_err(|error| io::Error::other(format!("start UDP DNS relay: {error}")))?; + + let tcp_active_workers = active_workers; + std::thread::Builder::new() + .name("openshell-dns-tcp".to_string()) + .spawn(move || { + for accepted in tcp.incoming() { + let Ok((stream, peer)) = accepted.and_then(|stream| { + let peer = stream.peer_addr()?; + Ok((stream, peer)) + }) else { + break; + }; + // One admission authorizes exactly one accepted TCP stream; + // no peer mapping needs to outlive this accept. + if lock(&tcp_admissions).remove(&peer).is_none() { + tracing::warn!(%peer, "dropping DNS stream from unregistered socket"); + continue; + } + let Ok(worker_slot) = acquire_pending_dns_slot(&tcp_active_workers) else { + tracing::warn!(%peer, "dropping DNS stream because the worker quota is full"); + continue; + }; + let tcp_pending = pending.clone(); + let _ = std::thread::Builder::new() + .name("openshell-dns-tcp-query".to_string()) + .spawn(move || { + let _worker_slot = worker_slot; + serve_dns_tcp(stream, tcp_pending); + }); + } + }) + .map_err(|error| io::Error::other(format!("start TCP DNS relay: {error}")))?; + Ok(relay) +} + +fn bind_dns_relay_sockets(address: SocketAddr) -> io::Result<(UdpSocket, TcpListener, SocketAddr)> { + const EPHEMERAL_BIND_ATTEMPTS: usize = 32; + + if address.port() != 0 { + let udp = UdpSocket::bind(address)?; + let tcp = TcpListener::bind(address)?; + return Ok((udp, tcp, address)); + } + + // TCP and UDP have independent ephemeral-port allocators. The port picked + // by the first bind can therefore already be occupied by the other + // protocol, especially while the test suite starts several brokers in + // parallel. Retry the pair rather than treating that collision as an + // unavailable network broker. + for _ in 0..EPHEMERAL_BIND_ATTEMPTS { + let udp = UdpSocket::bind(address)?; + let selected = udp.local_addr()?; + match TcpListener::bind(selected) { + Ok(tcp) => return Ok((udp, tcp, selected)), + Err(error) if error.kind() == io::ErrorKind::AddrInUse => {} + Err(error) => return Err(error), + } + } + + Err(io::Error::new( + io::ErrorKind::AddrInUse, + "could not reserve a shared ephemeral TCP/UDP DNS relay port", + )) +} + +fn pending_try_send( + pending: &mpsc::Sender, + query: PendingDnsQuery, +) -> Result<(), ()> { + pending.try_send(query).map_err(|error| { + tracing::warn!(%error, "dropping DNS query because mediation queue is unavailable"); + }) +} + +fn serve_dns_tcp(mut stream: TcpStream, pending: mpsc::Sender) { + use std::io::{Read as _, Write as _}; + + let _ = stream.set_read_timeout(Some(DNS_QUERY_TIMEOUT)); + let _ = stream.set_write_timeout(Some(DNS_QUERY_TIMEOUT)); + loop { + let mut length = [0_u8; 2]; + if stream.read_exact(&mut length).is_err() { + return; + } + let message_length = usize::from(u16::from_be_bytes(length)); + let mut request = Vec::with_capacity(message_length + 2); + request.extend_from_slice(&length); + request.resize(message_length + 2, 0); + if stream.read_exact(&mut request[2..]).is_err() { + return; + } + let (response_tx, response_rx) = std::sync::mpsc::sync_channel(1); + let query = PendingDnsQuery { + request, + transport: DnsTransport::Tcp, + identity: dns_sender_identity(), + notification_to_queue: Duration::ZERO, + queued_at: Instant::now(), + response: response_tx, + }; + if pending_try_send(&pending, query).is_err() { + return; + } + let Ok(Ok(response)) = response_rx.recv_timeout(DNS_QUERY_TIMEOUT) else { + return; + }; + if stream.write_all(&response).is_err() { + return; + } + } +} + +fn dispatch_notification( + registry: Arc>, + listener: Arc, + notification: Notification, + queues: NotificationQueues, +) -> io::Result<()> { + let syscall = i64::from(notification.syscall); + if matches!(syscall, libc::SYS_kill | libc::SYS_rt_sigqueueinfo) { + return openshell_isolation_interface::linux::process_signal::mediate_process_signal( + &listener, + notification, + std::process::id(), + ); + } + if syscall == libc::SYS_tkill { + return openshell_isolation_interface::linux::process_signal::mediate_thread_signal( + &listener, + notification, + std::process::id(), + ); + } + if syscall == libc::SYS_socket { + return create_socket(®istry, &listener, notification); + } + if syscall == libc::SYS_connect { + return connect_socket(registry, listener, notification, queues); + } + if syscall == libc::SYS_bind { + return bind_socket(®istry, &listener, notification); + } + if syscall == libc::SYS_listen { + return listen_socket(®istry, &listener, notification); + } + if matches!(syscall, libc::SYS_accept | libc::SYS_accept4) { + return accept_socket( + registry, + listener, + notification, + queues.active_accepts, + queues.accept_registrar, + ); + } + if matches!( + syscall, + libc::SYS_sendto | libc::SYS_sendmsg | libc::SYS_sendmmsg + ) { + return classify_send(®istry, &listener, notification, &queues.dns_relay); + } + if syscall == libc::SYS_getpeername { + return get_peer_name(®istry, &listener, notification); + } + if syscall == libc::SYS_setsockopt { + let level = i32::try_from(notification.args[1]) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; + let option = i32::try_from(notification.args[2]) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; + if (level == libc::IPPROTO_TCP && option == libc::TCP_FASTOPEN_CONNECT) + || (level == libc::IPPROTO_IPV6 && option == libc::IPV6_ADDRFORM) + { + return Err(io::Error::from_raw_os_error(libc::EPERM)); + } + return listener.respond_continue(notification.id); + } + Err(io::Error::from_raw_os_error(libc::EPERM)) +} + +fn create_socket( + registry: &Mutex, + listener: &NotificationListener, + notification: Notification, +) -> io::Result<()> { + let domain = i32::try_from(notification.args[0]) + .map_err(|_| io::Error::from_raw_os_error(libc::EAFNOSUPPORT))?; + if !matches!(domain, libc::AF_INET | libc::AF_INET6) { + return listener.respond_continue(notification.id); + } + let raw_kind = i32::try_from(notification.args[1]) + .map_err(|_| io::Error::from_raw_os_error(libc::EPROTONOSUPPORT))?; + let protocol = i32::try_from(notification.args[2]) + .map_err(|_| io::Error::from_raw_os_error(libc::EPROTONOSUPPORT))?; + let base_kind = raw_kind & !(libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK); + let kind = match (base_kind, protocol) { + (libc::SOCK_STREAM, 0 | libc::IPPROTO_TCP) => InetKind::Tcp, + (libc::SOCK_DGRAM, 0 | libc::IPPROTO_UDP) => InetKind::DnsUdp, + _ => return Err(io::Error::from_raw_os_error(libc::EPROTONOSUPPORT)), + }; + let family = if domain == libc::AF_INET { + InetFamily::V4 + } else { + InetFamily::V6 + }; + // SAFETY: arguments were reduced to the supported native INET matrix. A + // successful call returns one newly owned descriptor. + let mut source = unsafe { libc::socket(domain, raw_kind, protocol) }; + if source < 0 && io::Error::last_os_error().raw_os_error() == Some(libc::EMFILE) { + collect_closed_socket_entries(registry)?; + // SAFETY: same validated native INET socket creation after reclaiming + // broker-held descriptors for closed workload sockets. + source = unsafe { libc::socket(domain, raw_kind, protocol) }; + } + if source < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: successful socket returned one owned descriptor. + let source = unsafe { OwnedFd::from_raw_fd(source) }; + let metadata = SocketMetadata { + family, + kind, + close_on_exec: raw_kind & libc::SOCK_CLOEXEC != 0, + nonblocking: raw_kind & libc::SOCK_NONBLOCK != 0, + creator_generation: u64::from(notification.tid), + }; + let mut registry = lock(registry); + if registry.is_full() { + collect_closed_socket_entries_locked(&mut registry)?; + } + let tentative = registry.stage(source, metadata)?; + listener.add_fd_and_send( + notification.id, + tentative.source_fd(), + metadata.close_on_exec, + )?; + registry.commit(tentative)?; + Ok(()) +} + +fn reject_protected_control_destination( + destination: SocketAddr, + protected_port: Option, +) -> io::Result<()> { + // Reserve the listener's port across loopback aliases, IPv4-mapped IPv6, + // and wildcard Pod listeners. A workload must never reach its control + // endpoint, including through a supervisor-authorized external relay. + if protected_port == Some(destination.port()) { + return Err(io::Error::from_raw_os_error(libc::EACCES)); + } + Ok(()) +} + +fn connect_socket( + registry: Arc>, + listener: Arc, + notification: Notification, + queues: NotificationQueues, +) -> io::Result<()> { + let NotificationQueues { + pending, + dns_relay, + active_opens, + identity_resolver, + decision_timeout, + protected_control_port, + .. + } = queues; + let notification_started = Instant::now(); + let fd = raw_fd(notification.args[0])?; + let address_family = + read_socket_family(notification.tid, notification.args[1], notification.args[2])?; + if !matches!(address_family, libc::AF_INET | libc::AF_INET6) { + if address_family == libc::AF_UNSPEC { + let mut registry = lock(®istry); + if let Ok(entry) = registry.resolve_mut(notification.tid, fd) + && entry.metadata().kind == InetKind::DnsUdp + && matches!( + entry.state(), + SocketState::Created | SocketState::Bound { .. } + ) + { + // Address-selection implementations disconnect a temporary + // UDP route-probe socket with AF_UNSPEC before trying the next + // candidate. The probe below never connects the real OFD, so + // this is an idempotent no-op rather than a kernel CONTINUE. + return listener.respond_value(notification.id, 0); + } + } + if lock(®istry).resolve(notification.tid, fd).is_ok() { + // Every registered descriptor is an injected INET socket. Never + // CONTINUE based on a mutable workload sockaddr for such an FD. + return Err(io::Error::from_raw_os_error(libc::EAFNOSUPPORT)); + } + // Native non-INET descriptors remain kernel-driven. + return listener.respond_continue(notification.id); + } + let destination = + read_socket_addr(notification.tid, notification.args[1], notification.args[2])?; + reject_protected_control_destination(destination, protected_control_port)?; + let (kind, socket_identity, nonblocking) = { + let registry = lock(®istry); + let entry = registry.resolve(notification.tid, fd)?; + ( + entry.metadata().kind, + entry.identity(), + entry.metadata().nonblocking, + ) + }; + if kind == InetKind::DnsUdp && destination.port() == 0 { + let mut registry = lock(®istry); + let entry = registry.resolve_mut(notification.tid, fd)?; + if !matches!( + entry.state(), + SocketState::Created | SocketState::Bound { .. } + ) { + return Err(io::Error::from_raw_os_error(libc::EISCONN)); + } + let destination_family = match destination { + SocketAddr::V4(_) => InetFamily::V4, + SocketAddr::V6(_) => InetFamily::V6, + }; + if entry.metadata().family != destination_family { + return Err(io::Error::from_raw_os_error(libc::EAFNOSUPPORT)); + } + // glibc and uv use UDP connect(..., port 0), getsockname(), and an + // AF_UNSPEC disconnect to rank resolved addresses. Bind only to the + // matching loopback family and report success; never connect the + // kernel socket to the external candidate. write(2) therefore remains + // EDESTADDRREQ and destination-bearing sends remain broker-denied. + let local = ensure_dns_source_bound( + entry.retained_preconnect()?.as_raw_fd(), + entry.metadata().family, + )?; + entry.set_state(SocketState::Bound { local }); + return listener.respond_value(notification.id, 0); + } + if destination == dns_relay.address { + let mut registry = lock(®istry); + let entry = registry.resolve_mut(notification.tid, fd)?; + if !matches!( + entry.state(), + SocketState::Created | SocketState::Bound { .. } + ) { + return Err(io::Error::from_raw_os_error(libc::EISCONN)); + } + let source_fd = entry.retained_preconnect()?.as_raw_fd(); + let peer = ensure_dns_source_bound(source_fd, entry.metadata().family)?; + let admissions = match kind { + InetKind::Tcp => &dns_relay.tcp_admissions, + InetKind::DnsUdp => &dns_relay.udp_admissions, + }; + register_dns_socket(admissions, peer, entry.identity())?; + if let Err(error) = connect_exact(source_fd, destination) { + lock(admissions).remove(&peer); + return Err(error); + } + entry.set_state(match kind { + InetKind::Tcp => SocketState::DnsTcp { relay: destination }, + InetKind::DnsUdp => SocketState::DnsUdp { relay: destination }, + }); + entry.release_preconnect(); + return listener.respond_value(notification.id, 0); + } + if destination.ip().is_loopback() { + let mut registry = lock(®istry); + let entry = registry.resolve_mut(notification.tid, fd)?; + connect_exact(entry.retained_preconnect()?.as_raw_fd(), destination)?; + entry.set_state(SocketState::Local { peer: destination }); + entry.release_preconnect(); + return listener.respond_value(notification.id, 0); + } + if kind != InetKind::Tcp { + return Err(io::Error::from_raw_os_error(libc::EACCES)); + } + + let identity = identity_resolver.resolve(notification.tid); + let (decision_tx, decision_rx) = std::sync::mpsc::sync_channel(1); + let (relay_tx, relay_rx) = oneshot::channel(); + let slot = acquire_pending_open_slot(&active_opens)?; + pending + .try_send(PendingTcpOpen { + destination, + identity, + socket: NetworkSocketMetadata { + socket_cookie: socket_identity.cookie, + nonblocking, + process_generation: u64::from(notification.tid), + }, + notification_to_queue: notification_started.elapsed(), + queued_at: Instant::now(), + decision: decision_tx, + relay: relay_rx, + }) + .map_err(|error| match error { + mpsc::error::TrySendError::Full(_) => io::Error::from_raw_os_error(libc::EAGAIN), + mpsc::error::TrySendError::Closed(_) => { + io::Error::new(io::ErrorKind::BrokenPipe, "network-open queue closed") + } + })?; + let worker_listener = Arc::clone(&listener); + std::thread::Builder::new() + .name("openshell-network-open".to_string()) + .spawn(move || { + // The worker owns its quota: an unresponsive supervisor must not + // retain a blocked syscall or worker slot indefinitely. + let _slot = slot; + let result = match decision_rx.recv_timeout(decision_timeout) { + Ok(decision) => decision, + Err(std::sync::mpsc::RecvTimeoutError::Timeout) => { + let _ = worker_listener.respond_errno(notification.id, libc::ETIMEDOUT); + return; + } + Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => { + TcpOpenDecision::Denied(TcpOpenDenial::MediationUnavailable) + } + }; + match result { + TcpOpenDecision::Denied(reason) => { + let _ = + worker_listener.respond_errno(notification.id, tcp_denial_errno(reason)); + } + TcpOpenDecision::RelayReady => { + match worker_listener.validate_id(notification.id).and_then(|()| { + establish_relay( + ®istry, + notification.tid, + fd, + socket_identity, + destination, + ) + }) { + Ok(stream) => { + let result = worker_listener + .respond_value(notification.id, 0) + .map(|()| stream); + let _ = relay_tx.send(result); + } + Err(error) => { + let _ = worker_listener + .respond_errno(notification.id, error_to_errno(&error)); + let _ = relay_tx.send(Err(error)); + } + } + } + } + }) + .map_err(|error| io::Error::other(format!("start network-open worker: {error}")))?; + Ok(()) +} + +const fn tcp_denial_errno(reason: TcpOpenDenial) -> i32 { + match reason { + TcpOpenDenial::PolicyDenied + | TcpOpenDenial::IdentityUnavailable + | TcpOpenDenial::InvalidDestination => libc::EACCES, + TcpOpenDenial::ResourceExhausted => libc::EAGAIN, + TcpOpenDenial::MediationUnavailable => libc::ECANCELED, + } +} + +fn ensure_dns_source_bound(fd: RawFd, family: InetFamily) -> io::Result { + let mut address = socket_local_addr(fd)?; + let loopback = match family { + InetFamily::V4 => IpAddr::V4(Ipv4Addr::LOCALHOST), + InetFamily::V6 => IpAddr::V6(Ipv6Addr::LOCALHOST), + }; + if address.port() == 0 { + bind_exact(fd, SocketAddr::new(loopback, 0))?; + address = socket_local_addr(fd)?; + } + // Async resolvers commonly bind an unspecified address before sendto(2). + // A loopback destination makes the kernel select loopback as the actual + // source, so key attribution by that effective peer rather than by the + // wildcard returned before connect/send. Otherwise the relay observes + // 127.0.0.1: (or ::1:) and drops a valid query registered as + // 0.0.0.0: (or [::]:). + if address.ip().is_unspecified() { + address.set_ip(loopback); + } + Ok(address) +} + +fn establish_relay( + registry: &Mutex, + tid: u32, + fd: RawFd, + expected_socket: SocketIdentity, + destination: SocketAddr, +) -> io::Result { + let relay = TcpListener::bind(match destination { + SocketAddr::V4(_) => SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0), + SocketAddr::V6(_) => SocketAddr::new(IpAddr::V6(Ipv6Addr::LOCALHOST), 0), + })?; + relay.set_nonblocking(false)?; + let relay_address = relay.local_addr()?; + let expected_peer = { + let mut registry = lock(registry); + let entry = registry.resolve_mut(tid, fd)?; + if entry.identity() != expected_socket { + return Err(io::Error::from_raw_os_error(libc::EBADF)); + } + connect_exact(entry.retained_preconnect()?.as_raw_fd(), relay_address)?; + let expected_peer = socket_local_addr(entry.retained_preconnect()?.as_raw_fd())?; + entry.set_state(SocketState::Connected { + original_peer: destination, + }); + entry.release_preconnect(); + expected_peer + }; + relay.set_nonblocking(true)?; + let deadline = Instant::now() + RELAY_CONNECT_TIMEOUT; + let stream = loop { + let now = Instant::now(); + if now >= deadline { + return Err(io::Error::from_raw_os_error(libc::ETIMEDOUT)); + } + let timeout = deadline.saturating_duration_since(now); + let mut poll = libc::pollfd { + fd: relay.as_raw_fd(), + events: libc::POLLIN, + revents: 0, + }; + let timeout = i32::try_from(timeout.as_millis()).unwrap_or(i32::MAX); + // SAFETY: poll points to one live descriptor record. + if unsafe { libc::poll(&raw mut poll, 1, timeout) } <= 0 { + return Err(io::Error::from_raw_os_error(libc::ETIMEDOUT)); + } + match relay.accept() { + Ok((stream, peer)) if peer == expected_peer => break stream, + Ok((_stream, peer)) => { + tracing::warn!(%peer, %expected_peer, "rejected unexpected sandbox relay peer"); + } + Err(error) if error.kind() == io::ErrorKind::WouldBlock => {} + Err(error) => return Err(error), + } + }; + stream.set_nodelay(true)?; + Ok(stream) +} + +fn bind_socket( + registry: &Mutex, + listener: &NotificationListener, + notification: Notification, +) -> io::Result<()> { + let fd = raw_fd(notification.args[0])?; + if !socket_address_is_inet(notification.tid, notification.args[1], notification.args[2])? { + if lock(registry).resolve(notification.tid, fd).is_ok() { + return Err(io::Error::from_raw_os_error(libc::EAFNOSUPPORT)); + } + return listener.respond_continue(notification.id); + } + let local = read_socket_addr(notification.tid, notification.args[1], notification.args[2])?; + if !local.ip().is_loopback() && !local.ip().is_unspecified() { + return Err(io::Error::from_raw_os_error(libc::EACCES)); + } + let bind_result = { + let mut registry = lock(registry); + let entry = registry.resolve_mut(notification.tid, fd)?; + bind_exact(entry.retained_preconnect()?.as_raw_fd(), local) + }; + if bind_result + .as_ref() + .is_err_and(|error| error.raw_os_error() == Some(libc::EADDRINUSE)) + { + collect_closed_socket_entries(registry)?; + let mut registry = lock(registry); + let entry = registry.resolve_mut(notification.tid, fd)?; + bind_exact(entry.retained_preconnect()?.as_raw_fd(), local)?; + entry.set_state(SocketState::Bound { local }); + } else { + bind_result?; + lock(registry) + .resolve_mut(notification.tid, fd)? + .set_state(SocketState::Bound { local }); + } + listener.respond_value(notification.id, 0) +} + +fn collect_closed_socket_entries(registry: &Mutex) -> io::Result<()> { + let mut registry = lock(registry); + collect_closed_socket_entries_locked(&mut registry) +} + +fn collect_closed_socket_entries_locked(registry: &mut SocketRegistry) -> io::Result<()> { + let installed = + openshell_isolation_interface::linux::proc_fd::installed_socket_inodes_excluding( + std::process::id(), + )?; + registry.retain_installed(&installed); + Ok(()) +} + +fn listen_socket( + registry: &Mutex, + listener: &NotificationListener, + notification: Notification, +) -> io::Result<()> { + let fd = raw_fd(notification.args[0])?; + let backlog = i32::try_from(notification.args[1]).unwrap_or(i32::MAX); + let mut registry = lock(registry); + let Ok(entry) = registry.resolve_mut(notification.tid, fd) else { + return listener.respond_continue(notification.id); + }; + // SAFETY: retained descriptor is the exact registered socket OFD. + if unsafe { libc::listen(entry.retained_preconnect()?.as_raw_fd(), backlog) } < 0 { + return Err(io::Error::last_os_error()); + } + let local = socket_local_addr(entry.retained_preconnect()?.as_raw_fd())?; + entry.set_state(SocketState::Listening { local }); + listener.respond_value(notification.id, 0) +} + +fn accept_socket( + registry: Arc>, + listener: Arc, + notification: Notification, + active_accepts: Arc, + accept_registrar: crate::accept_interrupt::AcceptRegistrar, +) -> io::Result<()> { + let fd = raw_fd(notification.args[0])?; + let flags = if i64::from(notification.syscall) == libc::SYS_accept4 { + i32::try_from(notification.args[3]) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))? + } else { + 0 + }; + if flags & !(libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK) != 0 { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); + } + if (notification.args[1] == 0) != (notification.args[2] == 0) { + return Err(io::Error::from_raw_os_error(libc::EFAULT)); + } + let (listener_inode, metadata, source) = { + let registry = lock(®istry); + let Ok(entry) = registry.resolve(notification.tid, fd) else { + return listener.respond_continue(notification.id); + }; + if !matches!(entry.state(), SocketState::Listening { .. }) + || entry.metadata().kind != InetKind::Tcp + { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); + } + let source = duplicate_close_on_exec(entry.retained_preconnect()?.as_raw_fd())?; + (entry.identity().inode, entry.metadata(), source) + }; + let slot = acquire_pending_accept_slot(&active_accepts)?; + let worker_listener = Arc::clone(&listener); + std::thread::Builder::new() + .name("openshell-local-accept".to_string()) + .spawn(move || { + let _slot = slot; + let registration = match accept_registrar.register(notification.id) { + Ok(registration) => registration, + Err(error) => { + let _ = worker_listener.respond_errno(notification.id, error_to_errno(&error)); + return; + } + }; + if let Err(error) = accept_and_inject( + ®istry, + &worker_listener, + notification, + AcceptOperation { + flags, + listener_inode, + metadata, + source, + registration, + }, + ) { + let _ = worker_listener.respond_errno(notification.id, error_to_errno(&error)); + } + }) + .map_err(|error| io::Error::other(format!("start local-accept worker: {error}")))?; + Ok(()) +} + +struct AcceptOperation { + flags: i32, + listener_inode: u64, + metadata: SocketMetadata, + source: OwnedFd, + registration: crate::accept_interrupt::AcceptRegistration, +} + +fn accept_and_inject( + registry: &Mutex, + listener: &NotificationListener, + notification: Notification, + operation: AcceptOperation, +) -> io::Result<()> { + let AcceptOperation { + flags, + listener_inode, + metadata, + source, + registration, + } = operation; + let mut poll = libc::pollfd { + fd: source.as_raw_fd(), + events: libc::POLLIN, + revents: 0, + }; + // SAFETY: F_GETFL reads the live listener OFD flags. + let current_flags = unsafe { libc::fcntl(source.as_raw_fd(), libc::F_GETFL) }; + if current_flags < 0 { + return Err(io::Error::last_os_error()); + } + let nonblocking = current_flags & libc::O_NONBLOCK != 0; + let timeout = if nonblocking { + 0 + } else { + i32::try_from(ACCEPT_POLL_INTERVAL.as_millis()).map_err(io::Error::other)? + }; + // Readiness may disappear before accept (another accept or an aborted + // connection). The registered watchdog interrupts a blocked syscall when + // its notification dies or the broker shuts down. No workload OFD flags + // are changed, and no worker can outlive its cancellation registration. + loop { + registration.ensure_running()?; + listener.validate_id(notification.id)?; + // SAFETY: poll references one live pollfd for this call. + let ready = unsafe { libc::poll(&raw mut poll, 1, timeout) }; + if ready < 0 { + let error = io::Error::last_os_error(); + if error.kind() == io::ErrorKind::Interrupted { + continue; + } + return Err(error); + } + if ready == 0 { + if nonblocking { + return Err(io::Error::from_raw_os_error(libc::EAGAIN)); + } + continue; + } + break; + } + + let mut storage = std::mem::MaybeUninit::::zeroed(); + let mut length = + libc::socklen_t::try_from(size_of::()).map_err(io::Error::other)?; + // Always keep the broker-side descriptor close-on-exec. ADDFD separately + // applies the workload's requested descriptor flag. + let accepted_flags = flags | libc::SOCK_CLOEXEC; + // SAFETY: storage and length are live outputs and source is a listening + // socket proven by the registry. + let accepted = unsafe { + libc::accept4( + source.as_raw_fd(), + storage.as_mut_ptr().cast(), + &raw mut length, + accepted_flags, + ) + }; + if accepted < 0 { + return Err(io::Error::last_os_error()); + } + // Only the blocking accept phase needs asynchronous interruption. Stop + // monitoring before ADDFD completes the notification, otherwise a normal + // successful response could be mistaken for cancellation during commit. + drop(registration); + // SAFETY: successful accept4 returned one newly owned descriptor. + let accepted = unsafe { OwnedFd::from_raw_fd(accepted) }; + // SAFETY: accept4 initialized the reported prefix of storage. + let peer = decode_sockaddr( + unsafe { storage.assume_init() }, + usize::try_from(length).unwrap_or(0), + )?; + if !peer.ip().is_loopback() { + return Err(io::Error::from_raw_os_error(libc::EACCES)); + } + if notification.args[1] != 0 { + write_socket_addr( + listener, + notification.id, + notification.tid, + notification.args[1], + notification.args[2], + peer, + )?; + } + + let accepted_metadata = SocketMetadata { + family: metadata.family, + kind: InetKind::Tcp, + close_on_exec: flags & libc::SOCK_CLOEXEC != 0, + nonblocking: flags & libc::SOCK_NONBLOCK != 0, + creator_generation: u64::from(notification.tid), + }; + let mut registry = lock(registry); + let notifying_fd = raw_fd(notification.args[0])?; + if registry + .resolve(notification.tid, notifying_fd)? + .identity() + .inode + != listener_inode + { + return Err(io::Error::from_raw_os_error(libc::EBADF)); + } + if registry.is_full() { + collect_closed_socket_entries_locked(&mut registry)?; + } + let tentative = registry.stage(accepted, accepted_metadata)?; + listener.add_fd_and_send( + notification.id, + tentative.source_fd(), + accepted_metadata.close_on_exec, + )?; + registry.commit_with_state(tentative, SocketState::AcceptedLocal { peer })?; + Ok(()) +} + +fn duplicate_close_on_exec(fd: RawFd) -> io::Result { + // SAFETY: F_DUPFD_CLOEXEC returns an independent owned descriptor for the + // same open-file description. + let duplicate = unsafe { libc::fcntl(fd, libc::F_DUPFD_CLOEXEC, 3) }; + if duplicate < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: successful fcntl returned one newly owned descriptor. + Ok(unsafe { OwnedFd::from_raw_fd(duplicate) }) +} + +fn classify_send( + registry: &Mutex, + listener: &NotificationListener, + notification: Notification, + dns_relay: &DnsRelay, +) -> io::Result<()> { + let fd = raw_fd(notification.args[0])?; + let syscall = i64::from(notification.syscall); + let (state, metadata) = { + let registry = lock(registry); + let Ok(entry) = registry.resolve(notification.tid, fd) else { + // Non-INET sockets are never injected into the registry. Leave + // their native sendmsg/control-message semantics to the kernel. + return listener.respond_continue(notification.id); + }; + (entry.state().clone(), entry.metadata()) + }; + if matches!( + &state, + SocketState::Connected { .. } | SocketState::AcceptedLocal { .. } + ) || (metadata.kind == InetKind::Tcp && matches!(&state, SocketState::Local { .. })) + { + return listener.respond_continue(notification.id); + } + let messages = match syscall { + libc::SYS_sendto => vec![read_sendto_message(notification)?], + libc::SYS_sendmsg => vec![read_sendmsg_message( + notification.tid, + notification.args[1], + i32::try_from(notification.args[2]) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?, + None, + )?], + libc::SYS_sendmmsg => read_sendmmsg_messages(notification)?, + _ => return Err(io::Error::from_raw_os_error(libc::ENOSYS)), + }; + + let mut registry = lock(registry); + let resolution = registry.resolve(notification.tid, fd); + match resolution { + Ok(entry) + if entry.metadata().kind == InetKind::DnsUdp + && matches!(entry.state(), SocketState::Local { .. }) => + { + if messages.iter().all(|message| message.destination.is_none()) { + listener.respond_continue(notification.id) + } else { + Err(io::Error::from_raw_os_error(libc::EACCES)) + } + } + Ok(entry) if matches!(entry.state(), SocketState::DnsUdp { .. }) => { + let SocketState::DnsUdp { relay } = entry.state() else { + unreachable!("guard requires DNS UDP state"); + }; + // musl-based resolvers, including the statically linked `uv` + // client, send A and AAAA as separate destination-bearing + // datagrams on one socket. The first send pins the socket to the + // private relay; permit later sends only when their copied + // destination is absent or names that same relay. The mandatory + // outer network fence remains the fail-closed backstop for the + // sibling-thread pointer race inherent in seccomp CONTINUE. + if messages.iter().all(|message| { + message + .destination + .is_none_or(|destination| destination == *relay) + }) { + listener.respond_continue(notification.id) + } else { + Err(io::Error::from_raw_os_error(libc::EACCES)) + } + } + Ok(entry) + if entry.metadata().kind == InetKind::DnsUdp + && matches!( + entry.state(), + SocketState::Created | SocketState::Bound { .. } + ) + && messages.iter().all(|message| { + message + .destination + .is_some_and(|value| value == dns_relay.address) + }) => + { + let entry = registry.resolve_mut(notification.tid, fd)?; + let source_fd = entry.retained_preconnect()?.as_raw_fd(); + let peer = ensure_dns_source_bound(source_fd, entry.metadata().family)?; + register_dns_socket(&dns_relay.udp_admissions, peer, entry.identity())?; + if let Err(error) = connect_exact(source_fd, dns_relay.address) { + lock(&dns_relay.udp_admissions).remove(&peer); + return Err(error); + } + for message in &messages { + send_dns_message(source_fd, message)?; + if let Some(length_address) = message.result_length_address { + let length = u32::try_from(message.data.len()) + .map_err(|_| io::Error::from_raw_os_error(libc::EMSGSIZE))?; + listener.validate_id(notification.id)?; + task_memory::write_exact( + notification.tid, + length_address, + &length.to_ne_bytes(), + )?; + } + } + entry.set_state(SocketState::DnsUdp { + relay: dns_relay.address, + }); + entry.release_preconnect(); + let result = if syscall == libc::SYS_sendmmsg { + i64::try_from(messages.len()).unwrap_or(i64::MAX) + } else { + i64::try_from(messages[0].data.len()).unwrap_or(i64::MAX) + }; + listener.respond_value(notification.id, result) + } + Ok(_) => Err(io::Error::from_raw_os_error(libc::EDESTADDRREQ)), + // Non-INET sockets and accepted local sockets were never registered. + // The mandatory outer fence still prevents an external kernel route. + Err(_) => listener.respond_continue(notification.id), + } +} + +struct SendMessage { + data: Vec, + destination: Option, + flags: i32, + result_length_address: Option, +} + +fn read_sendto_message(notification: Notification) -> io::Result { + let length = usize::try_from(notification.args[2]) + .map_err(|_| io::Error::from_raw_os_error(libc::EMSGSIZE))?; + if u16::try_from(length).is_err() { + return Err(io::Error::from_raw_os_error(libc::EMSGSIZE)); + } + let mut data = vec![0_u8; length]; + task_memory::read_exact(notification.tid, notification.args[1], &mut data)?; + let destination = if notification.args[4] == 0 { + None + } else { + Some(read_socket_addr( + notification.tid, + notification.args[4], + notification.args[5], + )?) + }; + Ok(SendMessage { + data, + destination, + flags: i32::try_from(notification.args[3]) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?, + result_length_address: None, + }) +} + +fn read_sendmsg_message( + tid: u32, + address: u64, + flags: i32, + result_length_address: Option, +) -> io::Result { + let header = read_task_value::(tid, address)?; + if header.msg_controllen != 0 { + return Err(io::Error::from_raw_os_error(libc::EOPNOTSUPP)); + } + let destination = if header.msg_name.is_null() { + None + } else { + Some(read_socket_addr( + tid, + header.msg_name as u64, + u64::from(header.msg_namelen), + )?) + }; + #[cfg(target_env = "musl")] + let iov_count = usize::try_from(header.msg_iovlen) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; + #[cfg(not(target_env = "musl"))] + let iov_count = header.msg_iovlen; + if iov_count > 32 { + return Err(io::Error::from_raw_os_error(libc::EMSGSIZE)); + } + let mut data = Vec::new(); + for index in 0..iov_count { + let offset = index + .checked_mul(size_of::()) + .ok_or_else(|| io::Error::from_raw_os_error(libc::EOVERFLOW))?; + let iov = read_task_value::( + tid, + (header.msg_iov as u64) + .checked_add(u64::try_from(offset).unwrap_or(u64::MAX)) + .ok_or_else(|| io::Error::from_raw_os_error(libc::EOVERFLOW))?, + )?; + let start = data.len(); + let end = start + .checked_add(iov.iov_len) + .filter(|length| u16::try_from(*length).is_ok()) + .ok_or_else(|| io::Error::from_raw_os_error(libc::EMSGSIZE))?; + data.resize(end, 0); + task_memory::read_exact(tid, iov.iov_base as u64, &mut data[start..end])?; + } + Ok(SendMessage { + data, + destination, + flags, + result_length_address, + }) +} + +fn read_sendmmsg_messages(notification: Notification) -> io::Result> { + let count = usize::try_from(notification.args[2]) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; + if count == 0 || count > 32 { + return Err(io::Error::from_raw_os_error(libc::EMSGSIZE)); + } + let flags = i32::try_from(notification.args[3]) + .map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; + (0..count) + .map(|index| { + let offset = index + .checked_mul(size_of::()) + .ok_or_else(|| io::Error::from_raw_os_error(libc::EOVERFLOW))?; + let base = notification.args[1] + .checked_add(u64::try_from(offset).unwrap_or(u64::MAX)) + .ok_or_else(|| io::Error::from_raw_os_error(libc::EOVERFLOW))?; + read_sendmsg_message( + notification.tid, + base, + flags, + Some( + base.checked_add( + u64::try_from(std::mem::offset_of!(libc::mmsghdr, msg_len)) + .unwrap_or(u64::MAX), + ) + .ok_or_else(|| io::Error::from_raw_os_error(libc::EOVERFLOW))?, + ), + ) + }) + .collect() +} + +fn read_task_value(tid: u32, address: u64) -> io::Result { + let mut bytes = vec![0_u8; size_of::()]; + task_memory::read_exact(tid, address, &mut bytes)?; + // SAFETY: `bytes` contains exactly one copied native value; unaligned read + // avoids imposing alignment on the task-memory scratch allocation. + Ok(unsafe { std::ptr::read_unaligned(bytes.as_ptr().cast::()) }) +} + +fn send_dns_message(fd: RawFd, message: &SendMessage) -> io::Result<()> { + // SAFETY: `fd` is the retained exact UDP socket and the buffer remains + // valid for the duration of the syscall. + let sent = unsafe { + libc::send( + fd, + message.data.as_ptr().cast(), + message.data.len(), + message.flags, + ) + }; + if sent < 0 { + return Err(io::Error::last_os_error()); + } + if usize::try_from(sent).ok() == Some(message.data.len()) { + Ok(()) + } else { + Err(io::Error::from_raw_os_error(libc::EIO)) + } +} + +fn get_peer_name( + registry: &Mutex, + listener: &NotificationListener, + notification: Notification, +) -> io::Result<()> { + let fd = raw_fd(notification.args[0])?; + let registry = lock(registry); + let Ok(entry) = registry.resolve(notification.tid, fd) else { + return listener.respond_continue(notification.id); + }; + let peer = match entry.state() { + SocketState::Connected { original_peer } => *original_peer, + SocketState::Local { peer } | SocketState::AcceptedLocal { peer } => *peer, + _ => return Err(io::Error::from_raw_os_error(libc::ENOTCONN)), + }; + write_socket_addr( + listener, + notification.id, + notification.tid, + notification.args[1], + notification.args[2], + peer, + )?; + listener.respond_value(notification.id, 0) +} + +fn connect_exact(fd: RawFd, address: SocketAddr) -> io::Result<()> { + // Never let a blocking connect pin the single notification dispatcher. + // O_NONBLOCK is an OFD flag, so restore the workload's original setting + // after the bounded connect attempt completes. + // SAFETY: F_GETFL/F_SETFL operate on the live retained socket descriptor. + let original_flags = unsafe { libc::fcntl(fd, libc::F_GETFL) }; + if original_flags < 0 { + return Err(io::Error::last_os_error()); + } + let changed_flags = original_flags & libc::O_NONBLOCK == 0; + if changed_flags + && unsafe { libc::fcntl(fd, libc::F_SETFL, original_flags | libc::O_NONBLOCK) } < 0 + { + return Err(io::Error::last_os_error()); + } + let result = with_sockaddr(address, |pointer, length| { + // SAFETY: pointer/length describe a live native sockaddr and `fd` is + // the retained exact socket OFD. + let result = unsafe { libc::connect(fd, pointer, length) }; + if result == 0 { + return Ok(()); + } + let error = io::Error::last_os_error(); + if error.raw_os_error() != Some(libc::EINPROGRESS) { + return Err(error); + } + let mut poll = libc::pollfd { + fd, + events: libc::POLLOUT, + revents: 0, + }; + // SAFETY: poll points to one live pollfd. + let timeout = i32::try_from(RELAY_CONNECT_TIMEOUT.as_millis()).map_err(io::Error::other)?; + if unsafe { libc::poll(&raw mut poll, 1, timeout) } <= 0 { + return Err(io::Error::from_raw_os_error(libc::ETIMEDOUT)); + } + let mut socket_error = 0_i32; + let mut size = libc::socklen_t::try_from(size_of::()).map_err(io::Error::other)?; + // SAFETY: getsockopt writes one i32 into live storage. + if unsafe { + libc::getsockopt( + fd, + libc::SOL_SOCKET, + libc::SO_ERROR, + (&raw mut socket_error).cast(), + &raw mut size, + ) + } < 0 + { + return Err(io::Error::last_os_error()); + } + if socket_error == 0 { + Ok(()) + } else { + Err(io::Error::from_raw_os_error(socket_error)) + } + }); + let restore = if changed_flags && unsafe { libc::fcntl(fd, libc::F_SETFL, original_flags) } < 0 + { + Err(io::Error::last_os_error()) + } else { + Ok(()) + }; + result.and(restore) +} + +fn bind_exact(fd: RawFd, address: SocketAddr) -> io::Result<()> { + with_sockaddr(address, |pointer, length| { + // SAFETY: pointer/length describe a live native sockaddr. + if unsafe { libc::bind(fd, pointer, length) } == 0 { + Ok(()) + } else { + Err(io::Error::last_os_error()) + } + }) +} + +fn socket_local_addr(fd: RawFd) -> io::Result { + let mut storage = std::mem::MaybeUninit::::zeroed(); + let mut length = + libc::socklen_t::try_from(size_of::()).map_err(io::Error::other)?; + // SAFETY: storage and length are live output buffers. + if unsafe { libc::getsockname(fd, storage.as_mut_ptr().cast(), &raw mut length) } < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: getsockname initialized `length` bytes, including the family. + decode_sockaddr( + unsafe { storage.assume_init() }, + usize::try_from(length).unwrap_or(0), + ) +} + +fn read_socket_addr(tid: u32, address: u64, length: u64) -> io::Result { + let length = usize::try_from(length).map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; + if length < size_of::() || length > size_of::() { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); + } + let mut bytes = vec![0_u8; length]; + task_memory::read_exact(tid, address, &mut bytes)?; + let mut storage = std::mem::MaybeUninit::::zeroed(); + // SAFETY: destination spans sockaddr_storage and `length` was bounded. + unsafe { + std::ptr::copy_nonoverlapping(bytes.as_ptr(), storage.as_mut_ptr().cast(), length); + decode_sockaddr(storage.assume_init(), length) + } +} + +fn socket_address_is_inet(tid: u32, address: u64, length: u64) -> io::Result { + Ok(matches!( + read_socket_family(tid, address, length)?, + libc::AF_INET | libc::AF_INET6 + )) +} + +fn read_socket_family(tid: u32, address: u64, length: u64) -> io::Result { + let length = usize::try_from(length).map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?; + if address == 0 || length < size_of::() { + return Err(io::Error::from_raw_os_error(libc::EFAULT)); + } + let mut family = [0_u8; size_of::()]; + task_memory::read_exact(tid, address, &mut family)?; + Ok(i32::from(libc::sa_family_t::from_ne_bytes(family))) +} + +fn decode_sockaddr(storage: libc::sockaddr_storage, length: usize) -> io::Result { + match i32::from(storage.ss_family) { + libc::AF_INET if length >= size_of::() => { + // SAFETY: family and length establish sockaddr_in layout. + let address = unsafe { *(&raw const storage).cast::() }; + Ok(SocketAddr::new( + IpAddr::V4(Ipv4Addr::from(address.sin_addr.s_addr.to_ne_bytes())), + u16::from_be(address.sin_port), + )) + } + libc::AF_INET6 if length >= size_of::() => { + // SAFETY: family and length establish sockaddr_in6 layout. + let address = unsafe { *(&raw const storage).cast::() }; + Ok(SocketAddr::new( + IpAddr::V6(Ipv6Addr::from(address.sin6_addr.s6_addr)), + u16::from_be(address.sin6_port), + )) + } + _ => Err(io::Error::from_raw_os_error(libc::EAFNOSUPPORT)), + } +} + +fn write_socket_addr( + listener: &NotificationListener, + notification_id: u64, + tid: u32, + address: u64, + length_address: u64, + value: SocketAddr, +) -> io::Result<()> { + let mut supplied_length = [0_u8; size_of::()]; + task_memory::read_exact(tid, length_address, &mut supplied_length)?; + let supplied_length = libc::socklen_t::from_ne_bytes(supplied_length); + let (bytes, actual_length) = sockaddr_bytes(value)?; + let copied = usize::try_from(supplied_length) + .unwrap_or(0) + .min(bytes.len()); + listener.validate_id(notification_id)?; + if copied != 0 { + task_memory::write_exact(tid, address, &bytes[..copied])?; + } + listener.validate_id(notification_id)?; + task_memory::write_exact(tid, length_address, &actual_length.to_ne_bytes()) +} + +fn sockaddr_bytes(address: SocketAddr) -> io::Result<(Vec, libc::socklen_t)> { + with_sockaddr(address, |native, length| { + let length_usize = usize::try_from(length).map_err(io::Error::other)?; + // SAFETY: with_sockaddr lends fully initialized storage for this call. + let bytes = unsafe { std::slice::from_raw_parts(native.cast::(), length_usize) }; + Ok((bytes.to_vec(), length)) + }) +} + +fn with_sockaddr( + address: SocketAddr, + operation: impl FnOnce(*const libc::sockaddr, libc::socklen_t) -> io::Result, +) -> io::Result { + match address { + SocketAddr::V4(address) => { + let native = libc::sockaddr_in { + sin_family: libc::sa_family_t::try_from(libc::AF_INET).map_err(io::Error::other)?, + sin_port: address.port().to_be(), + sin_addr: libc::in_addr { + s_addr: u32::from_ne_bytes(address.ip().octets()), + }, + sin_zero: [0; 8], + }; + operation( + (&raw const native).cast(), + libc::socklen_t::try_from(size_of::()) + .map_err(io::Error::other)?, + ) + } + SocketAddr::V6(address) => { + let native = libc::sockaddr_in6 { + sin6_family: libc::sa_family_t::try_from(libc::AF_INET6) + .map_err(io::Error::other)?, + sin6_port: address.port().to_be(), + sin6_flowinfo: address.flowinfo(), + sin6_addr: libc::in6_addr { + s6_addr: address.ip().octets(), + }, + sin6_scope_id: address.scope_id(), + }; + operation( + (&raw const native).cast(), + libc::socklen_t::try_from(size_of::()) + .map_err(io::Error::other)?, + ) + } + } +} + +fn raw_fd(value: u64) -> io::Result { + RawFd::try_from(value).map_err(|_| io::Error::from_raw_os_error(libc::EBADF)) +} + +fn lock(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { + mutex + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) +} + +fn error_to_errno(error: &io::Error) -> i32 { + error.raw_os_error().unwrap_or(libc::EACCES).max(1) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Read as _, Write as _}; + use std::os::unix::net::{UnixListener, UnixStream}; + + #[test] + fn relay_rejects_descriptor_replaced_after_policy_decision() { + let metadata = SocketMetadata { + family: InetFamily::V4, + kind: InetKind::Tcp, + close_on_exec: true, + nonblocking: false, + creator_generation: 1, + }; + let mut registry = SocketRegistry::new(1, 2).unwrap(); + let mut create = || { + // SAFETY: a successful socket call returns a new owned descriptor. + let fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_STREAM, 0) }; + assert!(fd >= 0); + let socket = unsafe { OwnedFd::from_raw_fd(fd) }; + let installed = duplicate_close_on_exec(fd).unwrap(); + let tentative = registry.stage(socket, metadata).unwrap(); + let identity = registry.commit(tentative).unwrap(); + (installed, identity) + }; + let (original, original_identity) = create(); + let (replacement, replacement_identity) = create(); + // SAFETY: both descriptors are live; replace only the test-owned FD. + assert_eq!( + unsafe { libc::dup2(replacement.as_raw_fd(), original.as_raw_fd()) }, + original.as_raw_fd() + ); + let registry = Mutex::new(registry); + let error = establish_relay( + ®istry, + std::process::id(), + original.as_raw_fd(), + original_identity, + "203.0.113.7:443".parse().unwrap(), + ) + .expect_err("an approval for the old socket must not connect its replacement"); + assert_eq!(error.raw_os_error(), Some(libc::EBADF)); + let registry = lock(®istry); + let entry = registry + .resolve(std::process::id(), original.as_raw_fd()) + .unwrap(); + assert_eq!(entry.identity(), replacement_identity); + assert_eq!(entry.state(), &SocketState::Created); + assert_eq!( + socket_local_addr(replacement.as_raw_fd()).unwrap().port(), + 0 + ); + } + + #[test] + fn external_connect_times_out_when_supervisor_retains_the_decision() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_with_decision_timeout( + listener, + "127.0.0.1:0".parse().unwrap(), + None, + Duration::from_millis(50), + ) + .unwrap(); + let client = std::thread::spawn(move || { + launcher + .execute(|| TcpStream::connect("203.0.113.7:443")) + .unwrap() + }); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let pending = runtime.block_on(broker.accept()).unwrap(); + // Keep the request alive: channel disconnection must not be what + // releases the workload's blocked connect. + let error = client + .join() + .unwrap() + .expect_err("unanswered connect must time out"); + assert_eq!(error.raw_os_error(), Some(libc::ETIMEDOUT)); + assert!( + runtime + .block_on(pending.complete(TcpOpenDecision::RelayReady)) + .is_err() + ); + } + + #[test] + fn dns_admissions_reclaim_closed_sockets_at_the_bound() { + let admissions = Mutex::new(HashMap::new()); + let stale = SocketIdentity { + listener_generation: 1, + inode: 0, + cookie: 1, + }; + for port in 1..=SOCKET_CAPACITY { + lock(&admissions).insert( + SocketAddr::from(([127, 0, 0, 1], u16::try_from(port).unwrap())), + stale, + ); + } + let peer = "127.0.0.1:50000".parse().unwrap(); + register_dns_socket(&admissions, peer, stale).unwrap(); + assert_eq!(lock(&admissions).len(), 1); + assert_eq!(lock(&admissions).get(&peer), Some(&stale)); + } + + #[test] + fn inherited_dns_socket_after_exec_never_claims_the_connecting_binary() { + use std::process::{Command, Stdio}; + + for transport in [DnsTransport::Udp, DnsTransport::Tcp] { + let (launcher, listener) = + openshell_isolation_interface::linux::workload_launcher::start().unwrap(); + let broker = NetworkBroker::start_for_test(listener).unwrap(); + let address = broker.dns_address(); + let child = std::thread::spawn(move || { + launcher + .execute(move || -> io::Result<()> { + let (socket, script): (OwnedFd, &str) = match transport { + DnsTransport::Udp => { + let socket = UdpSocket::bind("127.0.0.1:0")?; + socket.connect(address)?; + (socket.into(), "printf dns >&0") + } + DnsTransport::Tcp => ( + TcpStream::connect(address)?.into(), + "printf '\\000\\003dns' >&0", + ), + }; + // The socket was connected by this executable. A forked + // child inherits it, execs a different binary, and writes + // without a new connect or a destination-bearing send. + let status = Command::new("/bin/sh") + .args(["-c", script]) + .stdin(Stdio::from(socket)) + .status()?; + if !status.success() { + return Err(io::Error::other("DNS-writing child failed")); + } + Ok(()) + }) + .unwrap() + }); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let query = runtime.block_on(async { + tokio::time::timeout(Duration::from_secs(5), broker.accept_dns()) + .await + .unwrap() + .unwrap() + }); + assert_eq!(query.transport, transport); + assert!(matches!( + query.identity, + Err(ResolveError::Failed(ref message)) if message.contains("unavailable") + )); + query.complete(Ok(Vec::new())).unwrap(); + child.join().unwrap().unwrap(); + } + } + + #[test] + fn protected_control_port_rejects_loopback_aliases_and_pod_addresses() { + for address in [ + "127.0.0.1:7443", + "127.0.0.2:7443", + "[::1]:7443", + "[::ffff:127.0.0.1]:7443", + "10.42.0.8:7443", + ] { + assert_eq!( + reject_protected_control_destination(address.parse().unwrap(), Some(7443)) + .unwrap_err() + .raw_os_error(), + Some(libc::EACCES) + ); + } + assert!( + reject_protected_control_destination("127.0.0.1:8080".parse().unwrap(), Some(7443)) + .is_ok() + ); + } + + #[test] + fn workload_cannot_fill_control_listener_with_loopback_connections() { + let control = TcpListener::bind("127.0.0.1:0").unwrap(); + control.set_nonblocking(true).unwrap(); + let address = control.local_addr().unwrap(); + let (launcher, listener) = + openshell_isolation_interface::linux::workload_launcher::start().unwrap(); + let _broker = NetworkBroker::start_with_dns_address( + listener, + "127.0.0.1:0".parse().unwrap(), + Some(address.port()), + ) + .unwrap(); + launcher + .execute(move || -> io::Result<()> { + for _ in 0..160 { + let error = + TcpStream::connect(address).expect_err("control port must be unreachable"); + assert_eq!(error.raw_os_error(), Some(libc::EACCES)); + } + Ok(()) + }) + .unwrap() + .unwrap(); + assert_eq!( + control.accept().unwrap_err().kind(), + io::ErrorKind::WouldBlock + ); + } + + #[test] + fn pending_external_open_slots_are_bounded_and_reusable() { + let active = Arc::new(AtomicUsize::new(OPEN_QUEUE_CAPACITY - 1)); + let last = acquire_pending_open_slot(&active).expect("last available slot"); + assert_eq!( + acquire_pending_open_slot(&active) + .expect_err("open limit must fail closed") + .raw_os_error(), + Some(libc::EAGAIN) + ); + drop(last); + let reused = acquire_pending_open_slot(&active).expect("released slot"); + drop(reused); + assert_eq!(active.load(Ordering::Acquire), OPEN_QUEUE_CAPACITY - 1); + } + + #[test] + fn dns_worker_slots_are_bounded_and_reusable() { + let active = Arc::new(AtomicUsize::new(DNS_WORKER_CAPACITY - 1)); + let last = acquire_pending_dns_slot(&active).expect("last available slot"); + assert_eq!( + acquire_pending_dns_slot(&active) + .expect_err("DNS worker limit must fail closed") + .raw_os_error(), + Some(libc::EAGAIN) + ); + drop(last); + let reused = acquire_pending_dns_slot(&active).expect("released slot"); + drop(reused); + assert_eq!(active.load(Ordering::Acquire), DNS_WORKER_CAPACITY - 1); + } + + #[test] + fn unix_connect_remains_kernel_driven() { + let directory = tempfile::tempdir().expect("temporary Unix socket directory"); + let path = directory.path().join("service.sock"); + let service = UnixListener::bind(&path).expect("bind Unix service"); + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let client = std::thread::spawn(move || { + launcher + .execute(move || -> io::Result<()> { + let mut stream = UnixStream::connect(path)?; + stream.write_all(b"unix") + }) + .expect("launcher result") + }); + let (mut stream, _) = service.accept().expect("accept Unix client"); + let mut payload = [0_u8; 4]; + stream.read_exact(&mut payload).expect("read Unix payload"); + assert_eq!(&payload, b"unix"); + client.join().expect("join client").expect("Unix client"); + } + + #[test] + fn accepted_loopback_stream_is_registered_for_notified_operations() { + let reservation = TcpListener::bind("127.0.0.1:0").expect("reserve loopback port"); + let address = reservation.local_addr().expect("reserved address"); + drop(reservation); + + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let (ready_tx, ready_rx) = std::sync::mpsc::sync_channel(1); + let workload = std::thread::spawn(move || { + launcher + .execute(move || -> io::Result { + let listener = TcpListener::bind(address)?; + ready_tx + .send(()) + .map_err(|_| io::Error::other("test client disappeared"))?; + let (stream, _) = listener.accept()?; + let peer = stream.peer_addr()?; + let payload = b"accepted"; + let iov = libc::iovec { + iov_base: payload.as_ptr().cast_mut().cast(), + iov_len: payload.len(), + }; + let message = libc::msghdr { + msg_name: std::ptr::null_mut(), + msg_namelen: 0, + msg_iov: (&raw const iov).cast_mut(), + msg_iovlen: 1, + msg_control: std::ptr::null_mut(), + msg_controllen: 0, + msg_flags: 0, + }; + // SAFETY: message references one live immutable payload; + // the accepted stream remains open for the call. + let sent = unsafe { libc::sendmsg(stream.as_raw_fd(), &raw const message, 0) }; + if sent != isize::try_from(payload.len()).expect("payload fits isize") { + return Err(io::Error::last_os_error()); + } + Ok(peer) + }) + .expect("launcher result") + }); + + ready_rx.recv().expect("listener ready"); + let mut client = TcpStream::connect(address).expect("connect loopback client"); + let mut payload = [0_u8; 8]; + client + .read_exact(&mut payload) + .expect("read accepted stream"); + assert_eq!(&payload, b"accepted"); + assert!( + workload + .join() + .expect("join workload") + .expect("accepted workload") + .ip() + .is_loopback() + ); + } + + #[test] + fn external_connect_waits_for_explicit_relay_decision() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let client = std::thread::spawn(move || { + launcher + .execute(|| -> io::Result<()> { + let mut stream = TcpStream::connect("203.0.113.7:443")?; + stream.write_all(b"request")?; + let mut response = [0_u8; 8]; + stream.read_exact(&mut response)?; + if &response != b"response" { + return Err(io::Error::other("relay returned wrong response")); + } + Ok(()) + }) + .expect("launcher result") + }); + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let pending = runtime.block_on(broker.accept()).expect("pending TCP open"); + assert_eq!(pending.destination, "203.0.113.7:443".parse().unwrap()); + assert!(pending.socket.socket_cookie != 0); + let mut relay = runtime + .block_on(pending.complete(TcpOpenDecision::RelayReady)) + .expect("complete relay") + .expect("authorized relay stream"); + let mut request = [0_u8; 7]; + relay + .read_exact(&mut request) + .expect("read relayed request"); + assert_eq!(&request, b"request"); + relay.write_all(b"response").expect("write relay response"); + client.join().expect("join client").expect("client relay"); + } + + #[test] + fn denied_external_connect_keeps_socket_unconnected() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let client = std::thread::spawn(move || { + launcher + .execute(|| TcpStream::connect("198.51.100.9:80")) + .expect("launcher result") + }); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let pending = runtime.block_on(broker.accept()).expect("pending TCP open"); + assert!( + runtime + .block_on(pending.complete(TcpOpenDecision::Denied(TcpOpenDenial::PolicyDenied)),) + .expect("complete denial") + .is_none() + ); + assert_eq!( + client + .join() + .expect("join client") + .expect_err("connect must be denied") + .raw_os_error(), + Some(libc::EACCES) + ); + } + + #[test] + fn udp_dns_normalizes_wildcard_source_for_relay_attribution() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let dns_address = broker.dns_address(); + let client = std::thread::spawn(move || { + launcher + .execute(move || -> io::Result { + // Tokio/Hickory-style resolvers bind a wildcard source + // before sending to the configured nameserver. + let socket = UdpSocket::bind("0.0.0.0:0")?; + socket.set_read_timeout(Some(Duration::from_secs(5)))?; + socket.send_to(b"dns-query", dns_address)?; + let mut response = [0_u8; 32]; + let (length, source) = socket.recv_from(&mut response)?; + if &response[..length] != b"dns-response" { + return Err(io::Error::other("wrong DNS response")); + } + Ok(source) + }) + .expect("launcher result") + }); + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let query = runtime.block_on(broker.accept_dns()).expect("DNS query"); + assert_eq!(query.transport, DnsTransport::Udp); + assert_eq!(query.request, b"dns-query"); + query.complete(Ok(b"dns-response".to_vec())).unwrap(); + assert_eq!( + client.join().expect("join client").expect("DNS client"), + dns_address + ); + } + + #[test] + fn udp_dns_allows_repeated_destination_sends_to_the_pinned_relay() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let dns_address = broker.dns_address(); + let client = std::thread::spawn(move || { + launcher + .execute(move || -> io::Result>> { + // Static musl clients send A and AAAA with two sendto(2) + // calls on the same initially-unconnected socket. + let socket = UdpSocket::bind("0.0.0.0:0")?; + socket.set_read_timeout(Some(Duration::from_secs(5)))?; + socket.send_to(b"dns-query-a", dns_address)?; + socket.send_to(b"dns-query-aaaa", dns_address)?; + let mut responses = Vec::new(); + for _ in 0..2 { + let mut response = [0_u8; 32]; + let (length, source) = socket.recv_from(&mut response)?; + if source != dns_address { + return Err(io::Error::other("wrong DNS response source")); + } + responses.push(response[..length].to_vec()); + } + responses.sort(); + Ok(responses) + }) + .expect("launcher result") + }); + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + for _ in 0..2 { + let query = runtime.block_on(broker.accept_dns()).expect("DNS query"); + let response = if query.request == b"dns-query-a" { + b"dns-response-a".to_vec() + } else if query.request == b"dns-query-aaaa" { + b"dns-response-aaaa".to_vec() + } else { + panic!("unexpected DNS query: {:?}", query.request); + }; + query.complete(Ok(response)).unwrap(); + } + assert_eq!( + client.join().expect("join client").expect("DNS client"), + vec![b"dns-response-a".to_vec(), b"dns-response-aaaa".to_vec()] + ); + } + + #[test] + fn udp_port_zero_route_probes_are_local_and_reusable() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + launcher + .execute(|| -> io::Result<()> { + // Address-selection probes create an unbound datagram socket; + // binding to INADDR_ANY first would intentionally preserve an + // unspecified local address and would not model that path. + // SAFETY: the return value is checked before ownership moves + // into UdpSocket. + let raw_socket = unsafe { + libc::socket( + libc::AF_INET, + libc::SOCK_DGRAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_UDP, + ) + }; + if raw_socket < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: raw_socket is a new, owned socket descriptor. + let socket = unsafe { UdpSocket::from_raw_fd(raw_socket) }; + socket.connect("198.51.100.7:0")?; + let local = socket.local_addr()?; + if !local.ip().is_loopback() || local.port() == 0 { + return Err(io::Error::other(format!( + "route probe did not expose a local source: {local}" + ))); + } + + let unspecified = libc::sockaddr { + sa_family: libc::sa_family_t::try_from(libc::AF_UNSPEC) + .expect("AF_UNSPEC fits sa_family_t"), + sa_data: [0; 14], + }; + // SAFETY: unspecified is a live native sockaddr used for the + // conventional UDP disconnect operation. + let disconnected = unsafe { + libc::connect( + socket.as_raw_fd(), + (&raw const unspecified).cast(), + libc::socklen_t::try_from(size_of::()) + .expect("sockaddr size fits socklen_t"), + ) + }; + if disconnected != 0 { + return Err(io::Error::last_os_error()); + } + socket.connect("203.0.113.9:0")?; + + // The route probe never commits an external UDP peer. A + // destination-free send must therefore remain kernel-denied. + // SAFETY: payload is live for the duration of this syscall. + let sent = unsafe { + libc::send( + socket.as_raw_fd(), + b"blocked".as_ptr().cast(), + b"blocked".len(), + 0, + ) + }; + if sent >= 0 { + return Err(io::Error::other("route probe became a data path")); + } + let error = io::Error::last_os_error(); + if !matches!( + error.raw_os_error(), + Some(libc::EDESTADDRREQ | libc::ENOTCONN) + ) { + return Err(error); + } + Ok(()) + }) + .expect("launcher result") + .expect("route-probe workload"); + } + + #[test] + fn tcp_dns_preserves_length_framing() { + let (launcher, listener) = openshell_isolation_interface::linux::workload_launcher::start() + .expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let dns_address = broker.dns_address(); + let client = std::thread::spawn(move || { + launcher + .execute(move || -> io::Result> { + let mut stream = TcpStream::connect(dns_address)?; + stream.set_read_timeout(Some(Duration::from_secs(5)))?; + stream.write_all(&[0, 3, 1, 2, 3])?; + let mut response = vec![0_u8; 5]; + stream.read_exact(&mut response)?; + Ok(response) + }) + .expect("launcher result") + }); + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let query = runtime.block_on(broker.accept_dns()).expect("DNS query"); + assert_eq!(query.transport, DnsTransport::Tcp); + assert_eq!(query.request, [0, 3, 1, 2, 3]); + query.complete(Ok(vec![0, 3, 4, 5, 6])).unwrap(); + assert_eq!( + client.join().expect("join client").expect("DNS client"), + [0, 3, 4, 5, 6] + ); + } +} diff --git a/crates/openshell-sandbox/src/perf.rs b/crates/openshell-sandbox/src/perf.rs new file mode 100644 index 0000000000..100eac5edb --- /dev/null +++ b/crates/openshell-sandbox/src/perf.rs @@ -0,0 +1,444 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Opt-in microbenchmarks for the production seccomp network listener. + +use std::io::{self, Read as _, Write as _}; +use std::net::{IpAddr, Ipv4Addr, SocketAddr, TcpListener, TcpStream}; +use std::process::{Command, Stdio}; +use std::sync::mpsc; +use std::thread; +use std::time::{Duration, Instant}; + +use anyhow::{Context as _, bail}; +use clap::ValueEnum; +use openshell_isolation_interface::linux::workload_launcher; +use serde::{Deserialize, Serialize}; +use socket2::{Domain, Socket, Type}; + +use crate::network_broker::NetworkBroker; + +#[derive(Clone, Copy, Debug, Deserialize, Serialize, ValueEnum)] +#[serde(rename_all = "kebab-case")] +pub enum Layer { + Native, + Filtered, +} + +impl Layer { + pub fn selection(value: &str) -> anyhow::Result> { + match value { + "all" => Ok(vec![Self::Native, Self::Filtered]), + "native" => Ok(vec![Self::Native]), + "filtered" => Ok(vec![Self::Filtered]), + _ => bail!("unknown layer {value}"), + } + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Serialize, ValueEnum)] +#[serde(rename_all = "kebab-case")] +pub enum Protocol { + TcpConnect, + TcpStream, +} + +impl Protocol { + pub fn selection(value: &str) -> anyhow::Result> { + match value { + "all" => Ok(vec![Self::TcpConnect, Self::TcpStream]), + _ => Ok(vec![ + Self::from_str(value, true).map_err(|error| anyhow::anyhow!("{error}"))?, + ]), + } + } +} + +#[derive(Debug)] +pub struct BenchmarkOptions { + pub layers: Vec, + pub protocols: Vec, + pub iterations: u64, + pub warmup: u64, + pub concurrency: usize, + pub payload_bytes: usize, +} + +#[derive(Debug, Deserialize, Serialize)] +pub struct BenchmarkReport { + pub layer: Layer, + pub protocol: Protocol, + pub iterations: u64, + pub concurrency: usize, + pub payload_bytes: usize, + pub capability_scope: String, + pub elapsed_ms: f64, + pub operations_per_second: f64, + pub throughput_mbit_per_second: f64, + pub latency_ns_p50: u64, + pub latency_ns_p95: u64, + pub latency_ns_p99: u64, +} + +pub fn run(options: BenchmarkOptions) -> anyhow::Result> { + if options.iterations == 0 || options.concurrency == 0 { + bail!("iterations and concurrency must be greater than zero"); + } + if options.payload_bytes == 0 || options.payload_bytes > 65_507 { + bail!("payload-bytes must be between 1 and 65507"); + } + + let executable = std::env::current_exe().context("resolve benchmark executable")?; + let mut reports = Vec::new(); + let mut filtered_runtime = None; + for layer in options.layers { + if matches!(layer, Layer::Filtered) && filtered_runtime.is_none() { + let (launcher, listener) = workload_launcher::start()?; + let broker = NetworkBroker::start_for_test(listener)?; + broker.confirm_healthy()?; + filtered_runtime = Some((launcher, broker)); + } + for protocol in &options.protocols { + let fixture = Fixture::start(*protocol, filtered_runtime.as_ref().map(|(_, b)| b))?; + let mut command = Command::new(&executable); + command + .arg("worker") + .arg("--protocol") + .arg( + protocol + .to_possible_value() + .expect("protocol value") + .get_name(), + ) + .arg("--target") + .arg(fixture.target.to_string()) + .arg("--iterations") + .arg(options.iterations.to_string()) + .arg("--warmup") + .arg(options.warmup.to_string()) + .arg("--concurrency") + .arg(options.concurrency.to_string()) + .arg("--payload-bytes") + .arg(options.payload_bytes.to_string()) + .stdout(Stdio::piped()) + .stderr(Stdio::inherit()); + let output = match layer { + Layer::Native => command.output(), + Layer::Filtered => filtered_runtime + .as_ref() + .expect("filtered runtime") + .0 + .execute(move || command.output())?, + } + .context("run benchmark worker")?; + if !output.status.success() { + bail!("benchmark worker exited with {}", output.status); + } + let mut report: BenchmarkReport = + serde_json::from_slice(&output.stdout).context("decode worker report")?; + report.layer = layer; + reports.push(report); + } + } + Ok(reports) +} + +struct Fixture { + target: SocketAddr, +} + +const OPERATION_TIMEOUT: Duration = Duration::from_secs(10); +const START_TIMEOUT: Duration = Duration::from_secs(30); + +impl Fixture { + fn start(protocol: Protocol, _broker: Option<&NetworkBroker>) -> io::Result { + match protocol { + Protocol::TcpConnect => start_tcp_fixture(false), + Protocol::TcpStream => start_tcp_fixture(true), + } + } +} + +fn start_tcp_fixture(echo: bool) -> io::Result { + let socket = Socket::new(Domain::IPV4, Type::STREAM, None)?; + socket.set_reuse_address(true)?; + socket.bind(&SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0).into())?; + socket.listen(4_096)?; + let listener: TcpListener = socket.into(); + let target = listener.local_addr()?; + thread::Builder::new() + .name("seccomp-perf-tcp-fixture".into()) + .spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { break }; + let _ = stream.set_nodelay(true); + if echo { + let _ = thread::Builder::new() + .name("seccomp-perf-tcp-echo".into()) + .spawn(move || { + let mut buffer = vec![0_u8; 65_507]; + while let Ok(length) = stream.read(&mut buffer) { + if length == 0 || stream.write_all(&buffer[..length]).is_err() { + break; + } + } + }); + } + } + })?; + Ok(Fixture { target }) +} + +#[allow(clippy::cast_precision_loss)] +pub fn run_worker( + protocol: Protocol, + target: SocketAddr, + iterations: u64, + warmup: u64, + concurrency: usize, + payload_bytes: usize, +) -> anyhow::Result { + let preparation_deadline = Instant::now() + START_TIMEOUT; + let (sender, receiver) = mpsc::channel(); + let mut threads = Vec::with_capacity(concurrency); + let mut starters = Vec::with_capacity(concurrency); + let mut readiness = Vec::with_capacity(concurrency); + for _ in 0..concurrency { + let sender = sender.clone(); + let (start_tx, start_rx) = mpsc::channel(); + let (ready_tx, ready_rx) = mpsc::channel(); + starters.push(start_tx); + readiness.push(ready_rx); + threads.push(thread::spawn(move || { + let result = worker_loop( + WorkerOptions { + protocol, + target, + iterations, + warmup, + payload_bytes, + preparation_deadline, + }, + start_rx, + ready_tx, + ); + let _ = sender.send(result); + })); + } + drop(sender); + let preparation = readiness.into_iter().try_for_each(|ready| { + ready + .recv_timeout(preparation_deadline.saturating_duration_since(Instant::now())) + .context("benchmark worker did not finish preparation")? + .map_err(anyhow::Error::msg) + }); + if let Err(error) = preparation { + // Releasing the senders wakes workers that completed preparation while + // a peer failed, so every thread can be joined without hanging. + drop(starters); + for worker in threads { + worker + .join() + .map_err(|_| anyhow::anyhow!("worker panicked"))?; + } + return Err(error); + } + let started = Instant::now(); + for starter in starters { + starter + .send(()) + .context("benchmark worker exited before measurement")?; + } + let mut samples = Vec::new(); + for result in receiver { + samples.extend(result?); + } + let elapsed = started.elapsed(); + for worker in threads { + worker + .join() + .map_err(|_| anyhow::anyhow!("worker panicked"))?; + } + samples.sort_unstable(); + let operations = iterations.saturating_mul(concurrency as u64); + let seconds = elapsed.as_secs_f64(); + Ok(BenchmarkReport { + layer: Layer::Native, + protocol, + iterations: operations, + concurrency, + payload_bytes, + capability_scope: protocol.capability_scope().to_string(), + elapsed_ms: seconds * 1_000.0, + operations_per_second: operations as f64 / seconds, + throughput_mbit_per_second: throughput_mbit_per_second( + protocol, + operations, + payload_bytes, + seconds, + ), + latency_ns_p50: percentile(&samples, 50), + latency_ns_p95: percentile(&samples, 95), + latency_ns_p99: percentile(&samples, 99), + }) +} + +/// Report application bytes transferred by the measured operation. A TCP +/// stream operation writes one payload and reads its echo, so both directions +/// count. A connect-only operation transfers no application payload. +#[allow(clippy::cast_precision_loss)] +fn throughput_mbit_per_second( + protocol: Protocol, + operations: u64, + payload_bytes: usize, + seconds: f64, +) -> f64 { + let bytes_per_operation = match protocol { + Protocol::TcpConnect => 0, + Protocol::TcpStream => payload_bytes.saturating_mul(2), + }; + operations as f64 * bytes_per_operation as f64 * 8.0 / seconds / 1_000_000.0 +} + +impl Protocol { + const fn capability_scope(self) -> &'static str { + match self { + Self::TcpConnect => "implemented local TCP socket/connect interception", + Self::TcpStream => "implemented established TCP fast path", + } + } +} + +#[derive(Clone, Copy)] +struct WorkerOptions { + protocol: Protocol, + target: SocketAddr, + iterations: u64, + warmup: u64, + payload_bytes: usize, + preparation_deadline: Instant, +} + +fn worker_loop( + options: WorkerOptions, + start: mpsc::Receiver<()>, + ready: mpsc::Sender>, +) -> anyhow::Result> { + let WorkerOptions { + protocol, + target, + iterations, + warmup, + payload_bytes, + preparation_deadline, + } = options; + let prepared = (|| -> anyhow::Result<_> { + let payload = vec![0x5a; payload_bytes]; + let mut tcp = if matches!(protocol, Protocol::TcpStream) { + Some(connect(target)?) + } else { + None + }; + let mut response = vec![0_u8; payload_bytes]; + for _ in 0..warmup { + if Instant::now() >= preparation_deadline { + bail!("benchmark warmup exceeded preparation deadline"); + } + one_operation(protocol, target, &payload, &mut response, tcp.as_mut())?; + } + Ok((payload, response, tcp)) + })(); + let (payload, mut response, mut tcp) = match prepared { + Ok(prepared) => { + ready + .send(Ok(())) + .context("benchmark coordinator exited during preparation")?; + prepared + } + Err(error) => { + let message = format!("{error:#}"); + let _ = ready.send(Err(message)); + return Err(error); + } + }; + start.recv().context("benchmark measurement cancelled")?; + let mut samples = Vec::with_capacity(usize::try_from(iterations).unwrap_or(0)); + for _ in 0..iterations { + let started = Instant::now(); + one_operation(protocol, target, &payload, &mut response, tcp.as_mut())?; + samples.push(u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX)); + } + Ok(samples) +} + +fn one_operation( + protocol: Protocol, + target: SocketAddr, + payload: &[u8], + response: &mut [u8], + tcp: Option<&mut TcpStream>, +) -> io::Result<()> { + match protocol { + Protocol::TcpConnect => { + connect(target)?; + } + Protocol::TcpStream => { + let stream = tcp.expect("TCP stream initialized"); + stream.write_all(payload)?; + stream.read_exact(response)?; + } + } + Ok(()) +} + +fn connect(target: SocketAddr) -> io::Result { + let stream = TcpStream::connect_timeout(&target, OPERATION_TIMEOUT)?; + stream.set_nodelay(true)?; + stream.set_read_timeout(Some(OPERATION_TIMEOUT))?; + stream.set_write_timeout(Some(OPERATION_TIMEOUT))?; + Ok(stream) +} + +fn percentile(samples: &[u64], percentile: usize) -> u64 { + if samples.is_empty() { + return 0; + } + let index = (samples.len() - 1) * percentile / 100; + samples[index] +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn percentiles_are_stable() { + let samples = (1..=100).collect::>(); + assert_eq!(percentile(&samples, 50), 50); + assert_eq!(percentile(&samples, 95), 95); + assert_eq!(percentile(&samples, 99), 99); + } + + #[test] + fn tcp_connect_never_reports_payload_throughput() { + let small_payload = throughput_mbit_per_second(Protocol::TcpConnect, 1_000, 64, 1.0); + let large_payload = throughput_mbit_per_second(Protocol::TcpConnect, 1_000, 65_507, 1.0); + assert!(small_payload.abs() < f64::EPSILON); + assert!(large_payload.abs() < f64::EPSILON); + } + + #[test] + fn tcp_stream_counts_request_and_echo_bytes() { + let throughput = throughput_mbit_per_second(Protocol::TcpStream, 1_000, 64, 1.0); + assert!((throughput - 1.024).abs() < f64::EPSILON); + } + + #[test] + fn preparation_failure_does_not_wait_for_other_workers() { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind test listener"); + let address = listener.local_addr().expect("test listener address"); + drop(listener); + + assert!(run_worker(Protocol::TcpStream, address, 1, 1, 2, 8).is_err()); + } +} diff --git a/crates/openshell-supervisor-process/src/process.rs b/crates/openshell-sandbox/src/process.rs similarity index 87% rename from crates/openshell-supervisor-process/src/process.rs rename to crates/openshell-sandbox/src/process.rs index 0ab1dd3187..5e07337214 100644 --- a/crates/openshell-supervisor-process/src/process.rs +++ b/crates/openshell-sandbox/src/process.rs @@ -6,33 +6,25 @@ use crate::child_env; #[cfg(target_os = "linux")] use crate::managed_children; -#[cfg(target_os = "linux")] -use crate::netns::NetworkNamespace; use crate::sandbox; #[cfg(target_os = "linux")] use miette::WrapErr; use miette::{IntoDiagnostic, Result}; use nix::sys::signal::{self, Signal}; use nix::unistd::{Gid, Group, Pid, Uid, User}; -use openshell_core::policy::{NetworkMode, SandboxPolicy}; +use openshell_core::policy::SandboxPolicy; use std::collections::HashMap; use std::ffi::CString; #[cfg(unix)] use std::os::fd::AsRawFd; -#[cfg(target_os = "linux")] -use std::os::fd::RawFd; -#[cfg(target_os = "linux")] -use std::os::unix::ffi::OsStrExt; #[cfg(unix)] use std::os::unix::fs::{MetadataExt, PermissionsExt}; #[cfg(any(test, unix))] use std::path::Path; use std::path::PathBuf; use std::process::Stdio; -#[cfg(target_os = "linux")] -use std::sync::OnceLock; -#[cfg(target_os = "linux")] -use std::sync::mpsc; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; use tokio::process::{Child, ChildStderr, ChildStdin, ChildStdout, Command}; use tracing::{debug, info}; @@ -48,18 +40,6 @@ fn set_controlling_tty(fd: libc::c_int) -> std::io::Result<()> { Ok(()) } -/// Process/filesystem enforcement performed by the process supervisor. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ProcessEnforcementMode { - /// Preserve the existing supervisor behavior: prepare filesystem policy, - /// drop privileges, and apply Landlock/seccomp to workload processes. - Full, - /// Preserve process launch and SSH/session behavior, but skip controls - /// that require root or extra Linux capabilities. Kubernetes sidecar mode - /// uses this when network policy is enforced by the network sidecar. - NetworkOnly, -} - /// Numeric identity components resolved once from driver-owned metadata. /// /// A component is `None` when the corresponding policy field was explicit and @@ -128,34 +108,43 @@ impl ResolvedWorkspace { } } -impl ProcessEnforcementMode { - #[must_use] - pub const fn uses_privileged_process_setup(self) -> bool { - matches!(self, Self::Full) - } - - #[must_use] - pub const fn enforces_child_sandbox(self) -> bool { - matches!(self, Self::Full | Self::NetworkOnly) - } -} - #[cfg(target_os = "linux")] pub(crate) fn prepare_child_sandbox( policy: &SandboxPolicy, workdir: Option<&str>, - enforcement_mode: ProcessEnforcementMode, + runtime_read_only: &[PathBuf], ) -> Result> { - if !enforcement_mode.enforces_child_sandbox() { - return Ok(None); + let effective_policy = policy_with_runtime_read_only(policy, runtime_read_only); + let prepared = sandbox::linux::prepare_capability_free(&effective_policy, workdir)?; + Ok(Some(prepared)) +} + +#[cfg(target_os = "linux")] +fn policy_with_runtime_read_only( + policy: &SandboxPolicy, + runtime_read_only: &[PathBuf], +) -> SandboxPolicy { + let mut effective_policy = policy.clone(); + for path in runtime_read_only { + if !effective_policy.filesystem.read_only.contains(path) { + effective_policy.filesystem.read_only.push(path.clone()); + } } + effective_policy +} - let prepared = if enforcement_mode.uses_privileged_process_setup() { - sandbox::linux::prepare(policy, workdir) - } else { - sandbox::linux::prepare_current_user(policy, workdir) - }?; - Ok(Some(prepared)) +#[cfg(target_os = "linux")] +pub(crate) fn ca_runtime_read_only_paths(ca_paths: Option<&(PathBuf, PathBuf)>) -> Vec { + let Some((certificate, bundle)) = ca_paths else { + return Vec::new(); + }; + let mut paths = Vec::with_capacity(3); + if let Some(directory) = certificate.parent() { + paths.push(directory.to_path_buf()); + } + paths.push(certificate.clone()); + paths.push(bundle.clone()); + paths } const SUPERVISOR_ONLY_ENV_VARS: &[&str] = &[ @@ -172,6 +161,19 @@ const SUPERVISOR_ONLY_ENV_VARS: &[&str] = &[ openshell_core::sandbox_env::NETWORK_RUNTIME_CAPABILITIES, ]; +const PROXY_ENV_VARS: &[&str] = &[ + "ALL_PROXY", + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "all_proxy", + "http_proxy", + "https_proxy", + "no_proxy", + "grpc_proxy", + "NODE_USE_ENV_PROXY", +]; + pub fn is_supervisor_only_env_var(key: &str) -> bool { SUPERVISOR_ONLY_ENV_VARS.contains(&key) } @@ -182,6 +184,27 @@ fn strip_supervisor_only_env(cmd: &mut Command) { } } +/// Remove ambient proxy routing from a transparently mediated child. +pub fn strip_proxy_env(cmd: &mut Command) { + for key in PROXY_ENV_VARS { + cmd.env_remove(key); + } +} + +/// [`strip_proxy_env`] for synchronous exec commands. +pub fn strip_proxy_env_std(cmd: &mut std::process::Command) { + for key in PROXY_ENV_VARS { + cmd.env_remove(key); + } +} + +/// Whether an environment key can redirect a child around transparent +/// network mediation. +#[must_use] +pub fn is_proxy_env_var(key: &str) -> bool { + PROXY_ENV_VARS.contains(&key) +} + fn inject_provider_env(cmd: &mut Command, provider_env: &HashMap) { for (key, value) in provider_env { if is_supervisor_only_env_var(key) { @@ -396,262 +419,24 @@ fn validate_capability_bounding_set_clear( } } -// Pins the pre-seccomp child mount namespace where supervisor identity sockets -// are shadowed. Children enter it with setns before dropping privileges. -#[cfg(target_os = "linux")] -static SUPERVISOR_IDENTITY_MOUNT_NS: OnceLock> = - OnceLock::new(); - -#[cfg(target_os = "linux")] -pub struct SupervisorIdentityMountNamespace { - spawn_tx: mpsc::Sender, -} - -#[cfg(target_os = "linux")] -type SupervisorIdentityNsRef = &'static SupervisorIdentityMountNamespace; -#[cfg(target_os = "linux")] -type SupervisorIdentitySpawnJob = Box; - -#[cfg(target_os = "linux")] -impl SupervisorIdentityMountNamespace { - fn from_socket_path(socket_path: &str) -> Result> { - let Some(target) = supervisor_identity_mount_target(socket_path)? else { - return Ok(None); - }; - Ok(Some(Self { - spawn_tx: start_supervisor_identity_spawn_worker(target)?, - })) - } -} - -#[cfg(target_os = "linux")] -pub fn prepare_supervisor_identity_mount_namespace_from_env() -> Result<()> { - if SUPERVISOR_IDENTITY_MOUNT_NS.get().is_some() { - return Ok(()); - } - - let Some((_env_name, socket_path)) = supervisor_identity_socket_path_from_env() else { - let _ = SUPERVISOR_IDENTITY_MOUNT_NS.set(None); - return Ok(()); - }; - let namespace = SupervisorIdentityMountNamespace::from_socket_path(&socket_path)?; - let _ = SUPERVISOR_IDENTITY_MOUNT_NS.set(namespace); - Ok(()) -} - #[cfg(target_os = "linux")] -pub fn supervisor_identity_mount_from_env() -> Result> { - let Some(namespace) = SUPERVISOR_IDENTITY_MOUNT_NS.get() else { - if supervisor_identity_socket_path_from_env().is_some() { - return Err(miette::miette!( - "supervisor identity mount namespace was not prepared before startup hardening" - )); - } - return Ok(None); - }; - Ok(namespace.as_ref()) -} - -#[cfg(target_os = "linux")] -pub fn spawn_command_with_supervisor_identity_namespace( +pub fn spawn_command_with_workload_launcher( + launcher: &openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, mut cmd: Command, ) -> std::io::Result { - let namespace = supervisor_identity_mount_from_env() - .map_err(|err| std::io::Error::other(err.to_string()))?; - let Some(namespace) = namespace else { - return cmd.spawn(); - }; - namespace.spawn_tokio_command(cmd) + let runtime = tokio::runtime::Handle::current(); + launcher.execute(move || { + let _guard = runtime.enter(); + cmd.spawn() + })? } #[cfg(target_os = "linux")] -pub fn spawn_std_command_with_supervisor_identity_namespace( +pub fn spawn_std_command_with_workload_launcher( + launcher: &openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, mut cmd: std::process::Command, ) -> std::io::Result { - let namespace = supervisor_identity_mount_from_env() - .map_err(|err| std::io::Error::other(err.to_string()))?; - let Some(namespace) = namespace else { - return cmd.spawn(); - }; - namespace.spawn_std_command(cmd) -} - -#[cfg(target_os = "linux")] -impl SupervisorIdentityMountNamespace { - fn spawn_tokio_command(&self, mut cmd: Command) -> std::io::Result { - let (result_tx, result_rx) = mpsc::channel(); - let handle = tokio::runtime::Handle::current(); - self.spawn_tx - .send(Box::new(move || { - let _guard = handle.enter(); - let _ = result_tx.send(cmd.spawn()); - })) - .map_err(|_| std::io::Error::other("supervisor identity spawn worker stopped"))?; - result_rx - .recv() - .map_err(|_| std::io::Error::other("supervisor identity spawn worker dropped result"))? - } - - fn spawn_std_command( - &self, - mut cmd: std::process::Command, - ) -> std::io::Result { - let (result_tx, result_rx) = mpsc::channel(); - self.spawn_tx - .send(Box::new(move || { - let _ = result_tx.send(cmd.spawn()); - })) - .map_err(|_| std::io::Error::other("supervisor identity spawn worker stopped"))?; - result_rx - .recv() - .map_err(|_| std::io::Error::other("supervisor identity spawn worker dropped result"))? - } -} - -#[cfg(target_os = "linux")] -fn start_supervisor_identity_spawn_worker( - target: PathBuf, -) -> Result> { - let (spawn_tx, spawn_rx) = mpsc::channel::(); - let (ready_tx, ready_rx) = mpsc::channel::>(); - std::thread::Builder::new() - .name("openshell-identity-spawn".into()) - .spawn(move || { - let setup = (|| -> std::io::Result<()> { - private_mount_namespace()?; - let target = - cstring_path(&target).map_err(|err| std::io::Error::other(err.to_string()))?; - mount_empty_tmpfs(&target) - })(); - let ready = match &setup { - Ok(()) => Ok(()), - Err(err) => Err(std::io::Error::new( - err.kind(), - format!("supervisor identity setup failed: {err}"), - )), - }; - let _ = ready_tx.send(ready); - if setup.is_err() { - return; - } - while let Ok(job) = spawn_rx.recv() { - job(); - } - }) - .map_err(|err| miette::miette!("failed to spawn supervisor identity worker: {err}"))?; - ready_rx - .recv() - .map_err(|err| miette::miette!("supervisor identity worker did not start: {err}"))? - .map_err(|err| miette::miette!("{err}"))?; - Ok(spawn_tx) -} - -#[cfg(target_os = "linux")] -fn supervisor_identity_socket_path_from_env() -> Option<(&'static str, String)> { - std::env::var(openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET) - .ok() - .filter(|socket_path| !socket_path.trim().is_empty()) - .map(|socket_path| { - ( - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, - socket_path, - ) - }) -} - -#[cfg(any(test, target_os = "linux"))] -fn supervisor_identity_mount_target(socket_path: &str) -> Result> { - let trimmed = socket_path.trim(); - if trimmed.is_empty() { - return Ok(None); - } - if trimmed.starts_with("tcp:") { - return Ok(None); - } - let path = trimmed.strip_prefix("unix:").unwrap_or(trimmed); - let path = Path::new(path); - if !path.is_absolute() { - return Err(miette::miette!( - "{} must be an absolute UNIX socket path", - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET - )); - } - let Some(parent) = path.parent() else { - return Err(miette::miette!( - "{} has no parent directory", - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET - )); - }; - if parent == Path::new("/") { - return Err(miette::miette!( - "{} must live below a dedicated directory, not directly under /", - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET - )); - } - if is_shared_root_mount_shadow(parent) { - return Err(miette::miette!( - "{} must live below a dedicated subdirectory; refusing to hide shared directory {}", - openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, - parent.display() - )); - } - Ok(Some(parent.to_path_buf())) -} - -#[cfg(any(test, target_os = "linux"))] -fn is_shared_root_mount_shadow(parent: &Path) -> bool { - matches!(parent.to_str(), Some("/run" | "/var" | "/tmp" | "/etc")) -} - -#[cfg(target_os = "linux")] -fn cstring_path(path: &Path) -> Result { - CString::new(path.as_os_str().as_bytes()) - .map_err(|_| miette::miette!("path contains an interior NUL byte: {}", path.display())) -} - -#[cfg(target_os = "linux")] -fn private_mount_namespace() -> std::io::Result<()> { - #[allow(unsafe_code)] - let rc = unsafe { libc::unshare(libc::CLONE_NEWNS) }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - - #[allow(unsafe_code)] - let rc = unsafe { - let flags: libc::c_ulong = libc::MS_REC | libc::MS_PRIVATE; - libc::mount( - std::ptr::null(), - c"/".as_ptr(), - std::ptr::null(), - flags, - std::ptr::null(), - ) - }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - Ok(()) -} - -#[cfg(target_os = "linux")] -fn mount_empty_tmpfs(target: &CString) -> std::io::Result<()> { - #[allow(unsafe_code)] - let rc = unsafe { - let flags: libc::c_ulong = - libc::MS_NOSUID | libc::MS_NODEV | libc::MS_NOEXEC | libc::MS_RDONLY; - libc::mount( - c"tmpfs".as_ptr(), - target.as_ptr(), - c"tmpfs".as_ptr(), - flags, - c"mode=0555,size=4k".as_ptr().cast(), - ) - }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - Ok(()) + launcher.execute(move || cmd.spawn())? } /// Handle to a running process. @@ -659,6 +444,10 @@ pub struct ProcessHandle { child: Child, pid: u32, io: Option, + terminal: Arc, + signal_lock: Arc>, + #[cfg(target_os = "linux")] + managed_child: Option, } /// Supervisor-owned canonical-process I/O. These handles outlive individual @@ -681,26 +470,22 @@ impl ProcessHandle { #[cfg(target_os = "linux")] #[allow(clippy::too_many_arguments)] pub fn spawn( + launcher: &openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, program: &str, args: &[String], workspace: &ResolvedWorkspace, interactive: bool, policy: &SandboxPolicy, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - netns: Option<&NetworkNamespace>, ca_paths: Option<&(PathBuf, PathBuf)>, provider_env: &HashMap, ) -> Result { Self::spawn_impl( + launcher, program, args, workspace, interactive, policy, - resolved_identity, - enforcement_mode, - netns.and_then(NetworkNamespace::ns_fd), ca_paths, provider_env, ) @@ -719,8 +504,6 @@ impl ProcessHandle { workspace: &ResolvedWorkspace, interactive: bool, policy: &SandboxPolicy, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, ca_paths: Option<&(PathBuf, PathBuf)>, provider_env: &HashMap, ) -> Result { @@ -730,8 +513,6 @@ impl ProcessHandle { workspace, interactive, policy, - resolved_identity, - enforcement_mode, ca_paths, provider_env, ) @@ -740,14 +521,12 @@ impl ProcessHandle { #[cfg(target_os = "linux")] #[allow(clippy::too_many_arguments)] fn spawn_impl( + launcher: &openshell_isolation_interface::linux::workload_launcher::WorkloadLauncher, program: &str, args: &[String], workspace: &ResolvedWorkspace, interactive: bool, policy: &SandboxPolicy, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - netns_fd: Option, ca_paths: Option<&(PathBuf, PathBuf)>, provider_env: &HashMap, ) -> Result { @@ -798,29 +577,7 @@ impl ProcessHandle { cmd.current_dir(dir); } - if matches!(policy.network.mode, NetworkMode::Proxy) { - let proxy = policy.network.proxy.as_ref().ok_or_else(|| { - miette::miette!( - "Network mode is set to proxy but no proxy configuration was provided" - ) - })?; - // When using network namespace, set proxy URL to the veth host IP - if netns_fd.is_some() { - // The proxy is on 10.200.0.1:3128 (or configured port) - let port = proxy.http_addr.map_or(3128, |addr| addr.port()); - let proxy_url = format!("http://10.200.0.1:{port}"); - // Both uppercase and lowercase variants: curl/wget use uppercase, - // gRPC C-core (libgrpc) checks lowercase http_proxy/https_proxy. - for (key, value) in child_env::proxy_env_vars(&proxy_url) { - cmd.env(key, value); - } - } else if let Some(http_addr) = proxy.http_addr { - let proxy_url = format!("http://{http_addr}"); - for (key, value) in child_env::proxy_env_vars(&proxy_url) { - cmd.env(key, value); - } - } - } + strip_proxy_env(&mut cmd); // Set TLS trust store env vars so sandbox processes trust the ephemeral CA if let Some((ca_cert_path, combined_bundle_path)) = ca_paths { @@ -833,25 +590,26 @@ impl ProcessHandle { // process where the tracing subscriber is functional. The child's // pre_exec context cannot reliably emit structured logs. #[cfg(target_os = "linux")] - if enforcement_mode.enforces_child_sandbox() { - sandbox::linux::log_sandbox_readiness(policy, workspace.root()); - } + sandbox::linux::log_sandbox_readiness(policy, workspace.root()); - // Phase 1: Prepare Landlock ruleset by opening PathFds. - // In full mode this runs before drop_privileges() so root-only paths - // can be opened. In sidecar network-only mode the container already - // runs as the sandbox UID, so inaccessible paths are unavailable to - // the workload and best-effort compatibility skips them. + // Prepare the Landlock ruleset as the workload UID. Inaccessible paths + // are already unavailable to the child and remain omitted. #[cfg(target_os = "linux")] - let prepared_sandbox = prepare_child_sandbox(policy, workspace.root(), enforcement_mode) + let runtime_read_only = ca_runtime_read_only_paths(ca_paths); + let prepared_sandbox = prepare_child_sandbox(policy, workspace.root(), &runtime_read_only) .map_err(|err| miette::miette!("Failed to prepare sandbox: {err}"))?; + #[cfg(target_os = "linux")] + let mut child_hardening = + openshell_isolation_interface::linux::child_seccomp::prepare(std::process::id()) + .map_err(|error| { + miette::miette!("prepare child self-protection filter: {error}") + })?; // Set up process group for signal handling (non-interactive mode only). // In interactive mode, we inherit the parent's process group to maintain // proper terminal control for shells and interactive programs. // SAFETY: pre_exec runs after fork but before exec in the child process. // setpgid and setns are async-signal-safe and safe to call in this context. { - let policy = policy.clone(); // Wrap in Option so we can .take() it out of the FnMut closure. // pre_exec is only called once (after fork, before exec). #[cfg(target_os = "linux")] @@ -868,25 +626,6 @@ impl ProcessHandle { return Err(std::io::Error::last_os_error()); } - // Enter network namespace before applying other restrictions. - if let Some(fd) = netns_fd { - let result = libc::setns(fd, libc::CLONE_NEWNET); - if result != 0 { - return Err(std::io::Error::other(format!( - "failed to enter network namespace: {}", - std::io::Error::last_os_error() - ))); - } - } - - // Drop privileges. initgroups/setgid/setuid need access to - // /etc/group and /etc/passwd which would be blocked if - // Landlock were already enforced. - if enforcement_mode.uses_privileged_process_setup() { - drop_privileges_with_identity(&policy, resolved_identity) - .map_err(|err| std::io::Error::other(err.to_string()))?; - } - harden_child_process().map_err(|err| std::io::Error::other(err.to_string()))?; // Phase 2 (as unprivileged user): Enforce the prepared @@ -894,7 +633,7 @@ impl ProcessHandle { // restrict_self() does not require root. #[cfg(target_os = "linux")] if let Some(prepared) = prepared_sandbox.take() { - sandbox::linux::enforce(prepared) + sandbox::linux::enforce_capability_free(prepared, &mut child_hardening) .map_err(|err| std::io::Error::other(err.to_string()))?; } @@ -908,7 +647,9 @@ impl ProcessHandle { // or interpreter, and is a common failure on images that lack the // requested shell/binary (e.g. bash on Alpine). #[cfg(target_os = "linux")] - let mut child = spawn_command_with_supervisor_identity_namespace(cmd) + let mut child_registry = managed_children::lock(); + #[cfg(target_os = "linux")] + let mut child = spawn_command_with_workload_launcher(launcher, cmd) .into_diagnostic() .wrap_err_with(|| format!("failed to spawn sandbox entrypoint process '{program}'"))?; #[cfg(not(target_os = "linux"))] @@ -917,7 +658,8 @@ impl ProcessHandle { .into_diagnostic() .wrap_err_with(|| format!("failed to spawn sandbox entrypoint process '{program}'"))?; let pid = child.id().unwrap_or(0); - managed_children::register(pid); + let managed_child = child_registry.register(pid); + drop(child_registry); let io = if let Some(master) = pty_master { ProcessIo::Pty(master) @@ -935,6 +677,10 @@ impl ProcessHandle { child, pid, io: Some(io), + terminal: Arc::new(AtomicBool::new(false)), + signal_lock: Arc::new(std::sync::Mutex::new(())), + #[cfg(target_os = "linux")] + managed_child, }) } @@ -946,8 +692,6 @@ impl ProcessHandle { workspace: &ResolvedWorkspace, interactive: bool, policy: &SandboxPolicy, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, ca_paths: Option<&(PathBuf, PathBuf)>, provider_env: &HashMap, ) -> Result { @@ -998,19 +742,7 @@ impl ProcessHandle { cmd.current_dir(dir); } - if matches!(policy.network.mode, NetworkMode::Proxy) { - let proxy = policy.network.proxy.as_ref().ok_or_else(|| { - miette::miette!( - "Network mode is set to proxy but no proxy configuration was provided" - ) - })?; - if let Some(http_addr) = proxy.http_addr { - let proxy_url = format!("http://{http_addr}"); - for (key, value) in child_env::proxy_env_vars(&proxy_url) { - cmd.env(key, value); - } - } - } + strip_proxy_env(&mut cmd); // Set TLS trust store env vars so sandbox processes trust the ephemeral CA if let Some((ca_cert_path, combined_bundle_path)) = ca_paths { @@ -1040,20 +772,9 @@ impl ProcessHandle { return Err(std::io::Error::last_os_error()); } - // Drop privileges before applying sandbox restrictions. - // initgroups/setgid/setuid need access to /etc/group and /etc/passwd - // which may be blocked by Landlock. - if enforcement_mode.uses_privileged_process_setup() { - drop_privileges_with_identity(&policy, resolved_identity) - .map_err(|err| std::io::Error::other(err.to_string()))?; - } - harden_child_process().map_err(|err| std::io::Error::other(err.to_string()))?; - - if enforcement_mode.enforces_child_sandbox() { - sandbox::apply(&policy, workdir.as_deref()) - .map_err(|err| std::io::Error::other(err.to_string()))?; - } + sandbox::apply(&policy, workdir.as_deref()) + .map_err(|err| std::io::Error::other(err.to_string()))?; Ok(()) }); @@ -1081,6 +802,8 @@ impl ProcessHandle { child, pid, io: Some(io), + terminal: Arc::new(AtomicBool::new(false)), + signal_lock: Arc::new(std::sync::Mutex::new(())), }) } @@ -1095,6 +818,12 @@ impl ProcessHandle { self.io.take().expect("canonical process I/O already taken") } + /// Shared state used by an independent boundary signal handle. + #[must_use] + pub fn signaling_state(&self) -> (Arc, Arc>) { + (self.terminal.clone(), self.signal_lock.clone()) + } + /// Wait for the process to exit. /// /// # Errors @@ -1102,8 +831,15 @@ impl ProcessHandle { /// Returns an error if waiting fails. pub async fn wait(&mut self) -> std::io::Result { let status = self.child.wait().await; + let _signal_guard = self + .signal_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + self.terminal.store(true, Ordering::Release); #[cfg(target_os = "linux")] - managed_children::unregister(self.pid); + if let Some(child) = self.managed_child.take() { + managed_children::unregister(child); + } let status = status?; Ok(ProcessStatus::from(status)) } @@ -1112,8 +848,15 @@ impl ProcessHandle { pub fn try_wait(&mut self) -> std::io::Result> { let status = self.child.try_wait()?; if status.is_some() { + let _signal_guard = self + .signal_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + self.terminal.store(true, Ordering::Release); #[cfg(target_os = "linux")] - managed_children::unregister(self.pid); + if let Some(child) = self.managed_child.take() { + managed_children::unregister(child); + } } Ok(status.map(ProcessStatus::from)) } @@ -1124,6 +867,13 @@ impl ProcessHandle { /// /// Returns an error if the signal cannot be sent. pub fn signal(&self, sig: Signal) -> Result<()> { + let _signal_guard = self + .signal_lock + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if self.terminal.load(Ordering::Acquire) { + return Err(miette::miette!("process has exited")); + } let pid = i32::try_from(self.pid).unwrap_or(i32::MAX); signal::kill(Pid::from_raw(pid), sig).into_diagnostic() } @@ -1163,7 +913,9 @@ impl ProcessHandle { impl Drop for ProcessHandle { fn drop(&mut self) { #[cfg(target_os = "linux")] - managed_children::unregister(self.pid); + if let Some(child) = self.managed_child.take() { + managed_children::unregister(child); + } } } @@ -2567,18 +2319,6 @@ mod tests { assert!(validate_sandbox_group_with_identity(&policy, resolved).is_ok()); } - #[test] - fn full_enforcement_uses_privileged_setup_and_child_sandbox() { - assert!(ProcessEnforcementMode::Full.uses_privileged_process_setup()); - assert!(ProcessEnforcementMode::Full.enforces_child_sandbox()); - } - - #[test] - fn network_only_enforcement_keeps_child_sandbox_without_privileged_setup() { - assert!(!ProcessEnforcementMode::NetworkOnly.uses_privileged_process_setup()); - assert!(ProcessEnforcementMode::NetworkOnly.enforces_child_sandbox()); - } - #[cfg(target_os = "linux")] fn capability_bounding_set_clear_available() -> bool { capctl::caps::CapState::get_current() @@ -3429,6 +3169,85 @@ mod tests { } } + #[cfg(target_os = "linux")] + #[test] + fn runtime_ca_paths_are_added_to_the_effective_read_only_policy() { + let mut policy = policy_with_process(ProcessPolicy::default()); + policy.filesystem.read_only = vec![PathBuf::from("/usr")]; + let certificate = PathBuf::from(format!( + "{}/ca.crt", + openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_DIR + )); + let bundle = PathBuf::from(format!( + "{}/ca-bundle.crt", + openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_DIR + )); + + let effective = policy_with_runtime_read_only( + &policy, + &[certificate.clone(), bundle.clone(), certificate.clone()], + ); + + assert_eq!(policy.filesystem.read_only, vec![PathBuf::from("/usr")]); + assert_eq!( + effective.filesystem.read_only, + vec![PathBuf::from("/usr"), certificate, bundle] + ); + } + + #[cfg(target_os = "linux")] + #[test] + #[allow(unsafe_code)] + fn runtime_ca_material_remains_readable_after_landlock_for_non_root_workload() { + let root = tempfile::tempdir_in("/tmp").unwrap(); + std::fs::set_permissions(root.path(), std::fs::Permissions::from_mode(0o755)).unwrap(); + let ca_directory = root.path().join("openshell-supervisor-ca"); + std::fs::create_dir(&ca_directory).unwrap(); + std::fs::set_permissions(&ca_directory, std::fs::Permissions::from_mode(0o755)).unwrap(); + let certificate = ca_directory.join("ca.crt"); + let bundle = ca_directory.join("ca-bundle.crt"); + let denied = root.path().join("not-authorized"); + for path in [&certificate, &bundle, &denied] { + std::fs::write(path, b"public certificate material").unwrap(); + std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o444)).unwrap(); + } + + let mut policy = policy_with_process(ProcessPolicy::default()); + policy.landlock = LandlockPolicy { + compatibility: openshell_core::policy::LandlockCompatibility::HardRequirement, + }; + let runtime_paths = + ca_runtime_read_only_paths(Some(&(certificate.clone(), bundle.clone()))); + let Ok(Some(prepared)) = prepare_child_sandbox(&policy, None, &runtime_paths) else { + return; + }; + + match unsafe { fork() }.expect("fork should succeed") { + ForkResult::Child => { + let dropped = if nix::unistd::geteuid().is_root() { + unsafe { + libc::setgroups(0, std::ptr::null()) == 0 + && libc::setgid(42_235) == 0 + && libc::setuid(42_234) == 0 + } + } else { + true + }; + let valid = dropped + && sandbox::linux::enforce(prepared).is_ok() + && std::fs::read(&certificate).is_ok() + && std::fs::read(&bundle).is_ok() + && std::fs::read(&denied).is_err(); + unsafe { libc::_exit(i32::from(!valid)) }; + } + ForkResult::Parent { child } => assert_eq!( + waitpid(child, None).expect("waitpid should succeed"), + WaitStatus::Exited(child, 0), + "Landlock must preserve non-root access only to admitted public CA material" + ), + } + } + #[cfg(unix)] #[test] fn validate_oci_workspace_rejects_restrictive_parent() { @@ -3890,63 +3709,64 @@ mod tests { assert!(stdout.contains("OPENSHELL_ENDPOINT=https://gateway.example.test")); } - #[test] - fn supervisor_identity_mount_target_uses_socket_parent() { - assert_eq!( - supervisor_identity_mount_target("/spiffe-workload-api/spire-agent.sock") - .expect("plain path should parse"), - Some(PathBuf::from("/spiffe-workload-api")) - ); - assert_eq!( - supervisor_identity_mount_target("unix:/spiffe-workload-api/spire-agent.sock") - .expect("unix path should parse"), - Some(PathBuf::from("/spiffe-workload-api")) - ); - } + #[tokio::test] + async fn transparent_mediation_removes_ambient_proxy_routing() { + let mut cmd = Command::new("/usr/bin/env"); + cmd.env_clear() + .stdin(StdStdio::null()) + .stdout(StdStdio::piped()) + .stderr(StdStdio::null()) + .env("PATH", "/usr/bin:/bin"); + for key in PROXY_ENV_VARS { + cmd.env(key, "http://ambient-proxy.invalid:3128"); + } - #[test] - fn supervisor_identity_mount_target_ignores_empty_socket_path() { - assert_eq!( - supervisor_identity_mount_target(" ").expect("empty path should be ignored"), - None - ); - } + strip_proxy_env(&mut cmd); - #[test] - fn supervisor_identity_mount_target_rejects_unhideable_endpoints() { - assert_eq!( - supervisor_identity_mount_target("tcp:127.0.0.1:8081") - .expect("tcp endpoint should not require mount hiding"), - None - ); - assert!(supervisor_identity_mount_target("spiffe-workload-api/spire-agent.sock").is_err()); - assert!(supervisor_identity_mount_target("/spire-agent.sock").is_err()); + let output = cmd.output().await.expect("spawn env"); + assert!(output.status.success()); + let stdout = String::from_utf8(output.stdout).expect("utf8"); + for key in PROXY_ENV_VARS { + assert!( + !stdout + .lines() + .any(|line| line.starts_with(&format!("{key}="))), + "{key} must not redirect a transparently mediated process" + ); + } + assert!(stdout.contains("PATH=/usr/bin:/bin")); } - #[test] - fn supervisor_identity_mount_target_rejects_shared_root_shadowing() { - for socket_path in [ - "/run/spire-agent.sock", - "/var/spire-agent.sock", - "/tmp/spire-agent.sock", - "/etc/spire-agent.sock", - ] { - let err = supervisor_identity_mount_target(socket_path) - .expect_err("shared root shadowing should be rejected"); - assert!(err.to_string().contains("dedicated subdirectory")); - } + // ---- Numeric UID tests (Phase 2) ---- - assert_eq!( - supervisor_identity_mount_target("/run/spire/spire-agent.sock") - .expect("dedicated subdirectory should be accepted"), - Some(PathBuf::from("/run/spire")) + // Even a failing setuid(0) probe synchronizes libc credentials across all + // threads. Other tests own seccomp-notified launcher threads in this same + // process; signaling those while they await their broker can deadlock the + // parallel harness. Re-exec just the credential probe, without those threads. + fn numeric_uid_probe_runs_in_child(test_name: &str) -> bool { + const MARKER: &str = "OPENSHELL_TEST_ISOLATED_NUMERIC_UID_PROBE"; + if std::env::var(MARKER).as_deref() == Ok(test_name) { + return true; + } + let output = std::process::Command::new(std::env::current_exe().expect("test executable")) + .args(["--exact", test_name, "--test-threads=1", "--nocapture"]) + .env(MARKER, test_name) + .output() + .expect("run isolated credential probe"); + assert!( + output.status.success(), + "isolated credential probe failed: {}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) ); + false } - // ---- Numeric UID tests (Phase 2) ---- - #[test] fn drop_privileges_accepts_numeric_uid() { + if !numeric_uid_probe_runs_in_child("process::tests::drop_privileges_accepts_numeric_uid") { + return; + } // When running as non-root, a numeric UID/GID that matches the // current process should succeed without any passwd lookup. if nix::unistd::geteuid().is_root() { @@ -3969,6 +3789,11 @@ mod tests { #[test] fn drop_privileges_numeric_uid_skips_initgroups() { + if !numeric_uid_probe_runs_in_child( + "process::tests::drop_privileges_numeric_uid_skips_initgroups", + ) { + return; + } // When running as non-root with a numeric user but group matches, // initgroups should not be called (guard: target_uid != geteuid()). if nix::unistd::geteuid().is_root() { diff --git a/crates/openshell-sandbox/src/pty.rs b/crates/openshell-sandbox/src/pty.rs new file mode 100644 index 0000000000..d887342306 --- /dev/null +++ b/crates/openshell-sandbox/src/pty.rs @@ -0,0 +1,144 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Workload-side PTY and audited pre-exec setup. + +use std::os::fd::RawFd; +use std::process::Command; + +use nix::pty::Winsize; +use nix::unistd::setsid; +use openshell_core::policy::SandboxPolicy; +#[cfg(unix)] +use std::os::unix::process::CommandExt as _; + +#[allow(unsafe_code)] +pub fn set_winsize(fd: RawFd, winsize: Winsize) -> std::io::Result<()> { + // SAFETY: fd is the owned PTY master and winsize is initialized. + let rc = unsafe { libc::ioctl(fd, libc::TIOCSWINSZ, &winsize) }; + if rc != 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) +} + +/// Install a pre-exec hook that gives the child a dedicated process group. +#[allow(unsafe_code)] +pub fn install_dedicated_process_group(command: &mut Command) { + // SAFETY: the hook invokes only the async-signal-safe setpgid syscall. + unsafe { + command.pre_exec(|| { + if libc::setpgid(0, 0) < 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) + }); + } +} + +#[allow(unsafe_code, clippy::useless_conversion)] +fn set_controlling_tty(fd: RawFd) -> std::io::Result<()> { + // SAFETY: fd is the slave PTY inherited by this pre-exec child. + let rc = unsafe { libc::ioctl(fd, libc::TIOCSCTTY.into(), 0) }; + if rc != 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) +} + +#[allow( + unsafe_code, + clippy::unnecessary_wraps, + reason = "pre-exec installation remains fallible as the prepared policy evolves" +)] +pub fn install_pre_exec( + command: &mut Command, + policy: SandboxPolicy, + _workdir: Option, + slave_fd: RawFd, + #[cfg(target_os = "linux")] prepared: Option, + #[cfg(target_os = "linux")] + child_hardening: openshell_isolation_interface::linux::child_seccomp::ChildHardeningProgram, +) -> anyhow::Result<()> { + #[cfg(target_os = "linux")] + let mut prepared = prepared; + #[cfg(target_os = "linux")] + let mut child_hardening = child_hardening; + // SAFETY: all allocations and policy compilation happened before spawn; + // the hook performs only the audited child transition. + unsafe { + command.pre_exec(move || { + setsid().map_err(|error| std::io::Error::other(error.to_string()))?; + set_controlling_tty(slave_fd)?; + enter_sandbox( + &policy, + #[cfg(target_os = "linux")] + prepared.take(), + #[cfg(target_os = "linux")] + &mut child_hardening, + ) + }); + } + Ok(()) +} + +#[allow( + unsafe_code, + clippy::unnecessary_wraps, + reason = "pre-exec installation remains fallible as the prepared policy evolves" +)] +pub fn install_pre_exec_no_pty( + command: &mut Command, + policy: SandboxPolicy, + _workdir: Option, + #[cfg(target_os = "linux")] prepared: Option, + #[cfg(target_os = "linux")] + child_hardening: openshell_isolation_interface::linux::child_seccomp::ChildHardeningProgram, +) -> anyhow::Result<()> { + #[cfg(target_os = "linux")] + let mut prepared = prepared; + #[cfg(target_os = "linux")] + let mut child_hardening = child_hardening; + // SAFETY: all allocations and policy compilation happened before spawn; + // the hook performs only the audited child transition. + unsafe { + command.pre_exec(move || { + if libc::setpgid(0, 0) != 0 { + return Err(std::io::Error::last_os_error()); + } + enter_sandbox( + &policy, + #[cfg(target_os = "linux")] + prepared.take(), + #[cfg(target_os = "linux")] + &mut child_hardening, + ) + }); + } + Ok(()) +} + +fn enter_sandbox( + policy: &SandboxPolicy, + #[cfg(target_os = "linux")] prepared: Option, + #[cfg(target_os = "linux")] + child_hardening: &mut openshell_isolation_interface::linux::child_seccomp::ChildHardeningProgram, +) -> std::io::Result<()> { + crate::process::harden_child_process() + .map_err(|error| std::io::Error::other(error.to_string()))?; + + #[cfg(target_os = "linux")] + if let Some(prepared) = prepared { + crate::sandbox::linux::enforce_capability_free(prepared, child_hardening) + .map_err(|error| std::io::Error::other(error.to_string()))?; + } + + #[cfg(not(target_os = "linux"))] + crate::sandbox::apply(policy, None) + .map_err(|error| std::io::Error::other(error.to_string()))?; + + #[cfg(target_os = "linux")] + let _ = policy; + + Ok(()) +} diff --git a/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs b/crates/openshell-sandbox/src/sandbox/linux/landlock.rs similarity index 76% rename from crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs rename to crates/openshell-sandbox/src/sandbox/linux/landlock.rs index 0a5bbfdc79..c18b633e75 100644 --- a/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/landlock.rs @@ -9,7 +9,7 @@ use landlock::{ }; use miette::{IntoDiagnostic, Result}; use openshell_core::policy::{LandlockCompatibility, SandboxPolicy}; -use std::os::fd::AsFd; +use std::os::fd::{AsFd, OwnedFd}; use std::path::{Path, PathBuf}; use tracing::debug; @@ -115,8 +115,8 @@ pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result Result { + prepare_capability_free_baseline_at(Path::new("/")) +} + +fn prepare_capability_free_baseline_at(root: &Path) -> Result { + // Unlike optional filesystem policy, self-protection must cover pathname + // truncation as well as opens. Never silently downgrade this ABI requirement. + let abi = ABI::V3; + let access = AccessFs::from_all(abi); + let mut ruleset = Ruleset::default() + .set_compatibility(CompatLevel::HardRequirement) + .handle_access(access) + .into_diagnostic()? + .create() + .into_diagnostic()?; + let entries = capability_free_baseline_entries(root)?; + if entries.is_empty() { + return Err(miette::miette!( + "capability-free Landlock baseline found no usable root entries" + )); + } + for (_, fd) in entries { + let allowed = access_for_path_fd(&fd, access, abi)?; + ruleset = ruleset + .add_rule(PathBeneath::new(fd, allowed)) + .into_diagnostic()?; + } + Ok(PreparedRuleset { + ruleset, + compatibility: LandlockCompatibility::HardRequirement, + }) +} + +fn capability_free_baseline_entries(root: &Path) -> Result> { + use rustix::fs::{Mode, OFlags, open, openat}; + const PRIVATE_ROOT: &str = ".openshell"; + + let root_fd = open( + root, + OFlags::PATH | OFlags::DIRECTORY | OFlags::CLOEXEC, + Mode::empty(), + ) + .into_diagnostic()?; + // The reserved root itself must not redirect private child mounts into an + // allowed subtree. Pin and validate it independently of the public entries. + // Absence is allowed for qualification before driver bootstrap is staged. + let _private_root = match openat( + &root_fd, + PRIVATE_ROOT, + OFlags::PATH | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) { + Ok(fd) => Some(fd), + Err(rustix::io::Errno::NOENT) => None, + Err(error) => { + return Err(miette::miette!( + "private sandbox root must be a real directory: {error}" + )); + } + }; + let mut entries = Vec::new(); + for entry in std::fs::read_dir(root).into_diagnostic()? { + let entry = entry.into_diagnostic()?; + if entry.file_name() == PRIVATE_ROOT { + continue; + } + // Open relative to the pinned root and classify this exact descriptor. + // O_PATH|O_NOFOLLOW opens a symlink itself, never its target. A root + // alias to `/` or `/.openshell` therefore cannot broaden the allowlist. + let fd = match openat( + &root_fd, + entry.file_name(), + OFlags::PATH | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) { + Ok(fd) => fd, + Err(rustix::io::Errno::NOENT | rustix::io::Errno::ACCESS) => continue, + Err(error) => return Err(error).into_diagnostic(), + }; + let stat = rustix::fs::fstat(&fd).into_diagnostic()?; + if rustix::fs::FileType::from_raw_mode(stat.st_mode) == rustix::fs::FileType::Symlink { + continue; + } + entries.push((entry.path(), fd)); + } + entries.sort_by(|left, right| left.0.cmp(&right.0)); + Ok(entries) +} + fn prepare_with_path_open_mode( policy: &SandboxPolicy, workdir: Option<&str>, @@ -144,14 +239,6 @@ fn prepare_with_path_open_mode( } if read_only.is_empty() && read_write.is_empty() { - if matches!( - policy.landlock.compatibility, - LandlockCompatibility::HardRequirement - ) { - miette::bail!( - "landlock.compatibility is hard_requirement but no filesystem paths are configured" - ); - } return Ok(None); } @@ -195,7 +282,10 @@ fn prepare_with_path_open_mode( } let total_paths = read_only.len() + read_write.len(); - let abi = ABI::V2; + // Read-only policy must also deny pathname truncation. The mandatory + // baseline already qualifies ABI v3; optional best-effort policy keeps its + // independent compatibility behavior for other callers. + let abi = ABI::V3; openshell_ocsf::ocsf_emit!( openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) .severity(openshell_ocsf::SeverityId::Informational) @@ -344,24 +434,13 @@ pub fn enforce(prepared: PreparedRuleset) -> Result<()> { Ok(()) } -/// Legacy single-phase apply. Kept for non-Linux platforms and tests. -/// On Linux, callers should use [`prepare`] + [`enforce`] for correct -/// privilege ordering. -#[allow(dead_code)] // Retained for backward compat; live callers use prepare+enforce. -pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> { - if let Some(prepared) = prepare(policy, workdir)? { - enforce(prepared)?; - } - Ok(()) -} - /// Tailor a rule's access mask to the inode referenced by its already-open FD. /// /// Landlock directory-only rights such as `ReadDir` are invalid for regular /// files and device nodes in hard-requirement mode. Classifying through the /// same `PathFd` used by the rule avoids a pathname TOCTOU race. fn access_for_path_fd( - path_fd: &PathFd, + path_fd: &impl AsFd, requested_access: BitFlags, abi: ABI, ) -> Result> { @@ -530,68 +609,108 @@ mod tests { panic!("hard_requirement should accept mixed directory and device paths: {err}"); } } + #[test] - fn prepare_hard_requirement_no_paths_aborts() { - // Zero configured paths under hard_requirement must fail startup rather - // than silently running without filesystem restrictions. - let policy = hard_requirement_policy(vec![], vec![]); - let Err(err) = prepare(&policy, None) else { - panic!("should abort with no paths"); - }; - let msg = err.to_string(); - assert!( - msg.contains("hard_requirement") && msg.contains("no filesystem paths"), - "error should explain the empty hard_requirement policy: {msg}" + fn capability_free_baseline_omits_only_private_root() { + let root = tempfile::tempdir().unwrap(); + for name in ["bin", "etc", "sandbox", ".openshell"] { + std::fs::create_dir(root.path().join(name)).unwrap(); + } + + std::os::unix::fs::symlink(root.path(), root.path().join("root-alias")).unwrap(); + std::os::unix::fs::symlink( + root.path().join(".openshell"), + root.path().join("private-alias"), + ) + .unwrap(); + let paths: Vec<_> = capability_free_baseline_entries(root.path()) + .unwrap() + .into_iter() + .map(|(path, _)| path) + .collect(); + assert_eq!( + paths, + ["bin", "etc", "sandbox"] + .map(|name| root.path().join(name)) + .to_vec() ); } #[test] - fn prepare_best_effort_no_paths_is_noop() { - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy { - read_only: vec![], - read_write: vec![], - include_workdir: false, - }, - network: NetworkPolicy::default(), - landlock: LandlockPolicy { - compatibility: LandlockCompatibility::BestEffort, - }, - process: ProcessPolicy::default(), - }; - let prepared = prepare(&policy, None).expect("best_effort no-op should succeed"); - assert!(prepared.is_none(), "no paths should produce no ruleset"); + fn capability_free_baseline_rejects_private_root_redirect() { + let root = tempfile::tempdir().unwrap(); + let public = root.path().join("public"); + let private = root.path().join(".openshell"); + std::fs::create_dir(&public).unwrap(); + std::fs::write(public.join("secret"), b"private mount contents").unwrap(); + std::os::unix::fs::symlink(&public, &private).unwrap(); + assert!(capability_free_baseline_entries(root.path()).is_err()); + std::fs::remove_file(&private).unwrap(); + std::fs::write(&private, b"not a directory").unwrap(); + assert!(capability_free_baseline_entries(root.path()).is_err()); } #[test] - fn prepare_include_workdir_counts_as_configured_path() { - // With no explicit paths but include_workdir set, the workdir must be - // treated as a configured path — so the zero-path abort must NOT fire. - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy { - read_only: vec![], - read_write: vec![], - include_workdir: true, - }, - network: NetworkPolicy::default(), - landlock: LandlockPolicy { - compatibility: LandlockCompatibility::HardRequirement, - }, - process: ProcessPolicy::default(), - }; - // Any error (e.g. Landlock unavailable on this host) is acceptable, but - // it must not be the "no filesystem paths" abort. - if let Err(err) = prepare(&policy, Some("/tmp")) { - let msg = err.to_string(); - assert!( - !msg.contains("no filesystem paths"), - "workdir should count as a configured path: {msg}" - ); + fn capability_free_baseline_denies_alias_reads_and_path_truncation() { + if !matches!(probe_availability(), LandlockAvailability::Available { abi } if abi >= 3) { + return; } + let root = tempfile::tempdir().unwrap(); + let public = root.path().join("public"); + let private = root.path().join(".openshell"); + std::fs::create_dir(&public).unwrap(); + std::fs::create_dir(&private).unwrap(); + std::fs::write(public.join("sentinel"), b"allowed").unwrap(); + std::fs::write(private.join("secret"), b"protected").unwrap(); + std::os::unix::fs::symlink(root.path(), root.path().join("root-alias")).unwrap(); + std::os::unix::fs::symlink(&private, root.path().join("private-alias")).unwrap(); + let path = root.path().to_path_buf(); + std::thread::spawn(move || { + enforce(prepare_capability_free_baseline_at(&path).unwrap()).unwrap(); + assert_eq!( + std::fs::read(path.join("public/sentinel")).unwrap(), + b"allowed" + ); + for name in [ + ".openshell/secret", + "root-alias/.openshell/secret", + "private-alias/secret", + ] { + let target = path.join(name); + assert_eq!( + std::fs::read(&target).unwrap_err().kind(), + std::io::ErrorKind::PermissionDenied + ); + let target = std::ffi::CString::new(target.as_os_str().as_encoded_bytes()).unwrap(); + // SAFETY: target is a live, NUL-terminated path. The syscall + // tests pathname truncation without opening a file first. + #[allow(unsafe_code)] + let result = unsafe { libc::truncate(target.as_ptr(), 0) }; + assert_eq!(result, -1); + assert_eq!( + std::io::Error::last_os_error().kind(), + std::io::ErrorKind::PermissionDenied + ); + } + let policy = hard_requirement_policy(vec![path.join("public")], Vec::new()); + enforce(prepare_current_user(&policy, None).unwrap().unwrap()).unwrap(); + let read_only = + std::ffi::CString::new(path.join("public/sentinel").as_os_str().as_encoded_bytes()) + .unwrap(); + // SAFETY: live NUL-terminated pathname; optional read-only policy + // must handle truncation independently of the protected baseline. + #[allow(unsafe_code)] + let truncated = unsafe { libc::truncate(read_only.as_ptr(), 0) }; + assert_eq!(truncated, -1); + assert_eq!( + std::io::Error::last_os_error().kind(), + std::io::ErrorKind::PermissionDenied + ); + }) + .join() + .unwrap(); + assert_eq!(std::fs::read(private.join("secret")).unwrap(), b"protected"); } - fn tailored_access(path: &Path, requested_access: BitFlags) -> BitFlags { let path_fd = PathFd::new(path).unwrap(); access_for_path_fd(&path_fd, requested_access, ABI::V2).unwrap() diff --git a/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs b/crates/openshell-sandbox/src/sandbox/linux/mod.rs similarity index 63% rename from crates/openshell-supervisor-process/src/sandbox/linux/mod.rs rename to crates/openshell-sandbox/src/sandbox/linux/mod.rs index 8afb3caef2..3f0084450e 100644 --- a/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/mod.rs @@ -14,7 +14,7 @@ use std::sync::Once; /// Opaque handle to a prepared-but-not-yet-enforced sandbox. /// Holds the Landlock ruleset with `PathFds` opened before child exec. pub struct PreparedSandbox { - landlock: Option, + landlock: Vec, policy: SandboxPolicy, } @@ -25,20 +25,41 @@ pub struct PreparedSandbox { pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result { let landlock = landlock::prepare(policy, workdir)?; Ok(PreparedSandbox { - landlock, + landlock: landlock.into_iter().collect(), policy: policy.clone(), }) } /// Phase 1 for already-unprivileged workloads. /// -/// Opens Landlock `PathFds` as the current UID. This is used by Kubernetes -/// sidecar mode, where the agent container already runs as the sandbox user. +/// Opens Landlock `PathFds` as the current workload UID. pub fn prepare_current_user( policy: &SandboxPolicy, workdir: Option<&str>, ) -> Result { let landlock = landlock::prepare_current_user(policy, workdir)?; + Ok(PreparedSandbox { + landlock: landlock.into_iter().collect(), + policy: policy.clone(), + }) +} + +/// Prepare the mandatory capability-free filesystem baseline plus the +/// optional user policy. +/// +/// The baseline is always a hard requirement. It grants access to each +/// top-level filesystem entry independently while deliberately omitting the +/// driver-owned `/.openshell` hierarchy. Applying the user ruleset after the +/// baseline intersects the two policies; it can narrow the baseline but can +/// never make the private hierarchy visible. +pub fn prepare_capability_free( + policy: &SandboxPolicy, + workdir: Option<&str>, +) -> Result { + let baseline = landlock::prepare_capability_free_baseline()?; + let user = landlock::prepare_current_user(policy, workdir)?; + let mut landlock = vec![baseline]; + landlock.extend(user); Ok(PreparedSandbox { landlock, policy: policy.clone(), @@ -50,9 +71,28 @@ pub fn prepare_current_user( /// Calls `restrict_self()` for Landlock and applies seccomp filters. /// Neither operation requires root privileges. pub fn enforce(prepared: PreparedSandbox) -> Result<()> { - if let Some(ruleset) = prepared.landlock { + for ruleset in prepared.landlock { + landlock::enforce(ruleset)?; + } + seccomp::apply(&prepared.policy)?; + Ok(()) +} + +/// Enforce the capability-free child filter stack. +/// +/// Landlock precedes sandbox-TGID self-protection. The ordinary workload +/// filter is installed last. The final filter blocks any later seccomp +/// installation, so this order is mandatory for capability-free children. +pub fn enforce_capability_free( + prepared: PreparedSandbox, + child_hardening: &mut openshell_isolation_interface::linux::child_seccomp::ChildHardeningProgram, +) -> Result<()> { + for ruleset in prepared.landlock { landlock::enforce(ruleset)?; } + child_hardening + .install() + .map_err(|error| miette::miette!("install child self-protection filter: {error}"))?; seccomp::apply(&prepared.policy)?; Ok(()) } @@ -62,15 +102,6 @@ pub fn apply_supervisor_prelude() -> Result<()> { seccomp::apply_supervisor_prelude() } -/// Legacy single-phase apply. Kept for backward compatibility. -/// New callers should use [`prepare`] + [`enforce`] for correct privilege ordering. -#[allow(dead_code)] // Retained for backward compat; live callers use prepare+enforce. -pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> { - landlock::apply(policy, workdir)?; - seccomp::apply(policy)?; - Ok(()) -} - /// Probe Landlock availability and emit OCSF logs from the parent process. /// /// This must be called **before** `pre_exec` / `fork()` so that the OCSF events @@ -99,60 +130,14 @@ pub fn log_sandbox_readiness(policy: &SandboxPolicy, workdir: Option<&str>) { let total_paths = read_only.len() + read_write.len(); if total_paths == 0 { - if matches!( - policy.landlock.compatibility, - openshell_core::policy::LandlockCompatibility::HardRequirement - ) { - // hard_requirement with no paths is fatal (see `landlock::prepare`). - // Emit a failure state so operators don't see a misleading "skipped" - // success event immediately before startup aborts. - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::High) - .status(openshell_ocsf::StatusId::Failure) - .state(openshell_ocsf::StateId::Other, "invalid") - .message( - "Landlock hard_requirement but no filesystem paths configured; \ - sandbox startup will abort" - .to_string(), - ) - .build() - ); - // Dual-emit a security finding for the unsafe policy (per OCSF - // guidance: pair the domain event with a DetectionFinding). - openshell_ocsf::ocsf_emit!( - openshell_ocsf::DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(openshell_ocsf::ActivityId::Open) - .severity(openshell_ocsf::SeverityId::High) - .confidence(openshell_ocsf::ConfidenceId::High) - .is_alert(true) - .finding_info( - openshell_ocsf::FindingInfo::new( - "landlock-hard-requirement-no-paths", - "Landlock Hard Requirement Without Paths", - ) - .with_desc( - "landlock.compatibility is hard_requirement but no filesystem \ - paths are configured; sandbox startup will abort.", - ), - ) - .message( - "Landlock hard_requirement with no filesystem paths configured".to_string(), - ) - .build() - ); - } else { - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Other, "skipped") - .message( - "Landlock filesystem sandbox skipped: no paths configured".to_string(), - ) - .build() - ); - } + openshell_ocsf::ocsf_emit!( + openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) + .severity(openshell_ocsf::SeverityId::Informational) + .status(openshell_ocsf::StatusId::Success) + .state(openshell_ocsf::StateId::Other, "skipped") + .message("Landlock filesystem sandbox skipped: no paths configured".to_string()) + .build() + ); return; } diff --git a/crates/openshell-supervisor-process/src/sandbox/linux/seccomp.rs b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs similarity index 88% rename from crates/openshell-supervisor-process/src/sandbox/linux/seccomp.rs rename to crates/openshell-sandbox/src/sandbox/linux/seccomp.rs index a9c67af95a..f3da87aeef 100644 --- a/crates/openshell-supervisor-process/src/sandbox/linux/seccomp.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs @@ -25,10 +25,9 @@ //! The risk is contained by existing sandbox layers: //! - **Privilege drop**: `CAP_NET_ADMIN` is not granted, so all write operations //! (add/delete routes, addresses, interfaces) fail with `EPERM` regardless. -//! - **Network namespace**: the sandboxed process sees only `lo` and one veth; -//! no host interfaces are visible. -//! - **nftables bypass rules**: all non-proxy traffic is rejected at the -//! netfilter level regardless of what the sandbox learns about its interfaces. +//! - **Driver outer fence**: direct workload egress is rejected outside this +//! process by Docker network-none, Kubernetes `NetworkPolicy`, or a NIC-less +//! VM. //! //! Every other netlink protocol (`NETLINK_SOCK_DIAG`, `NETLINK_NETFILTER`, //! `NETLINK_AUDIT`, `NETLINK_XFRM`, `NETLINK_GENERIC`, etc.) remains blocked. @@ -72,10 +71,10 @@ pub fn apply_supervisor_prelude() -> Result<()> { pub fn apply(policy: &SandboxPolicy) -> Result<()> { let allow_inet = matches!(policy.network.mode, NetworkMode::Proxy | NetworkMode::Allow); let main_filter = build_filter(allow_inet)?; - let clone3_filter = build_clone3_filter()?; + let compatibility_filter = build_compatibility_filter()?; set_no_new_privs()?; - apply_runtime_filters(&main_filter, &clone3_filter)?; + apply_runtime_filters(&main_filter, &compatibility_filter)?; Ok(()) } @@ -148,7 +147,7 @@ fn compile_filter( filter.try_into().into_diagnostic() } -/// Build a minimal BPF filter that blocks clone3 with ENOSYS. +/// Build a minimal BPF filter for unavailable process APIs. /// /// This is a separate filter from the main one because seccomp BPF cannot /// dereference the `struct clone_args *` pointer that clone3 takes as arg 0, @@ -156,27 +155,28 @@ fn compile_filter( /// unconditionally with ENOSYS so glibc falls back to the older clone /// syscall (where flags are a direct register argument and CAN be filtered). /// -/// glibc's clone3 wrapper checks for ENOSYS specifically — EPERM would be -/// treated as a hard failure and propagated to the caller instead of -/// triggering the clone fallback. -fn build_clone3_filter() -> Result { +/// glibc's clone3 wrapper and process launchers that opportunistically use +/// pidfds check for ENOSYS specifically. EPERM is treated as a hard policy +/// failure instead of triggering their portable fallback paths. +fn build_compatibility_filter() -> Result { let mut rules: BTreeMap> = BTreeMap::new(); rules.entry(libc::SYS_clone3).or_default(); + rules.entry(libc::SYS_pidfd_open).or_default(); compile_filter(rules, SeccompAction::Errno(libc::ENOSYS as u32)) } /// Install the sandbox seccomp filters in the required order. /// /// Order matters: -/// 1. Install the dedicated clone3 filter first so it can still call +/// 1. Install the compatibility filter first so it can still call /// `seccomp(SECCOMP_SET_MODE_FILTER)`. /// 2. Install the main filter second. It blocks further seccomp filter /// installation with `EPERM`, preserving the original hardening intent. fn apply_runtime_filters( main_filter: seccompiler::BpfProgramRef<'_>, - clone3_filter: seccompiler::BpfProgramRef<'_>, + compatibility_filter: seccompiler::BpfProgramRef<'_>, ) -> Result<()> { - apply_filter(clone3_filter).into_diagnostic()?; + apply_filter(compatibility_filter).into_diagnostic()?; apply_filter(main_filter).into_diagnostic()?; Ok(()) } @@ -227,8 +227,9 @@ fn build_filter_rules(allow_inet: bool) -> Result rules.entry(libc::SYS_process_vm_readv).or_default(); // Cross-process memory write (symmetric with process_vm_readv). rules.entry(libc::SYS_process_vm_writev).or_default(); - // Process handle acquisition, fd theft, and signalling via pidfd. - rules.entry(libc::SYS_pidfd_open).or_default(); + // Process fd theft and signalling via pidfd. pidfd_open is made + // unavailable with ENOSYS by the compatibility filter so runtimes can + // fall back without gaining a handle to the trusted sandbox boundary. rules.entry(libc::SYS_pidfd_getfd).or_default(); rules.entry(libc::SYS_pidfd_send_signal).or_default(); // Async I/O subsystem with extensive CVE history. @@ -279,7 +280,7 @@ fn build_filter_rules(allow_inet: bool) -> Result 0, // flags argument libc::CLONE_NEWUSER as u64, )?; - // clone3 is handled by a separate filter — see build_clone3_filter(). + // clone3 is handled by the ENOSYS compatibility filter. // seccomp(SECCOMP_SET_MODE_FILTER) would let sandboxed code replace the active filter. let condition = SeccompCondition::new( @@ -426,7 +427,6 @@ mod tests { libc::SYS_bpf, libc::SYS_process_vm_readv, libc::SYS_process_vm_writev, - libc::SYS_pidfd_open, libc::SYS_pidfd_getfd, libc::SYS_pidfd_send_signal, libc::SYS_io_uring_setup, @@ -552,19 +552,25 @@ mod tests { } #[test] - fn clone3_filter_compiles_and_blocks_clone3() { - let bpf = build_clone3_filter(); - assert!(bpf.is_ok(), "clone3 ENOSYS filter should compile"); + fn compatibility_filter_compiles() { + let bpf = build_compatibility_filter(); + assert!( + bpf.is_ok(), + "process API compatibility filter should compile" + ); } #[test] - fn clone3_not_in_main_filter() { - // clone3 must NOT be in the main filter; it has its own ENOSYS filter. + fn compatibility_syscalls_are_not_in_main_filter() { + // These APIs must NOT be in the EPERM filter; the compatibility filter + // reports them unavailable so process launchers can fall back. let filter_rules = build_filter_rules(true).unwrap(); - assert!( - !filter_rules.contains_key(&libc::SYS_clone3), - "clone3 should not be in the main filter — it uses a separate ENOSYS filter" - ); + for syscall in [libc::SYS_clone3, libc::SYS_pidfd_open] { + assert!( + !filter_rules.contains_key(&syscall), + "syscall {syscall} should use the ENOSYS compatibility filter" + ); + } } // --- Behavioral tests --- @@ -611,10 +617,10 @@ mod tests { unsafe fn install_runtime_filters_in_child( main_filter: &seccompiler::BpfProgram, - clone3_filter: &seccompiler::BpfProgram, + compatibility_filter: &seccompiler::BpfProgram, ) { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); - if let Err(err) = apply_runtime_filters(main_filter, clone3_filter) { + if let Err(err) = apply_runtime_filters(main_filter, compatibility_filter) { let msg = format!("failed to install runtime seccomp filters: {err}\n"); libc::write(2, msg.as_ptr().cast(), msg.len()); libc::_exit(1); @@ -702,13 +708,13 @@ mod tests { // clone3 uses a separate filter that returns ENOSYS (not EPERM) so // glibc falls back to clone. let main_filter = build_filter(true).unwrap(); - let clone3_filter = build_clone3_filter().unwrap(); - // Apply in the same order as apply(): clone3 filter first, main filter second. + let compatibility_filter = build_compatibility_filter().unwrap(); + // Apply in the same order as apply(): compatibility filter first, main filter second. let pid = unsafe { libc::fork() }; assert!(pid >= 0, "fork failed"); if pid == 0 { unsafe { - install_runtime_filters_in_child(&main_filter, &clone3_filter); + install_runtime_filters_in_child(&main_filter, &compatibility_filter); let ret = libc::syscall(libc::SYS_clone3, 0 as libc::c_ulong, 0 as libc::c_ulong); let errno = *libc::__errno_location(); if ret == -1 && errno == libc::ENOSYS { @@ -728,17 +734,23 @@ mod tests { ); } + #[test] + fn behavioral_pidfd_open_returns_enosys() { + let filter = build_compatibility_filter().unwrap(); + unsafe { assert_blocked_in_child(&filter, libc::SYS_pidfd_open, libc::ENOSYS) }; + } + #[test] fn behavioral_third_filter_install_blocked_after_startup() { let main_filter = build_filter(true).unwrap(); - let clone3_filter = build_clone3_filter().unwrap(); - let third_filter = build_clone3_filter().unwrap(); + let compatibility_filter = build_compatibility_filter().unwrap(); + let third_filter = build_compatibility_filter().unwrap(); let pid = unsafe { libc::fork() }; assert!(pid >= 0, "fork failed"); if pid == 0 { unsafe { - install_runtime_filters_in_child(&main_filter, &clone3_filter); + install_runtime_filters_in_child(&main_filter, &compatibility_filter); match apply_filter(&third_filter) { Err(seccompiler::Error::Seccomp(e)) if e.raw_os_error() == Some(libc::EPERM) => @@ -838,4 +850,43 @@ mod tests { "socket(AF_NETLINK, SOCK_RAW, NETLINK_SOCK_DIAG) should be blocked with EPERM" ); } + + #[test] + fn behavioral_block_mode_denies_inet_and_packet_sockets() { + let filter = build_filter(false).unwrap(); + let pid = unsafe { libc::fork() }; + assert!(pid >= 0, "fork failed"); + if pid == 0 { + unsafe { + libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); + apply_filter(&filter).expect("apply block-mode filter"); + for (domain, socket_type, protocol) in [ + (libc::AF_INET, libc::SOCK_STREAM, 0), + (libc::AF_INET6, libc::SOCK_DGRAM, 0), + (libc::AF_PACKET, libc::SOCK_RAW, 0), + ] { + let fd = libc::socket(domain, socket_type, protocol); + let errno = *libc::__errno_location(); + if fd >= 0 || errno != libc::EPERM { + if fd >= 0 { + libc::close(fd); + } + libc::_exit(1); + } + } + let unix_fd = libc::socket(libc::AF_UNIX, libc::SOCK_STREAM, 0); + if unix_fd < 0 { + libc::_exit(1); + } + libc::close(unix_fd); + libc::_exit(0); + } + } + let mut status: libc::c_int = 0; + unsafe { libc::waitpid(pid, &mut status, 0) }; + assert!( + unsafe { libc::WIFEXITED(status) && libc::WEXITSTATUS(status) == 0 }, + "block mode must deny IPv4, IPv6, and packet sockets while retaining Unix IPC" + ); + } } diff --git a/crates/openshell-supervisor-process/src/sandbox/mod.rs b/crates/openshell-sandbox/src/sandbox/mod.rs similarity index 87% rename from crates/openshell-supervisor-process/src/sandbox/mod.rs rename to crates/openshell-sandbox/src/sandbox/mod.rs index ff44f8ba10..bcdb800f79 100644 --- a/crates/openshell-supervisor-process/src/sandbox/mod.rs +++ b/crates/openshell-sandbox/src/sandbox/mod.rs @@ -4,6 +4,7 @@ //! Platform sandboxing implementation. use miette::Result; +#[cfg(not(target_os = "linux"))] use openshell_core::policy::SandboxPolicy; #[cfg(target_os = "linux")] @@ -16,15 +17,9 @@ pub mod linux; /// Returns an error if the sandbox cannot be applied. // On Linux the spawn path uses `prepare`+`enforce` directly; this single-phase // apply is only invoked from the non-Linux spawn_impl. -#[cfg_attr(target_os = "linux", allow(dead_code))] -#[cfg_attr(not(target_os = "linux"), allow(clippy::unnecessary_wraps))] +#[cfg(not(target_os = "linux"))] +#[allow(clippy::unnecessary_wraps)] pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> { - #[cfg(target_os = "linux")] - { - linux::apply(policy, workdir) - } - - #[cfg(not(target_os = "linux"))] { let _ = (policy, workdir); openshell_ocsf::ocsf_emit!( diff --git a/crates/openshell-sandbox/src/sidecar_control.rs b/crates/openshell-sandbox/src/sidecar_control.rs deleted file mode 100644 index 11f3e68e23..0000000000 --- a/crates/openshell-sandbox/src/sidecar_control.rs +++ /dev/null @@ -1,1210 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Local control channel for Kubernetes sidecar topology. -//! -//! The network sidecar owns gateway credentials. The process supervisor in the -//! agent container connects over this Unix socket to receive policy/provider -//! state without mounting gateway credentials into the agent container. - -use miette::{IntoDiagnostic, Result, WrapErr}; -use prost::Message; -use serde::{Deserialize, Serialize}; -use std::collections::HashMap; -use std::path::{Path, PathBuf}; -use std::sync::{Arc, RwLock}; -use std::time::Duration; -use tokio::io::{AsyncBufReadExt, AsyncWrite, AsyncWriteExt, BufReader}; -use tokio::net::UnixListener; -use tokio::net::unix::OwnedWriteHalf; -use tokio::sync::{Mutex, broadcast, mpsc}; -use tracing::{debug, info, warn}; - -#[derive(Debug, Clone)] -pub struct BootstrapData { - pub policy_proto: openshell_core::proto::SandboxPolicy, - pub provider_env_revision: u64, - pub provider_env_generation: u64, - pub provider_child_env: HashMap, - pub agent_proposals_enabled: bool, - pub proxy_ca_cert_path: Option, - pub proxy_ca_bundle_path: Option, -} - -#[derive(Debug, Clone)] -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -pub struct EntrypointStarted { - pub pid: u32, - pub start_session: bool, - pub instance_id: String, - pub exit_code: Option, - pub finalized: bool, -} - -#[derive(Debug, Clone, Copy)] -pub struct ExpectedPeer { - pub uid: u32, - pub gid: u32, -} - -#[derive(Debug, Clone)] -pub enum ControlUpdate { - ProviderEnv { - revision: u64, - generation: u64, - provider_child_env: HashMap, - }, - Policy { - policy_proto: Box, - policy_hash: String, - config_revision: u64, - }, - AgentProposals { - enabled: bool, - config_revision: u64, - }, - MainProcessExitAck { - instance_id: String, - }, -} - -#[derive(Clone)] -pub struct Publisher { - state: Arc>, - updates: broadcast::Sender, -} - -impl Publisher { - pub fn publish_provider_env(&self, revision: u64, provider_child_env: HashMap) { - let mut state = self.state.write().expect("sidecar control state poisoned"); - if revision == state.provider_env_revision { - return; - } - state.provider_env_revision = revision; - state.provider_env_generation = state - .provider_env_generation - .checked_add(1) - .expect("sidecar provider environment generation overflow"); - state.provider_child_env.clone_from(&provider_child_env); - - // Keep generation assignment, bootstrap state, and publication under - // one lock so cloned publishers cannot emit generations out of order. - let _ = self.updates.send(WireServerMessage::ProviderEnvUpdated { - revision, - generation: state.provider_env_generation, - provider_child_env, - }); - } - - pub fn publish_policy( - &self, - policy_proto: openshell_core::proto::SandboxPolicy, - policy_hash: String, - config_revision: u64, - ) { - { - let mut state = self.state.write().expect("sidecar control state poisoned"); - state.policy_proto = policy_proto.clone(); - } - - let _ = self.updates.send(WireServerMessage::PolicyUpdated { - policy_proto: policy_proto.encode_to_vec(), - policy_hash, - config_revision, - }); - } - - pub fn publish_agent_proposals(&self, enabled: bool, config_revision: u64) { - { - let mut state = self.state.write().expect("sidecar control state poisoned"); - if state.agent_proposals_enabled == enabled { - return; - } - state.agent_proposals_enabled = enabled; - } - - let _ = self.updates.send(WireServerMessage::AgentProposalsUpdated { - enabled, - config_revision, - }); - } - - #[cfg(any(target_os = "linux", test))] - pub fn publish_main_process_exit_ack(&self, instance_id: String) { - let _ = self - .updates - .send(WireServerMessage::MainProcessExitAck { instance_id }); - } -} - -pub struct ServerHandle { - publisher: Publisher, - #[cfg_attr(not(target_os = "linux"), allow(dead_code))] - entrypoint_rx: mpsc::Receiver, - connection_task: tokio::task::JoinHandle<()>, -} - -impl ServerHandle { - pub fn publisher(&self) -> Publisher { - self.publisher.clone() - } - - #[cfg(test)] - pub fn into_entrypoint_receiver(self) -> mpsc::Receiver { - self.entrypoint_rx - } - - #[cfg_attr(not(target_os = "linux"), allow(dead_code))] - pub fn into_runtime_parts( - self, - ) -> ( - mpsc::Receiver, - tokio::task::JoinHandle<()>, - ) { - (self.entrypoint_rx, self.connection_task) - } -} - -pub struct ProcessConnection { - pub writer: Arc>, - pub updates: mpsc::UnboundedReceiver, - pub closed: tokio::sync::oneshot::Receiver<()>, -} - -#[derive(Debug, Serialize, Deserialize)] -#[serde(tag = "type", rename_all = "snake_case")] -enum WireClientMessage { - BootstrapRequest { supervisor_pid: u32 }, - EntrypointStarted { pid: u32, instance_id: String }, - MainProcessExited { instance_id: String, exit_code: i32 }, - MainProcessFinalized { instance_id: String }, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(tag = "type", rename_all = "snake_case")] -enum WireServerMessage { - BootstrapResponse { - policy_proto: Vec, - provider_env_revision: u64, - provider_env_generation: u64, - provider_child_env: HashMap, - agent_proposals_enabled: bool, - proxy_ca_cert_path: Option, - proxy_ca_bundle_path: Option, - }, - ProviderEnvUpdated { - revision: u64, - generation: u64, - provider_child_env: HashMap, - }, - PolicyUpdated { - policy_proto: Vec, - policy_hash: String, - config_revision: u64, - }, - AgentProposalsUpdated { - enabled: bool, - config_revision: u64, - }, - MainProcessExitAck { - instance_id: String, - }, -} - -impl BootstrapData { - #[cfg_attr(not(target_os = "linux"), allow(dead_code))] - fn to_wire(&self) -> WireServerMessage { - WireServerMessage::BootstrapResponse { - policy_proto: self.policy_proto.encode_to_vec(), - provider_env_revision: self.provider_env_revision, - provider_env_generation: self.provider_env_generation, - provider_child_env: self.provider_child_env.clone(), - agent_proposals_enabled: self.agent_proposals_enabled, - proxy_ca_cert_path: self - .proxy_ca_cert_path - .as_ref() - .map(|path| path.display().to_string()), - proxy_ca_bundle_path: self - .proxy_ca_bundle_path - .as_ref() - .map(|path| path.display().to_string()), - } - } -} - -impl TryFrom for BootstrapData { - type Error = miette::Report; - - fn try_from(message: WireServerMessage) -> Result { - let WireServerMessage::BootstrapResponse { - policy_proto, - provider_env_revision, - provider_env_generation, - provider_child_env, - agent_proposals_enabled, - proxy_ca_cert_path, - proxy_ca_bundle_path, - } = message - else { - return Err(miette::miette!( - "expected sidecar bootstrap response, received update message" - )); - }; - - let policy_proto = openshell_core::proto::SandboxPolicy::decode(policy_proto.as_slice()) - .into_diagnostic() - .wrap_err("failed to decode sidecar bootstrap policy")?; - let policy_proto = canonicalize_sidecar_policy( - policy_proto, - "sidecar bootstrap policy failed validation", - )?; - - Ok(Self { - policy_proto, - provider_env_revision, - provider_env_generation, - provider_child_env, - agent_proposals_enabled, - proxy_ca_cert_path: proxy_ca_cert_path.map(PathBuf::from), - proxy_ca_bundle_path: proxy_ca_bundle_path.map(PathBuf::from), - }) - } -} - -impl TryFrom for ControlUpdate { - type Error = miette::Report; - - fn try_from(message: WireServerMessage) -> Result { - match message { - WireServerMessage::ProviderEnvUpdated { - revision, - generation, - provider_child_env, - } => Ok(Self::ProviderEnv { - revision, - generation, - provider_child_env, - }), - WireServerMessage::PolicyUpdated { - policy_proto, - policy_hash, - config_revision, - } => { - let policy_proto = - openshell_core::proto::SandboxPolicy::decode(policy_proto.as_slice()) - .into_diagnostic() - .wrap_err("failed to decode sidecar policy update")?; - let policy_proto = canonicalize_sidecar_policy( - policy_proto, - "sidecar policy update failed validation", - )?; - Ok(Self::Policy { - policy_proto: Box::new(policy_proto), - policy_hash, - config_revision, - }) - } - WireServerMessage::AgentProposalsUpdated { - enabled, - config_revision, - } => Ok(Self::AgentProposals { - enabled, - config_revision, - }), - WireServerMessage::MainProcessExitAck { instance_id } => { - Ok(Self::MainProcessExitAck { instance_id }) - } - WireServerMessage::BootstrapResponse { .. } => Err(miette::miette!( - "unexpected sidecar bootstrap response after initial handshake" - )), - } - } -} - -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -pub fn spawn_server( - path: &Path, - bootstrap: BootstrapData, - expected_peer: ExpectedPeer, -) -> Result { - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent) - .into_diagnostic() - .wrap_err_with(|| { - format!( - "failed to create sidecar control socket dir {}", - parent.display() - ) - })?; - } - match std::fs::remove_file(path) { - Ok(()) => {} - Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} - Err(err) => { - return Err(err).into_diagnostic().wrap_err_with(|| { - format!( - "failed to remove stale sidecar control socket {}", - path.display() - ) - }); - } - } - - let listener = UnixListener::bind(path) - .into_diagnostic() - .wrap_err_with(|| format!("failed to bind sidecar control socket {}", path.display()))?; - - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt; - std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o660)) - .into_diagnostic() - .wrap_err_with(|| { - format!( - "failed to set permissions on sidecar control socket {}", - path.display() - ) - })?; - } - - let state = Arc::new(RwLock::new(bootstrap)); - let (updates, _) = broadcast::channel(32); - let (entrypoint_tx, entrypoint_rx) = mpsc::channel(8); - let publisher = Publisher { - state: state.clone(), - updates: updates.clone(), - }; - - let connection_task = tokio::spawn(accept_authoritative_connection( - listener, - path.to_path_buf(), - expected_peer, - state, - updates, - entrypoint_tx, - )); - info!(path = %path.display(), "Sidecar control socket listening"); - - Ok(ServerHandle { - publisher, - entrypoint_rx, - connection_task, - }) -} - -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -async fn accept_authoritative_connection( - listener: UnixListener, - socket_path: PathBuf, - expected_peer: ExpectedPeer, - state: Arc>, - updates: broadcast::Sender, - entrypoint_tx: mpsc::Sender, -) { - let stream = match listener.accept().await { - Ok((stream, _addr)) => stream, - Err(err) => { - warn!(error = %err, "Failed to accept authoritative sidecar control connection"); - return; - } - }; - - // The process supervisor connects before it launches the workload. Drop - // the listener and unlink its pathname after that first accept so workload - // processes can neither open a second control channel nor impersonate a - // restarted server at the trusted path. - drop(listener); - if let Err(err) = std::fs::remove_file(&socket_path) - && err.kind() != std::io::ErrorKind::NotFound - { - warn!( - path = %socket_path.display(), - error = %err, - "Failed to unlink accepted sidecar control socket" - ); - } - - if let Err(err) = handle_connection(stream, expected_peer, state, updates, entrypoint_tx).await - { - warn!(error = %err, "Authoritative sidecar control connection closed"); - } -} - -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -async fn handle_connection( - stream: tokio::net::UnixStream, - expected_peer: ExpectedPeer, - state: Arc>, - updates: broadcast::Sender, - entrypoint_tx: mpsc::Sender, -) -> Result<()> { - let credentials = stream - .peer_cred() - .into_diagnostic() - .wrap_err("failed to read sidecar control peer credentials")?; - if credentials.uid() != expected_peer.uid || credentials.gid() != expected_peer.gid { - return Err(miette::miette!( - "sidecar control peer identity mismatch: expected uid:gid {}:{}, got {}:{}", - expected_peer.uid, - expected_peer.gid, - credentials.uid(), - credentials.gid(), - )); - } - let peer_pid = credentials - .pid() - .and_then(|pid| u32::try_from(pid).ok()) - .ok_or_else(|| miette::miette!("sidecar control peer PID is unavailable"))?; - - let (reader, mut writer) = stream.into_split(); - let mut lines = BufReader::new(reader).lines(); - - let first_line = - lines.next_line().await.into_diagnostic()?.ok_or_else(|| { - miette::miette!("sidecar control client disconnected before bootstrap") - })?; - match decode_client_message(&first_line)? { - WireClientMessage::BootstrapRequest { supervisor_pid } => { - if supervisor_pid == 0 || supervisor_pid != peer_pid { - return Err(miette::miette!( - "sidecar bootstrap PID mismatch: peer PID {peer_pid}, claimed PID {supervisor_pid}" - )); - } - entrypoint_tx - .send(EntrypointStarted { - pid: supervisor_pid, - start_session: false, - instance_id: String::new(), - exit_code: None, - finalized: false, - }) - .await - .map_err(|_| miette::miette!("sidecar entrypoint receiver closed"))?; - } - WireClientMessage::EntrypointStarted { .. } - | WireClientMessage::MainProcessExited { .. } - | WireClientMessage::MainProcessFinalized { .. } => { - return Err(miette::miette!( - "sidecar control client sent entrypoint event before bootstrap" - )); - } - } - - // Subscribe before taking the bootstrap snapshot so an update can neither - // be missed between the snapshot and the live update stream nor omitted - // from the snapshot itself. - let mut update_rx = updates.subscribe(); - let bootstrap = { - let state = state.read().expect("sidecar control state poisoned"); - state.to_wire() - }; - write_json_line(&mut writer, &bootstrap).await?; - - loop { - tokio::select! { - line = lines.next_line() => { - let Some(line) = line.into_diagnostic()? else { - return Ok(()); - }; - match decode_client_message(&line)? { - WireClientMessage::BootstrapRequest { .. } => { - debug!("Ignoring duplicate sidecar bootstrap request"); - } - WireClientMessage::EntrypointStarted { pid, instance_id } => { - if pid == 0 { - warn!("Ignoring sidecar entrypoint event with pid=0"); - continue; - } - entrypoint_tx - .send(EntrypointStarted { - pid, - start_session: true, - instance_id, - exit_code: None, - finalized: false, - }) - .await - .map_err(|_| miette::miette!("sidecar entrypoint receiver closed"))?; - } - WireClientMessage::MainProcessExited { - instance_id, - exit_code, - } => { - entrypoint_tx - .send(EntrypointStarted { - pid: 0, - start_session: false, - instance_id, - exit_code: Some(exit_code), - finalized: false, - }) - .await - .map_err(|_| miette::miette!("sidecar entrypoint receiver closed"))?; - } - WireClientMessage::MainProcessFinalized { instance_id } => { - entrypoint_tx - .send(EntrypointStarted { - pid: 0, - start_session: false, - instance_id, - exit_code: None, - finalized: true, - }) - .await - .map_err(|_| miette::miette!("sidecar entrypoint receiver closed"))?; - } - } - } - update = update_rx.recv() => { - match update { - Ok(message) => write_json_line(&mut writer, &message).await?, - Err(broadcast::error::RecvError::Lagged(skipped)) => { - warn!(skipped, "Sidecar control client lagged behind updates"); - } - Err(broadcast::error::RecvError::Closed) => return Ok(()), - } - } - } - } -} - -pub async fn connect_process_client( - path: &Path, - timeout: Duration, -) -> Result<(BootstrapData, ProcessConnection)> { - let stream = connect_with_retry(path, timeout).await?; - let (reader, mut writer) = stream.into_split(); - write_json_line( - &mut writer, - &WireClientMessage::BootstrapRequest { - supervisor_pid: std::process::id(), - }, - ) - .await?; - - let mut lines = BufReader::new(reader).lines(); - let first_line = lines - .next_line() - .await - .into_diagnostic()? - .ok_or_else(|| miette::miette!("sidecar control closed before bootstrap response"))?; - let bootstrap = BootstrapData::try_from(decode_server_message(&first_line)?)?; - - let (update_tx, updates) = mpsc::unbounded_channel(); - let (closed_tx, closed) = tokio::sync::oneshot::channel(); - tokio::spawn(async move { - while let Ok(Some(line)) = lines.next_line().await { - match decode_server_message(&line).and_then(ControlUpdate::try_from) { - Ok(update) => { - if update_tx.send(update).is_err() { - break; - } - } - Err(err) => { - warn!(error = %err, "Ignoring invalid sidecar control update"); - } - } - } - let _ = closed_tx.send(()); - }); - - Ok(( - bootstrap, - ProcessConnection { - writer: Arc::new(Mutex::new(writer)), - updates, - closed, - }, - )) -} - -async fn connect_with_retry(path: &Path, timeout: Duration) -> Result { - let deadline = tokio::time::Instant::now() + timeout; - loop { - match tokio::net::UnixStream::connect(path).await { - Ok(stream) => return Ok(stream), - Err(err) if tokio::time::Instant::now() < deadline => { - debug!( - path = %path.display(), - error = %err, - "Waiting for sidecar control socket" - ); - tokio::time::sleep(Duration::from_millis(100)).await; - } - Err(err) => { - return Err(err).into_diagnostic().wrap_err_with(|| { - format!( - "timed out waiting for sidecar control socket {}", - path.display() - ) - }); - } - } - } -} - -pub async fn send_entrypoint_started( - writer: &Arc>, - pid: u32, - instance_id: String, -) -> Result<()> { - let message = WireClientMessage::EntrypointStarted { pid, instance_id }; - let mut writer = writer.lock().await; - write_json_line(&mut *writer, &message).await -} - -pub async fn send_main_process_exited( - writer: &Arc>, - instance_id: String, - exit_code: i32, -) -> Result<()> { - let message = WireClientMessage::MainProcessExited { - instance_id, - exit_code, - }; - let mut writer = writer.lock().await; - write_json_line(&mut *writer, &message).await -} - -pub async fn send_main_process_finalized( - writer: &Arc>, - instance_id: String, -) -> Result<()> { - let message = WireClientMessage::MainProcessFinalized { instance_id }; - let mut writer = writer.lock().await; - write_json_line(&mut *writer, &message).await -} - -async fn write_json_line(writer: &mut W, value: &T) -> Result<()> -where - W: AsyncWrite + Unpin + Send, - T: Serialize + Sync, -{ - let bytes = serde_json::to_vec(value).into_diagnostic()?; - writer.write_all(&bytes).await.into_diagnostic()?; - writer.write_all(b"\n").await.into_diagnostic()?; - writer.flush().await.into_diagnostic()?; - Ok(()) -} - -#[cfg_attr(not(target_os = "linux"), allow(dead_code))] -fn decode_client_message(line: &str) -> Result { - serde_json::from_str(line) - .into_diagnostic() - .wrap_err("failed to decode sidecar client message") -} - -fn decode_server_message(line: &str) -> Result { - serde_json::from_str(line) - .into_diagnostic() - .wrap_err("failed to decode sidecar server message") -} - -fn canonicalize_sidecar_policy( - policy: openshell_core::proto::SandboxPolicy, - error_message: &'static str, -) -> Result { - // Bootstrap and update messages must expose the same canonical typed - // policy to every process-supervisor consumer. Keep validation details - // out of this channel error because they can contain authored values. - openshell_policy::validate_and_canonicalize_sandbox_policy(policy) - .map_err(|_| miette::miette!(error_message)) -} - -#[cfg(test)] -mod tests { - use super::*; - use openshell_core::proto::{McpOptions, NetworkEndpoint, NetworkPolicyRule, SandboxPolicy}; - - fn defaultable_mcp_policy(mcp: Option) -> SandboxPolicy { - let mut policy = openshell_policy::restrictive_default_policy(); - policy.network_policies.insert( - "mcp".to_string(), - NetworkPolicyRule { - name: "mcp".to_string(), - endpoints: vec![NetworkEndpoint { - host: "mcp.example.com".to_string(), - port: 443, - protocol: "mcp".to_string(), - mcp, - rules: vec![openshell_core::proto::L7Rule { - allow: Some(openshell_core::proto::L7Allow { - method: "tools/list".to_string(), - ..Default::default() - }), - }], - ..Default::default() - }], - ..Default::default() - }, - ); - policy - } - - fn mcp_versions(policy: &SandboxPolicy) -> &[String] { - policy.network_policies["mcp"].endpoints[0] - .mcp - .as_ref() - .expect("canonical MCP options") - .versions - .as_slice() - } - - fn bootstrap_message(policy: &SandboxPolicy) -> WireServerMessage { - WireServerMessage::BootstrapResponse { - policy_proto: policy.encode_to_vec(), - provider_env_revision: 0, - provider_env_generation: 0, - provider_child_env: HashMap::new(), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - } - } - - fn policy_update_message(policy: &SandboxPolicy) -> WireServerMessage { - WireServerMessage::PolicyUpdated { - policy_proto: policy.encode_to_vec(), - policy_hash: "hash".to_string(), - config_revision: 1, - } - } - - fn current_peer() -> ExpectedPeer { - ExpectedPeer { - uid: nix::unistd::Uid::current().as_raw(), - gid: nix::unistd::Gid::current().as_raw(), - } - } - - #[test] - fn policy_messages_canonicalize_defaultable_mcp_versions() { - for raw in [ - defaultable_mcp_policy(None), - defaultable_mcp_policy(Some(McpOptions::default())), - ] { - let bootstrap = BootstrapData::try_from(bootstrap_message(&raw)) - .expect("defaultable MCP policy must pass bootstrap ingress"); - assert_eq!(mcp_versions(&bootstrap.policy_proto), ["2025-11-25"]); - - let update = ControlUpdate::try_from(policy_update_message(&raw)) - .expect("defaultable MCP policy must pass update ingress"); - let ControlUpdate::Policy { policy_proto, .. } = update else { - panic!("expected policy update"); - }; - assert_eq!(mcp_versions(&policy_proto), ["2025-11-25"]); - } - } - - #[test] - fn policy_messages_reject_invalid_mcp_versions_without_echoing_values() { - let invalid = defaultable_mcp_policy(Some(McpOptions { - versions: vec!["latest".to_string()], - ..Default::default() - })); - - let bootstrap_error = BootstrapData::try_from(bootstrap_message(&invalid)) - .expect_err("invalid MCP policy must not pass bootstrap ingress") - .to_string(); - assert_eq!( - bootstrap_error, - "sidecar bootstrap policy failed validation" - ); - assert!(!bootstrap_error.contains("latest")); - - let update_error = ControlUpdate::try_from(policy_update_message(&invalid)) - .expect_err("invalid MCP policy must not pass update ingress") - .to_string(); - assert_eq!(update_error, "sidecar policy update failed validation"); - assert!(!update_error.contains("latest")); - } - - #[tokio::test] - async fn bootstrap_round_trips_policy_and_provider_env() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let mut env = HashMap::new(); - env.insert("GITHUB_TOKEN".to_string(), "secret".to_string()); - let bootstrap = BootstrapData { - policy_proto: SandboxPolicy { - version: 7, - ..SandboxPolicy::default() - }, - provider_env_revision: 3, - provider_env_generation: 0, - provider_child_env: env.clone(), - agent_proposals_enabled: true, - proxy_ca_cert_path: Some(PathBuf::from("/tmp/ca.pem")), - proxy_ca_bundle_path: Some(PathBuf::from("/tmp/bundle.pem")), - }; - - let _server = spawn_server(&socket, bootstrap, current_peer()).unwrap(); - let (received, _connection) = connect_process_client(&socket, Duration::from_secs(1)) - .await - .unwrap(); - - assert_eq!(received.policy_proto.version, 7); - assert_eq!(received.provider_env_revision, 3); - assert_eq!(received.provider_env_generation, 0); - assert_eq!(received.provider_child_env, env); - assert!(received.agent_proposals_enabled); - assert_eq!( - received.proxy_ca_cert_path, - Some(PathBuf::from("/tmp/ca.pem")) - ); - assert_eq!( - received.proxy_ca_bundle_path, - Some(PathBuf::from("/tmp/bundle.pem")) - ); - } - - #[tokio::test] - async fn provider_env_updates_use_generation_not_fingerprint_order() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let server = spawn_server( - &socket, - BootstrapData { - policy_proto: SandboxPolicy::default(), - provider_env_revision: u64::MAX, - provider_env_generation: 7, - provider_child_env: HashMap::from([("TOKEN".to_string(), "first".to_string())]), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - }, - current_peer(), - ) - .unwrap(); - let publisher = server.publisher(); - let (_bootstrap, mut connection) = connect_process_client(&socket, Duration::from_secs(1)) - .await - .unwrap(); - - publisher.publish_provider_env( - 1, - HashMap::from([("TOKEN".to_string(), "second".to_string())]), - ); - - let update = tokio::time::timeout(Duration::from_secs(1), connection.updates.recv()) - .await - .unwrap() - .unwrap(); - match update { - ControlUpdate::ProviderEnv { - revision, - generation, - provider_child_env, - } => { - assert_eq!(revision, 1); - assert_eq!(generation, 8); - assert_eq!( - provider_child_env.get("TOKEN").map(String::as_str), - Some("second") - ); - } - other => panic!("unexpected sidecar update: {other:?}"), - } - - publisher.publish_provider_env( - 1, - HashMap::from([("TOKEN".to_string(), "duplicate".to_string())]), - ); - assert!( - tokio::time::timeout(Duration::from_millis(50), connection.updates.recv()) - .await - .is_err(), - "an identical fingerprint must remain a no-op" - ); - - publisher.publish_provider_env( - u64::MAX, - HashMap::from([("TOKEN".to_string(), "third".to_string())]), - ); - let update = tokio::time::timeout(Duration::from_secs(1), connection.updates.recv()) - .await - .unwrap() - .unwrap(); - match update { - ControlUpdate::ProviderEnv { - revision, - generation, - provider_child_env, - } => { - assert_eq!(revision, u64::MAX); - assert_eq!(generation, 9); - assert_eq!( - provider_child_env.get("TOKEN").map(String::as_str), - Some("third") - ); - } - other => panic!("unexpected sidecar update: {other:?}"), - } - } - - #[tokio::test] - async fn agent_proposals_update_is_delivered_to_process_client() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let server = spawn_server( - &socket, - BootstrapData { - policy_proto: SandboxPolicy::default(), - provider_env_revision: 0, - provider_env_generation: 0, - provider_child_env: HashMap::new(), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - }, - current_peer(), - ) - .unwrap(); - let publisher = server.publisher(); - let (_bootstrap, mut connection) = connect_process_client(&socket, Duration::from_secs(1)) - .await - .unwrap(); - - publisher.publish_agent_proposals(true, 9); - - let update = tokio::time::timeout(Duration::from_secs(1), connection.updates.recv()) - .await - .unwrap() - .unwrap(); - match update { - ControlUpdate::AgentProposals { - enabled, - config_revision, - } => { - assert!(enabled); - assert_eq!(config_revision, 9); - } - other => panic!("unexpected sidecar update: {other:?}"), - } - } - - #[tokio::test] - async fn entrypoint_started_is_delivered_to_server() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let server = spawn_server( - &socket, - BootstrapData { - policy_proto: SandboxPolicy::default(), - provider_env_revision: 0, - provider_env_generation: 0, - provider_child_env: HashMap::new(), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - }, - current_peer(), - ) - .unwrap(); - let publisher = server.publisher(); - let mut entrypoint_rx = server.into_entrypoint_receiver(); - let (_bootstrap, mut connection) = connect_process_client(&socket, Duration::from_secs(1)) - .await - .unwrap(); - - let anchor = tokio::time::timeout(Duration::from_secs(1), entrypoint_rx.recv()) - .await - .unwrap() - .unwrap(); - assert_eq!(anchor.pid, std::process::id()); - assert!(!anchor.start_session); - - send_entrypoint_started(&connection.writer, 4242, "instance-1".to_string()) - .await - .unwrap(); - - let started = tokio::time::timeout(Duration::from_secs(1), entrypoint_rx.recv()) - .await - .unwrap() - .unwrap(); - assert_eq!(started.pid, 4242); - assert!(started.start_session); - assert_eq!(started.instance_id, "instance-1"); - assert!(started.exit_code.is_none()); - - send_main_process_exited(&connection.writer, "instance-1".to_string(), 0) - .await - .unwrap(); - let terminal = tokio::time::timeout(Duration::from_secs(1), entrypoint_rx.recv()) - .await - .unwrap() - .unwrap(); - assert_eq!(terminal.exit_code, Some(0)); - assert!(!terminal.finalized); - - assert!( - tokio::time::timeout(Duration::from_millis(20), connection.updates.recv()) - .await - .is_err(), - "process side must not observe a durable ACK before gateway persistence" - ); - publisher.publish_main_process_exit_ack("instance-1".to_string()); - let ack = tokio::time::timeout(Duration::from_secs(1), connection.updates.recv()) - .await - .unwrap() - .unwrap(); - assert!(matches!( - ack, - ControlUpdate::MainProcessExitAck { instance_id } if instance_id == "instance-1" - )); - - send_main_process_finalized(&connection.writer, "instance-1".to_string()) - .await - .unwrap(); - let delivered = tokio::time::timeout(Duration::from_secs(1), entrypoint_rx.recv()) - .await - .unwrap() - .unwrap(); - assert!(delivered.exit_code.is_none()); - assert!(delivered.finalized); - } - - #[tokio::test] - async fn second_control_client_is_rejected_after_authoritative_bootstrap() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let _server = spawn_server( - &socket, - BootstrapData { - policy_proto: SandboxPolicy::default(), - provider_env_revision: 0, - provider_env_generation: 0, - provider_child_env: HashMap::new(), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - }, - current_peer(), - ) - .unwrap(); - - let (_bootstrap, _connection) = connect_process_client(&socket, Duration::from_secs(1)) - .await - .unwrap(); - - let err = tokio::net::UnixStream::connect(&socket) - .await - .expect_err("control listener must be removed after the first bootstrap"); - assert!( - matches!( - err.kind(), - std::io::ErrorKind::NotFound | std::io::ErrorKind::ConnectionRefused - ), - "unexpected second-client error: {err}" - ); - } - - #[tokio::test] - async fn authoritative_connection_task_ends_when_process_supervisor_disconnects() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let server = spawn_server( - &socket, - BootstrapData { - policy_proto: SandboxPolicy::default(), - provider_env_revision: 0, - provider_env_generation: 0, - provider_child_env: HashMap::new(), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - }, - current_peer(), - ) - .unwrap(); - let (_entrypoint_rx, connection_task) = server.into_runtime_parts(); - let (_bootstrap, connection) = connect_process_client(&socket, Duration::from_secs(1)) - .await - .unwrap(); - - drop(connection); - tokio::time::timeout(Duration::from_secs(1), connection_task) - .await - .expect("server must observe authoritative client disconnect") - .expect("control task must not panic"); - } - - #[tokio::test] - async fn process_client_reports_network_sidecar_restart() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let server = spawn_server( - &socket, - BootstrapData { - policy_proto: SandboxPolicy::default(), - provider_env_revision: 0, - provider_env_generation: 0, - provider_child_env: HashMap::new(), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - }, - current_peer(), - ) - .unwrap(); - let (_entrypoint_rx, connection_task) = server.into_runtime_parts(); - let (_bootstrap, connection) = connect_process_client(&socket, Duration::from_secs(1)) - .await - .unwrap(); - - connection_task.abort(); - let _ = connection_task.await; - tokio::time::timeout(Duration::from_secs(1), connection.closed) - .await - .expect("process supervisor must observe network sidecar disconnect") - .expect("disconnect notifier must remain live"); - } - - #[tokio::test] - async fn bootstrap_rejects_claimed_pid_that_does_not_match_peer_credentials() { - let dir = tempfile::tempdir().unwrap(); - let socket = dir.path().join("control.sock"); - let server = spawn_server( - &socket, - BootstrapData { - policy_proto: SandboxPolicy::default(), - provider_env_revision: 0, - provider_env_generation: 0, - provider_child_env: HashMap::new(), - agent_proposals_enabled: false, - proxy_ca_cert_path: None, - proxy_ca_bundle_path: None, - }, - current_peer(), - ) - .unwrap(); - let mut entrypoint_rx = server.into_entrypoint_receiver(); - - let mut stream = tokio::net::UnixStream::connect(&socket).await.unwrap(); - write_json_line( - &mut stream, - &WireClientMessage::BootstrapRequest { - supervisor_pid: std::process::id().saturating_add(1), - }, - ) - .await - .unwrap(); - - assert!( - tokio::time::timeout(Duration::from_secs(1), entrypoint_rx.recv()) - .await - .unwrap() - .is_none(), - "mismatched bootstrap must not publish a process anchor" - ); - } - - #[test] - fn malformed_client_message_is_rejected() { - let err = decode_client_message("not-json").unwrap_err(); - assert!( - err.to_string() - .contains("failed to decode sidecar client message") - ); - } -} diff --git a/crates/openshell-sandbox/tests/stdout_logging.rs b/crates/openshell-sandbox/tests/stdout_logging.rs index c4f5213de8..ed3a2d5523 100644 --- a/crates/openshell-sandbox/tests/stdout_logging.rs +++ b/crates/openshell-sandbox/tests/stdout_logging.rs @@ -6,21 +6,16 @@ use std::process::Command; #[test] fn startup_logs_go_to_stderr_not_stdout() { let output = Command::new(env!("CARGO_BIN_EXE_openshell-sandbox")) - .arg("--") - .arg("/usr/bin/printf") - .arg("hello") + .arg("--bootstrap") + .arg("/does/not/exist/openshell-boundary.json") .env("OPENSHELL_LOG_LEVEL", "info") .env_remove("RUST_LOG") - .env_remove("OPENSHELL_POLICY_RULES") - .env_remove("OPENSHELL_POLICY_DATA") - .env_remove("OPENSHELL_SANDBOX_ID") - .env_remove("OPENSHELL_ENDPOINT") .output() .expect("spawn openshell-sandbox"); assert!( !output.status.success(), - "expected sandbox startup to fail without a policy source" + "expected sandbox startup to fail without bootstrap material" ); let stdout = String::from_utf8_lossy(&output.stdout); @@ -31,11 +26,9 @@ fn startup_logs_go_to_stderr_not_stdout() { "expected startup logs on stderr only, got stdout: {stdout}" ); assert!( - stderr.contains("Starting sandbox"), - "expected startup log on stderr, got: {stderr}" - ); - assert!( - stderr.contains("Sandbox policy required"), - "expected missing-policy error on stderr, got: {stderr}" + stderr.contains("capability-free sandbox probe") + || stderr.contains("read boundary config") + || stderr.contains("openshell-sandbox requires Linux"), + "expected startup qualification or bootstrap error on stderr, got: {stderr}" ); } diff --git a/crates/openshell-server/Cargo.toml b/crates/openshell-server/Cargo.toml index 505c37d50f..516d9a27df 100644 --- a/crates/openshell-server/Cargo.toml +++ b/crates/openshell-server/Cargo.toml @@ -12,7 +12,7 @@ repository.workspace = true [dependencies] openshell-bootstrap = { path = "../openshell-bootstrap" } -openshell-core = { path = "../openshell-core", default-features = false, features = ["oauth"] } +openshell-core = { path = "../openshell-core", default-features = false, features = ["jwt", "oauth"] } openshell-driver-db-credstore = { path = "../openshell-driver-db-credstore" } openshell-driver-kubernetes-secrets = { path = "../openshell-driver-kubernetes-secrets" } openshell-driver-vault = { path = "../openshell-driver-vault" } diff --git a/crates/openshell-server/src/auth/mod.rs b/crates/openshell-server/src/auth/mod.rs index b39ff7bfa9..d3433a557e 100644 --- a/crates/openshell-server/src/auth/mod.rs +++ b/crates/openshell-server/src/auth/mod.rs @@ -21,6 +21,7 @@ pub mod oidc; pub mod principal; pub mod sandbox_jwt; pub mod sandbox_methods; +pub mod sandbox_session; pub mod workspace_authz; pub use http::router; diff --git a/crates/openshell-server/src/auth/sandbox_jwt.rs b/crates/openshell-server/src/auth/sandbox_jwt.rs index f6da5dba6c..4bfca1e4f5 100644 --- a/crates/openshell-server/src/auth/sandbox_jwt.rs +++ b/crates/openshell-server/src/auth/sandbox_jwt.rs @@ -29,12 +29,21 @@ pub use openshell_extension_core::{ use serde::{Deserialize, Serialize}; use std::{ io::Cursor, + sync::Arc, time::{Duration, SystemTime, UNIX_EPOCH}, }; use tonic::Status; use tracing::{debug, warn}; use x509_parser::{oid_registry::OID_SIG_ED25519, prelude::FromDer, x509::SubjectPublicKeyInfo}; +use openshell_core::SandboxSessionId; +use openshell_core::jwt::{ + AuthenticatedSandboxSession, CredentialEpoch, GATEWAY_SESSION_JWT_TYPE, SandboxId, + SandboxLaunchAuthentication, SandboxRuntimeIdentity, SessionJwtIssuer, SessionJwtVerifier, + SessionTokenProfile, SessionVerificationKey, SupervisorAuthBundle, SystemJwtClock, +}; +use openshell_core::sandbox_generation::SandboxGenerationId; + /// SPIFFE-shaped subject prefix. Embedded in the `sub` claim of every /// minted token so a future migration to per-sandbox certs or SPIRE can /// reuse the same subject namespace without breaking handler equality @@ -105,6 +114,177 @@ pub struct MintedToken { pub expires_at_ms: i64, } +/// Issuer and verifier for launch-scoped supervisor credentials. +pub struct SandboxSessionJwtAuthority { + issuer: SessionJwtIssuer, + gateway_verifier: SessionJwtVerifier, + gateway_id: String, + verification_keys: Vec, +} + +impl std::fmt::Debug for SandboxSessionJwtAuthority { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("SandboxSessionJwtAuthority") + .field("gateway_id", &self.gateway_id) + .field( + "verification_key_ids", + &self + .verification_keys + .iter() + .map(|key| key.key_id.as_str()) + .collect::>(), + ) + .finish_non_exhaustive() + } +} + +impl SandboxSessionJwtAuthority { + pub fn from_pem( + signing_key_pem: &[u8], + public_key_pem: &[u8], + key_id: String, + gateway_id: &str, + ttl: Duration, + ) -> Result { + let clock = Arc::new(SystemJwtClock); + let issuer = SessionJwtIssuer::from_ed25519_pem( + signing_key_pem, + key_id.clone(), + gateway_id, + ttl, + clock.clone(), + ) + .map_err(|error| error.to_string())?; + let verification_keys = vec![SessionVerificationKey { + key_id, + public_key_pem: public_key_pem.to_vec(), + }]; + let gateway_verifier = SessionJwtVerifier::new( + gateway_id, + SessionTokenProfile::Gateway, + verification_keys.clone(), + clock, + ) + .map_err(|error| error.to_string())?; + Ok(Self { + issuer, + gateway_verifier, + gateway_id: gateway_id.to_string(), + verification_keys, + }) + } + + #[allow(clippy::result_large_err)] + pub fn mint_persisted_launch( + &self, + sandbox_id: &str, + identity: &crate::auth::sandbox_session::PersistedSandboxIdentity, + ) -> Result { + self.mint_launch( + sandbox_id, + identity.runtime_generation.clone(), + identity.auth_epoch, + ) + } + + #[allow(clippy::result_large_err)] + pub fn mint_launch( + &self, + sandbox_id: &str, + runtime_generation: SandboxGenerationId, + auth_epoch: CredentialEpoch, + ) -> Result { + let identity = SandboxRuntimeIdentity { + sandbox_id: SandboxId::parse(sandbox_id) + .map_err(|_| Status::invalid_argument("sandbox ID is invalid"))?, + runtime_generation: runtime_generation.clone(), + auth_epoch, + }; + let pair = self.issuer.mint_pair(&identity).map_err(|error| { + warn!(%error, "failed to mint launch-scoped sandbox credentials"); + Status::internal("failed to mint sandbox launch credentials") + })?; + Ok(SandboxLaunchAuthentication { + supervisor: SupervisorAuthBundle { + session_id: SandboxSessionId::new(), + runtime_generation, + session_rotation: openshell_core::jwt::SessionRotation::new(1) + .map_err(|error| Status::internal(error.to_string()))?, + auth_epoch: pair.auth_epoch, + gateway_token: pair.gateway.token, + gateway_expires_at: pair.gateway.expires_at, + sandbox_token: pair.sandbox.token, + sandbox_expires_at: pair.sandbox.expires_at, + }, + gateway_id: self.gateway_id.clone(), + verification_keys: self.verification_keys.clone(), + }) + } + + pub fn verify_gateway_token(&self, token: &str) -> Result { + self.gateway_verifier + .verify(token) + .map_err(|error| Status::unauthenticated(format!("invalid gateway session: {error}"))) + } +} + +/// Authenticates launch-scoped supervisor tokens and checks their identity +/// against the durable sandbox record. +pub struct SandboxSessionJwtAuthenticator { + authority: Arc, + store: Arc, +} + +impl SandboxSessionJwtAuthenticator { + pub fn new( + authority: Arc, + store: Arc, + ) -> Self { + Self { authority, store } + } +} + +impl std::fmt::Debug for SandboxSessionJwtAuthenticator { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("SandboxSessionJwtAuthenticator") + .finish_non_exhaustive() + } +} + +#[async_trait] +impl Authenticator for SandboxSessionJwtAuthenticator { + async fn authenticate( + &self, + headers: &http::HeaderMap, + _path: &str, + ) -> Result, Status> { + let Some(token) = headers + .get("authorization") + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.strip_prefix("Bearer ")) + else { + return Ok(None); + }; + let Ok(header) = decode_header(token) else { + return Ok(None); + }; + if header.typ.as_deref() != Some(GATEWAY_SESSION_JWT_TYPE) { + return Ok(None); + } + let authenticated = self.authority.verify_gateway_token(token)?; + crate::auth::sandbox_session::authorize_persisted(&self.store, &authenticated).await?; + Ok(Some(Principal::Sandbox(SandboxPrincipal { + sandbox_id: authenticated.sandbox_id.to_string(), + source: SandboxIdentitySource::BootstrapJwt { + issuer: "launch-session".to_string(), + }, + trust_domain: Some("openshell".to_string()), + }))) + } +} + impl SandboxJwtIssuer { pub fn from_pem( signing_key_pem: &[u8], diff --git a/crates/openshell-server/src/auth/sandbox_session.rs b/crates/openshell-server/src/auth/sandbox_session.rs new file mode 100644 index 0000000000..9589eb8a31 --- /dev/null +++ b/crates/openshell-server/src/auth/sandbox_session.rs @@ -0,0 +1,190 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Durable sandbox runtime identity used to authorize session JWTs. + +use std::collections::HashMap; + +use openshell_core::jwt::{AuthenticatedSandboxSession, CredentialEpoch, SessionJwtError}; +use openshell_core::proto::{Sandbox, SandboxPhase}; +use openshell_core::sandbox_generation::SandboxGenerationId; +use tonic::Status; + +use crate::persistence::Store; + +pub const RUNTIME_GENERATION_ANNOTATION: &str = "internal.openshell.ai/runtime-generation"; +pub const AUTH_EPOCH_ANNOTATION: &str = "internal.openshell.ai/auth-epoch"; + +/// The complete durable authorization identity for one sandbox runtime. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PersistedSandboxIdentity { + pub runtime_generation: SandboxGenerationId, + pub auth_epoch: CredentialEpoch, +} + +impl PersistedSandboxIdentity { + pub fn new() -> Result { + Ok(Self { + runtime_generation: SandboxGenerationId::parse(uuid::Uuid::new_v4().to_string()) + .map_err(|_| SessionJwtError::InvalidRuntimeIdentity)?, + auth_epoch: CredentialEpoch::new(1)?, + }) + } + + pub fn read(annotations: &HashMap) -> Result { + let runtime_generation = annotations + .get(RUNTIME_GENERATION_ANNOTATION) + .ok_or(SessionJwtError::MissingRuntimeIdentity) + .and_then(|value| { + SandboxGenerationId::parse(value.clone()) + .map_err(|_| SessionJwtError::InvalidRuntimeIdentity) + })?; + let auth_epoch = annotations + .get(AUTH_EPOCH_ANNOTATION) + .ok_or(SessionJwtError::MissingRuntimeIdentity)? + .parse::() + .map_err(|_| SessionJwtError::InvalidCredentialEpoch) + .and_then(CredentialEpoch::new)?; + Ok(Self { + runtime_generation, + auth_epoch, + }) + } + + pub fn write(&self, annotations: &mut HashMap) { + annotations.insert( + RUNTIME_GENERATION_ANNOTATION.to_string(), + self.runtime_generation.to_string(), + ); + annotations.insert( + AUTH_EPOCH_ANNOTATION.to_string(), + self.auth_epoch.get().to_string(), + ); + } +} + +/// Load the authoritative runtime identity and compare it with a signed JWT. +/// +/// Every gateway replica performs this check against shared persistence. No +/// raw token, token ID, or refresh lineage needs to be replicated. +#[allow(clippy::result_large_err)] +pub async fn authorize_persisted( + store: &Store, + principal: &AuthenticatedSandboxSession, +) -> Result { + let sandbox = store + .get_message::(principal.sandbox_id.as_str()) + .await + .map_err(|error| Status::unavailable(format!("load sandbox identity failed: {error}")))? + .ok_or_else(|| Status::unauthenticated("sandbox identity does not exist"))?; + + let phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); + if !matches!( + phase, + SandboxPhase::Provisioning | SandboxPhase::Ready | SandboxPhase::Starting + ) { + return Err(Status::failed_precondition( + "sandbox runtime identity is not active", + )); + } + + let metadata = sandbox + .metadata + .as_ref() + .ok_or_else(|| Status::unauthenticated("sandbox identity metadata is missing"))?; + let identity = PersistedSandboxIdentity::read(&metadata.annotations) + .map_err(|_| Status::unauthenticated("sandbox runtime identity is invalid"))?; + if principal.runtime_generation != identity.runtime_generation + || principal.auth_epoch != identity.auth_epoch + { + return Err(Status::unauthenticated( + "gateway token does not match the active sandbox identity", + )); + } + Ok(identity) +} + +#[cfg(test)] +mod tests { + use super::*; + use openshell_core::jwt::SandboxId; + use openshell_core::proto::SandboxStatus; + use openshell_core::proto::datamodel::v1::ObjectMeta; + use uuid::Uuid; + + fn principal(auth_epoch: u64) -> AuthenticatedSandboxSession { + AuthenticatedSandboxSession { + sandbox_id: SandboxId::parse("sandbox-a").expect("sandbox ID"), + runtime_generation: SandboxGenerationId::parse("generation-a") + .expect("runtime generation"), + auth_epoch: CredentialEpoch::new(auth_epoch).expect("auth epoch"), + token_id: Uuid::new_v4(), + issued_at: 1, + expires_at: 2, + } + } + + #[tokio::test] + async fn shared_identity_authorizes_every_replica_and_revokes_old_epochs() { + let store = Store::connect("sqlite::memory:").await.expect("store"); + let identity = PersistedSandboxIdentity { + runtime_generation: SandboxGenerationId::parse("generation-a") + .expect("runtime generation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + }; + let mut metadata = ObjectMeta { + id: "sandbox-a".to_string(), + name: "sandbox-a".to_string(), + workspace: "default".to_string(), + ..Default::default() + }; + identity.write(&mut metadata.annotations); + let sandbox = Sandbox { + metadata: Some(metadata), + status: Some(SandboxStatus { + phase: SandboxPhase::Ready as i32, + ..Default::default() + }), + ..Default::default() + }; + store.put_message(&sandbox).await.expect("persist sandbox"); + + authorize_persisted(&store, &principal(1)) + .await + .expect("first replica authorizes from persistence"); + authorize_persisted(&store, &principal(1)) + .await + .expect("second replica authorizes without local state"); + + store + .update_message_cas::("sandbox-a", 0, |sandbox| { + let next = PersistedSandboxIdentity { + runtime_generation: SandboxGenerationId::parse("generation-a") + .expect("runtime generation"), + auth_epoch: CredentialEpoch::new(2).expect("auth epoch"), + }; + next.write(&mut sandbox.metadata.as_mut().expect("metadata").annotations); + }) + .await + .expect("advance auth epoch"); + + let error = authorize_persisted(&store, &principal(1)) + .await + .expect_err("old epoch must be revoked on every replica"); + assert_eq!(error.code(), tonic::Code::Unauthenticated); + authorize_persisted(&store, &principal(2)) + .await + .expect("new epoch is active"); + + store + .update_message_cas::("sandbox-a", 0, |sandbox| { + sandbox.set_phase(SandboxPhase::Stopped as i32); + }) + .await + .expect("stop sandbox"); + let error = authorize_persisted(&store, &principal(2)) + .await + .expect_err("stopped runtime must reject its token"); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + } +} diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 70ffb5fe5d..c8b00ea134 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -31,8 +31,9 @@ use openshell_core::proto::compute::v1::{ ListSandboxesRequest, ResourceCapabilities as DriverResourceCapabilities, ResourceRequirements as DriverSandboxResourceRequirements, StartSandboxRequest, StopSandboxRequest, ValidateSandboxCreateRequest, WatchSandboxesEvent, WatchSandboxesRequest, - compute_driver_client::ComputeDriverClient, compute_driver_server::ComputeDriver, - gateway_listener_requirement::Selector, watch_sandboxes_event, + WorkloadIdentityRequest, compute_driver_client::ComputeDriverClient, + compute_driver_server::ComputeDriver, gateway_listener_requirement::Selector, + watch_sandboxes_event, }; use openshell_core::proto::{ PlatformEvent, Sandbox, SandboxCondition, SandboxPhase, SandboxSpec, SandboxStatus, @@ -43,6 +44,7 @@ use openshell_core::{ObjectLabels, ObjectWorkspace}; use prost::Message; use std::collections::HashMap; use std::fmt; +use std::future::Future; use std::net::SocketAddr; use std::path::{Path, PathBuf}; use std::pin::Pin; @@ -172,6 +174,7 @@ mod traced_driver { } const DELETE_PHASE_CAS_RETRY_LIMIT: usize = 3; +const SUPERVISOR_SESSION_CAS_RETRY_LIMIT: usize = 3; #[derive(Clone, Debug, Eq, PartialEq)] pub enum GatewayListenerRequirement { @@ -933,6 +936,22 @@ impl ComputeRuntime { sandbox: Sandbox, sandbox_token: Option, await_main_process_attachment: bool, + ) -> Result { + self.create_sandbox_authenticated( + sandbox, + sandbox_token, + None, + await_main_process_attachment, + ) + .await + } + + pub async fn create_sandbox_authenticated( + &self, + sandbox: Sandbox, + sandbox_token: Option, + launch_authentication: Option>, + await_main_process_attachment: bool, ) -> Result { let sandbox_id = sandbox.object_id().to_string(); let mut sandbox = sandbox; @@ -996,6 +1015,7 @@ impl ComputeRuntime { } if let Some(spec) = driver_sandbox.spec.as_mut() { spec.await_main_process_attachment = await_main_process_attachment; + spec.launch_authentication = launch_authentication.unwrap_or_default(); } match self .driver @@ -1216,10 +1236,21 @@ impl ComputeRuntime { } } + #[cfg(test)] pub(crate) async fn start_sandbox( &self, workspace: &str, name: &str, + ) -> Result { + self.start_sandbox_authenticated(workspace, name, Vec::new()) + .await + } + + pub(crate) async fn start_sandbox_authenticated( + &self, + workspace: &str, + name: &str, + launch_authentication: Vec, ) -> Result { let candidate = self .store @@ -1296,6 +1327,7 @@ impl ComputeRuntime { previous, starting, lifecycle_guard, + launch_authentication, ) .await } @@ -1316,7 +1348,11 @@ impl ComputeRuntime { previous: Sandbox, starting: Sandbox, lifecycle_guard: SandboxLifecycleGuard, + launch_authentication: Vec, ) -> Result { + let generation_id = sandbox_runtime_generation(&starting) + .map_err(Status::failed_precondition)? + .into_string(); let result = self .driver .call( @@ -1330,6 +1366,8 @@ impl ComputeRuntime { .start_sandbox(Request::new(StartSandboxRequest { sandbox_id, sandbox_name, + launch_authentication, + generation_id, })) .await } @@ -2254,6 +2292,35 @@ impl ComputeRuntime { /// Should be called once at gateway startup, before watchers spawn, /// so the watch loop sees the post-start state on its first poll. pub async fn start_persisted_sandboxes(&self) -> Result<(), String> { + self.start_persisted_sandboxes_with_authentication( + |_| async { Ok(Vec::new()) }, + |_| async { Ok(()) }, + |_| {}, + ) + .await + } + + /// Reconcile persisted running intent and provision fresh launch + /// authentication before a restored runtime reconnects. + pub async fn start_persisted_sandboxes_with_authentication< + Authentication, + AuthenticationFuture, + Committed, + CommittedFuture, + Failed, + >( + &self, + launch_authentication_for: Authentication, + authentication_committed: Committed, + authentication_failed: Failed, + ) -> Result<(), String> + where + Authentication: Fn(&Sandbox) -> AuthenticationFuture, + AuthenticationFuture: Future, String>>, + Committed: Fn(&str) -> CommittedFuture, + CommittedFuture: Future>, + Failed: Fn(&str), + { self.recover_persisted_lifecycle_transitions().await?; if !self.driver_info.gateway_manages_lifecycle { return Ok(()); @@ -2290,6 +2357,38 @@ impl ComputeRuntime { } let sandbox_name = sandbox.object_name().to_string(); + let generation_id = match sandbox_runtime_generation(&sandbox) { + Ok(generation) => generation.into_string(), + Err(error) => { + warn!(sandbox_id, %error, "Persisted sandbox runtime identity is invalid"); + authentication_failed(sandbox.object_id()); + failed += 1; + continue; + } + }; + let launch_authentication = match launch_authentication_for(&sandbox).await { + Ok(authentication) => authentication, + Err(err) => { + warn!( + sandbox_id = %sandbox.object_id(), + sandbox_name = %sandbox.object_name(), + error = %err, + "Failed to prepare sandbox authentication during gateway startup" + ); + if !recoverable_error { + self.mark_sandbox_error( + &sandbox, + "AuthenticationFailed", + &format!( + "Failed to prepare sandbox authentication during gateway startup: {err}" + ), + ) + .await; + } + failed += 1; + continue; + } + }; match self .driver .call( @@ -2298,11 +2397,14 @@ impl ComputeRuntime { |driver| { let sandbox_id = sandbox_id.clone(); let sandbox_name = sandbox_name.clone(); + let launch_authentication = launch_authentication.clone(); async move { driver .start_sandbox(Request::new(StartSandboxRequest { sandbox_id, sandbox_name, + launch_authentication, + generation_id, })) .await } @@ -2311,6 +2413,13 @@ impl ComputeRuntime { .await { Ok(_) => { + if let Err(err) = authentication_committed(sandbox.object_id()).await { + warn!( + sandbox_id = %sandbox.object_id(), + error = %err, + "Failed to commit sandbox authentication successor; it will be retried" + ); + } let did_recover = if recoverable_error { self.clear_recoverable_error(&sandbox).await } else { @@ -2329,6 +2438,7 @@ impl ComputeRuntime { } } Err(err) if err.code() == Code::NotFound => { + authentication_failed(sandbox.object_id()); // Backend resource is gone but the store still // remembers the sandbox. Mark Error so the UI // surfaces the inconsistency; the reconcile loop @@ -2350,6 +2460,7 @@ impl ComputeRuntime { missing += 1; } Err(err) => { + authentication_failed(sandbox.object_id()); warn!( sandbox_id = %sandbox.object_id(), sandbox_name = %sandbox.object_name(), @@ -2465,6 +2576,13 @@ impl ComputeRuntime { let sandbox_id = sandbox.object_id().to_string(); let sandbox_name = sandbox.object_name().to_string(); let driver_sandbox_id = sandbox_id.clone(); + let generation_id = match sandbox_runtime_generation(&sandbox) { + Ok(generation) => generation.into_string(), + Err(error) => { + warn!(sandbox_id, %error, "Persisted sandbox runtime identity is invalid"); + continue; + } + }; if let Err(err) = self .driver .call( @@ -2475,6 +2593,8 @@ impl ComputeRuntime { .start_sandbox(Request::new(StartSandboxRequest { sandbox_id: driver_sandbox_id, sandbox_name, + launch_authentication: Vec::new(), + generation_id, })) .await }, @@ -2885,18 +3005,19 @@ impl ComputeRuntime { SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown) }); - if !driver_snapshot_reports_terminal_container_exit(&incoming) - || existing_phase != SandboxPhase::Starting - { + if existing_phase != SandboxPhase::Starting { return self.apply_sandbox_update_locked(incoming, existing).await; } - // A terminal snapshot can already be queued when StartSandbox moves - // the durable phase to Starting. Release the global watch lock, wait - // for that lifecycle operation, and then reread both the driver and - // store before applying the terminal observation. Taking the - // per-sandbox gate only for this ambiguous phase avoids delaying - // unrelated watch events behind slow lifecycle operations. + // Any snapshot can already be queued when StartSandbox moves the + // durable phase to Starting. In particular, an old-generation Ready + // event followed by its terminal event can otherwise promote and then + // stop the new generation before the replacement supervisor connects. + // Release the global watch lock, wait for that lifecycle operation, + // and then reread both the driver and store before applying an + // authoritative observation. Taking the per-sandbox gate only for + // this ambiguous phase avoids delaying unrelated watch events behind + // slow lifecycle operations. let existing_name = existing_sandbox.as_ref().map_or_else( || incoming.name.clone(), |sandbox| sandbox.object_name().to_string(), @@ -2925,7 +3046,7 @@ impl ComputeRuntime { { warn!( sandbox_id = %incoming.id, - "Could not validate terminal driver snapshot; retaining current sandbox state" + "Could not validate driver snapshot during sandbox start; retaining current sandbox state" ); return Ok(()); } @@ -3037,82 +3158,129 @@ impl ComputeRuntime { instance_id: Option<&str>, terminal_delivery_finalized: bool, ) -> Result<(), String> { - let guard = self.sync_lock.lock().await; - - let Some(existing) = self + let _guard = self.sync_lock.lock().await; + let existing = self .store .get_message::(sandbox_id) .await - .map_err(|err| err.to_string())? - else { - return Ok(()); - }; - let current_phase = - SandboxPhase::try_from(existing.phase()).unwrap_or(SandboxPhase::Unknown); - if !connected - && matches!(current_phase, SandboxPhase::Error | SandboxPhase::Completed) - && terminal_delivery_finalized - { - drop(guard); - self.schedule_ephemeral_sandbox_delete(&existing); - return Ok(()); - } - if matches!( - current_phase, - SandboxPhase::Deleting - | SandboxPhase::Error - | SandboxPhase::Stopping - | SandboxPhase::Stopped - | SandboxPhase::Completed - ) { - return Ok(()); - } - if !connected && current_phase != SandboxPhase::Ready { - return Ok(()); - } - let expected_resource_version = sandbox_resource_version(&existing); - - // Use CAS to update sandbox phase based on supervisor session state - let result = self - .store - .update_message_cas::(sandbox_id, expected_resource_version, |sandbox| { - let sandbox_name = sandbox.object_name().to_string(); - if connected { - ensure_supervisor_ready_status(&mut sandbox.status, &sandbox_name); - let status = sandbox.status.get_or_insert_with(Default::default); - status.main_process_instance_id = instance_id.unwrap_or_default().to_string(); - status.exit_code = None; - sandbox.set_phase(SandboxPhase::Ready as i32); - } else { - ensure_supervisor_not_ready_status(&mut sandbox.status, &sandbox_name); - sandbox.set_phase(SandboxPhase::Provisioning as i32); - } - }) - .await; + .map_err(|err| err.to_string())?; + self.set_supervisor_session_state_from_snapshot( + sandbox_id, + connected, + instance_id, + terminal_delivery_finalized, + existing, + ) + .await + } - // Handle not found gracefully (sandbox may have been deleted) - let sandbox = match result { - Ok(s) => s, - Err(crate::persistence::PersistenceError::Database(ref msg)) - if msg.contains("not found") => - { + async fn set_supervisor_session_state_from_snapshot( + &self, + sandbox_id: &str, + connected: bool, + instance_id: Option<&str>, + terminal_delivery_finalized: bool, + mut existing: Option, + ) -> Result<(), String> { + for attempt in 1..=SUPERVISOR_SESSION_CAS_RETRY_LIMIT { + let Some(current) = existing else { return Ok(()); - } - Err(crate::persistence::PersistenceError::Conflict { - current_resource_version, - }) => { + }; + let current_phase = + SandboxPhase::try_from(current.phase()).unwrap_or(SandboxPhase::Unknown); + if connected + && matches!( + current_phase, + SandboxPhase::Deleting | SandboxPhase::Stopping | SandboxPhase::Stopped + ) + { return Err(format!( - "concurrent modification detected (current resource_version: {})", - current_resource_version - .map_or_else(|| "unknown".to_string(), |v| v.to_string()) + "sandbox is not accepting supervisor sessions while {current_phase:?}" )); } - Err(e) => return Err(e.to_string()), - }; + if !connected + && matches!(current_phase, SandboxPhase::Error | SandboxPhase::Completed) + && terminal_delivery_finalized + { + self.schedule_ephemeral_sandbox_delete(¤t); + return Ok(()); + } + if matches!( + current_phase, + SandboxPhase::Deleting + | SandboxPhase::Error + | SandboxPhase::Stopping + | SandboxPhase::Stopped + | SandboxPhase::Completed + ) { + return Ok(()); + } + if !connected && current_phase != SandboxPhase::Ready { + return Ok(()); + } + let expected_resource_version = sandbox_resource_version(¤t); + let result = self + .store + .update_message_cas::( + sandbox_id, + expected_resource_version, + |sandbox| { + let sandbox_name = sandbox.object_name().to_string(); + if connected { + ensure_supervisor_ready_status(&mut sandbox.status, &sandbox_name); + let status = sandbox.status.get_or_insert_with(Default::default); + status.main_process_instance_id = + instance_id.unwrap_or_default().to_string(); + status.exit_code = None; + sandbox.set_phase(SandboxPhase::Ready as i32); + } else { + ensure_supervisor_not_ready_status(&mut sandbox.status, &sandbox_name); + sandbox.set_phase(SandboxPhase::Provisioning as i32); + } + }, + ) + .await; - self.sandbox_index.update_from_sandbox(&sandbox); - self.sandbox_watch_bus.notify(sandbox_id); - Ok(()) + match result { + Ok(sandbox) => { + self.sandbox_index.update_from_sandbox(&sandbox); + self.sandbox_watch_bus.notify(sandbox_id); + return Ok(()); + } + Err(crate::persistence::PersistenceError::Database(ref message)) + if message.contains("not found") => + { + return Ok(()); + } + Err(crate::persistence::PersistenceError::Conflict { + current_resource_version, + }) if attempt < SUPERVISOR_SESSION_CAS_RETRY_LIMIT => { + debug!( + sandbox_id, + attempt, + ?current_resource_version, + "Retrying supervisor session state after concurrent modification" + ); + existing = self + .store + .get_message::(sandbox_id) + .await + .map_err(|error| error.to_string())?; + } + Err(crate::persistence::PersistenceError::Conflict { + current_resource_version, + }) => { + return Err(format!( + "concurrent modification detected after {attempt} attempts (current resource_version: {})", + current_resource_version + .map_or_else(|| "unknown".to_string(), |version| version.to_string()) + )); + } + Err(error) => return Err(error.to_string()), + } + } + + unreachable!("supervisor session CAS retry loop always returns") } /// Persist a terminal canonical-process result. Successful completion is @@ -3916,6 +4084,19 @@ fn driver_sandbox_spec_from_public( command: spec.command.clone(), tty: spec.tty, await_main_process_attachment: false, + workload_identity: Some(WorkloadIdentityRequest { + user: spec + .policy + .as_ref() + .and_then(|policy| policy.process.as_ref()) + .map_or_else(String::new, |process| process.run_as_user.clone()), + group: spec + .policy + .as_ref() + .and_then(|policy| policy.process.as_ref()) + .map_or_else(String::new, |process| process.run_as_group.clone()), + }), + launch_authentication: Vec::new(), }) } @@ -4190,6 +4371,7 @@ fn driver_status_from_public(status: &SandboxStatus) -> DriverSandboxStatus { .map(driver_condition_from_public) .collect(), deleting: SandboxPhase::try_from(status.phase) == Ok(SandboxPhase::Deleting), + ..Default::default() } } @@ -4234,6 +4416,19 @@ fn sandbox_resource_version(sandbox: &Sandbox) -> u64 { .map_or(0, |metadata| metadata.resource_version) } +fn sandbox_runtime_generation( + sandbox: &Sandbox, +) -> Result { + let persisted = sandbox.metadata.as_ref().and_then(|metadata| { + metadata + .annotations + .get(crate::auth::sandbox_session::RUNTIME_GENERATION_ANNOTATION) + }); + let value = persisted.ok_or_else(|| "sandbox runtime generation is missing".to_string())?; + openshell_core::sandbox_generation::SandboxGenerationId::parse(value.clone()) + .map_err(|error| error.to_string()) +} + fn public_status_from_driver( status: &DriverSandboxStatus, phase: SandboxPhase, @@ -4328,6 +4523,12 @@ fn apply_driver_snapshot( SandboxPhase::Stopping } SandboxPhase::Stopping if phase != SandboxPhase::Error => SandboxPhase::Stopping, + // A driver's explicit bootstrap condition is authoritative evidence + // that an accepted StartSandbox operation is provisioning a new + // generation. This observation must be able to recover a stale + // Stopped view so the replacement supervisor can register. A genuine + // stop includes Suspended=True and does not satisfy this predicate. + SandboxPhase::Stopped if driver_snapshot_confirms_starting(incoming) => phase, SandboxPhase::Stopped => SandboxPhase::Stopped, SandboxPhase::Completed => SandboxPhase::Completed, SandboxPhase::Starting if !matches!(phase, SandboxPhase::Ready | SandboxPhase::Error) => { @@ -4400,15 +4601,17 @@ fn driver_snapshot_confirms_stopped(incoming: &DriverSandbox) -> bool { }) } -fn driver_snapshot_reports_terminal_container_exit(incoming: &DriverSandbox) -> bool { +fn driver_snapshot_confirms_starting(incoming: &DriverSandbox) -> bool { incoming.status.as_ref().is_some_and(|status| { - status.conditions.iter().any(|condition| { - condition.status.eq_ignore_ascii_case("false") - && matches!( - condition.reason.to_ascii_lowercase().as_str(), - "containerexited" | "containerstopped" | "containerruntimerestart" - ) - }) + !status.deleting + && status.conditions.iter().any(|condition| { + condition.r#type.eq_ignore_ascii_case("Bootstrapping") + && condition.status.eq_ignore_ascii_case("true") + }) + && !status.conditions.iter().any(|condition| { + condition.r#type.eq_ignore_ascii_case("Suspended") + && condition.status.eq_ignore_ascii_case("true") + }) }) } @@ -4995,6 +5198,28 @@ mod tests { assert_eq!(gpu.count, Some(2)); } + #[test] + fn driver_sandbox_spec_carries_admitted_identity_selectors() { + let public = SandboxSpec { + policy: Some(openshell_core::proto::sandbox::v1::SandboxPolicy { + process: Some(openshell_core::proto::sandbox::v1::ProcessPolicy { + run_as_user: "10001".to_string(), + run_as_group: "10002".to_string(), + }), + ..Default::default() + }), + ..Default::default() + }; + + let driver = driver_sandbox_spec_from_public(&public, "test-driver") + .expect("driver spec should map"); + let identity = driver + .workload_identity + .expect("identity request is mandatory"); + assert_eq!(identity.user, "10001"); + assert_eq!(identity.group, "10002"); + } + #[test] fn select_driver_config_forwards_only_matching_driver_block() { let config = prost_types::Struct { @@ -5409,6 +5634,7 @@ mod tests { start_blocked: AtomicBool, start_calls: AtomicUsize, start_requests: TestMutex>, + start_authentications: TestMutex>>, start_outcome: TestMutex, get_started: Notify, get_release: Semaphore, @@ -5442,6 +5668,7 @@ mod tests { start_blocked: AtomicBool::new(false), start_calls: AtomicUsize::new(0), start_requests: TestMutex::new(Vec::new()), + start_authentications: TestMutex::new(Vec::new()), start_outcome: TestMutex::new(ControlledLifecycleOutcome::Ok), get_started: Notify::new(), get_release: Semaphore::new(0), @@ -5540,6 +5767,13 @@ mod tests { .clone() } + fn start_authentications(&self) -> Vec> { + self.start_authentications + .lock() + .expect("start authentications lock poisoned") + .clone() + } + fn send_event(&self, event: WatchSandboxesEvent) { self.watch_tx .send(Ok(event)) @@ -5685,6 +5919,10 @@ mod tests { .lock() .expect("start requests lock poisoned") .push((request.sandbox_id, request.sandbox_name)); + self.start_authentications + .lock() + .expect("start authentications lock poisoned") + .push(request.launch_authentication); self.start_calls.fetch_add(1, Ordering::SeqCst); self.start_started.notify_one(); if self.start_blocked.load(Ordering::SeqCst) { @@ -5828,6 +6066,15 @@ mod tests { } fn sandbox_record(id: &str, name: &str, phase: SandboxPhase) -> Sandbox { + let mut annotations = HashMap::new(); + crate::auth::sandbox_session::PersistedSandboxIdentity { + runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse( + format!("test-{id}"), + ) + .expect("test runtime generation"), + auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).expect("test auth epoch"), + } + .write(&mut annotations); let mut sandbox = Sandbox { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { id: id.to_string(), @@ -5835,7 +6082,7 @@ mod tests { created_at_ms: 1_000_000, labels: HashMap::new(), resource_version: 0, - annotations: HashMap::new(), + annotations, workspace: "default".to_string(), deletion_timestamp_ms: 0, }), @@ -6369,6 +6616,7 @@ mod tests { sandbox_fd: String::new(), conditions: vec![condition], deleting: false, + ..Default::default() } } @@ -6392,6 +6640,7 @@ mod tests { last_transition_time: String::new(), }], deleting: false, + ..Default::default() }), } } @@ -7077,6 +7326,9 @@ mod tests { ); register_test_supervisor_session(&runtime, sandbox.object_id()); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new( + ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()), + ))); runtime .apply_sandbox_update(ready_driver_sandbox( sandbox.object_id(), @@ -7664,6 +7916,63 @@ mod tests { } } + #[tokio::test] + async fn stale_ready_snapshot_queued_before_start_is_revalidated() { + let driver = ControlledDriver::new(); + driver.block_start(); + let sandbox = sandbox_record( + "sb-start-ready-race", + "sandbox-start-ready-race", + SandboxPhase::Stopped, + ); + let mut current = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + current.status = Some(make_driver_status(make_driver_condition( + "ContainerStarting", + "replacement workload is still starting", + ))); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new(current))); + let mut runtime = test_runtime(driver.clone()).await; + runtime.driver_info.driver_reports_runtime_readiness = true; + runtime.store.put_message(&sandbox).await.unwrap(); + + let start_runtime = runtime.clone(); + let sandbox_name = sandbox.object_name().to_string(); + let start = + tokio::spawn( + async move { start_runtime.start_sandbox("default", &sandbox_name).await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.start_started.notified()) + .await + .expect("start did not reach the driver"); + + let stale_ready = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + let update_runtime = runtime.clone(); + let mut update = + tokio::spawn(async move { update_runtime.apply_sandbox_update(stale_ready).await }); + assert!( + tokio::time::timeout(Duration::from_millis(50), &mut update) + .await + .is_err(), + "queued Ready event must wait for the active start operation" + ); + + driver.release_start(); + start.await.unwrap().unwrap(); + update.await.unwrap().unwrap(); + + let stored = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + let phase = SandboxPhase::try_from(stored.phase()).unwrap_or(SandboxPhase::Unknown); + assert!( + matches!(phase, SandboxPhase::Starting | SandboxPhase::Provisioning), + "the queued Ready event must not promote the sandbox; got {phase:?}" + ); + } + #[tokio::test] async fn live_container_exit_during_start_still_transitions_to_error() { for reason in [ @@ -7826,7 +8135,8 @@ mod tests { // (PodTerminated). Starting from the Starting phase that `start` sets, the // reconciled sandbox must advance to Ready rather than being pinned at // Starting by the stale Suspended condition. - let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let driver = ControlledDriver::new(); + let runtime = test_runtime(driver.clone()).await; let sandbox = sandbox_record("sb-resumed", "sandbox-resumed", SandboxPhase::Starting); runtime.store.put_message(&sandbox).await.unwrap(); register_test_supervisor_session(&runtime, sandbox.object_id()); @@ -7854,6 +8164,7 @@ mod tests { ..Default::default() }); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new(resumed.clone()))); runtime.apply_sandbox_update(resumed).await.unwrap(); let current = runtime @@ -7891,6 +8202,73 @@ mod tests { assert_eq!(current.phase(), SandboxPhase::Stopped as i32); } + #[tokio::test] + async fn active_bootstrap_snapshot_recovers_stale_stopped_view() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-restart", "sandbox-restart", SandboxPhase::Stopped); + runtime.store.put_message(&sandbox).await.unwrap(); + let mut bootstrapping = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + let mut status = make_driver_status(make_driver_condition( + "DependenciesNotReady", + "replacement supervisor is starting", + )); + status.conditions.push(DriverCondition { + r#type: "Bootstrapping".to_string(), + status: "True".to_string(), + reason: "GenerationStarting".to_string(), + message: "Replacement generation is starting".to_string(), + last_transition_time: String::new(), + }); + bootstrapping.status = Some(status); + + runtime.apply_sandbox_update(bootstrapping).await.unwrap(); + + let current = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(current.phase(), SandboxPhase::Provisioning as i32); + } + + #[tokio::test] + async fn suspended_bootstrap_snapshot_does_not_revive_stopped_sandbox() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-stopped", "sandbox-stopped", SandboxPhase::Stopped); + runtime.store.put_message(&sandbox).await.unwrap(); + let mut status = make_driver_status(make_driver_condition( + "DependenciesNotReady", + "dependencies are unavailable", + )); + status.conditions.push(DriverCondition { + r#type: "Bootstrapping".to_string(), + status: "True".to_string(), + reason: "GenerationStarting".to_string(), + message: "Replacement generation is starting".to_string(), + last_transition_time: String::new(), + }); + status.conditions.push(DriverCondition { + r#type: "Suspended".to_string(), + status: "True".to_string(), + reason: "PodTerminated".to_string(), + message: "Sandbox is suspended".to_string(), + last_transition_time: String::new(), + }); + let mut suspended = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + suspended.status = Some(status); + + runtime.apply_sandbox_update(suspended).await.unwrap(); + + let current = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(current.phase(), SandboxPhase::Stopped as i32); + } + #[tokio::test] async fn begin_sandbox_delete_retries_after_stale_snapshot_conflict() { let runtime = test_runtime(Arc::new(TestDriver::default())).await; @@ -7981,6 +8359,7 @@ mod tests { last_transition_time: String::new(), }], deleting: false, + ..Default::default() }), workspace: "default".to_string(), }) @@ -9523,6 +9902,66 @@ mod tests { ); } + #[tokio::test] + async fn supervisor_session_connected_rejects_stopped_sandbox() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Stopped); + runtime.store.put_message(&sandbox).await.unwrap(); + + let error = runtime + .supervisor_session_connected("sb-1", "stale-generation") + .await + .unwrap_err(); + + assert!(error.contains("Stopped")); + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!(stored.phase(), SandboxPhase::Stopped as i32); + } + + #[tokio::test] + async fn supervisor_session_connected_retries_a_stale_store_snapshot() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); + runtime.store.put_message(&sandbox).await.unwrap(); + let stale = runtime.store.get_message::("sb-1").await.unwrap(); + + runtime + .store + .update_message_cas::("sb-1", 0, |sandbox| { + sandbox.set_current_policy_version(7); + }) + .await + .unwrap(); + + runtime + .set_supervisor_session_state_from_snapshot( + "sb-1", + true, + Some("test-generation"), + false, + stale, + ) + .await + .unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Ready + ); + assert_eq!(stored.current_policy_version(), 7); + } + #[tokio::test] async fn supervisor_session_disconnected_demotes_ready_sandbox() { let runtime = test_runtime(Arc::new(TestDriver::default())).await; @@ -9587,6 +10026,7 @@ mod tests { last_transition_time: String::new(), }], deleting: false, + ..Default::default() } } @@ -9604,6 +10044,7 @@ mod tests { last_transition_time: String::new(), }], deleting: true, + ..Default::default() } } @@ -9881,6 +10322,7 @@ mod tests { last_transition_time: String::new(), }], deleting: false, + ..Default::default() }), workspace: "default".to_string(), }], @@ -9902,6 +10344,7 @@ mod tests { last_transition_time: String::new(), }], deleting: false, + ..Default::default() }), workspace: "default".to_string(), }], @@ -10115,6 +10558,7 @@ mod tests { last_transition_time: String::new(), }], deleting: false, + ..Default::default() }), workspace: "default".to_string(), }], @@ -10532,6 +10976,35 @@ mod tests { ); } + #[tokio::test] + async fn start_persisted_sandboxes_supplies_fresh_authentication() { + let driver = ControlledDriver::new(); + let runtime = + test_runtime_with_gateway_managed_lifecycle(driver.clone(), "arbitrary").await; + runtime + .store + .put_message(&sandbox_record("sb-1", "sandbox", SandboxPhase::Ready)) + .await + .unwrap(); + + runtime + .start_persisted_sandboxes_with_authentication( + |sandbox| { + let sandbox_id = sandbox.object_id().to_string(); + async move { Ok(format!("authentication:{sandbox_id}").into_bytes()) } + }, + |_| async { Ok(()) }, + |_| {}, + ) + .await + .unwrap(); + + assert_eq!( + driver.start_authentications(), + vec![b"authentication:sb-1".to_vec()] + ); + } + #[tokio::test] async fn startup_sweep_rechecks_intent_after_acquiring_gate() { let driver = ControlledDriver::new(); diff --git a/crates/openshell-server/src/grpc/auth_rpc.rs b/crates/openshell-server/src/grpc/auth_rpc.rs index bc9b91989a..abd2912b6f 100644 --- a/crates/openshell-server/src/grpc/auth_rpc.rs +++ b/crates/openshell-server/src/grpc/auth_rpc.rs @@ -141,10 +141,33 @@ pub async fn handle_refresh_sandbox_token( ); Status::unavailable("sandbox JWT minting is not configured on this gateway") })?; - - ensure_sandbox_exists(state, &sandbox.sandbox_id).await?; - - let minted = issuer.mint(&sandbox.sandbox_id)?; + let session_authority = state + .sandbox_session_jwt_authority + .as_ref() + .ok_or_else(|| Status::unavailable("sandbox session minting is not configured"))?; + + let authorization_values = request.metadata().get_all("authorization"); + let mut authorization_values = authorization_values.iter(); + let authorization = authorization_values + .next() + .ok_or_else(|| Status::unauthenticated("missing authorization metadata"))?; + if authorization_values.next().is_some() { + return Err(Status::unauthenticated("duplicate authorization metadata")); + } + let gateway_token = authorization + .to_str() + .ok() + .and_then(|value| value.strip_prefix("Bearer ")) + .ok_or_else(|| Status::unauthenticated("invalid bearer authorization metadata"))?; + let principal = session_authority.verify_gateway_token(gateway_token)?; + if principal.sandbox_id.as_str() != sandbox.sandbox_id { + return Err(Status::unauthenticated( + "gateway token does not match the authenticated sandbox", + )); + } + let identity = + crate::auth::sandbox_session::authorize_persisted(&state.store, &principal).await?; + let authentication = session_authority.mint_persisted_launch(&sandbox.sandbox_id, &identity)?; let extension_credentials = if requested_extension_services.is_empty() { Vec::new() } else if !state @@ -186,9 +209,27 @@ pub async fn handle_refresh_sandbox_token( ); Ok(Response::new(RefreshSandboxTokenResponse { - token: minted.token, - expires_at_ms: minted.expires_at_ms, + token: authentication + .supervisor + .gateway_token + .expose_secret() + .to_string(), + expires_at_ms: authentication + .supervisor + .gateway_expires_at + .saturating_mul(1000), extension_credentials, + sandbox_token: authentication + .supervisor + .sandbox_token + .expose_secret() + .to_string(), + sandbox_expires_at_ms: authentication + .supervisor + .sandbox_expires_at + .saturating_mul(1000), + session_id: authentication.supervisor.runtime_generation.to_string(), + credential_epoch: authentication.supervisor.auth_epoch.get(), })) } @@ -284,7 +325,7 @@ mod tests { use crate::ServerState; use crate::auth::identity::Identity; use crate::auth::principal::{Principal, SandboxPrincipal, UserPrincipal}; - use crate::auth::sandbox_jwt::SandboxJwtIssuer; + use crate::auth::sandbox_jwt::{SandboxJwtIssuer, SandboxSessionJwtAuthority}; use crate::compute::new_test_runtime; use crate::persistence::Store; use crate::sandbox_index::SandboxIndex; @@ -321,18 +362,35 @@ mod tests { // We don't need the authenticator for these tests; only the issuer. let issuer = SandboxJwtIssuer::from_pem( mat.signing_key_pem.as_bytes(), - mat.kid, + mat.kid.clone(), "test-gateway", Some(Duration::from_hours(1)), ) .unwrap(); state.sandbox_jwt_issuer = Some(Arc::new(issuer)); + let authority = Arc::new( + SandboxSessionJwtAuthority::from_pem( + mat.signing_key_pem.as_bytes(), + mat.public_key_pem.as_bytes(), + mat.kid, + "test-gateway", + Duration::from_hours(1), + ) + .expect("session authority"), + ); + let identity = crate::auth::sandbox_session::PersistedSandboxIdentity::new() + .expect("runtime identity"); + state.sandbox_session_jwt_authority = Some(authority); let state = Arc::new(state); - insert_sandbox(&state, "sandbox-a").await; + insert_sandbox(&state, "sandbox-a", &identity).await; state } - async fn insert_sandbox(state: &Arc, sandbox_id: &str) { + async fn insert_sandbox( + state: &Arc, + sandbox_id: &str, + identity: &crate::auth::sandbox_session::PersistedSandboxIdentity, + ) { let mut sandbox = Sandbox { metadata: Some(ObjectMeta { id: sandbox_id.to_string(), @@ -350,6 +408,7 @@ mod tests { }), ..Default::default() }; + identity.write(&mut sandbox.metadata.as_mut().expect("metadata").annotations); sandbox.set_phase(SandboxPhase::Ready as i32); state.store.put_message(&sandbox).await.unwrap(); } @@ -365,6 +424,35 @@ mod tests { }) } + async fn authorize_refresh( + state: &ServerState, + request: &mut Request, + ) { + let sandbox = state + .store + .get_message::("sandbox-a") + .await + .expect("load sandbox") + .expect("sandbox"); + let identity = crate::auth::sandbox_session::PersistedSandboxIdentity::read( + &sandbox.metadata.expect("metadata").annotations, + ) + .expect("persisted identity"); + let authentication = state + .sandbox_session_jwt_authority + .as_ref() + .expect("session authority") + .mint_persisted_launch("sandbox-a", &identity) + .expect("active authentication"); + let value = format!( + "Bearer {}", + authentication.supervisor.gateway_token.expose_secret() + ) + .parse() + .expect("authorization metadata"); + request.metadata_mut().insert("authorization", value); + } + #[tokio::test] async fn current_user_returns_gateway_validated_identity() { let mut req = Request::new(GetCurrentUserRequest {}); @@ -396,6 +484,7 @@ mod tests { extension_service_names: Vec::new(), }); req.extensions_mut().insert(sandbox_principal("sandbox-a")); + authorize_refresh(&state, &mut req).await; let resp = handle_refresh_sandbox_token(&state, req) .await .expect("refresh OK") @@ -502,7 +591,7 @@ mod tests { let err = handle_refresh_sandbox_token(&state, req) .await .expect_err("missing sandbox must not refresh"); - assert_eq!(err.code(), tonic::Code::NotFound); + assert_eq!(err.code(), tonic::Code::Unauthenticated); } #[tokio::test] @@ -615,7 +704,18 @@ mod tests { Arc::new(SupervisorSessionRegistry::new()), None, )); - insert_sandbox(&state, "sandbox-a").await; + insert_sandbox( + &state, + "sandbox-a", + &crate::auth::sandbox_session::PersistedSandboxIdentity { + runtime_generation: openshell_core::sandbox_generation::SandboxGenerationId::parse( + "generation-1", + ) + .expect("runtime generation"), + auth_epoch: openshell_core::jwt::CredentialEpoch::new(1).expect("auth epoch"), + }, + ) + .await; let mut req = Request::new(RefreshSandboxTokenRequest { extension_service_names: Vec::new(), }); diff --git a/crates/openshell-server/src/grpc/mod.rs b/crates/openshell-server/src/grpc/mod.rs index 95bd6eaa3b..b16f0c66c7 100644 --- a/crates/openshell-server/src/grpc/mod.rs +++ b/crates/openshell-server/src/grpc/mod.rs @@ -7,6 +7,7 @@ mod auth_rpc; pub mod policy; pub mod provider; mod sandbox; +pub use sandbox::mint_persisted_authentication; mod service; mod validation; pub mod workspace; diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index d428e64569..befb49597e 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -372,7 +372,7 @@ async fn handle_create_sandbox_inner( }; // Leave an omitted command empty rather than persisting a concrete shell: - // the supervisor resolves the default login shell against the sandbox image + // the sandbox boundary resolves the default login shell against the agent image // (bash when present, otherwise /bin/sh on minimal images like Alpine), // which the gateway cannot do since it does not see the sandbox filesystem. // The default is an interactive login shell, so request a TTY. @@ -499,26 +499,44 @@ async fn handle_create_sandbox_inner( status })?; - // Mint a gateway JWT whenever the issuer is configured. Compute runtimes - // that bootstrap through another authentication mechanism may ignore it. - let sandbox_token = state.sandbox_jwt_issuer.as_ref().map(|issuer| { - issuer.mint(&id).map(|minted| { - tracing::info!( - sandbox_id = %id, - "minted sandbox JWT" - ); - minted.token - }) - }); - let sandbox_token = match sandbox_token { - Some(Ok(token)) => Some(token), - Some(Err(status)) => return Err(status), - None => None, + let runtime_identity = crate::auth::sandbox_session::PersistedSandboxIdentity::new() + .map_err(|error| Status::internal(error.to_string()))?; + if let Some(metadata) = sandbox.metadata.as_mut() { + runtime_identity.write(&mut metadata.annotations); + } + let launch_authentication = if let Some(authority) = &state.sandbox_session_jwt_authority { + Some(authority.mint_persisted_launch(&id, &runtime_identity)?) + } else { + None }; + let sandbox_token = if let Some(authentication) = &launch_authentication { + Some( + authentication + .supervisor + .gateway_token + .expose_secret() + .to_string(), + ) + } else if let Some(issuer) = &state.sandbox_jwt_issuer { + Some(issuer.mint(&id)?.token) + } else { + None + }; + let launch_authentication = launch_authentication + .map(|authentication| { + serde_json::to_vec(&authentication) + .map_err(|error| Status::internal(format!("encode launch authentication: {error}"))) + }) + .transpose()?; let sandbox = state .compute - .create_sandbox(sandbox, sandbox_token, await_main_process_attachment) + .create_sandbox_authenticated( + sandbox, + sandbox_token, + launch_authentication, + await_main_process_attachment, + ) .await?; info!( @@ -1418,13 +1436,94 @@ async fn handle_start_sandbox_inner( let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) .await? .name; - let sandbox = state.compute.start_sandbox(&workspace, &req.name).await?; + let current = sandbox_by_name(state, &workspace, &req.name).await?; + let current_phase = SandboxPhase::try_from(current.phase()).unwrap_or(SandboxPhase::Unknown); + let launch_authentication = if current_phase == SandboxPhase::Ready { + Vec::new() + } else if state.sandbox_session_jwt_authority.is_some() { + let authentication = if matches!( + current_phase, + SandboxPhase::Stopped | SandboxPhase::Completed + ) { + mint_next_runtime_authentication(state, ¤t).await? + } else { + mint_persisted_authentication(state, ¤t)? + }; + serde_json::to_vec(&authentication) + .map_err(|error| Status::internal(format!("encode launch authentication: {error}")))? + } else { + Vec::new() + }; + let sandbox = state + .compute + .start_sandbox_authenticated(&workspace, &req.name, launch_authentication) + .await?; info!(sandbox_name = %req.name, "StartSandbox request completed successfully"); Ok(Response::new(SandboxResponse { sandbox: Some(sandbox), })) } +pub fn mint_persisted_authentication( + state: &ServerState, + sandbox: &Sandbox, +) -> Result { + let authority = state + .sandbox_session_jwt_authority + .as_ref() + .ok_or_else(|| Status::failed_precondition("sandbox session authority is unavailable"))?; + let metadata = sandbox + .metadata + .as_ref() + .ok_or_else(|| Status::failed_precondition("sandbox metadata is missing"))?; + let identity = + crate::auth::sandbox_session::PersistedSandboxIdentity::read(&metadata.annotations) + .map_err(|error| Status::failed_precondition(error.to_string()))?; + authority.mint_persisted_launch(sandbox.object_id(), &identity) +} + +async fn mint_next_runtime_authentication( + state: &Arc, + sandbox: &Sandbox, +) -> Result { + let authority = state + .sandbox_session_jwt_authority + .as_ref() + .ok_or_else(|| Status::failed_precondition("sandbox session authority is unavailable"))?; + let metadata = sandbox + .metadata + .as_ref() + .ok_or_else(|| Status::failed_precondition("sandbox metadata is missing"))?; + let current = + crate::auth::sandbox_session::PersistedSandboxIdentity::read(&metadata.annotations) + .map_err(|error| Status::failed_precondition(error.to_string()))?; + let next_epoch = current + .auth_epoch + .get() + .checked_add(1) + .and_then(|epoch| openshell_core::jwt::CredentialEpoch::new(epoch).ok()) + .ok_or_else(|| Status::internal("sandbox authorization epoch overflow"))?; + let next = crate::auth::sandbox_session::PersistedSandboxIdentity { + runtime_generation: current.runtime_generation, + auth_epoch: next_epoch, + }; + let authentication = authority.mint_persisted_launch(sandbox.object_id(), &next)?; + state + .store + .update_message_cas::( + sandbox.object_id(), + metadata.resource_version, + |updated| { + if let Some(metadata) = updated.metadata.as_mut() { + next.write(&mut metadata.annotations); + } + }, + ) + .await + .map_err(|error| Status::aborted(format!("persist sandbox runtime identity: {error}")))?; + Ok(authentication) +} + async fn sandbox_by_name( state: &Arc, workspace: &str, @@ -2269,10 +2368,7 @@ fn sandbox_relay_reachable(state: &ServerState, sandbox: &Sandbox) -> bool { let phase = SandboxPhase::try_from(sandbox.phase()).ok(); matches!(phase, Some(SandboxPhase::Ready)) || (matches!(phase, Some(SandboxPhase::Completed | SandboxPhase::Error)) - && state.supervisor_sessions.has_session(sandbox.object_id()) - && !state - .supervisor_sessions - .terminal_delivery_finalized(sandbox.object_id())) + && state.supervisor_sessions.has_session(sandbox.object_id())) } pub(super) async fn handle_create_ssh_session( @@ -5928,6 +6024,9 @@ mod tests { .supervisor_sessions .finalize_main_process_exit("sandbox-work") ); + assert!(sandbox_relay_reachable(&state, &sandbox)); + + assert!(state.supervisor_sessions.disconnect("sandbox-work")); assert!(!sandbox_relay_reachable(&state, &sandbox)); } @@ -6716,7 +6815,16 @@ mod tests { let mut sandbox = test_sandbox("cross-ws", Vec::new()); sandbox.metadata.as_mut().unwrap().workspace = "other-workspace".to_string(); + sandbox.set_phase(SandboxPhase::Completed as i32); state.store.put_message(&sandbox).await.unwrap(); + let (tx, _rx) = mpsc::channel(1); + let (shutdown_tx, _shutdown_rx) = oneshot::channel(); + let _ = state.supervisor_sessions.register( + sandbox.object_id().to_string(), + "retained-terminal-session".to_string(), + tx, + shutdown_tx, + ); // --- handle_watch_sandbox --- let err = handle_watch_sandbox( diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index f76d8d02e7..90dfb8dba0 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -315,6 +315,9 @@ pub struct ServerState { /// material that `certgen` writes. pub sandbox_jwt_issuer: Option>, + /// Launch-scoped gateway and Sandbox Protocol token authority. + pub sandbox_session_jwt_authority: Option>, + /// Authenticator that validates gateway-minted sandbox JWTs on every /// inbound request. Always set when `sandbox_jwt_issuer` is, so callers /// presenting a freshly minted token are recognized. @@ -423,6 +426,7 @@ impl ServerState { middleware_registry: Arc::new(MiddlewareRegistry::default()), oidc_cache, sandbox_jwt_issuer: None, + sandbox_session_jwt_authority: None, sandbox_jwt_authenticator: None, compute_driver_authenticator: None, grpc_rate_limiter, @@ -465,57 +469,71 @@ pub(crate) async fn run_server( // Load signing material before connecting remote extensions so their // startup Describe calls can authenticate with gateway-caller tokens. - let (sandbox_jwt_issuer, sandbox_jwt_authenticator) = if let Some(ref jwt) = config.gateway_jwt - { - let signing_pem = std::fs::read(&jwt.signing_key_path).map_err(|e| { - Error::config(format!( - "failed to read sandbox JWT signing key from {}: {e}", - jwt.signing_key_path.display() - )) - })?; - let public_pem = std::fs::read(&jwt.public_key_path).map_err(|e| { - Error::config(format!( - "failed to read sandbox JWT public key from {}: {e}", - jwt.public_key_path.display() - )) - })?; - let kid = std::fs::read_to_string(&jwt.kid_path) - .map_err(|e| { + let (sandbox_jwt_issuer, sandbox_jwt_authenticator, sandbox_session_jwt_authority) = + if let Some(ref jwt) = config.gateway_jwt { + let signing_pem = std::fs::read(&jwt.signing_key_path).map_err(|e| { Error::config(format!( - "failed to read sandbox JWT kid from {}: {e}", - jwt.kid_path.display() + "failed to read sandbox JWT signing key from {}: {e}", + jwt.signing_key_path.display() )) - })? - .trim() - .to_string(); - if kid.is_empty() { - return Err(Error::config(format!( - "sandbox JWT kid file {} is empty", - jwt.kid_path.display() - ))); - } - let issuer = Arc::new( - auth::sandbox_jwt::SandboxJwtIssuer::from_pem( - &signing_pem, - kid.clone(), - &jwt.gateway_id, - jwt.sandbox_token_ttl(), - ) - .map_err(Error::config)?, - ); - let authenticator = Arc::new( - auth::sandbox_jwt::SandboxJwtAuthenticator::from_pem(&public_pem, kid, &jwt.gateway_id) + })?; + let public_pem = std::fs::read(&jwt.public_key_path).map_err(|e| { + Error::config(format!( + "failed to read sandbox JWT public key from {}: {e}", + jwt.public_key_path.display() + )) + })?; + let kid = std::fs::read_to_string(&jwt.kid_path) + .map_err(|e| { + Error::config(format!( + "failed to read sandbox JWT kid from {}: {e}", + jwt.kid_path.display() + )) + })? + .trim() + .to_string(); + if kid.is_empty() { + return Err(Error::config(format!( + "sandbox JWT kid file {} is empty", + jwt.kid_path.display() + ))); + } + let issuer = Arc::new( + auth::sandbox_jwt::SandboxJwtIssuer::from_pem( + &signing_pem, + kid.clone(), + &jwt.gateway_id, + jwt.sandbox_token_ttl(), + ) .map_err(Error::config)?, - ); - info!( - gateway_id = %jwt.gateway_id, - ttl_secs = jwt.ttl_secs.map(std::num::NonZeroU64::get), - "gateway-minted sandbox JWT enabled" - ); - (Some(issuer), Some(authenticator)) - } else { - (None, None) - }; + ); + let authenticator = Arc::new( + auth::sandbox_jwt::SandboxJwtAuthenticator::from_pem( + &public_pem, + kid.clone(), + &jwt.gateway_id, + ) + .map_err(Error::config)?, + ); + let session_authority = Arc::new( + auth::sandbox_jwt::SandboxSessionJwtAuthority::from_pem( + &signing_pem, + &public_pem, + kid, + &jwt.gateway_id, + jwt.sandbox_token_ttl().unwrap_or(Duration::from_mins(15)), + ) + .map_err(Error::config)?, + ); + info!( + gateway_id = %jwt.gateway_id, + ttl_secs = jwt.ttl_secs.map(std::num::NonZeroU64::get), + "gateway-minted sandbox JWT enabled" + ); + (Some(issuer), Some(authenticator), Some(session_authority)) + } else { + (None, None, None) + }; let middleware_registrations = config_file .as_ref() @@ -666,6 +684,7 @@ pub(crate) async fn run_server( state.provider_profile_sources = provider_profile_sources; state.sandbox_jwt_issuer = sandbox_jwt_issuer.clone(); state.sandbox_jwt_authenticator = sandbox_jwt_authenticator; + state.sandbox_session_jwt_authority = sandbox_session_jwt_authority; if let Some(issuer) = sandbox_jwt_issuer { spawn_gateway_extension_token_refresh(issuer, gateway_extension_credentials); } @@ -701,15 +720,6 @@ pub(crate) async fn run_server( ) .await?; - if let Err(err) = state.compute.start_persisted_sandboxes().await { - warn!(error = %err, "Failed to start persisted sandboxes during startup"); - } - - state.compute.spawn_watchers(shutdown_rx.clone()); - ssh_sessions::spawn_session_reaper(store.clone(), Duration::from_hours(1)); - supervisor_session::spawn_relay_reaper(state.clone(), Duration::from_secs(30)); - provider_refresh::spawn_refresh_worker(state.clone(), Duration::from_mins(1)); - // Create the multiplexed service let service = MultiplexService::new(state.clone()); @@ -794,6 +804,39 @@ pub(crate) async fn run_server( ))); } + // Restored supervisors need the callback listeners while the compute + // driver reconciles persisted sandboxes. Serve them before starting that + // reconciliation so policy fetch and supervisor-session registration + // cannot deadlock gateway startup. + if let Err(err) = state + .compute + .start_persisted_sandboxes_with_authentication( + |sandbox| { + let state = state.clone(); + let sandbox = sandbox.clone(); + async move { + if state.sandbox_session_jwt_authority.is_none() { + return Ok(Vec::new()); + } + let authentication = grpc::mint_persisted_authentication(&state, &sandbox) + .map_err(|error| error.to_string())?; + serde_json::to_vec(&authentication) + .map_err(|error| format!("encode launch authentication: {error}")) + } + }, + |_| async { Ok(()) }, + |_| {}, + ) + .await + { + warn!(error = %err, "Failed to start persisted sandboxes during startup"); + } + + state.compute.spawn_watchers(shutdown_rx.clone()); + ssh_sessions::spawn_session_reaper(store.clone(), Duration::from_hours(1)); + supervisor_session::spawn_relay_reaper(state.clone(), Duration::from_secs(30)); + provider_refresh::spawn_refresh_worker(state.clone(), Duration::from_mins(1)); + shutdown_signal().await; info!("Shutdown signal received; stopping gateway"); state.gateway_shutting_down.store(true, Ordering::Release); diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index f03aa4d07f..f112522558 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -940,6 +940,14 @@ fn build_authenticator_chain(state: &ServerState) -> Option if let Some(driver) = state.compute_driver_authenticator.clone() { authenticators.push(driver); } + if let Some(authority) = state.sandbox_session_jwt_authority.clone() { + authenticators.push(Arc::new( + crate::auth::sandbox_jwt::SandboxSessionJwtAuthenticator::new( + authority, + state.store.clone(), + ), + )); + } if let Some(jwt) = state.sandbox_jwt_authenticator.clone() { authenticators.push(jwt); } diff --git a/crates/openshell-server/src/storage_proto.rs b/crates/openshell-server/src/storage_proto.rs index a1beb39ca8..282624e339 100644 --- a/crates/openshell-server/src/storage_proto.rs +++ b/crates/openshell-server/src/storage_proto.rs @@ -119,7 +119,7 @@ mod tests { const STORAGE_V1_SCHEMA_SHA256: &str = "79c72615d957fc0653c672f61998bf7d8d21b757bc05d07b3fff92bd70fc8f52"; const PUBLIC_RPC_SCHEMA_SHA256: &str = - "8ac68c71d93e6a5e56406b8df1882ee40c6066270969e03eb99803f0e6396fc1"; + "c6b4e1ae4d09e5d7de3831601953aa36901e51a24913a86f00559c53511fedde"; const DURABLE_SCHEMA_SHA256: &str = "369b36511c2e38b9df9621704a00123516c7538d8ee89a499158d7de5cee1882"; const PUBLIC_DURABLE_OVERLAP_SHA256: &str = diff --git a/crates/openshell-server/src/supervisor_session.rs b/crates/openshell-server/src/supervisor_session.rs index 24413f0f1b..ae20adac64 100644 --- a/crates/openshell-server/src/supervisor_session.rs +++ b/crates/openshell-server/src/supervisor_session.rs @@ -785,26 +785,36 @@ pub async fn handle_connect_supervisor( return Err(Status::internal("failed to send session accepted")); } - if superseded { - state - .supervisor_sessions - .replay_pending_relays(&sandbox_id, &tx) - .await; - } - if let Err(err) = state .compute .supervisor_session_connected(&sandbox_id, &hello.instance_id) .await { + // Do not expose SessionAccepted to the supervisor when the gateway + // could not durably record the connection. Dropping the buffered + // response forces a reconnect, which gives the state transition a + // fresh chance instead of leaving a healthy-looking supervisor tied + // to a sandbox that never reaches Ready. + state + .supervisor_sessions + .remove_if_current(&sandbox_id, &session_id); warn!( sandbox_id = %sandbox_id, session_id = %session_id, error = %err, "supervisor session: failed to mark sandbox ready" ); - } else { - state.telemetry.sandbox_session_connected(&sandbox_id); + return Err(Status::aborted( + "failed to persist supervisor session state; reconnect", + )); + } + state.telemetry.sandbox_session_connected(&sandbox_id); + + if superseded { + state + .supervisor_sessions + .replay_pending_relays(&sandbox_id, &tx) + .await; } // Step 4: Spawn the session loop that reads inbound messages. diff --git a/crates/openshell-supervisor-network/Cargo.toml b/crates/openshell-supervisor-network/Cargo.toml index 9961f4b31e..fa48e7a4bb 100644 --- a/crates/openshell-supervisor-network/Cargo.toml +++ b/crates/openshell-supervisor-network/Cargo.toml @@ -11,11 +11,15 @@ repository.workspace = true rust-version.workspace = true [dependencies] +openshell-binary-identity = { path = "../openshell-binary-identity" } openshell-core = { path = "../openshell-core", features = ["oauth"] } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } openshell-ocsf = { path = "../openshell-ocsf" } openshell-policy = { path = "../openshell-policy" } openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } +async-trait = "0.1" + apollo-parser = { workspace = true } aws-sigv4 = { version = "1", features = ["sign-http", "http1"] } aws-credential-types = { version = "1", features = ["hardcoded-credentials"] } @@ -24,6 +28,7 @@ http = { workspace = true } base64 = { workspace = true } bytes = { workspace = true } flate2 = "1" +futures = { workspace = true } glob = { workspace = true } hex = "0.4" hickory-proto = "0.26.1" @@ -61,7 +66,6 @@ tonic = { workspace = true } temp-env = "0.3" tokio = { workspace = true, features = ["test-util"] } tokio-tungstenite = { workspace = true } -futures = { workspace = true } tracing-subscriber = { workspace = true } tokio-stream = { workspace = true, features = ["net"] } diff --git a/crates/openshell-supervisor-network/data/sandbox-policy.rego b/crates/openshell-supervisor-network/data/sandbox-policy.rego index 278291a288..4053b23c9a 100644 --- a/crates/openshell-supervisor-network/data/sandbox-policy.rego +++ b/crates/openshell-supervisor-network/data/sandbox-policy.rego @@ -870,8 +870,8 @@ _matching_endpoint_configs := [cfg | # Full matched endpoint records are kept separate from the legacy # endpoint-config list, which intentionally contains only connection/L7 # metadata. The policy name and array index identify the endpoint within this -# policy generation while the complete endpoint preserves explicit protocol -# markers needed by later policy-DNS correlation. +# policy generation while the complete endpoint preserves protocol markers +# needed by later policy-DNS correlation. _policy_endpoint_records(policy_name, policy) := [record | some endpoint_index, ep in policy.endpoints @@ -892,12 +892,15 @@ _matching_endpoint_records := [record | # Endpoints eligible for policy DNS are a policy-data snapshot, not an # authorization decision. In particular, they do not depend on input.exec or -# grant access to any process. Only endpoints that explicitly opt into raw TCP -# and provide a resolvable host plus concrete ports are materialized. +# grant access to any process. Every supported endpoint protocol is carried by +# TCP, and an omitted protocol is the default L4 TCP form. Endpoints with a +# resolvable host plus concrete ports are therefore materialized regardless of +# whether later stream handling is L4, HTTP, WebSocket, or another L7 adapter. policy_dns_eligible_endpoint_records := [record | some policy_name, policy in data.network_policies some endpoint_index, ep in policy.endpoints - lower(object.get(ep, "protocol", "")) == "tcp" + protocol := lower(object.get(ep, "protocol", "tcp")) + protocol in {"tcp", "rest", "websocket", "graphql", "sql", "json-rpc", "mcp"} object.get(ep, "host", "") != "" ports := object.get(ep, "ports", []) count(ports) > 0 diff --git a/crates/openshell-supervisor-network/src/identity_source.rs b/crates/openshell-supervisor-network/src/identity_source.rs new file mode 100644 index 0000000000..f5446ddbda --- /dev/null +++ b/crates/openshell-supervisor-network/src/identity_source.rs @@ -0,0 +1,137 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! The in-pod binary-identity resolver (RFC 0012 runtime contract). +//! +//! RFC 0012 delivers executable identity on every +//! [`MediatedConnection`](openshell_isolation_interface::contract::MediatedConnection): +//! the backend resolves identity for the accepted connection before mediation. +//! An unresolved identity denies that connection. This is the in-pod +//! resolution mechanism — procfs, keyed by the workload-side TCP peer port — +//! kept in this crate on purpose: the proxy that consumes identity is here, and +//! so are procfs and the binary identity cache. Stronger backends may use a +//! different resolution mechanism without changing the contract. The result +//! type lives in the lower `openshell-isolation-interface` crate (network -> +//! interface -> core, acyclic). +//! +//! The legacy listener still resolves identity in the proxy hot path. The RFC +//! 0012 co-located source invokes this resolver before returning each accepted +//! connection, so mediation consumes the bound identity result. + +use std::sync::Arc; +use std::sync::atomic::AtomicU32; + +#[cfg(target_os = "linux")] +use openshell_binary_identity::ProcfsIdentityResolver as SharedProcfsIdentityResolver; +use openshell_isolation_interface::contract::{BinaryIdentity, ResolveError}; + +/// In-pod binary-identity resolver: reads and hashes the executable resolved +/// for an accepted connection from procfs. Resolution fails closed; it never +/// fabricates identity fields. +#[derive(Clone)] +pub struct ProcfsIdentityResolver { + /// The workload entrypoint PID, whose network namespace owns the peer + /// sockets the proxy resolves. Published once the agent starts. + pub entrypoint_pid: Arc, +} + +impl ProcfsIdentityResolver { + /// Resolve the executable identity behind an accepted workload connection. + pub fn resolve_connection( + &self, + workload_addr: std::net::SocketAddr, + proxy_addr: std::net::SocketAddr, + ) -> Result { + // procfs resolution is Linux-only; on other targets the supervisor has + // no procfs to read, so resolution fails closed. + #[cfg(target_os = "linux")] + { + self.resolve_via_procfs(workload_addr, proxy_addr) + } + #[cfg(not(target_os = "linux"))] + { + let _ = (workload_addr, proxy_addr); + Err(ResolveError::Failed( + "no procfs on this platform; identity resolution unavailable".to_string(), + )) + } + } +} + +#[cfg(target_os = "linux")] +impl ProcfsIdentityResolver { + fn resolve_via_procfs( + &self, + workload_addr: std::net::SocketAddr, + proxy_addr: std::net::SocketAddr, + ) -> Result { + use std::sync::atomic::Ordering; + + let entrypoint_pid = self.entrypoint_pid.load(Ordering::Acquire); + if entrypoint_pid == 0 { + // No workload yet: nothing to attribute the connection to. Fail + // closed so a binary-scoped rule cannot match an unattributed peer. + return Err(ResolveError::NotFound); + } + + let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); + let owners = crate::procfs::resolve_tcp_peer_socket_owners(entrypoint_pid, connection) + .map_err(|_| ResolveError::NotFound)?; + let resolver = SharedProcfsIdentityResolver::for_process_tree(entrypoint_pid); + let mut identities = Vec::with_capacity(owners.owners.len()); + for owner in owners.owners { + identities.push(resolver.resolve(owner.pid)?); + } + let Some(identity) = identities.first().cloned() else { + return Err(ResolveError::NotFound); + }; + if identities.iter().skip(1).any(|candidate| { + candidate.binary_path != identity.binary_path + || candidate.binary_digest != identity.binary_digest + || candidate.ancestors != identity.ancestors + || candidate.cmdline_paths != identity.cmdline_paths + }) { + return Err(ResolveError::Failed( + "shared socket owners have different policy identities".to_string(), + )); + } + Ok(identity) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Stands in for the mediation service: a binary-scoped rule can only be + /// authorized by a resolved identity carrying the fields it requires. + fn admits_binary_rule(result: Result) -> bool { + matches!(result, Ok(identity) if identity.binary_digest.is_some()) + } + + #[test] + fn fails_closed_before_the_workload_starts() { + // entrypoint_pid == 0 means no agent yet; identity must fail closed so a + // binary-scoped rule cannot be satisfied by an unattributed connection. + let resolver = ProcfsIdentityResolver { + entrypoint_pid: Arc::new(AtomicU32::new(0)), + }; + assert!(!admits_binary_rule(resolver.resolve_connection( + "127.0.0.1:12345".parse().unwrap(), + "127.0.0.1:3128".parse().unwrap(), + ))); + } + + #[test] + fn unknown_peer_fails_closed() { + // A peer port no live workload connection owns must resolve to an error, + // never a fabricated identity. + let resolver = ProcfsIdentityResolver { + entrypoint_pid: Arc::new(AtomicU32::new(u32::MAX - 1)), + }; + assert!(!admits_binary_rule(resolver.resolve_connection( + "127.0.0.1:1".parse().unwrap(), + "127.0.0.1:3128".parse().unwrap(), + ))); + } +} diff --git a/crates/openshell-supervisor-network/src/l7/rest.rs b/crates/openshell-supervisor-network/src/l7/rest.rs index d34a05bd1b..55ddc67734 100644 --- a/crates/openshell-supervisor-network/src/l7/rest.rs +++ b/crates/openshell-supervisor-network/src/l7/rest.rs @@ -3308,16 +3308,30 @@ fn parse_status_code(headers: &str) -> Option { code_str.parse().ok() } -/// Check if the response headers contain `Connection: close`. +/// Check whether the response is delimited by closing the connection. +/// +/// HTTP/1.0 closes by default unless the server explicitly negotiates +/// keep-alive. HTTP/1.1 keeps connections alive by default unless the server +/// sends `Connection: close`. fn parse_connection_close(headers: &str) -> bool { + let http_1_0 = headers + .lines() + .next() + .is_some_and(|line| line.starts_with("HTTP/1.0 ")); for line in headers.lines().skip(1) { let lower = line.to_ascii_lowercase(); if lower.starts_with("connection:") { let val = lower.split_once(':').map_or("", |(_, v)| v.trim()); - return val.contains("close"); + return if http_1_0 { + !val.split(',') + .any(|token| token.trim().eq_ignore_ascii_case("keep-alive")) + } else { + val.split(',') + .any(|token| token.trim().eq_ignore_ascii_case("close")) + }; } } - false + http_1_0 } fn response_is_event_stream(headers: &str) -> bool { @@ -5484,6 +5498,10 @@ mod tests { assert!(!parse_connection_close( "HTTP/1.1 200 OK\r\nHost: x\r\n\r\n" )); + assert!(parse_connection_close("HTTP/1.0 200 OK\r\nHost: x\r\n\r\n")); + assert!(!parse_connection_close( + "HTTP/1.0 200 OK\r\nConnection: keep-alive\r\n\r\n" + )); } #[test] @@ -5556,6 +5574,41 @@ mod tests { ); } + #[tokio::test] + async fn relay_response_http_1_0_defaults_to_connection_close() { + let response = b"HTTP/1.0 200 OK\r\nServer: test\r\n\r\nhello world"; + let (mut upstream_read, mut upstream_write) = tokio::io::duplex(4096); + let (mut client_read, mut client_write) = tokio::io::duplex(4096); + + tokio::spawn(async move { + upstream_write.write_all(response).await.unwrap(); + upstream_write.shutdown().await.unwrap(); + }); + + let outcome = tokio::time::timeout( + std::time::Duration::from_secs(2), + relay_response( + "POST", + &mut upstream_read, + &mut client_write, + RelayResponseOptions::default(), + ), + ) + .await + .expect("HTTP/1.0 close-delimited response should not deadlock") + .expect("HTTP/1.0 response should relay"); + assert!(matches!(outcome, RelayOutcome::Consumed)); + + client_write.shutdown().await.unwrap(); + let mut received = Vec::new(); + client_read.read_to_end(&mut received).await.unwrap(); + assert!( + received + .windows(b"hello world".len()) + .any(|value| value == b"hello world") + ); + } + #[tokio::test] async fn relay_response_no_framing_event_stream_reads_until_eof() { let response = diff --git a/crates/openshell-supervisor-network/src/l7/tls.rs b/crates/openshell-supervisor-network/src/l7/tls.rs index 63e04089cf..323561b0e3 100644 --- a/crates/openshell-supervisor-network/src/l7/tls.rs +++ b/crates/openshell-supervisor-network/src/l7/tls.rs @@ -8,7 +8,7 @@ //! store, terminates TLS from the client (presenting dynamic certs per hostname), //! inspects the plaintext HTTP, then re-encrypts to upstream using real root CAs. -use miette::{IntoDiagnostic, Result, miette}; +use miette::{IntoDiagnostic, Result, WrapErr, miette}; use rcgen::{CertificateParams, DnType, IsCa, KeyPair, KeyUsagePurpose}; use rustls::pki_types::{CertificateDer, PrivateKeyDer, ServerName}; use rustls::{ClientConfig, ServerConfig}; @@ -17,7 +17,6 @@ use std::io::BufReader; use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex}; use tokio::io::{AsyncRead, AsyncWrite}; -use tokio::net::TcpStream; use tokio_rustls::{TlsAcceptor, TlsConnector}; const MAX_CACHED_CERTS: usize = 256; @@ -67,6 +66,76 @@ impl SandboxCa { pub fn cert_pem(&self) -> &str { &self.ca_cert_pem } + + /// Returns the CA private key in PKCS#8 PEM format. + pub fn private_key_pem(&self) -> String { + self.ca_key.serialize_pem() + } + + /// Load a durable CA certificate and matching private key from absolute paths. + pub fn load_from_paths(certificate_path: &Path, private_key_path: &Path) -> Result { + if !certificate_path.is_absolute() || !private_key_path.is_absolute() { + return Err(miette!( + "proxy CA certificate and key paths must be absolute" + )); + } + if certificate_path == private_key_path { + return Err(miette!( + "proxy CA certificate and private key must use different paths" + )); + } + let certificate_pem = std::fs::read_to_string(certificate_path) + .into_diagnostic() + .wrap_err_with(|| { + format!("read proxy CA certificate {}", certificate_path.display()) + })?; + let private_key_pem = std::fs::read_to_string(private_key_path) + .into_diagnostic() + .wrap_err_with(|| { + format!("read proxy CA private key {}", private_key_path.display()) + })?; + Self::from_pem(&certificate_pem, &private_key_pem) + } + + /// Load a durable CA while preserving the exact certificate bytes supplied + /// by the provisioner for boundary launch replay. + pub fn from_pem(certificate_pem: &str, private_key_pem: &str) -> Result { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let ca_key = KeyPair::from_pem(private_key_pem) + .into_diagnostic() + .wrap_err("parse proxy CA private key")?; + let certificates = rustls_pemfile::certs(&mut certificate_pem.as_bytes()) + .collect::, _>>() + .into_diagnostic() + .wrap_err("parse proxy CA certificate")?; + if certificates.len() != 1 { + return Err(miette!( + "proxy CA certificate file must contain exactly one certificate" + )); + } + let private_key = rustls_pemfile::private_key(&mut private_key_pem.as_bytes()) + .into_diagnostic() + .wrap_err("parse proxy CA private key")? + .ok_or_else(|| miette!("proxy CA private key file contains no private key"))?; + ServerConfig::builder() + .with_no_client_auth() + .with_single_cert(certificates, private_key) + .into_diagnostic() + .wrap_err("proxy CA certificate and private key do not match")?; + + let params = CertificateParams::from_ca_cert_pem(certificate_pem) + .into_diagnostic() + .wrap_err("parse proxy CA signing certificate")?; + let ca_cert = params + .self_signed(&ca_key) + .into_diagnostic() + .wrap_err("initialize proxy CA signer")?; + Ok(Self { + ca_cert, + ca_key, + ca_cert_pem: certificate_pem.to_string(), + }) + } } /// A leaf certificate chain and private key for a specific hostname. @@ -170,11 +239,14 @@ impl ProxyTlsState { /// Accept TLS from a sandbox client, presenting a dynamic cert for the hostname. /// /// Returns a TLS stream that can be used for plaintext HTTP inspection. -pub async fn tls_terminate_client( - client: TcpStream, +pub async fn tls_terminate_client( + client: S, tls_state: &ProxyTlsState, hostname: &str, -) -> Result { +) -> Result +where + S: AsyncRead + AsyncWrite + Unpin + Send, +{ let acceptor = tls_state.acceptor_for(hostname)?; let tls_stream = acceptor.accept(client).await.into_diagnostic()?; Ok(tls_stream) @@ -558,4 +630,23 @@ mod tests { "bundle should contain at least one cert", ); } + + #[test] + fn durable_ca_round_trip_preserves_certificate_bytes() { + let generated = SandboxCa::generate().unwrap(); + let certificate = generated.cert_pem().to_string(); + let private_key = generated.private_key_pem(); + let loaded = SandboxCa::from_pem(&certificate, &private_key).unwrap(); + + assert_eq!(loaded.cert_pem(), certificate); + assert_eq!(loaded.private_key_pem(), private_key); + } + + #[test] + fn durable_ca_rejects_mismatched_key_and_relative_paths() { + let certificate = SandboxCa::generate().unwrap(); + let other_key = SandboxCa::generate().unwrap(); + assert!(SandboxCa::from_pem(certificate.cert_pem(), &other_key.private_key_pem()).is_err()); + assert!(SandboxCa::load_from_paths(Path::new("ca.pem"), Path::new("ca.key")).is_err()); + } } diff --git a/crates/openshell-supervisor-network/src/lib.rs b/crates/openshell-supervisor-network/src/lib.rs index b815e826c4..458e236f01 100644 --- a/crates/openshell-supervisor-network/src/lib.rs +++ b/crates/openshell-supervisor-network/src/lib.rs @@ -11,6 +11,7 @@ #[cfg(target_os = "windows")] pub mod host; pub mod identity; +pub mod identity_source; pub mod l7; pub mod opa; pub(crate) mod policy_dns; @@ -19,6 +20,7 @@ pub mod procfs; pub mod proxy; pub mod run; pub mod sigv4; +mod spiffe_endpoint; mod token_grant; pub mod upstream_proxy; diff --git a/crates/openshell-supervisor-network/src/opa.rs b/crates/openshell-supervisor-network/src/opa.rs index c8f303d64f..2bace98825 100644 --- a/crates/openshell-supervisor-network/src/opa.rs +++ b/crates/openshell-supervisor-network/src/opa.rs @@ -92,15 +92,6 @@ pub struct NetworkInput { pub cmdline_paths: Vec, } -pub(crate) fn network_binary_identity_required() -> bool { - std::env::var(openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY).map_or(true, |value| { - !matches!( - value.as_str(), - "relaxed" | "disabled" | "endpoint-only" | "false" | "0" - ) - }) -} - fn inject_runtime_policy_data(data: &mut serde_json::Value, require_binary_identity: bool) { let Some(obj) = data.as_object_mut() else { return; @@ -150,6 +141,7 @@ pub struct SandboxConfig { /// (one eval per CONNECT request). pub struct OpaEngine { engine: Mutex, + binary_identity_required: bool, generation: Arc, middleware_runner: RwLock, websocket_assembly_budget: crate::l7::websocket::WebSocketAssemblyBudget, @@ -278,11 +270,12 @@ impl OpaEngine { self.websocket_assembly_budget.clone() } - fn with_engine(engine: regorus::Engine) -> Self { + fn with_engine(engine: regorus::Engine, binary_identity_required: bool) -> Self { let generation = Arc::new(AtomicU64::new(0)); let (generation_tx, _) = watch::channel(0); Self { engine: Mutex::new(engine), + binary_identity_required, generation, middleware_runner: RwLock::new(ChainRunner::default()), websocket_assembly_budget: crate::l7::websocket::WebSocketAssemblyBudget::default(), @@ -291,6 +284,11 @@ impl OpaEngine { } } + /// Whether network authorization requires a workload binary identity. + pub const fn binary_identity_required(&self) -> bool { + self.binary_identity_required + } + fn advance_generation(&self) -> u64 { let generation = self.generation.fetch_add(1, Ordering::AcqRel) + 1; self.generation_tx.send_replace(generation); @@ -310,6 +308,36 @@ impl OpaEngine { policy_path: &Path, data_path: &Path, validate_middleware_config: Option<&MiddlewareConfigValidator>, + ) -> Result { + Self::from_files_with_identity_requirement( + policy_path, + data_path, + true, + validate_middleware_config, + ) + } + + /// Load local policy for a standalone proxy that cannot observe the + /// calling process. Authorization is based on the requested endpoint and + /// protocol rules instead of binary identity. + pub fn from_files_for_endpoint_only_proxy( + policy_path: &Path, + data_path: &Path, + validate_middleware_config: Option<&MiddlewareConfigValidator>, + ) -> Result { + Self::from_files_with_identity_requirement( + policy_path, + data_path, + false, + validate_middleware_config, + ) + } + + fn from_files_with_identity_requirement( + policy_path: &Path, + data_path: &Path, + require_binary_identity: bool, + validate_middleware_config: Option<&MiddlewareConfigValidator>, ) -> Result { let yaml_str = std::fs::read_to_string(data_path).map_err(|e| { miette::miette!("failed to read YAML data from {}: {e}", data_path.display()) @@ -318,7 +346,6 @@ impl OpaEngine { engine .add_policy_from_file(policy_path) .map_err(|e| miette::miette!("{e}"))?; - let require_binary_identity = network_binary_identity_required(); emit_binary_identity_mode(require_binary_identity, "files"); let data_json = preprocess_yaml_data( &yaml_str, @@ -328,14 +355,14 @@ impl OpaEngine { engine .add_data_json(&data_json) .map_err(|e| miette::miette!("{e}"))?; - Ok(Self::with_engine(engine)) + Ok(Self::with_engine(engine, require_binary_identity)) } /// Load policy rules and data from strings (data is YAML). /// /// Preprocesses the YAML data to expand access presets and validate L7 config. pub fn from_strings(policy: &str, data_yaml: &str) -> Result { - Self::from_strings_with_options(policy, data_yaml, network_binary_identity_required(), None) + Self::from_strings_with_options(policy, data_yaml, true, None) } pub fn from_strings_with_middleware_config( @@ -343,12 +370,7 @@ impl OpaEngine { data_yaml: &str, validate_middleware_config: Option<&MiddlewareConfigValidator>, ) -> Result { - Self::from_strings_with_options( - policy, - data_yaml, - network_binary_identity_required(), - validate_middleware_config, - ) + Self::from_strings_with_options(policy, data_yaml, true, validate_middleware_config) } #[cfg(test)] @@ -379,7 +401,7 @@ impl OpaEngine { engine .add_data_json(&data_json) .map_err(|e| miette::miette!("{e}"))?; - Ok(Self::with_engine(engine)) + Ok(Self::with_engine(engine, require_binary_identity)) } /// Create OPA engine from a typed proto policy. @@ -401,11 +423,7 @@ impl OpaEngine { /// gap between user-specified symlink paths (e.g., `/usr/bin/python3`) and /// kernel-resolved canonical paths (e.g., `/usr/bin/python3.11`). pub fn from_proto_with_pid(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> Result { - Self::from_proto_with_pid_and_binary_identity_required( - proto, - entrypoint_pid, - network_binary_identity_required(), - ) + Self::from_proto_with_pid_and_binary_identity_required(proto, entrypoint_pid, true) } fn from_proto_with_pid_and_binary_identity_required( @@ -472,7 +490,7 @@ impl OpaEngine { engine .add_data_json(&data_json) .map_err(|e| miette::miette!("{e}"))?; - Ok(Self::with_engine(engine)) + Ok(Self::with_engine(engine, require_binary_identity)) } /// Evaluate a network access request against the loaded policy. @@ -816,6 +834,7 @@ impl OpaEngine { /// generation comparison and callback linearizes state derived from an OPA /// snapshot with every policy reload and fail-closed transition. Callers /// must not perform I/O or other long-running work in `operation`. + #[allow(dead_code)] pub(crate) fn with_current_generation( &self, expected_generation: u64, @@ -2495,13 +2514,13 @@ process: "#; #[test] - fn policy_dns_snapshot_is_tcp_only_stable_and_generation_consistent() { + fn policy_dns_snapshot_includes_every_tcp_carried_endpoint() { let engine = OpaEngine::from_strings(TEST_POLICY, POLICY_DNS_SNAPSHOT_DATA).unwrap(); let snapshot = engine.policy_dns_eligibility_snapshot().unwrap(); assert_eq!(snapshot.generation, engine.current_generation()); - assert_eq!(snapshot.endpoints.len(), 2); + assert_eq!(snapshot.endpoints.len(), 4); assert_eq!(snapshot.endpoints[0].policy_name, "dns_transport"); assert_eq!(snapshot.endpoints[0].endpoint_index, 0); assert_eq!( @@ -2514,15 +2533,17 @@ process: panic!("eligible endpoint must retain concrete ports"); }; assert_eq!(ports.as_ref(), &[53.into(), 853.into()]); - assert_eq!(snapshot.endpoints[1].endpoint_index, 4); + assert_eq!(snapshot.endpoints[1].endpoint_index, 1); + assert_eq!(snapshot.endpoints[2].endpoint_index, 2); + assert_eq!(snapshot.endpoints[3].endpoint_index, 4); engine .reload(TEST_POLICY, POLICY_DNS_SNAPSHOT_DATA) .unwrap(); let reloaded = engine.policy_dns_eligibility_snapshot().unwrap(); assert_eq!(reloaded.generation, snapshot.generation + 1); - assert_eq!(reloaded.endpoints.len(), 2); - assert_eq!(reloaded.endpoints[1].endpoint_index, 4); + assert_eq!(reloaded.endpoints.len(), 4); + assert_eq!(reloaded.endpoints[3].endpoint_index, 4); } #[test] @@ -2540,7 +2561,13 @@ process: fn policy_dns_snapshot_accepts_the_default_multi_policy_shape() { let engine = OpaEngine::from_strings(TEST_POLICY, TEST_DATA_YAML).unwrap(); let snapshot = engine.policy_dns_eligibility_snapshot().unwrap(); - assert!(snapshot.endpoints.is_empty()); + assert_eq!(snapshot.endpoints.len(), 15); + assert!( + snapshot + .endpoints + .iter() + .any(|endpoint| endpoint.policy_name == "claude_code") + ); } #[test] @@ -3779,7 +3806,7 @@ network_policies: .expect("policy should load"); rego.add_data_json(&data_json.to_string()) .expect("data should load"); - let engine = OpaEngine::with_engine(rego); + let engine = OpaEngine::with_engine(rego, true); let input = l7_websocket_graphql_input( "realtime.graphql.com", serde_json::json!([{ diff --git a/crates/openshell-supervisor-network/src/policy_dns/mod.rs b/crates/openshell-supervisor-network/src/policy_dns/mod.rs index 6c692e4959..7033e6d6fc 100644 --- a/crates/openshell-supervisor-network/src/policy_dns/mod.rs +++ b/crates/openshell-supervisor-network/src/policy_dns/mod.rs @@ -139,18 +139,29 @@ impl PolicyDnsService { // The trusted resolver is invoked only after the immutable snapshot // proved policy eligibility. It never consults sandbox resolver state. let endpoint_context = eligible_endpoint_context(&eligible); - let trusted_answer = match self.resolver.resolve(&normalized_name, family).await { - Ok(answer) => answer, - Err(error) => { - emit_dns_failure( - &normalized_name, - family, - &endpoint_context, - snapshot.generation, - resolver_failure_detail(&error), - "Policy DNS trusted resolver query failed", - ); - return Err(PolicyDnsError::Resolver(error)); + let trusted_answer = if is_host_gateway_alias(normalized_name.as_str()) { + let address = self + .trusted_host_gateway + .filter(|address| family.accepts(*address)) + .ok_or(PolicyDnsError::NoValidAddress)?; + TrustedAnswer { + addresses: vec![address], + ttl: MAX_MAPPING_TTL, + } + } else { + match self.resolver.resolve(&normalized_name, family).await { + Ok(answer) => answer, + Err(error) => { + emit_dns_failure( + &normalized_name, + family, + &endpoint_context, + snapshot.generation, + resolver_failure_detail(&error), + "Policy DNS trusted resolver query failed", + ); + return Err(PolicyDnsError::Resolver(error)); + } } }; let ttl = clamp_mapping_ttl(trusted_answer.ttl); @@ -286,6 +297,7 @@ fn eligible_endpoints( name.as_str(), name.as_str(), trusted_host_gateway, + None, &raw_allowed_ips, exact_declared_host, ) @@ -805,26 +817,42 @@ process: { run_as_user: sandbox, run_as_group: sandbox } .unwrap(); assert_eq!(mapping.record.contracts[0].pinned_addresses, [trusted]); + assert_eq!(service.resolver.calls.load(Ordering::SeqCst), 0); } #[tokio::test] - async fn reserved_gateway_alias_rejects_mismatch_metadata_private_and_wrong_family_answers() { + async fn reserved_gateway_alias_accepts_an_exact_private_backend_gateway() { + let trusted: IpAddr = "172.23.0.1".parse().unwrap(); + let service = gateway_service(Vec::new(), Some(trusted)); + let now = Instant::now(); + + let answer = service + .answer_query("host.openshell.internal", AddressFamily::Ipv4, now) + .await + .unwrap(); + let mapping = service + .store + .lookup(answer.address, 8080, answer.policy_generation, now) + .unwrap(); + + assert_eq!(mapping.record.contracts[0].pinned_addresses, [trusted]); + assert_eq!(service.resolver.calls.load(Ordering::SeqCst), 0); + } + + #[tokio::test] + async fn reserved_gateway_alias_rejects_wrong_address_family_without_resolver_fallback() { let trusted: IpAddr = "169.254.1.2".parse().unwrap(); - for (family, address) in [ - (AddressFamily::Ipv4, "169.254.1.3"), - (AddressFamily::Ipv4, "169.254.169.254"), - (AddressFamily::Ipv4, "10.2.3.4"), - (AddressFamily::Ipv6, "fe80::2"), - ] { - let service = gateway_service(vec![address.parse().unwrap()], Some(trusted)); - let result = service - .answer_query("host.openshell.internal", family, Instant::now()) - .await; - assert!( - matches!(result, Err(PolicyDnsError::NoValidAddress)), - "{address} must not satisfy the trusted gateway contract" - ); - } + let service = gateway_service(vec!["fe80::2".parse().unwrap()], Some(trusted)); + let result = service + .answer_query( + "host.openshell.internal", + AddressFamily::Ipv6, + Instant::now(), + ) + .await; + + assert!(matches!(result, Err(PolicyDnsError::NoValidAddress))); + assert_eq!(service.resolver.calls.load(Ordering::SeqCst), 0); } struct BlockingResolver { diff --git a/crates/openshell-supervisor-network/src/policy_dns/runtime.rs b/crates/openshell-supervisor-network/src/policy_dns/runtime.rs index ad6095efa9..23b8a6c0b5 100644 --- a/crates/openshell-supervisor-network/src/policy_dns/runtime.rs +++ b/crates/openshell-supervisor-network/src/policy_dns/runtime.rs @@ -3,12 +3,16 @@ //! Runtime-owned policy DNS listeners for combined Linux supervisors. -use super::resolver::MAX_DNS_MESSAGE_BYTES; -use super::store::{ResolvedEndpointStore, StoreConfig, SyntheticPools}; -use super::{PolicyDnsService, SocketTrustedResolver, wire}; use crate::opa::OpaEngine; +use crate::policy_dns::resolver::MAX_DNS_MESSAGE_BYTES; +use crate::policy_dns::store::{ResolvedEndpointStore, StoreConfig, SyntheticPools}; +use crate::policy_dns::{PolicyDnsService, SocketTrustedResolver, wire}; +use futures::{FutureExt as _, StreamExt as _, stream::FuturesUnordered}; use miette::{IntoDiagnostic, Result, WrapErr}; use openshell_core::net::set_tcp_nodelay_best_effort; +use openshell_isolation_interface::contract::{ + DnsTransport, NetworkMediationSource, PendingDnsQuery, +}; use openshell_ocsf::{ConfigStateChangeBuilder, SeverityId, StateId, StatusId, ocsf_emit}; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; use std::sync::Arc; @@ -20,6 +24,23 @@ const IPV6_POOL_PREFIX: u8 = 119; const IPV4_EPOCH_WINDOWS: u64 = 1 << (IPV4_POOL_PREFIX - 15); const MAX_MAPPINGS: usize = 1024; const MAX_CONCURRENT_UDP_QUERIES: usize = 64; +const MEDIATION_ACCEPT_WINDOW: usize = 32; +const MEDIATION_RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(100); +const MEDIATION_MAX_RETRY_DELAY: std::time::Duration = std::time::Duration::from_secs(5); + +async fn accept_mediated_dns(source: Arc) -> PendingDnsQuery { + let mut delay = MEDIATION_RETRY_DELAY; + loop { + match source.accept_dns().await { + Ok(query) => return query, + Err(error) => { + tracing::warn!(%error, "mediated DNS accept failed; retrying"); + tokio::time::sleep(delay).await; + delay = delay.saturating_mul(2).min(MEDIATION_MAX_RETRY_DELAY); + } + } + } +} #[derive(Debug, Clone)] pub(crate) struct PolicyDnsRuntimeConfig { @@ -63,6 +84,97 @@ pub(crate) struct PolicyDnsRuntime { } impl PolicyDnsRuntime { + /// Start policy DNS over an isolation-backend exchange source. No UDP or + /// TCP listener is bound in the supervisor namespace. + pub(crate) fn start_mediated( + policy: Arc, + source: Arc, + trusted_host_gateway: Option, + config: PolicyDnsRuntimeConfig, + mut engine_ready: tokio::sync::watch::Receiver, + ) -> Result { + let upstream = trusted_resolver_from_resolv_conf()?; + let store = Arc::new(ResolvedEndpointStore::new( + StoreConfig::new(config.pools, MAX_MAPPINGS) + .map_err(|error| miette::miette!(error.to_string()))?, + )); + let service = Arc::new(PolicyDnsService::new( + policy, + SocketTrustedResolver::new(upstream), + store.clone(), + trusted_host_gateway, + )); + let task = tokio::spawn(async move { + if engine_ready.wait_for(|ready| *ready).await.is_err() { + return; + } + let mut accepts = FuturesUnordered::new(); + for _ in 0..MEDIATION_ACCEPT_WINDOW { + let source = source.clone(); + accepts.push(accept_mediated_dns(source).boxed()); + } + loop { + let Some(query) = accepts.next().await else { + return; + }; + let source = source.clone(); + accepts.push(accept_mediated_dns(source).boxed()); + let service = service.clone(); + tokio::spawn(async move { + let timing = query.timing.clone(); + let response = match query.transport { + DnsTransport::Udp => { + wire::handle_udp_query_with_ipv6(&service, &query.message, false).await + } + DnsTransport::Tcp => { + wire::handle_tcp_query_with_ipv6(&service, &query.message, false).await + } + } + .map_err(|error| { + openshell_isolation_interface::contract::BackendError::Process(format!( + "policy DNS response failed: {error}" + )) + }); + if query.response.send(response).is_err() { + tracing::warn!("sandbox DNS response channel closed before delivery"); + } + tracing::debug!( + target: "openshell::dns_timing", + notification_to_queue_us = timing + .sandbox_notification_to_queue + .as_micros(), + queue_wait_us = timing.sandbox_queue_wait.as_micros(), + supervisor_processing_us = timing + .supervisor_received_at + .elapsed() + .as_micros(), + "mediated DNS query timing" + ); + }); + } + }); + let expiry_store = store.clone(); + let expiry_task = tokio::spawn(async move { + let mut interval = tokio::time::interval(std::time::Duration::from_secs(1)); + loop { + interval.tick().await; + let _ = expiry_store.expire(std::time::Instant::now()); + } + }); + ocsf_emit!( + ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "ready") + .message("Policy DNS connected to isolation boundary") + .build() + ); + Ok(Self { + store, + tasks: vec![task, expiry_task], + }) + } + pub(crate) fn start( policy: Arc, udp: tokio::net::UdpSocket, @@ -213,6 +325,56 @@ fn trusted_resolver_from_resolv_conf() -> Result { mod tests { use super::*; + #[tokio::test(start_paused = true)] + async fn mediated_dns_accept_retries_errors_with_backoff() { + use openshell_isolation_interface::contract::{ + BackendError, MediationTiming, ResolveError, + }; + use std::sync::atomic::{AtomicUsize, Ordering}; + + struct RecoveringSource(AtomicUsize); + + #[async_trait::async_trait] + impl NetworkMediationSource for RecoveringSource { + async fn accept_tcp( + &self, + ) -> std::result::Result< + openshell_isolation_interface::contract::PendingTcpOpen, + BackendError, + > { + Err(BackendError::Unavailable( + "TCP not used by this test".into(), + )) + } + + async fn accept_dns(&self) -> std::result::Result { + if self.0.fetch_add(1, Ordering::AcqRel) < 2 { + return Err(BackendError::Unavailable("injected disconnect".into())); + } + let (response, _) = tokio::sync::oneshot::channel(); + Ok(PendingDnsQuery { + message: b"recovered query".to_vec(), + transport: DnsTransport::Udp, + binary_identity: Err(ResolveError::Failed("unknown sender".into())), + timing: MediationTiming::default(), + response, + }) + } + } + + let source = Arc::new(RecoveringSource(AtomicUsize::new(0))); + let start = tokio::time::Instant::now(); + let query = accept_mediated_dns(source.clone()).await; + assert_eq!(query.message, b"recovered query"); + assert_eq!(source.0.load(Ordering::Acquire), 3); + assert!(start.elapsed() >= MEDIATION_RETRY_DELAY * 3); + // The same source remains usable when the accept window replenishes. + assert_eq!( + accept_mediated_dns(source).await.message, + b"recovered query" + ); + } + #[test] fn production_pool_is_disjoint_from_workload_veth() { let workload: ipnet::IpNet = "10.200.0.0/24".parse().unwrap(); diff --git a/crates/openshell-supervisor-network/src/policy_dns/store.rs b/crates/openshell-supervisor-network/src/policy_dns/store.rs index ea2cd7c0f8..439cdcfa36 100644 --- a/crates/openshell-supervisor-network/src/policy_dns/store.rs +++ b/crates/openshell-supervisor-network/src/policy_dns/store.rs @@ -73,6 +73,17 @@ pub(crate) struct MappingLookup { } impl MappingLookup { + pub(crate) fn pinned_addresses(&self) -> Vec { + let mut seen = HashSet::new(); + self.record + .contracts + .iter() + .filter(|contract| contract.port == self.port) + .flat_map(|contract| contract.pinned_addresses.iter().copied()) + .filter(|address| seen.insert(*address)) + .collect() + } + pub(crate) fn endpoint_ids(&self) -> impl Iterator { self.record .contracts diff --git a/crates/openshell-supervisor-network/src/proxy.rs b/crates/openshell-supervisor-network/src/proxy.rs index 6689725368..8c0f728de8 100644 --- a/crates/openshell-supervisor-network/src/proxy.rs +++ b/crates/openshell-supervisor-network/src/proxy.rs @@ -7,14 +7,15 @@ pub(crate) mod destination; mod egress; mod relay; -#[cfg(target_os = "linux")] use crate::identity::BinaryIdentityCache; use crate::l7::tls::ProxyTlsState; use crate::opa::{NetworkAction, OpaEngine, PolicyGenerationGuard}; #[cfg(target_os = "linux")] -use crate::policy_dns::{MappingLookupError, PolicyEndpointId, ResolvedEndpointStore}; +use crate::policy_dns::PolicyEndpointId; +use crate::policy_dns::{MappingLookupError, ResolvedEndpointStore}; use crate::policy_local::{POLICY_LOCAL_HOST, PolicyLocalContext}; use crate::upstream_proxy::{self, UpstreamProxyConfig}; +use futures::{FutureExt as _, StreamExt as _, stream::FuturesUnordered}; use miette::{IntoDiagnostic, Result}; use openshell_core::activity::{ActivitySender, try_record_activity}; use openshell_core::denial::DenialEvent; @@ -25,6 +26,10 @@ use openshell_core::net::{ use openshell_core::policy::ProxyPolicy; use openshell_core::provider_credentials::{ProviderCredentialSnapshot, ProviderCredentialState}; use openshell_core::secrets::{self, SecretResolver, rewrite_header_line_checked}; +use openshell_isolation_interface::contract::{ + BinaryIdentity as ContractBinaryIdentity, BoundaryDuplexStream, MediationTiming, + NetworkMediationSource, PendingTcpOpen, ResolveError, TcpOpenDecision, TcpOpenDenial, +}; use openshell_ocsf::{ ActionId, ActivityId, DispositionId, Endpoint, HttpActivityBuilder, HttpRequest, NetworkActivityBuilder, Process, SeverityId, StatusId, Url as OcsfUrl, ocsf_emit, @@ -34,19 +39,39 @@ use std::mem::size_of; use std::net::{IpAddr, SocketAddr}; use std::path::PathBuf; use std::sync::Arc; -#[cfg(target_os = "linux")] -use std::sync::atomic::AtomicU32; +use std::sync::atomic::{AtomicU32, Ordering}; use tokio::io::{ - AsyncRead as TokioAsyncRead, AsyncReadExt, AsyncWrite as TokioAsyncWrite, AsyncWriteExt, + AsyncBufReadExt, AsyncRead as TokioAsyncRead, AsyncReadExt, AsyncWrite as TokioAsyncWrite, + AsyncWriteExt, }; -use tokio::net::{TcpListener, TcpStream}; +use tokio::net::TcpListener; +#[cfg(any(target_os = "linux", test))] +use tokio::net::TcpStream; use tokio::sync::mpsc; use tokio::task::JoinHandle; use tracing::{debug, warn}; +type ProxyClient = tokio::io::BufReader; +type AcceptedProxyConnection = ( + BoundaryDuplexStream, + Option>, + Option<(SocketAddr, SocketAddr)>, + Option, +); + +struct TransparentOpen { + destination: SocketAddr, + authorization: Option<(EgressDecision, destination::UpstreamConnector)>, +} + +enum ProxyAcceptError { + Listener(std::io::Error), + Source(openshell_isolation_interface::contract::BackendError), +} + use self::destination::{ - DestinationDenial, DestinationDenialKind, DestinationRequest, build_validation_plan, - validate_destination, + DestinationDenial, DestinationDenialKind, DestinationRequest, build_pinned_validation_plan, + build_validation_plan, validate_destination, }; use self::egress::{ EgressDecision, EgressIntent, EndpointDecision, IdentityUnavailableReason, L7ConfigSnapshot, @@ -54,6 +79,25 @@ use self::egress::{ }; const MAX_HEADER_BYTES: usize = 8192; +const MEDIATION_ACCEPT_WINDOW: usize = 32; + +struct NetworkOpenTimingGuard { + timing: MediationTiming, + operation: &'static str, +} + +impl Drop for NetworkOpenTimingGuard { + fn drop(&mut self) { + tracing::debug!( + target: "openshell::network_open_timing", + operation = self.operation, + notification_to_queue_us = self.timing.sandbox_notification_to_queue.as_micros(), + queue_wait_us = self.timing.sandbox_queue_wait.as_micros(), + supervisor_processing_us = self.timing.supervisor_received_at.elapsed().as_micros(), + "mediated network open timing" + ); + } +} const TUNNEL_PROTOCOL_PEEK_BYTES: usize = crate::l7::rest::HTTP2_PRIOR_KNOWLEDGE_PREFACE.len(); #[cfg(not(test))] const TUNNEL_PROTOCOL_PEEK_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(100); @@ -65,8 +109,6 @@ const TUNNEL_PROTOCOL_PEEK_POLL: std::time::Duration = std::time::Duration::from const TUNNEL_PROTOCOL_PEEK_POLL: std::time::Duration = std::time::Duration::from_millis(1); const FORWARD_ENCODED_SLASH_REJECTION_DETAIL: &str = "request-target contains an encoded '/' (%2F) which is not allowed on this endpoint"; -#[cfg(target_os = "linux")] -const SIDECAR_SUPERVISOR_TOPOLOGY: &str = "sidecar"; fn emit_credential_endpoint_mismatch(host: &str, port: u16, policy_name: &str) { let event = HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) @@ -92,11 +134,11 @@ fn emit_credential_endpoint_mismatch(host: &str, port: u16, policy_name: &str) { ocsf_emit!(finding); } -/// Hostnames injected by compute drivers as hosts-file aliases for the host +/// Hostnames injected by compute drivers as `/etc/hosts` aliases for the host /// machine. Traffic to these names is eligible for the trusted-gateway SSRF /// exemption when the resolved IP matches the driver-injected value read from -/// the platform hosts file at proxy startup. -const HOST_GATEWAY_ALIASES: &[&str] = &[ +/// `/etc/hosts` at proxy startup. +pub(crate) const HOST_GATEWAY_ALIASES: &[&str] = &[ "host.openshell.internal", "host.containers.internal", "host.docker.internal", @@ -138,60 +180,6 @@ pub struct ProxyHandle { exited_rx: Option>, } -#[derive(Clone)] -pub(crate) enum ProxyIdentityMode { - /// Linux supervisor mode: bind a CONNECT request to the process that owns - /// the redirected TCP socket via procfs. - #[cfg(target_os = "linux")] - Procfs { - identity_cache: Arc, - entrypoint_pid: Arc, - }, - /// Host-side mode for platforms where procfs socket ownership is - /// unavailable. MXC uses this on Windows: every connection redirected to - /// the per-sandbox listener is evaluated as the configured sandbox agent - /// identity. - #[cfg(any(not(target_os = "linux"), test))] - Static { - binary_path: PathBuf, - binary_sha256: String, - }, -} - -impl ProxyIdentityMode { - #[cfg(target_os = "linux")] - pub(crate) fn procfs( - identity_cache: Arc, - entrypoint_pid: Arc, - ) -> Self { - Self::Procfs { - identity_cache, - entrypoint_pid, - } - } - - #[cfg(any(not(target_os = "linux"), test))] - pub(crate) fn static_binary(path: impl Into) -> Result { - let binary_path = path.into(); - let binary_sha256 = crate::procfs::file_sha256(&binary_path)?; - Ok(Self::Static { - binary_path, - binary_sha256, - }) - } - - fn entrypoint_pid(&self) -> u32 { - match self { - #[cfg(target_os = "linux")] - Self::Procfs { entrypoint_pid, .. } => { - entrypoint_pid.load(std::sync::atomic::Ordering::Acquire) - } - #[cfg(any(not(target_os = "linux"), test))] - Self::Static { .. } => 0, - } - } -} - impl ProxyHandle { /// Start the proxy with OPA engine for policy evaluation. /// @@ -202,7 +190,8 @@ impl ProxyHandle { policy: &ProxyPolicy, bind_addr: Option, opa_engine: Arc, - identity_mode: Arc, + identity_cache: Arc, + entrypoint_pid: Arc, tls_state: Option>, provider_credentials: Option, policy_local_ctx: Option>, @@ -210,6 +199,9 @@ impl ProxyHandle { activity_tx: Option, engine_ready: tokio::sync::watch::Receiver, upstream_proxy_args: &upstream_proxy::UpstreamProxyArgs, + backend_host_gateway: Option, + network_mediation_source: Option>, + policy_dns_store: Option>, ) -> Result { // Use override bind_addr, fall back to policy http_addr, then default // to loopback:3128. The default allows the proxy to function when no @@ -225,27 +217,40 @@ impl ProxyHandle { )); } - let listener = TcpListener::bind(http_addr).await.into_diagnostic()?; - let local_addr = listener.local_addr().into_diagnostic()?; + let source_backed = network_mediation_source.is_some(); + let listener = if source_backed { + None + } else { + Some(TcpListener::bind(http_addr).await.into_diagnostic()?) + }; + let local_addr = match listener.as_ref() { + Some(listener) => listener.local_addr().into_diagnostic()?, + None => http_addr, + }; { let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(ActivityId::Listen) .severity(SeverityId::Informational) .status(StatusId::Success) .dst_endpoint(Endpoint::from_ip(local_addr.ip(), local_addr.port())) - .message(format!("Proxy listening on {local_addr}")) + .message(if source_backed { + "Proxy consuming isolation-boundary streams".to_string() + } else { + format!("Proxy listening on {local_addr}") + }) .build(); ocsf_emit!(event); } - // Detect the trusted host gateway IP from the platform hosts file - // before user code runs. This is read once at startup so later hosts - // file modifications by sandbox workloads cannot influence it. + // Detect the trusted host gateway IP from /etc/hosts before user code + // runs. This is read once at startup so later /etc/hosts modifications + // by sandbox workloads cannot influence the stored value. let trusted_host_gateway: Arc> = Arc::new(detect_trusted_host_gateway()); + let backend_host_gateway = Arc::new(backend_host_gateway); if let Some(ref ip) = *trusted_host_gateway { tracing::info!( %ip, - "Trusted host gateway detected from platform hosts file; \ + "Trusted host gateway detected from /etc/hosts; \ host-gateway aliases exempt from SSRF always-blocked check" ); } @@ -323,22 +328,92 @@ impl ProxyHandle { } } + let mut network_accepts = network_mediation_source.as_ref().map(|source| { + let accepts = FuturesUnordered::new(); + for _ in 0..MEDIATION_ACCEPT_WINDOW { + let source = source.clone(); + accepts.push(async move { source.accept_tcp().await }.boxed()); + } + accepts + }); + // Transparent opens require policy evaluation and destination + // validation before the sandbox may complete connect(2). Keep + // those potentially expensive operations out of the accept loop: + // serial preauthorization turns bursts of DNS-driven TCP opens + // into head-of-line blocking even though the source itself can + // accept a window of requests concurrently. + let (preauthorized_tx, mut preauthorized_rx) = + mpsc::channel(MEDIATION_ACCEPT_WINDOW * 2); let mut consecutive_resource_errors: u32 = 0; let mut consecutive_unknown_errors: u32 = 0; loop { - match listener.accept().await { - Ok((stream, _addr)) => { + let accepted = if let Some(source) = network_mediation_source.as_ref() { + let accepts = network_accepts + .as_mut() + .expect("mediation source has an accept window"); + tokio::select! { + pending = accepts.next() => { + let pending = pending.expect("accept window is never empty"); + let source = source.clone(); + accepts.push(async move { source.accept_tcp().await }.boxed()); + match pending { + Ok(connection) => { + let tx = preauthorized_tx.clone(); + let dns_store = policy_dns_store.clone(); + let opa = opa_engine.clone(); + let backend_gateway = *backend_host_gateway; + let trusted_gateway = *trusted_host_gateway; + tokio::spawn(async move { + if let Some(connection) = preauthorize_transparent_open( + connection, + dns_store.as_ref(), + &opa, + backend_gateway, + trusted_gateway, + ) + .await + { + let _ = tx.send(connection).await; + } + }); + continue; + } + Err(error) => Err(ProxyAcceptError::Source(error)), + } + } + Some(connection) = preauthorized_rx.recv() => Ok(connection), + } + } else { + let listener = listener + .as_ref() + .expect("listener exists without a mediation source"); + listener + .accept() + .await + .map(|(stream, _)| { + set_tcp_nodelay_best_effort(&stream); + let workload_addr = stream.peer_addr().ok(); + let proxy_addr = stream.local_addr().ok(); + let stream: BoundaryDuplexStream = Box::new(stream); + (stream, None, workload_addr.zip(proxy_addr), None) + }) + .map_err(ProxyAcceptError::Listener) + }; + match accepted { + Ok((stream, supplied_identity, socket_addrs, transparent_destination)) => { consecutive_resource_errors = 0; consecutive_unknown_errors = 0; - set_tcp_nodelay_best_effort(&stream); let opa = opa_engine.clone(); - let identity = identity_mode.clone(); + let cache = identity_cache.clone(); + let spid = entrypoint_pid.clone(); let tls = tls_state.clone(); let policy_local = policy_local_ctx.clone(); let proposals = agent_proposals.clone(); let gw = trusted_host_gateway.clone(); + let backend_gw = backend_host_gateway.clone(); let up_proxy = upstream_proxy.clone(); let credentials = provider_credentials.clone(); + let dns_store = policy_dns_store.clone(); let resolver = provider_credentials .as_ref() .and_then(ProviderCredentialState::resolver); @@ -351,13 +426,19 @@ impl ProxyHandle { let atx = activity_tx.clone(); tokio::spawn(async move { #[allow(clippy::large_futures)] - if let Err(err) = handle_tcp_connection( - stream, + if let Err(err) = handle_mediated_connection( + tokio::io::BufReader::new(stream), + supplied_identity, + socket_addrs, + transparent_destination, + dns_store, opa, - identity, + cache, + spid, tls, policy_local, proposals, + backend_gw, gw, up_proxy, credentials, @@ -378,7 +459,19 @@ impl ProxyHandle { } }); } - Err(err) => { + Err(ProxyAcceptError::Source(err)) => { + let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(ActivityId::Fail) + .severity(SeverityId::High) + .status(StatusId::Failure) + .message(format!( + "Network-mediation source failed; proxy accept loop exiting: {err}" + )) + .build(); + ocsf_emit!(event); + break; + } + Err(ProxyAcceptError::Listener(err)) => { match classify_accept_error( &err, &mut consecutive_resource_errors, @@ -416,7 +509,7 @@ impl ProxyHandle { }); Ok(Self { - http_addr: Some(local_addr), + http_addr: (!source_backed).then_some(local_addr), join, exited_rx: Some(exited_rx), }) @@ -432,6 +525,199 @@ impl ProxyHandle { } } +async fn preauthorize_transparent_open( + connection: PendingTcpOpen, + policy_dns_store: Option<&Arc>, + opa_engine: &OpaEngine, + backend_host_gateway: Option, + trusted_host_gateway: Option, +) -> Option { + let PendingTcpOpen { + stream, + binary_identity, + destination, + socket: _, + policy_generation: _, + timing, + decision: completion, + } = connection; + let _timing = NetworkOpenTimingGuard { + timing, + operation: "tcp", + }; + let host = match transparent_destination_host(destination, policy_dns_store, opa_engine) { + Ok(host) => host, + Err(error) => { + warn!(%destination, %error, "Denied staged transparent connection"); + emit_staged_transparent_denial( + destination, + &binary_identity, + &error.to_string(), + "transparent_tcp_mapping_denied", + ); + let _ = completion.send(TcpOpenDecision::Denied(TcpOpenDenial::InvalidDestination)); + return None; + } + }; + let mut decision = authorize_supplied_identity( + opa_engine, + EgressIntent::connect(host.clone(), destination.port()), + &binary_identity, + ); + if let NetworkAction::Deny { reason } = &decision.action { + warn!(%destination, %reason, "Denied staged transparent connection"); + emit_staged_transparent_denial( + destination, + &binary_identity, + reason, + "transparent_tcp_policy_denied", + ); + let denial = if binary_identity.is_err() { + TcpOpenDenial::IdentityUnavailable + } else { + TcpOpenDenial::PolicyDenied + }; + let _ = completion.send(TcpOpenDecision::Denied(denial)); + return None; + } + if let Err(denial) = + hydrate_destination_plan(&mut decision, backend_host_gateway, trusted_host_gateway) + { + warn!(%destination, reason = %denial.reason, "Denied staged transparent destination"); + emit_staged_transparent_denial( + destination, + &binary_identity, + &denial.reason, + "transparent_tcp_destination_denied", + ); + let _ = completion.send(TcpOpenDecision::Denied(TcpOpenDenial::InvalidDestination)); + return None; + } + if let Some(mapping) = policy_dns_store.and_then(|store| { + store + .lookup( + destination.ip(), + destination.port(), + opa_engine.current_generation(), + std::time::Instant::now(), + ) + .ok() + }) { + let Ok(plan) = build_pinned_validation_plan(mapping.pinned_addresses()) else { + emit_staged_transparent_denial( + destination, + &binary_identity, + "policy DNS produced an invalid pinned destination", + "transparent_tcp_destination_denied", + ); + let _ = completion.send(TcpOpenDecision::Denied(TcpOpenDenial::InvalidDestination)); + return None; + }; + decision.endpoint.destination = Some(plan); + } + let plan = decision + .endpoint + .destination + .as_ref() + .expect("destination plan hydrated"); + let connector = match validate_destination(DestinationRequest { + host: &host, + port: destination.port(), + sandbox_entrypoint_pid: 0, + plan, + }) + .await + { + Ok(connector) => connector, + Err(denial) => { + warn!(%destination, reason = %denial.reason, "Denied staged transparent destination"); + emit_staged_transparent_denial( + destination, + &binary_identity, + &denial.reason, + "transparent_tcp_destination_denied", + ); + let _ = completion.send(TcpOpenDecision::Denied(TcpOpenDenial::InvalidDestination)); + return None; + } + }; + if completion.send(TcpOpenDecision::RelayReady).is_err() { + return None; + } + Some(( + stream, + Some(binary_identity), + None, + Some(TransparentOpen { + destination, + authorization: Some((decision, connector)), + }), + )) +} + +fn emit_staged_transparent_denial( + destination: SocketAddr, + identity: &Result, + reason: &str, + status_detail: &'static str, +) { + let (binary, ancestors, cmdline) = identity.as_ref().map_or_else( + |_| ("-".to_string(), "-".to_string(), "-".to_string()), + |identity| { + ( + identity.binary_path.display().to_string(), + identity + .ancestors + .iter() + .map(|path| path.display().to_string()) + .collect::>() + .join(" -> "), + identity + .cmdline_paths + .iter() + .map(|path| path.display().to_string()) + .collect::>() + .join(", "), + ) + }, + ); + ocsf_emit!( + NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(ActivityId::Open) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .dst_endpoint(Endpoint::from_ip(destination.ip(), destination.port())) + .actor_process(Process::from_bypass(&binary, "-", &ancestors).with_cmd_line(&cmdline)) + .message(format!("Transparent TCP denied before relay: {reason}")) + .status_detail(status_detail) + .build() + ); +} + +fn transparent_destination_host( + destination: SocketAddr, + policy_dns_store: Option<&Arc>, + opa_engine: &OpaEngine, +) -> Result { + let Some(store) = policy_dns_store else { + return Ok(destination.ip().to_string()); + }; + match store.lookup( + destination.ip(), + destination.port(), + opa_engine.current_generation(), + std::time::Instant::now(), + ) { + Ok(mapping) => Ok(mapping.record.normalized_name.as_str().to_string()), + Err(MappingLookupError::Missing) => Ok(destination.ip().to_string()), + Err(error) => Err(miette::miette!( + "transparent destination mapping is unavailable: {error}" + )), + } +} + impl Drop for ProxyHandle { fn drop(&mut self) { self.join.abort(); @@ -576,7 +862,7 @@ async fn handle_transparent_tcp_connection( let cache = identity_cache.clone(); let pid = entrypoint_pid.clone(); let decision = tokio::task::spawn_blocking(move || { - authorize_egress_intent_procfs(connection, &engine, &cache, &pid, intent) + authorize_egress_intent(connection, &engine, &cache, &pid, intent) }) .await .map_err(|error| miette::miette!("identity resolution task panicked: {error}"))?; @@ -1143,21 +1429,24 @@ fn middleware_uninspectable_gate( Ok(crate::l7::middleware::uninspectable_traffic_gate(&chain)) } -async fn peek_tunnel_protocol(client: &TcpStream) -> Result> { - let mut peek_buf = [0u8; TUNNEL_PROTOCOL_PEEK_BYTES]; +async fn peek_tunnel_protocol(client: &mut C) -> Result> +where + C: tokio::io::AsyncBufRead + Unpin, +{ let deadline = tokio::time::Instant::now() + TUNNEL_PROTOCOL_PEEK_TIMEOUT; loop { - let n = client.peek(&mut peek_buf).await.into_diagnostic()?; - if n == 0 { + let available = client.fill_buf().await.into_diagnostic()?; + if available.is_empty() { return Ok(None); } - let peek = &peek_buf[..n]; + let n = available.len().min(TUNNEL_PROTOCOL_PEEK_BYTES); + let peek = &available[..n]; let protocol = classify_tunnel_protocol(peek); if protocol != TunnelProtocol::Unsupported || !could_be_supported_tunnel_protocol_prefix(peek) - || n == peek_buf.len() + || n == TUNNEL_PROTOCOL_PEEK_BYTES || tokio::time::Instant::now() >= deadline { return Ok(Some(protocol)); @@ -1535,8 +1824,8 @@ fn build_forward_destination_deny_ocsf_event( } #[allow(clippy::too_many_arguments)] -async fn deny_connect_destination( - client: &mut TcpStream, +async fn deny_connect_destination( + client: &mut C, denial: &DestinationDenial, peer_addr: SocketAddr, host: &str, @@ -1548,7 +1837,10 @@ async fn deny_connect_destination( decision: &EgressDecision, denial_tx: &Option>, activity_tx: &Option, -) -> Result<()> { +) -> Result<()> +where + C: TokioAsyncWrite + Unpin, +{ let detail = destination_denial_detail(denial.kind); ocsf_emit!(build_connect_destination_deny_ocsf_event( denial, peer_addr, host, port, binary, pid, ancestors, cmdline, @@ -1581,8 +1873,8 @@ async fn deny_connect_destination( } #[allow(clippy::too_many_arguments)] -async fn deny_forward_destination( - client: &mut TcpStream, +async fn deny_forward_destination( + client: &mut C, denial: &DestinationDenial, peer_addr: SocketAddr, method: &str, @@ -1597,7 +1889,10 @@ async fn deny_forward_destination( decision: &EgressDecision, denial_tx: Option<&mpsc::UnboundedSender>, activity_tx: Option<&ActivitySender>, -) -> Result<()> { +) -> Result<()> +where + C: TokioAsyncWrite + Unpin, +{ let detail = destination_denial_detail(denial.kind); ocsf_emit!(build_forward_destination_deny_ocsf_event( denial, peer_addr, method, host, port, path, binary, pid, ancestors, cmdline, policy, @@ -1632,14 +1927,17 @@ async fn deny_forward_destination( // Many distinct, non-related context parameters are required for a CONNECT // dispatch; bundling them into a struct would just shift the noise into call // sites. +#[cfg(test)] #[allow(clippy::too_many_arguments)] async fn handle_tcp_connection( - mut client: TcpStream, + client: TcpStream, opa_engine: Arc, - identity_mode: Arc, + identity_cache: Arc, + entrypoint_pid: Arc, tls_state: Option>, policy_local_ctx: Option>, agent_proposals: openshell_core::proposals::AgentProposals, + backend_host_gateway: Arc>, trusted_host_gateway: Arc>, upstream_proxy: Arc>, provider_credentials: Option, @@ -1654,6 +1952,116 @@ async fn handle_tcp_connection( denial_tx: Option>, activity_tx: Option, ) -> Result<()> { + let socket_addrs = client.peer_addr().ok().zip(client.local_addr().ok()); + let stream: BoundaryDuplexStream = Box::new(client); + Box::pin(handle_mediated_connection( + tokio::io::BufReader::new(stream), + None, + socket_addrs, + None, + None, + opa_engine, + identity_cache, + entrypoint_pid, + tls_state, + policy_local_ctx, + agent_proposals, + backend_host_gateway, + trusted_host_gateway, + upstream_proxy, + provider_credentials, + secret_resolver, + dynamic_credentials, + denial_tx, + activity_tx, + )) + .await +} + +/// Adapt a transparent application stream to the existing CONNECT pipeline. +/// The synthetic CONNECT request is supervisor-owned and its successful 200 +/// response is consumed before bytes are returned to the workload. +fn virtual_connect_stream( + workload: BoundaryDuplexStream, + authority: String, +) -> BoundaryDuplexStream { + let (handler, bridge) = tokio::io::duplex(64 * 1024); + let (mut bridge_read, mut bridge_write) = tokio::io::split(bridge); + let (mut workload_read, mut workload_write) = tokio::io::split(workload); + tokio::spawn(async move { + let request = format!("CONNECT {authority} HTTP/1.1\r\nHost: {authority}\r\n\r\n"); + if bridge_write.write_all(request.as_bytes()).await.is_ok() { + let _ = tokio::io::copy(&mut workload_read, &mut bridge_write).await; + } + let _ = bridge_write.shutdown().await; + }); + tokio::spawn(async move { + let mut header = Vec::with_capacity(256); + let mut byte = [0_u8; 1]; + while header.len() < MAX_HEADER_BYTES { + match bridge_read.read(&mut byte).await { + Ok(0) | Err(_) => return, + Ok(_) => header.push(byte[0]), + } + if header.ends_with(b"\r\n\r\n") { + break; + } + } + if !header.starts_with(b"HTTP/1.1 200 ") && !header.starts_with(b"HTTP/1.0 200 ") { + let _ = workload_write.shutdown().await; + return; + } + let _ = tokio::io::copy(&mut bridge_read, &mut workload_write).await; + let _ = workload_write.shutdown().await; + }); + Box::new(handler) +} + +#[allow(clippy::too_many_arguments)] +async fn handle_mediated_connection( + mut client: ProxyClient, + supplied_identity: Option>, + socket_addrs: Option<(SocketAddr, SocketAddr)>, + transparent_open: Option, + policy_dns_store: Option>, + opa_engine: Arc, + identity_cache: Arc, + entrypoint_pid: Arc, + tls_state: Option>, + policy_local_ctx: Option>, + agent_proposals: openshell_core::proposals::AgentProposals, + backend_host_gateway: Arc>, + trusted_host_gateway: Arc>, + upstream_proxy: Arc>, + provider_credentials: Option, + secret_resolver: Option>, + dynamic_credentials: Option< + Arc< + std::sync::RwLock< + std::collections::HashMap, + >, + >, + >, + denial_tx: Option>, + activity_tx: Option, +) -> Result<()> { + let (mut preauthorized_decision, prevalidated_connector) = if let Some(transparent) = + transparent_open + { + let destination = transparent.destination; + let host = + transparent_destination_host(destination, policy_dns_store.as_ref(), &opa_engine)?; + let (decision, connector) = transparent + .authorization + .map_or((None, None), |(decision, connector)| { + (Some(decision), Some(connector)) + }); + let authority = format!("{host}:{}", destination.port()); + client = tokio::io::BufReader::new(virtual_connect_stream(client.into_inner(), authority)); + (decision, connector) + } else { + (None, None) + }; let mut buf = vec![0u8; MAX_HEADER_BYTES]; let mut used = 0usize; @@ -1706,10 +2114,14 @@ async fn handle_tcp_connection( &buf[..], used, &mut client, + supplied_identity.as_ref(), + socket_addrs, opa_engine, - identity_mode, + identity_cache, + entrypoint_pid, policy_local_ctx, agent_proposals, + backend_host_gateway, trusted_host_gateway, provider_credentials, secret_resolver, @@ -1723,22 +2135,35 @@ async fn handle_tcp_connection( let (raw_host, port) = parse_target(target)?; let host = normalize_host(&raw_host); let (host_lc, raw_host_lc) = (host.to_ascii_lowercase(), raw_host.to_ascii_lowercase()); - - let workload_addr = client.peer_addr().into_diagnostic()?; - let proxy_addr = client.local_addr().into_diagnostic()?; - let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); + let workload_addr = socket_addrs.map_or_else( + || SocketAddr::from(([0, 0, 0, 0], 0)), + |(workload, _)| workload, + ); // Evaluate OPA policy with process-identity binding. // Wrapped in spawn_blocking because identity resolution does heavy sync I/O: // /proc scanning + SHA256 hashing of binaries (e.g. node at 124MB). - let opa_clone = opa_engine.clone(); - let identity_clone = identity_mode.clone(); let intent = EgressIntent::connect(host_lc.clone(), port); - let mut decision = tokio::task::spawn_blocking(move || { - authorize_egress_intent(connection, &opa_clone, &identity_clone, intent) - }) - .await - .map_err(|e| miette::miette!("identity resolution task panicked: {e}"))?; + let mut decision = if let Some(decision) = preauthorized_decision.take() { + decision + } else if let Some(identity) = supplied_identity.as_ref() { + authorize_supplied_identity(&opa_engine, intent, identity) + } else if !opa_engine.binary_identity_required() { + evaluate_endpoint_only_opa(&opa_engine, intent) + } else { + let (workload_addr, proxy_addr) = socket_addrs.ok_or_else(|| { + miette::miette!("legacy proxy connection is missing socket addresses") + })?; + let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); + let opa_clone = opa_engine.clone(); + let cache_clone = identity_cache.clone(); + let pid_clone = entrypoint_pid.clone(); + tokio::task::spawn_blocking(move || { + authorize_egress_intent(connection, &opa_clone, &cache_clone, &pid_clone, intent) + }) + .await + .map_err(|e| miette::miette!("identity resolution task panicked: {e}"))? + }; debug!( transport = ?decision.intent.transport, @@ -1853,27 +2278,30 @@ async fn handle_tcp_connection( let effective_tls_skip = decision.endpoint.tls_mode == crate::l7::TlsMode::Skip; let credential_guard = query_endpoint_credential_guard(&opa_engine, &decision, &host_lc, port)?; - let sandbox_entrypoint_pid = identity_mode.entrypoint_pid(); + let sandbox_entrypoint_pid = entrypoint_pid.load(Ordering::Acquire); - match hydrate_destination_plan(&mut decision, *trusted_host_gateway) { - Ok(()) => {} - Err(denial) => { - deny_connect_destination( - &mut client, - &denial, - workload_addr, - &host_lc, - port, - &binary_str, - &pid_str, - &ancestors_str, - &cmdline_str, - &decision, - &denial_tx, - &activity_tx, - ) - .await?; - return Ok(()); + if prevalidated_connector.is_none() { + match hydrate_destination_plan(&mut decision, *backend_host_gateway, *trusted_host_gateway) + { + Ok(()) => {} + Err(denial) => { + deny_connect_destination( + &mut client, + &denial, + workload_addr, + &host_lc, + port, + &binary_str, + &pid_str, + &ancestors_str, + &cmdline_str, + &decision, + &denial_tx, + &activity_tx, + ) + .await?; + return Ok(()); + } } } let destination_plan = decision @@ -1884,32 +2312,36 @@ async fn handle_tcp_connection( // Defense-in-depth: resolve DNS and reject connections to internal IPs. let dns_connect_start = std::time::Instant::now(); - let connector = match validate_destination(DestinationRequest { - host: &raw_host, - port, - sandbox_entrypoint_pid, - plan: destination_plan, - }) - .await - { - Ok(connector) => connector, - Err(denial) => { - deny_connect_destination( - &mut client, - &denial, - workload_addr, - &host_lc, - port, - &binary_str, - &pid_str, - &ancestors_str, - &cmdline_str, - &decision, - &denial_tx, - &activity_tx, - ) - .await?; - return Ok(()); + let connector = if let Some(connector) = prevalidated_connector { + connector + } else { + match validate_destination(DestinationRequest { + host: &raw_host, + port, + sandbox_entrypoint_pid, + plan: destination_plan, + }) + .await + { + Ok(connector) => connector, + Err(denial) => { + deny_connect_destination( + &mut client, + &denial, + workload_addr, + &host_lc, + port, + &binary_str, + &pid_str, + &ancestors_str, + &cmdline_str, + &decision, + &denial_tx, + &activity_tx, + ) + .await?; + return Ok(()); + } } }; @@ -2125,7 +2557,7 @@ async fn handle_tcp_connection( // Auto-detect the tunnel payload. L7-configured endpoints must only // enter relays that can enforce their configured protocol; unsupported // bytes fail closed below instead of falling through to raw relay. - let Some(tunnel_protocol) = peek_tunnel_protocol(&client).await? else { + let Some(tunnel_protocol) = peek_tunnel_protocol(&mut client).await? else { return Ok(()); }; @@ -2527,7 +2959,7 @@ fn resolve_process_identity( /// Evaluate OPA policy for a TCP connection with identity binding via /proc/net/tcp. #[cfg(target_os = "linux")] -fn authorize_egress_intent_procfs( +fn authorize_egress_intent( connection: crate::procfs::WorkloadProxyTcpConnection, engine: &OpaEngine, identity_cache: &BinaryIdentityCache, @@ -2557,18 +2989,6 @@ fn authorize_egress_intent_procfs( } }; - if !crate::opa::network_binary_identity_required() { - let result = evaluate_endpoint_only_opa(engine, intent); - debug!( - "authorize_egress_intent endpoint-only: host={} port={} transport={:?} action={:?}", - result.intent.destination.host, - result.intent.destination.port, - result.intent.transport, - result.action - ); - return result; - } - let entrypoint_pid = entrypoint_pid.load(Ordering::Acquire); let Some(proc_net_anchor_pid) = proc_net_anchor_pid(entrypoint_pid) else { return deny( @@ -2646,16 +3066,7 @@ fn authorize_egress_intent_procfs( #[cfg(target_os = "linux")] fn proc_net_anchor_pid(entrypoint_pid: u32) -> Option { - if entrypoint_pid != 0 { - return Some(entrypoint_pid); - } - sidecar_topology_enabled().then(std::process::id) -} - -#[cfg(target_os = "linux")] -fn sidecar_topology_enabled() -> bool { - std::env::var(openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY) - .is_ok_and(|value| value == SIDECAR_SUPERVISOR_TOPOLOGY) + (entrypoint_pid != 0).then_some(entrypoint_pid) } fn evaluate_endpoint_only_opa(engine: &OpaEngine, intent: EgressIntent) -> EgressDecision { @@ -2673,98 +3084,127 @@ fn evaluate_endpoint_only_opa(engine: &OpaEngine, intent: EgressIntent) -> Egres intent, action: authorization.action.clone(), policy_generation: authorization.generation, - identity: ProcessIdentityEvidence::Unavailable( - IdentityUnavailableReason::EndpointOnlyMode, - ), + identity: ProcessIdentityEvidence::Unavailable( + IdentityUnavailableReason::EndpointOnlyMode, + ), + endpoint: EndpointDecision::from_authorization(&authorization), + binary: None, + binary_pid: None, + ancestors: vec![], + cmdline_paths: vec![], + }, + Err(e) => EgressDecision { + intent, + action: NetworkAction::Deny { + reason: format!("policy evaluation error: {e}"), + }, + policy_generation: engine.current_generation(), + identity: ProcessIdentityEvidence::Unavailable( + IdentityUnavailableReason::EndpointOnlyMode, + ), + endpoint: EndpointDecision::default(), + binary: None, + binary_pid: None, + ancestors: vec![], + cmdline_paths: vec![], + }, + } +} + +/// Evaluate an egress intent using identity already bound to the accepted +/// connection by an isolation backend. This is the RFC 0012 path; legacy +/// listeners continue to resolve through procfs in `authorize_egress_intent`. +fn authorize_supplied_identity( + engine: &OpaEngine, + intent: EgressIntent, + identity: &Result, +) -> EgressDecision { + let deny = |reason: String, + binary: Option, + ancestors: Vec, + cmdline_paths: Vec| EgressDecision { + intent: intent.clone(), + action: NetworkAction::Deny { reason }, + policy_generation: engine.current_generation(), + identity: ProcessIdentityEvidence::Unavailable(IdentityUnavailableReason::LookupFailed), + endpoint: EndpointDecision::default(), + binary, + binary_pid: None, + ancestors, + cmdline_paths, + }; + + let identity = match identity { + Ok(identity) => identity, + Err(error) => { + return deny( + format!("backend identity resolution failed: {error}"), + None, + vec![], + vec![], + ); + } + }; + let Some(digest) = identity.binary_digest else { + return deny( + "backend identity did not include the required binary digest".to_string(), + Some(identity.binary_path.clone()), + identity.ancestors.clone(), + identity.cmdline_paths.clone(), + ); + }; + let input = crate::opa::NetworkInput { + host: intent.destination.host.clone(), + port: intent.destination.port, + binary_path: identity.binary_path.clone(), + binary_sha256: digest.to_string(), + ancestors: identity.ancestors.clone(), + cmdline_paths: identity.cmdline_paths.clone(), + }; + match engine.authorize_egress(&input) { + Ok(authorization) => EgressDecision { + intent, + action: authorization.action.clone(), + policy_generation: authorization.generation, + identity: ProcessIdentityEvidence::Available, endpoint: EndpointDecision::from_authorization(&authorization), - binary: None, - binary_pid: None, - ancestors: vec![], - cmdline_paths: vec![], - }, - Err(e) => EgressDecision { - intent, - action: NetworkAction::Deny { - reason: format!("policy evaluation error: {e}"), - }, - policy_generation: engine.current_generation(), - identity: ProcessIdentityEvidence::Unavailable( - IdentityUnavailableReason::EndpointOnlyMode, - ), - endpoint: EndpointDecision::default(), - binary: None, + binary: Some(identity.binary_path.clone()), binary_pid: None, - ancestors: vec![], - cmdline_paths: vec![], + ancestors: identity.ancestors.clone(), + cmdline_paths: identity.cmdline_paths.clone(), }, + Err(error) => deny( + format!("policy evaluation error: {error}"), + Some(identity.binary_path.clone()), + identity.ancestors.clone(), + identity.cmdline_paths.clone(), + ), } } +/// Non-Linux stub: OPA identity binding requires /proc. +#[cfg(not(target_os = "linux"))] fn authorize_egress_intent( - connection: crate::procfs::WorkloadProxyTcpConnection, + _connection: crate::procfs::WorkloadProxyTcpConnection, engine: &OpaEngine, - identity_mode: &ProxyIdentityMode, + _identity_cache: &BinaryIdentityCache, + _entrypoint_pid: &AtomicU32, intent: EgressIntent, ) -> EgressDecision { - #[cfg(not(target_os = "linux"))] - let _ = &connection; - - if !crate::opa::network_binary_identity_required() { - return evaluate_endpoint_only_opa(engine, intent); - } - - match identity_mode { - #[cfg(target_os = "linux")] - ProxyIdentityMode::Procfs { - identity_cache, - entrypoint_pid, - } => authorize_egress_intent_procfs( - connection, - engine, - identity_cache, - entrypoint_pid, - intent, + EgressDecision { + intent, + action: NetworkAction::Deny { + reason: "identity binding unavailable on this platform".into(), + }, + policy_generation: engine.current_generation(), + identity: ProcessIdentityEvidence::Unavailable( + IdentityUnavailableReason::UnsupportedPlatform, ), - #[cfg(any(not(target_os = "linux"), test))] - ProxyIdentityMode::Static { - binary_path, - binary_sha256, - } => { - let input = crate::opa::NetworkInput { - host: intent.destination.host.clone(), - port: intent.destination.port, - binary_path: binary_path.clone(), - binary_sha256: binary_sha256.clone(), - ancestors: Vec::new(), - cmdline_paths: Vec::new(), - }; - match engine.authorize_egress(&input) { - Ok(authorization) => EgressDecision { - intent, - action: authorization.action.clone(), - policy_generation: authorization.generation, - identity: ProcessIdentityEvidence::Available, - endpoint: EndpointDecision::from_authorization(&authorization), - binary: Some(binary_path.clone()), - binary_pid: None, - ancestors: Vec::new(), - cmdline_paths: Vec::new(), - }, - Err(error) => EgressDecision { - intent, - action: NetworkAction::Deny { - reason: format!("policy evaluation error: {error}"), - }, - policy_generation: engine.current_generation(), - identity: ProcessIdentityEvidence::Available, - endpoint: EndpointDecision::default(), - binary: Some(binary_path.clone()), - binary_pid: None, - ancestors: Vec::new(), - cmdline_paths: Vec::new(), - }, - } - } + endpoint: EndpointDecision::default(), + binary: None, + binary_pid: None, + ancestors: vec![], + cmdline_paths: vec![], } } @@ -2783,13 +3223,16 @@ fn emit_l7_tunnel_close_after_policy_change(host: &str, port: u16, error: miette ocsf_emit!(event); } -async fn reject_stale_connect_policy( - client: &mut TcpStream, +async fn reject_stale_connect_policy( + client: &mut C, host: &str, port: u16, activity_tx: Option<&ActivitySender>, error: miette::Report, -) -> Result<()> { +) -> Result<()> +where + C: TokioAsyncWrite + Unpin, +{ warn!( host, port, @@ -2828,6 +3271,7 @@ fn hydrate_tls_mode(decision: &mut EgressDecision) { fn hydrate_destination_plan( decision: &mut EgressDecision, + backend_host_gateway: Option, trusted_host_gateway: Option, ) -> std::result::Result<(), DestinationDenial> { let host = decision.intent.destination.host.clone(); @@ -2836,6 +3280,7 @@ fn hydrate_destination_plan( let plan = build_validation_plan( &host, &host.to_ascii_lowercase(), + backend_host_gateway, trusted_host_gateway, &raw_allowed_ips, exact_declared_host, @@ -3017,27 +3462,16 @@ fn is_cloud_metadata_ip(ip: IpAddr) -> bool { } } -/// Read the proxy's own platform hosts file at startup and return the IP mapped to +/// Read the proxy's own `/etc/hosts` at startup and return the IP mapped to /// `host.openshell.internal`, if present and safe. /// /// This is called once before user code runs, so the returned value is immune -/// to later hosts-file tampering by sandbox workloads. Returns `None` if the -/// hosts file cannot be read, no entry exists, the entry cannot be parsed, or -/// the mapped IP is a cloud metadata address. -#[cfg(any(target_os = "linux", target_os = "windows", test))] +/// to later `/etc/hosts` tampering by sandbox workloads. Returns `None` if no +/// entry exists, the entry cannot be parsed, or the mapped IP is a cloud +/// metadata address. +#[cfg(any(target_os = "linux", test))] pub(crate) fn detect_trusted_host_gateway() -> Option { - let hosts_path = platform_hosts_path(); - let contents = match std::fs::read_to_string(&hosts_path) { - Ok(contents) => contents, - Err(error) => { - warn!( - path = %hosts_path.display(), - %error, - "failed to read platform hosts file; trusted-gateway SSRF exemption disabled" - ); - return None; - } - }; + let contents = std::fs::read_to_string("/etc/hosts").ok()?; let ips = parse_hosts_file_for_host(&contents, "host.openshell.internal"); // Multiple distinct IPs for the alias is unexpected — compute drivers @@ -3049,7 +3483,7 @@ pub(crate) fn detect_trusted_host_gateway() -> Option { if ips.len() > 1 { warn!( ips = ?ips, - "host.openshell.internal has {} distinct IPs in the platform hosts file; \ + "host.openshell.internal has {} distinct IPs in /etc/hosts; \ expected exactly one. Using first entry. \ Connections resolving to any other IP will be rejected.", ips.len() @@ -3083,38 +3517,11 @@ pub(crate) fn detect_trusted_host_gateway() -> Option { Some(ip) } -#[cfg(not(any(target_os = "linux", target_os = "windows", test)))] +#[cfg(not(any(target_os = "linux", test)))] pub(crate) fn detect_trusted_host_gateway() -> Option { None } -#[cfg(target_os = "linux")] -fn platform_hosts_path() -> PathBuf { - PathBuf::from("/etc/hosts") -} - -#[cfg(target_os = "windows")] -fn platform_hosts_path() -> PathBuf { - windows_hosts_path_from_system_root(std::env::var("SystemRoot").ok().as_deref()) -} - -#[cfg(target_os = "windows")] -fn windows_hosts_path_from_system_root(system_root: Option<&str>) -> PathBuf { - let root = system_root - .filter(|value| !value.trim().is_empty()) - .unwrap_or(r"C:\Windows"); - PathBuf::from(root) - .join("System32") - .join("drivers") - .join("etc") - .join("hosts") -} - -#[cfg(all(test, not(any(target_os = "linux", target_os = "windows"))))] -fn platform_hosts_path() -> PathBuf { - PathBuf::from("/etc/hosts") -} - /// Resolve `host:port` and validate that every resolved address matches the /// trusted host gateway IP. /// @@ -3181,7 +3588,7 @@ fn resolve_ip_literal(host: &str, port: u16) -> Option> { .map(|ip| vec![SocketAddr::new(ip, port)]) } -#[cfg(any(target_os = "linux", target_os = "windows", test))] +#[cfg(any(target_os = "linux", test))] fn parse_hosts_file_for_host(contents: &str, host: &str) -> Vec { let lookup_host = normalize_host_lookup_key(host); let mut addrs = Vec::new(); @@ -4204,11 +4611,15 @@ async fn handle_forward_proxy( target_uri: &str, buf: &[u8], used: usize, - client: &mut TcpStream, + client: &mut ProxyClient, + supplied_identity: Option<&Result>, + socket_addrs: Option<(SocketAddr, SocketAddr)>, opa_engine: Arc, - identity_mode: Arc, + identity_cache: Arc, + entrypoint_pid: Arc, policy_local_ctx: Option>, agent_proposals: openshell_core::proposals::AgentProposals, + backend_host_gateway: Arc>, trusted_host_gateway: Arc>, provider_credentials: Option, secret_resolver: Option>, @@ -4307,18 +4718,29 @@ async fn handle_forward_proxy( canonicalize_forward_host_header(&buf[..used], &canonical_authority)?; // 2. Evaluate OPA policy (same identity binding as CONNECT) - let workload_addr = client.peer_addr().into_diagnostic()?; - let proxy_addr = client.local_addr().into_diagnostic()?; - let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); - - let opa_clone = opa_engine.clone(); - let identity_clone = identity_mode.clone(); + let workload_addr = socket_addrs.map_or_else( + || SocketAddr::from(([0, 0, 0, 0], 0)), + |(workload, _)| workload, + ); let intent = EgressIntent::forward_http(host_lc.clone(), port); - let mut decision = tokio::task::spawn_blocking(move || { - authorize_egress_intent(connection, &opa_clone, &identity_clone, intent) - }) - .await - .map_err(|e| miette::miette!("identity resolution task panicked: {e}"))?; + let mut decision = if let Some(identity) = supplied_identity { + authorize_supplied_identity(&opa_engine, intent, identity) + } else if !opa_engine.binary_identity_required() { + evaluate_endpoint_only_opa(&opa_engine, intent) + } else { + let (workload_addr, proxy_addr) = socket_addrs.ok_or_else(|| { + miette::miette!("legacy proxy connection is missing socket addresses") + })?; + let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); + let opa_clone = opa_engine.clone(); + let cache_clone = identity_cache.clone(); + let pid_clone = entrypoint_pid.clone(); + tokio::task::spawn_blocking(move || { + authorize_egress_intent(connection, &opa_clone, &cache_clone, &pid_clone, intent) + }) + .await + .map_err(|e| miette::miette!("identity resolution task panicked: {e}"))? + }; debug!( transport = ?decision.intent.transport, @@ -4406,7 +4828,7 @@ async fn handle_forward_proxy( action = ?decision.action, "Forward proxy L4 policy decision" ); - let sandbox_entrypoint_pid = identity_mode.entrypoint_pid(); + let sandbox_entrypoint_pid = entrypoint_pid.load(Ordering::Acquire); let forward_generation_guard = match relay::pin_policy_generation( &opa_engine, decision.policy_generation, @@ -4979,7 +5401,7 @@ async fn handle_forward_proxy( // - Otherwise: reject internal IPs, allow public IPs through. // When the policy host is already a literal IP address, treat it as // implicitly allowed — the user explicitly declared the destination. - match hydrate_destination_plan(&mut decision, *trusted_host_gateway) { + match hydrate_destination_plan(&mut decision, *backend_host_gateway, *trusted_host_gateway) { Ok(()) => {} Err(denial) => { deny_forward_destination( @@ -5282,6 +5704,17 @@ async fn handle_forward_proxy( ), ) .await?; + client.shutdown().await.into_diagnostic()?; + let mut discard = [0_u8; 1024]; + let _ = tokio::time::timeout(std::time::Duration::from_secs(1), async { + loop { + match client.read(&mut discard).await { + Ok(0) | Err(_) => break, + Ok(_) => {} + } + } + }) + .await; } else { respond( client, @@ -5601,8 +6034,9 @@ fn normalize_host(raw_host: &str) -> &str { raw_host.strip_suffix('.').unwrap_or(raw_host) } -async fn respond(client: &mut TcpStream, bytes: &[u8]) -> Result<()> { +async fn respond(client: &mut (impl TokioAsyncWrite + Unpin), bytes: &[u8]) -> Result<()> { client.write_all(bytes).await.into_diagnostic()?; + client.flush().await.into_diagnostic()?; Ok(()) } @@ -5741,11 +6175,14 @@ const TLS_TERMINATION_UNAVAILABLE_DETAIL: &str = "TLS termination unavailable (C /// HTTP status (the flaw this replaces). Returns `true` when the connection was /// refused (the caller must stop) and `false` when the caller should proceed to /// establish the tunnel. -async fn refuse_connect_when_tls_unavailable( - client: &mut TcpStream, +async fn refuse_connect_when_tls_unavailable( + client: &mut C, tls_state_present: bool, effective_tls_skip: bool, -) -> Result { +) -> Result +where + C: TokioAsyncWrite + Unpin, +{ if tls_state_present || effective_tls_skip { return Ok(false); } @@ -5789,15 +6226,204 @@ fn is_benign_relay_error(err: &miette::Report) -> bool { mod tests { use super::*; use openshell_core::proposals::AgentProposals; - use openshell_core::proto::{ - NetworkBinary, NetworkEndpoint, NetworkPolicyRule, SandboxPolicy as ProtoSandboxPolicy, - }; use std::collections::HashMap as TestHashMap; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; use std::sync::Arc; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; + #[test] + fn supplied_identity_preserves_authorized_endpoint_metadata() { + let engine = OpaEngine::from_strings( + include_str!("../data/sandbox-policy.rego"), + r#" +network_policies: + inspected: + name: inspected + endpoints: + - host: api.example.com + port: 443 + protocol: rest + enforcement: enforce + request_body_credential_rewrite: true + allowed_ips: ["192.0.2.0/24"] + rules: + - allow: { method: GET, path: /allowed } + binaries: + - path: /usr/bin/python3 +filesystem_policy: + include_workdir: true + read_only: [] + read_write: [] +landlock: + compatibility: best_effort +process: + run_as_user: sandbox + run_as_group: sandbox +"#, + ) + .expect("load policy"); + let identity = Ok(ContractBinaryIdentity { + binary_path: PathBuf::from("/usr/bin/python3"), + binary_digest: Some("00".repeat(32).parse().expect("digest")), + ancestors: Vec::new(), + cmdline_paths: Vec::new(), + }); + + let mut decision = authorize_supplied_identity( + &engine, + EgressIntent::connect("api.example.com".to_string(), 443), + &identity, + ); + + assert_eq!(query_allowed_ips(&decision), ["192.0.2.0/24"]); + hydrate_l7_route(&mut decision); + let route = decision + .endpoint + .l7_route + .expect("supplied identity must retain L7 metadata"); + assert_eq!(route.configs.len(), 1); + assert!(route.configs[0].config.request_body_credential_rewrite); + } + + #[tokio::test] + async fn staged_transparent_open_waits_for_l4_policy() { + let engine = OpaEngine::from_strings( + include_str!("../data/sandbox-policy.rego"), + r#" +network_policies: + allowed: + name: allowed + endpoints: + - host: 203.0.113.7 + port: 443 + - host: 169.254.169.254 + port: 80 + binaries: + - path: /usr/bin/curl +filesystem_policy: + include_workdir: true + read_only: [] + read_write: [] +landlock: + compatibility: best_effort +process: + run_as_user: sandbox + run_as_group: sandbox +"#, + ) + .unwrap(); + let identity = || { + Ok(ContractBinaryIdentity { + binary_path: PathBuf::from("/usr/bin/curl"), + binary_digest: Some("00".repeat(32).parse().unwrap()), + ancestors: Vec::new(), + cmdline_paths: Vec::new(), + }) + }; + let pending = |destination: &str| { + let (stream, _peer) = tokio::io::duplex(64); + let (decision, completion) = tokio::sync::oneshot::channel(); + ( + PendingTcpOpen { + stream: Box::new(stream), + binary_identity: identity(), + destination: destination.parse().unwrap(), + socket: openshell_isolation_interface::contract::NetworkSocketMetadata { + socket_cookie: 7, + nonblocking: false, + process_generation: 1, + }, + policy_generation: engine.current_generation(), + timing: MediationTiming::default(), + decision, + }, + completion, + ) + }; + + let (allowed, allowed_result) = pending("203.0.113.7:443"); + assert!( + preauthorize_transparent_open(allowed, None, &engine, None, None) + .await + .is_some() + ); + assert_eq!(allowed_result.await.unwrap(), TcpOpenDecision::RelayReady); + + let (unsafe_destination, unsafe_result) = pending("169.254.169.254:80"); + assert!( + preauthorize_transparent_open(unsafe_destination, None, &engine, None, None) + .await + .is_none() + ); + assert_eq!( + unsafe_result.await.unwrap(), + TcpOpenDecision::Denied(TcpOpenDenial::InvalidDestination) + ); + + let (denied, denied_result) = pending("203.0.113.8:443"); + assert!( + preauthorize_transparent_open(denied, None, &engine, None, None) + .await + .is_none() + ); + assert_eq!( + denied_result.await.unwrap(), + TcpOpenDecision::Denied(TcpOpenDenial::PolicyDenied) + ); + } + + struct FailedMediationSource; + + #[tokio::test] + async fn virtual_connect_is_portless_and_hides_the_synthetic_handshake() { + let (workload, mut workload_peer) = tokio::io::duplex(1024); + let mut handler = virtual_connect_stream(Box::new(workload), "api.example.com:443".into()); + + workload_peer.write_all(b"client-tls").await.unwrap(); + let mut request = vec![0_u8; 128]; + let length = handler.read(&mut request).await.unwrap(); + let request = &request[..length]; + assert!(request.starts_with(b"CONNECT api.example.com:443 HTTP/1.1\r\n")); + + handler + .write_all(b"HTTP/1.1 200 Connection Established\r\n\r\nserver-tls") + .await + .unwrap(); + let mut response = [0_u8; 10]; + workload_peer.read_exact(&mut response).await.unwrap(); + assert_eq!(&response, b"server-tls"); + } + + #[async_trait::async_trait] + impl NetworkMediationSource for FailedMediationSource { + async fn accept_tcp( + &self, + ) -> std::result::Result< + PendingTcpOpen, + openshell_isolation_interface::contract::BackendError, + > { + Err( + openshell_isolation_interface::contract::BackendError::Unavailable( + "test source unavailable".to_string(), + ), + ) + } + + async fn accept_dns( + &self, + ) -> std::result::Result< + openshell_isolation_interface::contract::PendingDnsQuery, + openshell_isolation_interface::contract::BackendError, + > { + Err( + openshell_isolation_interface::contract::BackendError::Unavailable( + "test source unavailable".to_string(), + ), + ) + } + } + struct DenyWebSocketPreflight; #[tonic::async_trait] @@ -5965,12 +6591,14 @@ network_policies: {} Box::pin(handle_tcp_connection( server, engine, - Arc::new(ProxyIdentityMode::static_binary(std::env::current_exe().unwrap()).unwrap()), + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(std::process::id())), None, None, AgentProposals::default(), Arc::new(None), Arc::new(None), + Arc::new(None), None, None, None, @@ -5982,6 +6610,48 @@ network_policies: {} client.await.unwrap() } + #[tokio::test] + async fn terminal_mediation_source_failure_stops_proxy() { + let policy = include_str!("../data/sandbox-policy.rego"); + let engine = Arc::new( + OpaEngine::from_strings_with_binary_identity_required( + policy, + "network_policies: {}", + true, + ) + .expect("engine"), + ); + let (_ready_tx, ready_rx) = tokio::sync::watch::channel(true); + let mut handle = ProxyHandle::start_with_bind_addr( + &ProxyPolicy { http_addr: None }, + Some(([127, 0, 0, 1], 3128).into()), + engine, + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(1)), + None, + None, + None, + None, + None, + ready_rx, + &upstream_proxy::UpstreamProxyArgs::default(), + None, + Some(Arc::new(FailedMediationSource)), + None, + ) + .await + .expect("proxy starts before source accept"); + let exited = handle + .take_exit_receiver() + .expect("proxy exposes its exit receiver"); + + tokio::time::timeout(std::time::Duration::from_secs(1), exited) + .await + .expect("source failure must stop the proxy") + .expect_err("proxy task drops the exit sender"); + assert!(handle.join.is_finished()); + } + #[tokio::test] async fn malformed_forward_headers_are_rejected_before_route_or_middleware_dispatch() { for host in ["api.example.com", "unmatched.example.com"] { @@ -6105,36 +6775,37 @@ network_policies: socket.read_to_end(&mut response).await.unwrap(); response }); - let (mut proxy_connection, _) = proxy_listener.accept().await.unwrap(); - #[cfg(target_os = "linux")] - let identity_mode = Arc::new(ProxyIdentityMode::procfs( - Arc::new(BinaryIdentityCache::new()), - Arc::new(AtomicU32::new(std::process::id())), - )); - #[cfg(not(target_os = "linux"))] - let identity_mode = Arc::new( - ProxyIdentityMode::static_binary(executable.clone()).expect("hash test executable"), - ); + let (proxy_connection, _) = proxy_listener.accept().await.unwrap(); + let socket_addrs = proxy_connection + .peer_addr() + .ok() + .zip(proxy_connection.local_addr().ok()); + let mut proxy_connection: ProxyClient = + tokio::io::BufReader::new(Box::new(proxy_connection)); tokio::time::timeout( std::time::Duration::from_secs(30), - handle_forward_proxy( + Box::pin(handle_forward_proxy( "POST", &target, request.as_bytes(), request.len(), &mut proxy_connection, + None, + socket_addrs, engine, - identity_mode, + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(std::process::id())), None, AgentProposals::default(), Arc::new(None), + Arc::new(None), None, None, None, None, None, - ), + )), ) .await .expect("MCP forwarding should complete") @@ -6227,29 +6898,37 @@ network_policies: .expect("read proxy response"); response }); - let (mut proxy_connection, _) = proxy_listener.accept().await.unwrap(); + let (proxy_connection, _) = proxy_listener.accept().await.unwrap(); + let socket_addrs = proxy_connection + .peer_addr() + .ok() + .zip(proxy_connection.local_addr().ok()); + let stream: BoundaryDuplexStream = Box::new(proxy_connection); + let mut proxy_connection = tokio::io::BufReader::new(stream); tokio::time::timeout( std::time::Duration::from_secs(30), - handle_forward_proxy( + Box::pin(handle_forward_proxy( "GET", &target, request.as_bytes(), request.len(), &mut proxy_connection, + None, + socket_addrs, engine, - Arc::new( - ProxyIdentityMode::static_binary(std::env::current_exe().unwrap()).unwrap(), - ), + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(std::process::id())), None, AgentProposals::default(), Arc::new(None), + Arc::new(None), None, None, None, None, None, - ), + )), ) .await .expect("denied preflight must complete without an upstream response") @@ -6362,7 +7041,13 @@ network_policies: .await .unwrap(); }); - let (mut proxy_connection, _) = proxy_listener.accept().await.unwrap(); + let (proxy_connection, _) = proxy_listener.accept().await.unwrap(); + let socket_addrs = proxy_connection + .peer_addr() + .ok() + .zip(proxy_connection.local_addr().ok()); + let stream: BoundaryDuplexStream = Box::new(proxy_connection); + let mut proxy_connection = tokio::io::BufReader::new(stream); let handler = tokio::spawn(async move { handle_forward_proxy( @@ -6371,13 +7056,15 @@ network_policies: request.as_bytes(), request.len(), &mut proxy_connection, + None, + socket_addrs, engine, - Arc::new( - ProxyIdentityMode::static_binary(std::env::current_exe().unwrap()).unwrap(), - ), + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(std::process::id())), None, AgentProposals::default(), Arc::new(None), + Arc::new(None), None, None, None, @@ -6942,97 +7629,6 @@ network_policies: ); } - #[test] - fn static_binary_hashes_configured_file() { - let tmp = tempfile::NamedTempFile::new().unwrap(); - std::fs::write(tmp.path(), b"abc").unwrap(); - - match ProxyIdentityMode::static_binary(tmp.path()).unwrap() { - ProxyIdentityMode::Static { - binary_path, - binary_sha256, - } => { - assert_eq!(binary_path, tmp.path()); - assert_eq!( - binary_sha256, - "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad" - ); - } - #[cfg(target_os = "linux")] - ProxyIdentityMode::Procfs { .. } => panic!("expected static identity mode"), - } - } - - #[test] - fn static_identity_evaluate_opa_tcp_allows_and_denies_with_proto_policy() { - let tmp = tempfile::NamedTempFile::new().unwrap(); - std::fs::write(tmp.path(), b"static-agent").unwrap(); - let identity = ProxyIdentityMode::static_binary(tmp.path()).unwrap(); - let binary_path = tmp.path().to_string_lossy().into_owned(); - let policy_name = "static_agent"; - let engine = OpaEngine::from_proto(&ProtoSandboxPolicy { - version: 1, - network_policies: std::collections::HashMap::from([( - policy_name.to_string(), - NetworkPolicyRule { - name: policy_name.to_string(), - endpoints: vec![NetworkEndpoint { - host: "api.example.test".to_string(), - port: 443, - ..Default::default() - }], - binaries: vec![NetworkBinary { path: binary_path }], - }, - )]), - ..Default::default() - }) - .unwrap(); - let peer_addr: SocketAddr = ([127, 0, 0, 1], 49152).into(); - let connection = crate::procfs::WorkloadProxyTcpConnection::new( - peer_addr, - ([127, 0, 0, 1], 18080).into(), - ); - - let allowed = authorize_egress_intent( - connection, - &engine, - &identity, - EgressIntent::connect("api.example.test".to_string(), 443), - ); - match allowed.action { - NetworkAction::Allow { matched_policy } => { - assert_eq!(matched_policy.as_deref(), Some(policy_name)); - } - NetworkAction::Deny { reason } => panic!("expected allow, got deny: {reason}"), - } - assert_eq!(allowed.binary.as_deref(), Some(tmp.path())); - assert_eq!(allowed.binary_pid, None); - assert!(allowed.ancestors.is_empty()); - assert!(allowed.cmdline_paths.is_empty()); - - let denied = authorize_egress_intent( - connection, - &engine, - &identity, - EgressIntent::connect("blocked.example.test".to_string(), 443), - ); - match denied.action { - NetworkAction::Allow { matched_policy } => { - panic!("expected deny, got allow from policy {matched_policy:?}"); - } - NetworkAction::Deny { reason } => { - assert!( - reason.contains("endpoint blocked.example.test:443 is not allowed"), - "unexpected deny reason: {reason}" - ); - } - } - assert_eq!(denied.binary.as_deref(), Some(tmp.path())); - assert_eq!(denied.binary_pid, None); - assert!(denied.ancestors.is_empty()); - assert!(denied.cmdline_paths.is_empty()); - } - #[test] fn tunnel_protocol_prefix_detection_waits_for_partial_supported_prefixes() { assert!(could_be_supported_tunnel_protocol_prefix(&[0x16])); @@ -7048,14 +7644,14 @@ network_policies: let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); let mut client = TcpStream::connect(addr).await.unwrap(); - let (server, _) = listener.accept().await.unwrap(); + let (mut server, _) = listener.accept().await.unwrap(); client .write_all(crate::l7::rest::HTTP2_PRIOR_KNOWLEDGE_PREFACE) .await .unwrap(); - let protocol = peek_tunnel_protocol(&server) + let protocol = peek_tunnel_protocol(&mut tokio::io::BufReader::new(&mut server)) .await .expect("peek should succeed") .expect("client sent bytes"); @@ -11062,7 +11658,8 @@ network_policies: }); let (server, _peer) = listener.accept().await.unwrap(); - let identity_mode = Arc::new(ProxyIdentityMode::static_binary(exe).unwrap()); + let entrypoint_pid = Arc::new(AtomicU32::new(std::process::id())); + let cache = Arc::new(BinaryIdentityCache::new()); let (denial_tx, mut denial_rx) = mpsc::unbounded_channel(); let completed = tokio::time::timeout( @@ -11070,10 +11667,12 @@ network_policies: Box::pin(handle_tcp_connection( server, engine, - identity_mode, + cache, + entrypoint_pid, None, // tls_state — ephemeral CA unavailable None, // policy_local_ctx AgentProposals::default(), // agent_proposals + Arc::new(None), // backend_host_gateway Arc::new(None), // trusted_host_gateway Arc::new(None), // upstream_proxy None, // provider_credentials @@ -11106,15 +11705,15 @@ network_policies: const POLICY_REGO: &str = include_str!("../data/sandbox-policy.rego"); let exe = std::env::current_exe().expect("current_exe"); - let exe_yaml = serde_json::to_string(&exe.to_string_lossy()).expect("serialize exe path"); let data = format!( r#"network_policies: test_allow: name: test_allow endpoints: {endpoint_yaml} binaries: - - {{ path: {exe_yaml} }} + - {{ path: "{exe}" }} "#, + exe = exe.display(), ); let engine = Arc::new(OpaEngine::from_strings(POLICY_REGO, &data).expect("load policy")); @@ -11135,12 +11734,14 @@ network_policies: Box::pin(handle_tcp_connection( server, engine, - Arc::new(ProxyIdentityMode::static_binary(exe).unwrap()), + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(std::process::id())), None, None, AgentProposals::default(), Arc::new(None), Arc::new(None), + Arc::new(None), None, None, None, diff --git a/crates/openshell-supervisor-network/src/proxy/destination.rs b/crates/openshell-supervisor-network/src/proxy/destination.rs index 1ce514133a..16e0e03999 100644 --- a/crates/openshell-supervisor-network/src/proxy/destination.rs +++ b/crates/openshell-supervisor-network/src/proxy/destination.rs @@ -29,6 +29,10 @@ pub(crate) enum AddressAuthorization { TrustedGatewayAlias { expected_ip: IpAddr, }, + /// A backend-provided host-side dial target. The backend is the trusted + /// authority for this mapping, so the supervisor does not consult its own + /// resolver before dialing it. + BackendPinnedGateway(IpAddr), /// Addresses already resolved and authorized by policy DNS. This mode must /// never resolve `DestinationRequest::host` again before constructing the /// unopened connector. @@ -79,11 +83,16 @@ impl DestinationDenial { pub(crate) fn build_validation_plan( host: &str, normalized_host: &str, + backend_host_gateway: Option, trusted_host_gateway: Option, raw_allowed_ips: &[String], exact_declared_endpoint_host: bool, ) -> Result { let address_authorization = if is_host_gateway_alias(normalized_host) + && let Some(expected_ip) = backend_host_gateway + { + AddressAuthorization::BackendPinnedGateway(expected_ip) + } else if is_host_gateway_alias(normalized_host) && let Some(expected_ip) = trusted_host_gateway { AddressAuthorization::TrustedGatewayAlias { expected_ip } @@ -140,7 +149,8 @@ pub(crate) fn filter_resolved_addresses( resolved_ips: &[IpAddr], ) -> Result, DestinationDenial> { let (kind, control_plane_blocked) = match &plan.address_authorization { - AddressAuthorization::TrustedGatewayAlias { .. } => { + AddressAuthorization::TrustedGatewayAlias { .. } + | AddressAuthorization::BackendPinnedGateway(_) => { (DestinationDenialKind::TrustedGateway, true) } AddressAuthorization::ExplicitAllowedIps(_) @@ -211,6 +221,20 @@ pub(crate) fn filter_resolved_addresses( None } } + AddressAuthorization::BackendPinnedGateway(expected_ip) => { + if is_cloud_metadata_ip(ip) { + Some(format!( + "{host} resolves to cloud metadata address {ip}, connection rejected" + )) + } else if ip != *expected_ip { + Some(format!( + "{host} resolves to {ip} which does not match backend host gateway \ + {expected_ip}, connection rejected" + )) + } else { + None + } + } AddressAuthorization::PinnedResolved(pinned) if !pinned.contains(&ip) => Some(format!( "{host} resolves to unpinned address {ip}, connection rejected" )), @@ -296,6 +320,23 @@ pub(crate) async fn validate_destination( DestinationDenial::new(DestinationDenialKind::TrustedGateway, reason) })? } + AddressAuthorization::BackendPinnedGateway(ip) => { + if BLOCKED_CONTROL_PLANE_PORTS.contains(&port) { + return Err(DestinationDenial::new( + DestinationDenialKind::TrustedGateway, + format!("port {port} is a blocked control-plane port, connection rejected"), + )); + } + if is_cloud_metadata_ip(*ip) { + return Err(DestinationDenial::new( + DestinationDenialKind::TrustedGateway, + format!( + "backend host gateway resolves to cloud metadata address {ip}, connection rejected" + ), + )); + } + vec![SocketAddr::new(*ip, port)] + } AddressAuthorization::ExplicitAllowedIps(networks) => { resolve_and_check_allowed_ips(host, port, networks, sandbox_entrypoint_pid) .await @@ -381,6 +422,7 @@ mod tests { "api.example.test", "api.example.test", None, + None, &["not-an-ip".to_string()], false, ) @@ -516,10 +558,26 @@ mod tests { #[test] fn validation_mode_precedence_is_explicit_and_stable() { + let backend_ip = IpAddr::V4(Ipv4Addr::LOCALHOST); let trusted_ip = IpAddr::V4(Ipv4Addr::new(169, 254, 1, 2)); + let backend = build_validation_plan( + "host.openshell.internal", + "host.openshell.internal", + Some(backend_ip), + Some(trusted_ip), + &["10.0.0.0/8".to_string()], + true, + ) + .unwrap(); + assert_eq!( + backend.address_authorization, + AddressAuthorization::BackendPinnedGateway(backend_ip) + ); + let trusted = build_validation_plan( "host.openshell.internal", "host.openshell.internal", + None, Some(trusted_ip), &["10.0.0.0/8".to_string()], true, @@ -536,6 +594,7 @@ mod tests { "10.2.3.4", "10.2.3.4", None, + None, &["10.0.0.0/8".to_string()], true, ) @@ -545,21 +604,24 @@ mod tests { AddressAuthorization::ExplicitAllowedIps(vec!["10.0.0.0/8".parse().unwrap()]) ); - let implicit = build_validation_plan("10.2.3.4", "10.2.3.4", None, &[], true).unwrap(); + let implicit = + build_validation_plan("10.2.3.4", "10.2.3.4", None, None, &[], true).unwrap(); assert_eq!( implicit.address_authorization, AddressAuthorization::ImplicitIpLiteral("10.2.3.4".parse().unwrap()) ); let declared = - build_validation_plan("private.example", "private.example", None, &[], true).unwrap(); + build_validation_plan("private.example", "private.example", None, None, &[], true) + .unwrap(); assert_eq!( declared.address_authorization, AddressAuthorization::ExactDeclaredHost ); let default = - build_validation_plan("*.example.com", "*.example.com", None, &[], false).unwrap(); + build_validation_plan("*.example.com", "*.example.com", None, None, &[], false) + .unwrap(); assert_eq!( default.address_authorization, AddressAuthorization::DefaultPublicOnly diff --git a/crates/openshell-supervisor-network/src/proxy/egress.rs b/crates/openshell-supervisor-network/src/proxy/egress.rs index 314596b048..55c4dd9099 100644 --- a/crates/openshell-supervisor-network/src/proxy/egress.rs +++ b/crates/openshell-supervisor-network/src/proxy/egress.rs @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +#![allow(dead_code)] + //! Transport-neutral egress inputs and authorization results. //! //! Explicit proxy adapters normalize their protocol-specific request into an diff --git a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs index c0e78484ac..296f987ece 100644 --- a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs +++ b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs @@ -381,27 +381,28 @@ network_policies: #[cfg(not(target_os = "linux"))] #[test] -fn static_identity_is_supported_off_linux() { +fn identity_required_mode_is_explicitly_unsupported_off_linux() { let engine = OpaEngine::from_strings( include_str!("../../../data/sandbox-policy.rego"), "network_policies: {}\n", ) .unwrap(); - let binary = tempfile::NamedTempFile::new().unwrap(); - std::fs::write(binary.path(), b"agent").unwrap(); - let identity = ProxyIdentityMode::static_binary(binary.path()).unwrap(); let decision = authorize_egress_intent( crate::procfs::WorkloadProxyTcpConnection::new( "127.0.0.1:41000".parse().unwrap(), "127.0.0.1:3000".parse().unwrap(), ), &engine, - &identity, + &BinaryIdentityCache::new(), + &AtomicU32::new(1), EgressIntent::connect("target.example".to_string(), 443), ); assert!(matches!(decision.action, NetworkAction::Deny { .. })); - assert_eq!(decision.identity, ProcessIdentityEvidence::Available); + assert_eq!( + decision.identity, + ProcessIdentityEvidence::Unavailable(IdentityUnavailableReason::UnsupportedPlatform) + ); } #[test] @@ -489,26 +490,20 @@ async fn exercise_benchmark_request(proxy_addr: SocketAddr, target: SocketAddr, #[test] #[ignore = "manual proxy allocation/query/latency baseline"] fn proxy_performance_baseline() { - temp_env::with_vars( - [( - openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY, - Some("endpoint-only"), - )], - || { - tokio::runtime::Builder::new_multi_thread() - .worker_threads(2) - .enable_all() - .build() - .unwrap() - .block_on(async { - // Benchmark the full fail-closed path using a declared loopback - // destination. This is deterministic and never opens a listener - // outside the local process, so it does not trigger host firewall - // prompts during manual baseline collection. - let target: SocketAddr = "127.0.0.1:18080".parse().unwrap(); - - let policy = format!( - r#" + tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + .unwrap() + .block_on(async { + // Benchmark the full fail-closed path using a declared loopback + // destination. This is deterministic and never opens a listener + // outside the local process, so it does not trigger host firewall + // prompts during manual baseline collection. + let target: SocketAddr = "127.0.0.1:18080".parse().unwrap(); + + let policy = format!( + r#" network_policies: proxy_compatibility: name: proxy_compatibility @@ -519,94 +514,89 @@ network_policies: binaries: - path: "/**" "#, - host = target.ip(), - port = target.port(), - ); - let engine = Arc::new( - OpaEngine::from_strings_with_binary_identity_required( - include_str!("../../../data/sandbox-policy.rego"), - &policy, - false, - ) - .unwrap(), - ); - let proxy_listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let proxy_addr = proxy_listener.local_addr().unwrap(); - let proxy_engine = engine.clone(); - let proxy_task = tokio::spawn(async move { - while let Ok((stream, _)) = proxy_listener.accept().await { - let engine = proxy_engine.clone(); - tokio::spawn(async move { - Box::pin(handle_tcp_connection( - stream, - engine, - Arc::new( - ProxyIdentityMode::static_binary( - std::env::current_exe().unwrap(), - ) - .unwrap(), - ), - None, - None, - AgentProposals::default(), - Arc::new(None), - Arc::new(None), - None, - None, - None, - None, - None, - )) - .await - .unwrap(); - }); - } + host = target.ip(), + port = target.port(), + ); + let engine = Arc::new( + OpaEngine::from_strings_with_binary_identity_required( + include_str!("../../../data/sandbox-policy.rego"), + &policy, + false, + ) + .unwrap(), + ); + let proxy_listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let proxy_addr = proxy_listener.local_addr().unwrap(); + let proxy_engine = engine.clone(); + let proxy_task = tokio::spawn(async move { + while let Ok((stream, _)) = proxy_listener.accept().await { + let engine = proxy_engine.clone(); + tokio::spawn(async move { + Box::pin(handle_tcp_connection( + stream, + engine, + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(0)), + None, + None, + AgentProposals::default(), + Arc::new(None), + Arc::new(None), + Arc::new(None), + None, + None, + None, + None, + None, + )) + .await + .unwrap(); }); - - for connect in [true, false] { - exercise_benchmark_request(proxy_addr, target, connect).await; - } - - let iterations = std::env::var("OPENSHELL_PROXY_BASELINE_ITERATIONS") - .ok() - .and_then(|value| value.parse::().ok()) - .filter(|value| *value > 0) - .unwrap_or(25); - let mut results = serde_json::Map::new(); - for (name, connect) in [("connect", true), ("forward", false)] { - crate::test_alloc::reset(); - crate::opa::reset_test_opa_query_count(); - let started = std::time::Instant::now(); - for _ in 0..iterations { - exercise_benchmark_request(proxy_addr, target, connect).await; - } - let elapsed = started.elapsed(); - let queries = crate::opa::test_opa_query_count(); - let (allocations, allocated_bytes) = crate::test_alloc::snapshot(); - let expected_queries = 4; - assert_eq!(queries, expected_queries * iterations); - results.insert( - name.to_string(), - serde_json::json!({ - "allocated_bytes_per_request": allocated_bytes / iterations, - "allocations_per_request": allocations / iterations, - "latency_ns_per_request": elapsed.as_nanos() / u128::from(iterations), - "opa_queries_per_request": queries / iterations, - }), - ); - } - println!( - "{}", - serde_json::json!({ - "iterations": iterations, - "proxy_performance_baseline": results, - "scenario": "declared_loopback_destination_denied", - "schema_version": 1, - }) - ); - - proxy_task.abort(); - }); - }, - ); + } + }); + + for connect in [true, false] { + exercise_benchmark_request(proxy_addr, target, connect).await; + } + + let iterations = std::env::var("OPENSHELL_PROXY_BASELINE_ITERATIONS") + .ok() + .and_then(|value| value.parse::().ok()) + .filter(|value| *value > 0) + .unwrap_or(25); + let mut results = serde_json::Map::new(); + for (name, connect) in [("connect", true), ("forward", false)] { + crate::test_alloc::reset(); + crate::opa::reset_test_opa_query_count(); + let started = std::time::Instant::now(); + for _ in 0..iterations { + exercise_benchmark_request(proxy_addr, target, connect).await; + } + let elapsed = started.elapsed(); + let queries = crate::opa::test_opa_query_count(); + let (allocations, allocated_bytes) = crate::test_alloc::snapshot(); + let expected_queries = 4; + assert_eq!(queries, expected_queries * iterations); + results.insert( + name.to_string(), + serde_json::json!({ + "allocated_bytes_per_request": allocated_bytes / iterations, + "allocations_per_request": allocations / iterations, + "latency_ns_per_request": elapsed.as_nanos() / u128::from(iterations), + "opa_queries_per_request": queries / iterations, + }), + ); + } + println!( + "{}", + serde_json::json!({ + "iterations": iterations, + "proxy_performance_baseline": results, + "scenario": "declared_loopback_destination_denied", + "schema_version": 1, + }) + ); + + proxy_task.abort(); + }); } diff --git a/crates/openshell-supervisor-network/src/run.rs b/crates/openshell-supervisor-network/src/run.rs index b11d9aa0d6..ca35c6e631 100644 --- a/crates/openshell-supervisor-network/src/run.rs +++ b/crates/openshell-supervisor-network/src/run.rs @@ -29,7 +29,6 @@ use openshell_core::denial::DenialEvent; use openshell_core::proposals::AgentProposals; use tokio::sync::mpsc::UnboundedSender; -#[cfg(target_os = "linux")] use crate::identity::BinaryIdentityCache; use crate::l7::tls::{ CertCache, ProxyTlsState, SandboxCa, build_upstream_client_config, read_system_ca_bundle, @@ -37,7 +36,8 @@ use crate::l7::tls::{ }; use crate::opa::OpaEngine; use crate::policy_local::PolicyLocalContext; -use crate::proxy::{ProxyHandle, ProxyIdentityMode}; +use crate::proxy::ProxyHandle; +use openshell_isolation_interface::contract::NetworkMediationSource; #[cfg(target_os = "linux")] pub struct TransparentRuntimeSetup { @@ -156,19 +156,13 @@ pub struct Networking { /// loop so it can publish updated `SandboxPolicy` snapshots that the /// `policy.local` route handler returns to the workload. pub policy_local_ctx: Arc, + _mediated_policy_dns: Option, #[cfg(target_os = "linux")] _policy_dns: Option, #[cfg(target_os = "linux")] _transparent_tcp: Option, } -#[cfg(not(target_os = "linux"))] -fn current_exe_static_identity_path() -> Result { - std::env::current_exe().map_err(|e| { - miette::miette!("failed to resolve supervisor executable for static proxy identity: {e}") - }) -} - /// Set up the networking stack: ephemeral CA + TLS state, proxy server, /// and the SSH-side proxy URL / netns FD. /// @@ -203,7 +197,10 @@ pub async fn run_networking( agent_proposals: AgentProposals, workspace_rx: tokio::sync::watch::Receiver, upstream_proxy_args: &crate::upstream_proxy::UpstreamProxyArgs, + proxy_tls_dir: Option<&std::path::Path>, + host_gateway_ip: Option, #[cfg(target_os = "linux")] transparent_runtime: Option, + network_mediation_source: Option>, ) -> Result { // Build the policy-local route context. The orchestrator's policy poll // loop also holds an `Arc` clone (via `Networking::policy_local_ctx`) so @@ -316,19 +313,42 @@ pub async fn run_networking( let _ = engine_ready_tx.send(true); } - // Linux procfs identity mode uses a SHA256 TOFU cache. Windows host mode - // uses an explicit static sandbox identity instead. - #[cfg(target_os = "linux")] + // Identity cache for SHA256 TOFU when OPA is active. Only consumed by + // the proxy, so it's owned here. let identity_cache = opa_engine.map(|_| Arc::new(BinaryIdentityCache::new())); - // Generate ephemeral CA and TLS state for HTTPS L7 inspection. + // Load a provisioned CA when the boundary lifetime outlives this control + // process; otherwise generate an ephemeral CA. // The CA cert is written to disk so sandbox processes can trust it. let (tls_state, ca_file_paths) = if matches!(policy.network.mode, NetworkMode::Proxy) { - match SandboxCa::generate() { + let configured_ca = match ( + std::env::var_os(openshell_core::sandbox_env::PROXY_CA_CERT), + std::env::var_os(openshell_core::sandbox_env::PROXY_CA_KEY), + ) { + (Some(certificate), Some(private_key)) => Some(SandboxCa::load_from_paths( + std::path::Path::new(&certificate), + std::path::Path::new(&private_key), + )?), + (None, None) => None, + _ => { + return Err(miette::miette!( + "{} and {} must be configured together", + openshell_core::sandbox_env::PROXY_CA_CERT, + openshell_core::sandbox_env::PROXY_CA_KEY, + )); + } + }; + let durable_ca = configured_ca.is_some(); + match configured_ca.map_or_else(SandboxCa::generate, Ok) { Ok(ca) => { - let tls_dir = std::env::var(openshell_core::sandbox_env::PROXY_TLS_DIR) - .unwrap_or_else(|_| openshell_core::container_paths::TLS_ROOT.to_string()); - let tls_dir = std::path::Path::new(&tls_dir); + let configured_tls_dir = + std::env::var_os(openshell_core::sandbox_env::PROXY_TLS_DIR) + .map(std::path::PathBuf::from); + let tls_dir = proxy_tls_dir + .or(configured_tls_dir.as_deref()) + .unwrap_or_else(|| { + std::path::Path::new(openshell_core::container_paths::TLS_ROOT) + }); let mut system_ca_bundle = read_system_ca_bundle(); // A TLS-intercepting corporate proxy (issue #1792) re-signs // tunneled server certificates with the corporate CA, so the @@ -364,7 +384,11 @@ pub async fn run_networking( .severity(SeverityId::Informational) .status(StatusId::Success) .state(StateId::Enabled, "enabled") - .message("TLS termination enabled: ephemeral CA generated") + .message(if durable_ca { + "TLS termination enabled: provisioned CA loaded" + } else { + "TLS termination enabled: ephemeral CA generated" + }) .build() ); (Some(state), Some(paths)) @@ -410,6 +434,21 @@ pub async fn run_networking( (None, None) }; + let mediated_policy_dns = if let Some(source) = network_mediation_source.clone() { + let engine = opa_engine + .cloned() + .ok_or_else(|| miette::miette!("Mediated DNS requires an OPA engine"))?; + Some(crate::policy_dns::PolicyDnsRuntime::start_mediated( + engine, + source, + host_gateway_ip, + crate::policy_dns::PolicyDnsRuntimeConfig::for_epoch(0)?, + engine_ready_rx.clone(), + )?) + } else { + None + }; + let proxy_handle = if matches!(policy.network.mode, NetworkMode::Proxy) { let proxy_policy = policy.network.proxy.as_ref().ok_or_else(|| { miette::miette!("Network mode is set to proxy but no proxy configuration was provided") @@ -419,6 +458,10 @@ pub async fn run_networking( miette::miette!("Proxy mode requires an OPA engine (--rego-policy and --rego-data)") })?; + let cache = identity_cache.clone().ok_or_else(|| { + miette::miette!("Proxy mode requires an identity cache (OPA engine must be configured)") + })?; + // If the orchestrator gave us a proxy bind IP (the host-side veth IP // from the workload's netns on Linux), use it so only traffic // originating inside the namespace can reach the proxy. Otherwise the @@ -429,25 +472,12 @@ pub async fn run_networking( SocketAddr::new(ip, port) }); - #[cfg(target_os = "linux")] - let identity_mode = { - let cache = identity_cache.clone().ok_or_else(|| { - miette::miette!( - "Proxy mode requires an identity cache (OPA engine must be configured)" - ) - })?; - ProxyIdentityMode::procfs(cache, entrypoint_pid.clone()) - }; - #[cfg(target_os = "windows")] - let identity_mode = ProxyIdentityMode::static_binary(current_exe_static_identity_path()?)?; - #[cfg(all(not(target_os = "linux"), not(target_os = "windows")))] - let identity_mode = ProxyIdentityMode::static_binary(current_exe_static_identity_path()?)?; - let proxy_handle = ProxyHandle::start_with_bind_addr( proxy_policy, bind_addr, engine, - Arc::new(identity_mode), + cache, + entrypoint_pid.clone(), tls_state, Some(provider_credentials.clone()), Some(policy_local_ctx.clone()), @@ -455,6 +485,11 @@ pub async fn run_networking( activity_tx.clone(), engine_ready_rx, upstream_proxy_args, + host_gateway_ip, + network_mediation_source, + mediated_policy_dns + .as_ref() + .map(|runtime| runtime.store.clone()), ) .await?; Some(proxy_handle) @@ -500,6 +535,7 @@ pub async fn run_networking( proxy: proxy_handle, ca_file_paths, policy_local_ctx, + _mediated_policy_dns: mediated_policy_dns, #[cfg(target_os = "linux")] _policy_dns: policy_dns, #[cfg(target_os = "linux")] diff --git a/crates/openshell-supervisor-network/src/spiffe_endpoint.rs b/crates/openshell-supervisor-network/src/spiffe_endpoint.rs new file mode 100644 index 0000000000..b3b6816f1e --- /dev/null +++ b/crates/openshell-supervisor-network/src/spiffe_endpoint.rs @@ -0,0 +1,18 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::path::Path; + +/// Convert a path to a SPIFFE Workload API endpoint URL. +/// +/// If the path already has a scheme (`unix:` or `tcp:`), use it as-is. +/// Otherwise, assume it is a Unix socket path and prepend `unix:`. +#[allow(dead_code)] +pub fn workload_api_endpoint(path: &Path) -> String { + let path = path.to_string_lossy(); + if path.starts_with("unix:") || path.starts_with("tcp:") { + path.into_owned() + } else { + format!("unix:{path}") + } +} diff --git a/crates/openshell-supervisor-network/src/upstream_proxy.rs b/crates/openshell-supervisor-network/src/upstream_proxy.rs index 9cb5481dfd..ab5ccafece 100644 --- a/crates/openshell-supervisor-network/src/upstream_proxy.rs +++ b/crates/openshell-supervisor-network/src/upstream_proxy.rs @@ -83,6 +83,9 @@ const CONNECT_HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(30); pub struct ProxyEndpoint { host: String, port: u16, + /// Optional driver-pinned address used only for the TCP dial. The + /// configured host remains authoritative for TLS identity and logging. + dial_ip: Option, /// Pre-computed `Basic ` header value from the proxy auth file. /// Never logged. proxy_authorization: Option, @@ -106,6 +109,7 @@ impl std::fmt::Debug for ProxyEndpoint { f.debug_struct("ProxyEndpoint") .field("host", &self.host) .field("port", &self.port) + .field("dial_ip", &self.dial_ip) .field("proxy_authorization", &self.proxy_authorization.is_some()) .field("tls", &self.tls.is_some()) .finish() @@ -334,6 +338,9 @@ pub struct UpstreamProxyArgs { /// `http://host:port` or `https://host:port` corporate proxy URL, or /// `None` for direct egress. pub https_proxy: Option, + /// Optional compute-driver-selected IP for reaching the proxy from the + /// supervisor's network namespace without changing its TLS identity. + pub proxy_dial_ip: Option, /// Comma-separated `NO_PROXY` list. pub no_proxy: Option, /// Path to the root-only credential mount (`user:pass`). @@ -354,6 +361,7 @@ pub struct UpstreamProxyArgs { // Supervisor CLI flag names for the corporate-proxy settings, used as the // dispatch keys in `from_lookup` and in operator-facing error messages. const ARG_HTTPS_PROXY: &str = "--upstream-proxy"; +const ARG_PROXY_DIAL_IP: &str = "--upstream-proxy-dial-ip"; const ARG_NO_PROXY: &str = "--upstream-no-proxy"; const ARG_PROXY_AUTH_FILE: &str = "--upstream-proxy-auth-file"; const ARG_PROXY_AUTH_ALLOW_INSECURE: &str = "--upstream-proxy-auth-allow-insecure"; @@ -391,6 +399,8 @@ impl UpstreamProxyConfig { Self::from_lookup(|name| { if name == ARG_HTTPS_PROXY { args.https_proxy.clone() + } else if name == ARG_PROXY_DIAL_IP { + args.proxy_dial_ip.map(|ip| ip.to_string()) } else if name == ARG_NO_PROXY { args.no_proxy.clone() } else if name == ARG_PROXY_AUTH_FILE { @@ -424,6 +434,12 @@ impl UpstreamProxyConfig { let https = var(ARG_HTTPS_PROXY)? .map(|url| parse_proxy_url(&url, ARG_HTTPS_PROXY)) .transpose()?; + let proxy_dial_ip = var(ARG_PROXY_DIAL_IP)? + .map(|raw| { + raw.parse::() + .map_err(|error| format!("{ARG_PROXY_DIAL_IP} is invalid: {error}")) + }) + .transpose()?; let auth_file = var(ARG_PROXY_AUTH_FILE)?; let auth_allow_insecure = var(ARG_PROXY_AUTH_ALLOW_INSECURE)?; let connect_by_hostname_raw = var(ARG_PROXY_CONNECT_BY_HOSTNAME)?; @@ -435,6 +451,7 @@ impl UpstreamProxyConfig { // silently running with direct egress. for (name, value) in [ (ARG_PROXY_AUTH_FILE, &auth_file), + (ARG_PROXY_DIAL_IP, &proxy_dial_ip.map(|ip| ip.to_string())), (ARG_PROXY_AUTH_ALLOW_INSECURE, &auth_allow_insecure), (ARG_PROXY_CONNECT_BY_HOSTNAME, &connect_by_hostname_raw), (ARG_NO_PROXY, &no_proxy_list), @@ -446,6 +463,7 @@ impl UpstreamProxyConfig { } return Ok(None); }; + https.dial_ip = proxy_dial_ip; // CONNECT-target mode. The default binds the tunnel to a validated // address; hostname CONNECT re-opens proxy-side DNS resolution and @@ -592,6 +610,7 @@ fn parse_proxy_url(raw: &str, var_name: &str) -> Result<(ProxyEndpoint, bool), S ProxyEndpoint { host: addr.host, port: addr.port, + dial_ip: None, proxy_authorization: None, tls: None, }, @@ -1000,7 +1019,10 @@ async fn connect_via_inner( port: u16, target: ConnectTarget, ) -> std::io::Result { - let tcp = TcpStream::connect((endpoint.host.as_str(), endpoint.port)).await?; + let tcp = match endpoint.dial_ip { + Some(ip) => TcpStream::connect(SocketAddr::new(ip, endpoint.port)).await?, + None => TcpStream::connect((endpoint.host.as_str(), endpoint.port)).await?, + }; set_tcp_nodelay_best_effort(&tcp); // For an `https://` proxy, wrap the connection in TLS (verifying the proxy // certificate against the configured roots) before the CONNECT handshake. @@ -1113,6 +1135,7 @@ mod tests { ARG_PROXY_AUTH_ALLOW_INSECURE as PROXY_AUTH_ALLOW_INSECURE, ARG_PROXY_AUTH_FILE as PROXY_AUTH_FILE, ARG_PROXY_CA_BUNDLE as PROXY_CA_BUNDLE, ARG_PROXY_CONNECT_BY_HOSTNAME as PROXY_CONNECT_BY_HOSTNAME, + ARG_PROXY_DIAL_IP as PROXY_DIAL_IP, }; fn config_from(pairs: &[(&str, &str)]) -> Result, String> { @@ -1822,6 +1845,7 @@ mod tests { ProxyEndpoint { host: addr.ip().to_string(), port: addr.port(), + dial_ip: None, proxy_authorization: auth.map(str::to_string), tls: None, } @@ -2227,13 +2251,16 @@ mod tests { // -- TLS (https://) proxies -- - /// A fake `https://` proxy: a TLS server with a self-signed cert for - /// 127.0.0.1 that answers CONNECT with 200. Returns the listen address, + /// A fake `https://` proxy: a TLS server with a self-signed cert for the + /// requested identity that answers CONNECT with 200. Returns the listen address, /// the server task (yielding the received CONNECT request), and the /// server certificate PEM to use as the corporate CA bundle. - async fn fake_tls_proxy() -> (SocketAddr, tokio::task::JoinHandle, String) { + async fn fake_tls_proxy( + tls_identity: &str, + ) -> (SocketAddr, tokio::task::JoinHandle, String) { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); let key = rcgen::KeyPair::generate().unwrap(); - let cert = rcgen::CertificateParams::new(vec!["127.0.0.1".to_string()]) + let cert = rcgen::CertificateParams::new(vec![tls_identity.to_string()]) .unwrap() .self_signed(&key) .unwrap(); @@ -2273,18 +2300,23 @@ mod tests { #[tokio::test] async fn connect_via_https_proxy_with_corporate_ca_bundle() { - let (addr, handle, cert_pem) = fake_tls_proxy().await; + const PROXY_IDENTITY: &str = "proxy.corp.test"; + let (addr, handle, cert_pem) = fake_tls_proxy(PROXY_IDENTITY).await; let ca_file = tempfile::NamedTempFile::new().unwrap(); std::fs::write(ca_file.path(), cert_pem).unwrap(); - let proxy_url = format!("https://{addr}"); + let proxy_url = format!("https://{PROXY_IDENTITY}:{}", addr.port()); let ca_path = ca_file.path().to_string_lossy().into_owned(); + let dial_ip = addr.ip().to_string(); let cfg = config_ok(&[ (HTTPS_PROXY, proxy_url.as_str()), + (PROXY_DIAL_IP, dial_ip.as_str()), (PROXY_CA_BUNDLE, ca_path.as_str()), ]); let endpoint = &cfg.https; assert!(endpoint.tls.is_some()); + assert_eq!(endpoint.host, PROXY_IDENTITY); + assert_eq!(endpoint.dial_ip, Some(addr.ip())); let stream = connect_via(endpoint, "api.example.com", 443, ConnectTarget::Hostname) .await @@ -2299,7 +2331,7 @@ mod tests { async fn connect_via_https_proxy_rejects_untrusted_cert() { // No corporate CA bundle: the self-signed proxy cert must not verify // against the built-in / system roots, so the handshake fails closed. - let (addr, _handle, _cert_pem) = fake_tls_proxy().await; + let (addr, _handle, _cert_pem) = fake_tls_proxy("127.0.0.1").await; let proxy_url = format!("https://{addr}"); let cfg = config_ok(&[(HTTPS_PROXY, proxy_url.as_str())]); let endpoint = &cfg.https; diff --git a/crates/openshell-supervisor-process/Cargo.toml b/crates/openshell-supervisor-process/Cargo.toml index 2e2120f1d0..cfcea38555 100644 --- a/crates/openshell-supervisor-process/Cargo.toml +++ b/crates/openshell-supervisor-process/Cargo.toml @@ -3,7 +3,7 @@ [package] name = "openshell-supervisor-process" -description = "Process component of the OpenShell supervisor: entrypoint spawn, SSH server, supervisor session, netns, bypass monitor" +description = "Process access and gateway session runtime for the OpenShell supervisor" version.workspace = true edition.workspace = true license.workspace = true @@ -12,14 +12,14 @@ rust-version.workspace = true [dependencies] openshell-core = { path = "../openshell-core" } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } openshell-ocsf = { path = "../openshell-ocsf" } -openshell-policy = { path = "../openshell-policy" } anyhow = { workspace = true } +async-trait = "0.1" base64 = { workspace = true } bytes = { workspace = true } hex = "0.4" -ipnet = "2" miette = { workspace = true } nix = { workspace = true } rand = "0.10" @@ -35,14 +35,6 @@ uuid = { workspace = true } [target.'cfg(unix)'.dependencies] libc = "0.2" -rustix = { workspace = true } - -[target.'cfg(target_os = "linux")'.dependencies] -capctl = "0.2.4" -landlock = "0.4" -seccompiler = "0.5" -socket2 = { workspace = true } -tempfile = "3" [dev-dependencies] tempfile = "3" diff --git a/crates/openshell-supervisor-process/src/bypass_monitor/mod.rs b/crates/openshell-supervisor-process/src/bypass_monitor/mod.rs deleted file mode 100644 index 44847b0d13..0000000000 --- a/crates/openshell-supervisor-process/src/bypass_monitor/mod.rs +++ /dev/null @@ -1,651 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Bypass detection monitor — reads kernel log messages from `/dev/kmsg` to -//! detect and report direct connection attempts that bypass the HTTP CONNECT -//! proxy. -//! -//! When the sandbox network namespace has nftables log rules installed (see -//! `NetworkNamespace::install_bypass_rules`), the kernel writes a log line for -//! each dropped packet. This module reads those messages, parses the nftables -//! LOG format, and emits structured tracing events + denial aggregator entries. -//! -//! ## Graceful degradation -//! -//! If `/dev/kmsg` cannot be opened (e.g., restricted container environment), -//! the monitor logs a one-time warning and returns. The nftables reject rules -//! still provide fast-fail UX — the monitor only adds diagnostic visibility. - -mod procfs; - -use openshell_core::activity::{ActivitySender, try_record_activity}; -use openshell_core::denial::DenialEvent; -use openshell_ocsf::{ - ActionId, ActivityId, ConfidenceId, DetectionFindingBuilder, DispositionId, Endpoint, - FindingInfo, NetworkActivityBuilder, Process, SeverityId, ocsf_emit, -}; -use std::sync::Arc; -use std::sync::atomic::{AtomicU32, Ordering}; -use tokio::sync::mpsc; -use tracing::debug; - -/// A parsed nftables log entry from `/dev/kmsg`. -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct BypassEvent { - /// Destination IP address. - pub dst_addr: String, - /// Destination port. - pub dst_port: u16, - /// Source port (used for process identity resolution). - pub src_port: u16, - /// Protocol (TCP or UDP). - pub proto: String, - /// UID of the process that initiated the connection. - pub uid: Option, -} - -/// Parse a nftables log line from `/dev/kmsg`. -/// -/// Expected format (from the kernel LOG target): -/// ```text -/// ...,;openshell:bypass::IN= OUT=veth-s-... SRC=10.200.0.2 DST=93.184.216.34 -/// LEN=60 ... PROTO=TCP SPT=48012 DPT=443 ... UID=1000 -/// ``` -/// -/// Returns `None` if the line doesn't match the expected prefix or is malformed. -pub fn parse_kmsg_line(line: &str, namespace_prefix: &str) -> Option { - // Check that this line contains our namespace prefix. - let prefix_pos = line.find(namespace_prefix)?; - let relevant = &line[prefix_pos + namespace_prefix.len()..]; - - let dst_addr = extract_field(relevant, "DST=")?; - let dst_port = extract_field(relevant, "DPT=")?.parse::().ok()?; - let src_port = extract_field(relevant, "SPT=") - .and_then(|s| s.parse::().ok()) - .unwrap_or(0); - let proto = extract_field(relevant, "PROTO=") - .unwrap_or_else(|| "unknown".to_string()) - .to_lowercase(); - let uid = extract_field(relevant, "UID=").and_then(|s| s.parse::().ok()); - - Some(BypassEvent { - dst_addr, - dst_port, - src_port, - proto, - uid, - }) -} - -fn build_bypass_ocsf_events( - event: &BypassEvent, - binary: &str, - binary_pid: &str, - ancestors: &str, -) -> (openshell_ocsf::OcsfEvent, openshell_ocsf::OcsfEvent) { - let hint = hint_for_event(event); - let reason = "direct connection bypassed HTTP CONNECT proxy"; - let dst_port = event.dst_port.to_string(); - let dst_ep = event.dst_addr.parse::().map_or_else( - |_| Endpoint::from_domain(&event.dst_addr, event.dst_port), - |ip| Endpoint::from_ip(ip, event.dst_port), - ); - - let net_event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Refuse) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .dst_endpoint(dst_ep) - .actor_process(Process::from_bypass(binary, binary_pid, ancestors)) - .firewall_rule("bypass-detect", "nftables") - .observation_point(3) - .message(format!( - "BYPASS_DETECT {}:{} proto={} binary={binary} action=reject reason={reason}", - event.dst_addr, event.dst_port, event.proto, - )) - .build(); - - let finding_event = DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .is_alert(true) - .confidence(ConfidenceId::High) - .finding_info(FindingInfo::new("bypass-detect", "Proxy Bypass Detected").with_desc(reason)) - .remediation(hint) - .evidence_pairs(&[ - ("dst_addr", event.dst_addr.as_str()), - ("dst_port", dst_port.as_str()), - ("proto", event.proto.as_str()), - ("binary", binary), - ("binary_pid", binary_pid), - ("ancestors", ancestors), - ]) - .message(format!( - "BYPASS_DETECT {}:{} proto={} binary={binary} hint={hint}", - event.dst_addr, event.dst_port, event.proto, - )) - .build(); - - (net_event, finding_event) -} - -/// Extract a single space-delimited field value from a nftables log line. -/// -/// Given `"DST="` and a string like `"...DST=93.184.216.34 LEN=60..."`, -/// returns `Some("93.184.216.34")`. -fn extract_field(s: &str, key: &str) -> Option { - let start = s.find(key)? + key.len(); - let rest = &s[start..]; - let end = rest.find(' ').unwrap_or(rest.len()); - let value = &rest[..end]; - if value.is_empty() { - None - } else { - Some(value.to_string()) - } -} - -/// Generate a protocol-appropriate hint for the bypass event. -fn hint_for_event(event: &BypassEvent) -> &'static str { - if event.proto == "udp" && event.dst_port == 53 { - "DNS queries should route through the sandbox proxy; check resolver configuration" - } else if event.proto == "udp" { - "UDP traffic must route through the sandbox proxy" - } else { - "ensure process honors HTTP_PROXY/HTTPS_PROXY; for Node.js set NODE_USE_ENV_PROXY=1" - } -} - -/// Spawn the bypass monitor as a background tokio task. -/// -/// Uses `dmesg --follow` to tail the kernel ring buffer for nftables log -/// entries matching the given namespace. Falls back gracefully if `dmesg` -/// is not available. -/// -/// We use `dmesg` rather than reading `/dev/kmsg` directly because the -/// container runtime's device cgroup policy blocks direct `/dev/kmsg` access -/// even with `CAP_SYSLOG`. The `dmesg` command reads via the `syslog(2)` -/// syscall which is permitted with `CAP_SYSLOG`. -/// -/// Returns a `JoinHandle` if the monitor was started, or `None` if `dmesg` -/// is not available. -pub fn spawn( - namespace_name: String, - entrypoint_pid: Arc, - denial_tx: Option>, - activity_tx: Option, -) -> Option> { - use std::io::BufRead; - use std::process::{Command, Stdio}; - - // Verify dmesg is available before spawning the monitor. - let dmesg_check = Command::new("dmesg") - .arg("--version") - .stdout(Stdio::null()) - .stderr(Stdio::null()) - .status(); - - if !dmesg_check.is_ok_and(|s| s.success()) { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Other) - .severity(SeverityId::Low) - .message( - "dmesg not available; bypass detection monitor will not run. \ - Bypass REJECT rules still provide fast-fail behavior.", - ) - .build(); - ocsf_emit!(event); - return None; - } - - let namespace_prefix = format!("openshell:bypass:{namespace_name}:"); - debug!( - namespace = %namespace_name, - "Starting bypass detection monitor via dmesg --follow" - ); - - let handle = tokio::task::spawn_blocking(move || { - // Start dmesg in follow mode to tail new kernel messages. - let mut child = match Command::new("dmesg") - .args(["--follow", "--notime"]) - .stdout(Stdio::piped()) - .stderr(Stdio::null()) - .spawn() - { - Ok(c) => c, - Err(e) => { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Other) - .severity(SeverityId::Low) - .message(format!( - "Failed to start dmesg --follow; bypass monitor will not run: {e}" - )) - .build(); - ocsf_emit!(event); - return; - } - }; - - let Some(stdout) = child.stdout.take() else { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Other) - .severity(SeverityId::Low) - .message("dmesg --follow produced no stdout; bypass monitor will not run") - .build(); - ocsf_emit!(event); - return; - }; - - let reader = std::io::BufReader::new(stdout); - for line in reader.lines() { - let line = match line { - Ok(l) => l, - Err(e) => { - debug!(error = %e, "Error reading dmesg line, continuing"); - continue; - } - }; - - let Some(event) = parse_kmsg_line(&line, &namespace_prefix) else { - continue; - }; - - // Attempt process identity resolution (best-effort, TCP only). - let pid = entrypoint_pid.load(Ordering::Acquire); - let (binary, binary_pid, ancestors) = - if event.proto == "tcp" && event.src_port > 0 && pid > 0 { - resolve_process_identity(pid, event.src_port) - } else { - ("-".to_string(), "-".to_string(), "-".to_string()) - }; - - // Dual-emit: Network Activity [4001] + Detection Finding [2004] - let (net_event, finding_event) = - build_bypass_ocsf_events(&event, &binary, &binary_pid, &ancestors); - ocsf_emit!(net_event); - ocsf_emit!(finding_event); - - // Send to denial aggregator if available. - if let Some(ref tx) = denial_tx { - let ancestors_vec: Vec = if ancestors == "-" { - vec![] - } else { - ancestors.split(" -> ").map(String::from).collect() - }; - - let _ = tx.send(DenialEvent { - host: event.dst_addr.clone(), - port: event.dst_port, - binary: binary.clone(), - ancestors: ancestors_vec, - deny_reason: "direct connection bypassed HTTP CONNECT proxy".to_string(), - denial_stage: "bypass".to_string(), - l7_method: None, - l7_path: None, - }); - } - if let Some(ref tx) = activity_tx { - let _ = try_record_activity(tx, true, "bypass"); - } - } - - // Clean up the dmesg child process. - let _ = child.kill(); - let _ = child.wait(); - debug!("Bypass monitor: dmesg reader exited"); - }); - - Some(handle) -} - -/// Resolve process identity from a TCP source port. -/// -/// Returns `(binary_path, pid, ancestors)` as display strings. -/// Falls back to `("-", "-", "-")` on any failure (race condition, etc.). -fn resolve_process_identity(entrypoint_pid: u32, src_port: u16) -> (String, String, String) { - match procfs::resolve_tcp_peer_socket_owners(entrypoint_pid, src_port) { - Ok(socket_owners) => { - let mut identities = Vec::new(); - for owner in &socket_owners.owners { - let Ok(binary_path) = procfs::binary_path(owner.pid.cast_signed()) else { - continue; - }; - let ancestors = procfs::collect_ancestor_binaries(owner.pid, entrypoint_pid); - identities.push((owner.pid, binary_path, ancestors)); - } - - if identities.is_empty() { - return ("-".to_string(), "-".to_string(), "-".to_string()); - } - - identities.sort_by_key(|(pid, _, _)| *pid); - let first_identity = (identities[0].1.clone(), identities[0].2.clone()); - let ambiguous = identities - .iter() - .skip(1) - .any(|(_, binary_path, ancestors)| { - binary_path != &first_identity.0 || ancestors != &first_identity.1 - }); - - if ambiguous { - let pids = identities - .iter() - .map(|(pid, _, _)| pid.to_string()) - .collect::>() - .join(", "); - let owner_summary = identities - .iter() - .map(|(pid, binary_path, ancestors)| { - let ancestors_str = if ancestors.is_empty() { - "-".to_string() - } else { - ancestors - .iter() - .map(|p| p.display().to_string()) - .collect::>() - .join(" -> ") - }; - format!( - "pid={pid} binary={} ancestors=[{ancestors_str}]", - binary_path.display() - ) - }) - .collect::>() - .join("; "); - return ("ambiguous".to_string(), pids, owner_summary); - } - - let (pid, binary_path, ancestors) = identities.remove(0); - let ancestors_str = if ancestors.is_empty() { - "-".to_string() - } else { - ancestors - .iter() - .map(|p| p.display().to_string()) - .collect::>() - .join(" -> ") - }; - ( - binary_path.display().to_string(), - pid.to_string(), - ancestors_str, - ) - } - Err(_) => ("-".to_string(), "-".to_string(), "-".to_string()), - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn parse_kmsg_line_tcp_bypass() { - let line = "6,1234,5678,-;openshell:bypass:sandbox-abcd1234:IN= OUT=veth-s-abcd1234 \ - SRC=10.200.0.2 DST=93.184.216.34 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=12345 \ - DF PROTO=TCP SPT=48012 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0 UID=1000"; - - let event = parse_kmsg_line(line, "openshell:bypass:sandbox-abcd1234:").unwrap(); - assert_eq!(event.dst_addr, "93.184.216.34"); - assert_eq!(event.dst_port, 443); - assert_eq!(event.src_port, 48012); - assert_eq!(event.proto, "tcp"); - assert_eq!(event.uid, Some(1000)); - } - - #[test] - fn parse_kmsg_line_udp_dns_bypass() { - let line = "6,5678,9012,-;openshell:bypass:sandbox-abcd1234:IN= OUT=veth-s-abcd1234 \ - SRC=10.200.0.2 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 \ - DF PROTO=UDP SPT=53421 DPT=53 LEN=32 UID=1000"; - - let event = parse_kmsg_line(line, "openshell:bypass:sandbox-abcd1234:").unwrap(); - assert_eq!(event.dst_addr, "8.8.8.8"); - assert_eq!(event.dst_port, 53); - assert_eq!(event.src_port, 53421); - assert_eq!(event.proto, "udp"); - assert_eq!(event.uid, Some(1000)); - } - - #[test] - fn parse_kmsg_line_no_uid() { - let line = "6,1234,5678,-;openshell:bypass:sandbox-abcd1234:IN= OUT=veth-s-abcd1234 \ - SRC=10.200.0.2 DST=10.0.0.5 LEN=60 PROTO=TCP SPT=12345 DPT=6379"; - - let event = parse_kmsg_line(line, "openshell:bypass:sandbox-abcd1234:").unwrap(); - assert_eq!(event.dst_addr, "10.0.0.5"); - assert_eq!(event.dst_port, 6379); - assert_eq!(event.proto, "tcp"); - assert_eq!(event.uid, None); - } - - #[test] - fn parse_kmsg_line_wrong_namespace_returns_none() { - let line = "6,1234,5678,-;openshell:bypass:sandbox-other:IN= OUT=veth \ - SRC=10.200.0.2 DST=1.2.3.4 PROTO=TCP SPT=1111 DPT=80"; - - let result = parse_kmsg_line(line, "openshell:bypass:sandbox-abcd1234:"); - assert!(result.is_none()); - } - - #[test] - fn parse_kmsg_line_unrelated_message_returns_none() { - let line = "6,1234,5678,-;audit: type=1400 audit(1234567890.123:1): something else"; - let result = parse_kmsg_line(line, "openshell:bypass:sandbox-abcd1234:"); - assert!(result.is_none()); - } - - #[test] - fn parse_kmsg_line_missing_dst_returns_none() { - let line = "6,1234,5678,-;openshell:bypass:sandbox-abcd1234:IN= OUT=veth \ - SRC=10.200.0.2 PROTO=TCP SPT=1111 DPT=80"; - // Missing DST= field - let result = parse_kmsg_line(line, "openshell:bypass:sandbox-abcd1234:"); - assert!(result.is_none()); - } - - #[test] - fn parse_kmsg_line_ipv6_address() { - let line = "6,1234,5678,-;openshell:bypass:sandbox-abcd1234:IN= OUT=veth-s-abcd1234 \ - SRC=fd00::2 DST=2001:4860:4860::8888 LEN=60 PROTO=TCP SPT=55555 DPT=443 UID=1000"; - - let event = parse_kmsg_line(line, "openshell:bypass:sandbox-abcd1234:").unwrap(); - assert_eq!(event.dst_addr, "2001:4860:4860::8888"); - assert_eq!(event.dst_port, 443); - assert_eq!(event.proto, "tcp"); - } - - #[test] - fn hint_for_tcp_event() { - let event = BypassEvent { - dst_addr: "1.2.3.4".to_string(), - dst_port: 443, - src_port: 12345, - proto: "tcp".to_string(), - uid: None, - }; - assert!(hint_for_event(&event).contains("HTTP_PROXY")); - } - - #[test] - fn hint_for_dns_bypass() { - let event = BypassEvent { - dst_addr: "8.8.8.8".to_string(), - dst_port: 53, - src_port: 12345, - proto: "udp".to_string(), - uid: None, - }; - assert!(hint_for_event(&event).contains("DNS")); - } - - #[test] - fn hint_for_non_dns_udp() { - let event = BypassEvent { - dst_addr: "1.2.3.4".to_string(), - dst_port: 5060, - src_port: 12345, - proto: "udp".to_string(), - uid: None, - }; - assert!(hint_for_event(&event).contains("UDP")); - } - - #[test] - fn bypass_ocsf_contract_is_stable() { - let event = BypassEvent { - dst_addr: "93.184.216.34".to_string(), - dst_port: 443, - src_port: 48012, - proto: "tcp".to_string(), - uid: Some(1000), - }; - let (network, finding) = - build_bypass_ocsf_events(&event, "/usr/bin/curl", "42", "/usr/bin/sh"); - let network = serde_json::to_value(network).unwrap(); - assert_eq!(network["class_name"], "Network Activity"); - assert_eq!(network["activity_name"], "Refuse"); - assert_eq!(network["action"], "Denied"); - assert_eq!(network["disposition"], "Blocked"); - assert_eq!(network["severity"], "Medium"); - assert!(network.get("status").is_none()); - assert_eq!(network["dst_endpoint"]["ip"], "93.184.216.34"); - assert_eq!(network["dst_endpoint"]["port"], 443); - assert_eq!(network["actor"]["process"]["name"], "/usr/bin/curl"); - assert_eq!(network["firewall_rule"]["name"], "bypass-detect"); - assert_eq!(network["firewall_rule"]["type"], "nftables"); - assert_eq!(network["observation_point_id"], 3); - assert!( - network["message"] - .as_str() - .unwrap() - .contains("action=reject") - ); - - let finding = serde_json::to_value(finding).unwrap(); - assert_eq!(finding["class_name"], "Detection Finding"); - assert_eq!(finding["action"], "Denied"); - assert_eq!(finding["disposition"], "Blocked"); - assert_eq!(finding["severity"], "Medium"); - assert_eq!(finding["confidence"], "High"); - assert_eq!(finding["is_alert"], true); - assert_eq!(finding["finding_info"]["uid"], "bypass-detect"); - assert_eq!(finding["finding_info"]["title"], "Proxy Bypass Detected"); - assert_eq!(finding["evidences"][0]["data"]["dst_port"], "443"); - } - - #[test] - fn resolve_process_identity_surfaces_ambiguous_shared_socket() { - use std::ffi::CString; - use std::net::{TcpListener, TcpStream}; - use std::os::fd::AsRawFd; - use std::time::{Duration, Instant}; - - if !std::path::Path::new("/bin/sleep").exists() { - eprintln!("skipping: /bin/sleep not available"); - return; - } - - let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); - let listener_port = listener.local_addr().unwrap().port(); - let stream = TcpStream::connect(("127.0.0.1", listener_port)).expect("connect"); - let peer_port = stream.local_addr().unwrap().port(); - let (_accepted, _) = listener.accept().expect("accept"); - - let fd = stream.as_raw_fd(); - // libc/syscall FFI requires unsafe - #[allow(unsafe_code)] - unsafe { - let flags = libc::fcntl(fd, libc::F_GETFD); - assert!(flags >= 0, "F_GETFD failed"); - assert_eq!( - libc::fcntl(fd, libc::F_SETFD, flags & !libc::FD_CLOEXEC), - 0, - "F_SETFD failed" - ); - } - - let sleep_path = CString::new("/bin/sleep").unwrap(); - let arg0 = CString::new("sleep").unwrap(); - let arg1 = CString::new("30").unwrap(); - // libc/syscall FFI requires unsafe - #[allow(unsafe_code)] - let child_pid = unsafe { libc::fork() }; - assert!(child_pid >= 0, "fork failed"); - if child_pid == 0 { - // libc/syscall FFI requires unsafe - #[allow(unsafe_code)] - unsafe { - libc::execl( - sleep_path.as_ptr(), - arg0.as_ptr(), - arg1.as_ptr(), - std::ptr::null::(), - ); - libc::_exit(127); - } - } - - if std::fs::read_link(format!("/proc/{child_pid}/exe")).is_err() - || std::fs::read_dir(format!("/proc/{child_pid}/fd")).is_err() - { - #[allow(unsafe_code)] - unsafe { - libc::kill(child_pid, libc::SIGKILL); - libc::waitpid(child_pid, std::ptr::null_mut(), 0); - } - eprintln!("skipping: cannot read /proc/{child_pid} (restricted /proc)"); - return; - } - - let deadline = Instant::now() + Duration::from_secs(2); - loop { - if let Ok(link) = std::fs::read_link(format!("/proc/{child_pid}/exe")) - && link.to_string_lossy().contains("sleep") - { - break; - } - assert!( - Instant::now() < deadline, - "child pid {child_pid} did not exec into sleep within 2s" - ); - std::thread::sleep(Duration::from_millis(20)); - } - - let (binary, pid, ancestors) = resolve_process_identity(std::process::id(), peer_port); - - // libc/syscall FFI requires unsafe - #[allow(unsafe_code)] - unsafe { - libc::kill(child_pid, libc::SIGKILL); - libc::waitpid(child_pid, std::ptr::null_mut(), 0); - } - - assert_eq!(binary, "ambiguous"); - assert!(pid.contains(&std::process::id().to_string())); - assert!(pid.contains(&child_pid.to_string())); - assert!(ancestors.contains("binary=")); - } - - #[test] - fn extract_field_basic() { - let s = "DST=1.2.3.4 LEN=60"; - assert_eq!(extract_field(s, "DST="), Some("1.2.3.4".to_string())); - assert_eq!(extract_field(s, "LEN="), Some("60".to_string())); - } - - #[test] - fn extract_field_missing() { - let s = "DST=1.2.3.4 LEN=60"; - assert_eq!(extract_field(s, "PROTO="), None); - } - - #[test] - fn extract_field_at_end_of_string() { - let s = "DST=1.2.3.4"; - assert_eq!(extract_field(s, "DST="), Some("1.2.3.4".to_string())); - } -} diff --git a/crates/openshell-supervisor-process/src/bypass_monitor/procfs.rs b/crates/openshell-supervisor-process/src/bypass_monitor/procfs.rs deleted file mode 100644 index 98bf9634a6..0000000000 --- a/crates/openshell-supervisor-process/src/bypass_monitor/procfs.rs +++ /dev/null @@ -1,318 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Linux `/proc` filesystem reading for bypass-monitor process identity. -//! -//! Trimmed copy of `openshell-supervisor-network`'s `procfs` module: only -//! the helpers the bypass monitor calls when resolving the originating PID -//! and binary for an nftables LOG entry. The networking leaf keeps its own -//! richer copy because it also needs sha256 hashing, cmdline scraping, and -//! ambiguity-failure helpers for its proxy identity cache. - -use miette::Result; -use std::collections::HashSet; -use std::path::PathBuf; - -/// Where a socket owner was discovered while scanning `/proc`. -#[derive(Clone, Debug, Eq, PartialEq)] -pub enum SocketOwnerSource { - /// Owner was found in the entrypoint process tree at the given BFS depth. - Descendant { depth: usize }, - /// Owner was found by scanning all of `/proc` after the descendant scan. - ProcFallback, -} - -/// A process with an fd pointing at a target socket inode. -#[derive(Clone, Debug, Eq, PartialEq)] -pub struct SocketOwner { - pub pid: u32, - pub source: SocketOwnerSource, -} - -/// All process owners for a TCP peer socket. -#[derive(Clone, Debug, Eq, PartialEq)] -pub struct TcpPeerSocketOwners { - pub inode: u64, - pub owners: Vec, -} - -#[derive(Clone, Debug, Eq, PartialEq)] -struct DescendantPid { - pid: u32, - depth: usize, -} - -/// Read the binary path of a process via `/proc/{pid}/exe` symlink. -/// -/// Strips the kernel-added `" (deleted)"` suffix when the raw readlink -/// target cannot be stat'd, so callers see a clean path. See the networking -/// crate's procfs documentation for the full rationale. -pub fn binary_path(pid: i32) -> Result { - use std::ffi::OsString; - use std::io::ErrorKind; - use std::os::unix::ffi::{OsStrExt, OsStringExt}; - - const DELETED_SUFFIX: &[u8] = b" (deleted)"; - - let link = format!("/proc/{pid}/exe"); - let target = std::fs::read_link(&link).map_err(|e| { - miette::miette!( - "Failed to read /proc/{pid}/exe: {e}. \ - Cannot determine binary identity — denying request. \ - Hint: the proxy may need CAP_SYS_PTRACE or to run as the same user." - ) - })?; - - let raw_target_missing = - matches!(std::fs::metadata(&target), Err(err) if err.kind() == ErrorKind::NotFound); - - let bytes = target.as_os_str().as_bytes(); - if raw_target_missing && bytes.ends_with(DELETED_SUFFIX) { - let stripped = bytes[..bytes.len() - DELETED_SUFFIX.len()].to_vec(); - return Ok(PathBuf::from(OsString::from_vec(stripped))); - } - - Ok(target) -} - -/// Resolve all process owners for the TCP peer inside a sandbox network namespace. -pub fn resolve_tcp_peer_socket_owners( - entrypoint_pid: u32, - peer_port: u16, -) -> Result { - let inode = parse_proc_net_tcp(entrypoint_pid, peer_port)?; - let owners = find_socket_inode_owners(inode, entrypoint_pid)?; - Ok(TcpPeerSocketOwners { inode, owners }) -} - -/// Read the `PPid` (parent PID) from `/proc//status`. -fn read_ppid(pid: u32) -> Option { - let status = std::fs::read_to_string(format!("/proc/{pid}/status")).ok()?; - for line in status.lines() { - if let Some(rest) = line.strip_prefix("PPid:") { - return rest.trim().parse().ok(); - } - } - None -} - -/// Walk the process tree upward from `pid`, collecting binary paths. -/// -/// Stops at PID 1 (init), `stop_pid` (the entrypoint process), or after -/// 64 ancestors. The returned vec excludes `pid` itself. -#[allow(clippy::similar_names)] -pub fn collect_ancestor_binaries(pid: u32, stop_pid: u32) -> Vec { - const MAX_DEPTH: usize = 64; - let mut ancestors = Vec::new(); - let mut current = pid; - - for _ in 0..MAX_DEPTH { - let ppid = match read_ppid(current) { - Some(p) if p > 0 && p != current => p, - _ => break, - }; - - if let Ok(path) = binary_path(ppid.cast_signed()) { - ancestors.push(path); - } - - if ppid == stop_pid || ppid == 1 { - break; - } - current = ppid; - } - - ancestors -} - -fn parse_proc_net_tcp(pid: u32, peer_port: u16) -> Result { - for suffix in &["tcp", "tcp6"] { - let path = format!("/proc/{pid}/net/{suffix}"); - let Ok(content) = std::fs::read_to_string(&path) else { - continue; - }; - - for line in content.lines().skip(1) { - let fields: Vec<&str> = line.split_whitespace().collect(); - if fields.len() < 10 { - continue; - } - - let local_addr = fields[1]; - let local_port = match local_addr.rsplit_once(':') { - Some((_, port_hex)) => u16::from_str_radix(port_hex, 16).unwrap_or(0), - None => continue, - }; - - let state = fields[3]; - if state != "01" { - continue; - } - - if local_port == peer_port { - let inode: u64 = fields[9] - .parse() - .map_err(|_| miette::miette!("Failed to parse inode from {}", fields[9]))?; - if inode == 0 { - continue; - } - return Ok(inode); - } - } - } - - Err(miette::miette!( - "No ESTABLISHED TCP connection found for port {} in /proc/{}/net/tcp{{,6}}", - peer_port, - pid - )) -} - -fn find_socket_inode_owners(inode: u64, entrypoint_pid: u32) -> Result> { - let target = format!("socket:[{inode}]"); - let mut owners = Vec::new(); - let mut checked = HashSet::new(); - - let descendants = collect_descendant_pids_with_depth(entrypoint_pid); - - for descendant in &descendants { - checked.insert(descendant.pid); - if check_pid_fds(descendant.pid, &target) { - owners.push(SocketOwner { - pid: descendant.pid, - source: SocketOwnerSource::Descendant { - depth: descendant.depth, - }, - }); - } - } - - if let Ok(proc_dir) = std::fs::read_dir("/proc") { - let mut proc_pids = Vec::new(); - for entry in proc_dir.flatten() { - let name = entry.file_name(); - if let Ok(pid) = name.to_string_lossy().parse::() { - proc_pids.push(pid); - } - } - proc_pids.sort_unstable(); - - for pid in proc_pids { - if checked.contains(&pid) { - continue; - } - checked.insert(pid); - if check_pid_fds(pid, &target) { - owners.push(SocketOwner { - pid, - source: SocketOwnerSource::ProcFallback, - }); - } - } - } - - if !owners.is_empty() { - return Ok(owners); - } - - Err(miette::miette!( - "No process found owning socket inode {} \ - (scanned {} descendants of entrypoint PID {}). \ - Hint: the container may need --cap-add=SYS_PTRACE to read /proc//fd/ \ - for processes running as a different user.", - inode, - descendants.len(), - entrypoint_pid - )) -} - -fn check_pid_fds(pid: u32, target: &str) -> bool { - let fd_dir = format!("/proc/{pid}/fd"); - let Some(fds) = std::fs::read_dir(&fd_dir).ok() else { - return false; - }; - for fd_entry in fds.flatten() { - if let Ok(link) = std::fs::read_link(fd_entry.path()) - && link.to_string_lossy() == target - { - return true; - } - } - false -} - -fn collect_descendant_pids_with_depth(root_pid: u32) -> Vec { - let mut pids = vec![DescendantPid { - pid: root_pid, - depth: 0, - }]; - let mut seen = HashSet::from([root_pid]); - let mut i = 0; - while i < pids.len() { - let pid = pids[i].pid; - let child_depth = pids[i].depth + 1; - let task_dir = format!("/proc/{pid}/task"); - if let Ok(tasks) = std::fs::read_dir(&task_dir) { - for task_entry in tasks.flatten() { - let children_path = task_entry.path().join("children"); - if let Ok(children_str) = std::fs::read_to_string(&children_path) { - for child in children_str.split_whitespace() { - if let Ok(child_pid) = child.parse::() - && seen.insert(child_pid) - { - pids.push(DescendantPid { - pid: child_pid, - depth: child_depth, - }); - } - } - } - } - } - i += 1; - } - pids -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn binary_path_reads_current_process() { - let pid = std::process::id().cast_signed(); - let path = binary_path(pid).unwrap(); - assert!(path.exists()); - } - - #[test] - #[allow(clippy::similar_names)] - fn read_ppid_returns_parent() { - let pid = std::process::id(); - let ppid = read_ppid(pid); - assert!(ppid.is_some(), "Should be able to read PPid of self"); - assert!(ppid.unwrap() > 0, "PPid should be > 0"); - } - - #[test] - fn read_ppid_nonexistent_pid() { - let result = read_ppid(999_999_999); - assert!(result.is_none()); - } - - #[test] - fn collect_ancestor_binaries_returns_parents() { - let pid = std::process::id(); - let ancestors = collect_ancestor_binaries(pid, 1); - assert!( - !ancestors.is_empty(), - "Should have at least one ancestor binary" - ); - for path in &ancestors { - assert!( - !path.as_os_str().is_empty(), - "Ancestor path should not be empty" - ); - } - } -} diff --git a/crates/openshell-supervisor-process/src/delegated.rs b/crates/openshell-supervisor-process/src/delegated.rs new file mode 100644 index 0000000000..28792b2d6f --- /dev/null +++ b/crates/openshell-supervisor-process/src/delegated.rs @@ -0,0 +1,271 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Supervisor-owned access-plane assembly for a remote sandbox. + +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; + +use miette::Result; +use openshell_isolation_interface::contract::{ + BoundaryExec, BoundaryLoopbackConnector, BoundaryProcess, +}; +use openshell_ocsf::{ActivityId, AppLifecycleBuilder, SeverityId, StatusId, ocsf_emit}; + +fn ocsf_ctx() -> &'static openshell_ocsf::EventContext { + openshell_ocsf::ctx::ctx() +} + +/// Supervisor-owned SSH and gateway-session tasks for a running sandbox. +pub struct BoundaryAccess { + instance_id: String, + terminating: Arc, + ssh_task: Option>, + session_task: Option>, + session_readiness: Option>, + main_session: Option>, +} + +impl BoundaryAccess { + /// Stable supervisor instance ID used for lifecycle reporting. + #[must_use] + pub fn instance_id(&self) -> &str { + &self.instance_id + } + + /// Observe whether the gateway has accepted the current supervisor + /// session. The value returns to false while the session reconnects. + #[must_use] + pub fn session_readiness(&self) -> Option> { + self.session_readiness.clone() + } + + /// Publish the canonical process's terminal status to attached clients. + pub async fn publish_main_exit(&self, exit_code: i32, attachment_expected: bool) { + let Some(main_session) = self.main_session.as_ref() else { + return; + }; + let _ = main_session + .finish_remote(exit_code, attachment_expected) + .await; + } + + /// Release terminal delivery after the gateway acknowledges the exit, then + /// wait for attached clients to consume the terminal status. + pub async fn drain_main_terminal_delivery(&self) { + let Some(main_session) = self.main_session.as_ref() else { + return; + }; + main_session.mark_terminal_reported(); + main_session.wait_for_terminal_attachments().await; + } +} + +impl Drop for BoundaryAccess { + fn drop(&mut self) { + self.terminating.store(true, Ordering::Release); + if let Some(task) = self.ssh_task.take() { + task.abort(); + } + if let Some(task) = self.session_task.take() { + task.abort(); + } + } +} + +/// Start the supervisor access plane using sandbox-supplied exec and +/// loopback-forwarding capabilities. +#[allow(clippy::too_many_arguments)] +pub async fn start_boundary_access( + sandbox_id: Option<&str>, + openshell_endpoint: Option<&str>, + ssh_socket_path: Option<&str>, + shared_ssh_socket: bool, + ca_file_paths: Option<(std::path::PathBuf, std::path::PathBuf)>, + boundary_exec: Arc, + port_forward: Arc, + agent: Arc, +) -> Result { + let instance_id = uuid::Uuid::new_v4().to_string(); + let terminating = Arc::new(AtomicBool::new(false)); + let Some(ssh_socket_path) = ssh_socket_path.map(std::path::PathBuf::from) else { + return Ok(BoundaryAccess { + instance_id, + terminating, + ssh_task: None, + session_task: None, + session_readiness: None, + main_session: None, + }); + }; + + let attachment = agent + .attach() + .await + .map_err(|error| miette::miette!(error.to_string()))?; + let main_session = crate::main_session::MainSession::from_boundary(attachment, agent); + + let (ssh_ready_tx, ssh_ready_rx) = tokio::sync::oneshot::channel(); + let listen_path = ssh_socket_path.clone(); + let ssh_port_forward = port_forward.clone(); + let ssh_main_session = main_session.clone(); + let ssh_task = tokio::spawn(async move { + if let Err(error) = crate::ssh::run_ssh_server( + listen_path, + ssh_ready_tx, + ca_file_paths, + shared_ssh_socket, + ssh_port_forward, + boundary_exec, + Some(ssh_main_session), + ) + .await + { + ocsf_emit!( + AppLifecycleBuilder::new(ocsf_ctx()) + .activity(ActivityId::Fail) + .severity(SeverityId::Critical) + .status(StatusId::Failure) + .message(format!("SSH server failed: {error}")) + .build() + ); + } + }); + + match tokio::time::timeout(Duration::from_secs(10), ssh_ready_rx).await { + Ok(Ok(Ok(()))) => {} + Ok(Ok(Err(error))) => { + ssh_task.abort(); + return Err(error.context("SSH server failed during startup")); + } + Ok(Err(_)) => { + ssh_task.abort(); + return Err(miette::miette!( + "SSH server task ended before signaling readiness" + )); + } + Err(_) => { + ssh_task.abort(); + return Err(miette::miette!( + "SSH server did not start within 10 seconds" + )); + } + } + + let (session_task, session_readiness) = match (openshell_endpoint, sandbox_id) { + (Some(endpoint), Some(id)) => { + let (task, mut accepted) = crate::supervisor_session::spawn_with_readiness( + endpoint.to_string(), + id.to_string(), + ssh_socket_path, + port_forward, + None, + terminating.clone(), + instance_id.clone(), + ); + let accepted_result = + tokio::time::timeout(Duration::from_secs(10), accepted.wait_for(|ready| *ready)) + .await + .map(|result| result.map(|_| ())); + match accepted_result { + Ok(Ok(())) => (Some(task), Some(accepted)), + Ok(Err(_)) => { + task.abort(); + return Err(miette::miette!( + "supervisor session ended before gateway acceptance" + )); + } + Err(_) => { + task.abort(); + return Err(miette::miette!( + "gateway did not accept supervisor session within 10 seconds" + )); + } + } + } + _ => (None, None), + }; + + Ok(BoundaryAccess { + instance_id, + terminating, + ssh_task: Some(ssh_task), + session_task, + session_readiness, + main_session: Some(main_session), + }) +} + +/// Report the canonical process exit until the gateway acknowledges it. +pub async fn report_main_process_exit( + endpoint: &str, + sandbox_id: &str, + instance_id: &str, + exit_code: i32, +) { + let mut delay = Duration::from_millis(250); + loop { + match crate::supervisor_session::report_main_process_exit( + endpoint, + sandbox_id, + instance_id, + exit_code, + ) + .await + { + Ok(()) => break, + Err(error) => { + tracing::warn!(%error, "main-process exit report failed; retrying"); + tokio::time::sleep(delay).await; + delay = (delay * 2).min(Duration::from_secs(2)); + } + } + } +} + +/// Finalize canonical process terminal delivery until acknowledged. +pub async fn finalize_main_process_exit(endpoint: &str, sandbox_id: &str, instance_id: &str) { + let mut delay = Duration::from_millis(250); + loop { + match crate::supervisor_session::finalize_main_process_exit( + endpoint, + sandbox_id, + instance_id, + ) + .await + { + Ok(()) => break, + Err(error) => { + tracing::warn!(%error, "main-process finalization failed; retrying"); + tokio::time::sleep(delay).await; + delay = (delay * 2).min(Duration::from_secs(2)); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn expected_post_exit_attachment_is_preserved_for_remote_main() { + let main_session = crate::main_session::MainSession::inert(); + let access = BoundaryAccess { + instance_id: "instance".to_string(), + terminating: Arc::new(AtomicBool::new(false)), + ssh_task: None, + session_task: None, + session_readiness: None, + main_session: Some(main_session.clone()), + }; + + access.publish_main_exit(7, true).await; + + main_session + .begin_terminal_attachment() + .expect("declared CLI attachment must remain valid after a fast remote main exits"); + main_session.end_terminal_attachment(); + } +} diff --git a/crates/openshell-supervisor-process/src/lib.rs b/crates/openshell-supervisor-process/src/lib.rs index 743942faa4..023a6c8e73 100644 --- a/crates/openshell-supervisor-process/src/lib.rs +++ b/crates/openshell-supervisor-process/src/lib.rs @@ -1,30 +1,18 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Process component of the `OpenShell` supervisor. +//! Access-plane component of the `OpenShell` supervisor. //! -//! Owns the entrypoint process spawn, SSH server, supervisor session, network -//! namespace, bypass monitor, child environment construction, skills install, -//! and log push. Populated by follow-up commits as modules migrate out of -//! `openshell-sandbox`. +//! Owns SSH access, retained canonical-process I/O, gateway supervisor +//! sessions, skills, and log forwarding. Workload spawning and in-sandbox +//! enforcement live exclusively in `openshell-sandbox`. -pub mod child_env; pub mod debug_rpc; -#[cfg(unix)] -pub mod identity; +pub mod delegated; pub mod log_push; pub mod main_session; -pub mod managed_children; -pub mod process; -pub mod run; -pub mod sandbox; pub mod skills; pub mod ssh; pub mod supervisor_session; mod unix_socket; - -#[cfg(target_os = "linux")] -pub mod bypass_monitor; -#[cfg(target_os = "linux")] -pub mod netns; diff --git a/crates/openshell-supervisor-process/src/main_session.rs b/crates/openshell-supervisor-process/src/main_session.rs index 00dd2ea53c..3ffa24d1d0 100644 --- a/crates/openshell-supervisor-process/src/main_session.rs +++ b/crates/openshell-supervisor-process/src/main_session.rs @@ -17,10 +17,22 @@ use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::sync::Notify; use tokio::sync::watch; -use crate::process::ProcessIo; +use openshell_isolation_interface::contract::{ + BoundaryProcess, BoundarySignal, BoundaryTerminal, ProcessAttachment, +}; const OUTPUT_BUFFER_BYTES: usize = 1024 * 1024; +/// Canonical-process I/O retained by the supervisor session multiplexer. +pub enum ProcessIo { + Pty(std::fs::File), + Pipes { + stdin: tokio::process::ChildStdin, + stdout: tokio::process::ChildStdout, + stderr: tokio::process::ChildStderr, + }, +} + #[derive(Clone, Debug)] pub enum MainOutput { Stdout(Bytes), @@ -186,6 +198,8 @@ pub struct MainSession { input_owner: Mutex>, next_owner: AtomicU64, pty_master: Option>, + boundary_process: Option>, + boundary_terminal: Option>, readers_remaining: AtomicUsize, readers_done: Notify, finished: std::sync::atomic::AtomicBool, @@ -194,6 +208,7 @@ pub struct MainSession { } impl MainSession { + const REMOTE_OUTPUT_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2); #[cfg(test)] pub fn inert() -> Arc { let (input, _input_rx) = tokio::sync::mpsc::channel(64); @@ -205,6 +220,8 @@ impl MainSession { input_owner: Mutex::new(None), next_owner: AtomicU64::new(1), pty_master: None, + boundary_process: None, + boundary_terminal: None, readers_remaining: AtomicUsize::new(0), readers_done: Notify::new(), finished: std::sync::atomic::AtomicBool::new(false), @@ -256,6 +273,8 @@ impl MainSession { input_owner: Mutex::new(None), next_owner: AtomicU64::new(1), pty_master, + boundary_process: None, + boundary_terminal: None, readers_remaining: AtomicUsize::new(if terminal { 1 } else { 2 }), readers_done: Notify::new(), finished: std::sync::atomic::AtomicBool::new(false), @@ -270,6 +289,81 @@ impl MainSession { session } + /// Build the control-side multiplexer around a boundary-owned admitted + /// process. Process lifecycle and PTY operations remain delegated to the + /// boundary process handle. + #[must_use] + pub fn from_boundary( + attachment: ProcessAttachment, + process: Arc, + ) -> Arc { + let ProcessAttachment { + stdin, + stdout, + stderr, + terminal, + } = attachment; + let terminal_mode = terminal.is_some(); + let (input, mut input_rx) = tokio::sync::mpsc::channel::>(64); + let session = Arc::new(Self { + pid: 0, + terminal: terminal_mode, + input, + output: OutputLog::new(), + input_owner: Mutex::new(None), + next_owner: AtomicU64::new(1), + pty_master: None, + boundary_process: Some(process), + boundary_terminal: terminal, + readers_remaining: AtomicUsize::new(if terminal_mode { 1 } else { 2 }), + readers_done: Notify::new(), + finished: std::sync::atomic::AtomicBool::new(false), + terminal_attachments: Mutex::new(TerminalAttachmentState { + active: 0, + process_finished: false, + expectation: AttachmentExpectation::None, + }), + terminal_attachments_done: Notify::new(), + }); + let stdout_session = Arc::clone(&session); + tokio::spawn(async move { + let mut stdout = stdout; + let mut buffer = [0u8; 4096]; + loop { + match stdout.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(read) => stdout_session + .publish(MainOutput::Stdout(Bytes::copy_from_slice(&buffer[..read]))), + } + } + stdout_session.reader_finished(); + }); + if let Some(mut stderr) = stderr { + let stderr_session = Arc::clone(&session); + tokio::spawn(async move { + let mut buffer = [0u8; 4096]; + loop { + match stderr.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(read) => stderr_session + .publish(MainOutput::Stderr(Bytes::copy_from_slice(&buffer[..read]))), + } + } + stderr_session.reader_finished(); + }); + } + tokio::spawn(async move { + let mut stdin = stdin; + while let Some(data) = input_rx.recv().await { + if stdin.write_all(&data).await.is_err() { + break; + } + let _ = stdin.flush().await; + } + }); + session + } + fn start_io( this: &Arc, io: ProcessIo, @@ -380,10 +474,39 @@ impl MainSession { /// /// Returns whether terminal delivery must complete before shutdown. pub async fn finish(&self, exit_code: i32, attachment_expected: bool) -> bool { + self.wait_for_output_readers().await; + self.complete_finish(exit_code, attachment_expected) + } + + /// Finish a remotely owned process without allowing descendants that keep + /// inherited output descriptors open to block terminal publication forever. + pub async fn finish_remote(&self, exit_code: i32, attachment_expected: bool) -> bool { + self.finish_remote_with_timeout( + exit_code, + attachment_expected, + Self::REMOTE_OUTPUT_DRAIN_TIMEOUT, + ) + .await + } + + async fn finish_remote_with_timeout( + &self, + exit_code: i32, + attachment_expected: bool, + timeout: std::time::Duration, + ) -> bool { + let _ = tokio::time::timeout(timeout, self.wait_for_output_readers()).await; + self.complete_finish(exit_code, attachment_expected) + } + + async fn wait_for_output_readers(&self) { let notified = self.readers_done.notified(); if self.readers_remaining.load(Ordering::Acquire) != 0 { notified.await; } + } + + fn complete_finish(&self, exit_code: i32, attachment_expected: bool) -> bool { let delivery_pending = { let mut state = self .terminal_attachments @@ -410,6 +533,23 @@ impl MainSession { self.output.subscribe() } + /// Return the bounded output sequence range currently retained for a + /// replacement supervisor. A nonzero first sequence is an explicit + /// truncation watermark rather than silent data loss. + #[must_use] + pub fn output_window(&self) -> (u64, u64, bool) { + let state = self + .output + .state + .lock() + .expect("main output log lock poisoned"); + let first_sequence = state + .events + .front() + .map_or(state.next_sequence, |event| event.sequence); + (first_sequence, state.next_sequence, first_sequence != 0) + } + /// Wait until the gateway durably acknowledges the main-process result. pub async fn wait_for_terminal_reported(&self) { let notified = self.output.terminal_reported_notify.notified(); @@ -499,7 +639,16 @@ impl MainSession { } } - pub fn resize(&self, columns: u32, rows: u32, pixel_width: u32, pixel_height: u32) { + pub async fn resize(&self, columns: u32, rows: u32, pixel_width: u32, pixel_height: u32) { + if let Some(terminal) = self.boundary_terminal.as_ref() { + let _ = terminal + .resize( + u16::try_from(columns.max(1)).unwrap_or(u16::MAX), + u16::try_from(rows.max(1)).unwrap_or(u16::MAX), + ) + .await; + return; + } let Some(master) = self.pty_master.as_ref() else { return; }; @@ -515,9 +664,23 @@ impl MainSession { } } - pub fn signal_group(&self, signal: nix::sys::signal::Signal) -> Result<(), nix::errno::Errno> { + pub async fn signal_group(&self, signal: nix::sys::signal::Signal) -> Result<(), String> { + if let Some(process) = self.boundary_process.as_ref() { + let signal = match signal { + nix::sys::signal::Signal::SIGHUP => BoundarySignal::Hup, + nix::sys::signal::Signal::SIGINT => BoundarySignal::Int, + nix::sys::signal::Signal::SIGKILL => BoundarySignal::Kill, + nix::sys::signal::Signal::SIGTERM => BoundarySignal::Term, + other => return Err(format!("boundary signal {other:?} is unsupported")), + }; + return process + .signal(signal) + .await + .map_err(|error| error.to_string()); + } let pid = i32::try_from(self.pid).unwrap_or(i32::MAX); nix::sys::signal::kill(nix::unistd::Pid::from_raw(-pid), signal) + .map_err(|error| error.to_string()) } #[must_use] @@ -544,6 +707,83 @@ fn set_nonblocking(file: &std::fs::File) -> Result<(), nix::errno::Errno> { #[cfg(test)] mod tests { use super::*; + use openshell_isolation_interface::contract::{ + BackendError, BoundaryExitStatus, BoundaryInput, BoundaryOutput, + }; + + struct TestBoundaryProcess { + signals: Mutex>, + } + + #[async_trait::async_trait] + impl BoundaryProcess for TestBoundaryProcess { + async fn wait(&self) -> Result { + Ok(BoundaryExitStatus::Exited(0)) + } + + async fn signal(&self, signal: BoundarySignal) -> Result<(), BackendError> { + self.signals.lock().unwrap().push(signal); + Ok(()) + } + + async fn terminate(&self) -> Result<(), BackendError> { + Ok(()) + } + } + + struct TestBoundaryTerminal { + size: Mutex>, + } + + #[async_trait::async_trait] + impl BoundaryTerminal for TestBoundaryTerminal { + async fn resize(&self, cols: u16, rows: u16) -> Result<(), BackendError> { + *self.size.lock().unwrap() = Some((cols, rows)); + Ok(()) + } + } + + #[tokio::test] + async fn boundary_attachment_drives_main_io_signal_and_terminal() { + let (stdin, mut stdin_peer) = tokio::io::duplex(1024); + let (stdout, mut stdout_peer) = tokio::io::duplex(1024); + let process = Arc::new(TestBoundaryProcess { + signals: Mutex::new(Vec::new()), + }); + let terminal = Arc::new(TestBoundaryTerminal { + size: Mutex::new(None), + }); + let stdin: BoundaryInput = Box::new(stdin); + let stdout: BoundaryOutput = Box::new(stdout); + let attachment = ProcessAttachment { + stdin, + stdout, + stderr: None, + terminal: Some(terminal.clone()), + }; + let session = MainSession::from_boundary(attachment, process.clone()); + let mut output = session.subscribe(); + + stdout_peer.write_all(b"ready\n").await.unwrap(); + assert!(matches!( + output.recv().await.unwrap(), + MainOutput::Stdout(data) if data == b"ready\n"[..] + )); + + let (_owner, input) = session.acquire_input().unwrap(); + input.send(b"hello\n".to_vec()).await.unwrap(); + let mut received = [0_u8; 6]; + stdin_peer.read_exact(&mut received).await.unwrap(); + assert_eq!(&received, b"hello\n"); + + session.resize(120, 40, 0, 0).await; + assert_eq!(*terminal.size.lock().unwrap(), Some((120, 40))); + session + .signal_group(nix::sys::signal::Signal::SIGINT) + .await + .unwrap(); + assert_eq!(*process.signals.lock().unwrap(), vec![BoundarySignal::Int]); + } #[test] fn input_lease_has_one_owner_and_can_be_reacquired() { @@ -632,6 +872,27 @@ mod tests { .expect("closing the attachment should wake the waiter"); } + #[tokio::test] + async fn remote_finish_bounds_output_drain_before_publishing_exit() { + let mut session = MainSession::inert(); + Arc::get_mut(&mut session) + .expect("sole test session reference") + .readers_remaining = AtomicUsize::new(1); + let mut output = session.subscribe(); + + session + .finish_remote_with_timeout(19, false, std::time::Duration::from_millis(10)) + .await; + + assert!(matches!( + output + .recv() + .await + .expect("terminal status after bounded drain"), + MainOutput::Exit(19) + )); + } + #[tokio::test] async fn declared_attachment_waits_for_connection_then_natural_close() { let session = MainSession::inert(); diff --git a/crates/openshell-supervisor-process/src/managed_children.rs b/crates/openshell-supervisor-process/src/managed_children.rs deleted file mode 100644 index 311c80693f..0000000000 --- a/crates/openshell-supervisor-process/src/managed_children.rs +++ /dev/null @@ -1,53 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Process-wide tracker for sandbox-managed child PIDs. -//! -//! The supervisor spawns several long-lived children (the entrypoint, SSH -//! sessions). Each registers its PID here on spawn and removes it on exit so -//! the orchestrator's `SIGCHLD` reaper can distinguish supervised processes -//! from incidental zombies. - -#![cfg(target_os = "linux")] - -use std::collections::HashSet; -use std::sync::{LazyLock, Mutex}; - -static MANAGED_CHILDREN: LazyLock>> = - LazyLock::new(|| Mutex::new(HashSet::new())); - -/// Add `pid` to the supervised-child set. Non-positive or out-of-range values -/// are silently ignored. -pub fn register(pid: u32) { - let Ok(pid) = i32::try_from(pid) else { - return; - }; - if pid <= 0 { - return; - } - if let Ok(mut children) = MANAGED_CHILDREN.lock() { - children.insert(pid); - } -} - -/// Remove `pid` from the supervised-child set. Non-positive or out-of-range -/// values are silently ignored. -pub fn unregister(pid: u32) { - let Ok(pid) = i32::try_from(pid) else { - return; - }; - if pid <= 0 { - return; - } - if let Ok(mut children) = MANAGED_CHILDREN.lock() { - children.remove(&pid); - } -} - -/// Return `true` if `pid` is currently in the supervised-child set. -#[must_use] -pub fn is_managed(pid: i32) -> bool { - MANAGED_CHILDREN - .lock() - .is_ok_and(|children| children.contains(&pid)) -} diff --git a/crates/openshell-supervisor-process/src/netns/mod.rs b/crates/openshell-supervisor-process/src/netns/mod.rs deleted file mode 100644 index 2b4ea554ed..0000000000 --- a/crates/openshell-supervisor-process/src/netns/mod.rs +++ /dev/null @@ -1,1239 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Network namespace isolation for sandboxed processes. -//! -//! Creates an isolated network namespace with a veth pair connecting -//! the sandbox to the host. This ensures the sandboxed process can only -//! communicate through the proxy running on the host side of the veth. - -mod nft_ruleset; - -use miette::{IntoDiagnostic, Result}; -use std::net::IpAddr; -use std::os::unix::io::RawFd; -use std::path::Path; -use std::process::Command; -use tracing::{debug, warn}; -use uuid::Uuid; - -/// Default subnet for sandbox networking. -const SUBNET_PREFIX: &str = "10.200.0"; -const HOST_IP_SUFFIX: u8 = 1; -const SANDBOX_IP_SUFFIX: u8 = 2; -/// Unprivileged port owned by the supervisor's policy DNS service. Workload -/// queries still target the standard DNS port and nftables redirects them to -/// this listener before the bypass fence runs. -pub const POLICY_DNS_PORT: u16 = 15_053; -pub const TRANSPARENT_TCP_PORT: u16 = 15_001; -const IP_SEARCH_PATHS: &[&str] = &["/usr/sbin/ip", "/sbin/ip", "/usr/bin/ip", "/bin/ip"]; -const NSENTER_SEARCH_PATHS: &[&str] = &[ - "/usr/bin/nsenter", - "/bin/nsenter", - "/usr/sbin/nsenter", - "/sbin/nsenter", -]; - -/// Handle to a network namespace with veth pair. -/// -/// The namespace and veth interfaces are automatically cleaned up on drop. -#[derive(Debug)] -pub struct NetworkNamespace { - /// Namespace name (e.g., "sandbox-{uuid}") - name: String, - /// Host-side veth interface name - veth_host: String, - /// Sandbox-side veth interface name (inside namespace, used only during setup) - _veth_sandbox: String, - /// Host-side IP address (proxy binds here) - host_ip: IpAddr, - /// Sandbox-side IP address - sandbox_ip: IpAddr, - /// File descriptor for the namespace (for setns) - ns_fd: Option, -} - -impl NetworkNamespace { - /// Create a new isolated network namespace with veth pair. - /// - /// Sets up: - /// - A new network namespace named `sandbox-{uuid}` - /// - A veth pair connecting host and sandbox - /// - IP addresses on both ends (10.200.0.1/24 and 10.200.0.2/24) - /// - Default route in sandbox pointing to host - /// - /// # Errors - /// - /// Returns an error if namespace creation or network setup fails. - pub fn create() -> Result { - let id = Uuid::new_v4(); - let short_id = &id.to_string()[..8]; - let name = format!("sandbox-{short_id}"); - let veth_host = format!("veth-h-{short_id}"); - let veth_sandbox = format!("veth-s-{short_id}"); - - let host_ip: IpAddr = format!("{SUBNET_PREFIX}.{HOST_IP_SUFFIX}").parse().unwrap(); - let sandbox_ip: IpAddr = format!("{SUBNET_PREFIX}.{SANDBOX_IP_SUFFIX}") - .parse() - .unwrap(); - - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Enabled, "creating") - .message(format!( - "Creating network namespace [ns:{name} host_veth:{veth_host} sandbox_veth:{veth_sandbox}]" - )) - .build() - ); - - // Create the namespace - run_ip(&["netns", "add", &name])?; - - // Create veth pair - if let Err(e) = run_ip(&[ - "link", - "add", - &veth_host, - "type", - "veth", - "peer", - "name", - &veth_sandbox, - ]) { - // Cleanup namespace on failure - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - // Move sandbox veth into namespace - if let Err(e) = run_ip(&["link", "set", &veth_sandbox, "netns", &name]) { - let _ = run_ip(&["link", "delete", &veth_host]); - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - // Configure host side - let host_cidr = format!("{host_ip}/24"); - if let Err(e) = run_ip(&["addr", "add", &host_cidr, "dev", &veth_host]) { - let _ = run_ip(&["link", "delete", &veth_host]); - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - if let Err(e) = run_ip(&["link", "set", &veth_host, "up"]) { - let _ = run_ip(&["link", "delete", &veth_host]); - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - // Configure sandbox side (inside namespace) - let sandbox_cidr = format!("{sandbox_ip}/24"); - if let Err(e) = run_ip_netns(&name, &["addr", "add", &sandbox_cidr, "dev", &veth_sandbox]) { - let _ = run_ip(&["link", "delete", &veth_host]); - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - if let Err(e) = run_ip_netns(&name, &["link", "set", &veth_sandbox, "up"]) { - let _ = run_ip(&["link", "delete", &veth_host]); - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - // Bring up loopback in namespace - if let Err(e) = run_ip_netns(&name, &["link", "set", "lo", "up"]) { - let _ = run_ip(&["link", "delete", &veth_host]); - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - // Add default route via host - let host_ip_str = host_ip.to_string(); - if let Err(e) = run_ip_netns(&name, &["route", "add", "default", "via", &host_ip_str]) { - let _ = run_ip(&["link", "delete", &veth_host]); - let _ = run_ip(&["netns", "delete", &name]); - return Err(e); - } - - // Open the namespace file descriptor for later use with setns - let ns_path = openshell_core::container_paths::netns_path(&name); - let ns_fd = match nix::fcntl::open( - ns_path.as_path(), - nix::fcntl::OFlag::O_RDONLY, - nix::sys::stat::Mode::empty(), - ) { - Ok(fd) => Some(fd), - Err(e) => { - warn!(error = %e, "Failed to open namespace fd, will use nsenter fallback"); - None - } - }; - - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Enabled, "created") - .message(format!( - "Network namespace created [ns:{name} host_ip:{host_ip} sandbox_ip:{sandbox_ip}]" - )) - .build() - ); - - Ok(Self { - name, - veth_host, - _veth_sandbox: veth_sandbox, - host_ip, - sandbox_ip, - ns_fd, - }) - } - - /// Get the host-side IP address (proxy should bind to this). - #[must_use] - pub const fn host_ip(&self) -> IpAddr { - self.host_ip - } - - /// Get the sandbox-side IP address. - #[must_use] - pub const fn sandbox_ip(&self) -> IpAddr { - self.sandbox_ip - } - - /// Get the namespace name. - #[must_use] - pub fn name(&self) -> &str { - &self.name - } - - /// Enter this network namespace. - /// - /// Must be called from the child process after fork, before exec. - /// Uses `setns()` to switch the calling process into the namespace. - /// - /// # Errors - /// - /// Returns an error if setns fails. - /// - /// # Safety - /// - /// This function should only be called in a `pre_exec` context after fork. - pub fn enter(&self) -> Result<()> { - if let Some(fd) = self.ns_fd { - debug!(namespace = %self.name, "Entering network namespace via setns"); - // SAFETY: setns is safe to call after fork, before exec - // libc/syscall FFI requires unsafe - #[allow(unsafe_code)] - let result = unsafe { libc::setns(fd, libc::CLONE_NEWNET) }; - if result != 0 { - return Err(miette::miette!( - "setns failed: {}", - std::io::Error::last_os_error() - )); - } - Ok(()) - } else { - Err(miette::miette!( - "No namespace file descriptor available for setns" - )) - } - } - - /// Get the namespace file descriptor for use with clone/unshare. - #[must_use] - pub const fn ns_fd(&self) -> Option { - self.ns_fd - } - - /// Install nftables rules for bypass detection inside the namespace. - /// - /// Sets up OUTPUT chain rules that: - /// 1. ACCEPT traffic destined for the proxy (`host_ip:proxy_port`) - /// 2. ACCEPT loopback traffic - /// 3. ACCEPT established/related connections (response packets) - /// 4. LOG + REJECT all other TCP/UDP traffic (bypass attempts) - /// - /// This provides two benefits: - /// - **Fast-fail UX**: applications get immediate ECONNREFUSED instead of - /// a 30-second timeout when they bypass the proxy - /// - **Diagnostics**: nftables LOG entries are picked up by the bypass - /// monitor to emit structured tracing events - /// - /// Degrades gracefully if `nft` is not available — the namespace - /// still provides isolation via routing, just without fast-fail and - /// diagnostic logging. - pub fn install_bypass_rules(&self, proxy_port: u16) -> Result<()> { - let Some(nft_path) = find_nft() else { - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Medium) - .status(openshell_ocsf::StatusId::Failure) - .state(openshell_ocsf::StateId::Disabled, "degraded") - .message(format!( - "nft not found; bypass detection rules will not be installed [ns:{}]", - self.name - )) - .build() - ); - return Ok(()); - }; - - let host_ip_str = self.host_ip.to_string(); - let log_prefix = format!("openshell:bypass:{}:", &self.name); - - // The kernel's nf_log_syslog module suppresses log output from - // non-init network namespaces by default. Enable it so the bypass - // monitor can see log entries from the sandbox namespace. - enable_nf_log_all_netns(); - - let commands = - nft_ruleset::generate_bypass_commands(&host_ip_str, proxy_port, Some(&log_prefix)); - - if let Err(e) = run_nft_commands_netns(&self.name, &nft_path, &commands) { - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Medium) - .status(openshell_ocsf::StatusId::Failure) - .state(openshell_ocsf::StateId::Disabled, "failed") - .message(format!( - "Failed to install bypass detection rules [ns:{}]: {e}", - self.name - )) - .build() - ); - return Err(e); - } - - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Enabled, "installed") - .message(format!( - "Bypass detection rules installed [ns:{}]", - self.name - )) - .build() - ); - - Ok(()) - } - - /// Replace the ordinary bypass fence with the policy-DNS and transparent - /// TCP ruleset. This is fail-closed: callers must not release workload - /// execution unless every required rule was installed. - pub fn install_transparent_tcp_rules( - &self, - proxy_port: u16, - synthetic_ipv4_cidr: &str, - synthetic_ipv6_cidr: &str, - ) -> Result<()> { - self.validate_synthetic_pool_routes(synthetic_ipv4_cidr, synthetic_ipv6_cidr)?; - // The inner namespace has an IPv4 default route, but not an IPv6 - // default route. Install only the active synthetic IPv6 epoch so the - // kernel reaches the nft OUTPUT hook; REDIRECT then reroutes it to - // the local transparent listener. - run_ip_netns( - &self.name, - &["-6", "route", "replace", synthetic_ipv6_cidr, "dev", "lo"], - )?; - let nft_path = find_nft().ok_or_else(|| { - miette::miette!( - "trusted nft helper not found; policy DNS and transparent TCP require nftables" - ) - })?; - let host_ip = self.host_ip.to_string(); - let log_prefix = format!("openshell:bypass:{}:", self.name); - let commands = nft_ruleset::generate_transparent_tcp_commands( - &host_ip, - proxy_port, - POLICY_DNS_PORT, - TRANSPARENT_TCP_PORT, - synthetic_ipv4_cidr, - synthetic_ipv6_cidr, - Some(&log_prefix), - ); - run_nft_commands_netns(&self.name, &nft_path, &commands)?; - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Enabled, "installed") - .message(format!( - "Policy DNS and transparent TCP capture installed [ns:{}]", - self.name - )) - .build() - ); - Ok(()) - } - - fn validate_synthetic_pool_routes( - &self, - synthetic_ipv4_cidr: &str, - synthetic_ipv6_cidr: &str, - ) -> Result<()> { - let reserved = [ - synthetic_ipv4_cidr - .parse::() - .into_diagnostic()?, - synthetic_ipv6_cidr - .parse::() - .into_diagnostic()?, - ]; - for family in ["-4", "-6"] { - let routes = - run_ip_netns_output(&self.name, &[family, "route", "show", "table", "all"])?; - if let Some((route, pool)) = first_route_overlap(&routes, &reserved) { - return Err(miette::miette!( - "synthetic address pool {pool} overlaps workload route {route}; refusing to enable policy DNS" - )); - } - } - Ok(()) - } - - /// Bind IPv4 and IPv6 transparent listeners inside the workload network - /// namespace without moving an async runtime worker into that namespace. - pub async fn bind_transparent_tcp_listeners( - &self, - ) -> std::io::Result> { - let ns_fd = self - .ns_fd - .ok_or_else(|| std::io::Error::other("no namespace fd available for bind"))?; - let (tx, rx) = tokio::sync::oneshot::channel(); - std::thread::spawn(move || { - let result = (|| -> std::io::Result> { - #[allow(unsafe_code)] - if unsafe { libc::setns(ns_fd, libc::CLONE_NEWNET) } != 0 { - return Err(std::io::Error::last_os_error()); - } - let mut listeners = Vec::with_capacity(2); - for (domain, address) in [ - ( - socket2::Domain::IPV4, - format!("0.0.0.0:{TRANSPARENT_TCP_PORT}"), - ), - ( - socket2::Domain::IPV6, - format!("[::]:{TRANSPARENT_TCP_PORT}"), - ), - ] { - let socket = socket2::Socket::new( - domain, - socket2::Type::STREAM, - Some(socket2::Protocol::TCP), - )?; - socket.set_reuse_address(true)?; - if domain == socket2::Domain::IPV6 { - socket.set_only_v6(true)?; - } - let address: std::net::SocketAddr = address.parse().map_err(|error| { - std::io::Error::other(format!("invalid listener address: {error}")) - })?; - socket.bind(&address.into())?; - socket.listen(128)?; - let listener: std::net::TcpListener = socket.into(); - listener.set_nonblocking(true)?; - listeners.push(listener); - } - Ok(listeners) - })(); - let _ = tx.send(result); - }); - rx.await - .map_err(|_| std::io::Error::other("netns bind thread panicked"))?? - .into_iter() - .map(tokio::net::TcpListener::from_std) - .collect() - } - - /// Bind UDP and TCP DNS listeners inside the workload network namespace. - /// The workload keeps its image-provided resolver configuration; nftables - /// redirects port 53 to these sockets before the bypass fence runs. - pub async fn bind_policy_dns_sockets( - &self, - ) -> std::io::Result<(tokio::net::UdpSocket, tokio::net::TcpListener)> { - let ns_fd = self - .ns_fd - .ok_or_else(|| std::io::Error::other("no namespace fd available for bind"))?; - let (tx, rx) = tokio::sync::oneshot::channel(); - std::thread::spawn(move || { - let result = (|| -> std::io::Result<(std::net::UdpSocket, std::net::TcpListener)> { - #[allow(unsafe_code)] - if unsafe { libc::setns(ns_fd, libc::CLONE_NEWNET) } != 0 { - return Err(std::io::Error::last_os_error()); - } - // Bind the exact REDIRECT destination instead of INADDR_ANY. - // For UDP this keeps replies sourced from loopback so - // conntrack can reverse the port/address translation before - // delivering them to libc in nested rootless namespaces. - let address: std::net::SocketAddr = format!("127.0.0.1:{POLICY_DNS_PORT}") - .parse() - .map_err(|error| { - std::io::Error::other(format!("invalid DNS listener address: {error}")) - })?; - - let udp = socket2::Socket::new( - socket2::Domain::IPV4, - socket2::Type::DGRAM, - Some(socket2::Protocol::UDP), - )?; - udp.set_reuse_address(true)?; - udp.bind(&address.into())?; - udp.set_nonblocking(true)?; - - let tcp = socket2::Socket::new( - socket2::Domain::IPV4, - socket2::Type::STREAM, - Some(socket2::Protocol::TCP), - )?; - tcp.set_reuse_address(true)?; - tcp.bind(&address.into())?; - tcp.listen(128)?; - tcp.set_nonblocking(true)?; - - Ok((udp.into(), tcp.into())) - })(); - let _ = tx.send(result); - }); - let (udp, tcp) = rx - .await - .map_err(|_| std::io::Error::other("netns DNS bind thread panicked"))??; - Ok(( - tokio::net::UdpSocket::from_std(udp)?, - tokio::net::TcpListener::from_std(tcp)?, - )) - } - - /// Bind a TCP listener inside this network namespace on a dedicated thread. - /// - /// Spawns a short-lived OS thread that enters the namespace via `setns`, - /// binds a `std::net::TcpListener`, then exits. The listener fd is handed - /// back as a non-blocking `tokio::net::TcpListener`. Using a dedicated - /// thread (not `spawn_blocking`) avoids contaminating the tokio thread - /// pool's namespace state. - /// - /// Returns `Err` if the namespace has no fd, `setns` fails, or bind fails. - pub async fn bind_tcp_in_netns(&self, addr: &str) -> std::io::Result { - let ns_fd = self - .ns_fd - .ok_or_else(|| std::io::Error::other("no namespace fd available for bind"))?; - let addr = addr.to_string(); - let (tx, rx) = tokio::sync::oneshot::channel(); - std::thread::spawn(move || { - let result = (|| -> std::io::Result { - // SAFETY: setns is safe to call; this is a dedicated thread - // that exits after binding. The thread's namespace state does - // not contaminate any thread pool. - #[allow(unsafe_code)] - let rc = unsafe { libc::setns(ns_fd, libc::CLONE_NEWNET) }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - std::net::TcpListener::bind(&addr) - })(); - let _ = tx.send(result); - }); - - let std_listener = rx - .await - .map_err(|_| std::io::Error::other("netns bind thread panicked"))??; - std_listener.set_nonblocking(true)?; - tokio::net::TcpListener::from_std(std_listener) - } -} - -impl Drop for NetworkNamespace { - fn drop(&mut self) { - debug!(namespace = %self.name, "Cleaning up network namespace"); - - // Close the fd if we have one - if let Some(fd) = self.ns_fd.take() { - let _ = nix::unistd::close(fd); - } - - // Delete the host-side veth (this also removes the peer) - if let Err(e) = run_ip(&["link", "delete", &self.veth_host]) { - warn!( - error = %e, - veth = %self.veth_host, - "Failed to delete veth interface" - ); - } - - // Delete the namespace - if let Err(e) = run_ip(&["netns", "delete", &self.name]) { - warn!( - error = %e, - namespace = %self.name, - "Failed to delete network namespace" - ); - } - - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Disabled, "cleaned_up") - .message(format!("Network namespace cleaned up [ns:{}]", self.name)) - .build() - ); - } -} - -/// Create the workload's network namespace and install bypass detection -/// rules. Returns `None` when the policy is not in proxy mode. -/// -/// The namespace is shared infrastructure: the proxy binds to its host-side -/// veth IP and reads /dev/kmsg from inside it for bypass detection, while -/// the workload child and SSH sessions enter it via `setns()`. -/// -/// # Errors -/// -/// Returns an error if proxy mode is requested but the namespace cannot be -/// created (e.g., missing `CAP_NET_ADMIN` / `CAP_SYS_ADMIN` or `iproute2`). -/// Failure to install nftables bypass-detection rules is non-fatal and is -/// reported via OCSF instead. -pub fn create_netns_for_proxy( - policy: &openshell_core::policy::SandboxPolicy, -) -> Result> { - use openshell_core::policy::NetworkMode; - use openshell_ocsf::{ConfigStateChangeBuilder, SeverityId, StateId, StatusId, ocsf_emit}; - - if !matches!(policy.network.mode, NetworkMode::Proxy) { - return Ok(None); - } - match NetworkNamespace::create() { - Ok(ns) => { - let proxy_port = policy - .network - .proxy - .as_ref() - .and_then(|p| p.http_addr) - .map_or(3128, |addr| addr.port()); - if let Err(e) = ns.install_bypass_rules(proxy_port) { - ocsf_emit!( - ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Disabled, "degraded") - .message(format!( - "Failed to install bypass detection rules (non-fatal): {e}" - )) - .build() - ); - } - Ok(Some(ns)) - } - Err(e) => Err(miette::miette!( - "Network namespace creation failed and proxy mode requires isolation. \ - Ensure CAP_NET_ADMIN and CAP_SYS_ADMIN are available and iproute2 is installed. \ - Error: {e}" - )), - } -} - -/// Install pod-network bypass enforcement for Kubernetes sidecar topology. -/// -/// This runs in the current network namespace, not in a per-workload netns. -/// The rules allow loopback and the sidecar proxy UID, then reject direct -/// TCP/UDP egress from other UIDs so traffic must use the sidecar's local -/// proxy. -/// -/// # Errors -/// -/// Returns an error when `nft` is unavailable or the ruleset cannot be loaded. -pub fn install_sidecar_bypass_rules(proxy_uid: u32) -> Result<()> { - match install_sidecar_nft_bypass_rules(proxy_uid) { - Ok(()) => Ok(()), - Err(nft_error) => { - warn!( - error = %nft_error, - "Failed to install nftables sidecar rules; trying iptables-legacy fallback" - ); - install_sidecar_iptables_legacy_bypass_rules(proxy_uid).map_err(|iptables_error| { - miette::miette!( - "sidecar nft ruleset load failed: {nft_error}; sidecar iptables-legacy fallback failed: {iptables_error}" - ) - }) - } - } -} - -fn install_sidecar_nft_bypass_rules(proxy_uid: u32) -> Result<()> { - let nft_cmd = find_nft().ok_or_else(|| { - miette::miette!( - "trusted nft helper not found; sidecar network enforcement requires nftables" - ) - })?; - let log_prefix = Some("openshell:sidecar-bypass:"); - let commands = nft_ruleset::generate_sidecar_bypass_commands(proxy_uid, log_prefix); - run_nft_commands_current_namespace(&nft_cmd, &commands) -} - -const SIDECAR_IPTABLES_CHAIN: &str = "OPENSHELL_SIDECAR_BYPASS"; -const PROC_NET_IF_INET6_PATH: &str = "/proc/net/if_inet6"; - -fn install_sidecar_iptables_legacy_bypass_rules(proxy_uid: u32) -> Result<()> { - let ipv4_filter_tool = find_iptables_legacy().ok_or_else(|| { - miette::miette!( - "trusted iptables-legacy helper not found; sidecar network enforcement fallback unavailable" - ) - })?; - - let ipv6_fence_tool = if current_namespace_has_non_loopback_ipv6()? { - Some(find_ip6tables_legacy().ok_or_else(|| { - miette::miette!( - "trusted ip6tables-legacy helper not found; sidecar network enforcement fallback cannot fence IPv6" - ) - })?) - } else { - warn!( - "Skipping IPv6 sidecar iptables-legacy fallback because the current namespace has no non-loopback IPv6 interface" - ); - None - }; - - cleanup_sidecar_iptables_legacy_rule_families(&ipv4_filter_tool, ipv6_fence_tool.as_deref()); - - if let Err(e) = install_sidecar_iptables_legacy_family_rules( - &ipv4_filter_tool, - proxy_uid, - "icmp-port-unreachable", - ) { - cleanup_sidecar_iptables_legacy_rule_families( - &ipv4_filter_tool, - ipv6_fence_tool.as_deref(), - ); - return Err(e); - } - - if let Some(ipv6_fence_tool) = ipv6_fence_tool - && let Err(e) = install_sidecar_iptables_legacy_family_rules( - &ipv6_fence_tool, - proxy_uid, - "icmp6-port-unreachable", - ) - { - cleanup_sidecar_iptables_legacy_rule_families(&ipv4_filter_tool, Some(&ipv6_fence_tool)); - return Err(e); - } - - Ok(()) -} - -fn current_namespace_has_non_loopback_ipv6() -> Result { - match std::fs::read_to_string(PROC_NET_IF_INET6_PATH) { - Ok(content) => Ok(has_non_loopback_ipv6_interface(&content)), - Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(false), - Err(e) => Err(miette::miette!( - "failed to inspect {PROC_NET_IF_INET6_PATH} before installing sidecar IPv6 fence: {e}" - )), - } -} - -fn has_non_loopback_ipv6_interface(content: &str) -> bool { - content.lines().any(|line| { - line.split_whitespace() - .nth(5) - .is_some_and(|iface| iface != "lo") - }) -} - -fn install_sidecar_iptables_legacy_family_rules( - cmd: &str, - proxy_uid: u32, - udp_reject_with: &str, -) -> Result<()> { - let proxy_uid_arg = proxy_uid.to_string(); - let commands: Vec> = vec![ - vec!["-N", SIDECAR_IPTABLES_CHAIN], - vec!["-A", SIDECAR_IPTABLES_CHAIN, "-o", "lo", "-j", "ACCEPT"], - vec![ - "-A", - SIDECAR_IPTABLES_CHAIN, - "-m", - "conntrack", - "--ctstate", - "ESTABLISHED,RELATED", - "-j", - "ACCEPT", - ], - vec![ - "-A", - SIDECAR_IPTABLES_CHAIN, - "-m", - "owner", - "--uid-owner", - &proxy_uid_arg, - "-j", - "ACCEPT", - ], - vec![ - "-A", - SIDECAR_IPTABLES_CHAIN, - "-p", - "tcp", - "-j", - "REJECT", - "--reject-with", - "tcp-reset", - ], - vec![ - "-A", - SIDECAR_IPTABLES_CHAIN, - "-p", - "udp", - "-j", - "REJECT", - "--reject-with", - udp_reject_with, - ], - vec!["-A", "OUTPUT", "-j", SIDECAR_IPTABLES_CHAIN], - ]; - - for args in commands { - if let Err(e) = run_iptables_legacy_current_namespace(cmd, &args) { - cleanup_sidecar_iptables_legacy_rules(cmd); - return Err(e); - } - } - - Ok(()) -} - -fn cleanup_sidecar_iptables_legacy_rules(iptables_cmd: &str) { - while run_iptables_legacy_current_namespace( - iptables_cmd, - &["-D", "OUTPUT", "-j", SIDECAR_IPTABLES_CHAIN], - ) - .is_ok() - {} - let _ = run_iptables_legacy_current_namespace(iptables_cmd, &["-F", SIDECAR_IPTABLES_CHAIN]); - let _ = run_iptables_legacy_current_namespace(iptables_cmd, &["-X", SIDECAR_IPTABLES_CHAIN]); -} - -fn cleanup_sidecar_iptables_legacy_rule_families(ipv4_cmd: &str, ipv6_cmd: Option<&str>) { - cleanup_sidecar_iptables_legacy_rules(ipv4_cmd); - if let Some(ipv6_cmd) = ipv6_cmd { - cleanup_sidecar_iptables_legacy_rules(ipv6_cmd); - } -} - -/// Run an `ip` command on the host. -fn run_ip(args: &[&str]) -> Result<()> { - let ip_path = find_trusted_binary("ip", IP_SEARCH_PATHS)?; - - debug!(command = %format!("{ip_path} {}", args.join(" ")), "Running ip command"); - - let output = Command::new(ip_path) - .args(args) - .output() - .into_diagnostic()?; - - if !output.status.success() { - let stderr = String::from_utf8_lossy(&output.stderr); - return Err(miette::miette!( - "{ip_path} {} failed: {}", - args.join(" "), - stderr.trim() - )); - } - - Ok(()) -} - -fn run_iptables_legacy_current_namespace(iptables_cmd: &str, args: &[&str]) -> Result<()> { - debug!( - command = %format!("{iptables_cmd} {}", args.join(" ")), - "Running iptables-legacy sidecar command" - ); - - let output = Command::new(iptables_cmd) - .args(args) - .output() - .into_diagnostic()?; - - if !output.status.success() { - let stderr = String::from_utf8_lossy(&output.stderr); - return Err(miette::miette!( - "{iptables_cmd} {} failed: {}", - args.join(" "), - stderr.trim() - )); - } - - Ok(()) -} - -/// Run a sequence of nft commands in the current network namespace. -/// -/// Each command is executed as a separate `nft` invocation to avoid atomic -/// batch rollback (where one unsupported expression like `ct state` or `log` -/// causes the entire transaction, including table creation, to fail). -/// -/// Commands marked as non-required are allowed to fail with a warning. -/// Required commands that fail abort the sequence immediately. -fn run_nft_commands_current_namespace( - nft_cmd: &str, - commands: &[nft_ruleset::NftCommand], -) -> Result<()> { - for cmd in commands { - let args_str = cmd.args.join(" "); - debug!(command = %format!("{nft_cmd} {args_str}"), "Running nft command"); - - let output = Command::new(nft_cmd) - .args(&cmd.args) - .output() - .into_diagnostic()?; - - if !output.status.success() { - let stderr = String::from_utf8_lossy(&output.stderr); - if cmd.required { - return Err(miette::miette!( - "{nft_cmd} {args_str} failed: {}", - stderr.trim() - )); - } - warn!( - command = %args_str, - error = %stderr.trim(), - "non-required nft command failed (continuing)" - ); - } - } - Ok(()) -} - -/// Run an `ip` command inside a network namespace via `nsenter --net=`. -/// -/// We use `nsenter` instead of `ip netns exec` because `ip netns exec` -/// remounts `/sys` to reflect the target namespace's sysfs entries. That -/// sysfs remount requires real `CAP_SYS_ADMIN` in the host user namespace, -/// which is unavailable in rootless container runtimes (e.g. rootless -/// Podman). `nsenter --net=` enters only the network namespace without -/// changing the mount namespace, avoiding the sysfs remount entirely. -/// The supervisor's operations (addr add, link set, route add) are all -/// netlink-based and do not need sysfs access. -fn run_ip_netns(netns: &str, args: &[&str]) -> Result<()> { - run_ip_netns_output(netns, args).map(|_| ()) -} - -fn run_ip_netns_output(netns: &str, args: &[&str]) -> Result { - let ip_path = find_trusted_binary("ip", IP_SEARCH_PATHS)?; - let nsenter_path = find_trusted_binary("nsenter", NSENTER_SEARCH_PATHS)?; - let ns_path = openshell_core::container_paths::netns_path(netns); - let net_flag = format!("--net={}", ns_path.display()); - - let mut full_args = vec![net_flag.as_str(), "--", ip_path]; - full_args.extend(args); - - debug!( - command = %format!("{nsenter_path} {}", full_args.join(" ")), - "Running ip in namespace via nsenter" - ); - - let output = Command::new(nsenter_path) - .args(&full_args) - .output() - .into_diagnostic()?; - - if !output.status.success() { - let stderr = String::from_utf8_lossy(&output.stderr); - return Err(miette::miette!( - "{nsenter_path} --net={} {ip_path} {} failed: {}", - ns_path.display(), - args.join(" "), - stderr.trim() - )); - } - - Ok(String::from_utf8_lossy(&output.stdout).into_owned()) -} - -fn first_route_overlap( - routes: &str, - reserved: &[ipnet::IpNet], -) -> Option<(ipnet::IpNet, ipnet::IpNet)> { - routes.lines().find_map(|line| { - line.split_whitespace().find_map(|token| { - let route = token - .parse::() - .ok() - .or_else(|| token.parse::().ok().map(ipnet::IpNet::from))?; - reserved - .iter() - .copied() - .find(|pool| { - let same_family = route.addr().is_ipv4() == pool.addr().is_ipv4(); - let overlaps = - route.contains(&pool.network()) || pool.contains(&route.network()); - same_family && overlaps - }) - .map(|pool| (route, pool)) - }) - }) -} - -/// Run a sequence of nft commands inside a network namespace via `nsenter --net=`. -/// -/// Each command is executed as a separate invocation to avoid atomic batch -/// rollback. See [`run_nft_commands_current_namespace`] for rationale. -fn run_nft_commands_netns( - netns: &str, - nft_cmd: &str, - commands: &[nft_ruleset::NftCommand], -) -> Result<()> { - let nsenter_path = find_trusted_binary("nsenter", NSENTER_SEARCH_PATHS)?; - let ns_path = openshell_core::container_paths::netns_path(netns); - let net_flag = format!("--net={}", ns_path.display()); - - for cmd in commands { - let args_str = cmd.args.join(" "); - debug!( - command = %format!("{nsenter_path} {net_flag} -- {nft_cmd} {args_str}"), - "Running nft command in namespace" - ); - - let mut full_args = vec![net_flag.as_str(), "--", nft_cmd]; - let arg_refs: Vec<&str> = cmd.args.iter().map(String::as_str).collect(); - full_args.extend(&arg_refs); - - let output = Command::new(nsenter_path) - .args(&full_args) - .output() - .into_diagnostic()?; - - if !output.status.success() { - let stderr = String::from_utf8_lossy(&output.stderr); - if cmd.required { - return Err(miette::miette!( - "nft {args_str} failed in netns {netns}: {}", - stderr.trim() - )); - } - warn!( - command = %args_str, - error = %stderr.trim(), - netns = %netns, - "non-required nft command failed in namespace (continuing)" - ); - } - } - Ok(()) -} - -const NF_LOG_ALL_NETNS_PATH: &str = "/proc/sys/net/netfilter/nf_log_all_netns"; - -/// Enable nftables logging from non-init network namespaces. -/// -/// The kernel's `nf_log_syslog` module silently suppresses log output from -/// non-init network namespaces unless `net.netfilter.nf_log_all_netns` is -/// set to 1. Since sandbox bypass rules live in a per-sandbox network -/// namespace, the bypass monitor can't see log entries without this. -fn enable_nf_log_all_netns() { - use std::path::Path; - if !Path::new(NF_LOG_ALL_NETNS_PATH).exists() { - debug!("nf_log_all_netns sysctl not available (may already be set by init)"); - return; - } - match std::fs::write(NF_LOG_ALL_NETNS_PATH, "1") { - Ok(()) => { - debug!("Enabled nf_log_all_netns for non-init namespace logging"); - } - Err(e) => { - debug!( - error = %e, - "Could not enable nf_log_all_netns; bypass log rules may not produce output" - ); - } - } -} - -/// Well-known paths where nft may be installed. -const NFT_SEARCH_PATHS: &[&str] = &["/usr/sbin/nft", "/sbin/nft", "/usr/bin/nft"]; -const IPTABLES_LEGACY_SEARCH_PATHS: &[&str] = &[ - "/usr/sbin/iptables-legacy", - "/sbin/iptables-legacy", - "/usr/bin/iptables-legacy", -]; -const IP6TABLES_LEGACY_SEARCH_PATHS: &[&str] = &[ - "/usr/sbin/ip6tables-legacy", - "/sbin/ip6tables-legacy", - "/usr/bin/ip6tables-legacy", -]; - -fn find_trusted_binary<'a>(name: &str, paths: &'a [&str]) -> Result<&'a str> { - paths - .iter() - .copied() - .find(|path| { - let path = Path::new(path); - path.is_absolute() && path.is_file() - }) - .ok_or_else(|| { - miette::miette!( - "trusted {name} helper not found; checked {}", - paths.join(", ") - ) - }) -} - -/// Find the nft binary path, checking well-known locations. -fn find_nft() -> Option { - find_trusted_binary("nft", NFT_SEARCH_PATHS) - .ok() - .map(String::from) -} - -fn find_iptables_legacy() -> Option { - find_trusted_binary("iptables-legacy", IPTABLES_LEGACY_SEARCH_PATHS) - .ok() - .map(String::from) -} - -fn find_ip6tables_legacy() -> Option { - find_trusted_binary("ip6tables-legacy", IP6TABLES_LEGACY_SEARCH_PATHS) - .ok() - .map(String::from) -} - -#[cfg(test)] -mod tests { - use super::*; - use std::fs; - - // These tests require root and network namespace support - // Run with: sudo cargo test -- --ignored - - #[test] - fn find_trusted_binary_uses_absolute_existing_file() { - let tempdir = tempfile::tempdir().unwrap(); - let helper = tempdir.path().join("ip"); - fs::write(&helper, b"test helper").unwrap(); - let helper = helper.to_str().unwrap(); - - assert_eq!( - find_trusted_binary("ip", &["relative-ip", "/missing/ip", helper]).unwrap(), - helper - ); - } - - #[test] - fn find_trusted_binary_rejects_missing_helpers() { - let err = - find_trusted_binary("nsenter", &["relative-nsenter", "/missing/nsenter"]).unwrap_err(); - - assert!(err.to_string().contains("trusted nsenter helper not found")); - } - - #[test] - fn nft_search_paths_are_absolute() { - for path in NFT_SEARCH_PATHS { - assert!( - path.starts_with('/'), - "NFT_SEARCH_PATHS entry must be absolute: {path}" - ); - } - } - - #[test] - fn iptables_legacy_search_paths_are_absolute() { - for path in IPTABLES_LEGACY_SEARCH_PATHS { - assert!( - path.starts_with('/'), - "IPTABLES_LEGACY_SEARCH_PATHS entry must be absolute: {path}" - ); - } - } - - #[test] - fn ip6tables_legacy_search_paths_are_absolute() { - for path in IP6TABLES_LEGACY_SEARCH_PATHS { - assert!( - path.starts_with('/'), - "IP6TABLES_LEGACY_SEARCH_PATHS entry must be absolute: {path}" - ); - } - } - - #[test] - fn non_loopback_ipv6_detector_ignores_empty_input() { - assert!(!has_non_loopback_ipv6_interface("")); - assert!(!has_non_loopback_ipv6_interface("\n\n")); - } - - #[test] - fn non_loopback_ipv6_detector_ignores_loopback() { - let content = "00000000000000000000000000000001 01 80 10 80 lo\n"; - - assert!(!has_non_loopback_ipv6_interface(content)); - } - - #[test] - fn non_loopback_ipv6_detector_detects_pod_interface() { - let content = "\ -00000000000000000000000000000001 01 80 10 80 lo -fe800000000000000000000000000001 02 40 20 80 eth0 -"; - - assert!(has_non_loopback_ipv6_interface(content)); - } - - #[test] - fn route_overlap_detects_reserved_pool_collision() { - let reserved = [ - "198.18.1.0/25".parse().unwrap(), - "fd23:6f70:656e:1::/120".parse().unwrap(), - ]; - let routes = "default via 10.200.0.1 dev veth\n198.18.0.0/15 dev eth1\n"; - let (route, pool) = first_route_overlap(routes, &reserved).expect("collision"); - assert_eq!(route.to_string(), "198.18.0.0/15"); - assert_eq!(pool.to_string(), "198.18.1.0/25"); - } - - #[test] - fn route_overlap_ignores_default_and_unrelated_routes() { - let reserved = [ - "198.18.1.0/25".parse().unwrap(), - "fd23:6f70:656e:1::/120".parse().unwrap(), - ]; - let routes = "default via 10.200.0.1 dev veth\n10.200.0.0/24 dev veth\n"; - assert_eq!(first_route_overlap(routes, &reserved), None); - } - - #[test] - #[ignore = "requires root privileges"] - fn test_create_and_drop_namespace() { - let ns = NetworkNamespace::create().expect("Failed to create namespace"); - let name = ns.name().to_string(); - - // Verify namespace exists - let ns_path = openshell_core::container_paths::netns_path(&name); - assert!(ns_path.exists(), "Namespace file should exist"); - - // Verify IPs are set correctly - assert_eq!( - ns.host_ip().to_string(), - format!("{SUBNET_PREFIX}.{HOST_IP_SUFFIX}") - ); - assert_eq!( - ns.sandbox_ip().to_string(), - format!("{SUBNET_PREFIX}.{SANDBOX_IP_SUFFIX}") - ); - - // Drop should clean up - drop(ns); - - // Verify namespace is gone - assert!( - !Path::new(&ns_path).exists(), - "Namespace should be cleaned up" - ); - } -} diff --git a/crates/openshell-supervisor-process/src/netns/nft_ruleset.rs b/crates/openshell-supervisor-process/src/netns/nft_ruleset.rs deleted file mode 100644 index aef95b6068..0000000000 --- a/crates/openshell-supervisor-process/src/netns/nft_ruleset.rs +++ /dev/null @@ -1,825 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! nftables ruleset generation for sandbox network bypass enforcement. -//! -//! This module provides pure functions to generate nftables rulesets that enforce -//! the sandbox network policy: all traffic must go through the proxy, with bypass -//! attempts logged and rejected. -//! -//! Rulesets are returned as a sequence of individual nft commands rather than a -//! monolithic file. Running each command as a separate `nft` invocation avoids -//! `nft -f` atomic batch semantics, where a single unsupported expression (e.g. -//! `ct state` without `nf_conntrack`, `log` without `nf_log`) rolls back the -//! entire transaction including table/chain creation. - -const DNS_DESTINATION_PORT: &str = "53"; - -/// A single nft command with metadata about whether it is required. -pub struct NftCommand { - /// The nft command arguments (e.g. `["add", "table", "inet", "openshell_bypass"]`). - pub args: Vec, - /// When false, failure of this command is non-fatal; the caller should - /// log a warning and continue with the remaining commands. - pub required: bool, -} - -/// Generate nft commands for sandbox network bypass enforcement. -/// -/// Creates an `inet` family table (handles both IPv4 and IPv6) with rules that: -/// 1. Accept traffic to the proxy (IPv4 only) -/// 2. Accept loopback traffic -/// 3. Accept established/related connections (optional; requires `nf_conntrack`) -/// 4. Reject TCP and UDP bypass attempts (both IPv4 and IPv6) -/// -/// If `log_prefix` is provided, log rules are inserted before each reject rule -/// so that bypass attempts are recorded in the kernel ring buffer before being -/// rejected. Log rules are always non-required since they need `nf_log` support. -pub fn generate_bypass_commands( - host_ip: &str, - proxy_port: u16, - log_prefix: Option<&str>, -) -> Vec { - let table = "openshell_bypass"; - let mut cmds = vec![ - nft_cmd(true, &["add", "table", "inet", table]), - nft_cmd(true, &["flush", "table", "inet", table]), - nft_cmd( - true, - &[ - "add", - "chain", - "inet", - table, - "output", - "{ type filter hook output priority 0; policy accept; }", - ], - ), - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "ip", - "daddr", - host_ip, - "tcp", - "dport", - &proxy_port.to_string(), - "accept", - ], - ), - nft_cmd( - true, - &[ - "add", "rule", "inet", table, "output", "oifname", "lo", "accept", - ], - ), - nft_cmd( - false, - &[ - "add", - "rule", - "inet", - table, - "output", - "ct", - "state", - "established,related", - "accept", - ], - ), - ]; - - if let Some(prefix) = log_prefix { - let quoted = nft_quote(prefix); - cmds.push(nft_cmd( - false, - &[ - "add", "rule", "inet", table, "output", "tcp", "flags", "syn", "limit", "rate", - "5/second", "burst", "10", "packets", "log", "prefix", "ed, "flags", "skuid", - ], - )); - } - - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv4", - "meta", - "l4proto", - "tcp", - "reject", - "with", - "icmp", - "type", - "port-unreachable", - ], - )); - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv6", - "meta", - "l4proto", - "tcp", - "reject", - "with", - "icmpv6", - "type", - "port-unreachable", - ], - )); - - if let Some(prefix) = log_prefix { - let quoted = nft_quote(prefix); - cmds.push(nft_cmd( - false, - &[ - "add", "rule", "inet", table, "output", "meta", "l4proto", "udp", "limit", "rate", - "5/second", "burst", "10", "packets", "log", "prefix", "ed, "flags", "skuid", - ], - )); - } - - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv4", - "meta", - "l4proto", - "udp", - "reject", - "with", - "icmp", - "type", - "port-unreachable", - ], - )); - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv6", - "meta", - "l4proto", - "udp", - "reject", - "with", - "icmpv6", - "type", - "port-unreachable", - ], - )); - - cmds -} - -/// Generate the combined policy-DNS, transparent-TCP, and bypass fence. -/// -/// DNS may reach only the supervisor's trusted listener. TCP addressed to the -/// reserved synthetic pools is redirected before the terminal bypass reject; -/// all other direct TCP/UDP retains the existing fast-fail behavior. -pub fn generate_transparent_tcp_commands( - host_ip: &str, - proxy_port: u16, - dns_port: u16, - transparent_port: u16, - synthetic_ipv4_cidr: &str, - synthetic_ipv6_cidr: &str, - log_prefix: Option<&str>, -) -> Vec { - let mut cmds = vec![ - nft_cmd(true, &["add", "table", "inet", "openshell_transparent"]), - nft_cmd(true, &["flush", "table", "inet", "openshell_transparent"]), - nft_cmd( - true, - &[ - "add", - "chain", - "inet", - "openshell_transparent", - "output", - "{ type nat hook output priority dstnat; policy accept; }", - ], - ), - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - "openshell_transparent", - "output", - "meta", - "nfproto", - "ipv4", - "udp", - "dport", - DNS_DESTINATION_PORT, - "redirect", - "to", - &format!(":{dns_port}"), - ], - ), - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - "openshell_transparent", - "output", - "ip", - "daddr", - synthetic_ipv4_cidr, - "tcp", - "dport", - "1-65535", - "redirect", - "to", - &format!(":{transparent_port}"), - ], - ), - // Synthetic destinations must take precedence over the generic TCP - // DNS capture. A policy endpoint may legitimately use TCP port 53; - // that connection belongs to transparent TCP, not the DNS listener. - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - "openshell_transparent", - "output", - "meta", - "nfproto", - "ipv4", - "tcp", - "dport", - DNS_DESTINATION_PORT, - "redirect", - "to", - &format!(":{dns_port}"), - ], - ), - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - "openshell_transparent", - "output", - "ip6", - "daddr", - synthetic_ipv6_cidr, - "tcp", - "dport", - "1-65535", - "redirect", - "to", - &format!(":{transparent_port}"), - ], - ), - ]; - let mut bypass = generate_bypass_commands(host_ip, proxy_port, log_prefix); - // NAT REDIRECT rewrites both DNS and synthetic TCP to loopback before the - // filter hook. Some kernels retain the packet's pre-REDIRECT output - // interface for filter matching, so `oifname lo accept` alone is not - // portable. Admit only packets that the kernel records as DNATed to the - // supervisor listeners. A direct dial to either port has no DNAT status - // and still reaches the terminal bypass reject. Transparent TCP - // authorization after accept remains bound by SO_ORIGINAL_DST plus the - // synthetic-address mapping. - let insertion = bypass - .iter() - .position(|command| { - command.args.iter().any(|arg| arg == "log") - || command.args.iter().any(|arg| arg == "reject") - }) - .unwrap_or(bypass.len()); - bypass.splice( - insertion..insertion, - [ - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - "openshell_bypass", - "output", - "ct", - "status", - "dnat", - "udp", - "dport", - &dns_port.to_string(), - "accept", - ], - ), - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - "openshell_bypass", - "output", - "ct", - "status", - "dnat", - "tcp", - "dport", - &dns_port.to_string(), - "accept", - ], - ), - nft_cmd( - true, - &[ - "add", - "rule", - "inet", - "openshell_bypass", - "output", - "ct", - "status", - "dnat", - "tcp", - "dport", - &transparent_port.to_string(), - "accept", - ], - ), - ], - ); - cmds.extend(bypass); - cmds -} - -/// Generate nft commands for Kubernetes sidecar enforcement. -/// -/// The network sidecar and the process supervisor share a pod network -/// namespace. The sidecar runs as `proxy_uid` and owns external egress; -/// sandbox traffic must use loopback services hosted by that sidecar -/// (gateway forward and HTTP CONNECT proxy). The generated fence rejects -/// TCP/UDP bypass attempts from non-proxy UIDs; other L4 protocols are outside -/// the sidecar policy fence. -pub fn generate_sidecar_bypass_commands( - proxy_uid: u32, - log_prefix: Option<&str>, -) -> Vec { - let table = "openshell_sidecar_bypass"; - let uid_str = proxy_uid.to_string(); - let mut cmds = vec![ - nft_cmd(true, &["add", "table", "inet", table]), - nft_cmd(true, &["flush", "table", "inet", table]), - nft_cmd( - true, - &[ - "add", - "chain", - "inet", - table, - "output", - "{ type filter hook output priority 0; policy accept; }", - ], - ), - nft_cmd( - true, - &[ - "add", "rule", "inet", table, "output", "oifname", "lo", "accept", - ], - ), - nft_cmd( - false, - &[ - "add", - "rule", - "inet", - table, - "output", - "ct", - "state", - "established,related", - "accept", - ], - ), - nft_cmd( - true, - &[ - "add", "rule", "inet", table, "output", "meta", "skuid", &uid_str, "accept", - ], - ), - ]; - - if let Some(prefix) = log_prefix { - let quoted = nft_quote(prefix); - cmds.push(nft_cmd( - false, - &[ - "add", "rule", "inet", table, "output", "tcp", "flags", "syn", "limit", "rate", - "5/second", "burst", "10", "packets", "log", "prefix", "ed, "flags", "skuid", - ], - )); - } - - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv4", - "meta", - "l4proto", - "tcp", - "reject", - "with", - "icmp", - "type", - "port-unreachable", - ], - )); - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv6", - "meta", - "l4proto", - "tcp", - "reject", - "with", - "icmpv6", - "type", - "port-unreachable", - ], - )); - - if let Some(prefix) = log_prefix { - let quoted = nft_quote(prefix); - cmds.push(nft_cmd( - false, - &[ - "add", "rule", "inet", table, "output", "meta", "l4proto", "udp", "limit", "rate", - "5/second", "burst", "10", "packets", "log", "prefix", "ed, "flags", "skuid", - ], - )); - } - - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv4", - "meta", - "l4proto", - "udp", - "reject", - "with", - "icmp", - "type", - "port-unreachable", - ], - )); - cmds.push(nft_cmd( - true, - &[ - "add", - "rule", - "inet", - table, - "output", - "meta", - "nfproto", - "ipv6", - "meta", - "l4proto", - "udp", - "reject", - "with", - "icmpv6", - "type", - "port-unreachable", - ], - )); - - cmds -} - -fn nft_cmd(required: bool, args: &[&str]) -> NftCommand { - NftCommand { - args: args.iter().map(|s| (*s).to_string()).collect(), - required, - } -} - -fn nft_quote(s: &str) -> String { - // nft quoted strings don't support escape sequences; strip any embedded - // double-quotes that would terminate the string early. - format!("\"{}\"", s.replace('"', "")) -} - -#[cfg(test)] -mod tests { - use super::*; - - fn cmd_str(cmd: &NftCommand) -> String { - cmd.args.join(" ") - } - - fn all_strs(cmds: &[NftCommand]) -> String { - cmds.iter().map(cmd_str).collect::>().join("\n") - } - - #[test] - fn generates_bypass_commands_with_proxy_rule() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, None); - let text = all_strs(&cmds); - assert!(text.contains("add table inet openshell_bypass")); - assert!(text.contains("add chain inet openshell_bypass output")); - assert!(text.contains("ip daddr 10.0.2.2 tcp dport 8080 accept")); - } - - #[test] - fn bypass_commands_have_table_and_chain() { - let cmds = generate_bypass_commands("192.168.1.1", 3128, None); - let text = all_strs(&cmds); - assert!(text.contains("add table inet openshell_bypass")); - assert!(text.contains("type filter hook output priority 0; policy accept;")); - } - - #[test] - fn proxy_accept_rule_uses_provided_ip_and_port() { - let cmds = generate_bypass_commands("172.16.0.1", 9999, None); - let text = all_strs(&cmds); - assert!(text.contains("ip daddr 172.16.0.1 tcp dport 9999 accept")); - } - - #[test] - fn rules_are_ordered_accept_then_reject() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, None); - let text = all_strs(&cmds); - let proxy_pos = text.find("ip daddr").unwrap(); - let lo_pos = text.find("oifname lo").unwrap(); - let ct_pos = text.find("ct state established,related").unwrap(); - let reject_pos = text.find("reject with icmp type").unwrap(); - - assert!(proxy_pos < lo_pos); - assert!(lo_pos < ct_pos); - assert!(ct_pos < reject_pos); - } - - #[test] - fn transparent_rules_precede_bypass_rejects_and_scope_dns() { - let commands = generate_transparent_tcp_commands( - "10.200.0.1", - 3128, - 15053, - 15001, - "198.18.0.0/24", - "fd23:6f70:656e::/48", - None, - ); - let text = all_strs(&commands); - assert!(text.contains("meta nfproto ipv4 udp dport 53 redirect to :15053")); - assert!(text.contains("meta nfproto ipv4 tcp dport 53 redirect to :15053")); - assert!(!text.contains("udp dport 53 accept")); - assert!(text.contains("ip daddr 198.18.0.0/24 tcp dport 1-65535 redirect to :15001")); - assert!( - text.contains("ip6 daddr fd23:6f70:656e::/48 tcp dport 1-65535 redirect to :15001") - ); - assert!(!text.contains("meta mark")); - assert!(text.contains("ct status dnat udp dport 15053 accept")); - assert!(text.contains("ct status dnat tcp dport 15053 accept")); - assert!(text.contains("ct status dnat tcp dport 15001 accept")); - for (protocol, port) in [("udp", "15053"), ("tcp", "15053"), ("tcp", "15001")] { - assert!(!commands.iter().any(|command| { - command.args.ends_with(&[ - protocol.to_string(), - "dport".to_string(), - port.to_string(), - "accept".to_string(), - ]) && !command.args.windows(3).any(|window| { - window == ["ct".to_string(), "status".to_string(), "dnat".to_string()] - }) - })); - } - assert!(text.contains("oifname lo accept")); - assert!( - text.find("ct status dnat tcp dport 15053 accept").unwrap() - < text - .find("meta nfproto ipv4 meta l4proto tcp reject") - .unwrap() - ); - assert!( - text.find("ct status dnat udp dport 15053 accept").unwrap() - < text - .find("meta nfproto ipv4 meta l4proto udp reject") - .unwrap() - ); - assert!( - text.find("ct status dnat tcp dport 15001 accept").unwrap() - < text - .find("meta nfproto ipv4 meta l4proto tcp reject") - .unwrap() - ); - assert!( - text.find("ip daddr 198.18.0.0/24 tcp dport 1-65535 redirect to :15001") - .unwrap() - < text - .find("meta nfproto ipv4 meta l4proto tcp reject") - .unwrap() - ); - assert!(!text.contains("meta nfproto ipv6 udp dport 53 redirect")); - assert!( - text.find("ip daddr 198.18.0.0/24 tcp dport 1-65535 redirect to :15001") - .unwrap() - < text - .find("meta nfproto ipv4 tcp dport 53 redirect to :15053") - .unwrap(), - "synthetic TCP:53 must reach transparent TCP before generic DNS capture" - ); - } - - #[test] - fn both_ipv4_and_ipv6_reject_types_are_present() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, None); - let text = all_strs(&cmds); - let icmp_count = text - .matches("reject with icmp type port-unreachable") - .count(); - let icmpv6_count = text - .matches("reject with icmpv6 type port-unreachable") - .count(); - assert_eq!(icmp_count, 2, "need IPv4 ICMP rejects for TCP + UDP"); - assert_eq!(icmpv6_count, 2, "need IPv6 ICMPv6 rejects for TCP + UDP"); - } - - #[test] - fn no_log_commands_omit_log_rules() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, None); - let text = all_strs(&cmds); - assert!( - !text.contains("log prefix"), - "no-log commands must not contain log rules" - ); - } - - #[test] - fn log_commands_contain_prefix_for_tcp_and_udp() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, Some("openshell:bypass:test:")); - let text = all_strs(&cmds); - let count = text - .matches("log prefix \"openshell:bypass:test:\"") - .count(); - assert_eq!(count, 2, "need log rules for both TCP and UDP"); - assert!(text.contains("tcp flags syn limit rate 5/second burst 10 packets")); - assert!(text.contains("meta l4proto udp limit rate 5/second burst 10 packets")); - } - - #[test] - fn log_rules_appear_before_reject_rules() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, Some("openshell:bypass:test:")); - let text = all_strs(&cmds); - let tcp_log_pos = text.find("tcp flags syn").unwrap(); - let tcp_reject_pos = text - .find("meta nfproto ipv4 meta l4proto tcp reject") - .unwrap(); - let udp_log_pos = text.find("meta l4proto udp limit rate").unwrap(); - let udp_reject_pos = text - .find("meta nfproto ipv4 meta l4proto udp reject") - .unwrap(); - - assert!( - tcp_log_pos < tcp_reject_pos, - "TCP log rule must come before TCP reject rule" - ); - assert!( - udp_log_pos < udp_reject_pos, - "UDP log rule must come before UDP reject rule" - ); - } - - #[test] - fn ct_state_rule_is_not_required() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, None); - let ct_cmd = cmds - .iter() - .find(|c| cmd_str(c).contains("ct state")) - .unwrap(); - assert!( - !ct_cmd.required, - "ct state rule should be non-required (needs nf_conntrack)" - ); - } - - #[test] - fn log_rules_are_not_required() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, Some("openshell:bypass:test:")); - for cmd in &cmds { - if cmd_str(cmd).contains("log prefix") { - assert!( - !cmd.required, - "log rules should be non-required (needs nf_log)" - ); - } - } - } - - #[test] - fn sidecar_commands_allow_supervisor_uid_and_loopback() { - let cmds = generate_sidecar_bypass_commands(1337, None); - let text = all_strs(&cmds); - assert!(text.contains("add table inet openshell_sidecar_bypass")); - assert!(text.contains("oifname lo accept")); - assert!(text.contains("meta skuid 1337 accept")); - } - - #[test] - fn sidecar_commands_reject_tcp_and_udp_egress() { - let cmds = generate_sidecar_bypass_commands(0, Some("openshell:sidecar:test:")); - let text = all_strs(&cmds); - assert!(text.contains("meta nfproto ipv4 meta l4proto tcp reject")); - assert!(text.contains("meta nfproto ipv6 meta l4proto tcp reject")); - assert!(text.contains("meta nfproto ipv4 meta l4proto udp reject")); - assert!(text.contains("meta nfproto ipv6 meta l4proto udp reject")); - assert_eq!( - text.matches("log prefix \"openshell:sidecar:test:\"") - .count(), - 2 - ); - } - - #[test] - fn log_prefix_is_quoted_as_nft_string_literal() { - let cmds = generate_bypass_commands("10.0.2.2", 8080, Some("openshell:bypass:test:")); - for cmd in &cmds { - let s = cmd_str(cmd); - if let Some(idx) = s.find("log prefix ") { - let after_prefix = &s[idx + "log prefix ".len()..]; - assert!( - after_prefix.starts_with('"'), - "log prefix value must be an nft-quoted string, got: {after_prefix}" - ); - } - } - } - - #[test] - fn nft_quote_wraps_in_double_quotes() { - assert_eq!(nft_quote("simple"), "\"simple\""); - assert_eq!(nft_quote("has:colons:"), "\"has:colons:\""); - assert_eq!(nft_quote("has\"quote"), "\"hasquote\""); - assert_eq!(nft_quote("has\\backslash"), "\"has\\backslash\""); - } -} diff --git a/crates/openshell-supervisor-process/src/run.rs b/crates/openshell-supervisor-process/src/run.rs deleted file mode 100644 index ef40fbd760..0000000000 --- a/crates/openshell-supervisor-process/src/run.rs +++ /dev/null @@ -1,830 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Workload supervision entry point. -//! -//! Spawns the SSH server, optional supervisor session, the entrypoint child -//! process, and waits for it to exit (with optional timeout). Long-running -//! background tasks that aren't strictly tied to the workload's lifetime -//! (policy poll loop, denial aggregator, symlink resolver) live in the -//! orchestrator, not here. - -use miette::{IntoDiagnostic, Result}; -use std::sync::Arc; -use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; -use std::time::Duration; -use tokio::time::timeout; -use tracing::info; - -use openshell_ocsf::{ - ActionId, ActivityId, AppLifecycleBuilder, DispositionId, LaunchTypeId, Process as OcsfProcess, - ProcessActivityBuilder, SeverityId, StatusId, ocsf_emit, -}; - -#[cfg(target_os = "linux")] -use crate::netns::NetworkNamespace; -use openshell_core::policy::{NetworkMode, SandboxPolicy}; -use openshell_core::proposals::AgentProposals; -use openshell_core::provider_credentials::ProviderCredentialState; - -#[cfg(target_os = "linux")] -use openshell_core::activity::ActivitySender; -#[cfg(target_os = "linux")] -use openshell_core::denial::DenialEvent; - -#[cfg(target_os = "linux")] -use crate::managed_children; -use crate::process::{ - ProcessEnforcementMode, ProcessHandle, ProcessStatus, ResolvedProcessIdentity, - ResolvedWorkspace, -}; - -pub enum SidecarExitReport { - Exited { - instance_id: String, - exit_code: i32, - ack: tokio::sync::oneshot::Sender>, - }, - Finalized { - instance_id: String, - ack: tokio::sync::oneshot::Sender>, - }, -} - -fn ocsf_ctx() -> &'static openshell_ocsf::EventContext { - openshell_ocsf::ctx::ctx() -} - -/// Spawn the workload entrypoint, wire up SSH and supervisor session, and -/// wait for the entrypoint child to exit. -/// -/// # Errors -/// -/// Returns an error if SSH server startup fails, if the entrypoint child -/// fails to spawn, or if waiting for the child returns an OS error. -#[allow(clippy::too_many_arguments, clippy::implicit_hasher)] -pub async fn run_process( - program: &str, - args: &[String], - workspace: ResolvedWorkspace, - timeout_secs: u64, - interactive: bool, - await_main_process_attachment: bool, - sandbox_id: Option<&str>, - openshell_endpoint: Option<&str>, - ssh_socket_path: Option, - shared_ssh_socket: bool, - ssh_exit_tx: Option>, - policy: &SandboxPolicy, - resolved_process_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - entrypoint_pid: Arc, - entrypoint_started_tx: Option>, - sidecar_exit_tx: Option>, - provider_credentials: ProviderCredentialState, - provider_env: std::collections::HashMap, - ca_file_paths: Option<(std::path::PathBuf, std::path::PathBuf)>, - agent_proposals: AgentProposals, - #[cfg(target_os = "linux")] netns: Option<&NetworkNamespace>, - #[cfg(target_os = "linux")] bypass_denial_tx: Option< - tokio::sync::mpsc::UnboundedSender, - >, - #[cfg(target_os = "linux")] bypass_activity_tx: Option, -) -> Result { - // Platform drivers with a resolved numeric UID/GID retain the legacy - // account-file update. OCI-image identity leaves those environment values - // empty, so the image's account files remain unchanged. - #[cfg(unix)] - if enforcement_mode.uses_privileged_process_setup() { - crate::process::update_sandbox_passwd_entries()?; - } - - // Validate the completed process identity before exposing a child. - #[cfg(unix)] - if enforcement_mode.uses_privileged_process_setup() { - crate::process::validate_sandbox_user_with_identity(policy, resolved_process_identity)?; - crate::process::validate_sandbox_group_with_identity(policy, resolved_process_identity)?; - } - - // Create read_write directories and chown newly-created ones to the - // sandbox user/group. Runs as the supervisor (root) before the child - // is forked so the workload sees writable paths it owns. - #[cfg(unix)] - if enforcement_mode.uses_privileged_process_setup() { - crate::process::prepare_filesystem_with_identity( - policy, - resolved_process_identity, - workspace.root(), - workspace.home().is_some(), - )?; - } - - // Eagerly fetch initial settings and install the agent skill if the - // proposals flag is on at startup, rather than waiting for the policy - // poll loop's first tick. In offline/file-mode there is no gateway, so - // the flag stays at its default (false) and no skill is installed. - install_initial_agent_skill(sandbox_id, openshell_endpoint, &agent_proposals).await; - - // Provider token grants may mount supervisor-only identity sockets such as - // the SPIFFE Workload API. Prepare the child mount namespace that hides - // those mounts before supervisor seccomp hardening removes the needed - // namespace syscalls. - #[cfg(target_os = "linux")] - crate::process::prepare_supervisor_identity_mount_namespace_from_env()?; - - // Install the supervisor seccomp prelude before spawning any workload-side - // tasks. By this point the orchestrator has finished privileged startup - // helpers (network namespace setup, identity mount namespace setup, - // nftables probes via run_networking), and the SSH listener and entrypoint - // child have not been exposed yet. - crate::sandbox::apply_supervisor_startup_hardening()?; - - // Spawn the bypass detection monitor. It tails dmesg for nftables LOG - // entries fired by rules installed on the workload's network namespace - // and reports direct connection attempts that would have bypassed the - // proxy. Spawn it before the entrypoint child so the first packets are - // not missed. Best-effort: returns None when dmesg is unavailable. - #[cfg(target_os = "linux")] - let _bypass_handle = netns.and_then(|ns| { - crate::bypass_monitor::spawn( - ns.name().to_string(), - entrypoint_pid.clone(), - bypass_denial_tx, - bypass_activity_tx, - ) - }); - - // Verify the runtime PID limit can accommodate the policy's pid_max. - #[cfg(target_os = "linux")] - { - let pid_limit_mode = if std::env::var_os("OPENSHELL_REQUIRE_RUNTIME_PID_LIMIT").is_some() { - crate::process::RuntimePidLimitMode::Require - } else { - crate::process::RuntimePidLimitMode::Warn - }; - crate::process::check_runtime_pid_limit(pid_limit_mode)?; - } - - // Zombie reaper — openshell-sandbox may run as PID 1 in containers and - // must reap orphaned grandchildren (e.g. background daemons started by - // coding agents) to prevent zombie accumulation. - // - // Use waitid(..., WNOWAIT) so we can inspect exited children before - // actually reaping them. This avoids racing explicit `child.wait()` calls - // for managed children (entrypoint and SSH session processes). - #[cfg(target_os = "linux")] - tokio::spawn(async { - use nix::sys::wait::{Id, WaitPidFlag, WaitStatus, waitid, waitpid}; - use tokio::signal::unix::{SignalKind, signal}; - use tokio::time::MissedTickBehavior; - - let mut sigchld = match signal(SignalKind::child()) { - Ok(s) => s, - Err(e) => { - tracing::warn!(error = %e, "Failed to register SIGCHLD handler for zombie reaping"); - return; - } - }; - let mut retry = tokio::time::interval(Duration::from_secs(5)); - retry.set_missed_tick_behavior(MissedTickBehavior::Skip); - - loop { - tokio::select! { - _ = sigchld.recv() => {} - _ = retry.tick() => {} - } - - loop { - let status = match waitid( - Id::All, - WaitPidFlag::WEXITED | WaitPidFlag::WNOHANG | WaitPidFlag::WNOWAIT, - ) { - Ok(WaitStatus::StillAlive) | Err(nix::errno::Errno::ECHILD) => break, - Ok(status) => status, - Err(nix::errno::Errno::EINTR) => continue, - Err(e) => { - tracing::debug!(error = %e, "waitid error during zombie reaping"); - break; - } - }; - - let Some(pid) = status.pid() else { - break; - }; - - if managed_children::is_managed(pid.as_raw()) { - // Let the explicit waiter own this child status. - break; - } - - match waitpid(pid, Some(WaitPidFlag::WNOHANG)) { - Ok(WaitStatus::StillAlive) - | Err(nix::errno::Errno::ECHILD | nix::errno::Errno::EINTR) => {} - Ok(reaped) => { - tracing::debug!(?reaped, "Reaped orphaned child process"); - } - Err(e) => { - tracing::debug!(error = %e, "waitpid error during orphan reap"); - break; - } - } - } - } - }); - - // Hard network policy enforcement for SSH sessions and the persistent - // supervisor session: each session's pre-exec hook calls setns(fd, - // CLONE_NEWNET) so it lands inside the workload's network namespace. - // Without this, SSH-spawned shells run in the host namespace and bypass - // the proxy entirely. - #[cfg(target_os = "linux")] - let ssh_netns_fd = netns.and_then(NetworkNamespace::ns_fd); - #[cfg(not(target_os = "linux"))] - let ssh_netns_fd: Option = None; - - #[cfg(target_os = "linux")] - let mut handle = ProcessHandle::spawn( - program, - args, - &workspace, - interactive, - policy, - resolved_process_identity, - enforcement_mode, - netns, - ca_file_paths.as_ref(), - &provider_env, - )?; - - #[cfg(not(target_os = "linux"))] - let mut handle = ProcessHandle::spawn( - program, - args, - &workspace, - interactive, - policy, - resolved_process_identity, - enforcement_mode, - ca_file_paths.as_ref(), - &provider_env, - )?; - - let main_pid = handle.pid(); - let main_session = crate::main_session::MainSession::new(handle.take_io(), main_pid); - let main_instance_id = uuid::Uuid::new_v4().to_string(); - - // SSH-spawned shells get http_proxy=http://: exported into - // their env so cooperative tools (curl, npm, Node) route through the - // CONNECT proxy. Linux uses the netns host_ip; on other targets fall back - // to the policy-declared http_addr directly. - #[cfg(target_os = "linux")] - let ssh_proxy_url = ssh_proxy_url_for_policy(policy, netns.map(NetworkNamespace::host_ip)); - #[cfg(not(target_os = "linux"))] - let ssh_proxy_url = ssh_proxy_url_for_policy(policy, None); - - let ssh_socket_path: Option = ssh_socket_path.map(std::path::PathBuf::from); - if let Some(listen_path) = ssh_socket_path.clone() { - let policy_clone = policy.clone(); - let workspace_clone = workspace.clone(); - let proxy_url = ssh_proxy_url; - let netns_fd = ssh_netns_fd; - let ca_paths = ca_file_paths.clone(); - let provider_credentials_clone = provider_credentials.clone(); - let main_session_clone = Arc::clone(&main_session); - let user_env_clone: std::collections::HashMap = - std::env::var(openshell_core::sandbox_env::USER_ENVIRONMENT) - .ok() - .and_then(|json| serde_json::from_str(&json).ok()) - .unwrap_or_default(); - - let (ssh_ready_tx, ssh_ready_rx) = tokio::sync::oneshot::channel(); - - tokio::spawn(async move { - let _ssh_exit_guard = ssh_exit_tx; - if let Err(err) = crate::ssh::run_ssh_server( - listen_path, - ssh_ready_tx, - policy_clone, - workspace_clone, - netns_fd, - proxy_url, - ca_paths, - provider_credentials_clone, - user_env_clone, - resolved_process_identity, - enforcement_mode, - shared_ssh_socket, - main_session_clone, - ) - .await - { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Critical) - .status(StatusId::Failure) - .message(format!("SSH server failed: {err}")) - .build() - ); - } - }); - - // Wait for the SSH server to bind before advertising its relay. The - // main process is already supervised; MainSession retains any output - // produced while this endpoint is being prepared. - match timeout(Duration::from_secs(10), ssh_ready_rx).await { - Ok(Ok(Ok(()))) => { - ocsf_emit!( - AppLifecycleBuilder::new(ocsf_ctx()) - .activity(ActivityId::Open) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .message("SSH server is ready to accept connections") - .build() - ); - } - Ok(Ok(Err(err))) => { - return Err(err.context("SSH server failed during startup")); - } - Ok(Err(_)) => { - return Err(miette::miette!( - "SSH server task panicked before signaling ready" - )); - } - Err(_) => { - return Err(miette::miette!( - "SSH server did not start within 10 seconds" - )); - } - } - } - - let supervisor_terminating = Arc::new(AtomicBool::new(false)); - // A canonical process may have completed while the SSH socket was being - // prepared. Detect that exit before entering the main wait path. - let early_exit = handle.try_wait().into_diagnostic()?; - - // Spawn the persistent supervisor session if we have a gateway endpoint - // and sandbox identity. The session provides relay channels for SSH - // connect and ExecSandbox through the gateway. - let supervisor_session_task = if let (Some(endpoint), Some(id), Some(socket)) = - (openshell_endpoint, sandbox_id, ssh_socket_path.as_ref()) - { - let task = crate::supervisor_session::spawn( - endpoint.to_string(), - id.to_string(), - socket.clone(), - ssh_netns_fd, - None, - Arc::clone(&supervisor_terminating), - main_instance_id.clone(), - ); - info!("supervisor session task spawned"); - Some(task) - } else { - None - }; - - // Store the entrypoint PID so the proxy can resolve TCP peer identity - entrypoint_pid.store(handle.pid(), Ordering::Release); - if let Some(tx) = entrypoint_started_tx { - let _ = tx.send((handle.pid(), main_instance_id.clone())); - } - ocsf_emit!( - ProcessActivityBuilder::new(ocsf_ctx()) - .activity(ActivityId::Open) - .action(ActionId::Allowed) - .disposition(DispositionId::Allowed) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .launch_type(LaunchTypeId::Spawn) - .process(OcsfProcess::new(program, i64::from(handle.pid()))) - .message(format!("Process started: pid={}", handle.pid())) - .build() - ); - - let outcome = if let Some(status) = early_exit { - ProcessWaitOutcome::Exited(status) - } else { - wait_for_process_exit_or_shutdown(&mut handle, timeout_secs, &supervisor_terminating) - .await? - }; - - let (rendered_code, drain_terminal) = match outcome { - ProcessWaitOutcome::Exited(status) => (status.code(), true), - ProcessWaitOutcome::TimedOut => { - ocsf_emit!( - ProcessActivityBuilder::new(ocsf_ctx()) - .activity(ActivityId::Close) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Critical) - .status(StatusId::Failure) - .message("Process timed out, killing") - .build() - ); - (124, false) - } - ProcessWaitOutcome::ShutdownSignal { signal, status } => { - info!( - signal, - exit_code = status.code(), - "Entrypoint exited after supervisor shutdown signal" - ); - (status.code(), false) - } - }; - let terminal_delivery_pending = main_session - .finish( - rendered_code, - drain_terminal && await_main_process_attachment, - ) - .await; - - ocsf_emit!( - ProcessActivityBuilder::new(ocsf_ctx()) - .activity(ActivityId::Close) - .action(ActionId::Allowed) - .disposition(DispositionId::Allowed) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .exit_code(rendered_code) - .message(format!("Process exited with code {rendered_code}")) - .build() - ); - - if outcome.should_report_main_process_exit() { - if let Some(tx) = sidecar_exit_tx.as_ref() { - report_sidecar_main_process_exit(tx, &main_instance_id, rendered_code).await?; - } else if let (Some(endpoint), Some(id)) = (openshell_endpoint, sandbox_id) { - report_main_process_exit_until_ack(endpoint, id, &main_instance_id, rendered_code) - .await; - info!(instance_id = %main_instance_id, "main-process exit acknowledged"); - } - } else { - info!( - instance_id = %main_instance_id, - "skipping main-process exit report during supervisor shutdown" - ); - } - main_session.mark_terminal_reported(); - if outcome.should_report_main_process_exit() && drain_terminal && terminal_delivery_pending { - // The peer's SSH channel-close confirms that the terminal frames sent - // above traversed russh and the relay. Detached commands have no active - // attachment and never enter this wait. - main_session.wait_for_terminal_attachments().await; - } - if outcome.should_report_main_process_exit() { - if let Some(tx) = sidecar_exit_tx.as_ref() { - finalize_sidecar_main_process_exit(tx, &main_instance_id).await?; - } else if let (Some(endpoint), Some(id)) = (openshell_endpoint, sandbox_id) { - finalize_main_process_exit_until_ack(endpoint, id, &main_instance_id).await; - info!(instance_id = %main_instance_id, "main-process terminal delivery finalized"); - } - } - - supervisor_terminating.store(true, Ordering::Release); - if let Some(task) = supervisor_session_task { - task.abort(); - } - - Ok(rendered_code) -} - -async fn report_main_process_exit_until_ack( - endpoint: &str, - sandbox_id: &str, - instance_id: &str, - exit_code: i32, -) { - let mut retry_delay = Duration::from_millis(250); - loop { - match crate::supervisor_session::report_main_process_exit( - endpoint, - sandbox_id, - instance_id, - exit_code, - ) - .await - { - Ok(()) => return, - Err(error) => { - tracing::warn!(%error, "main-process exit report failed; retrying"); - tokio::time::sleep(retry_delay).await; - retry_delay = (retry_delay * 2).min(Duration::from_secs(2)); - } - } - } -} - -async fn finalize_main_process_exit_until_ack(endpoint: &str, sandbox_id: &str, instance_id: &str) { - let mut retry_delay = Duration::from_millis(250); - loop { - match crate::supervisor_session::finalize_main_process_exit( - endpoint, - sandbox_id, - instance_id, - ) - .await - { - Ok(()) => return, - Err(error) => { - tracing::warn!(%error, "main-process terminal finalization failed; retrying"); - tokio::time::sleep(retry_delay).await; - retry_delay = (retry_delay * 2).min(Duration::from_secs(2)); - } - } - } -} - -async fn report_sidecar_main_process_exit( - tx: &tokio::sync::mpsc::Sender, - instance_id: &str, - exit_code: i32, -) -> Result<()> { - let (ack_tx, ack_rx) = tokio::sync::oneshot::channel(); - tx.send(SidecarExitReport::Exited { - instance_id: instance_id.to_string(), - exit_code, - ack: ack_tx, - }) - .await - .map_err(|_| miette::miette!("sidecar exit reporter closed"))?; - ack_rx - .await - .map_err(|_| miette::miette!("sidecar exit reporter dropped acknowledgement"))? - .map_err(|error| miette::miette!(error)) -} - -async fn finalize_sidecar_main_process_exit( - tx: &tokio::sync::mpsc::Sender, - instance_id: &str, -) -> Result<()> { - let (ack_tx, ack_rx) = tokio::sync::oneshot::channel(); - tx.send(SidecarExitReport::Finalized { - instance_id: instance_id.to_string(), - ack: ack_tx, - }) - .await - .map_err(|_| miette::miette!("sidecar exit reporter closed"))?; - ack_rx - .await - .map_err(|_| miette::miette!("sidecar exit reporter dropped acknowledgement"))? - .map_err(|error| miette::miette!(error)) -} - -enum ProcessWaitOutcome { - Exited(ProcessStatus), - TimedOut, - ShutdownSignal { - signal: &'static str, - status: ProcessStatus, - }, -} - -impl ProcessWaitOutcome { - /// A gateway acknowledgement is required for ordinary canonical-process - /// completion, but cannot be awaited after the supervisor itself has been - /// asked to terminate. At that point the gateway may already be shutting - /// down and no longer able to acknowledge the report. - fn should_report_main_process_exit(&self) -> bool { - !matches!(self, Self::ShutdownSignal { .. }) - } -} - -async fn wait_for_process_exit_or_shutdown( - handle: &mut ProcessHandle, - timeout_secs: u64, - terminating: &AtomicBool, -) -> Result { - let pid = handle.pid(); - let wait = handle.wait(); - tokio::pin!(wait); - - if timeout_secs > 0 { - let deadline = tokio::time::sleep(Duration::from_secs(timeout_secs)); - tokio::pin!(deadline); - tokio::select! { - result = &mut wait => { - Ok(ProcessWaitOutcome::Exited(result.into_diagnostic()?)) - } - () = &mut deadline => { - terminating.store(true, Ordering::Release); - terminate_then_kill_pid(pid).await; - Ok(ProcessWaitOutcome::TimedOut) - } - signal = wait_for_supervisor_shutdown_signal() => { - terminating.store(true, Ordering::Release); - signal_entrypoint_for_shutdown(pid, signal); - let status = (&mut wait).await.into_diagnostic()?; - Ok(ProcessWaitOutcome::ShutdownSignal { signal, status }) - } - } - } else { - tokio::select! { - result = &mut wait => { - Ok(ProcessWaitOutcome::Exited(result.into_diagnostic()?)) - } - signal = wait_for_supervisor_shutdown_signal() => { - terminating.store(true, Ordering::Release); - signal_entrypoint_for_shutdown(pid, signal); - let status = (&mut wait).await.into_diagnostic()?; - Ok(ProcessWaitOutcome::ShutdownSignal { signal, status }) - } - } - } -} - -#[cfg(unix)] -async fn terminate_then_kill_pid(pid: u32) { - signal_pid(pid, nix::sys::signal::Signal::SIGTERM, "process timeout"); - tokio::time::sleep(Duration::from_millis(100)).await; - signal_pid(pid, nix::sys::signal::Signal::SIGKILL, "process timeout"); -} - -#[cfg(not(unix))] -async fn terminate_then_kill_pid(_pid: u32) {} - -#[cfg(unix)] -fn signal_entrypoint_for_shutdown(pid: u32, signal: &'static str) { - signal_pid(pid, nix::sys::signal::Signal::SIGTERM, signal); -} - -#[cfg(not(unix))] -fn signal_entrypoint_for_shutdown(_pid: u32, _signal: &'static str) {} - -#[cfg(unix)] -fn signal_pid(pid: u32, signal: nix::sys::signal::Signal, reason: &'static str) { - let raw_pid = i32::try_from(pid).unwrap_or(i32::MAX); - if let Err(error) = nix::sys::signal::kill(nix::unistd::Pid::from_raw(-raw_pid), signal) { - tracing::warn!( - pid, - signal = ?signal, - reason, - error = %error, - "failed to signal entrypoint process group" - ); - } -} - -#[cfg(unix)] -async fn wait_for_supervisor_shutdown_signal() -> &'static str { - use tokio::signal::unix::{SignalKind, signal}; - - let mut sigterm = match signal(SignalKind::terminate()) { - Ok(signal) => signal, - Err(error) => { - tracing::warn!( - error = %error, - "Failed to install SIGTERM handler; supervisor shutdown detection disabled" - ); - return std::future::pending::<&'static str>().await; - } - }; - - let _ = sigterm.recv().await; - info!("Received SIGTERM, shutting down supervisor process"); - "SIGTERM" -} - -#[cfg(not(unix))] -async fn wait_for_supervisor_shutdown_signal() -> &'static str { - std::future::pending::<&'static str>().await -} - -fn ssh_proxy_url_for_policy( - policy: &SandboxPolicy, - netns_proxy_host: Option, -) -> Option { - if !matches!(policy.network.mode, NetworkMode::Proxy) { - return None; - } - - let proxy = policy.network.proxy.as_ref()?; - if let Some(host) = netns_proxy_host { - let port = proxy.http_addr.map_or(3128, |addr| addr.port()); - return Some(format!("http://{host}:{port}")); - } - - proxy.http_addr.map(|addr| format!("http://{addr}")) -} - -/// Eagerly fetch initial settings and install the agent-driven policy -/// proposal skill if the flag is on at startup. -/// -/// Without this, the skill would only get installed on the policy poll -/// loop's first false→true transition, which can be ~10 s after launch — -/// long enough for an agent to start running without seeing it. -/// -/// Best-effort: any failure (no gateway, RPC error, install failure) is -/// logged but does not fail sandbox startup. -async fn install_initial_agent_skill( - sandbox_id: Option<&str>, - openshell_endpoint: Option<&str>, - agent_proposals: &AgentProposals, -) { - use openshell_core::proto::setting_value; - - if let (Some(id), Some(endpoint)) = (sandbox_id, openshell_endpoint) - && let Ok(client) = - openshell_core::grpc_client::CachedOpenShellClient::connect(endpoint).await - && let Ok(result) = client.poll_settings(id).await - { - let initial = result - .settings - .get(openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY) - .and_then(|es| es.value.as_ref()) - .and_then(|sv| sv.value.as_ref()) - .and_then(|v| match v { - setting_value::Value::BoolValue(b) => Some(*b), - _ => None, - }) - .unwrap_or(false); - agent_proposals.set_enabled(initial); - } - - if agent_proposals.enabled() { - match crate::skills::install_static_skills() { - Ok(installed) => info!( - path = %installed.policy_advisor.display(), - "Installed sandbox agent skill" - ), - Err(error) => tracing::warn!( - error = %error, - "Failed to install sandbox agent skill" - ), - } - } else { - tracing::debug!( - "agent_policy_proposals_enabled is false at startup; skipping skill install" - ); - } -} - -#[cfg(test)] -mod tests { - use super::*; - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkMode, NetworkPolicy, ProcessPolicy, ProxyPolicy, - }; - - fn policy(mode: NetworkMode, http_addr: Option) -> SandboxPolicy { - SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy { - mode, - proxy: http_addr.map(|http_addr| ProxyPolicy { - http_addr: Some(http_addr), - }), - }, - landlock: LandlockPolicy::default(), - process: ProcessPolicy::default(), - } - } - - #[test] - fn ssh_proxy_url_uses_policy_addr_without_netns() { - let policy = policy(NetworkMode::Proxy, Some(([127, 0, 0, 1], 3128).into())); - - assert_eq!( - ssh_proxy_url_for_policy(&policy, None).as_deref(), - Some("http://127.0.0.1:3128") - ); - } - - #[test] - fn ssh_proxy_url_prefers_netns_host_with_policy_port() { - let policy = policy(NetworkMode::Proxy, Some(([127, 0, 0, 1], 8080).into())); - - assert_eq!( - ssh_proxy_url_for_policy(&policy, Some([10, 200, 0, 1].into())).as_deref(), - Some("http://10.200.0.1:8080") - ); - } - - #[test] - fn ssh_proxy_url_skips_non_proxy_mode() { - let policy = policy(NetworkMode::Allow, Some(([127, 0, 0, 1], 3128).into())); - - assert_eq!(ssh_proxy_url_for_policy(&policy, None), None); - } - - #[cfg(unix)] - #[test] - fn supervisor_shutdown_exit_skips_gateway_acknowledgement() { - use std::os::unix::process::ExitStatusExt; - - let status = ProcessStatus::from(std::process::ExitStatus::from_raw(libc::SIGTERM)); - - assert!(ProcessWaitOutcome::Exited(status).should_report_main_process_exit()); - assert!(ProcessWaitOutcome::TimedOut.should_report_main_process_exit()); - assert!( - !ProcessWaitOutcome::ShutdownSignal { - signal: "SIGTERM", - status, - } - .should_report_main_process_exit() - ); - } -} diff --git a/crates/openshell-supervisor-process/src/ssh.rs b/crates/openshell-supervisor-process/src/ssh.rs index fbd6d9275b..c93f361e24 100644 --- a/crates/openshell-supervisor-process/src/ssh.rs +++ b/crates/openshell-supervisor-process/src/ssh.rs @@ -3,24 +3,11 @@ //! Embedded SSH server for sandbox access. -use crate::child_env; use crate::main_session::{MainOutput, MainSession}; -#[cfg(target_os = "linux")] -use crate::managed_children; -use crate::process::{ - ProcessEnforcementMode, ResolvedProcessIdentity, ResolvedWorkspace, - drop_privileges_with_identity, is_supervisor_only_env_var, session_user_and_home, -}; -use crate::sandbox; #[cfg(unix)] use libc; use miette::{IntoDiagnostic, Result}; -use nix::pty::{Winsize, openpty}; -use nix::unistd::setsid; use openshell_core::VERSION; -use openshell_core::net::set_tcp_nodelay_best_effort; -use openshell_core::policy::SandboxPolicy; -use openshell_core::provider_credentials::ProviderCredentialState; use openshell_ocsf::{ ActionId, ActivityId, DispositionId, SeverityId, SshActivityBuilder, StatusId, ocsf_emit, }; @@ -29,16 +16,35 @@ use russh::server::{Auth, ChannelOpenHandle, Handle, Session}; use russh::{ChannelId, ChannelOpenFailure, Sig}; use std::borrow::Cow; use std::collections::HashMap; -use std::io::{Read, Write}; -use std::os::fd::{AsRawFd, RawFd}; use std::path::{Path, PathBuf}; -use std::process::{Command, Stdio}; use std::sync::{Arc, mpsc}; use std::time::Duration; use tokio::net::UnixListener; use tracing::warn; const NO_LOGIN_SHELL_ENV: (&str, &str) = ("OPENSHELL_NO_LOGIN_SHELL", "1"); +const MAIN_DETACH_PREFIX: u8 = 0x10; +const MAIN_DETACH_KEY: u8 = 0x11; + +fn filter_main_detach_sequence(prefix_pending: &mut bool, data: &[u8]) -> (Vec, bool) { + let mut forward = Vec::with_capacity(data.len() + usize::from(*prefix_pending)); + for &byte in data { + if *prefix_pending { + if byte == MAIN_DETACH_KEY { + *prefix_pending = false; + return (forward, true); + } + forward.push(MAIN_DETACH_PREFIX); + *prefix_pending = false; + } + if byte == MAIN_DETACH_PREFIX { + *prefix_pending = true; + } else { + forward.push(byte); + } + } + (forward, false) +} /// Perform SSH server initialization: generate a host key, build the config, /// and bind the Unix socket listener. Extracted so that startup errors can be @@ -52,13 +58,11 @@ type SshServerInit = ( fn ssh_server_init( listen_path: &Path, ca_file_paths: &Option<(PathBuf, PathBuf)>, - enforcement_mode: ProcessEnforcementMode, shared_socket: bool, ) -> Result { let mut rng = rand::rng(); let host_key = PrivateKey::random(&mut rng, Algorithm::Ed25519).into_diagnostic()?; - // TODO: while building the SSH config, refactor the server_id to be "SSH-2.0-OpenShell_" from `openshell_core::VERSION` let mut config = russh::server::Config { server_id: russh::SshId::Standard(Cow::Owned(format!("SSH-2.0-OpenShell_{VERSION}"))), auth_rejection_time: Duration::from_secs(1), @@ -69,17 +73,14 @@ fn ssh_server_init( let config = Arc::new(config); let ca_paths = ca_file_paths.as_ref().map(|p| Arc::new(p.clone())); - // In full enforcement mode the supervisor normally starts as root and can - // isolate the SSH socket in a root-only directory before spawning - // unprivileged children. Sidecar topology is different: the gateway relay - // runs in the network sidecar as a different UID, so the shared sidecar - // state directory must stay group-accessible. Sidecar mode uses a Linux - // abstract socket instead, so the workload cannot unlink the relay target. + // A driver may place the supervisor in another container, so an explicitly + // shared socket retains group access. Linux abstract sockets avoid a + // workload-replaceable filesystem inode. let abstract_socket = crate::unix_socket::is_abstract(listen_path); if !abstract_socket && let Some(parent) = listen_path.parent() { std::fs::create_dir_all(parent).into_diagnostic()?; #[cfg(unix)] - if enforcement_mode.uses_privileged_process_setup() && !shared_socket { + if !shared_socket { use std::os::unix::fs::PermissionsExt; let perms = std::fs::Permissions::from_mode(0o700); std::fs::set_permissions(parent, perms).into_diagnostic()?; @@ -94,7 +95,7 @@ fn ssh_server_init( let listener = UnixListener::bind(runtime_path.as_ref()).into_diagnostic()?; // Tighten filesystem-socket permissions. Abstract sockets have no inode; - // sidecar relay connections authenticate the listener with SO_PEERCRED. + // local relay connections authenticate the listener with SO_PEERCRED. #[cfg(unix)] if !abstract_socket { use std::os::unix::fs::PermissionsExt; @@ -119,71 +120,44 @@ fn ssh_server_init( pub async fn run_ssh_server( listen_path: PathBuf, ready_tx: tokio::sync::oneshot::Sender>, - policy: SandboxPolicy, - workspace: ResolvedWorkspace, - netns_fd: Option, - proxy_url: Option, ca_file_paths: Option<(PathBuf, PathBuf)>, - provider_credentials: ProviderCredentialState, - user_environment: HashMap, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, shared_socket: bool, - main_session: Arc, + port_forward: Arc, + boundary_exec: Arc, + main_session: Option>, ) -> Result<()> { - let (listener, config, ca_paths) = match ssh_server_init( - &listen_path, - &ca_file_paths, - enforcement_mode, - shared_socket, - ) { - Ok(v) => { - // Signal that the SSH server has bound the socket and is ready to - // accept connections. The parent task awaits this before spawning - // the entrypoint process, ensuring exec requests won't race - // against server startup. - let _ = ready_tx.send(Ok(())); - v - } - Err(err) => { - let _ = ready_tx.send(Err(err)); - return Ok(()); - } - }; - - let mut consecutive_resource_errors: u32 = 0; - let mut consecutive_unknown_errors: u32 = 0; + let (listener, config, _ca_paths) = + match ssh_server_init(&listen_path, &ca_file_paths, shared_socket) { + Ok(v) => { + // Signal that the SSH server has bound the socket and is ready to + // accept connections. The parent task awaits this before spawning + // the entrypoint process, ensuring exec requests won't race + // against server startup. + let _ = ready_tx.send(Ok(())); + v + } + Err(err) => { + let _ = ready_tx.send(Err(err)); + return Ok(()); + } + }; + let mut consecutive_resource_errors = 0; + let mut consecutive_unknown_errors = 0; loop { match listener.accept().await { Ok((stream, _peer)) => { consecutive_resource_errors = 0; consecutive_unknown_errors = 0; let config = config.clone(); - let policy = policy.clone(); - let workspace = workspace.clone(); - let proxy_url = proxy_url.clone(); - let ca_paths = ca_paths.clone(); - let provider_credentials = provider_credentials.clone(); - let user_environment = user_environment.clone(); - let main_session = Arc::clone(&main_session); + let port_forward = port_forward.clone(); + let boundary_exec = boundary_exec.clone(); + let main_session = main_session.clone(); tokio::spawn(async move { - if let Err(err) = handle_connection( - stream, - config, - policy, - workspace, - netns_fd, - proxy_url, - ca_paths, - provider_credentials, - user_environment, - resolved_identity, - enforcement_mode, - main_session, - ) - .await + if let Err(err) = + handle_connection(stream, config, port_forward, boundary_exec, main_session) + .await { ocsf_emit!( SshActivityBuilder::new(openshell_ocsf::ctx::ctx()) @@ -196,45 +170,31 @@ pub async fn run_ssh_server( } }); } - Err(err) => { - match classify_ssh_accept_error( - &err, - &mut consecutive_resource_errors, - &mut consecutive_unknown_errors, - ) { - SshAcceptAction::Terminal => { - ocsf_emit!( - SshActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::High) - .status(StatusId::Failure) - .message(format!( - "SSH accept loop exiting on terminal error: {err}" - )) - .build() - ); - break; - } - SshAcceptAction::Retry { backoff, severity } => { - ocsf_emit!( - SshActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(severity) - .status(StatusId::Failure) - .message(format!( - "SSH accept error (retrying in {}ms): {err}", - backoff.as_millis(), - )) - .build() - ); - tokio::time::sleep(backoff).await; - } + Err(error) => match classify_ssh_accept_error( + &error, + &mut consecutive_resource_errors, + &mut consecutive_unknown_errors, + ) { + SshAcceptAction::Terminal => { + return Err(error).into_diagnostic(); } - } + SshAcceptAction::Retry { backoff, severity } => { + ocsf_emit!( + SshActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(ActivityId::Fail) + .severity(severity) + .status(StatusId::Failure) + .message(format!( + "SSH accept error (retrying in {}ms): {error}", + backoff.as_millis() + )) + .build() + ); + tokio::time::sleep(backoff).await; + } + }, } } - - Ok(()) } const MAX_CONSECUTIVE_UNKNOWN_SSH_ACCEPT_ERRORS: u32 = 10; @@ -249,13 +209,13 @@ enum SshAcceptAction { } fn classify_ssh_accept_error( - err: &std::io::Error, + error: &std::io::Error, consecutive_resource_errors: &mut u32, consecutive_unknown_errors: &mut u32, ) -> SshAcceptAction { #[cfg(unix)] if matches!( - err.raw_os_error(), + error.raw_os_error(), Some(libc::EBADF | libc::EINVAL | libc::ENOTSOCK) ) { return SshAcceptAction::Terminal; @@ -263,7 +223,7 @@ fn classify_ssh_accept_error( #[cfg(unix)] if matches!( - err.raw_os_error(), + error.raw_os_error(), Some( libc::EMFILE | libc::ENFILE @@ -286,26 +246,20 @@ fn classify_ssh_accept_error( ) ) { *consecutive_unknown_errors = 0; - - #[cfg(unix)] - let is_resource_pressure = matches!( - err.raw_os_error(), + let resource_pressure = matches!( + error.raw_os_error(), Some(libc::EMFILE | libc::ENFILE | libc::ENOBUFS | libc::ENOMEM | libc::ENOSR) ); - #[cfg(not(unix))] - let is_resource_pressure = false; - - if is_resource_pressure { + if resource_pressure { *consecutive_resource_errors = consecutive_resource_errors.saturating_add(1); - let backoff_ms = 100u64 - .saturating_mul(1u64 << (*consecutive_resource_errors).min(7).saturating_sub(1)) + let backoff_ms = 100_u64 + .saturating_mul(1_u64 << (*consecutive_resource_errors).min(7).saturating_sub(1)) .min(5_000); return SshAcceptAction::Retry { backoff: Duration::from_millis(backoff_ms), severity: SeverityId::Medium, }; } - *consecutive_resource_errors = 0; return SshAcceptAction::Retry { backoff: Duration::from_millis(100), @@ -313,24 +267,25 @@ fn classify_ssh_accept_error( }; } - #[cfg(unix)] #[cfg(target_os = "linux")] - if matches!(err.raw_os_error(), Some(libc::ENONET)) { - *consecutive_unknown_errors = 0; + if error.raw_os_error() == Some(libc::ENONET) { *consecutive_resource_errors = 0; + *consecutive_unknown_errors = 0; return SshAcceptAction::Retry { backoff: Duration::from_millis(100), severity: SeverityId::Low, }; } + *consecutive_resource_errors = 0; *consecutive_unknown_errors = consecutive_unknown_errors.saturating_add(1); if *consecutive_unknown_errors >= MAX_CONSECUTIVE_UNKNOWN_SSH_ACCEPT_ERRORS { - return SshAcceptAction::Terminal; - } - SshAcceptAction::Retry { - backoff: Duration::from_millis(100), - severity: SeverityId::Low, + SshAcceptAction::Terminal + } else { + SshAcceptAction::Retry { + backoff: Duration::from_millis(100), + severity: SeverityId::Low, + } } } @@ -338,16 +293,9 @@ fn classify_ssh_accept_error( async fn handle_connection( stream: tokio::net::UnixStream, config: Arc, - policy: SandboxPolicy, - workspace: ResolvedWorkspace, - netns_fd: Option, - proxy_url: Option, - ca_file_paths: Option>, - provider_credentials: ProviderCredentialState, - user_environment: HashMap, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - main_session: Arc, + port_forward: Arc, + boundary_exec: Arc, + main_session: Option>, ) -> Result<()> { // Access is gated by the Unix-socket filesystem permissions (root-only), // not by an application-level preface. The supervisor bridges the @@ -363,18 +311,7 @@ async fn handle_connection( .build() ); - let handler = SshHandler::new( - policy, - workspace, - netns_fd, - proxy_url, - ca_file_paths, - provider_credentials, - user_environment, - resolved_identity, - enforcement_mode, - main_session, - ); + let handler = SshHandler::new(port_forward, boundary_exec, main_session); russh::server::run_stream(config, stream, handler) .await .map_err(|err| miette::miette!("ssh stream error: {err}"))?; @@ -387,13 +324,12 @@ async fn handle_connection( /// sender. This allows `window_change_request` to resize the correct PTY when /// multiple channels are open simultaneously (e.g. parallel shells, shell + /// sftp, etc.). -// Several independent per-channel boolean flags (login-shell opt-out and the -// main-attachment state bits) legitimately live side by side here. #[allow(clippy::struct_excessive_bools)] #[derive(Default)] struct ChannelState { input_sender: Option, - pty_master: Option, + process: Option>, + terminal: Option>, pty_request: Option, no_login_shell: bool, main_input_owner: Option, @@ -403,37 +339,6 @@ struct ChannelState { main_output_task: Option, } -const MAIN_DETACH_PREFIX: u8 = 0x10; // Ctrl-P -const MAIN_DETACH_KEY: u8 = 0x11; // Ctrl-Q - -/// Remove the `OpenShell` detach sequence from canonical-main input. -/// -/// A trailing Ctrl-P remains pending across SSH data frames. If the following -/// byte is not Ctrl-Q, both bytes are forwarded unchanged. Bytes after a -/// completed detach sequence are discarded because the attachment is closing. -fn filter_main_detach_sequence(prefix_pending: &mut bool, data: &[u8]) -> (Vec, bool) { - let mut forward = Vec::with_capacity(data.len() + usize::from(*prefix_pending)); - - for &byte in data { - if *prefix_pending { - if byte == MAIN_DETACH_KEY { - *prefix_pending = false; - return (forward, true); - } - forward.push(MAIN_DETACH_PREFIX); - *prefix_pending = false; - } - - if byte == MAIN_DETACH_PREFIX { - *prefix_pending = true; - } else { - forward.push(byte); - } - } - - (forward, false) -} - enum InputSender { Process(mpsc::Sender>), Main(tokio::sync::mpsc::Sender>), @@ -454,28 +359,26 @@ impl InputSender { } struct SshHandler { - policy: SandboxPolicy, - workspace: ResolvedWorkspace, - netns_fd: Option, - proxy_url: Option, - ca_file_paths: Option>, - provider_credentials: ProviderCredentialState, - user_environment: HashMap, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - main_session: Arc, + /// Loopback port-forward, injected by the orchestrator (RFC 0012). In-pod + /// this connects from inside the workload netns; a delegated backend + /// tunnels into its guest. The handler does not know which. + port_forward: Arc, + boundary_exec: Arc, + main_session: Option>, channels: HashMap, } impl Drop for SshHandler { fn drop(&mut self) { + let Some(main_session) = self.main_session.as_ref() else { + return; + }; for state in self.channels.values_mut() { if state.main_attached { - self.main_session.end_terminal_attachment(); - state.main_attached = false; + main_session.end_terminal_attachment(); } if let Some(owner) = state.main_input_owner.take() { - self.main_session.release_input(owner); + main_session.release_input(owner); } if let Some(task) = state.main_output_task.take() { task.abort(); @@ -485,29 +388,14 @@ impl Drop for SshHandler { } impl SshHandler { - #[allow(clippy::too_many_arguments)] fn new( - policy: SandboxPolicy, - workspace: ResolvedWorkspace, - netns_fd: Option, - proxy_url: Option, - ca_file_paths: Option>, - provider_credentials: ProviderCredentialState, - user_environment: HashMap, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - main_session: Arc, + port_forward: Arc, + boundary_exec: Arc, + main_session: Option>, ) -> Self { Self { - policy, - workspace, - netns_fd, - proxy_url, - ca_file_paths, - provider_credentials, - user_environment, - resolved_identity, - enforcement_mode, + port_forward, + boundary_exec, main_session, channels: HashMap::new(), } @@ -550,15 +438,23 @@ impl russh::server::Handler for SshHandler { channel: ChannelId, _session: &mut Session, ) -> Result<(), Self::Error> { - if let Some(state) = self.channels.remove(&channel) { - if state.main_attached { - self.main_session.end_terminal_attachment(); - } - if let Some(owner) = state.main_input_owner { - self.main_session.release_input(owner); + if let Some(mut state) = self.channels.remove(&channel) { + if state.main_attached + && let Some(main_session) = self.main_session.as_ref() + { + main_session.end_terminal_attachment(); + if let Some(owner) = state.main_input_owner.take() { + main_session.release_input(owner); + } + if let Some(task) = state.main_output_task.take() { + task.abort(); + } + return Ok(()); } - if let Some(task) = state.main_output_task { - task.abort(); + if let Some(process) = state.process { + // Channel ownership defines the exec lifetime. Closing an SSH + // channel must not strand an in-boundary process. + let _ = process.terminate().await; } } Ok(()) @@ -574,12 +470,6 @@ impl russh::server::Handler for SshHandler { reply: ChannelOpenHandle, _session: &mut Session, ) -> Result<(), Self::Error> { - if self.main_session.finished() { - reply - .reject(ChannelOpenFailure::AdministrativelyProhibited) - .await; - return Ok(()); - } // Validate port range before truncating u32 -> u16. The SSH protocol // uses u32 for ports, but valid TCP ports are 0-65535. Without this // check, port 65537 truncates to port 1 (privileged). @@ -599,9 +489,8 @@ impl russh::server::Handler for SshHandler { return Ok(()); } - // Only allow forwarding to loopback destinations to prevent the - // sandbox SSH server from being used as a generic proxy. - if !is_loopback_host(host_to_connect) { + let target = direct_tcpip_target(host_to_connect, port_to_connect); + if target.is_none() { ocsf_emit!(SshActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(ActivityId::Refuse) .action(ActionId::Denied) @@ -620,16 +509,13 @@ impl russh::server::Handler for SshHandler { let host = host_to_connect.to_string(); // SSH protocol port is bounded by u32 but only u16 is meaningful; // saturate as a guard for malformed clients. - let port = u16::try_from(port_to_connect).unwrap_or(u16::MAX); - let netns_fd = self.netns_fd; - - // Confirm the channel before spawning: the task below writes to it, and - // the peer must see the open-confirmation first. + let port = u16::try_from(port_to_connect).expect("port range checked above"); + let target = target.expect("loopback target checked above"); + let port_forward = self.port_forward.clone(); reply.accept().await; tokio::spawn(async move { - let addr = format!("{host}:{port}"); - let tcp = match connect_in_netns(&addr, netns_fd).await { + let mut tcp_stream = match port_forward.connect(target).await { Ok(stream) => stream, Err(err) => { ocsf_emit!( @@ -637,7 +523,9 @@ impl russh::server::Handler for SshHandler { .activity(ActivityId::Fail) .severity(SeverityId::Low) .status(StatusId::Failure) - .message(format!("direct-tcpip: failed to connect to {addr}: {err}")) + .message(format!( + "direct-tcpip: failed to connect to {host}:{port}: {err}" + )) .build() ); let _ = channel.close().await; @@ -646,7 +534,6 @@ impl russh::server::Handler for SshHandler { }; let mut channel_stream = channel.into_stream(); - let mut tcp_stream = tcp; let _ = tokio::io::copy_bidirectional(&mut channel_stream, &mut tcp_stream).await; }); @@ -694,18 +581,17 @@ impl russh::server::Handler for SshHandler { return Ok(()); }; if state.main_attached { - self.main_session - .resize(col_width, row_height, pixel_width, pixel_height); - } else if let Some(master) = state.pty_master.as_ref() { - let winsize = Winsize { - ws_row: to_u16(row_height.max(1)), - ws_col: to_u16(col_width.max(1)), - ws_xpixel: to_u16(pixel_width), - ws_ypixel: to_u16(pixel_height), - }; - if let Err(e) = unsafe_pty::set_winsize(master.as_raw_fd(), winsize) { - warn!("failed to resize PTY for channel {channel:?}: {e}"); + if let Some(main_session) = self.main_session.as_ref() { + main_session + .resize(col_width, row_height, pixel_width, pixel_height) + .await; } + } else if let Some(terminal) = state.terminal.as_ref() + && let Err(e) = terminal + .resize(to_u16(col_width.max(1)), to_u16(row_height.max(1))) + .await + { + warn!("failed to resize PTY for channel {channel:?}: {e}"); } Ok(()) } @@ -715,10 +601,6 @@ impl russh::server::Handler for SshHandler { channel: ChannelId, session: &mut Session, ) -> Result<(), Self::Error> { - if self.main_session.finished() { - session.channel_failure(channel)?; - return Ok(()); - } session.channel_success(channel)?; // Only allocate a PTY when the client explicitly requested one via // pty_request. VS Code Remote-SSH sends shell_request *without* a @@ -726,7 +608,7 @@ impl russh::server::Handler for SshHandler { // endings. Forcing a PTY here caused CRLF translation which made // VS Code misdetect the platform as Windows (and then try to run // `powershell`). - self.start_shell(channel, session.handle(), None)?; + self.start_shell(channel, session.handle(), None).await?; Ok(()) } @@ -736,16 +618,13 @@ impl russh::server::Handler for SshHandler { data: &[u8], session: &mut Session, ) -> Result<(), Self::Error> { - if self.main_session.finished() { - session.channel_failure(channel)?; - return Ok(()); - } session.channel_success(channel)?; let command = String::from_utf8_lossy(data).trim().to_string(); if command.is_empty() { return Ok(()); } - self.start_shell(channel, session.handle(), Some(command))?; + self.start_shell(channel, session.handle(), Some(command)) + .await?; Ok(()) } @@ -756,12 +635,11 @@ impl russh::server::Handler for SshHandler { session: &mut Session, ) -> Result<(), Self::Error> { if name == "openshell-main" { - if !self.channels.contains_key(&channel) { - return Err(anyhow::anyhow!( - "subsystem_request on unknown channel {channel:?}" - )); - } - if self.main_session.begin_terminal_attachment().is_err() { + let Some(main_session) = self.main_session.clone() else { + session.channel_failure(channel)?; + return Ok(()); + }; + if !begin_main_attachment(&main_session, self.channels.contains_key(&channel)) { session.channel_failure(channel)?; return Ok(()); } @@ -771,34 +649,33 @@ impl russh::server::Handler for SshHandler { .expect("main channel existence checked above"); state.main_attached = true; if let Some(pty) = state.pty_request.take() { - self.main_session.resize( - pty.col_width, - pty.row_height, - pty.pixel_width, - pty.pixel_height, - ); + main_session + .resize( + pty.col_width, + pty.row_height, + pty.pixel_width, + pty.pixel_height, + ) + .await; } - let (input, input_warning) = if state.main_read_only { + let (input, warning) = if state.main_read_only { (None, None) } else { - match self.main_session.acquire_input() { + match main_session.acquire_input() { Ok((owner, input)) => { state.main_input_owner = Some(owner); (Some(InputSender::Main(input)), None) } - Err(error) => { - warn!(%error, "main process input lease unavailable; attaching read-only"); - (None, Some(error)) - } + Err(error) => (None, Some(error)), } }; - state.main_detach_prefix_pending = false; state.input_sender = input; - let mut output = self.main_session.subscribe(); - let terminal_delivery = Arc::clone(&self.main_session); + state.main_detach_prefix_pending = false; + let mut output = main_session.subscribe(); + let terminal_delivery = main_session.clone(); let handle = session.handle(); session.channel_success(channel)?; - if let Some(error) = input_warning { + if let Some(error) = warning { let _ = handle .extended_data( channel, @@ -810,13 +687,13 @@ impl russh::server::Handler for SshHandler { let output_task = tokio::spawn(async move { loop { match output.recv().await { + Ok(MainOutput::Exit(code)) => { + terminal_delivery.wait_for_terminal_reported().await; + let _ = + send_main_output(&handle, channel, MainOutput::Exit(code)).await; + break; + } Ok(event) => { - if let MainOutput::Exit(code) = event { - terminal_delivery.wait_for_terminal_reported().await; - let _ = send_main_output(&handle, channel, MainOutput::Exit(code)) - .await; - break; - } let _ = send_main_output(&handle, channel, event).await; } Err(error) => { @@ -840,31 +717,24 @@ impl russh::server::Handler for SshHandler { if let Some(state) = self.channels.get_mut(&channel) { state.main_output_task = Some(output_task.abort_handle()); } - } else if name == "sftp" && !self.main_session.finished() { + } else if name == "sftp" { session.channel_success(channel)?; // sftp-server speaks the SFTP binary protocol over stdin/stdout, - // which is exactly what spawn_pipe_exec wires up. This enables + // which the boundary executor preserves as separate pipes. This enables // modern scp (SFTP-based, OpenSSH 9.0+) and SFTP clients to // transfer files into and out of the sandbox. - let input_sender = spawn_pipe_exec( - &self.policy, - &self.workspace, - Some("/usr/lib/openssh/sftp-server".to_string()), - false, - session.handle(), + self.start_exec_spec( channel, - self.netns_fd, - self.proxy_url.clone(), - self.ca_file_paths.clone(), - &self.provider_credentials.child_env_with_gcp_resolved(), - &self.user_environment, - self.resolved_identity, - self.enforcement_mode, - )?; - let state = self.channels.get_mut(&channel).ok_or_else(|| { - anyhow::anyhow!("subsystem_request on unknown channel {channel:?}") - })?; - state.input_sender = Some(InputSender::Process(input_sender)); + session.handle(), + openshell_isolation_interface::contract::ExecSpec { + program: "/usr/lib/openssh/sftp-server".to_string(), + args: vec![], + env: vec![], + workdir: None, + pty: false, + }, + ) + .await?; } else { ocsf_emit!( SshActivityBuilder::new(openshell_ocsf::ctx::ctx()) @@ -889,11 +759,9 @@ impl russh::server::Handler for SshHandler { ) -> Result<(), Self::Error> { // Accept the env request so the client knows we handled it, but we // don't actually propagate arbitrary variables — the sandbox - // environment is controlled via policy. We must reply so VSCode - // doesn't stall. Two exceptions carry supervisor signals the SSH - // protocol has no native field for: - // - OPENSHELL_NO_LOGIN_SHELL: gateway login-shell opt-out. - // - OPENSHELL_MAIN_READ_ONLY: read-only main attachment. + // environment is controlled via policy. The login-shell opt-out is a + // supervisor signal carried over SSH because the protocol has no + // native field for it. if variable_name == NO_LOGIN_SHELL_ENV.0 && let Some(state) = self.channels.get_mut(&channel) { @@ -919,38 +787,17 @@ impl russh::server::Handler for SshHandler { warn!("data on unknown channel {channel:?}"); return Ok(()); }; - - let main_attached = state.main_attached; - let (forward, detach) = if main_attached { + let (forward, detach) = if state.main_attached { filter_main_detach_sequence(&mut state.main_detach_prefix_pending, data) } else { (data.to_vec(), false) }; - let send_error = (!forward.is_empty()) + let error = (!forward.is_empty()) .then(|| state.input_sender.as_ref()?.send(forward).err()) .flatten(); - - if let Some(error) = send_error { - let handle = session.handle(); - if main_attached { - self.close_main_attachment(channel, handle, Some(error)) - .await; - } else { - let _ = handle - .extended_data( - channel, - 1, - format!("openshell: {error}; closing attachment\n").into_bytes(), - ) - .await; - let _ = handle.close(channel).await; - } - return Ok(()); - } - if detach { - self.close_main_attachment(channel, session.handle(), None) + if state.main_attached && (detach || error.is_some()) { + self.close_main_attachment(channel, session.handle(), error) .await; - return Ok(()); } Ok(()) } @@ -967,8 +814,12 @@ impl russh::server::Handler for SshHandler { if let Some(state) = self.channels.get_mut(&channel) { if state.main_attached && let Some(owner) = state.main_input_owner.take() + && let Some(main_session) = self.main_session.as_ref() { - self.main_session.release_input(owner); + // A canonical process outlives one SSH attachment. Release + // this channel's lease without closing process stdin so a + // replacement attachment can become the input owner. + main_session.release_input(owner); } state.input_sender.take(); state.main_detach_prefix_pending = false; @@ -984,47 +835,192 @@ impl russh::server::Handler for SshHandler { signal: Sig, _session: &mut Session, ) -> Result<(), Self::Error> { - if !self + if self .channels .get(&channel) .is_some_and(|state| state.main_attached) { + let signal = match signal { + Sig::HUP => Some(nix::sys::signal::Signal::SIGHUP), + Sig::INT => Some(nix::sys::signal::Signal::SIGINT), + Sig::KILL => Some(nix::sys::signal::Signal::SIGKILL), + Sig::QUIT => Some(nix::sys::signal::Signal::SIGQUIT), + Sig::TERM => Some(nix::sys::signal::Signal::SIGTERM), + _ => None, + }; + if let (Some(signal), Some(main_session)) = (signal, self.main_session.as_ref()) + && let Err(error) = main_session.signal_group(signal).await + { + warn!(%error, ?signal, "failed to signal canonical main process group"); + } return Ok(()); } + let Some(process) = self + .channels + .get(&channel) + .and_then(|state| state.process.clone()) + else { + return Ok(()); + }; let signal = match signal { - Sig::HUP => Some(nix::sys::signal::Signal::SIGHUP), - Sig::INT => Some(nix::sys::signal::Signal::SIGINT), - Sig::KILL => Some(nix::sys::signal::Signal::SIGKILL), - Sig::QUIT => Some(nix::sys::signal::Signal::SIGQUIT), - Sig::TERM => Some(nix::sys::signal::Signal::SIGTERM), + Sig::HUP => Some(openshell_isolation_interface::contract::BoundarySignal::Hup), + Sig::INT => Some(openshell_isolation_interface::contract::BoundarySignal::Int), + Sig::KILL => Some(openshell_isolation_interface::contract::BoundarySignal::Kill), + Sig::TERM => Some(openshell_isolation_interface::contract::BoundarySignal::Term), _ => None, }; if let Some(signal) = signal - && let Err(error) = self.main_session.signal_group(signal) + && let Err(error) = process.signal(signal).await { - warn!(%error, ?signal, "failed to signal canonical main process group"); + warn!(%error, ?signal, "failed to signal boundary exec process"); } Ok(()) } } -async fn send_main_output(handle: &Handle, channel: ChannelId, event: MainOutput) -> bool { - match event { - MainOutput::Stdout(data) => handle.data(channel, data).await.is_ok(), - MainOutput::Stderr(data) => handle.extended_data(channel, 1, data).await.is_ok(), - MainOutput::Exit(code) => { - let eof_sent = handle.eof(channel).await.is_ok(); - let status_sent = handle - .exit_status_request(channel, code.max(0).unsigned_abs()) +impl SshHandler { + async fn start_shell( + &mut self, + channel: ChannelId, + handle: Handle, + command: Option, + ) -> anyhow::Result<()> { + let state = self + .channels + .get_mut(&channel) + .ok_or_else(|| anyhow::anyhow!("start_shell on unknown channel {channel:?}"))?; + let no_login_shell = state.no_login_shell; + let pty = state.pty_request.take(); + let pty_requested = pty.is_some(); + let (program, args) = command.map_or_else( + || { + if pty_requested { + ("/bin/bash".to_string(), vec!["-i".to_string()]) + } else { + ("/bin/bash".to_string(), vec![]) + } + }, + |command| { + ( + "/bin/bash".to_string(), + vec![login_shell_flag(no_login_shell).to_string(), command], + ) + }, + ); + let env = pty + .as_ref() + .map(|request| vec![("TERM".to_string(), request.term.clone())]) + .unwrap_or_default(); + self.start_exec_spec( + channel, + handle, + openshell_isolation_interface::contract::ExecSpec { + program, + args, + env, + workdir: None, + pty: pty_requested, + }, + ) + .await?; + if let (Some(pty), Some(terminal)) = ( + pty, + self.channels + .get(&channel) + .and_then(|state| state.terminal.as_ref()), + ) { + terminal + .resize(to_u16(pty.col_width.max(1)), to_u16(pty.row_height.max(1))) .await - .is_ok(); - let close_sent = handle.close(channel).await.is_ok(); - eof_sent && status_sent && close_sent + .map_err(|error| anyhow::anyhow!(error.to_string()))?; } + Ok(()) + } + + async fn start_exec_spec( + &mut self, + channel: ChannelId, + handle: Handle, + spec: openshell_isolation_interface::contract::ExecSpec, + ) -> anyhow::Result<()> { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + let mut exec = self + .boundary_exec + .exec(spec) + .await + .map_err(|error| anyhow::anyhow!(error.to_string()))?; + let state = self + .channels + .get_mut(&channel) + .ok_or_else(|| anyhow::anyhow!("exec on unknown channel {channel:?}"))?; + state.process = Some(exec.process.clone()); + state.terminal = exec.terminal.take(); + + if let Some(mut stdin) = exec.stdin.take() { + let (sender, receiver) = mpsc::channel::>(); + let runtime = tokio::runtime::Handle::current(); + std::thread::spawn(move || { + while let Ok(bytes) = receiver.recv() { + if runtime.block_on(stdin.write_all(&bytes)).is_err() { + break; + } + } + }); + state.input_sender = Some(InputSender::Process(sender)); + } + + let mut stdout = exec.stdout; + let stdout_handle = handle.clone(); + let stdout_task = tokio::spawn(async move { + let mut buffer = [0_u8; 4096]; + loop { + match stdout.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(size) => { + let _ = stdout_handle.data(channel, buffer[..size].to_vec()).await; + } + } + } + }); + let stderr_task = exec.stderr.map(|mut stderr| { + let stderr_handle = handle.clone(); + tokio::spawn(async move { + let mut buffer = [0_u8; 4096]; + loop { + match stderr.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(size) => { + let _ = stderr_handle + .extended_data(channel, 1, buffer[..size].to_vec()) + .await; + } + } + } + }) + }); + tokio::spawn(async move { + let status = exec.process.wait().await; + let _ = stdout_task.await; + if let Some(task) = stderr_task { + let _ = task.await; + } + let code = match status { + Ok(openshell_isolation_interface::contract::BoundaryExitStatus::Exited(code)) => { + code.max(0).cast_unsigned() + } + Ok(openshell_isolation_interface::contract::BoundaryExitStatus::Signaled( + signal, + )) => (128_i32.saturating_add(signal)).max(0).cast_unsigned(), + Err(_) => 1, + }; + let _ = handle.eof(channel).await; + let _ = handle.exit_status_request(channel, code).await; + let _ = handle.close(channel).await; + }); + Ok(()) } -} -impl SshHandler { async fn close_main_attachment( &mut self, channel: ChannelId, @@ -1033,11 +1029,15 @@ impl SshHandler { ) { if let Some(state) = self.channels.get_mut(&channel) { if state.main_attached { - self.main_session.end_terminal_attachment(); + if let Some(main_session) = self.main_session.as_ref() { + main_session.end_terminal_attachment(); + } state.main_attached = false; } - if let Some(owner) = state.main_input_owner.take() { - self.main_session.release_input(owner); + if let Some(owner) = state.main_input_owner.take() + && let Some(main_session) = self.main_session.as_ref() + { + main_session.release_input(owner); } state.input_sender.take(); state.main_detach_prefix_pending = false; @@ -1058,118 +1058,33 @@ impl SshHandler { let _ = handle.exit_status_request(channel, 0).await; let _ = handle.close(channel).await; } +} - fn start_shell( - &mut self, - channel: ChannelId, - handle: Handle, - command: Option, - ) -> anyhow::Result<()> { - let provider_env = self.provider_credentials.child_env_with_gcp_resolved(); - let state = self - .channels - .get_mut(&channel) - .ok_or_else(|| anyhow::anyhow!("start_shell on unknown channel {channel:?}"))?; - let no_login_shell = state.no_login_shell; - if let Some(pty) = state.pty_request.take() { - // PTY was requested — allocate a real PTY (interactive shell or - // exec that explicitly asked for a terminal). - let (pty_master, input_sender) = spawn_pty_shell( - &self.policy, - &self.workspace, - command, - no_login_shell, - &pty, - handle, - channel, - self.netns_fd, - self.proxy_url.clone(), - self.ca_file_paths.clone(), - &provider_env, - &self.user_environment, - self.resolved_identity, - self.enforcement_mode, - )?; - state.pty_master = Some(pty_master); - state.input_sender = Some(InputSender::Process(input_sender)); - } else { - // No PTY requested — use plain pipes so stdout/stderr are - // separate and output has clean LF line endings. This is the - // path VSCode Remote-SSH exec commands take. - let input_sender = spawn_pipe_exec( - &self.policy, - &self.workspace, - command, - no_login_shell, - handle, - channel, - self.netns_fd, - self.proxy_url.clone(), - self.ca_file_paths.clone(), - &provider_env, - &self.user_environment, - self.resolved_identity, - self.enforcement_mode, - )?; - state.input_sender = Some(InputSender::Process(input_sender)); +fn begin_main_attachment(main_session: &MainSession, channel_exists: bool) -> bool { + channel_exists && main_session.begin_terminal_attachment().is_ok() +} + +async fn send_main_output(handle: &Handle, channel: ChannelId, event: MainOutput) -> bool { + match event { + MainOutput::Stdout(data) => handle.data(channel, data).await.is_ok(), + MainOutput::Stderr(data) => handle.extended_data(channel, 1, data).await.is_ok(), + MainOutput::Exit(code) => { + let eof = handle.eof(channel).await.is_ok(); + let status = handle + .exit_status_request(channel, code.max(0).unsigned_abs()) + .await + .is_ok(); + let close = handle.close(channel).await.is_ok(); + eof && status && close } - Ok(()) } } -/// Connect a TCP stream to `addr` inside the sandbox network namespace. -/// -/// The SSH supervisor runs in the host network namespace while sandbox child -/// processes run in an isolated network namespace (with their own loopback). -/// A plain `TcpStream::connect("127.0.0.1:port")` from the supervisor would -/// hit the host loopback, not the sandbox loopback where services are listening. -/// -/// On Linux, we spawn a dedicated OS thread, call `setns` to enter the sandbox -/// namespace, create the socket there, then convert it to a tokio `TcpStream`. -/// We use `std::thread::spawn` (not `spawn_blocking`) because `setns` changes -/// the calling thread's network namespace permanently — a tokio blocking-pool -/// thread could be reused for unrelated tasks and must not be contaminated. -/// On non-Linux platforms (no network namespace support), we connect directly. -pub async fn connect_in_netns( - addr: &str, - netns_fd: Option, -) -> std::io::Result { - #[cfg(target_os = "linux")] - if let Some(fd) = netns_fd { - let addr = addr.to_string(); - let (tx, rx) = tokio::sync::oneshot::channel(); - std::thread::spawn(move || { - let result = (|| -> std::io::Result { - // Enter the sandbox network namespace on this dedicated thread. - // SAFETY: setns is safe to call; this is a dedicated thread that - // will exit after the connection is established. - #[allow(unsafe_code)] - let rc = unsafe { libc::setns(fd, libc::CLONE_NEWNET) }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - std::net::TcpStream::connect(&addr) - })(); - let _ = tx.send(result); - }); - - let std_stream = rx - .await - .map_err(|_| std::io::Error::other("netns connect thread panicked"))??; - std_stream.set_nonblocking(true)?; - let stream = tokio::net::TcpStream::from_std(std_stream)?; - set_tcp_nodelay_best_effort(&stream); - return Ok(stream); - } - - #[cfg(not(target_os = "linux"))] - let _ = netns_fd; - - let stream = tokio::net::TcpStream::connect(addr).await?; - set_tcp_nodelay_best_effort(&stream); - Ok(stream) +const fn login_shell_flag(no_login_shell: bool) -> &'static str { + if no_login_shell { "-c" } else { "-lc" } } +#[allow(dead_code)] #[derive(Clone)] struct PtyRequest { term: String, @@ -1191,575 +1106,6 @@ impl Default for PtyRequest { } } -#[allow(clippy::too_many_arguments)] -fn apply_child_env( - cmd: &mut Command, - session_home: &str, - session_user: &str, - term: &str, - proxy_url: Option<&str>, - ca_file_paths: Option<&(PathBuf, PathBuf)>, - provider_env: &HashMap, - user_environment: &HashMap, -) { - let path = std::env::var("PATH").unwrap_or_else(|_| "/usr/local/bin:/usr/bin:/bin".into()); - - cmd.env_clear() - .env(openshell_core::sandbox_env::SANDBOX, "1") - .env("HOME", session_home) - .env("USER", session_user) - .env("SHELL", openshell_core::shell::detect_login_shell()) - .env("PATH", &path) - .env("TERM", term); - - for (key, value) in user_environment { - if !key.starts_with("OPENSHELL_") { - cmd.env(key, value); - } - } - - if let Some(url) = proxy_url { - for (key, value) in child_env::proxy_env_vars(url) { - cmd.env(key, value); - } - } - - if let Some((ca_cert_path, combined_bundle_path)) = ca_file_paths { - for (key, value) in child_env::tls_env_vars(ca_cert_path, combined_bundle_path) { - cmd.env(key, value); - } - } - - for (key, value) in provider_env { - if is_supervisor_only_env_var(key) { - continue; - } - cmd.env(key, value); - } -} - -const fn login_shell_flag(no_login_shell: bool) -> &'static str { - if no_login_shell { "-c" } else { "-lc" } -} - -/// Build the shell command for an SSH session using a shell that exists in the -/// sandbox image (minimal images such as Alpine ship only `/bin/sh`, not bash). -/// -/// `no_command_arg` is appended only when no explicit command is given: `-i` -/// for an interactive PTY session, or `None` for the non-PTY stdin path (a -/// bare shell already reads piped stdin line-by-line). With an explicit -/// command the login-shell flag is used per `no_login_shell`. -fn build_ssh_shell_command( - shell: &str, - command: Option, - no_login_shell: bool, - no_command_arg: Option<&str>, -) -> Command { - let mut cmd = Command::new(shell); - match command { - None => { - if let Some(arg) = no_command_arg { - cmd.arg(arg); - } - } - Some(command) => { - cmd.arg(login_shell_flag(no_login_shell)).arg(command); - } - } - cmd -} - -#[allow(clippy::too_many_arguments)] -fn spawn_pty_shell( - policy: &SandboxPolicy, - workspace: &ResolvedWorkspace, - command: Option, - no_login_shell: bool, - pty: &PtyRequest, - handle: Handle, - channel: ChannelId, - netns_fd: Option, - proxy_url: Option, - ca_file_paths: Option>, - provider_env: &HashMap, - user_environment: &HashMap, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, -) -> anyhow::Result<(std::fs::File, mpsc::Sender>)> { - let winsize = Winsize { - ws_row: to_u16(pty.row_height.max(1)), - ws_col: to_u16(pty.col_width.max(1)), - ws_xpixel: to_u16(pty.pixel_width), - ws_ypixel: to_u16(pty.pixel_height), - }; - let openpty = openpty(Some(&winsize), None)?; - let master = std::fs::File::from(openpty.master); - let slave = std::fs::File::from(openpty.slave); - let slave_fd = slave.as_raw_fd(); - - let stdin = slave.try_clone()?; - let stdout = slave.try_clone()?; - let stderr = slave; - let mut reader = master.try_clone()?; - let mut writer = master.try_clone()?; - - // Resolve a shell present in the sandbox image; interactive PTY sessions - // pass `-i` when no command is given. Runs in the supervisor, so it - // inspects the sandbox filesystem. - let shell = openshell_core::shell::detect_login_shell(); - let mut cmd = build_ssh_shell_command(&shell, command, no_login_shell, Some("-i")); - - let term = if pty.term.is_empty() { - "xterm-256color" - } else { - pty.term.as_str() - }; - - // Derive USER and HOME from the policy's run_as_user when available, - // falling back to "sandbox" / "/sandbox" for backward compatibility. - let (session_user, session_home) = session_user_and_home(policy, workspace.home()); - apply_child_env( - &mut cmd, - &session_home, - &session_user, - term, - proxy_url.as_deref(), - ca_file_paths.as_deref(), - provider_env, - user_environment, - ); - cmd.stdin(stdin).stdout(stdout).stderr(stderr); - - if let Some(dir) = workspace.root() { - cmd.current_dir(dir); - } - - // Probe Landlock availability from the parent process where tracing works. - #[cfg(target_os = "linux")] - if enforcement_mode.enforces_child_sandbox() { - sandbox::linux::log_sandbox_readiness(policy, workspace.root()); - } - - // Phase 1: Prepare Landlock ruleset before the child applies it. - #[cfg(target_os = "linux")] - let prepared_sandbox = - crate::process::prepare_child_sandbox(policy, workspace.root(), enforcement_mode) - .map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?; - - #[cfg(unix)] - { - unsafe_pty::install_pre_exec( - &mut cmd, - policy.clone(), - workspace.owned_root(), - slave_fd, - netns_fd, - resolved_identity, - enforcement_mode, - #[cfg(target_os = "linux")] - prepared_sandbox, - ); - } - - #[cfg(target_os = "linux")] - let mut child = crate::process::spawn_std_command_with_supervisor_identity_namespace(cmd)?; - #[cfg(not(target_os = "linux"))] - let mut child = cmd.spawn()?; - #[cfg(target_os = "linux")] - let child_pid = child.id(); - #[cfg(target_os = "linux")] - managed_children::register(child_pid); - let master_file = master; - - let (sender, receiver) = mpsc::channel::>(); - std::thread::spawn(move || { - while let Ok(bytes) = receiver.recv() { - if writer.write_all(&bytes).is_err() { - break; - } - let _ = writer.flush(); - } - }); - - let runtime = tokio::runtime::Handle::current(); - let runtime_reader = runtime.clone(); - let handle_clone = handle.clone(); - // Signal from the reader thread to the exit thread that all output has - // been forwarded. The exit thread waits for this before sending the - // exit-status and closing the channel, ensuring the correct SSH protocol - // ordering: data → EOF → exit-status → close. - let (reader_done_tx, reader_done_rx) = mpsc::channel::<()>(); - std::thread::spawn(move || { - let mut buf = [0u8; 4096]; - loop { - match reader.read(&mut buf) { - Ok(0) | Err(_) => break, - Ok(n) => { - let data = buf[..n].to_vec(); - let handle_clone = handle_clone.clone(); - let _ = runtime_reader - .block_on(async move { handle_clone.data(channel, data).await }); - } - } - } - // Send EOF to indicate no more data will be sent on this channel. - let eof_handle = handle_clone.clone(); - let _ = runtime_reader.block_on(async move { eof_handle.eof(channel).await }); - // Notify the exit thread that all output has been forwarded. - let _ = reader_done_tx.send(()); - }); - - let handle_exit = handle; - let runtime_exit = runtime; - std::thread::spawn(move || { - let status = child.wait().ok(); - #[cfg(target_os = "linux")] - managed_children::unregister(child_pid); - let code = status.and_then(|s| s.code()).unwrap_or(1).unsigned_abs(); - // Wait for the reader thread to finish forwarding all output before - // sending exit-status and closing the channel. This prevents the - // race where close() was called before exit_status_request(). - // - // Use a timeout because a backgrounded grandchild process (e.g. - // `nohup daemon &`) may hold the PTY slave open indefinitely, - // preventing the reader from reaching EOF. Two seconds is enough - // for any remaining buffered data to drain. - let _ = reader_done_rx.recv_timeout(Duration::from_secs(2)); - drop(runtime_exit.spawn(async move { - let _ = handle_exit.exit_status_request(channel, code).await; - let _ = handle_exit.close(channel).await; - })); - }); - - Ok((master_file, sender)) -} - -/// Spawn a command using plain pipes (no PTY). -/// -/// stdout is forwarded as SSH channel data and stderr as SSH extended data -/// (type 1), preserving the separation that clients like `VSCode` Remote-SSH -/// expect. Output retains clean LF line endings (no CRLF translation). -#[allow(clippy::too_many_arguments)] -fn spawn_pipe_exec( - policy: &SandboxPolicy, - workspace: &ResolvedWorkspace, - command: Option, - no_login_shell: bool, - handle: Handle, - channel: ChannelId, - netns_fd: Option, - proxy_url: Option, - ca_file_paths: Option>, - provider_env: &HashMap, - user_environment: &HashMap, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, -) -> anyhow::Result>> { - // Resolve a shell present in the sandbox image; minimal images (e.g. Alpine) - // don't ship bash, only `/bin/sh`. Runs in the supervisor, so it inspects - // the sandbox filesystem. No command → read from stdin with no `-i`: - // interactive mode reads .bashrc, writes prompts to stderr, and can add - // just enough latency for VS Code Remote-SSH's platform detection to time - // out and fall back to "windows". A plain shell with piped stdin already - // reads commands line-by-line (script mode), which is what VS Code expects. - let shell = openshell_core::shell::detect_login_shell(); - let mut cmd = build_ssh_shell_command(&shell, command, no_login_shell, None); - - let (session_user, session_home) = session_user_and_home(policy, workspace.home()); - apply_child_env( - &mut cmd, - &session_home, - &session_user, - "dumb", - proxy_url.as_deref(), - ca_file_paths.as_deref(), - provider_env, - user_environment, - ); - cmd.stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()); - - if let Some(dir) = workspace.root() { - cmd.current_dir(dir); - } - - // Probe Landlock availability from the parent process where tracing works. - #[cfg(target_os = "linux")] - if enforcement_mode.enforces_child_sandbox() { - sandbox::linux::log_sandbox_readiness(policy, workspace.root()); - } - - // Phase 1: Prepare Landlock ruleset before the child applies it. - #[cfg(target_os = "linux")] - let prepared_sandbox = - crate::process::prepare_child_sandbox(policy, workspace.root(), enforcement_mode) - .map_err(|err| anyhow::anyhow!("Failed to prepare sandbox: {err}"))?; - - #[cfg(unix)] - { - unsafe_pty::install_pre_exec_no_pty( - &mut cmd, - policy.clone(), - workspace.owned_root(), - netns_fd, - resolved_identity, - enforcement_mode, - #[cfg(target_os = "linux")] - prepared_sandbox, - ); - } - - #[cfg(target_os = "linux")] - let mut child = crate::process::spawn_std_command_with_supervisor_identity_namespace(cmd)?; - #[cfg(not(target_os = "linux"))] - let mut child = cmd.spawn()?; - #[cfg(target_os = "linux")] - let child_pid = child.id(); - #[cfg(target_os = "linux")] - managed_children::register(child_pid); - - let child_stdin = child.stdin.take(); - let child_stdout = child.stdout.take().expect("stdout must be piped"); - let child_stderr = child.stderr.take().expect("stderr must be piped"); - - // stdin writer thread - let (sender, receiver) = mpsc::channel::>(); - std::thread::spawn(move || { - let Some(mut stdin) = child_stdin else { - return; - }; - while let Ok(bytes) = receiver.recv() { - if stdin.write_all(&bytes).is_err() { - break; - } - let _ = stdin.flush(); - } - }); - - let runtime = tokio::runtime::Handle::current(); - - // Signal from the reader threads to the exit thread that all output has - // been forwarded. - let (reader_done_tx, reader_done_rx) = mpsc::channel::<()>(); - - // stdout reader - let stdout_handle = handle.clone(); - let stdout_runtime = runtime.clone(); - let reader_done_stdout = reader_done_tx.clone(); - std::thread::spawn(move || { - let mut reader = child_stdout; - let mut buf = [0u8; 4096]; - loop { - match reader.read(&mut buf) { - Ok(0) | Err(_) => break, - Ok(n) => { - let data = buf[..n].to_vec(); - let h = stdout_handle.clone(); - let _ = stdout_runtime.block_on(async move { h.data(channel, data).await }); - } - } - } - let _ = reader_done_stdout.send(()); - }); - - // stderr reader — sends as extended data (type 1) - let stderr_handle = handle.clone(); - let stderr_runtime = runtime.clone(); - std::thread::spawn(move || { - let mut reader = child_stderr; - let mut buf = [0u8; 4096]; - loop { - match reader.read(&mut buf) { - Ok(0) | Err(_) => break, - Ok(n) => { - let data = buf[..n].to_vec(); - let h = stderr_handle.clone(); - let _ = stderr_runtime - .block_on(async move { h.extended_data(channel, 1, data).await }); - } - } - } - let _ = reader_done_tx.send(()); - }); - - // Exit waiter thread - let handle_exit = handle; - let runtime_exit = runtime; - std::thread::spawn(move || { - let status = child.wait().ok(); - #[cfg(target_os = "linux")] - managed_children::unregister(child_pid); - let code = status.and_then(|s| s.code()).unwrap_or(1).unsigned_abs(); - // Wait for both reader threads. - let _ = reader_done_rx.recv_timeout(Duration::from_secs(2)); - let _ = reader_done_rx.recv_timeout(Duration::from_secs(1)); - drop(runtime_exit.spawn(async move { - let _ = handle_exit.eof(channel).await; - let _ = handle_exit.exit_status_request(channel, code).await; - let _ = handle_exit.close(channel).await; - })); - }); - - Ok(sender) -} - -mod unsafe_pty { - #[cfg(not(target_os = "linux"))] - use super::sandbox; - use super::{ - Command, ProcessEnforcementMode, RawFd, ResolvedProcessIdentity, SandboxPolicy, Winsize, - drop_privileges_with_identity, setsid, - }; - #[cfg(unix)] - use std::os::unix::process::CommandExt; - - #[allow(unsafe_code)] - pub fn set_winsize(fd: RawFd, winsize: Winsize) -> std::io::Result<()> { - let rc = unsafe { libc::ioctl(fd, libc::TIOCSWINSZ, &winsize) }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - Ok(()) - } - - #[allow(unsafe_code)] - // `libc::TIOCSCTTY` is `u32` on macOS/BSD and `u64` on Linux; allow the - // cross-platform conversion so the same expression compiles everywhere. - #[allow(clippy::useless_conversion)] - fn set_controlling_tty(fd: RawFd) -> std::io::Result<()> { - let rc = unsafe { libc::ioctl(fd, libc::TIOCSCTTY.into(), 0) }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - Ok(()) - } - - #[allow(unsafe_code)] - #[allow(clippy::too_many_arguments)] - #[cfg_attr( - not(target_os = "linux"), - allow( - clippy::unnecessary_wraps, - reason = "Linux pre_exec setup can fail while non-Linux setup cannot." - ) - )] - pub fn install_pre_exec( - cmd: &mut Command, - policy: SandboxPolicy, - _workdir: Option, - slave_fd: RawFd, - netns_fd: Option, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - #[cfg(target_os = "linux")] prepared: Option, - ) { - // Wrap in Option so we can .take() it out of the FnMut closure. - // pre_exec is only called once (after fork, before exec). - #[cfg(target_os = "linux")] - let mut prepared = prepared; - unsafe { - cmd.pre_exec(move || { - setsid().map_err(|err| std::io::Error::other(err.to_string()))?; - set_controlling_tty(slave_fd)?; - - enter_netns_and_sandbox( - netns_fd, - &policy, - resolved_identity, - enforcement_mode, - #[cfg(target_os = "linux")] - prepared.take(), - ) - }); - } - } - - /// Pre-exec hook for pipe-based (non-PTY) exec. - /// - /// Skips `setsid` and `TIOCSCTTY` since there is no controlling terminal. - #[allow(unsafe_code)] - #[cfg_attr( - not(target_os = "linux"), - allow( - clippy::unnecessary_wraps, - reason = "Linux pre_exec setup can fail while non-Linux setup cannot." - ) - )] - pub fn install_pre_exec_no_pty( - cmd: &mut Command, - policy: SandboxPolicy, - _workdir: Option, - netns_fd: Option, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - #[cfg(target_os = "linux")] prepared: Option, - ) { - #[cfg(target_os = "linux")] - let mut prepared = prepared; - unsafe { - cmd.pre_exec(move || { - enter_netns_and_sandbox( - netns_fd, - &policy, - resolved_identity, - enforcement_mode, - #[cfg(target_os = "linux")] - prepared.take(), - ) - }); - } - } - - fn enter_netns_and_sandbox( - netns_fd: Option, - policy: &SandboxPolicy, - resolved_identity: ResolvedProcessIdentity, - enforcement_mode: ProcessEnforcementMode, - #[cfg(target_os = "linux")] prepared: Option, - ) -> std::io::Result<()> { - // Enter network namespace before dropping privileges. - // This ensures SSH shell processes are isolated to the same - // network namespace as the entrypoint, forcing all traffic - // through the veth pair and CONNECT proxy. - #[cfg(target_os = "linux")] - if let Some(fd) = netns_fd { - #[allow(unsafe_code)] - let result = unsafe { libc::setns(fd, libc::CLONE_NEWNET) }; - if result != 0 { - return Err(std::io::Error::last_os_error()); - } - } - - #[cfg(not(target_os = "linux"))] - let _ = netns_fd; - - // Drop privileges. initgroups/setgid/setuid need /etc/group and - // /etc/passwd which would be blocked if Landlock were already enforced. - if enforcement_mode.uses_privileged_process_setup() { - drop_privileges_with_identity(policy, resolved_identity) - .map_err(|err| std::io::Error::other(err.to_string()))?; - } - crate::process::harden_child_process() - .map_err(|err| std::io::Error::other(err.to_string()))?; - - // Phase 2: Enforce the prepared Landlock ruleset + seccomp. - // restrict_self() does not require root. - #[cfg(target_os = "linux")] - if let Some(prepared) = prepared { - crate::sandbox::linux::enforce(prepared) - .map_err(|err| std::io::Error::other(err.to_string()))?; - } - - #[cfg(not(target_os = "linux"))] - if enforcement_mode.enforces_child_sandbox() { - sandbox::apply(policy, None).map_err(|err| std::io::Error::other(err.to_string()))?; - } - - Ok(()) - } -} - fn to_u16(value: u32) -> u16 { u16::try_from(value.min(u32::from(u16::MAX))).unwrap_or(u16::MAX) } @@ -1799,60 +1145,259 @@ fn is_loopback_host(host: &str) -> bool { } } +/// Resolve a (loopback-validated) destination host string to an `IpAddr`, +/// mapping `localhost` to `127.0.0.1`. +/// +/// Returns `None` for anything that does not parse to an IP, so +/// [`LoopbackTarget::new`] never sees a hostname. +fn loopback_ip(host: &str) -> Option { + let host = host + .strip_prefix('[') + .and_then(|h| h.strip_suffix(']')) + .unwrap_or(host); + if host.eq_ignore_ascii_case("localhost") { + return Some(std::net::Ipv4Addr::LOCALHOST.into()); + } + host.parse().ok() +} + +fn direct_tcpip_target( + host: &str, + port: u32, +) -> Option { + if !is_loopback_host(host) { + return None; + } + let port = u16::try_from(port).ok()?; + let ip = loopback_ip(host)?; + openshell_isolation_interface::contract::LoopbackTarget::new(ip, port).ok() +} + #[cfg(test)] #[allow( clippy::doc_markdown, - unsafe_code, - reason = "Test code: doc text references identifiers and uses libc::winsize zero-init." + reason = "Test documentation references protocol and API identifiers." )] mod tests { use super::*; - use std::ffi::OsStr; - use std::process::Stdio; + use std::io::Write as _; + use std::process::{Command, Stdio}; + + struct AcceptAnyServerKey; + + impl russh::client::Handler for AcceptAnyServerKey { + type Error = russh::Error; + + async fn check_server_key( + &mut self, + _server_public_key: &russh::keys::PublicKey, + ) -> Result { + Ok(true) + } + } + + struct TestLoopbackConnector; + + #[async_trait::async_trait] + impl openshell_isolation_interface::contract::BoundaryLoopbackConnector for TestLoopbackConnector { + async fn connect( + &self, + target: openshell_isolation_interface::contract::LoopbackTarget, + ) -> std::result::Result< + openshell_isolation_interface::contract::BoundaryDuplexStream, + openshell_isolation_interface::contract::BackendError, + > { + let stream = tokio::net::TcpStream::connect((target.host(), target.port())) + .await + .map_err(|error| { + openshell_isolation_interface::contract::BackendError::Process( + error.to_string(), + ) + })?; + Ok(Box::new(stream)) + } + } + + struct RejectingExec; + + #[async_trait::async_trait] + impl openshell_isolation_interface::contract::BoundaryExec for RejectingExec { + async fn exec( + &self, + _spec: openshell_isolation_interface::contract::ExecSpec, + ) -> std::result::Result< + openshell_isolation_interface::contract::ExecSession, + openshell_isolation_interface::contract::BackendError, + > { + Err( + openshell_isolation_interface::contract::BackendError::Unsupported( + "exec is not used by direct-tcpip tests".into(), + ), + ) + } + } + + async fn authenticated_test_client() -> russh::client::Handle { + let host_key = { + let mut rng = rand::rng(); + PrivateKey::random(&mut rng, Algorithm::Ed25519).expect("host key") + }; + let mut server_config = russh::server::Config { + auth_rejection_time: Duration::from_millis(1), + ..Default::default() + }; + server_config.keys.push(host_key); + + let handler = SshHandler::new( + Arc::new(TestLoopbackConnector), + Arc::new(RejectingExec), + Some(MainSession::inert()), + ); + let (server_stream, client_stream) = tokio::io::duplex(64 * 1024); + tokio::spawn(async move { + if let Ok(session) = + russh::server::run_stream(Arc::new(server_config), server_stream, handler).await + { + let _ = session.await; + } + }); + + let mut client = russh::client::connect_stream( + Arc::new(russh::client::Config::default()), + client_stream, + AcceptAnyServerKey, + ) + .await + .expect("SSH handshake should complete over the duplex"); + let auth = client + .authenticate_none("sandbox") + .await + .expect("auth_none should not error"); + assert!(matches!(auth, russh::client::AuthResult::Success)); + client + } + + #[cfg(unix)] + #[test] + fn transient_accept_errors_retry_with_bounded_backoff() { + let mut resource_errors = 0; + let mut unknown_errors = 0; + let aborted = std::io::Error::from_raw_os_error(libc::ECONNABORTED); + assert_eq!( + classify_ssh_accept_error(&aborted, &mut resource_errors, &mut unknown_errors), + SshAcceptAction::Retry { + backoff: Duration::from_millis(100), + severity: SeverityId::Low, + } + ); + + let exhausted = std::io::Error::from_raw_os_error(libc::EMFILE); + let first = + classify_ssh_accept_error(&exhausted, &mut resource_errors, &mut unknown_errors); + let second = + classify_ssh_accept_error(&exhausted, &mut resource_errors, &mut unknown_errors); + assert_eq!( + first, + SshAcceptAction::Retry { + backoff: Duration::from_millis(100), + severity: SeverityId::Medium, + } + ); + assert_eq!( + second, + SshAcceptAction::Retry { + backoff: Duration::from_millis(200), + severity: SeverityId::Medium, + } + ); + } - /// Regression test: SSH sessions run the shell they are given, never a - /// hardcoded bash, so sh-only images (e.g. Alpine) work. Covers both the - /// interactive PTY path (`-i` when no command) and the non-PTY path. + #[cfg(unix)] #[test] - fn build_ssh_shell_command_uses_given_shell() { - // PTY, no command → given shell + interactive flag. - let cmd = build_ssh_shell_command("/bin/sh", None, false, Some("-i")); - assert_eq!(cmd.get_program(), OsStr::new("/bin/sh")); - assert_eq!(cmd.get_args().collect::>(), vec![OsStr::new("-i")]); - - // Non-PTY, no command → bare shell, no args (reads piped stdin). - let cmd = build_ssh_shell_command("/bin/sh", None, false, None); - assert_eq!(cmd.get_program(), OsStr::new("/bin/sh")); - assert_eq!(cmd.get_args().count(), 0); - - // Explicit command → login-shell flag + command, still on the given shell. - let cmd = build_ssh_shell_command("/bin/sh", Some("echo hi".into()), false, Some("-i")); - assert_eq!(cmd.get_program(), OsStr::new("/bin/sh")); + fn invalid_listener_accept_error_is_terminal() { + let mut resource_errors = 0; + let mut unknown_errors = 0; + let error = std::io::Error::from_raw_os_error(libc::EBADF); assert_eq!( - cmd.get_args().collect::>(), - vec![OsStr::new("-lc"), OsStr::new("echo hi")] + classify_ssh_accept_error(&error, &mut resource_errors, &mut unknown_errors), + SshAcceptAction::Terminal ); + } + + #[test] + fn direct_tcpip_target_rejects_non_loopback_and_out_of_range_ports() { + assert!(direct_tcpip_target("10.0.0.1", 80).is_none()); + assert!(direct_tcpip_target("127.0.0.1", 65_537).is_none()); + } - // OPENSHELL_NO_LOGIN_SHELL → plain -c. - let cmd = build_ssh_shell_command("/bin/sh", Some("echo hi".into()), true, None); + #[test] + fn direct_tcpip_target_accepts_loopback_destinations() { + let target = direct_tcpip_target("localhost", 8_080).expect("loopback target"); assert_eq!( - cmd.get_args().collect::>(), - vec![OsStr::new("-c"), OsStr::new("echo hi")] + target.host(), + std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST) ); + assert_eq!(target.port(), 8_080); + } + + #[tokio::test] + async fn direct_tcpip_handler_rejects_invalid_destinations() { + for (host, port) in [("10.0.0.1", 80), ("127.0.0.1", 65_537)] { + let client = authenticated_test_client().await; + let error = client + .channel_open_direct_tcpip(host, port, "127.0.0.1", 0) + .await + .expect_err("invalid forwarding destination must be refused"); + assert!(matches!( + error, + russh::Error::ChannelOpenFailure(ChannelOpenFailure::AdministrativelyProhibited) + )); + } } - /// Regression test: the direct-tcpip connect path sets `TCP_NODELAY`. #[tokio::test] - async fn connect_in_netns_sets_tcp_nodelay() { + async fn direct_tcpip_handler_relays_loopback_bytes() { + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") .await - .expect("bind listener"); - let addr = listener.local_addr().expect("local addr"); + .expect("bind loopback echo listener"); + let port = listener.local_addr().expect("listener address").port(); + tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.expect("accept forwarded stream"); + let mut payload = [0_u8; 4]; + socket.read_exact(&mut payload).await.expect("read payload"); + socket.write_all(&payload).await.expect("echo payload"); + }); - let stream = connect_in_netns(&addr.to_string(), None) + let client = authenticated_test_client().await; + let channel = client + .channel_open_direct_tcpip("127.0.0.1", u32::from(port), "127.0.0.1", 0) .await - .expect("connect"); - assert!(stream.nodelay().expect("query TCP_NODELAY")); + .expect("loopback forwarding must be allowed"); + let mut stream = channel.into_stream(); + stream.write_all(b"ping").await.expect("write channel"); + let mut echoed = [0_u8; 4]; + tokio::time::timeout(Duration::from_secs(5), stream.read_exact(&mut echoed)) + .await + .expect("forwarded response timeout") + .expect("read channel"); + assert_eq!(&echoed, b"ping"); + } + + #[tokio::test] + async fn main_attachment_accepts_declared_session_after_process_exit() { + let main_session = MainSession::inert(); + assert!(main_session.finish(23, true).await); + assert!(main_session.finished()); + + assert!(begin_main_attachment(&main_session, true)); + let mut output = main_session.subscribe(); + assert!(matches!( + output.recv().await.expect("retained terminal status"), + MainOutput::Exit(23) + )); + main_session.end_terminal_attachment(); } #[cfg(unix)] @@ -1869,15 +1414,14 @@ mod tests { #[cfg(unix)] #[tokio::test] - async fn ssh_server_init_full_enforcement_keeps_private_socket() { + async fn ssh_server_init_keeps_private_socket() { let temp = tempfile::tempdir().unwrap(); let parent = temp.path().join("ssh"); std::fs::create_dir_all(&parent).unwrap(); set_file_mode(&parent, 0o775); let socket = parent.join("ssh.sock"); - let (listener, _, _) = - ssh_server_init(&socket, &None, ProcessEnforcementMode::Full, false).unwrap(); + let (listener, _, _) = ssh_server_init(&socket, &None, false).unwrap(); drop(listener); assert_eq!(file_mode(&parent), 0o700); @@ -1893,8 +1437,7 @@ mod tests { set_file_mode(&parent, 0o775); let socket = parent.join("ssh.sock"); - let (listener, _, _) = - ssh_server_init(&socket, &None, ProcessEnforcementMode::Full, true).unwrap(); + let (listener, _, _) = ssh_server_init(&socket, &None, true).unwrap(); drop(listener); assert_eq!(file_mode(&parent), 0o775); @@ -1905,8 +1448,7 @@ mod tests { #[tokio::test] async fn ssh_server_abstract_socket_cannot_be_replaced_while_bound() { let socket = PathBuf::from(format!("@openshell-ssh-test-{}", uuid::Uuid::new_v4())); - let (listener, _, _) = - ssh_server_init(&socket, &None, ProcessEnforcementMode::NetworkOnly, true).unwrap(); + let (listener, _, _) = ssh_server_init(&socket, &None, true).unwrap(); assert!( !socket.exists(), @@ -1971,41 +1513,9 @@ mod tests { assert_eq!(output.stdout, b"hello"); } - /// Command execution selects a login shell by default and a non-login shell - /// under `--no-login-shell`, so user startup files are sourced only in the - /// default case. - #[cfg(unix)] - #[test] - fn login_shell_flag_controls_profile_sourcing() { - let home = tempfile::tempdir().unwrap(); - std::fs::write(home.path().join(".bash_profile"), "echo LOGIN_MARKER\n").unwrap(); - - let run = |flag: &str| -> String { - let out = Command::new("bash") - .arg(flag) - .arg("true") - .env("HOME", home.path()) - .env_remove("BASH_ENV") // isolate: -c still reads BASH_ENV if set - .output() - .expect("spawn bash"); - String::from_utf8_lossy(&out.stdout).into_owned() - }; - - assert_eq!(login_shell_flag(true), "-c"); - assert_eq!(login_shell_flag(false), "-lc"); - assert!( - run("-lc").contains("LOGIN_MARKER"), - "login shell must source .bash_profile" - ); - assert!( - !run("-c").contains("LOGIN_MARKER"), - "non-login shell must not source it" - ); - } - - /// Verify that the stdin writer delivers all buffered data before exiting - /// when the sender is dropped. This ensures channel_eof doesn't cause - /// data loss — only signals "no more data after this". + /// Verify that the stdin writer delivers all buffered data before exiting + /// when the sender is dropped. This ensures channel_eof doesn't cause + /// data loss — only signals "no more data after this". #[test] fn stdin_writer_delivers_buffered_data_before_eof() { let (sender, receiver) = mpsc::channel::>(); @@ -2105,62 +1615,6 @@ mod tests { assert!(!is_loopback_host("[]")); } - // ----------------------------------------------------------------------- - // Per-channel PTY state tests (#543) - // ----------------------------------------------------------------------- - - #[test] - fn set_winsize_applies_to_correct_pty() { - // Verify that set_winsize applies to a specific PTY master FD, - // which is the mechanism that per-channel tracking relies on. - // With the old single-pty_master design, a window_change_request - // for channel N would resize whatever PTY was stored last — - // potentially belonging to a different channel. - let pty_a = openpty(None, None).expect("openpty a"); - let pty_b = openpty(None, None).expect("openpty b"); - let master_a = std::fs::File::from(pty_a.master); - let master_b = std::fs::File::from(pty_b.master); - let fd_a = master_a.as_raw_fd(); - let fd_b = master_b.as_raw_fd(); - assert_ne!(fd_a, fd_b, "two PTYs must have distinct FDs"); - - // Close the slave ends to avoid leaking FDs in the test. - drop(std::fs::File::from(pty_a.slave)); - drop(std::fs::File::from(pty_b.slave)); - - // Resize only PTY B. - let winsize_b = Winsize { - ws_row: 50, - ws_col: 120, - ws_xpixel: 0, - ws_ypixel: 0, - }; - unsafe_pty::set_winsize(fd_b, winsize_b).expect("set_winsize on PTY B"); - - // Resize PTY A to a different size. - let winsize_a = Winsize { - ws_row: 24, - ws_col: 80, - ws_xpixel: 0, - ws_ypixel: 0, - }; - unsafe_pty::set_winsize(fd_a, winsize_a).expect("set_winsize on PTY A"); - - // Read back sizes via ioctl to verify independence. - let mut actual_a: libc::winsize = unsafe { std::mem::zeroed() }; - let mut actual_b: libc::winsize = unsafe { std::mem::zeroed() }; - #[allow(unsafe_code)] - unsafe { - libc::ioctl(fd_a, libc::TIOCGWINSZ, &mut actual_a); - libc::ioctl(fd_b, libc::TIOCGWINSZ, &mut actual_b); - } - - assert_eq!(actual_a.ws_row, 24, "PTY A should be 24 rows"); - assert_eq!(actual_a.ws_col, 80, "PTY A should be 80 cols"); - assert_eq!(actual_b.ws_row, 50, "PTY B should be 50 rows"); - assert_eq!(actual_b.ws_col, 120, "PTY B should be 120 cols"); - } - #[test] fn channel_state_independent_input_senders() { // Verify that each channel gets its own input sender so that @@ -2211,604 +1665,4 @@ mod tests { .unwrap(); assert_eq!(rx_b.recv().unwrap(), b"still-alive"); } - - #[test] - fn main_detach_filter_forwards_ctrl_c_unchanged() { - let mut prefix_pending = false; - let (forward, detach) = - filter_main_detach_sequence(&mut prefix_pending, b"before\x03after"); - - assert_eq!(forward, b"before\x03after"); - assert!(!detach); - assert!(!prefix_pending); - } - - #[test] - fn main_detach_filter_removes_sequence_and_trailing_input() { - let mut prefix_pending = false; - let (forward, detach) = - filter_main_detach_sequence(&mut prefix_pending, b"before\x10\x11after"); - - assert_eq!(forward, b"before"); - assert!(detach); - assert!(!prefix_pending); - } - - #[test] - fn main_detach_filter_recognizes_sequence_across_frames() { - let mut prefix_pending = false; - let (forward, detach) = filter_main_detach_sequence(&mut prefix_pending, b"before\x10"); - assert_eq!(forward, b"before"); - assert!(!detach); - assert!(prefix_pending); - - let (forward, detach) = filter_main_detach_sequence(&mut prefix_pending, b"\x11"); - assert!(forward.is_empty()); - assert!(detach); - assert!(!prefix_pending); - } - - #[test] - fn main_detach_filter_forwards_unmatched_prefix() { - let mut prefix_pending = false; - let (forward, detach) = filter_main_detach_sequence(&mut prefix_pending, b"\x10"); - assert!(forward.is_empty()); - assert!(!detach); - assert!(prefix_pending); - - let (forward, detach) = filter_main_detach_sequence(&mut prefix_pending, b"x"); - assert_eq!(forward, b"\x10x"); - assert!(!detach); - assert!(!prefix_pending); - } - - // ----------------------------------------------------------------------- - // session_user_and_home tests (Phase 2: numeric UID support) - // ----------------------------------------------------------------------- - - #[test] - fn session_user_and_home_returns_numeric_uid_as_user() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, - }; - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: Some("1000".into()), - run_as_group: None, - }, - }; - let (user, home) = session_user_and_home(&policy, None); - assert_eq!(user, "1000"); - // Numeric UID has no passwd entry — defaults to /sandbox. - assert_eq!(home, "/sandbox"); - } - - #[test] - fn session_user_and_home_uses_driver_workspace_when_supplied() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, - }; - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: Some("1234".into()), - run_as_group: Some("1235".into()), - }, - }; - - let (user, home) = session_user_and_home(&policy, Some("/workspace/project")); - assert_eq!(user, "1234"); - assert_eq!(home, "/workspace/project"); - } - - #[test] - fn session_user_and_home_returns_name_from_passwd() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, - }; - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: Some("sandbox".into()), - run_as_group: None, - }, - }; - let (user, home) = session_user_and_home(&policy, None); - assert_eq!(user, "sandbox"); - // Name-based — should resolve via passwd (or /home/{user}). - assert!(!home.is_empty()); - } - - #[test] - fn session_user_and_home_defaults_to_sandbox_when_empty() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, - }; - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: Some(String::new()), - run_as_group: None, - }, - }; - let (user, home) = session_user_and_home(&policy, None); - assert_eq!(user, "sandbox"); - assert_eq!(home, "/sandbox"); - } - - #[test] - fn session_user_and_home_defaults_to_sandbox_when_none() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, - }; - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: None, - run_as_group: None, - }, - }; - let (user, home) = session_user_and_home(&policy, None); - assert_eq!(user, "sandbox"); - assert_eq!(home, "/sandbox"); - } - - #[test] - fn session_user_and_home_handles_large_numeric_uid() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, - }; - let policy = SandboxPolicy { - version: 1, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: Some("1000660000".into()), - run_as_group: None, - }, - }; - let (user, home) = session_user_and_home(&policy, None); - assert_eq!(user, "1000660000"); - assert_eq!(home, "/sandbox"); - } - - /// `install_pre_exec_no_pty` runs drop_privileges and succeeds when the - /// current user/group is already the configured one (no actual uid change). - /// - /// This exercises the pre_exec hook end-to-end without needing root: a policy - /// with no run_as_user/group is a no-op when the process is already unprivileged. - #[cfg(unix)] - #[test] - fn pre_exec_always_calls_drop_privileges() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, SandboxPolicy, - }; - - // No user/group configured and not running as root → drop_privileges is - // a no-op, so spawn succeeds regardless of the effective UID. - let policy = SandboxPolicy { - version: 0, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: None, - run_as_group: None, - }, - }; - - // Skip if running as root: drop_privileges would try to switch to - // "sandbox" which may not exist in the test environment. - if rustix::process::geteuid().is_root() { - return; - } - - let mut cmd = Command::new("echo"); - cmd.arg("drop-privileges-ok"); - cmd.stdout(Stdio::piped()); - - unsafe_pty::install_pre_exec_no_pty( - &mut cmd, - policy, - None, - None, // no netns fd - ResolvedProcessIdentity::default(), - ProcessEnforcementMode::Full, - #[cfg(target_os = "linux")] - Some( - sandbox::linux::prepare( - &SandboxPolicy { - version: 0, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: None, - run_as_group: None, - }, - }, - None, - ) - .expect("prepare should succeed in test environment"), - ), - ); - - let output = cmd - .spawn() - .expect("spawn must succeed") - .wait_with_output() - .expect("wait_with_output"); - assert!(output.status.success(), "echo should exit 0"); - assert!( - String::from_utf8_lossy(&output.stdout).contains("drop-privileges-ok"), - "echo output should contain 'drop-privileges-ok'" - ); - } - - /// SSH pre-exec uses the numeric identity resolved from OCI metadata rather - /// than looking the preserved declaration up through host NSS. - #[cfg(unix)] - #[test] - fn pre_exec_uses_resolved_oci_identity() { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, SandboxPolicy, - }; - - if rustix::process::geteuid().is_root() { - return; - } - - let policy = SandboxPolicy { - version: 0, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: Some("__oci_user_not_in_host_nss__".into()), - run_as_group: Some("__oci_group_not_in_host_nss__".into()), - }, - }; - let resolved = ResolvedProcessIdentity::new( - Some(rustix::process::geteuid().as_raw()), - Some(rustix::process::getegid().as_raw()), - ); - - let mut cmd = Command::new("echo"); - cmd.arg("resolved-identity-ok"); - cmd.stdout(Stdio::piped()); - - unsafe_pty::install_pre_exec_no_pty( - &mut cmd, - policy, - None, - None, - resolved, - ProcessEnforcementMode::Full, - #[cfg(target_os = "linux")] - None, - ); - - let output = cmd - .spawn() - .expect("spawn should use resolved numeric identity") - .wait_with_output() - .expect("wait should succeed"); - assert!(output.status.success()); - assert_eq!( - String::from_utf8_lossy(&output.stdout).trim(), - "resolved-identity-ok" - ); - } - - // ----------------------------------------------------------------------- - // direct-tcpip authorization wiring (SEC-007) - // - // The `loopback_host_*` tests above cover the predicate in isolation. - // These drive the real `russh::server::Handler` over an in-memory duplex - // so the deny path itself is covered: channel-open authorization travels - // through a reply handle rather than the handler's return value, so a - // handler that never rejects anything still type-checks and still passes - // every predicate test. - // ----------------------------------------------------------------------- - - struct AcceptAnyServerKey; - - impl russh::client::Handler for AcceptAnyServerKey { - type Error = russh::Error; - - async fn check_server_key( - &mut self, - _server_public_key: &russh::keys::PublicKey, - ) -> Result { - Ok(true) - } - } - - fn forwarding_test_policy() -> SandboxPolicy { - use openshell_core::policy::{ - FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, - }; - - SandboxPolicy { - version: 0, - filesystem: FilesystemPolicy::default(), - network: NetworkPolicy::default(), - landlock: LandlockPolicy::default(), - process: ProcessPolicy { - run_as_user: None, - run_as_group: None, - }, - } - } - - /// Serve `SshHandler` on one end of an in-memory duplex and return an - /// authenticated client handle for the other end. - /// - /// The handler gets `netns_fd: None` so `connect_in_netns` performs a plain - /// TCP connect, making the forwarding path reachable without a network - /// namespace. - async fn authenticated_test_client_with_main( - main_session: Arc, - ) -> russh::client::Handle { - // Scoped so the `!Send` ThreadRng is dropped before the first await. - let host_key = { - let mut rng = rand::rng(); - PrivateKey::random(&mut rng, Algorithm::Ed25519).expect("host key") - }; - let mut server_config = russh::server::Config { - auth_rejection_time: Duration::from_millis(1), - ..Default::default() - }; - server_config.keys.push(host_key); - - let handler = SshHandler::new( - forwarding_test_policy(), - ResolvedWorkspace::default(), - None, - None, - None, - ProviderCredentialState::from_child_env_snapshot(0, HashMap::new()), - HashMap::new(), - ResolvedProcessIdentity::default(), - ProcessEnforcementMode::NetworkOnly, - main_session, - ); - - let (server_stream, client_stream) = tokio::io::duplex(64 * 1024); - tokio::spawn(async move { - if let Ok(session) = - russh::server::run_stream(Arc::new(server_config), server_stream, handler).await - { - let _ = session.await; - } - }); - - let mut client = russh::client::connect_stream( - Arc::new(russh::client::Config::default()), - client_stream, - AcceptAnyServerKey, - ) - .await - .expect("SSH handshake should complete over the duplex"); - - let auth = client - .authenticate_none("sandbox") - .await - .expect("auth_none should not error"); - assert!( - matches!(auth, russh::client::AuthResult::Success), - "sandbox SSH server accepts the none auth method" - ); - - client - } - - async fn authenticated_test_client() -> russh::client::Handle { - authenticated_test_client_with_main(MainSession::inert()).await - } - - #[tokio::test] - async fn abrupt_transport_drop_releases_main_input_lease() { - let main_session = MainSession::inert(); - let client = authenticated_test_client_with_main(Arc::clone(&main_session)).await; - let channel = client.channel_open_session().await.expect("open session"); - channel - .request_subsystem(true, "openshell-main") - .await - .expect("attach main subsystem"); - tokio::time::timeout(Duration::from_secs(1), async { - loop { - match main_session.acquire_input() { - Err(_) => break, - Ok((owner, _)) => main_session.release_input(owner), - } - tokio::task::yield_now().await; - } - }) - .await - .expect("main subsystem should acquire canonical input lease"); - - drop(channel); - drop(client); - tokio::time::timeout(Duration::from_secs(1), async { - loop { - if main_session.acquire_input().is_ok() { - break; - } - tokio::task::yield_now().await; - } - }) - .await - .expect("handler drop should release canonical input lease"); - } - - #[tokio::test] - async fn main_attachment_closes_naturally_after_terminal_delivery() { - let main_session = MainSession::inert(); - let client = authenticated_test_client_with_main(Arc::clone(&main_session)).await; - let mut channel = client.channel_open_session().await.expect("open session"); - channel - .request_subsystem(true, "openshell-main") - .await - .expect("attach main subsystem"); - tokio::time::timeout(Duration::from_secs(1), async { - loop { - match main_session.acquire_input() { - Err(_) => break, - Ok((owner, _)) => main_session.release_input(owner), - } - tokio::task::yield_now().await; - } - }) - .await - .expect("main subsystem should register its attachment"); - - assert!(main_session.finish(7, false).await); - main_session.mark_terminal_reported(); - - let exit_status = tokio::time::timeout(Duration::from_secs(1), async { - let mut exit_status = None; - loop { - match channel.wait().await { - Some(russh::ChannelMsg::ExitStatus { - exit_status: status, - }) => { - exit_status = Some(status); - } - Some(russh::ChannelMsg::Close) => break exit_status, - None => panic!("main channel ended without a close message"), - Some(_) => {} - } - } - }) - .await - .expect("main channel should deliver its exit status"); - assert_eq!(exit_status, Some(7)); - drop(channel); - drop(client); - - tokio::time::timeout( - Duration::from_secs(1), - main_session.wait_for_terminal_attachments(), - ) - .await - .expect("peer channel close should release terminal delivery"); - } - - #[tokio::test] - async fn main_subsystem_applies_initial_pty_dimensions() { - let (main_session, _slave) = MainSession::terminal_for_test(); - let client = authenticated_test_client_with_main(Arc::clone(&main_session)).await; - let channel = client.channel_open_session().await.expect("open session"); - channel - .request_pty(true, "xterm-256color", 200, 60, 1600, 900, &[]) - .await - .expect("request PTY"); - channel - .request_subsystem(true, "openshell-main") - .await - .expect("attach main subsystem"); - - tokio::time::timeout(Duration::from_secs(1), async { - loop { - if main_session.terminal_size_for_test() == (200, 60) { - break; - } - tokio::task::yield_now().await; - } - }) - .await - .expect("main subsystem should apply the initial PTY dimensions"); - } - - #[tokio::test] - async fn direct_tcpip_rejects_non_loopback_destination() { - let client = authenticated_test_client().await; - - let err = client - .channel_open_direct_tcpip("10.0.0.1", 80, "127.0.0.1", 0) - .await - .expect_err("forwarding to a non-loopback host must be refused"); - - assert!( - matches!( - err, - russh::Error::ChannelOpenFailure(ChannelOpenFailure::AdministrativelyProhibited) - ), - "expected AdministrativelyProhibited, got {err:?}" - ); - } - - #[tokio::test] - async fn direct_tcpip_rejects_port_above_tcp_range() { - let client = authenticated_test_client().await; - - // 65_537 truncates to port 1 when cast to u16, so the guard has to - // reject it before the cast rather than forward to a privileged port. - let err = client - .channel_open_direct_tcpip("127.0.0.1", 65_537, "127.0.0.1", 0) - .await - .expect_err("a port outside the TCP range must be refused"); - - assert!( - matches!( - err, - russh::Error::ChannelOpenFailure(ChannelOpenFailure::AdministrativelyProhibited) - ), - "expected AdministrativelyProhibited, got {err:?}" - ); - } - - #[tokio::test] - async fn direct_tcpip_forwards_to_loopback_listener() { - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - - let listener = tokio::net::TcpListener::bind("127.0.0.1:0") - .await - .expect("bind loopback echo listener"); - let port = listener.local_addr().expect("listener address").port(); - tokio::spawn(async move { - if let Ok((mut socket, _)) = listener.accept().await { - let mut buf = [0u8; 64]; - if let Ok(n) = socket.read(&mut buf).await - && n > 0 - { - let _ = socket.write_all(&buf[..n]).await; - } - } - }); - - let client = authenticated_test_client().await; - let channel = client - .channel_open_direct_tcpip("127.0.0.1", u32::from(port), "127.0.0.1", 0) - .await - .expect("forwarding to a loopback listener must be allowed"); - - let mut stream = channel.into_stream(); - stream.write_all(b"ping").await.expect("write to channel"); - - let mut echoed = [0u8; 4]; - tokio::time::timeout(Duration::from_secs(10), stream.read_exact(&mut echoed)) - .await - .expect("relayed response should arrive before the timeout") - .expect("read from channel"); - assert_eq!(&echoed, b"ping", "bytes round-trip through the tunnel"); - } } diff --git a/crates/openshell-supervisor-process/src/supervisor_session.rs b/crates/openshell-supervisor-process/src/supervisor_session.rs index a8f60a5681..52a73b02b6 100644 --- a/crates/openshell-supervisor-process/src/supervisor_session.rs +++ b/crates/openshell-supervisor-process/src/supervisor_session.rs @@ -11,8 +11,6 @@ //! selection — it has no protocol awareness of the bytes flowing through. use std::net::IpAddr; -#[cfg(target_os = "linux")] -use std::os::fd::RawFd; use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; use std::time::Duration; @@ -23,6 +21,7 @@ use openshell_core::proto::{ RelayOpenResult, ReportMainProcessExitRequest, SupervisorHeartbeat, SupervisorHello, SupervisorMessage, TcpRelayTarget, gateway_message, relay_open, supervisor_message, }; +use openshell_isolation_interface::contract::{BoundaryLoopbackConnector, LoopbackTarget}; use openshell_ocsf::{ ActivityId, ConnectionInfo, Endpoint, EventContext, NetworkActivityBuilder, OcsfEvent, SeverityId, StatusId, ocsf_emit, @@ -33,7 +32,6 @@ use tokio_stream::StreamExt; use tracing::{debug, warn}; use openshell_core::grpc_client; -use openshell_core::net::set_tcp_nodelay_best_effort; use openshell_core::transport_errors::is_expected_transport_close_status; const INITIAL_BACKOFF: Duration = Duration::from_secs(1); @@ -274,31 +272,59 @@ pub fn spawn( endpoint: String, sandbox_id: String, ssh_socket_path: std::path::PathBuf, - netns_fd: Option, + port_forward: Arc, expected_ssh_peer_pid: Option, terminating: Arc, instance_id: String, ) -> tokio::task::JoinHandle<()> { + spawn_with_readiness( + endpoint, + sandbox_id, + ssh_socket_path, + port_forward, + expected_ssh_peer_pid, + terminating, + instance_id, + ) + .0 +} + +/// Spawn the supervisor session and expose when the gateway has accepted it. +pub fn spawn_with_readiness( + endpoint: String, + sandbox_id: String, + ssh_socket_path: std::path::PathBuf, + port_forward: Arc, + expected_ssh_peer_pid: Option, + terminating: Arc, + instance_id: String, +) -> ( + tokio::task::JoinHandle<()>, + tokio::sync::watch::Receiver, +) { + let (ready_tx, ready_rx) = tokio::sync::watch::channel(false); let config = SessionConfig { endpoint, sandbox_id, ssh_socket_path, - netns_fd, + port_forward, expected_ssh_peer_pid, terminating, instance_id, + ready_tx, }; - tokio::spawn(run_session_loop(config)) + (tokio::spawn(run_session_loop(config)), ready_rx) } struct SessionConfig { endpoint: String, sandbox_id: String, ssh_socket_path: std::path::PathBuf, - netns_fd: Option, + port_forward: Arc, expected_ssh_peer_pid: Option, terminating: Arc, instance_id: String, + ready_tx: tokio::sync::watch::Sender, } async fn run_session_loop(config: SessionConfig) { @@ -310,6 +336,7 @@ async fn run_session_loop(config: SessionConfig) { match run_single_session(&config).await { Ok(()) => { + config.ready_tx.send_replace(false); let event = session_closed_event( openshell_ocsf::ctx::ctx(), &config.endpoint, @@ -319,6 +346,7 @@ async fn run_session_loop(config: SessionConfig) { break; } Err(e) => { + config.ready_tx.send_replace(false); let event = session_failed_event( openshell_ocsf::ctx::ctx(), &config.endpoint, @@ -388,6 +416,8 @@ async fn run_single_session( heartbeat_secs, ); ocsf_emit!(event); + config.ready_tx.send_replace(true); + // Main loop: receive gateway messages + send heartbeats. let mut heartbeat_interval = tokio::time::interval(Duration::from_secs(u64::from(heartbeat_secs))); @@ -407,7 +437,7 @@ async fn run_single_session( let context = GatewayMessageContext { sandbox_id: &config.sandbox_id, ssh_socket_path: &config.ssh_socket_path, - netns_fd: config.netns_fd, + port_forward: &config.port_forward, expected_ssh_peer_pid: config.expected_ssh_peer_pid, channel: &channel, tx: &tx, @@ -475,7 +505,7 @@ pub async fn finalize_main_process_exit( struct GatewayMessageContext<'a> { sandbox_id: &'a str, ssh_socket_path: &'a std::path::Path, - netns_fd: Option, + port_forward: &'a Arc, expected_ssh_peer_pid: Option, channel: &'a grpc_client::AuthedChannel, tx: &'a mpsc::Sender, @@ -494,7 +524,7 @@ fn handle_gateway_message(msg: &GatewayMessage, context: &GatewayMessageContext< let channel = context.channel.clone(); let ssh_socket_path = context.ssh_socket_path.to_path_buf(); let tx = context.tx.clone(); - let netns_fd = context.netns_fd; + let port_forward = context.port_forward.clone(); let expected_ssh_peer_pid = context.expected_ssh_peer_pid; let terminating = Arc::clone(context.terminating); @@ -506,7 +536,7 @@ fn handle_gateway_message(msg: &GatewayMessage, context: &GatewayMessageContext< match handle_relay_open( relay_open, &ssh_socket_path, - netns_fd, + port_forward, expected_ssh_peer_pid, channel, tx, @@ -563,7 +593,7 @@ fn handle_gateway_message(msg: &GatewayMessage, context: &GatewayMessageContext< async fn handle_relay_open( relay_open: RelayOpen, ssh_socket_path: &std::path::Path, - netns_fd: Option, + port_forward: Arc, expected_ssh_peer_pid: Option, channel: grpc_client::AuthedChannel, tx: mpsc::Sender, @@ -573,7 +603,7 @@ async fn handle_relay_open( let target = match open_target( &relay_open, ssh_socket_path, - netns_fd, + &port_forward, expected_ssh_peer_pid, ) .await @@ -718,11 +748,11 @@ async fn send_relay_open_result( async fn open_target( relay_open: &RelayOpen, ssh_socket_path: &std::path::Path, - netns_fd: Option, + port_forward: &Arc, expected_ssh_peer_pid: Option, ) -> Result, Box> { match relay_open.target.as_ref() { - Some(relay_open::Target::Tcp(target)) => open_tcp_target(target, netns_fd).await, + Some(relay_open::Target::Tcp(target)) => open_tcp_target(target, port_forward).await, Some(relay_open::Target::Ssh(_)) | None => { let runtime_path = crate::unix_socket::runtime_path(ssh_socket_path); let stream = tokio::net::UnixStream::connect(runtime_path.as_ref()).await?; @@ -743,59 +773,26 @@ async fn open_target( async fn open_tcp_target( target: &TcpRelayTarget, - netns_fd: Option, + port_forward: &Arc, ) -> Result, Box> { let host = normalize_tcp_target_host(target)?; let port = u16::try_from(target.port).map_err(|_| "tcp target port must fit in u16")?; - let stream = connect_tcp_target(host, port, netns_fd).await?; + // `normalize_tcp_target_host` returns a loopback IP string; parse it and let + // `LoopbackTarget::new` re-validate before connecting. + let ip: IpAddr = host + .parse() + .map_err(|_| "tcp target host must be a loopback IP")?; + let target = LoopbackTarget::new(ip, port) + .map_err(|e| -> Box { e.to_string().into() })?; + // Connect through the sandbox-owned loopback-forward interface. The + // supervisor session remains independent of the driver's transport. + let stream = port_forward + .connect(target) + .await + .map_err(|e| -> Box { e.to_string().into() })?; Ok(Box::new(stream)) } -#[cfg(target_os = "linux")] -async fn connect_tcp_target( - host: String, - port: u16, - netns_fd: Option, -) -> Result> { - if let Some(fd) = netns_fd { - let (tx, rx) = tokio::sync::oneshot::channel(); - std::thread::spawn(move || { - let result = (|| -> std::io::Result { - #[allow(unsafe_code)] - let rc = unsafe { libc::setns(fd, libc::CLONE_NEWNET) }; - if rc != 0 { - return Err(std::io::Error::last_os_error()); - } - std::net::TcpStream::connect((host.as_str(), port)) - })(); - let _ = tx.send(result); - }); - - let stream = rx - .await - .map_err(|_| "netns tcp connect thread panicked")??; - stream.set_nonblocking(true)?; - let stream = tokio::net::TcpStream::from_std(stream)?; - set_tcp_nodelay_best_effort(&stream); - return Ok(stream); - } - - let stream = tokio::net::TcpStream::connect((host.as_str(), port)).await?; - set_tcp_nodelay_best_effort(&stream); - Ok(stream) -} - -#[cfg(not(target_os = "linux"))] -async fn connect_tcp_target( - host: String, - port: u16, - _netns_fd: Option, -) -> Result> { - let stream = tokio::net::TcpStream::connect((host.as_str(), port)).await?; - set_tcp_nodelay_best_effort(&stream); - Ok(stream) -} - #[cfg(test)] fn validate_tcp_target(target: &TcpRelayTarget) -> Result<(), String> { normalize_tcp_target_host(target).map(|_| ()) @@ -835,20 +832,6 @@ mod target_tests { } } - /// Regression test: the TCP relay connect path sets `TCP_NODELAY`. - #[tokio::test] - async fn connect_tcp_target_sets_tcp_nodelay() { - let listener = tokio::net::TcpListener::bind("127.0.0.1:0") - .await - .expect("bind listener"); - let addr = listener.local_addr().expect("local addr"); - - let stream = connect_tcp_target(addr.ip().to_string(), addr.port(), None) - .await - .expect("connect"); - assert!(stream.nodelay().expect("query TCP_NODELAY")); - } - #[test] fn tcp_target_allows_loopback_hosts() { validate_tcp_target(&tcp("127.0.0.1", 8080)).expect("ipv4 loopback"); @@ -891,6 +874,23 @@ mod target_tests { mod ocsf_event_tests { use super::*; + #[cfg(target_os = "linux")] + struct UnusedLoopbackConnector; + + #[cfg(target_os = "linux")] + #[async_trait::async_trait] + impl BoundaryLoopbackConnector for UnusedLoopbackConnector { + async fn connect( + &self, + _target: LoopbackTarget, + ) -> Result< + openshell_isolation_interface::contract::BoundaryDuplexStream, + openshell_isolation_interface::contract::BackendError, + > { + unreachable!("SSH relay does not use loopback port forwarding") + } + } + fn ctx() -> EventContext { EventContext { sandbox_id: "sbx-1".into(), @@ -1131,7 +1131,11 @@ mod ocsf_event_tests { }); let relay = ssh_relay_open("peer-check"); - let trusted = open_target(&relay, &socket, None, Some(std::process::id())) + // The SSH relay path does not use the port-forward (that is the TCP + // target path); connect from the supervisor's own namespace. + let port_forward: Arc = Arc::new(UnusedLoopbackConnector); + + let trusted = open_target(&relay, &socket, &port_forward, Some(std::process::id())) .await .expect("matching peer PID should be accepted"); drop(trusted); @@ -1139,7 +1143,7 @@ mod ocsf_event_tests { let Err(err) = open_target( &relay, &socket, - None, + &port_forward, Some(std::process::id().saturating_add(1)), ) .await diff --git a/crates/openshell-supervisor/Cargo.toml b/crates/openshell-supervisor/Cargo.toml new file mode 100644 index 0000000000..351e4cce66 --- /dev/null +++ b/crates/openshell-supervisor/Cargo.toml @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-supervisor" +description = "OpenShell policy and workload supervisor" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[[bin]] +name = "openshell-supervisor" +path = "src/main.rs" + +[dependencies] +openshell-core = { path = "../openshell-core", default-features = false, features = ["jwt"] } +openshell-extension-core = { path = "../openshell-extension-core" } +openshell-isolation-interface = { path = "../openshell-isolation-interface" } +openshell-sandbox-backend = { path = "../openshell-sandbox-backend" } +openshell-ocsf = { path = "../openshell-ocsf" } +openshell-policy = { path = "../openshell-policy" } +openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } +openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" } +openshell-supervisor-network = { path = "../openshell-supervisor-network", default-features = false } +openshell-supervisor-process = { path = "../openshell-supervisor-process" } + +clap = { workspace = true } +miette = { workspace = true } +nix = { workspace = true } +prost = { workspace = true } +prost-types = { workspace = true } +rustls = { workspace = true } +rustix = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +tokio = { workspace = true } +tonic = { workspace = true, features = ["channel", "tls-native-roots"] } +tracing = { workspace = true } +tracing-appender = { workspace = true } +tracing-subscriber = { workspace = true } +uuid = { workspace = true } + +[features] +default = ["telemetry", "bundled-ca-roots"] +system-ca-roots = ["telemetry"] +defaults-without-telemetry = ["bundled-ca-roots"] +telemetry = ["openshell-core/telemetry"] +bundled-ca-roots = ["openshell-supervisor-network/bundled-ca-roots"] + +[dev-dependencies] +futures = { workspace = true } +temp-env = "0.3" +tempfile = "3" +tokio-tungstenite = { workspace = true } + +[lints] +workspace = true diff --git a/crates/openshell-supervisor/src/activity_aggregator.rs b/crates/openshell-supervisor/src/activity_aggregator.rs new file mode 100644 index 0000000000..33605c1df9 --- /dev/null +++ b/crates/openshell-supervisor/src/activity_aggregator.rs @@ -0,0 +1,224 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Anonymous supervised network activity counter aggregation. +//! +//! Producer-side types (`ActivityEvent`, `ActivitySender`, +//! `ACTIVITY_EVENT_QUEUE_CAPACITY`, `try_record_activity`) live in +//! `openshell_core::activity` so the supervisor leaves can emit without +//! depending on the orchestrator. This module hosts the aggregator that +//! runs orchestrator-side and flushes summaries to the gateway. + +use std::collections::HashMap; +use std::future::Future; +use tokio::sync::mpsc; +use tracing::{debug, warn}; + +pub use openshell_core::activity::ActivityEvent; + +const ACTIVITY_FLUSH_QUEUE_CAPACITY: usize = 1; +pub const DEFAULT_ACTIVITY_FLUSH_INTERVAL_SECS: u64 = 10; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct FlushableActivitySummary { + pub network_activity_count: u32, + pub denied_action_count: u32, + pub denials_by_group: Vec<(String, u32)>, +} + +pub struct ActivityAggregator { + rx: mpsc::Receiver, + network_activity_count: u32, + denied_action_count: u32, + denials_by_group: HashMap, + flush_interval_secs: u64, +} + +impl ActivityAggregator { + pub fn new(rx: mpsc::Receiver, flush_interval_secs: u64) -> Self { + Self { + rx, + network_activity_count: 0, + denied_action_count: 0, + denials_by_group: HashMap::new(), + flush_interval_secs, + } + } + + /// `ready_gate` is checked before each flush. When it returns `false`, + /// the drain is skipped and events stay in the buffer until the next tick. + pub async fn run(mut self, flush_callback: F, ready_gate: G) + where + F: Fn(FlushableActivitySummary) -> Fut + Send + 'static, + Fut: Future + Send + 'static, + G: Fn() -> bool, + { + let (flush_tx, mut flush_rx) = + mpsc::channel::(ACTIVITY_FLUSH_QUEUE_CAPACITY); + tokio::spawn(async move { + while let Some(summary) = flush_rx.recv().await { + flush_callback(summary).await; + } + }); + + let mut flush_interval = + tokio::time::interval(std::time::Duration::from_secs(self.flush_interval_secs)); + flush_interval.tick().await; + + loop { + tokio::select! { + event = self.rx.recv() => { + if let Some(event) = event { + self.ingest(event); + } else { + if self.network_activity_count > 0 { + if ready_gate() { + if let Some(summary) = self.drain() { + queue_flush_summary(&flush_tx, summary); + } + } else { + warn!( + count = self.network_activity_count, + "ActivityAggregator: dropping unflushed events, workspace not yet known" + ); + } + } + debug!("ActivityAggregator: channel closed, exiting"); + return; + } + } + _ = flush_interval.tick() => { + if ready_gate() + && let Some(summary) = self.drain() + { + debug!( + count = summary.network_activity_count, + denied = summary.denied_action_count, + "ActivityAggregator: flushing anonymous activity summary" + ); + queue_flush_summary(&flush_tx, summary); + } + } + } + } + } + + fn ingest(&mut self, event: ActivityEvent) { + self.network_activity_count = self.network_activity_count.saturating_add(1); + if event.denied { + self.denied_action_count = self.denied_action_count.saturating_add(1); + let group = sanitize_deny_group(event.deny_group).to_string(); + let count = self.denials_by_group.entry(group).or_default(); + *count = count.saturating_add(1); + } + } + + fn drain(&mut self) -> Option { + if self.network_activity_count == 0 { + return None; + } + let mut denials_by_group: Vec<(String, u32)> = self.denials_by_group.drain().collect(); + denials_by_group.sort_by(|left, right| left.0.cmp(&right.0)); + let summary = FlushableActivitySummary { + network_activity_count: self.network_activity_count, + denied_action_count: self.denied_action_count, + denials_by_group, + }; + self.network_activity_count = 0; + self.denied_action_count = 0; + Some(summary) + } +} + +pub fn activity_flush_interval_secs_from_env(value: Option<&str>) -> u64 { + value + .and_then(|value| value.parse::().ok()) + .filter(|value| *value > 0) + .unwrap_or(DEFAULT_ACTIVITY_FLUSH_INTERVAL_SECS) +} + +fn queue_flush_summary( + tx: &mpsc::Sender, + summary: FlushableActivitySummary, +) -> bool { + tx.try_send(summary).is_ok() +} + +pub fn sanitize_deny_group(raw: &str) -> &'static str { + match raw { + "connect_policy" | "connect" | "l4_deny" => "connect_policy", + "forward_policy" | "forward" => "forward_policy", + "l7_policy" | "l7" | "l7_deny" | "forward-l7-deny" => "l7_policy", + "l7_parse_rejection" | "parse_rejection" => "l7_parse_rejection", + "ssrf" => "ssrf", + "bypass" => "bypass", + "policy_stale" => "policy_stale", + _ => "unknown", + } +} + +#[cfg(test)] +fn denial_rate_pct(network_activity_count: u32, denied_action_count: u32) -> f64 { + if network_activity_count == 0 { + return 0.0; + } + ((f64::from(denied_action_count) / f64::from(network_activity_count)) * 100.0).clamp(0.0, 100.0) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn assert_float_eq(actual: f64, expected: f64) { + assert!((actual - expected).abs() <= f64::EPSILON); + } + + #[test] + fn deny_group_sanitization_uses_allowlist() { + assert_eq!(sanitize_deny_group("connect"), "connect_policy"); + assert_eq!(sanitize_deny_group("forward-l7-deny"), "l7_policy"); + assert_eq!(sanitize_deny_group("host=example.test/path"), "unknown"); + assert_eq!(sanitize_deny_group("acme.internal:443"), "unknown"); + assert_eq!( + sanitize_deny_group("binary=/usr/local/bin/private"), + "unknown" + ); + } + + #[test] + fn denial_rate_handles_zero_and_clamps() { + assert_float_eq(denial_rate_pct(0, 10), 0.0); + assert_float_eq(denial_rate_pct(4, 1), 25.0); + assert_float_eq(denial_rate_pct(4, 10), 100.0); + } + + #[test] + fn flush_summary_drops_when_queue_is_full() { + let (tx, _rx) = mpsc::channel(1); + let summary = FlushableActivitySummary { + network_activity_count: 1, + denied_action_count: 0, + denials_by_group: Vec::new(), + }; + + assert!(queue_flush_summary(&tx, summary.clone())); + assert!(!queue_flush_summary(&tx, summary)); + } + + #[test] + fn activity_flush_interval_uses_positive_values_only() { + assert_eq!( + activity_flush_interval_secs_from_env(None), + DEFAULT_ACTIVITY_FLUSH_INTERVAL_SECS + ); + assert_eq!( + activity_flush_interval_secs_from_env(Some("not-a-number")), + DEFAULT_ACTIVITY_FLUSH_INTERVAL_SECS + ); + assert_eq!( + activity_flush_interval_secs_from_env(Some("0")), + DEFAULT_ACTIVITY_FLUSH_INTERVAL_SECS + ); + assert_eq!(activity_flush_interval_secs_from_env(Some("5")), 5); + } +} diff --git a/crates/openshell-supervisor/src/denial_aggregator.rs b/crates/openshell-supervisor/src/denial_aggregator.rs new file mode 100644 index 0000000000..d80d28db50 --- /dev/null +++ b/crates/openshell-supervisor/src/denial_aggregator.rs @@ -0,0 +1,210 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Supervisor denial aggregator — collects and deduplicates proxy deny events. +//! +//! The proxy emits a [`DenialEvent`] each time a connection or request is +//! denied. The [`DenialAggregator`] receives these events via an MPSC channel, +//! deduplicates them by `(host, port, binary)` key, and maintains running +//! counters. Periodically, the aggregator flushes accumulated summaries +//! upstream to the gateway via `SubmitPolicyAnalysis`. + +use std::collections::HashMap; +use std::future::Future; +use tokio::sync::mpsc; +use tracing::{debug, warn}; + +use openshell_core::denial::DenialEvent; + +/// Aggregated denial summary keyed by `(host, port, binary)`. +#[derive(Debug, Clone)] +struct AggregatedDenial { + host: String, + port: u16, + binary: String, + ancestors: Vec, + deny_reason: String, + denial_stage: String, + first_seen_ms: i64, + last_seen_ms: i64, + count: u32, + sample_cmdlines: Vec, + l7_samples: Vec, +} + +/// A single L7 request sample for aggregation. +#[derive(Debug, Clone)] +struct L7Sample { + method: String, + path: String, + count: u32, +} + +/// The denial aggregator collects proxy deny events and periodically flushes +/// summaries. It is designed to be spawned as a background tokio task. +pub struct DenialAggregator { + rx: mpsc::UnboundedReceiver, + /// Accumulated denials keyed by `(host, port, binary)`. + summaries: HashMap<(String, u16, String), AggregatedDenial>, + /// Flush interval in seconds. + flush_interval_secs: u64, +} + +impl DenialAggregator { + /// Create a new aggregator that reads from the given channel. + pub fn new(rx: mpsc::UnboundedReceiver, flush_interval_secs: u64) -> Self { + Self { + rx, + summaries: HashMap::new(), + flush_interval_secs, + } + } + + /// Run the aggregator loop. This consumes `self` and runs until the + /// channel is closed (all senders are dropped). + /// + /// `flush_callback` is called periodically with the accumulated summaries. + /// In production this calls `SubmitPolicyAnalysis` on the gateway. + /// + /// `ready_gate` is checked before each flush. When it returns `false`, + /// the drain is skipped and events stay in the buffer until the next tick. + pub async fn run(mut self, flush_callback: F, ready_gate: G) + where + F: Fn(Vec) -> Fut, + Fut: Future, + G: Fn() -> bool, + { + let mut flush_interval = + tokio::time::interval(std::time::Duration::from_secs(self.flush_interval_secs)); + // Don't fire immediately on first tick. + flush_interval.tick().await; + + loop { + tokio::select! { + event = self.rx.recv() => { + if let Some(evt) = event { + self.ingest(evt); + } else { + // Channel closed; do a final flush and exit. + if !self.summaries.is_empty() { + if ready_gate() { + let batch = self.drain(); + flush_callback(batch).await; + } else { + warn!( + count = self.summaries.len(), + "DenialAggregator: dropping unflushed summaries, workspace not yet known" + ); + } + } + debug!("DenialAggregator: channel closed, exiting"); + return; + } + } + _ = flush_interval.tick() => { + if ready_gate() && !self.summaries.is_empty() { + let batch = self.drain(); + debug!(count = batch.len(), "DenialAggregator: flushing summaries"); + flush_callback(batch).await; + } + } + } + } + } + + /// Ingest a single denial event, merging into existing summary or creating + /// a new one. + fn ingest(&mut self, event: DenialEvent) { + let now_ms = openshell_core::time::now_ms(); + let key = (event.host.clone(), event.port, event.binary.clone()); + + let entry = self + .summaries + .entry(key) + .or_insert_with(|| AggregatedDenial { + host: event.host.clone(), + port: event.port, + binary: event.binary.clone(), + ancestors: event.ancestors.clone(), + deny_reason: event.deny_reason.clone(), + denial_stage: event.denial_stage.clone(), + first_seen_ms: now_ms, + last_seen_ms: now_ms, + count: 0, + sample_cmdlines: Vec::new(), + l7_samples: Vec::new(), + }); + + entry.count += 1; + entry.last_seen_ms = now_ms; + + // Merge L7 samples. + if let (Some(method), Some(path)) = (&event.l7_method, &event.l7_path) { + if let Some(sample) = entry + .l7_samples + .iter_mut() + .find(|s| s.method == *method && s.path == *path) + { + sample.count += 1; + } else if entry.l7_samples.len() < 20 { + entry.l7_samples.push(L7Sample { + method: method.clone(), + path: path.clone(), + count: 1, + }); + } + } + } + + /// Drain all accumulated summaries into a flushable batch. + fn drain(&mut self) -> Vec { + self.summaries + .drain() + .map(|(_, v)| FlushableDenialSummary { + host: v.host, + port: v.port, + binary: v.binary, + ancestors: v.ancestors, + deny_reason: v.deny_reason, + denial_stage: v.denial_stage, + first_seen_ms: v.first_seen_ms, + last_seen_ms: v.last_seen_ms, + count: v.count, + sample_cmdlines: v.sample_cmdlines, + l7_samples: v + .l7_samples + .into_iter() + .map(|s| FlushableL7Sample { + method: s.method, + path: s.path, + count: s.count, + }) + .collect(), + }) + .collect() + } +} + +/// A denial summary ready to be sent to the gateway. +#[derive(Debug, Clone)] +pub struct FlushableDenialSummary { + pub host: String, + pub port: u16, + pub binary: String, + pub ancestors: Vec, + pub deny_reason: String, + pub denial_stage: String, + pub first_seen_ms: i64, + pub last_seen_ms: i64, + pub count: u32, + pub sample_cmdlines: Vec, + pub l7_samples: Vec, +} + +/// L7 request sample in flushable form. +#[derive(Debug, Clone)] +pub struct FlushableL7Sample { + pub method: String, + pub path: String, + pub count: u32, +} diff --git a/crates/openshell-supervisor/src/lib.rs b/crates/openshell-supervisor/src/lib.rs new file mode 100644 index 0000000000..5fb6e0803b --- /dev/null +++ b/crates/openshell-supervisor/src/lib.rs @@ -0,0 +1,5752 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! `OpenShell` supervisor library. +//! +//! This crate provides process sandboxing and monitoring capabilities. + +// `defaults-without-telemetry` is an alias for the default feature set minus +// `telemetry`, not a switch that turns telemetry off. Cargo cannot subtract a +// default feature, so adding it on top of the defaults would otherwise produce +// a telemetry-on build that reads as telemetry-free. Fail the build instead. +#[cfg(all(feature = "telemetry", feature = "defaults-without-telemetry"))] +compile_error!( + "features `telemetry` and `defaults-without-telemetry` are mutually exclusive; \ + build a telemetry-free supervisor with `--no-default-features --features defaults-without-telemetry`" +); + +mod activity_aggregator; +mod denial_aggregator; +mod mechanistic_mapper; + +use miette::{IntoDiagnostic, Result, WrapErr}; +use std::future::Future; +use std::io::Write as _; +use std::pin::Pin; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicU32}; +use std::time::Duration; +use tracing::{debug, info, warn}; + +use openshell_core::PolicyValidationFailureMode; + +use openshell_ocsf::{ + ActionId, ActivityId, AppLifecycleBuilder, ConfidenceId, ConfigStateChangeBuilder, + DetectionFindingBuilder, DispositionId, EventContext, FindingInfo, OcsfEvent, SeverityId, + StateId, StatusId, ocsf_emit, +}; + +// --------------------------------------------------------------------------- +// OCSF Context +// --------------------------------------------------------------------------- +// +// The following log sites intentionally remain as plain `tracing` macros +// and are NOT migrated to OCSF builders: +// +// - DEBUG/TRACE events (zombie reaping, ip commands, gRPC connects, PTY state) +// - Transient "about to do X" events where the result is logged separately +// (e.g., "Fetching sandbox policy via gRPC", "Creating OPA engine from proto") +// - Internal SSH channel warnings (unknown channel, PTY resize failures) +// - Denial flush telemetry (the individual denials are already OCSF events) +// - Status reporting failures (sync to gateway, non-actionable) +// - Route refresh interval validation warnings +// +// These are operational plumbing that don't represent security decisions, +// policy changes, or observable sandbox behavior worth structuring. +// --------------------------------------------------------------------------- + +/// Re-export the process-wide OCSF sandbox context getter. +/// +/// The singleton lives in `openshell-ocsf` so both supervisor leaves can +/// reach it without depending on `openshell-sandbox`. Initialised once during +/// `run_sandbox()` startup via `openshell_ocsf::ctx::set_ctx`. +pub(crate) use openshell_ocsf::ctx::ctx as ocsf_ctx; + +async fn retain_remote_access_plane( + proxy_exited: impl Future, + shutdown_requested: impl Future, +) -> Result<()> { + tokio::pin!(proxy_exited); + tokio::pin!(shutdown_requested); + tokio::select! { + () = &mut proxy_exited => Err(miette::miette!( + "control-mode proxy accept loop exited unexpectedly" + )), + () = &mut shutdown_requested => Ok(()), + } +} + +async fn completion_phase_or_shutdown(phase: F, mut shutdown: Pin<&mut S>) -> bool +where + F: Future, + S: Future + ?Sized, +{ + tokio::pin!(phase); + tokio::select! { + () = &mut phase => false, + () = &mut shutdown => true, + } +} + +struct ControlReadiness { + task: tokio::task::JoinHandle<()>, + path: std::path::PathBuf, +} + +impl ControlReadiness { + fn start( + path: std::path::PathBuf, + mut session_readiness: Option>, + ) -> Result { + if session_readiness + .as_ref() + .is_some_and(|readiness| !*readiness.borrow()) + { + return Err(miette::miette!( + "supervisor session is not ready when starting health listener" + )); + } + prepare_control_readiness_path(&path)?; + let listener = tokio::net::UnixListener::bind(&path) + .into_diagnostic() + .wrap_err_with(|| format!("bind supervisor readiness socket on {}", path.display()))?; + let task_path = path.clone(); + let task = tokio::spawn(async move { + let mut listener = Some(listener); + loop { + let session_unready = session_readiness + .as_ref() + .is_some_and(|readiness| !*readiness.borrow()); + if listener.is_none() || session_unready { + if session_unready { + listener.take(); + let _ = std::fs::remove_file(&task_path); + let Some(readiness) = session_readiness.as_mut() else { + break; + }; + if readiness.wait_for(|ready| *ready).await.is_err() { + break; + } + } + match prepare_control_readiness_path(&task_path).and_then(|()| { + tokio::net::UnixListener::bind(&task_path) + .into_diagnostic() + .wrap_err_with(|| { + format!( + "rebind supervisor readiness socket on {}", + task_path.display() + ) + }) + }) { + Ok(rebound) => listener = Some(rebound), + Err(error) => { + tracing::warn!(%error, "control-mode readiness rebind failed; retrying"); + tokio::time::sleep(Duration::from_millis(100)).await; + continue; + } + } + continue; + } + + let Some(active_listener) = listener.as_ref() else { + continue; + }; + if let Some(readiness) = session_readiness.as_mut() { + tokio::select! { + accepted = active_listener.accept() => match accepted { + Ok((stream, _)) => drop(stream), + Err(error) => { + tracing::warn!(%error, "control-mode readiness accept failed; retrying"); + tokio::time::sleep(Duration::from_millis(100)).await; + } + }, + changed = readiness.changed() => { + if changed.is_err() { + break; + } + } + } + } else { + match active_listener.accept().await { + Ok((stream, _)) => drop(stream), + Err(error) => { + tracing::warn!(%error, "control-mode readiness accept failed; retrying"); + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + } + } + let _ = std::fs::remove_file(&task_path); + }); + Ok(Self { task, path }) + } +} + +#[cfg(unix)] +fn prepare_control_readiness_path(path: &std::path::Path) -> Result<()> { + use std::os::unix::fs::{FileTypeExt as _, MetadataExt as _}; + + if !path.is_absolute() { + return Err(miette::miette!( + "supervisor readiness socket path must be absolute" + )); + } + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .into_diagnostic() + .wrap_err_with(|| format!("create readiness directory {}", parent.display()))?; + } + match std::fs::symlink_metadata(path) { + Ok(metadata) => { + if !metadata.file_type().is_socket() + || metadata.uid() != rustix::process::getuid().as_raw() + { + return Err(miette::miette!( + "refusing unsafe existing readiness path {}", + path.display() + )); + } + std::fs::remove_file(path) + .into_diagnostic() + .wrap_err_with(|| format!("remove stale readiness socket {}", path.display()))?; + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => { + return Err(error) + .into_diagnostic() + .wrap_err_with(|| format!("inspect readiness path {}", path.display())); + } + } + Ok(()) +} + +impl Drop for ControlReadiness { + fn drop(&mut self) { + self.task.abort(); + let _ = std::fs::remove_file(&self.path); + } +} + +/// Check whether the live supervisor owns its private readiness socket. +#[cfg(unix)] +pub fn check_control_readiness(path: &std::path::Path) -> Result<()> { + if !path.is_absolute() { + return Err(miette::miette!("health socket path must be absolute")); + } + std::os::unix::net::UnixStream::connect(path) + .into_diagnostic() + .wrap_err_with(|| format!("connect supervisor readiness socket {}", path.display()))?; + Ok(()) +} + +/// Health subcommands are unsupported on non-Unix hosts. +#[cfg(not(unix))] +pub fn check_control_readiness(_path: &std::path::Path) -> Result<()> { + Err(miette::miette!( + "supervisor readiness sockets require a Unix host" + )) +} + +#[cfg(unix)] +async fn wait_for_control_shutdown_signal() { + use tokio::signal::unix::{SignalKind, signal}; + + let mut sigterm = signal(SignalKind::terminate()).expect("install control SIGTERM handler"); + let mut sigint = signal(SignalKind::interrupt()).expect("install control SIGINT handler"); + tokio::select! { + _ = sigterm.recv() => {} + _ = sigint.recv() => {} + } +} + +#[cfg(not(unix))] +async fn wait_for_control_shutdown_signal() { + let _ = tokio::signal::ctrl_c().await; +} + +use openshell_core::denial::DenialEvent; +use openshell_core::policy::{NetworkMode, NetworkPolicy, ProxyPolicy, SandboxPolicy}; +use openshell_core::proposals::AgentProposals; +use openshell_core::provider_credentials::ProviderCredentialState; +use openshell_supervisor_network::opa::OpaEngine; +use openshell_supervisor_network::proxy::ProxyHandle; +use openshell_supervisor_process::skills; +use tokio::sync::mpsc::UnboundedSender; +use tokio::time::timeout; + +fn shared_ssh_socket_from_env() -> bool { + std::env::var(openshell_core::sandbox_env::SSH_SOCKET_SHARED) + .is_ok_and(|value| shared_ssh_socket_value(&value)) +} + +fn shared_ssh_socket_value(value: &str) -> bool { + value == "1" || value.eq_ignore_ascii_case("true") +} + +/// Run the supervisor as an explicit HTTP/CONNECT network proxy. +/// +/// This role deliberately bypasses the Isolation Backend: it does not attach +/// a Sandbox Runtime, launch a workload, or claim process and binary identity. +/// It reuses the same local Rego/YAML policy engine and proxy implementation as +/// sandbox supervision. +/// +/// # Errors +/// +/// Returns an error when policy loading or proxy startup fails, or when the +/// proxy accept loop exits unexpectedly. +pub async fn run_network_proxy( + listen: std::net::SocketAddr, + policy_rules: String, + policy_data: String, + tls_dir: Option, + upstream_proxy_args: openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs, +) -> Result { + if !listen.ip().is_loopback() { + return Err(miette::miette!( + "network-proxy listener must use a loopback address: {listen}" + )); + } + + let hostname = std::fs::read_to_string("/etc/hostname").map_or_else( + |_| "openshell-supervisor".to_string(), + |value| value.trim().to_string(), + ); + if !openshell_ocsf::ctx::set_ctx(EventContext { + sandbox_id: String::new(), + sandbox_name: "network-proxy".to_string(), + container_image: String::new(), + hostname, + product_version: openshell_core::VERSION.to_string(), + proxy_ip: listen.ip(), + proxy_port: listen.port(), + }) { + debug!("OCSF context already initialized, keeping existing"); + } + + let extension_credentials = openshell_extension_core::ExtensionCredentialStore::new(); + let (mut policy, opa_engine, _, _, _, initial_agent_proposals_enabled, _) = load_policy( + None, + None, + None, + Some(policy_rules), + Some(policy_data), + &extension_credentials, + LocalPolicyIdentity::EndpointOnly, + ) + .await?; + policy.network = NetworkPolicy { + mode: NetworkMode::Proxy, + proxy: Some(ProxyPolicy { + http_addr: Some(listen), + }), + }; + + let provider_credentials = ProviderCredentialState::from_environment( + 0, + std::collections::HashMap::new(), + std::collections::HashMap::new(), + std::collections::HashMap::new(), + ); + let (_, workspace_rx) = tokio::sync::watch::channel(String::new()); + let tls_dir = tls_dir.unwrap_or_else(|| { + std::env::temp_dir().join(format!("openshell-supervisor-{}", std::process::id())) + }); + let mut networking = openshell_supervisor_network::run::run_networking( + &policy, + None, + opa_engine.as_ref(), + None, + Arc::new(AtomicU32::new(0)), + false, + &provider_credentials, + None, + Some("network-proxy"), + None, + None, + None, + AgentProposals::new(initial_agent_proposals_enabled), + workspace_rx, + &upstream_proxy_args, + Some(&tls_dir), + None, + #[cfg(target_os = "linux")] + None, + None, + ) + .await?; + + if let Some((ca_certificate, trust_bundle)) = networking.ca_file_paths.as_ref() { + info!( + ca_certificate = %ca_certificate.display(), + trust_bundle = %trust_bundle.display(), + "Network-proxy trust files ready" + ); + } + + let proxy = networking + .proxy + .as_mut() + .ok_or_else(|| miette::miette!("network-proxy role did not start a proxy listener"))?; + let bound = proxy + .http_addr() + .ok_or_else(|| miette::miette!("network-proxy role did not bind an explicit listener"))?; + let exited = proxy + .take_exit_receiver() + .ok_or_else(|| miette::miette!("network-proxy exit monitor is unavailable"))?; + info!(%bound, "Network-proxy role ready"); + + tokio::select! { + _ = exited => Err(miette::miette!("network-proxy accept loop exited unexpectedly")), + () = wait_for_control_shutdown_signal() => { + drop(networking); + Ok(0) + } + } +} + +/// Run a command in the sandbox. +/// +/// # Errors +/// +/// Returns an error if the command fails to start or encounters a fatal error. +#[allow( + clippy::too_many_arguments, + clippy::implicit_hasher, + clippy::similar_names, + clippy::fn_params_excessive_bools +)] +pub async fn run_sandbox( + command: Vec, + workdir: Option, + timeout_secs: u64, + interactive: bool, + await_main_process_attachment: bool, + sandbox_id: Option, + sandbox: Option, + openshell_endpoint: Option, + policy_rules: Option, + policy_data: Option, + ssh_socket_path: Option, + health_socket_path: Option, + ocsf_enabled: Arc, + upstream_proxy_args: openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs, + backend_descriptor: openshell_isolation_interface::contract::BackendDescriptor, + auth_bundle: openshell_core::jwt::SupervisorAuthBundle, + admitted_isolation_backend: Option, + main_exit_marker: Option, +) -> Result { + // An empty command is the versioned scratch-sandbox sentinel. The + // external supervisor cannot inspect the workload filesystem, so preserve + // it for openshell-sandbox to resolve against the agent image. + let (program, args) = command.split_first().map_or_else( + || (String::new(), Vec::new()), + |(program, args)| (program.clone(), args.to_vec()), + ); + + // Initialize the process-wide OCSF context early so that events emitted + // during policy loading (filesystem config, validation) have a context. + // Proxy IP/port use defaults here; the boundary mediation source carries + // workload-side connection metadata. + { + let hostname = std::fs::read_to_string("/etc/hostname").map_or_else( + |_| "openshell-sandbox".to_string(), + |s| s.trim().to_string(), + ); + + if !openshell_ocsf::ctx::set_ctx(EventContext { + sandbox_id: sandbox_id.clone().unwrap_or_default(), + sandbox_name: sandbox.as_deref().unwrap_or_default().to_string(), + container_image: std::env::var("OPENSHELL_CONTAINER_IMAGE").unwrap_or_default(), + hostname, + product_version: openshell_core::VERSION.to_string(), + proxy_ip: std::net::IpAddr::from([127, 0, 0, 1]), + proxy_port: 3128, + }) { + debug!("OCSF context already initialized, keeping existing"); + } + } + + // Extension credentials are owned by this supervisor and shared by every + // gateway connection it opens, so the middleware registry's bearer slots + // and the policy poll loop that rotates them stay the same objects. + let extension_credentials = openshell_extension_core::ExtensionCredentialStore::new(); + + // Load policy and initialize OPA engine + let openshell_endpoint_for_proxy = openshell_endpoint.clone(); + let sandbox_name_for_agg = sandbox.clone(); + let ( + policy, + opa_engine, + retained_proto, + middleware_registry_status, + loaded_policy_origin, + initial_agent_proposals_enabled, + initial_extension_authentication_enabled, + ) = load_policy( + sandbox_id.clone(), + sandbox, + openshell_endpoint.clone(), + policy_rules, + policy_data, + &extension_credentials, + LocalPolicyIdentity::Required, + ) + .await?; + + // Normalize the active driver's identity contract once, while both the + // policy and launched image filesystem are available. Kubernetes and + // OpenShift retain their authoritative numeric pair; Docker fills only + // omitted policy fields from OCI Config.User. A remote boundary resolves + // identity in its own filesystem instead; control must not interpret + // guest account data against the host's /etc/passwd and /etc/group. + let workspace = workdir; + + let provider_credentials = { + // Fetch provider environment variables from the server. + // This is done after loading the policy so the sandbox can still start + // even if provider env fetch fails (graceful degradation). + let ( + provider_env_revision, + provider_env, + provider_credential_expires_at_ms, + dynamic_credentials, + static_credential_bindings, + non_secret_environment_keys, + ) = if let (Some(id), Some(endpoint)) = (&sandbox_id, &openshell_endpoint) { + match openshell_core::grpc_client::fetch_provider_environment(endpoint, id).await { + Ok(result) => { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .message(format!( + "Fetched provider environment [env_count:{}]", + result.environment.len() + )) + .build() + ); + ( + result.provider_env_revision, + result.environment, + result.credential_expires_at_ms, + result.dynamic_credentials, + result.static_credential_bindings, + result.non_secret_environment_keys, + ) + } + Err(e) => { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::High) + .status(StatusId::Failure) + .state(StateId::Disabled, "fail_closed") + .message(format!( + "Failed to fetch provider environment; no provider credentials are active: {e}" + )) + .build() + ); + ( + 0, + std::collections::HashMap::new(), + std::collections::HashMap::new(), + std::collections::HashMap::new(), + std::collections::HashMap::new(), + Vec::new(), + ) + } + } + } else { + ( + 0, + std::collections::HashMap::new(), + std::collections::HashMap::new(), + std::collections::HashMap::new(), + std::collections::HashMap::new(), + Vec::new(), + ) + }; + + let dynamic_credentials_fallback = dynamic_credentials.clone(); + match ProviderCredentialState::from_bound_environment( + provider_env_revision, + provider_env, + provider_credential_expires_at_ms, + dynamic_credentials, + static_credential_bindings, + non_secret_environment_keys, + ) { + Ok(credentials) => credentials, + Err(error) => { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::High) + .status(StatusId::Failure) + .state(StateId::Disabled, "fail_closed") + .message(format!( + "Rejected provider environment bindings; static provider credentials were revoked; fetched dynamic token grants remain active: {error}" + )) + .build() + ); + ProviderCredentialState::from_environment( + provider_env_revision, + std::collections::HashMap::new(), + std::collections::HashMap::new(), + dynamic_credentials_fallback, + ) + } + } + }; + + if credential_gating_unavailable( + &loaded_policy_origin, + provider_credentials.resolver().is_some(), + true, + ) { + report_credential_gating_unavailable(); + } + + // Canonical-process overrides are deliberately applied only to the main + // child. Keep the provider snapshot pristine for later exec/editor/SFTP + // children launched by the sandbox. + + // Shared agent-proposals feature flag. Seed from the same initial settings + // snapshot that produced the policy so networking and process setup agree + // before the poll loop starts reconciling later changes. + let agent_proposals = AgentProposals::new(initial_agent_proposals_enabled); + + // Shared PID: set after process spawn so the proxy can look up + // the entrypoint process's /proc/net/tcp for identity binding. + let entrypoint_pid = Arc::new(AtomicU32::new(0)); + + // The sandbox runtime uses the shared authenticated boundary protocol. + // The admitted backend name is resolved independently of the protected + // descriptor, and generic supervisor code never imports a driver crate. + let admitted_backend_name = admitted_isolation_backend.ok_or_else(|| { + miette::miette!("runtime descriptor supplied without an admitted isolation backend") + })?; + let runtime_descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor = + serde_json::from_slice(&backend_descriptor.payload) + .map_err(|error| miette::miette!("decode sandbox runtime descriptor: {error}"))?; + if auth_bundle.runtime_generation.as_str() != runtime_descriptor.generation { + return Err(miette::miette!( + "supervisor authentication bundle does not match runtime generation" + )); + } + let sandbox_bearer = openshell_core::grpc_client::install_supervisor_auth_bundle(&auth_bundle)?; + let session_id = runtime_descriptor.session_id; + let ca_file_paths = Arc::new(std::sync::Mutex::new(None)); + let backend: Arc = + Arc::new(openshell_sandbox_backend::OpenShellRuntimeBackend::new( + ca_file_paths.clone(), + provider_credentials.clone(), + sandbox_bearer, + )); + let mut registry = openshell_isolation_interface::contract::BackendRegistry::new(); + registry + .register(backend) + .map_err(|error| miette::miette!(error.to_string()))?; + let (backend, verified) = registry + .resolve(backend_descriptor, &admitted_backend_name) + .map_err(|error| miette::miette!(error.to_string()))?; + let context = openshell_isolation_interface::contract::SandboxContext { + sandbox_id: sandbox_id.clone().unwrap_or_default(), + session_id, + policy: policy.clone(), + agent: openshell_isolation_interface::AgentSpec { + program, + args, + workdir: workspace, + timeout_secs, + interactive, + }, + identity: runtime_descriptor.workload_identity, + }; + let bound = backend + .attach(verified, context) + .await + .map_err(|error| miette::miette!(error.to_string()))?; + info!(backend = %admitted_backend_name, "Isolation boundary attached"); + let remote_boundary = (bound, admitted_backend_name, ca_file_paths); + + let transparent_tcp_capable = true; + let transparent_tcp_substrate_ready = true; + // The denial channel is owned by the orchestrator: the proxy (in the + // networking leaf) and the bypass monitor (in the process leaf) both + // produce DenialEvents that the denial aggregator (orchestrator-side) + // consumes via the matching receiver. Both leaves are pure producers; + // the orchestrator owns the consumer task spawned below. + let (denial_tx, denial_rx): (Option>, _) = if sandbox_id.is_some() + { + let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); + (Some(tx), Some(rx)) + } else { + (None, None) + }; + + // Anonymous activity channel: same orchestrator-owned pattern as the + // denial channel. The proxy and the bypass monitor both emit per-event + // activity records; the orchestrator-side aggregator drains, sanitizes, + // and flushes anonymous summaries to the gateway. + let (activity_tx, activity_rx) = if sandbox_id.is_some() { + let (tx, rx) = + tokio::sync::mpsc::channel(openshell_core::activity::ACTIVITY_EVENT_QUEUE_CAPACITY); + (Some(tx), Some(rx)) + } else { + (None, None) + }; + + // Workspace watch: the policy poll loop learns the workspace from + // GetSandboxConfig and broadcasts it. Flush tasks and the policy.local + // API read the current value so proposals target the correct workspace. + let (workspace_tx, workspace_rx) = tokio::sync::watch::channel(String::new()); + + let remote_network_source = remote_boundary.0.network_mediation_source(); + let remote_host_gateway_ip = remote_boundary.0.host_gateway_ip(); + let (remote_ready, backend_name, ca_file_paths) = { + let (bound, backend_name, ca_file_paths) = remote_boundary; + let ready = bound + .confirm() + .await + .map_err(|error| miette::miette!(error.to_string()))?; + info!(backend = %backend_name, "Isolation boundary enforcement confirmed"); + (ready, backend_name, ca_file_paths) + }; + + let mut networking = Some( + openshell_supervisor_network::run::run_networking( + &policy, + None, + opa_engine.as_ref(), + retained_proto.as_ref(), + entrypoint_pid.clone(), + // The sandbox supplies already-resolved identities across the + // boundary. The host supervisor cannot inspect its mount or PID + // namespace, so waiting for a host-visible entrypoint PID would + // unnecessarily delay DNS and network readiness. + false, + &provider_credentials, + sandbox_id.as_deref(), + sandbox_name_for_agg.as_deref(), + openshell_endpoint_for_proxy.as_deref(), + denial_tx, + activity_tx, + agent_proposals.clone(), + workspace_rx.clone(), + &upstream_proxy_args, + None, + remote_host_gateway_ip, + #[cfg(target_os = "linux")] + None, + Some(remote_network_source), + ) + .await?, + ); + + ca_file_paths + .lock() + .map_err(|_| miette::miette!("boundary CA path lock is poisoned"))? + .clone_from( + &networking + .as_ref() + .and_then(|runtime| runtime.ca_file_paths.clone()), + ); + let remote_ready = (remote_ready, backend_name); + + // Spawn the denial-aggregator flush task. The aggregator drains proxy + // denial events, batches them, and ships summaries to the gateway via + // `SubmitPolicyAnalysis`. + if let (Some(rx), Some(endpoint)) = (denial_rx, openshell_endpoint_for_proxy.as_deref()) { + // SubmitPolicyAnalysis resolves by sandbox *name*, not UUID — fall + // back to the ID when the name isn't set. + let agg_name = sandbox_name_for_agg + .clone() + .or_else(|| sandbox_id.clone()) + .unwrap_or_default(); + let agg_endpoint = endpoint.to_string(); + let flush_interval_secs: u64 = std::env::var("OPENSHELL_DENIAL_FLUSH_INTERVAL_SECS") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(10); + + let aggregator = denial_aggregator::DenialAggregator::new(rx, flush_interval_secs); + let denial_workspace_gate = workspace_rx.clone(); + let denial_workspace_rx = workspace_rx.clone(); + + tokio::spawn(async move { + aggregator + .run( + |summaries| { + let endpoint = agg_endpoint.clone(); + let sandbox_name = agg_name.clone(); + let workspace = denial_workspace_rx.borrow().clone(); + async move { + if let Err(e) = flush_proposals_to_gateway( + &endpoint, + &sandbox_name, + &workspace, + summaries, + ) + .await + { + warn!(error = %e, "Failed to flush denial summaries to gateway"); + } + } + }, + move || !denial_workspace_gate.borrow().is_empty(), + ) + .await; + }); + } + + // Spawn the activity-aggregator flush task. The aggregator drains + // anonymous activity events from the proxy, sanitizes deny groups, + // and ships periodic summaries to the gateway. + if let (Some(rx), Some(endpoint)) = (activity_rx, openshell_endpoint_for_proxy.as_deref()) { + let agg_name = sandbox_name_for_agg + .clone() + .or_else(|| sandbox_id.clone()) + .unwrap_or_default(); + let agg_endpoint = endpoint.to_string(); + let flush_interval_secs = activity_aggregator::activity_flush_interval_secs_from_env( + std::env::var("OPENSHELL_ACTIVITY_FLUSH_INTERVAL_SECS") + .ok() + .as_deref(), + ); + + let aggregator = activity_aggregator::ActivityAggregator::new(rx, flush_interval_secs); + let activity_workspace_gate = workspace_rx.clone(); + let activity_workspace_rx = workspace_rx.clone(); + + tokio::spawn(async move { + aggregator + .run( + move |summary| { + let endpoint = agg_endpoint.clone(); + let sandbox_name = agg_name.clone(); + let workspace = activity_workspace_rx.borrow().clone(); + async move { + if let Err(e) = flush_activity_to_gateway( + &endpoint, + &sandbox_name, + &workspace, + summary, + ) + .await + { + warn!(error = %e, "Failed to flush activity summary to gateway"); + } + } + }, + move || !activity_workspace_gate.borrow().is_empty(), + ) + .await; + }); + } + + // Spawn background policy poll task (gRPC mode only). + if let (Some(id), Some(endpoint), Some(engine)) = ( + sandbox_id.as_deref(), + openshell_endpoint.as_deref(), + opa_engine.as_ref(), + ) { + let poll_id = id.to_string(); + let poll_endpoint = endpoint.to_string(); + let poll_engine = engine.clone(); + let poll_ocsf_enabled = ocsf_enabled.clone(); + let poll_pid = entrypoint_pid.clone(); + let poll_provider_credentials = provider_credentials.clone(); + let poll_policy_local = networking.as_ref().map(|n| n.policy_local_ctx.clone()); + let poll_interval_secs: u64 = std::env::var("OPENSHELL_POLICY_POLL_INTERVAL_SECS") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(10); + let poll_ctx = PolicyPollLoopContext { + endpoint: poll_endpoint, + sandbox_id: poll_id, + opa_engine: poll_engine, + loaded_policy_origin, + entrypoint_pid: poll_pid, + interval_secs: poll_interval_secs, + ocsf_enabled: poll_ocsf_enabled, + provider_credentials: poll_provider_credentials, + policy_local_ctx: poll_policy_local, + agent_proposals: agent_proposals.clone(), + middleware_registry_status, + workspace_tx, + extension_credentials: extension_credentials.clone(), + extension_authentication_enabled: initial_extension_authentication_enabled, + middleware_connector: default_middleware_connector(), + transparent_tcp: TransparentTcpReloadState { + capable: transparent_tcp_capable, + substrate_ready: transparent_tcp_substrate_ready, + }, + }; + + tokio::spawn(async move { + if let Err(e) = run_policy_poll_loop(poll_ctx).await { + ocsf_emit!( + AppLifecycleBuilder::new(ocsf_ctx()) + .activity(ActivityId::Fail) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .message(format!("Policy poll loop exited with error: {e}")) + .build() + ); + } + }); + } + + let proxy_exited: Pin + Send>> = if let Some(rx) = networking + .as_mut() + .and_then(|n| n.proxy.as_mut()) + .and_then(ProxyHandle::take_exit_receiver) + { + Box::pin(async { + let _ = rx.await; + }) + } else { + Box::pin(std::future::pending()) + }; + tokio::pin!(proxy_exited); + + let (confirmed, backend_name) = remote_ready; + let exit_code = { + let running = confirmed + .into_boundary() + .start_agent() + .await + .map_err(|error| miette::miette!(error.to_string()))?; + info!(backend = %backend_name, "Isolation boundary agent started"); + let agent = running.agent(); + let boundary_access = openshell_supervisor_process::delegated::start_boundary_access( + sandbox_id.as_deref(), + openshell_endpoint.as_deref(), + ssh_socket_path.as_deref(), + shared_ssh_socket_from_env(), + networking + .as_ref() + .and_then(|runtime| runtime.ca_file_paths.clone()), + running.exec(), + running.loopback_connector(), + agent.clone(), + ) + .await?; + info!(backend = %backend_name, "Control-mode access plane started"); + let mut control_readiness = if let Some(path) = health_socket_path { + Some(ControlReadiness::start( + path, + boundary_access.session_readiness(), + )?) + } else { + None + }; + let instance_id = boundary_access.instance_id().to_string(); + let wait_agent = agent.clone(); + let shutdown_requested = wait_for_control_shutdown_signal(); + tokio::pin!(shutdown_requested); + let wait = async move { + wait_agent + .wait() + .await + .map(|status| match status { + openshell_isolation_interface::contract::BoundaryExitStatus::Exited(code) => { + code + } + openshell_isolation_interface::contract::BoundaryExitStatus::Signaled( + signal, + ) => 128_i32.saturating_add(signal), + }) + .map_err(|error| miette::miette!(error.to_string())) + }; + let (exit_code, mut retain_access) = tokio::select! { + result = wait => (result?, true), + () = &mut proxy_exited => { + let _ = running.terminate().await; + return Err(miette::miette!( + "control-mode proxy accept loop exited unexpectedly" + )); + } + () = &mut shutdown_requested => { + let _ = agent + .signal(openshell_isolation_interface::contract::BoundarySignal::Term) + .await; + let status = if let Ok(result) = timeout(Duration::from_secs(5), agent.wait()).await { + result + } else { + let _ = agent.terminate().await; + agent.wait().await + } + .map_err(|error| miette::miette!(error.to_string()))?; + let exit_code = match status { + openshell_isolation_interface::contract::BoundaryExitStatus::Exited(code) => code, + openshell_isolation_interface::contract::BoundaryExitStatus::Signaled(signal) => { + 128_i32.saturating_add(signal) + } + }; + running + .terminate() + .await + .map_err(|error| miette::miette!( + "sandbox did not acknowledge terminal state: {error}" + ))?; + (exit_code, false) + } + }; + if !retain_access { + control_readiness.take(); + } + boundary_access + .publish_main_exit(exit_code, await_main_process_attachment) + .await; + // `shutdown_requested` has already completed when shutdown won the + // lifecycle select above and must not be polled again. + let mut completion_cancelled = !retain_access; + if retain_access && let Some(marker) = main_exit_marker.as_deref() { + persist_main_exit_marker(marker, exit_code) + .into_diagnostic() + .wrap_err("persist canonical-process completion marker")?; + } + if !completion_cancelled + && let (Some(endpoint), Some(id)) = + (openshell_endpoint.as_deref(), sandbox_id.as_deref()) + { + let report = openshell_supervisor_process::delegated::report_main_process_exit( + endpoint, + id, + &instance_id, + exit_code, + ); + completion_cancelled = + completion_phase_or_shutdown(report, shutdown_requested.as_mut()).await; + } + if !completion_cancelled { + let drain = boundary_access.drain_main_terminal_delivery(); + completion_cancelled = + completion_phase_or_shutdown(drain, shutdown_requested.as_mut()).await; + } + if !completion_cancelled + && let (Some(endpoint), Some(id)) = + (openshell_endpoint.as_deref(), sandbox_id.as_deref()) + { + let finalize = openshell_supervisor_process::delegated::finalize_main_process_exit( + endpoint, + id, + &instance_id, + ); + completion_cancelled = + completion_phase_or_shutdown(finalize, shutdown_requested.as_mut()).await; + } + if completion_cancelled { + retain_access = false; + control_readiness.take(); + } + if retain_access { + info!(backend = %backend_name, "Canonical process exited; retaining control-mode access plane"); + retain_remote_access_plane(&mut proxy_exited, &mut shutdown_requested).await?; + } + drop(control_readiness); + drop(running); + drop(boundary_access); + exit_code + }; + + // Drop networking explicitly so proxy tasks tear down before we return. + drop(networking); + + Ok(exit_code) +} + +fn persist_main_exit_marker(path: &std::path::Path, exit_code: i32) -> std::io::Result<()> { + let parent = path + .parent() + .filter(|parent| !parent.as_os_str().is_empty()) + .ok_or_else(|| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + format!("completion marker has no parent: {}", path.display()), + ) + })?; + let name = path.file_name().ok_or_else(|| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + format!("completion marker has no file name: {}", path.display()), + ) + })?; + let temporary = parent.join(format!( + ".{}.tmp-{}", + name.to_string_lossy(), + std::process::id() + )); + let mut options = std::fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(0o600); + } + let mut file = options.open(&temporary)?; + writeln!(file, "exit_code={exit_code}")?; + file.sync_all()?; + std::fs::rename(&temporary, path)?; + std::fs::File::open(parent)?.sync_all() +} + +/// Flush aggregated denial summaries to the gateway via `SubmitPolicyAnalysis`. +async fn flush_proposals_to_gateway( + endpoint: &str, + sandbox_name: &str, + workspace: &str, + summaries: Vec, +) -> Result<()> { + use openshell_core::grpc_client::CachedOpenShellClient; + use openshell_core::proto::{DenialSummary, L7RequestSample}; + + let client = CachedOpenShellClient::connect(endpoint).await?; + client.set_workspace(workspace.to_string()); + + let proto_summaries: Vec = summaries + .into_iter() + .map(|s| DenialSummary { + sandbox_id: String::new(), + host: s.host, + port: u32::from(s.port), + binary: s.binary, + ancestors: s.ancestors, + deny_reason: s.deny_reason, + first_seen_ms: s.first_seen_ms, + last_seen_ms: s.last_seen_ms, + count: s.count, + suppressed_count: 0, + total_count: s.count, + sample_cmdlines: s.sample_cmdlines, + binary_sha256: String::new(), + persistent: false, + denial_stage: s.denial_stage, + l7_request_samples: s + .l7_samples + .into_iter() + .map(|l| L7RequestSample { + method: l.method, + path: l.path, + decision: "deny".to_string(), + count: l.count, + }) + .collect(), + l7_inspection_active: false, + }) + .collect(); + + // Run the mechanistic mapper sandbox-side to generate proposals. + // The gateway is a thin persistence + validation layer — it never + // generates proposals itself. + let proposals = mechanistic_mapper::generate_proposals(&proto_summaries); + + info!( + sandbox_name = %sandbox_name, + summaries = proto_summaries.len(), + proposals = proposals.len(), + "Flushed denial analysis to gateway" + ); + + client + .submit_policy_analysis( + sandbox_name, + proto_summaries, + proposals, + Vec::new(), + "mechanistic", + ) + .await?; + + Ok(()) +} + +/// Flush an anonymous activity summary to the gateway via `SubmitPolicyAnalysis`. +async fn flush_activity_to_gateway( + endpoint: &str, + sandbox_name: &str, + workspace: &str, + summary: activity_aggregator::FlushableActivitySummary, +) -> Result<()> { + use openshell_core::grpc_client::CachedOpenShellClient; + use openshell_core::proto::{DenialGroupCount, NetworkActivitySummary}; + + let client = CachedOpenShellClient::connect(endpoint).await?; + client.set_workspace(workspace.to_string()); + + let proto_summary = NetworkActivitySummary { + network_activity_count: summary.network_activity_count, + denied_action_count: summary.denied_action_count, + denials_by_group: summary + .denials_by_group + .into_iter() + .map(|(group, count)| DenialGroupCount { + deny_group: group, + denied_count: count, + }) + .collect(), + }; + + info!( + sandbox_name = %sandbox_name, + network_activity_count = proto_summary.network_activity_count, + denied_action_count = proto_summary.denied_action_count, + "Flushed activity summary to gateway" + ); + + client + .submit_policy_analysis( + sandbox_name, + Vec::new(), + Vec::new(), + vec![proto_summary], + "activity", + ) + .await?; + + Ok(()) +} + +// ============================================================================ +// Baseline filesystem path enrichment +// ============================================================================ + +/// Minimum read-only paths required for a proxy-mode sandbox child process to +/// function: dynamic linker, shared libraries, DNS resolution, CA certs, +/// Python venv, openshell logs, process info, and random bytes. +/// +/// `/proc` and `/dev/urandom` are included here for the same reasons they +/// appear in `restrictive_default_policy()`: virtually every process needs +/// them. Before the Landlock per-path fix (#677) these were effectively free +/// because a missing path silently disabled the entire ruleset; now they must +/// be explicit. +const PROXY_BASELINE_READ_ONLY: &[&str] = &[ + "/usr", + "/lib", + "/etc", + "/app", + "/var/log", + "/proc", + "/dev/urandom", +]; + +/// Minimum read-write paths required for a proxy-mode sandbox child process. +/// The active workspace is granted separately through `include_workdir`. +// `/dev/null` is opened by common child-process launchers when they construct +// piped or discarded stdio. Without it, tools such as uv report EACCES while +// probing an otherwise executable interpreter under an explicit filesystem +// policy. +const PROXY_BASELINE_READ_WRITE: &[&str] = &["/tmp", "/dev/null"]; + +/// GPU read-only paths. +/// +/// `/run/nvidia-persistenced`: NVML tries to connect to the persistenced +/// socket at init time. If the directory exists but Landlock denies traversal +/// (EACCES vs ECONNREFUSED), NVML returns `NVML_ERROR_INSUFFICIENT_PERMISSIONS` +/// even though the daemon is optional. Only read/traversal access is needed. +/// +/// `/usr/lib/wsl`: On WSL2, CDI bind-mounts GPU libraries (libdxcore.so, +/// libcuda.so.1.1, etc.) into paths under `/usr/lib/wsl/`. Although `/usr` +/// is already in `PROXY_BASELINE_READ_ONLY`, individual file bind-mounts may +/// not be covered by the parent-directory Landlock rule when the mount crosses +/// a filesystem boundary. Listing `/usr/lib/wsl` explicitly ensures traversal +/// is permitted regardless of Landlock's cross-mount behaviour. +const GPU_BASELINE_READ_ONLY: &[&str] = &[ + "/run/nvidia-persistenced", + "/usr/lib/wsl", // WSL2: CDI-injected GPU library directory +]; + +/// GPU read-write paths (static). +/// +/// `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools`, +/// `/dev/nvidia-modeset`: control and UVM devices injected by CDI on native +/// Linux. Landlock restricts `open(2)` on device files even when DAC allows +/// it; these need read-write because NVML/CUDA opens them with `O_RDWR`. +/// These devices do not exist on WSL2 and will be skipped by the existence +/// check in `enrich_proto_baseline_paths()`. +/// +/// `/dev/dxg`: On WSL2, NVIDIA GPUs are exposed through the DXG kernel driver +/// (DirectX Graphics) rather than the native nvidia* devices. CDI injects +/// `/dev/dxg` as the sole GPU device node; it does not exist on native Linux +/// and will be skipped there by the existence check. +/// +/// `/proc`: CUDA writes to `/proc//task//comm` during `cuInit()` +/// to set thread names. Without write access, `cuInit()` returns error 304. +/// Must use `/proc` (not `/proc/self/task`) because Landlock rules bind to +/// inodes and child processes have different procfs inodes than the parent. +/// +/// Per-GPU device files (`/dev/nvidia0`, …) are enumerated at runtime by +/// `enumerate_gpu_device_nodes()` since the count varies. +const GPU_BASELINE_READ_WRITE: &[&str] = &[ + "/dev/nvidiactl", + "/dev/nvidia-uvm", + "/dev/nvidia-uvm-tools", + "/dev/nvidia-modeset", + "/dev/dxg", // WSL2: DXG device (GPU via DirectX kernel driver, injected by CDI) + "/proc", +]; + +/// Returns true if GPU devices are present in the container. +/// +/// Checks both the native Linux NVIDIA control device (`/dev/nvidiactl`) and +/// the WSL2 DXG device (`/dev/dxg`). CDI injects exactly one of these +/// depending on the host kernel; the other will not exist. +fn has_gpu_devices() -> bool { + std::path::Path::new("/dev/nvidiactl").exists() || std::path::Path::new("/dev/dxg").exists() +} + +/// Enumerate per-GPU device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …). +fn enumerate_gpu_device_nodes() -> Vec { + let mut paths = Vec::new(); + if let Ok(entries) = std::fs::read_dir("/dev") { + for entry in entries.flatten() { + let name = entry.file_name(); + let name = name.to_string_lossy(); + if let Some(suffix) = name.strip_prefix("nvidia") { + if suffix.is_empty() || !suffix.chars().all(|c| c.is_ascii_digit()) { + continue; + } + paths.push(entry.path().to_string_lossy().into_owned()); + } + } + } + paths +} + +fn push_unique(paths: &mut Vec, path: String) { + if !paths.iter().any(|p| p == &path) { + paths.push(path); + } +} + +fn collect_baseline_enrichment_paths( + include_proxy: bool, + include_gpu: bool, + gpu_device_nodes: Vec, +) -> (Vec, Vec) { + let mut ro = Vec::new(); + let mut rw = Vec::new(); + + if include_proxy { + for &path in PROXY_BASELINE_READ_ONLY { + push_unique(&mut ro, path.to_string()); + } + for &path in PROXY_BASELINE_READ_WRITE { + push_unique(&mut rw, path.to_string()); + } + } + + if include_gpu { + for &path in GPU_BASELINE_READ_ONLY { + push_unique(&mut ro, path.to_string()); + } + for &path in GPU_BASELINE_READ_WRITE { + push_unique(&mut rw, path.to_string()); + } + for path in gpu_device_nodes { + push_unique(&mut rw, path); + } + } + + // A path promoted to read_write (e.g. /proc for GPU) should not also + // appear in read_only — Landlock handles the overlap correctly but the + // duplicate is confusing when inspecting the effective policy. + ro.retain(|p| !rw.contains(p)); + + (ro, rw) +} + +fn active_baseline_enrichment_paths(include_proxy: bool) -> (Vec, Vec) { + let include_gpu = has_gpu_devices(); + let gpu_device_nodes = if include_gpu { + enumerate_gpu_device_nodes() + } else { + Vec::new() + }; + collect_baseline_enrichment_paths(include_proxy, include_gpu, gpu_device_nodes) +} + +/// Collect all active baseline paths for tests and diagnostics. +/// Returns `(read_only, read_write)` as owned `String` vecs. +#[cfg(test)] +fn baseline_enrichment_paths() -> (Vec, Vec) { + active_baseline_enrichment_paths(true) +} + +fn enrich_proto_baseline_paths_with( + proto: &mut openshell_core::proto::SandboxPolicy, + ro: &[String], + rw: &[String], + path_exists: F, +) -> bool +where + F: Fn(&str) -> bool, +{ + if ro.is_empty() && rw.is_empty() { + return false; + } + + let fs = proto + .filesystem + .get_or_insert_with(|| openshell_core::proto::FilesystemPolicy { + include_workdir: true, + ..Default::default() + }); + + let mut modified = false; + for path in ro { + if !fs.read_only.iter().any(|p| p == path) && !fs.read_write.iter().any(|p| p == path) { + if !path_exists(path) { + debug!( + path, + "Baseline read-only path does not exist, skipping enrichment" + ); + continue; + } + fs.read_only.push(path.clone()); + modified = true; + } + } + for path in rw { + if fs.read_write.iter().any(|p| p == path) { + continue; + } + if !path_exists(path) { + debug!( + path, + "Baseline read-write path does not exist, skipping enrichment" + ); + continue; + } + if fs.read_only.iter().any(|p| p == path) { + if path == "/proc" { + info!( + path, + "Promoting /proc from read-only to read-write for GPU runtime compatibility" + ); + fs.read_only.retain(|p| p != path); + fs.read_write.push(path.clone()); + modified = true; + } + continue; + } + fs.read_write.push(path.clone()); + modified = true; + } + + modified +} + +/// Ensure a proto `SandboxPolicy` includes the baseline filesystem paths +/// required by proxy-mode sandboxes and GPU runtimes. Paths are only added if +/// missing; user-specified paths are never removed. +/// +/// Returns `true` if the policy was modified (caller may want to sync back). +fn enrich_proto_baseline_paths(proto: &mut openshell_core::proto::SandboxPolicy) -> bool { + let (ro, rw) = active_baseline_enrichment_paths(!proto.network_policies.is_empty()); + + // Baseline paths are system-injected, not user-specified. Skip paths + // that do not exist in this container image to avoid noisy warnings from + // Landlock and, more critically, to prevent a single missing baseline + // path from abandoning the entire Landlock ruleset under best-effort + // mode (see issue #664). + let modified = enrich_proto_baseline_paths_with(proto, &ro, &rw, |path| { + std::path::Path::new(path).exists() + }); + + if modified { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "enriched") + .message("Enriched policy with baseline filesystem paths for proxy mode") + .build() + ); + } + + modified +} + +fn strip_proto_provider_policy_entries(proto: &mut openshell_core::proto::SandboxPolicy) -> bool { + openshell_policy::strip_provider_rule_names(proto) +} + +fn proto_sync_payload_for_enriched_policy( + proto: &openshell_core::proto::SandboxPolicy, + enriched: bool, +) -> Option { + if !enriched { + return None; + } + + let mut sync_policy = proto.clone(); + strip_proto_provider_policy_entries(&mut sync_policy); + Some(sync_policy) +} + +/// Ensure a `SandboxPolicy` (Rust type) includes the baseline filesystem +/// paths required by proxy-mode sandboxes and GPU runtimes. Used for the +/// local-file code path where no proto is available. +fn enrich_sandbox_baseline_paths(policy: &mut SandboxPolicy) { + let (ro, rw) = + active_baseline_enrichment_paths(matches!(policy.network.mode, NetworkMode::Proxy)); + if ro.is_empty() && rw.is_empty() { + return; + } + + let mut modified = false; + for path in &ro { + let p = std::path::PathBuf::from(path); + if !policy.filesystem.read_only.contains(&p) && !policy.filesystem.read_write.contains(&p) { + if !p.exists() { + debug!( + path, + "Baseline read-only path does not exist, skipping enrichment" + ); + continue; + } + policy.filesystem.read_only.push(p); + modified = true; + } + } + for path in &rw { + let p = std::path::PathBuf::from(path); + if policy.filesystem.read_only.contains(&p) || policy.filesystem.read_write.contains(&p) { + continue; + } + if !p.exists() { + debug!( + path, + "Baseline read-write path does not exist, skipping enrichment" + ); + continue; + } + policy.filesystem.read_write.push(p); + modified = true; + } + + if modified { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "enriched") + .message("Enriched policy with baseline filesystem paths for proxy mode") + .build() + ); + } +} + +#[cfg(test)] +#[allow( + clippy::needless_raw_string_hashes, + clippy::iter_on_single_items, + clippy::similar_names, + clippy::manual_string_new, + clippy::doc_markdown, + reason = "Test code: test fixtures often use idiomatic forms not flagged in production." +)] +mod baseline_tests { + use super::*; + use openshell_core::policy::{FilesystemPolicy, LandlockPolicy, ProcessPolicy}; + use std::path::PathBuf; + + #[test] + fn proc_not_in_both_read_only_and_read_write_when_gpu_present() { + // When GPU devices are present, /proc is promoted to read_write + // (CUDA needs to write /proc//task//comm). It should + // NOT also appear in read_only. + if !has_gpu_devices() { + // Can't test GPU dedup without GPU devices; skip silently. + return; + } + let (ro, rw) = baseline_enrichment_paths(); + assert!( + rw.contains(&"/proc".to_string()), + "/proc should be in read_write when GPU is present" + ); + assert!( + !ro.contains(&"/proc".to_string()), + "/proc should NOT be in read_only when it is already in read_write" + ); + } + + #[test] + fn proc_in_read_only_without_gpu() { + if has_gpu_devices() { + // On a GPU host we can't test the non-GPU path; skip silently. + return; + } + let (ro, _rw) = baseline_enrichment_paths(); + assert!( + ro.contains(&"/proc".to_string()), + "/proc should be in read_only when GPU is not present" + ); + } + + #[test] + fn baseline_read_write_does_not_hardcode_sandbox() { + let (_ro, rw) = baseline_enrichment_paths(); + assert!(rw.contains(&"/tmp".to_string())); + assert!(rw.contains(&"/dev/null".to_string())); + assert!(!rw.contains(&"/sandbox".to_string())); + } + + #[test] + fn enumerate_gpu_device_nodes_skips_bare_nvidia() { + // "nvidia" (without a trailing digit) is a valid /dev entry on some + // systems but is not a per-GPU device node. The enumerator must + // not match it. + let nodes = enumerate_gpu_device_nodes(); + assert!( + !nodes.contains(&"/dev/nvidia".to_string()), + "bare /dev/nvidia should not be enumerated: {nodes:?}" + ); + } + + #[test] + fn no_duplicate_paths_in_baseline() { + let (ro, rw) = baseline_enrichment_paths(); + // No path should appear in both lists. + for path in &ro { + assert!( + !rw.contains(path), + "path {path} appears in both read_only and read_write" + ); + } + } + + #[test] + fn proto_enrichment_preserves_explicit_read_only_for_baseline_read_write_paths() { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.filesystem = Some(openshell_core::proto::FilesystemPolicy { + read_only: vec!["/tmp".to_string()], + read_write: vec![], + include_workdir: false, + }); + policy.network_policies.insert( + "test".into(), + openshell_core::proto::NetworkPolicyRule { + name: "test-rule".into(), + endpoints: vec![openshell_core::proto::NetworkEndpoint { + host: "example.com".into(), + port: 443, + ..Default::default() + }], + ..Default::default() + }, + ); + + enrich_proto_baseline_paths(&mut policy); + + let filesystem = policy.filesystem.expect("filesystem policy"); + assert!( + filesystem.read_only.contains(&"/tmp".to_string()), + "explicit read_only baseline path should be preserved" + ); + assert!( + !filesystem.read_write.contains(&"/tmp".to_string()), + "baseline enrichment must not promote explicit read_only /tmp to read_write" + ); + } + + #[test] + fn proto_strip_provider_policy_entries_removes_only_reserved_entries() { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.network_policies.insert( + "_provider_work_github".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: "_provider_work_github".to_string(), + ..Default::default() + }, + ); + policy.network_policies.insert( + "sandbox_only".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: "sandbox_only".to_string(), + ..Default::default() + }, + ); + + assert!(strip_proto_provider_policy_entries(&mut policy)); + assert!( + !policy + .network_policies + .contains_key("_provider_work_github") + ); + assert!(policy.network_policies.contains_key("sandbox_only")); + assert!(!strip_proto_provider_policy_entries(&mut policy)); + } + + #[test] + fn proto_sync_payload_not_created_for_provider_entries_without_enrichment() { + let mut runtime_policy = openshell_policy::restrictive_default_policy(); + runtime_policy.network_policies.insert( + "_provider_work_github".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: "_provider_work_github".to_string(), + ..Default::default() + }, + ); + + assert!(proto_sync_payload_for_enriched_policy(&runtime_policy, false).is_none()); + assert!( + runtime_policy + .network_policies + .contains_key("_provider_work_github"), + "provider-derived rules alone must not trigger sync or mutate runtime policy" + ); + } + + #[test] + fn proto_sync_payload_for_enrichment_strips_provider_entries_without_mutating_runtime_policy() { + let mut runtime_policy = openshell_policy::restrictive_default_policy(); + runtime_policy.network_policies.insert( + "_provider_work_github".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: "_provider_work_github".to_string(), + ..Default::default() + }, + ); + runtime_policy.network_policies.insert( + "sandbox_only".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: "sandbox_only".to_string(), + ..Default::default() + }, + ); + + let sync_policy = proto_sync_payload_for_enriched_policy(&runtime_policy, true) + .expect("enrichment should create a sync payload"); + + assert!( + runtime_policy + .network_policies + .contains_key("_provider_work_github"), + "runtime policy must retain provider-derived rules for OPA input" + ); + assert!( + !sync_policy + .network_policies + .contains_key("_provider_work_github") + ); + assert!(sync_policy.network_policies.contains_key("sandbox_only")); + } + + #[test] + fn proto_gpu_enrichment_promotes_proc_without_network_policy() { + let mut policy = openshell_policy::restrictive_default_policy(); + assert!( + policy.network_policies.is_empty(), + "regression setup must exercise the no-network default path" + ); + let (ro, rw) = + collect_baseline_enrichment_paths(false, true, vec!["/dev/nvidia0".to_string()]); + + let enriched = enrich_proto_baseline_paths_with(&mut policy, &ro, &rw, |path| { + matches!(path, "/proc" | "/dev/nvidia0") + }); + + let filesystem = policy.filesystem.expect("filesystem policy"); + assert!( + enriched, + "GPU enrichment should not require network policies" + ); + assert!( + filesystem.read_write.contains(&"/dev/nvidia0".to_string()), + "GPU enrichment should add enumerated device nodes without network policies" + ); + assert!( + !filesystem.read_only.contains(&"/proc".to_string()), + "GPU enrichment should remove /proc from read_only" + ); + assert!( + filesystem.read_write.contains(&"/proc".to_string()), + "GPU enrichment should promote /proc to read_write" + ); + } + + #[test] + fn gpu_baseline_read_write_contains_dxg() { + // /dev/dxg must be present so WSL2 sandboxes get the Landlock + // read-write rule for the CDI-injected DXG device. The existence + // check in enrich_proto_baseline_paths() skips it on native Linux. + assert!( + GPU_BASELINE_READ_WRITE.contains(&"/dev/dxg"), + "/dev/dxg must be in GPU_BASELINE_READ_WRITE for WSL2 support" + ); + } + + #[test] + fn local_enrichment_preserves_explicit_read_only_for_baseline_read_write_paths() { + let mut policy = SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy { + read_only: vec![PathBuf::from("/tmp")], + read_write: vec![], + include_workdir: false, + }, + network: NetworkPolicy { + mode: NetworkMode::Proxy, + proxy: Some(ProxyPolicy { http_addr: None }), + }, + landlock: LandlockPolicy::default(), + process: ProcessPolicy::default(), + }; + + enrich_sandbox_baseline_paths(&mut policy); + + assert!( + policy.filesystem.read_only.contains(&PathBuf::from("/tmp")), + "explicit read_only baseline path should be preserved" + ); + assert!( + !policy + .filesystem + .read_write + .contains(&PathBuf::from("/tmp")), + "baseline enrichment must not promote explicit read_only /tmp to read_write" + ); + } + + #[test] + fn gpu_baseline_read_only_contains_usr_lib_wsl() { + // /usr/lib/wsl must be present so CDI-injected WSL2 GPU library + // bind-mounts are accessible under Landlock. Skipped on native Linux. + assert!( + GPU_BASELINE_READ_ONLY.contains(&"/usr/lib/wsl"), + "/usr/lib/wsl must be in GPU_BASELINE_READ_ONLY for WSL2 CDI library paths" + ); + } + + #[test] + fn has_gpu_devices_reflects_dxg_or_nvidiactl() { + // Verify the OR logic: result must match the manual disjunction of + // the two path checks. Passes in all environments. + let nvidiactl = std::path::Path::new("/dev/nvidiactl").exists(); + let dxg = std::path::Path::new("/dev/dxg").exists(); + assert_eq!( + has_gpu_devices(), + nvidiactl || dxg, + "has_gpu_devices() should be true iff /dev/nvidiactl or /dev/dxg exists" + ); + } +} + +/// Returns `true` if the error is transient and worth retrying. +/// +/// Walks the `miette::Report` error chain looking for a `tonic::Status`. If +/// found, only the gRPC codes that represent transient failures are retryable. +/// If no `tonic::Status` is present (e.g. a raw connection error), assume the +/// failure is transient. +fn is_retryable_error(err: &miette::Report) -> bool { + let mut source: Option<&dyn std::error::Error> = Some(err.as_ref()); + while let Some(e) = source { + if let Some(status) = e.downcast_ref::() { + return matches!( + status.code(), + tonic::Code::Unavailable + | tonic::Code::DeadlineExceeded + | tonic::Code::ResourceExhausted + | tonic::Code::Aborted + | tonic::Code::Internal + | tonic::Code::Unknown + ); + } + source = e.source(); + } + true +} + +/// Retry a gRPC operation with exponential backoff (capped at 4 s). +/// +/// Non-transient gRPC errors (e.g. `NOT_FOUND`, `INVALID_ARGUMENT`, +/// `PERMISSION_DENIED`) are returned immediately without retrying. +async fn grpc_retry(op_name: &str, f: F) -> Result +where + F: Fn() -> Fut, + Fut: Future>, +{ + let mut last_err = None; + for attempt in 1..=5u32 { + match f().await { + Ok(val) => return Ok(val), + Err(e) => { + if !is_retryable_error(&e) { + return Err(e); + } + if attempt < 5 { + warn!( + attempt, + max_attempts = 5, + error = %e, + "{op_name} failed, retrying" + ); + let backoff = Duration::from_secs((1u64 << (attempt - 1)).min(4)); + tokio::time::sleep(backoff).await; + } + last_err = Some(e); + } + } + } + Err(miette::miette!( + "{op_name} failed after 5 attempts: {}", + last_err.expect("loop executed at least once") + )) +} + +/// Load sandbox policy from local files or gRPC. +/// +/// Priority: +/// 1. If `policy_rules` and `policy_data` are provided, load OPA engine from local files +/// 2. If `sandbox_id` and `openshell_endpoint` are provided, fetch via gRPC +/// 3. If the server returns no policy, discover from disk or use restrictive default +/// 4. Otherwise, return an error +/// +/// Returns the policy, the OPA engine, and (for gRPC mode) the original proto +/// policy. The proto is retained so the OPA engine can be rebuilt with symlink +/// resolution after the container entrypoint starts. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum LocalPolicyIdentity { + Required, + EndpointOnly, +} + +async fn load_policy( + sandbox_id: Option, + sandbox: Option, + openshell_endpoint: Option, + policy_rules: Option, + policy_data: Option, + extension_credentials: &openshell_extension_core::ExtensionCredentialStore, + local_policy_identity: LocalPolicyIdentity, +) -> Result<( + SandboxPolicy, + Option>, + Option, + MiddlewareRegistryStatus, + LoadedPolicyOrigin, + bool, + bool, +)> { + // File mode: load OPA engine from rego rules + YAML data (dev override) + if let (Some(policy_file), Some(data_file)) = (&policy_rules, &policy_data) { + ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Other, "loading") + .unmapped("policy_rules", serde_json::json!(policy_file)) + .unmapped("policy_data", serde_json::json!(data_file)) + .message(format!( + "Loading OPA policy engine from local files [rules:{policy_file} data:{data_file}]" + )) + .build()); + let validate_middleware_config = |implementation: &str, config: &prost_types::Struct| { + openshell_supervisor_middleware_builtins::validate_config(implementation, config) + .map_err(|error| error.to_string()) + }; + let policy_path = std::path::Path::new(policy_file); + let data_path = std::path::Path::new(data_file); + let engine = match local_policy_identity { + LocalPolicyIdentity::Required => OpaEngine::from_files_with_middleware_config( + policy_path, + data_path, + Some(&validate_middleware_config), + )?, + LocalPolicyIdentity::EndpointOnly => OpaEngine::from_files_for_endpoint_only_proxy( + policy_path, + data_path, + Some(&validate_middleware_config), + )?, + }; + let middleware_registry = + openshell_supervisor_middleware::MiddlewareRegistry::connect_services( + openshell_supervisor_middleware_builtins::services(), + Vec::new(), + ) + .await?; + engine.replace_middleware_registry(middleware_registry)?; + let config = engine.query_sandbox_config()?; + let mut policy = SandboxPolicy { + version: 1, + filesystem: config.filesystem, + network: NetworkPolicy { + mode: NetworkMode::Proxy, + proxy: Some(ProxyPolicy { http_addr: None }), + }, + landlock: config.landlock, + process: config.process, + }; + enrich_sandbox_baseline_paths(&mut policy); + // File mode has no operator-registered middleware to connect. + return Ok(( + policy, + Some(Arc::new(engine)), + None, + MiddlewareRegistryStatus::Synchronized, + LoadedPolicyOrigin::LocalOverride, + false, + false, + )); + } + + // gRPC mode: fetch typed proto policy, construct OPA engine from baked rules + proto data + if let (Some(id), Some(endpoint)) = (&sandbox_id, &openshell_endpoint) { + info!( + sandbox_id = %id, + endpoint = %endpoint, + "Fetching sandbox policy via gRPC" + ); + let mut snapshot = grpc_retry("Policy fetch", || { + openshell_core::grpc_client::fetch_settings_snapshot(endpoint, id) + }) + .await?; + + let mut proto_policy = if let Some(p) = snapshot.policy.clone() { + p + } else { + // No policy configured on the server. Discover from disk or + // fall back to the restrictive default, then sync to the + // gateway so it becomes the authoritative baseline. + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Other, "discovery") + .message("Server returned no policy; attempting local discovery") + .build() + ); + let mut discovered = discover_policy_from_disk_or_default(); + // Enrich before syncing so the gateway baseline includes + // baseline paths from the start. + enrich_proto_baseline_paths(&mut discovered); + strip_proto_provider_policy_entries(&mut discovered); + let sandbox = sandbox.as_deref().ok_or_else(|| { + miette::miette!( + "Cannot sync discovered policy: sandbox not available.\n\ + Set OPENSHELL_SANDBOX or --sandbox to enable policy sync." + ) + })?; + + // Sync and re-fetch over a single connection to avoid extra + // TLS handshakes. + let ws = snapshot.workspace.clone(); + snapshot = grpc_retry("Policy discovery sync", || { + openshell_core::grpc_client::sync_policy_and_fetch_snapshot( + endpoint, + id, + sandbox, + &discovered, + &ws, + ) + }) + .await?; + snapshot.policy.clone().ok_or_else(|| { + miette::miette!("Server still returned no policy after sync — this is a bug") + })? + }; + + // True only while `snapshot` describes the exact policy that will be + // constructed below. If enrichment cannot be synced and re-fetched, + // the policy remains enforceable but cannot be acknowledged by + // inferred structural equality. + let mut policy_bound_to_snapshot = true; + + // Ensure baseline filesystem paths are present for proxy-mode + // sandboxes. If the policy was enriched, sync the updated version + // back to the gateway so users can see the effective policy. + let enriched = enrich_proto_baseline_paths(&mut proto_policy); + let sync_policy = proto_sync_payload_for_enriched_policy(&proto_policy, enriched); + if let Some(sync_policy) = sync_policy { + if let Some(sandbox_name) = sandbox.as_deref() { + match openshell_core::grpc_client::sync_policy_and_fetch_snapshot( + endpoint, + id, + sandbox_name, + &sync_policy, + &snapshot.workspace, + ) + .await + { + Ok(canonical) => { + if let Some(policy) = canonical.policy.clone() { + proto_policy = policy; + snapshot = canonical; + } else { + policy_bound_to_snapshot = false; + warn!( + "Gateway returned no policy after enrichment sync; initial revision will be reconciled" + ); + } + } + Err(e) => { + policy_bound_to_snapshot = false; + warn!( + error = %e, + "Failed to sync enriched policy back to gateway; initial revision will be reconciled" + ); + } + } + } else { + policy_bound_to_snapshot = false; + } + } + + let mut loaded_policy_revision = + policy_bound_to_snapshot.then(|| LoadedPolicyRevision::from_snapshot(&snapshot)); + + // Build OPA engine from baked-in rules + typed proto data. + // In cluster mode, proxy networking is always enabled so OPA is + // always required for allow/deny decisions. + // The initial load uses pid=0 (no symlink resolution) because the + // container hasn't started yet. After the entrypoint spawns, the + // engine is rebuilt with the real PID for symlink resolution. + info!("Creating OPA engine from proto policy data"); + let mut has_last_valid_policy = true; + let engine = match OpaEngine::from_proto(&proto_policy) { + Ok(engine) => Arc::new(engine), + Err(e) => { + report_initial_policy_failure(endpoint, id, loaded_policy_revision.as_ref(), &e) + .await; + let validation_error = e.to_string(); + let candidate_version = snapshot.version; + let candidate_hash = snapshot.policy_hash.clone(); + // There is no in-memory last-known-good generation during + // startup, so both configured modes necessarily fail closed. + // Load the restrictive default atomically and keep the + // rejected revision unacknowledged for poll reconciliation. + has_last_valid_policy = false; + proto_policy = openshell_policy::restrictive_default_policy(); + let engine = Arc::new(OpaEngine::from_proto(&proto_policy)?); + let disposition = apply_policy_validation_failure( + &engine, + snapshot.policy_validation_failure_mode, + has_last_valid_policy, + candidate_version, + &validation_error, + )?; + emit_policy_validation_failure( + &disposition, + candidate_version, + &candidate_hash, + &validation_error, + ); + loaded_policy_revision = None; + engine + } + }; + + // Install the in-process catalog before any external connection can + // fail. A newly started sandbox must always be able to resolve built-in + // bindings, even while operator-run services are unavailable. + install_builtin_middleware_registry(&engine).await?; + + // Connect operator-registered middleware services. A connect/describe + // failure keeps the built-in registry active so each request's + // `on_error` policy governs matched traffic. The policy poll loop + // retries the install without waiting for a config change. + let middleware_services = snapshot.supervisor_middleware_services.clone(); + let middleware_registry_status = if middleware_services.is_empty() { + MiddlewareRegistryStatus::Synchronized + } else if let Err(error) = grpc_retry("Middleware connect", || { + let middleware_services = middleware_services.clone(); + let extension_credentials = extension_credentials.clone(); + let extension_authentication_enabled = snapshot.extension_authentication_enabled; + async move { + let credentials = if extension_authentication_enabled { + // Share the supervisor's store so the slots installed here + // are the ones the policy poll loop later rotates in place. + openshell_core::grpc_client::CachedOpenShellClient::connect_with_credentials( + endpoint, + extension_credentials, + ) + .await? + .refresh_extension_credentials(&middleware_services) + .await? + } else { + std::collections::HashMap::new() + }; + connect_middleware_registry( + &middleware_services, + &MiddlewareAuthentication { + credentials, + enabled: extension_authentication_enabled, + }, + ) + .await + } + }) + .await + .and_then(|registry| engine.replace_middleware_registry(registry)) + { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .state(StateId::Other, "degraded") + .unmapped( + "supervisor_middleware_service_count", + serde_json::json!(middleware_services.len()) + ) + .message(format!( + "Supervisor middleware connect failed at startup; continuing with built-in middleware only, per-request on_error governs matched requests [error:{error}]" + )) + .build() + ); + MiddlewareRegistryStatus::NeedsReconciliation + } else { + MiddlewareRegistryStatus::Synchronized + }; + let opa_engine = Some(engine); + + let policy = match SandboxPolicy::try_from(proto_policy.clone()) { + Ok(policy) => policy, + Err(e) => { + report_initial_policy_failure(endpoint, id, loaded_policy_revision.as_ref(), &e) + .await; + return Err(e); + } + }; + return Ok(( + policy, + opa_engine, + Some(proto_policy), + middleware_registry_status, + LoadedPolicyOrigin::Gateway { + revision: loaded_policy_revision, + has_last_valid_policy, + }, + agent_proposals_enabled_from_settings(&snapshot.settings), + snapshot.extension_authentication_enabled, + )); + } + + // No policy source available + Err(miette::miette!( + "Sandbox policy required. Provide one of:\n\ + - --policy-rules and --policy-data (or OPENSHELL_POLICY_RULES and OPENSHELL_POLICY_DATA env vars)\n\ + - --sandbox-id and --openshell-endpoint (or OPENSHELL_SANDBOX_ID and OPENSHELL_ENDPOINT env vars)" + )) +} + +/// Try to discover a sandbox policy from the well-known disk path, falling +/// back to the legacy path, then to the hardcoded restrictive default. +fn discover_policy_from_disk_or_default() -> openshell_core::proto::SandboxPolicy { + let primary = std::path::Path::new(openshell_policy::CONTAINER_POLICY_PATH); + if primary.exists() { + return discover_policy_from_path(primary); + } + let legacy = std::path::Path::new(openshell_policy::LEGACY_CONTAINER_POLICY_PATH); + if legacy.exists() { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped( + "legacy_path", + serde_json::json!(legacy.display().to_string()) + ) + .unmapped("new_path", serde_json::json!(primary.display().to_string())) + .message(format!( + "Policy found at legacy path; consider moving [legacy_path:{} new_path:{}]", + legacy.display(), + primary.display() + )) + .build() + ); + return discover_policy_from_path(legacy); + } + discover_policy_from_path(primary) +} + +/// Try to read a sandbox policy YAML from `path`, falling back to the +/// hardcoded restrictive default if the file is missing or invalid. +fn discover_policy_from_path(path: &std::path::Path) -> openshell_core::proto::SandboxPolicy { + use openshell_policy::{ + parse_sandbox_policy, restrictive_default_policy, validate_sandbox_policy, + }; + + let Ok(yaml) = std::fs::read_to_string(path) else { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "default") + .message(format!( + "No policy file on disk, using restrictive default [path:{}]", + path.display() + )) + .build() + ); + return restrictive_default_policy(); + }; + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .message(format!( + "Loaded sandbox policy from container disk [path:{}]", + path.display() + )) + .build() + ); + match parse_sandbox_policy(&yaml) { + Ok(policy) => { + // Validate the disk-loaded policy for safety. + if let Err(violations) = validate_sandbox_policy(&policy) { + let messages: Vec = violations.iter().map(ToString::to_string).collect(); + ocsf_emit!(DetectionFindingBuilder::new(ocsf_ctx()) + .activity(ActivityId::Open) + .severity(SeverityId::Medium) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .finding_info( + FindingInfo::new( + "unsafe-disk-policy", + "Unsafe Disk Policy Content", + ) + .with_desc(&format!( + "Disk policy at {} contains unsafe content: {}", + path.display(), + messages.join("; "), + )), + ) + .message(format!( + "Disk policy contains unsafe content, using restrictive default [path:{}]", + path.display() + )) + .build()); + return restrictive_default_policy(); + } + policy + } + Err(e) => { + ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .state(StateId::Other, "fallback") + .message(format!( + "Failed to parse disk policy, using restrictive default [path:{} error:{e}]", + path.display() + )) + .build()); + restrictive_default_policy() + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum MiddlewareRegistryStatus { + Synchronized, + NeedsReconciliation, +} + +#[derive(Debug)] +enum GatewayRuntimeReloadError { + PolicyValidation(miette::Report), + TransparentTcpPrerequisite(miette::Report), + MiddlewareRegistry(miette::Report), +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum GatewayRuntimeFailureClass { + PolicyValidation, + TransparentTcpPrerequisite, + MiddlewareRegistry, +} + +impl GatewayRuntimeReloadError { + fn class(&self) -> GatewayRuntimeFailureClass { + match self { + Self::PolicyValidation(_) => GatewayRuntimeFailureClass::PolicyValidation, + Self::TransparentTcpPrerequisite(_) => { + GatewayRuntimeFailureClass::TransparentTcpPrerequisite + } + Self::MiddlewareRegistry(_) => GatewayRuntimeFailureClass::MiddlewareRegistry, + } + } +} + +#[derive(Debug, PartialEq, Eq)] +struct FailedRuntimeRevision { + config_revision: u64, + policy_hash: String, + failure_class: GatewayRuntimeFailureClass, +} + +impl FailedRuntimeRevision { + fn new(config_revision: u64, policy_hash: &str, failure: &GatewayRuntimeReloadError) -> Self { + Self { + config_revision, + policy_hash: policy_hash.to_string(), + failure_class: failure.class(), + } + } +} + +struct MiddlewareReloadContext<'a> { + desired_services: &'a [openshell_core::proto::SupervisorMiddlewareService], + authentication: &'a MiddlewareAuthentication, + registry_changed: bool, + connector: &'a MiddlewareConnector, +} + +async fn reload_gateway_policy_runtime( + engine: &OpaEngine, + policy: Option<&openshell_core::proto::SandboxPolicy>, + entrypoint_pid: u32, + middleware: MiddlewareReloadContext<'_>, + transparent_tcp: TransparentTcpReloadState, +) -> std::result::Result<(), GatewayRuntimeReloadError> { + if let Some(policy) = policy + && policy_contains_explicit_tcp(policy) + { + if !transparent_tcp.capable { + return Err(GatewayRuntimeReloadError::TransparentTcpPrerequisite( + miette::miette!( + "candidate policy introduces protocol: tcp, but the runtime does not advertise transparent TCP support; previous policy remains active" + ), + )); + } + if !transparent_tcp.substrate_ready { + return Err(GatewayRuntimeReloadError::TransparentTcpPrerequisite( + miette::miette!( + "candidate policy introduces protocol: tcp, but this sandbox started without the transparent TCP substrate; recreate the sandbox to enable TCP; previous policy remains active" + ), + )); + } + } + match policy { + Some(policy) if middleware.registry_changed => { + let registry = (middleware.connector)( + middleware.desired_services.to_vec(), + middleware.authentication.clone(), + ) + .await + .map_err(GatewayRuntimeReloadError::MiddlewareRegistry)?; + engine + .reload_policy_and_middleware_from_proto_with_pid(policy, entrypoint_pid, registry) + .map_err(GatewayRuntimeReloadError::PolicyValidation) + } + // Policy-only change: the installed registry already matches the + // delivered service set, so swap the engine alone. This must not + // require middleware reachability. + Some(policy) => engine + .reload_from_proto_with_pid(policy, entrypoint_pid) + .map_err(GatewayRuntimeReloadError::PolicyValidation), + None => Err(GatewayRuntimeReloadError::PolicyValidation( + miette::miette!("runtime reload requires a policy payload but none was returned"), + )), + } +} + +fn policy_contains_explicit_tcp(policy: &openshell_core::proto::SandboxPolicy) -> bool { + policy.network_policies.values().any(|rule| { + rule.endpoints + .iter() + .any(|endpoint| endpoint.protocol.eq_ignore_ascii_case("tcp")) + }) +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +struct TransparentTcpReloadState { + capable: bool, + substrate_ready: bool, +} + +/// True when the installed middleware registry no longer matches the desired +/// service set and must be rebuilt (reconnecting every delivered service). +/// +/// A policy-only change never requires a rebuild: middleware configs were +/// validated at gateway admission and the installed registry's manifests +/// already cover the unchanged service set, so requiring the services to be +/// reachable would only let a middleware outage block the policy update. +fn middleware_registry_needs_rebuild( + registry_status: MiddlewareRegistryStatus, + current_services: &[openshell_core::proto::SupervisorMiddlewareService], + desired_services: &[openshell_core::proto::SupervisorMiddlewareService], +) -> bool { + registry_status == MiddlewareRegistryStatus::NeedsReconciliation + || current_services != desired_services +} + +fn gateway_policy_runtime_needs_reconciliation( + reloads_gateway_policy: bool, + current_policy_hash: &str, + desired_policy_hash: &str, + current_services: &[openshell_core::proto::SupervisorMiddlewareService], + desired_services: &[openshell_core::proto::SupervisorMiddlewareService], + registry_status: MiddlewareRegistryStatus, +) -> bool { + reloads_gateway_policy + && (current_policy_hash != desired_policy_hash + || middleware_registry_needs_rebuild( + registry_status, + current_services, + desired_services, + )) +} + +/// Identity returned with the exact policy snapshot used to construct OPA. +#[derive(Clone, Debug, PartialEq, Eq)] +struct LoadedPolicyRevision { + version: u32, + policy_hash: String, + config_revision: u64, + policy_source: openshell_core::proto::PolicySource, +} + +/// Identifies where the policy currently loaded into OPA came from. +/// +/// A missing gateway revision means the policy was loaded from the gateway but +/// could not be bound to an authoritative snapshot (for example, enrichment +/// sync failed). That state must reconcile on the first successful poll. A +/// local-file override is different: gateway policy revisions are observed for +/// settings/provider refreshes but must never replace the explicit local OPA +/// policy. +#[derive(Clone, Debug, PartialEq, Eq)] +enum LoadedPolicyOrigin { + LocalOverride, + Gateway { + revision: Option, + has_last_valid_policy: bool, + }, +} + +impl LoadedPolicyOrigin { + fn allows_gateway_policy_reload(&self) -> bool { + matches!(self, Self::Gateway { .. }) + } + + fn has_last_valid_policy(&self) -> bool { + match self { + Self::LocalOverride => true, + Self::Gateway { + has_last_valid_policy, + .. + } => *has_last_valid_policy, + } + } +} + +impl LoadedPolicyRevision { + fn from_snapshot(snapshot: &openshell_core::grpc_client::SettingsPollResult) -> Self { + Self { + version: snapshot.version, + policy_hash: snapshot.policy_hash.clone(), + config_revision: snapshot.config_revision, + policy_source: snapshot.policy_source, + } + } +} + +/// A sandbox-scoped policy revision that was constructed successfully at +/// startup and must be acknowledged to the gateway exactly once. +#[derive(Clone, Debug, PartialEq, Eq)] +struct InitialPolicyAck { + version: u32, + policy_hash: String, + config_revision: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +struct PolicyStatusUpdate { + version: u32, + loaded: bool, + error: String, + success_event: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +enum PolicyStatusSuccessEvent { + InitialAcknowledgement { policy_hash: String }, + UnchangedAcknowledgement { policy_hash: String }, +} + +impl PolicyStatusUpdate { + fn initial_loaded(ack: &InitialPolicyAck) -> Self { + Self { + version: ack.version, + loaded: true, + error: String::new(), + success_event: Some(PolicyStatusSuccessEvent::InitialAcknowledgement { + policy_hash: ack.policy_hash.clone(), + }), + } + } + + fn loaded(version: u32) -> Self { + Self { + version, + loaded: true, + error: String::new(), + success_event: None, + } + } + + fn unchanged_loaded(version: u32, policy_hash: String) -> Self { + Self { + version, + loaded: true, + error: String::new(), + success_event: Some(PolicyStatusSuccessEvent::UnchangedAcknowledgement { policy_hash }), + } + } + + fn failed(version: u32, error: String) -> Self { + Self { + version, + loaded: false, + error, + success_event: None, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +enum InitialPollDisposition { + Acknowledge(InitialPolicyAck), + Reconcile, + TrackOnly, +} + +/// Determine whether the initially loaded policy corresponds to an +/// authoritative sandbox-scoped revision that must be acknowledged. +/// +/// Returns `Some` only for sandbox-sourced revisions (version > 0) whose +/// captured gateway identity matches the current version and hash. Global +/// policies, local-file development policies, version zero, and changed +/// identities yield `None`, so those paths never emit a sandbox-revision +/// acknowledgement. +fn initial_policy_ack_candidate( + loaded: Option<&LoadedPolicyRevision>, + canonical: &openshell_core::grpc_client::SettingsPollResult, +) -> Option { + let loaded = loaded?; + if loaded.policy_source != openshell_core::proto::PolicySource::Sandbox + || canonical.policy_source != openshell_core::proto::PolicySource::Sandbox + { + return None; + } + if loaded.version == 0 || canonical.version == 0 { + return None; + } + if loaded.version != canonical.version + || loaded.policy_hash != canonical.policy_hash + || canonical.config_revision < loaded.config_revision + { + return None; + } + Some(InitialPolicyAck { + version: loaded.version, + policy_hash: loaded.policy_hash.clone(), + config_revision: canonical.config_revision, + }) +} + +fn initial_poll_disposition( + origin: &LoadedPolicyOrigin, + canonical: &openshell_core::grpc_client::SettingsPollResult, +) -> InitialPollDisposition { + match origin { + LoadedPolicyOrigin::LocalOverride => InitialPollDisposition::TrackOnly, + LoadedPolicyOrigin::Gateway { revision, .. } => { + initial_policy_ack_candidate(revision.as_ref(), canonical).map_or( + InitialPollDisposition::Reconcile, + InitialPollDisposition::Acknowledge, + ) + } + } +} + +fn unchanged_policy_revision_candidate( + reloads_gateway_policy: bool, + recovering_rejected_policy: bool, + current_policy_version: u32, + current_policy_hash: &str, + result: &openshell_core::grpc_client::SettingsPollResult, +) -> Option { + (reloads_gateway_policy + && !recovering_rejected_policy + && !current_policy_hash.is_empty() + && result.policy_source == openshell_core::proto::PolicySource::Sandbox + && result.version > current_policy_version + && result.policy_hash == current_policy_hash) + .then_some(result.version) +} + +fn unchanged_policy_revision_ready_to_ack( + candidate: Option, + policy_runtime_changed: bool, + policy_runtime_reconciled: bool, +) -> Option { + candidate.filter(|_| !policy_runtime_changed || policy_runtime_reconciled) +} + +/// Whether the credential-provenance gates cannot apply to the loaded policy. +/// +/// The gateway derives `provider_credentialed` and deliberately keeps it out of +/// the policy YAML schema, so a local-file policy never carries it and never +/// will: gateway revisions are observed for settings and providers but must not +/// replace the local OPA policy. Provider credentials still arrive from the +/// gateway on that path, so the raw-tunnel and WebSocket binary-frame refusals +/// have nothing to match on. The request-body backstop is unaffected because it +/// keys off the secret resolver rather than endpoint provenance. +fn credential_gating_unavailable( + origin: &LoadedPolicyOrigin, + has_resolver: bool, + network_enabled: bool, +) -> bool { + network_enabled && has_resolver && matches!(origin, LoadedPolicyOrigin::LocalOverride) +} + +/// Report that credential provenance is unavailable for the loaded policy. +/// +/// Carries no credential name, host, or value: the finding states which +/// controls are inactive, nothing about what they would have protected. +fn report_credential_gating_unavailable() { + ocsf_emit!( + DetectionFindingBuilder::new(ocsf_ctx()) + .activity(ActivityId::Open) + .severity(SeverityId::High) + .confidence(ConfidenceId::High) + .is_alert(true) + .finding_info( + FindingInfo::new( + "credential-gating-unavailable", + "Credential Provenance Unavailable", + ) + .with_desc( + "Provider credentials are injected, but the loaded policy comes from local \ + files and carries no gateway-derived credential provenance. Uninspected \ + credentialed tunnels and WebSocket binary frames are not refused. Load \ + policy from the gateway to enable these controls." + ), + ) + .evidence_pairs(&[ + ("policy_source", "local-override"), + ("uninspected_connect_gate", "inactive"), + ("websocket_binary_gate", "inactive"), + ("request_body_backstop", "active"), + ]) + .remediation( + "Remove the local policy override so the gateway-delivered effective policy \ + applies, or detach provider credentials from this sandbox." + ) + .message( + "Credential provenance unavailable for local-file policy; uninspected credential gates inactive" + ) + .build() + ); +} + +/// Deliver policy status updates independently from policy reconciliation. +/// +/// The channel is FIFO, so a delayed older status can never arrive after a +/// newer status and move the gateway's active version backward. Delivery uses +/// the existing bounded retry, but failures never delay policy enforcement. +#[tonic::async_trait] +trait PolicyGatewayClient: Clone + Send + Sync + 'static { + async fn poll_settings( + &self, + sandbox_id: &str, + ) -> Result; + + async fn report_policy_status( + &self, + sandbox_id: &str, + version: u32, + loaded: bool, + error: &str, + ) -> Result<()>; + + async fn refresh_installed_extension_credentials(&self) -> Result<()> { + Ok(()) + } + + async fn extension_credentials_for( + &self, + _services: &[openshell_core::proto::SupervisorMiddlewareService], + ) -> Result> { + Ok(std::collections::HashMap::new()) + } + + fn workspace(&self) -> String; +} + +#[tonic::async_trait] +impl PolicyGatewayClient for openshell_core::grpc_client::CachedOpenShellClient { + async fn poll_settings( + &self, + sandbox_id: &str, + ) -> Result { + self.poll_settings(sandbox_id).await + } + + async fn report_policy_status( + &self, + sandbox_id: &str, + version: u32, + loaded: bool, + error: &str, + ) -> Result<()> { + self.report_policy_status(sandbox_id, version, loaded, error) + .await + } + + async fn refresh_installed_extension_credentials(&self) -> Result<()> { + self.refresh_installed_extension_credentials().await + } + + async fn extension_credentials_for( + &self, + services: &[openshell_core::proto::SupervisorMiddlewareService], + ) -> Result> { + self.extension_credentials_for(services).await + } + + fn workspace(&self) -> String { + self.workspace() + } +} + +async fn run_policy_status_reporter( + client: C, + sandbox_id: String, + mut updates: tokio::sync::mpsc::UnboundedReceiver, +) { + 'updates: while let Some(update) = updates.recv().await { + let operation = if matches!( + update.success_event, + Some(PolicyStatusSuccessEvent::InitialAcknowledgement { .. }) + ) { + "Initial policy acknowledgement" + } else { + "Policy status report" + }; + let mut attempt = 1_u32; + loop { + let sandbox_id = sandbox_id.clone(); + let error = update.error.clone(); + let client = client.clone(); + match client + .report_policy_status(&sandbox_id, update.version, update.loaded, &error) + .await + { + Ok(()) => break, + Err(error) if is_retryable_error(&error) => { + let backoff = Duration::from_secs(1_u64 << attempt.saturating_sub(1).min(5)); + warn!( + %error, + attempt, + version = update.version, + loaded = update.loaded, + retry_in_secs = backoff.as_secs(), + "{operation} failed transiently; retaining ordered update" + ); + tokio::time::sleep(backoff).await; + attempt = attempt.saturating_add(1); + } + Err(error) => { + warn!( + %error, + version = update.version, + loaded = update.loaded, + "Discarding terminal policy status update" + ); + continue 'updates; + } + } + } + + if let Some(event) = update.success_event { + let (policy_hash, message) = match event { + PolicyStatusSuccessEvent::InitialAcknowledgement { policy_hash } => ( + policy_hash, + format!( + "Acknowledged initial policy revision as loaded [version:{}]", + update.version + ), + ), + PolicyStatusSuccessEvent::UnchangedAcknowledgement { policy_hash } => ( + policy_hash, + format!( + "Acknowledged unchanged policy revision as loaded [version:{}]", + update.version + ), + ), + }; + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped("version", serde_json::json!(update.version)) + .unmapped("policy_hash", serde_json::json!(policy_hash)) + .message(message) + .build() + ); + } + } +} + +fn enqueue_policy_status(sender: &UnboundedSender, update: PolicyStatusUpdate) { + let version = update.version; + if let Err(error) = sender.send(update) { + warn!( + %error, + version, + "Policy status reporter unavailable during shutdown" + ); + } +} + +/// Best-effort `FAILED` acknowledgement when initial policy construction or +/// conversion fails. +/// +/// Uses the revision identity captured with the policy that failed to build, +/// and preserves the original construction error as the reported message. A +/// delivery failure here is swallowed so it can never mask that error. +async fn report_initial_policy_failure( + endpoint: &str, + sandbox_id: &str, + revision: Option<&LoadedPolicyRevision>, + error: &miette::Report, +) { + let Some(revision) = revision.filter(|revision| { + revision.version > 0 + && revision.policy_source == openshell_core::proto::PolicySource::Sandbox + }) else { + return; + }; + let client = match openshell_core::grpc_client::CachedOpenShellClient::connect(endpoint).await { + Ok(client) => client, + Err(e) => { + warn!(error = %e, "Failed to connect to report initial policy failure"); + return; + } + }; + let message = error.to_string(); + if let Err(e) = grpc_retry("Initial policy failure report", || { + let client = client.clone(); + let message = message.clone(); + async move { + client + .report_policy_status(sandbox_id, revision.version, false, &message) + .await + } + }) + .await + { + warn!(error = %e, version = revision.version, "Failed to report initial policy failure"); + } +} + +/// Background loop that polls the server for policy updates. +/// +/// When a new version is detected, attempts to reload the OPA engine via +/// `reload_from_proto_with_pid()`. Reports load success/failure back to the +/// server. On failure, the previous engine is untouched (LKG behavior). +/// +/// When the entrypoint PID is available, policy reloads include symlink +/// resolution for binary paths via the container filesystem. +struct PolicyPollLoopContext { + endpoint: String, + sandbox_id: String, + opa_engine: Arc, + /// Source of the policy currently loaded into OPA. This distinguishes an + /// explicit local-file override from an unbound gateway revision so the + /// former is never replaced by policy polling. + loaded_policy_origin: LoadedPolicyOrigin, + entrypoint_pid: Arc, + interval_secs: u64, + ocsf_enabled: Arc, + provider_credentials: ProviderCredentialState, + policy_local_ctx: Option>, + agent_proposals: AgentProposals, + middleware_registry_status: MiddlewareRegistryStatus, + workspace_tx: tokio::sync::watch::Sender, + extension_credentials: openshell_extension_core::ExtensionCredentialStore, + extension_authentication_enabled: bool, + middleware_connector: MiddlewareConnector, + /// Immutable driver capability and startup substrate state. + transparent_tcp: TransparentTcpReloadState, +} + +type MiddlewareConnector = Arc< + dyn Fn( + Vec, + MiddlewareAuthentication, + ) -> Pin< + Box< + dyn std::future::Future< + Output = Result, + > + Send, + >, + > + Send + + Sync, +>; + +#[derive(Clone, Default)] +struct MiddlewareAuthentication { + credentials: std::collections::HashMap, + enabled: bool, +} + +fn default_middleware_connector() -> MiddlewareConnector { + Arc::new(|services, authentication| { + Box::pin(async move { connect_middleware_registry(&services, &authentication).await }) + }) +} + +async fn connect_middleware_registry( + services: &[openshell_core::proto::SupervisorMiddlewareService], + authentication: &MiddlewareAuthentication, +) -> Result { + if authentication.enabled { + openshell_supervisor_middleware::MiddlewareRegistry::connect_services_authenticated( + openshell_supervisor_middleware_builtins::services(), + services.to_vec(), + &authentication.credentials, + ) + .await + } else { + openshell_supervisor_middleware::MiddlewareRegistry::connect_services( + openshell_supervisor_middleware_builtins::services(), + services.to_vec(), + ) + .await + } +} + +async fn install_builtin_middleware_registry(opa_engine: &OpaEngine) -> Result<()> { + let registry = openshell_supervisor_middleware::MiddlewareRegistry::connect_services( + openshell_supervisor_middleware_builtins::services(), + Vec::new(), + ) + .await?; + opa_engine.replace_middleware_registry(registry) +} + +/// Wait the configured poll interval, but never past the point at which an +/// installed extension credential must be rotated. +fn next_poll_delay( + store: &openshell_extension_core::ExtensionCredentialStore, + interval: Duration, +) -> Duration { + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| { + i64::try_from(elapsed.as_millis()).unwrap_or(i64::MAX) + }); + store.next_refresh_delay(interval, now_ms) +} + +/// Drop credentials for services no longer in the installed registry. +/// +/// Call only after a registry swap succeeds, so a failed candidate cannot +/// invalidate the last-known-good clients. +fn retain_extension_credentials( + store: &openshell_extension_core::ExtensionCredentialStore, + installed: &[openshell_core::proto::SupervisorMiddlewareService], + extension_authentication_enabled: bool, +) { + let retained = if extension_authentication_enabled { + installed + .iter() + .map(|service| service.name.as_str()) + .collect() + } else { + std::collections::HashSet::default() + }; + store.retain(&retained); +} + +struct MiddlewareRegistryReconciliation<'a> { + desired_services: &'a [openshell_core::proto::SupervisorMiddlewareService], + authentication: MiddlewareAuthentication, + registry_changed: bool, + extension_credentials: &'a openshell_extension_core::ExtensionCredentialStore, + current_services: &'a mut Vec, + status: &'a mut MiddlewareRegistryStatus, +} + +async fn reconcile_middleware_registry( + opa_engine: &OpaEngine, + middleware_connector: &MiddlewareConnector, + reconciliation: MiddlewareRegistryReconciliation<'_>, +) { + if !reconciliation.registry_changed { + return; + } + + match middleware_connector( + reconciliation.desired_services.to_vec(), + reconciliation.authentication.clone(), + ) + .await + .and_then(|registry| opa_engine.replace_middleware_registry(registry)) + { + Ok(()) => { + retain_extension_credentials( + reconciliation.extension_credentials, + reconciliation.desired_services, + reconciliation.authentication.enabled, + ); + reconciliation.current_services.clear(); + reconciliation + .current_services + .extend_from_slice(reconciliation.desired_services); + *reconciliation.status = MiddlewareRegistryStatus::Synchronized; + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped( + "supervisor_middleware_service_count", + serde_json::json!(reconciliation.current_services.len()) + ) + .message(format!( + "Supervisor middleware registry reloaded [service_count:{}]", + reconciliation.current_services.len() + )) + .build() + ); + } + Err(error) => { + // Emit only on the transition into the failed state to avoid + // repeating the same finding on every poll during an outage. + if *reconciliation.status == MiddlewareRegistryStatus::Synchronized { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .state(StateId::Other, "failed") + .message(format!( + "Supervisor middleware registry reload failed, keeping last-known-good registry [error:{error}]" + )) + .build() + ); + } + *reconciliation.status = MiddlewareRegistryStatus::NeedsReconciliation; + } + } +} + +#[derive(Debug, PartialEq, Eq)] +struct PolicyValidationFailureDisposition { + configured_mode: PolicyValidationFailureMode, + mode: PolicyValidationFailureMode, + previous_policy_active: bool, + active_generation: u64, +} + +struct RejectedPolicyGeneration { + version: u32, + policy_hash: String, + validation_error: String, + configured_mode: PolicyValidationFailureMode, +} + +enum GatewayRuntimeFailureDisposition { + PolicyRejected { + error: String, + disposition: PolicyValidationFailureDisposition, + }, + MiddlewareUnavailable { + error: String, + }, + TransparentTcpExpansionRejected { + error: String, + active_generation: u64, + }, +} + +fn apply_gateway_runtime_reload_failure( + engine: &OpaEngine, + failure: GatewayRuntimeReloadError, + configured_mode: PolicyValidationFailureMode, + has_last_valid_policy: bool, + version: u32, +) -> Result { + match failure { + GatewayRuntimeReloadError::PolicyValidation(error) => { + let error = error.to_string(); + let disposition = apply_policy_validation_failure( + engine, + configured_mode, + has_last_valid_policy, + version, + &error, + )?; + Ok(GatewayRuntimeFailureDisposition::PolicyRejected { error, disposition }) + } + GatewayRuntimeReloadError::TransparentTcpPrerequisite(error) => Ok( + GatewayRuntimeFailureDisposition::TransparentTcpExpansionRejected { + error: error.to_string(), + active_generation: engine.current_generation(), + }, + ), + GatewayRuntimeReloadError::MiddlewareRegistry(error) => { + Ok(GatewayRuntimeFailureDisposition::MiddlewareUnavailable { + error: error.to_string(), + }) + } + } +} + +fn emit_transparent_tcp_expansion_rejection( + version: u32, + policy_hash: &str, + active_generation: u64, + error: &str, +) { + let message = format!( + "Transparent TCP policy expansion rejected; previous policy IS active [version:{version} active_generation:{active_generation} error:{error}]" + ); + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::High) + .status(StatusId::Failure) + .state(StateId::Enabled, "retained_previous_policy") + .unmapped("candidate_version", serde_json::json!(version)) + .unmapped("candidate_policy_hash", serde_json::json!(policy_hash)) + .unmapped("previous_policy_active", serde_json::json!(true)) + .unmapped("active_generation", serde_json::json!(active_generation)) + .unmapped("validation_error", serde_json::json!(error)) + .message(message) + .build() + ); +} + +fn apply_policy_validation_failure( + engine: &OpaEngine, + configured_mode: PolicyValidationFailureMode, + has_last_valid_policy: bool, + version: u32, + error: &str, +) -> Result { + let mode = if has_last_valid_policy { + configured_mode + } else { + PolicyValidationFailureMode::FailClosed + }; + match mode { + PolicyValidationFailureMode::FailClosed => { + let reason = format!( + "policy validation failed; fail-closed quarantine is active; candidate version {version} rejected: {error}" + ); + let active_generation = engine.enter_fail_closed(reason)?; + Ok(PolicyValidationFailureDisposition { + configured_mode, + mode, + previous_policy_active: false, + active_generation, + }) + } + PolicyValidationFailureMode::RetainLastValid => { + let active_generation = engine.exit_fail_closed()?; + Ok(PolicyValidationFailureDisposition { + configured_mode, + mode, + previous_policy_active: true, + active_generation, + }) + } + } +} + +fn policy_validation_failure_events( + disposition: &PolicyValidationFailureDisposition, + version: u32, + policy_hash: &str, + error: &str, +) -> [OcsfEvent; 2] { + let previous_policy_state = if disposition.previous_policy_active { + "IS active" + } else { + "IS NOT active" + }; + let state = if disposition.previous_policy_active { + (StateId::Enabled, "retained_last_valid") + } else { + (StateId::Disabled, "fail_closed") + }; + let message = format!( + "Policy validation failed; configured_mode={} effective_mode={}; previous policy {previous_policy_state} [version:{version} active_generation:{} error:{error}]", + disposition.configured_mode.as_str(), + disposition.mode.as_str(), + disposition.active_generation, + ); + let finding_uid = format!("policy-validation-failed-{version}"); + let version_string = version.to_string(); + let config = ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::High) + .status(StatusId::Failure) + .state(state.0, state.1) + .unmapped("candidate_version", serde_json::json!(version)) + .unmapped("candidate_policy_hash", serde_json::json!(policy_hash)) + .unmapped( + "validation_failure_mode", + serde_json::json!(disposition.mode.as_str()), + ) + .unmapped( + "configured_validation_failure_mode", + serde_json::json!(disposition.configured_mode.as_str()), + ) + .unmapped( + "previous_policy_active", + serde_json::json!(disposition.previous_policy_active), + ) + .unmapped( + "active_generation", + serde_json::json!(disposition.active_generation), + ) + .unmapped("validation_error", serde_json::json!(error)) + .message(message.clone()) + .build(); + let finding = DetectionFindingBuilder::new(ocsf_ctx()) + .activity(ActivityId::Open) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::High) + .is_alert(true) + .finding_info( + FindingInfo::new(&finding_uid, "Invalid policy generation rejected").with_desc(error), + ) + .evidence_pairs(&[ + ("candidate_version", &version_string), + ("candidate_policy_hash", policy_hash), + ("validation_failure_mode", disposition.mode.as_str()), + ( + "configured_validation_failure_mode", + disposition.configured_mode.as_str(), + ), + ( + "previous_policy_active", + if disposition.previous_policy_active { + "true" + } else { + "false" + }, + ), + ]) + .remediation("Submit a valid, unambiguous policy generation") + .message(message) + .build(); + [config, finding] +} + +fn emit_policy_validation_failure( + disposition: &PolicyValidationFailureDisposition, + version: u32, + policy_hash: &str, + error: &str, +) { + for event in policy_validation_failure_events(disposition, version, policy_hash, error) { + ocsf_emit!(event); + } +} + +async fn run_policy_poll_loop(ctx: PolicyPollLoopContext) -> Result<()> { + let client = openshell_core::grpc_client::CachedOpenShellClient::connect_with_credentials( + &ctx.endpoint, + ctx.extension_credentials.clone(), + ) + .await?; + run_policy_poll_loop_with_client(ctx, client).await +} + +async fn run_policy_poll_loop_with_client( + ctx: PolicyPollLoopContext, + client: C, +) -> Result<()> { + use openshell_core::proto::PolicySource; + use std::sync::atomic::Ordering; + + let (status_sender, status_receiver) = tokio::sync::mpsc::unbounded_channel(); + tokio::spawn(run_policy_status_reporter( + client.clone(), + ctx.sandbox_id.clone(), + status_receiver, + )); + + let mut current_config_revision: u64 = 0; + let mut current_provider_env_revision: u64 = ctx.provider_credentials.snapshot().revision; + let mut current_policy_version: u32 = 0; + let mut current_policy_hash = String::new(); + let mut current_middleware_services = Vec::new(); + let mut current_extension_authentication_enabled = ctx.extension_authentication_enabled; + let mut middleware_registry_status = ctx.middleware_registry_status; + let mut current_settings: std::collections::HashMap< + String, + openshell_core::proto::EffectiveSetting, + > = std::collections::HashMap::new(); + let reloads_gateway_policy = ctx.loaded_policy_origin.allows_gateway_policy_reload(); + let mut last_failed_runtime_revision: Option = None; + let mut rejected_policy_generation: Option = None; + let mut has_last_valid_policy = ctx.loaded_policy_origin.has_last_valid_policy(); + + // A first poll that does not match the policy already loaded into OPA must + // pass through the normal reconciliation path immediately. It must never + // seed the applied-state trackers before OPA actually loads it. + let mut pending_result = None; + + // Initialize revision from the first poll and acknowledge the initial + // policy revision the supervisor actually loaded. A mismatched result is + // reconciled below instead of being recorded as already applied. + match client.poll_settings(&ctx.sandbox_id).await { + Ok(result) => { + let _ = ctx.workspace_tx.send(client.workspace()); + match initial_poll_disposition(&ctx.loaded_policy_origin, &result) { + InitialPollDisposition::Acknowledge(candidate) => { + apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + apply_agent_proposals_enabled( + &ctx.agent_proposals, + agent_proposals_enabled_from_settings(&result.settings), + "initial settings poll", + Some(candidate.config_revision), + skills::install_static_skills, + ); + current_config_revision = candidate.config_revision; + current_policy_version = candidate.version; + current_policy_hash.clone_from(&candidate.policy_hash); + current_middleware_services = result.supervisor_middleware_services; + current_extension_authentication_enabled = + result.extension_authentication_enabled; + current_settings = result.settings; + enqueue_policy_status( + &status_sender, + PolicyStatusUpdate::initial_loaded(&candidate), + ); + debug!( + config_revision = current_config_revision, + "Settings poll: initial policy matches loaded revision" + ); + } + InitialPollDisposition::Reconcile => pending_result = Some(result), + InitialPollDisposition::TrackOnly => { + apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + apply_agent_proposals_enabled( + &ctx.agent_proposals, + agent_proposals_enabled_from_settings(&result.settings), + "initial settings poll", + Some(result.config_revision), + skills::install_static_skills, + ); + current_config_revision = result.config_revision; + current_policy_hash = result.policy_hash.clone(); + current_middleware_services = result.supervisor_middleware_services; + current_extension_authentication_enabled = + result.extension_authentication_enabled; + current_settings = result.settings; + debug!( + config_revision = current_config_revision, + "Settings poll: tracking gateway config while preserving local policy override" + ); + } + } + } + Err(e) => { + warn!(error = %e, "Settings poll: failed to fetch initial version, will retry"); + } + } + + let interval = Duration::from_secs(ctx.interval_secs); + loop { + let result = if let Some(result) = pending_result.take() { + result + } else { + tokio::time::sleep(next_poll_delay(&ctx.extension_credentials, interval)).await; + match client.poll_settings(&ctx.sandbox_id).await { + Ok(result) => { + let _ = ctx.workspace_tx.send(client.workspace()); + result + } + Err(e) => { + debug!(error = %e, "Settings poll: server unreachable, will retry"); + if current_extension_authentication_enabled + && let Err(refresh_error) = + client.refresh_installed_extension_credentials().await + { + warn!( + error = %refresh_error, + "Settings poll: extension credential refresh failed while configuration was unavailable" + ); + } + continue; + } + } + }; + + // Reuse installed per-service credentials, rotating only when one is + // missing or due. Rotation happens on the existing gateway channel and + // updates slots in place, so it is independent of config revision and + // registry equality. + let middleware_credentials = if result.extension_authentication_enabled { + match client + .extension_credentials_for(&result.supervisor_middleware_services) + .await + { + Ok(credentials) => credentials, + Err(error) => { + warn!(error = %error, "Settings poll: extension credential refresh failed"); + std::collections::HashMap::new() + } + } + } else { + std::collections::HashMap::new() + }; + + let config_changed = result.config_revision != current_config_revision; + let provider_env_changed = result.provider_env_revision != current_provider_env_revision; + let policy_changed = result.policy_hash != current_policy_hash; + let extension_authentication_changed = + current_extension_authentication_enabled != result.extension_authentication_enabled; + let middleware_registry_changed = extension_authentication_changed + || middleware_registry_needs_rebuild( + middleware_registry_status, + ¤t_middleware_services, + &result.supervisor_middleware_services, + ); + // A valid candidate may intentionally restore byte-for-byte policy + // content that was active before a rejected update. Its hash then + // equals `current_policy_hash`, but the runtime is still quarantined + // and must reload (or it would remain deny-all indefinitely). + let recovering_rejected_policy = reloads_gateway_policy + && rejected_policy_generation + .as_ref() + .is_some_and(|rejected| rejected.policy_hash != result.policy_hash); + let policy_runtime_changed = recovering_rejected_policy + || extension_authentication_changed + || gateway_policy_runtime_needs_reconciliation( + reloads_gateway_policy, + ¤t_policy_hash, + &result.policy_hash, + ¤t_middleware_services, + &result.supervisor_middleware_services, + middleware_registry_status, + ); + // Recovery already has its own acknowledgement path below. Giving it + // precedence here prevents a restored last-known-good policy from + // also being acknowledged as an ordinary same-hash revision. + let unchanged_policy_revision = unchanged_policy_revision_candidate( + reloads_gateway_policy, + recovering_rejected_policy, + current_policy_version, + ¤t_policy_hash, + &result, + ); + let mut policy_runtime_reconciled = false; + + // A local policy override is not coupled to the gateway policy + // snapshot, so its service registry can still be reconciled alone. + // Gateway policy snapshots, however, must install policy and registry + // as one generation below. + if !reloads_gateway_policy { + reconcile_middleware_registry( + &ctx.opa_engine, + &ctx.middleware_connector, + MiddlewareRegistryReconciliation { + desired_services: &result.supervisor_middleware_services, + authentication: MiddlewareAuthentication { + credentials: middleware_credentials.clone(), + enabled: result.extension_authentication_enabled, + }, + registry_changed: middleware_registry_changed, + extension_credentials: &ctx.extension_credentials, + current_services: &mut current_middleware_services, + status: &mut middleware_registry_status, + }, + ) + .await; + if middleware_registry_status == MiddlewareRegistryStatus::Synchronized { + current_extension_authentication_enabled = result.extension_authentication_enabled; + } + } + + if !config_changed + && !provider_env_changed + && !policy_runtime_changed + && unchanged_policy_revision.is_none() + { + continue; + } + + if config_changed || provider_env_changed { + // Log which settings changed. + log_setting_changes(¤t_settings, &result.settings); + + // A posture change after a rejected update takes effect immediately. + // The compiled last-known-good engine remains available beneath a + // fail-closed quarantine, so an explicit retain_last_valid selection + // can reactivate it without accepting any part of the invalid policy. + if !policy_changed && let Some(rejected) = rejected_policy_generation.as_mut() { + let mode = result.policy_validation_failure_mode; + if mode != rejected.configured_mode { + let disposition = apply_policy_validation_failure( + &ctx.opa_engine, + mode, + has_last_valid_policy, + rejected.version, + &rejected.validation_error, + )?; + emit_policy_validation_failure( + &disposition, + rejected.version, + &rejected.policy_hash, + &rejected.validation_error, + ); + rejected.configured_mode = mode; + } + } + + ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Other, "detected") + .unmapped("old_config_revision", serde_json::json!(current_config_revision)) + .unmapped("new_config_revision", serde_json::json!(result.config_revision)) + .unmapped("policy_changed", serde_json::json!(policy_changed)) + .unmapped("provider_env_changed", serde_json::json!(provider_env_changed)) + .message(format!( + "Settings poll: config change detected [old_revision:{current_config_revision} new_revision:{} policy_changed:{policy_changed} provider_env_changed:{provider_env_changed}]", + result.config_revision + )) + .build()); + } + + if provider_env_changed { + match openshell_core::grpc_client::fetch_provider_environment( + &ctx.endpoint, + &ctx.sandbox_id, + ) + .await + { + Ok(env_result) => { + let provider_env_revision = env_result.provider_env_revision; + let install_result = ctx.provider_credentials.install_bound_environment( + provider_env_revision, + env_result.environment, + env_result.credential_expires_at_ms, + env_result.dynamic_credentials, + env_result.static_credential_bindings, + env_result.non_secret_environment_keys, + ); + if let Err(error) = install_result { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::High) + .status(StatusId::Failure) + .state(StateId::Disabled, "fail_closed") + .message(format!( + "Rejected provider environment refresh; static provider credentials were revoked; fetched dynamic token grants remain active: {error}" + )) + .build() + ); + } else { + let env_count = + ctx.provider_credentials.child_env_with_gcp_resolved().len(); + current_provider_env_revision = provider_env_revision; + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped( + "provider_env_revision", + serde_json::json!(provider_env_revision) + ) + .message(format!( + "Provider environment refreshed [revision:{provider_env_revision} env_count:{env_count}]" + )) + .build() + ); + } + } + Err(e) => { + ctx.provider_credentials + .revoke_static_provider_environment(result.provider_env_revision); + warn!( + error = %e, + provider_env_revision = result.provider_env_revision, + "Settings poll: failed to refresh provider environment; static provider credentials were revoked; previous dynamic token grants remain active" + ); + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::High) + .status(StatusId::Failure) + .state(StateId::Disabled, "fail_closed") + .message( + "Provider environment refresh failed; static provider credentials were revoked; previous dynamic token grants remain active" + ) + .build() + ); + } + } + } + + if policy_runtime_changed { + let pid = ctx.entrypoint_pid.load(Ordering::Acquire); + let runtime_result = reload_gateway_policy_runtime( + &ctx.opa_engine, + result.policy.as_ref(), + pid, + MiddlewareReloadContext { + desired_services: &result.supervisor_middleware_services, + authentication: &MiddlewareAuthentication { + credentials: middleware_credentials.clone(), + enabled: result.extension_authentication_enabled, + }, + registry_changed: middleware_registry_changed, + connector: &ctx.middleware_connector, + }, + ctx.transparent_tcp, + ) + .await; + + match runtime_result { + Ok(()) => { + policy_runtime_reconciled = true; + let policy = result + .policy + .as_ref() + .expect("successful runtime reload requires a policy payload"); + has_last_valid_policy = true; + rejected_policy_generation = None; + if policy_changed { + if let Some(policy_local_ctx) = ctx.policy_local_ctx.as_ref() { + policy_local_ctx.set_current_policy(policy.clone()).await; + } + if result.global_policy_version > 0 { + ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped("policy_hash", serde_json::json!(&result.policy_hash)) + .unmapped("global_version", serde_json::json!(result.global_policy_version)) + .message(format!( + "Policy reloaded successfully (global) [policy_hash:{} global_version:{}]", + result.policy_hash, + result.global_policy_version + )) + .build()); + } else { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped("policy_hash", serde_json::json!(&result.policy_hash)) + .message(format!( + "Policy reloaded successfully [policy_hash:{}]", + result.policy_hash + )) + .build() + ); + } + if result.version > 0 && result.policy_source == PolicySource::Sandbox { + enqueue_policy_status( + &status_sender, + PolicyStatusUpdate::loaded(result.version), + ); + current_policy_version = result.version; + } + } else if recovering_rejected_policy + && result.version > 0 + && result.policy_source == PolicySource::Sandbox + { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped("policy_hash", serde_json::json!(&result.policy_hash)) + .message(format!( + "Policy reloaded successfully and fail-closed quarantine cleared [policy_hash:{}]", + result.policy_hash + )) + .build() + ); + enqueue_policy_status( + &status_sender, + PolicyStatusUpdate::loaded(result.version), + ); + current_policy_version = result.version; + } + + if middleware_registry_changed { + ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .unmapped( + "supervisor_middleware_service_count", + serde_json::json!(result.supervisor_middleware_services.len()) + ) + .message(format!( + "Supervisor policy runtime reloaded atomically [service_count:{}]", + result.supervisor_middleware_services.len() + )) + .build()); + } + + current_policy_hash.clone_from(&result.policy_hash); + current_middleware_services.clone_from(&result.supervisor_middleware_services); + current_extension_authentication_enabled = + result.extension_authentication_enabled; + retain_extension_credentials( + &ctx.extension_credentials, + &result.supervisor_middleware_services, + result.extension_authentication_enabled, + ); + middleware_registry_status = MiddlewareRegistryStatus::Synchronized; + last_failed_runtime_revision = None; + } + Err(failure) => { + let failed_revision = FailedRuntimeRevision::new( + result.config_revision, + &result.policy_hash, + &failure, + ); + if last_failed_runtime_revision.as_ref() != Some(&failed_revision) { + let failure_mode = result.policy_validation_failure_mode; + match apply_gateway_runtime_reload_failure( + &ctx.opa_engine, + failure, + failure_mode, + has_last_valid_policy, + result.version, + )? { + GatewayRuntimeFailureDisposition::PolicyRejected { + error, + disposition, + } => { + emit_policy_validation_failure( + &disposition, + result.version, + &result.policy_hash, + &error, + ); + rejected_policy_generation = Some(RejectedPolicyGeneration { + version: result.version, + policy_hash: result.policy_hash.clone(), + validation_error: error.clone(), + configured_mode: failure_mode, + }); + if policy_changed + && result.version > 0 + && result.policy_source == PolicySource::Sandbox + { + enqueue_policy_status( + &status_sender, + PolicyStatusUpdate::failed(result.version, error), + ); + } + } + GatewayRuntimeFailureDisposition::MiddlewareUnavailable { error } => { + ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .state(StateId::Other, "failed") + .unmapped("version", serde_json::json!(result.version)) + .unmapped("error", serde_json::json!(&error)) + .unmapped("previous_policy_active", serde_json::json!(true)) + .message(format!( + "Supervisor middleware registry unavailable, keeping last-known-good policy runtime active [version:{} error:{error}]", + result.version + )) + .build()); + } + GatewayRuntimeFailureDisposition::TransparentTcpExpansionRejected { + error, + active_generation, + } => { + emit_transparent_tcp_expansion_rejection( + result.version, + &result.policy_hash, + active_generation, + &error, + ); + if policy_changed + && result.version > 0 + && result.policy_source == PolicySource::Sandbox + { + enqueue_policy_status( + &status_sender, + PolicyStatusUpdate::failed(result.version, error), + ); + } + } + } + } + last_failed_runtime_revision = Some(failed_revision); + // Nothing was installed, so the registry status still + // describes the live registry. The retry is driven by the + // persisting hash/service-set mismatch (or an existing + // NeedsReconciliation), not by degrading the status here. + } + } + } + + if let Some(version) = unchanged_policy_revision_ready_to_ack( + unchanged_policy_revision, + policy_runtime_changed, + policy_runtime_reconciled, + ) { + enqueue_policy_status( + &status_sender, + PolicyStatusUpdate::unchanged_loaded(version, result.policy_hash.clone()), + ); + current_policy_version = version; + } + + // Apply OCSF JSON toggle from the `ocsf_json_enabled` setting. + apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + + // Apply the agent-proposals feature toggle. On a false→true transition + // we lazily install the skill so a sandbox that started with the flag + // off picks up the surface without a recreate. We never uninstall on + // a true→false transition: stale skill content on disk is harmless + // because route_request and agent_next_steps both gate on the live + // shared flag, so the agent that reads the skill will see 404s and an + // empty `next_steps` array regardless. + apply_agent_proposals_enabled( + &ctx.agent_proposals, + agent_proposals_enabled_from_settings(&result.settings), + "settings poll", + Some(result.config_revision), + skills::install_static_skills, + ); + + current_config_revision = result.config_revision; + if !reloads_gateway_policy { + current_policy_hash = result.policy_hash; + } + current_settings = result.settings; + } +} + +fn apply_ocsf_json_setting( + enabled: &AtomicBool, + settings: &std::collections::HashMap, +) { + use std::sync::atomic::Ordering; + + let new_ocsf = extract_bool_setting(settings, "ocsf_json_enabled").unwrap_or(false); + let prev_ocsf = enabled.swap(new_ocsf, Ordering::Relaxed); + if new_ocsf != prev_ocsf { + info!(ocsf_json_enabled = new_ocsf, "OCSF JSONL logging toggled"); + } +} + +/// Extract a bool value from an effective setting, if present. +fn extract_bool_setting( + settings: &std::collections::HashMap, + key: &str, +) -> Option { + use openshell_core::proto::setting_value; + settings + .get(key) + .and_then(|es| es.value.as_ref()) + .and_then(|sv| sv.value.as_ref()) + .and_then(|v| match v { + setting_value::Value::BoolValue(b) => Some(*b), + _ => None, + }) +} + +fn agent_proposals_enabled_from_settings( + settings: &std::collections::HashMap, +) -> bool { + extract_bool_setting( + settings, + openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY, + ) + .unwrap_or(false) +} + +fn apply_agent_proposals_enabled( + agent_proposals: &AgentProposals, + enabled: bool, + source: &'static str, + config_revision: Option, + install_static_skills: impl FnOnce() -> Result, +) { + let previously_enabled = agent_proposals.swap_enabled(enabled); + if enabled == previously_enabled { + return; + } + + info!( + agent_policy_proposals_enabled = enabled, + source, config_revision, "agent-driven policy proposals toggled" + ); + + if enabled && !previously_enabled { + match install_static_skills() { + Ok(installed) => info!( + path = %installed.policy_advisor.display(), + "Installed sandbox agent skill on toggle-on" + ), + Err(error) => warn!( + error = %error, + "Failed to install sandbox agent skill on toggle-on" + ), + } + } +} + +/// Log individual setting changes between two snapshots. +fn log_setting_changes( + old: &std::collections::HashMap, + new: &std::collections::HashMap, +) { + for (key, new_es) in new { + let new_val = format_setting_value(new_es); + match old.get(key) { + Some(old_es) => { + let old_val = format_setting_value(old_es); + if old_val != new_val { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "updated") + .unmapped("key", serde_json::json!(key)) + .unmapped("old", serde_json::json!(old_val.clone())) + .unmapped("new", serde_json::json!(new_val.clone())) + .message(format!( + "Setting changed [key:{key} old:{old_val} new:{new_val}]" + )) + .build() + ); + } + } + None => { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "enabled") + .unmapped("key", serde_json::json!(key)) + .unmapped("value", serde_json::json!(new_val.clone())) + .message(format!("Setting added [key:{key} value:{new_val}]")) + .build() + ); + } + } + } + for key in old.keys() { + if !new.contains_key(key) { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Disabled, "disabled") + .unmapped("key", serde_json::json!(key)) + .message(format!("Setting removed [key:{key}]")) + .build() + ); + } + } +} + +/// Format an `EffectiveSetting` value for log display. +fn format_setting_value(es: &openshell_core::proto::EffectiveSetting) -> String { + use openshell_core::proto::setting_value; + match es.value.as_ref().and_then(|sv| sv.value.as_ref()) { + None => "".to_string(), + Some(setting_value::Value::StringValue(v)) => v.clone(), + Some(setting_value::Value::BoolValue(v)) => v.to_string(), + Some(setting_value::Value::IntValue(v)) => v.to_string(), + Some(setting_value::Value::BytesValue(_)) => "".to_string(), + } +} + +#[cfg(test)] +#[allow( + clippy::needless_raw_string_hashes, + clippy::iter_on_single_items, + clippy::similar_names, + clippy::manual_string_new, + clippy::doc_markdown, + reason = "Test code: test fixtures often use idiomatic forms not flagged in production." +)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicUsize, Ordering}; + + fn effective_bool(value: bool) -> openshell_core::proto::EffectiveSetting { + openshell_core::proto::EffectiveSetting { + value: Some(openshell_core::proto::SettingValue { + value: Some(openshell_core::proto::setting_value::Value::BoolValue( + value, + )), + }), + scope: openshell_core::proto::SettingScope::Global.into(), + } + } + + #[test] + fn shared_ssh_socket_setting_is_explicit() { + assert!(shared_ssh_socket_value("1")); + assert!(shared_ssh_socket_value("true")); + assert!(shared_ssh_socket_value("TRUE")); + assert!(!shared_ssh_socket_value("0")); + assert!(!shared_ssh_socket_value("yes")); + } + + #[tokio::test] + async fn control_readiness_exists_only_while_guard_is_live() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("health.sock"); + let readiness = + ControlReadiness::start(path.clone(), None).expect("start readiness listener"); + check_control_readiness(&path).expect("running supervisor accepts readiness probes"); + + drop(readiness); + tokio::task::yield_now().await; + assert!(check_control_readiness(&path).is_err()); + } + + #[tokio::test] + async fn control_readiness_tracks_supervisor_session() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().join("health.sock"); + let (session_tx, session_rx) = tokio::sync::watch::channel(true); + let _readiness = ControlReadiness::start(path.clone(), Some(session_rx)) + .expect("start readiness listener"); + check_control_readiness(&path).expect("accepted session is ready"); + + session_tx.send_replace(false); + timeout(Duration::from_secs(1), async { + while check_control_readiness(&path).is_ok() { + tokio::task::yield_now().await; + } + }) + .await + .expect("lost session removes readiness socket"); + + session_tx.send_replace(true); + timeout(Duration::from_secs(1), async { + while check_control_readiness(&path).is_err() { + tokio::task::yield_now().await; + } + }) + .await + .expect("replacement session restores readiness socket"); + } + + #[test] + fn control_readiness_rejects_relative_path() { + let error = prepare_control_readiness_path(std::path::Path::new("health.sock")) + .expect_err("relative readiness path must be rejected"); + assert!(error.to_string().contains("must be absolute")); + } + + #[test] + fn main_exit_marker_atomically_replaces_previous_value() { + let directory = tempfile::tempdir().unwrap(); + let marker = directory.path().join("main-exited"); + std::fs::write(&marker, b"stale\n").unwrap(); + + persist_main_exit_marker(&marker, 23).unwrap(); + + assert_eq!(std::fs::read_to_string(&marker).unwrap(), "exit_code=23\n"); + assert!( + !directory + .path() + .join(format!(".main-exited.tmp-{}", std::process::id())) + .exists() + ); + } + + #[tokio::test] + async fn remote_access_plane_outlives_main_completion_until_teardown() { + let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel(); + let retained = retain_remote_access_plane(std::future::pending(), async { + let _ = shutdown_rx.await; + }); + tokio::pin!(retained); + + assert!( + timeout(Duration::from_millis(10), &mut retained) + .await + .is_err(), + "access plane must remain live after canonical process completion" + ); + shutdown_tx.send(()).expect("request teardown"); + timeout(Duration::from_secs(1), &mut retained) + .await + .expect("teardown should release retained access plane") + .expect("clean teardown"); + } + + #[tokio::test] + async fn completion_retry_phase_is_cancelled_by_shutdown() { + let mut shutdown = Box::pin(std::future::ready(())); + assert!( + completion_phase_or_shutdown(std::future::pending(), shutdown.as_mut()).await, + "shutdown must cancel an indefinitely retrying completion phase" + ); + } + + #[test] + fn apply_agent_proposals_enabled_installs_only_on_false_to_true() { + let agent_proposals = AgentProposals::default(); + let installs = AtomicUsize::new(0); + + apply_agent_proposals_enabled(&agent_proposals, true, "test", Some(1), || { + installs.fetch_add(1, Ordering::Relaxed); + Ok(skills::InstalledSkills { + policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), + policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), + agents: None, + }) + }); + assert!(agent_proposals.enabled()); + assert_eq!(installs.load(Ordering::Relaxed), 1); + + apply_agent_proposals_enabled(&agent_proposals, true, "test", Some(2), || { + installs.fetch_add(1, Ordering::Relaxed); + Ok(skills::InstalledSkills { + policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), + policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), + agents: None, + }) + }); + assert_eq!(installs.load(Ordering::Relaxed), 1); + + apply_agent_proposals_enabled(&agent_proposals, false, "test", Some(3), || { + installs.fetch_add(1, Ordering::Relaxed); + Ok(skills::InstalledSkills { + policy_advisor: std::path::PathBuf::from("/tmp/policy_advisor.md"), + policy_advisor_skill: std::path::PathBuf::from("/tmp/SKILL.md"), + agents: None, + }) + }); + assert!(!agent_proposals.enabled()); + assert_eq!(installs.load(Ordering::Relaxed), 1); + } + + #[test] + fn apply_ocsf_json_setting_enables_from_initial_settings_snapshot() { + let enabled = AtomicBool::new(false); + let mut settings = std::collections::HashMap::new(); + settings.insert("ocsf_json_enabled".to_string(), effective_bool(true)); + + apply_ocsf_json_setting(&enabled, &settings); + + assert!(enabled.load(Ordering::Relaxed)); + } + + #[test] + fn apply_ocsf_json_setting_disables_when_setting_is_unset() { + let enabled = AtomicBool::new(true); + let settings = std::collections::HashMap::new(); + + apply_ocsf_json_setting(&enabled, &settings); + + assert!(!enabled.load(Ordering::Relaxed)); + } + + #[test] + fn agent_proposals_setting_enables_from_initial_settings_snapshot() { + let mut settings = std::collections::HashMap::new(); + settings.insert( + openshell_core::settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY.to_string(), + effective_bool(true), + ); + + assert!(agent_proposals_enabled_from_settings(&settings)); + } + + #[test] + fn agent_proposals_setting_defaults_false_when_unset() { + let settings = std::collections::HashMap::new(); + + assert!(!agent_proposals_enabled_from_settings(&settings)); + } + + // ---- Policy disk discovery tests ---- + + #[test] + fn discover_policy_from_nonexistent_path_returns_restrictive_default() { + let path = std::path::Path::new("/nonexistent/policy.yaml"); + let policy = discover_policy_from_path(path); + // Restrictive default has no network policies. + assert!(policy.network_policies.is_empty()); + // It keeps filesystem restrictions while leaving identity to the + // active compute driver. + assert!(policy.filesystem.is_some()); + assert!(policy.process.is_none()); + } + + #[test] + fn discover_policy_from_valid_yaml_file() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("policy.yaml"); + std::fs::write( + &path, + r#" +version: 1 +filesystem_policy: + include_workdir: false + read_only: + - /usr + read_write: + - /tmp +network_policies: + test: + name: test + endpoints: + - { host: example.com, port: 443 } + binaries: + - { path: /usr/bin/curl } +"#, + ) + .unwrap(); + + let policy = discover_policy_from_path(&path); + assert_eq!(policy.network_policies.len(), 1); + assert!(policy.network_policies.contains_key("test")); + let fs = policy.filesystem.unwrap(); + assert!(!fs.include_workdir); + } + + #[test] + fn discover_policy_from_invalid_yaml_returns_restrictive_default() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("policy.yaml"); + std::fs::write(&path, "this is not valid yaml: [[[").unwrap(); + + let policy = discover_policy_from_path(&path); + // Falls back to restrictive default. + assert!(policy.network_policies.is_empty()); + assert!(policy.filesystem.is_some()); + } + + #[test] + fn discover_policy_from_unsafe_yaml_falls_back_to_default() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("policy.yaml"); + std::fs::write( + &path, + r#" +version: 1 +process: + run_as_user: root + run_as_group: root +filesystem_policy: + include_workdir: true + read_only: + - /usr + read_write: + - /tmp +"#, + ) + .unwrap(); + + let policy = discover_policy_from_path(&path); + // Falls back to restrictive default because of root user. + assert!(policy.process.is_none()); + } + + #[test] + fn discover_policy_restrictive_default_blocks_network() { + // In cluster mode we keep proxy mode enabled so `inference.local` + // can always be routed through proxy/OPA controls. + let proto = openshell_policy::restrictive_default_policy(); + let local_policy = SandboxPolicy::try_from(proto).expect("conversion should succeed"); + assert!(matches!(local_policy.network.mode, NetworkMode::Proxy)); + } + + // ---- Initial policy acknowledgement tests ---- + + fn proto_policy_fixture() -> openshell_core::proto::SandboxPolicy { + openshell_policy::restrictive_default_policy() + } + + fn proto_tcp_policy_fixture() -> openshell_core::proto::SandboxPolicy { + openshell_policy::parse_sandbox_policy( + r#" +version: 1 +network_policies: + redis: + name: redis + endpoints: + - host: redis.example.com + port: 6379 + protocol: tcp + binaries: + - path: /usr/bin/redis-cli +"#, + ) + .expect("parse TCP policy") + } + + fn settings_poll_result( + policy: Option, + version: u32, + source: openshell_core::proto::PolicySource, + ) -> openshell_core::grpc_client::SettingsPollResult { + openshell_core::grpc_client::SettingsPollResult { + policy, + version, + policy_hash: format!("hash-v{version}"), + config_revision: u64::from(version) * 100, + policy_source: source, + settings: std::collections::HashMap::new(), + global_policy_version: 0, + provider_env_revision: 0, + supervisor_middleware_services: Vec::new(), + workspace: String::new(), + policy_validation_failure_mode: PolicyValidationFailureMode::default(), + extension_authentication_enabled: false, + } + } + + #[derive(Clone)] + struct ScriptedPolicyGateway { + polls: Arc< + tokio::sync::Mutex< + tokio::sync::mpsc::UnboundedReceiver< + openshell_core::grpc_client::SettingsPollResult, + >, + >, + >, + reports: UnboundedSender<(u32, bool, String)>, + } + + #[tonic::async_trait] + impl PolicyGatewayClient for ScriptedPolicyGateway { + async fn poll_settings( + &self, + _sandbox_id: &str, + ) -> Result { + self.polls + .lock() + .await + .recv() + .await + .ok_or_else(|| miette::miette!("scripted policy poll channel closed")) + } + + async fn report_policy_status( + &self, + _sandbox_id: &str, + version: u32, + loaded: bool, + error: &str, + ) -> Result<()> { + self.reports + .send((version, loaded, error.to_string())) + .map_err(|_| miette::miette!("scripted policy report channel closed")) + } + + fn workspace(&self) -> String { + "test-workspace".to_string() + } + } + + #[derive(Clone)] + struct CredentialRejectingPolicyGateway { + inner: ScriptedPolicyGateway, + credential_requests: Arc, + } + + #[tonic::async_trait] + impl PolicyGatewayClient for CredentialRejectingPolicyGateway { + async fn poll_settings( + &self, + sandbox_id: &str, + ) -> Result { + self.inner.poll_settings(sandbox_id).await + } + + async fn report_policy_status( + &self, + sandbox_id: &str, + version: u32, + loaded: bool, + error: &str, + ) -> Result<()> { + self.inner + .report_policy_status(sandbox_id, version, loaded, error) + .await + } + + async fn extension_credentials_for( + &self, + _services: &[openshell_core::proto::SupervisorMiddlewareService], + ) -> Result> + { + self.credential_requests.fetch_add(1, Ordering::SeqCst); + Err(miette::miette!( + "gateway extension authentication is unavailable" + )) + } + + fn workspace(&self) -> String { + self.inner.workspace() + } + } + + fn scripted_policy_gateway() -> ( + ScriptedPolicyGateway, + UnboundedSender, + tokio::sync::mpsc::UnboundedReceiver<(u32, bool, String)>, + ) { + let (poll_tx, poll_rx) = tokio::sync::mpsc::unbounded_channel(); + let (report_tx, report_rx) = tokio::sync::mpsc::unbounded_channel(); + ( + ScriptedPolicyGateway { + polls: Arc::new(tokio::sync::Mutex::new(poll_rx)), + reports: report_tx, + }, + poll_tx, + report_rx, + ) + } + + fn policy_poll_test_context( + opa_engine: Arc, + loaded_policy_origin: LoadedPolicyOrigin, + middleware_connector: MiddlewareConnector, + ) -> PolicyPollLoopContext { + let (workspace_tx, _workspace_rx) = tokio::sync::watch::channel(String::new()); + PolicyPollLoopContext { + endpoint: String::new(), + sandbox_id: "sandbox-test".to_string(), + opa_engine, + loaded_policy_origin, + entrypoint_pid: Arc::new(AtomicU32::new(0)), + interval_secs: 0, + ocsf_enabled: Arc::new(AtomicBool::new(false)), + provider_credentials: ProviderCredentialState::from_child_env_snapshot( + 0, + std::collections::HashMap::new(), + ), + policy_local_ctx: None, + agent_proposals: AgentProposals::default(), + middleware_registry_status: MiddlewareRegistryStatus::Synchronized, + workspace_tx, + extension_credentials: openshell_extension_core::ExtensionCredentialStore::new(), + extension_authentication_enabled: false, + middleware_connector, + transparent_tcp: TransparentTcpReloadState::default(), + } + } + + async fn expect_policy_report( + reports: &mut tokio::sync::mpsc::UnboundedReceiver<(u32, bool, String)>, + version: u32, + ) { + let report = timeout(Duration::from_secs(1), reports.recv()) + .await + .expect("policy report timed out") + .expect("policy reporter stopped"); + assert_eq!(report, (version, true, String::new())); + } + + async fn expect_no_policy_report( + reports: &mut tokio::sync::mpsc::UnboundedReceiver<(u32, bool, String)>, + ) { + assert!( + timeout(Duration::from_millis(50), reports.recv()) + .await + .is_err(), + "unexpected policy status report" + ); + } + + #[tokio::test] + async fn same_hash_poll_revision_is_acknowledged_once_without_opa_reload() { + let mut v1 = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + v1.policy_hash = "same-policy".to_string(); + let mut v2 = v1.clone(); + v2.version = 2; + v2.config_revision = 200; + + let engine = + Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); + let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); + let ctx = policy_poll_test_context( + engine.clone(), + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_revision), + has_last_valid_policy: true, + }, + default_middleware_connector(), + ); + let (client, polls, mut reports) = scripted_policy_gateway(); + polls.send(v1).unwrap(); + + let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); + expect_policy_report(&mut reports, 1).await; + + polls.send(v2.clone()).unwrap(); + expect_policy_report(&mut reports, 2).await; + polls.send(v2).unwrap(); + expect_no_policy_report(&mut reports).await; + + assert_eq!( + engine.current_generation(), + 0, + "same-hash acknowledgement must not reload OPA" + ); + handle.abort(); + } + + #[tokio::test] + async fn poll_rejects_first_tcp_expansion_and_reports_previous_policy_active() { + let v1 = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + let v2 = settings_poll_result( + Some(proto_tcp_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Sandbox, + ); + let engine = + Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); + let active_generation = engine.current_generation(); + let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); + let mut ctx = policy_poll_test_context( + engine.clone(), + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_revision), + has_last_valid_policy: true, + }, + default_middleware_connector(), + ); + ctx.transparent_tcp = TransparentTcpReloadState { + capable: true, + substrate_ready: false, + }; + let (client, polls, mut reports) = scripted_policy_gateway(); + polls.send(v1).unwrap(); + + let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); + expect_policy_report(&mut reports, 1).await; + polls.send(v2).unwrap(); + let report = timeout(Duration::from_secs(1), reports.recv()) + .await + .expect("TCP rejection report timed out") + .expect("policy reporter stopped"); + + assert_eq!(report.0, 2); + assert!(!report.1); + assert!(report.2.contains("recreate the sandbox"), "{}", report.2); + assert!(report.2.contains("previous policy remains active")); + assert_eq!(engine.current_generation(), active_generation); + assert!(engine.fail_closed_reason().is_none()); + handle.abort(); + } + + #[tokio::test] + async fn same_hash_ack_waits_for_failed_middleware_reconciliation_and_retries_once() { + let mut v1 = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + v1.policy_hash = "same-policy".to_string(); + let mut v2 = v1.clone(); + v2.version = 2; + v2.config_revision = 200; + v2.supervisor_middleware_services = + vec![openshell_core::proto::SupervisorMiddlewareService { + name: "scripted-guard".to_string(), + grpc_endpoint: "http://scripted.invalid".to_string(), + ..Default::default() + }]; + + let connector_attempts = Arc::new(AtomicUsize::new(0)); + let (attempt_tx, mut attempt_rx) = tokio::sync::mpsc::unbounded_channel(); + let middleware_connector: MiddlewareConnector = { + let connector_attempts = connector_attempts.clone(); + Arc::new(move |_services, _authentication| { + let attempt = connector_attempts.fetch_add(1, Ordering::SeqCst) + 1; + attempt_tx.send(attempt).unwrap(); + Box::pin(async move { + if attempt == 1 { + Err(miette::miette!("scripted middleware connection failure")) + } else { + connect_middleware_registry(&[], &MiddlewareAuthentication::default()).await + } + }) + }) + }; + + let engine = + Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); + let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); + let ctx = policy_poll_test_context( + engine.clone(), + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_revision), + has_last_valid_policy: true, + }, + middleware_connector, + ); + let (client, polls, mut reports) = scripted_policy_gateway(); + polls.send(v1).unwrap(); + + let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); + expect_policy_report(&mut reports, 1).await; + + polls.send(v2.clone()).unwrap(); + assert_eq!( + timeout(Duration::from_secs(1), attempt_rx.recv()) + .await + .unwrap(), + Some(1) + ); + expect_no_policy_report(&mut reports).await; + assert_eq!(engine.current_generation(), 0); + + polls.send(v2.clone()).unwrap(); + assert_eq!( + timeout(Duration::from_secs(1), attempt_rx.recv()) + .await + .unwrap(), + Some(2) + ); + expect_policy_report(&mut reports, 2).await; + assert_eq!(engine.current_generation(), 1); + + polls.send(v2).unwrap(); + expect_no_policy_report(&mut reports).await; + assert_eq!(connector_attempts.load(Ordering::SeqCst), 2); + handle.abort(); + } + + #[tokio::test] + async fn no_signer_capability_uses_legacy_middleware_connector_without_credentials() { + let mut v1 = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + v1.policy_hash = "same-policy".to_string(); + let mut v2 = v1.clone(); + v2.version = 2; + v2.config_revision = 200; + v2.supervisor_middleware_services = + vec![openshell_core::proto::SupervisorMiddlewareService { + name: "legacy-guard".to_string(), + grpc_endpoint: "http://legacy.invalid".to_string(), + ..Default::default() + }]; + assert!(!v2.extension_authentication_enabled); + + let (inner, polls, mut reports) = scripted_policy_gateway(); + let credential_requests = Arc::new(AtomicUsize::new(0)); + let client = CredentialRejectingPolicyGateway { + inner, + credential_requests: credential_requests.clone(), + }; + let (connector_tx, mut connector_rx) = tokio::sync::mpsc::unbounded_channel(); + let connector: MiddlewareConnector = Arc::new(move |_services, authentication| { + connector_tx + .send((authentication.credentials.len(), authentication.enabled)) + .unwrap(); + Box::pin(async move { + connect_middleware_registry(&[], &MiddlewareAuthentication::default()).await + }) + }); + let engine = + Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); + let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); + let ctx = policy_poll_test_context( + engine, + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_revision), + has_last_valid_policy: true, + }, + connector, + ); + + polls.send(v1).unwrap(); + let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); + expect_policy_report(&mut reports, 1).await; + polls.send(v2).unwrap(); + assert_eq!( + timeout(Duration::from_secs(1), connector_rx.recv()) + .await + .unwrap(), + Some((0, false)) + ); + expect_policy_report(&mut reports, 2).await; + assert_eq!(credential_requests.load(Ordering::SeqCst), 0); + handle.abort(); + } + + #[tokio::test] + async fn enabled_extension_authentication_keeps_credential_failure_fail_closed() { + let mut v1 = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + v1.policy_hash = "same-policy".to_string(); + let mut v2 = v1.clone(); + v2.version = 2; + v2.config_revision = 200; + v2.extension_authentication_enabled = true; + v2.supervisor_middleware_services = + vec![openshell_core::proto::SupervisorMiddlewareService { + name: "authenticated-guard".to_string(), + grpc_endpoint: "https://guard.invalid".to_string(), + ..Default::default() + }]; + + let (inner, polls, mut reports) = scripted_policy_gateway(); + let credential_requests = Arc::new(AtomicUsize::new(0)); + let client = CredentialRejectingPolicyGateway { + inner, + credential_requests: credential_requests.clone(), + }; + let (connector_tx, mut connector_rx) = tokio::sync::mpsc::unbounded_channel(); + let connector: MiddlewareConnector = Arc::new(move |_services, authentication| { + connector_tx + .send((authentication.credentials.len(), authentication.enabled)) + .unwrap(); + Box::pin(async move { + if authentication.enabled && authentication.credentials.is_empty() { + Err(miette::miette!( + "missing authenticated middleware credential" + )) + } else { + connect_middleware_registry(&[], &authentication).await + } + }) + }); + let engine = + Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); + let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); + let ctx = policy_poll_test_context( + engine, + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_revision), + has_last_valid_policy: true, + }, + connector, + ); + + polls.send(v1).unwrap(); + let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); + expect_policy_report(&mut reports, 1).await; + polls.send(v2).unwrap(); + assert_eq!( + timeout(Duration::from_secs(1), connector_rx.recv()) + .await + .unwrap(), + Some((0, true)) + ); + expect_no_policy_report(&mut reports).await; + assert_eq!(credential_requests.load(Ordering::SeqCst), 1); + handle.abort(); + } + + async fn assert_poll_does_not_use_same_hash_acknowledgement( + initial: openshell_core::grpc_client::SettingsPollResult, + next: openshell_core::grpc_client::SettingsPollResult, + origin: LoadedPolicyOrigin, + initial_report: Option, + ) { + let engine = + Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); + let ctx = policy_poll_test_context(engine.clone(), origin, default_middleware_connector()); + let (client, polls, mut reports) = scripted_policy_gateway(); + polls.send(initial).unwrap(); + let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); + + if let Some(version) = initial_report { + expect_policy_report(&mut reports, version).await; + } else { + expect_no_policy_report(&mut reports).await; + } + + polls.send(next).unwrap(); + expect_no_policy_report(&mut reports).await; + assert_eq!( + engine.current_generation(), + 0, + "negative same-hash scope must not reload OPA" + ); + handle.abort(); + } + + #[tokio::test] + async fn same_hash_ack_poll_loop_rejects_local_global_empty_equal_and_older_scopes() { + let mut sandbox_v1 = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + sandbox_v1.policy_hash = "same-policy".to_string(); + let loaded_v1 = LoadedPolicyRevision::from_snapshot(&sandbox_v1); + let mut sandbox_v2 = sandbox_v1.clone(); + sandbox_v2.version = 2; + sandbox_v2.config_revision = 200; + + assert_poll_does_not_use_same_hash_acknowledgement( + sandbox_v1.clone(), + sandbox_v2.clone(), + LoadedPolicyOrigin::LocalOverride, + None, + ) + .await; + + let mut global_v2 = sandbox_v2.clone(); + global_v2.policy_source = openshell_core::proto::PolicySource::Global; + assert_poll_does_not_use_same_hash_acknowledgement( + sandbox_v1.clone(), + global_v2, + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_v1.clone()), + has_last_valid_policy: true, + }, + Some(1), + ) + .await; + + let mut empty_v1 = sandbox_v1.clone(); + empty_v1.policy_hash.clear(); + let empty_loaded = LoadedPolicyRevision::from_snapshot(&empty_v1); + let mut empty_v2 = sandbox_v2.clone(); + empty_v2.policy_hash.clear(); + assert_poll_does_not_use_same_hash_acknowledgement( + empty_v1, + empty_v2, + LoadedPolicyOrigin::Gateway { + revision: Some(empty_loaded), + has_last_valid_policy: true, + }, + Some(1), + ) + .await; + + assert_poll_does_not_use_same_hash_acknowledgement( + sandbox_v1.clone(), + sandbox_v1.clone(), + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_v1.clone()), + has_last_valid_policy: true, + }, + Some(1), + ) + .await; + + let loaded_v2 = LoadedPolicyRevision::from_snapshot(&sandbox_v2); + assert_poll_does_not_use_same_hash_acknowledgement( + sandbox_v2, + sandbox_v1, + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_v2), + has_last_valid_policy: true, + }, + Some(2), + ) + .await; + } + + #[tokio::test] + async fn changed_hash_poll_uses_normal_opa_reload_and_status_path() { + let v1 = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + let v2 = settings_poll_result( + Some(proto_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Sandbox, + ); + let loaded_revision = LoadedPolicyRevision::from_snapshot(&v1); + let engine = + Arc::new(OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine")); + let ctx = policy_poll_test_context( + engine.clone(), + LoadedPolicyOrigin::Gateway { + revision: Some(loaded_revision), + has_last_valid_policy: true, + }, + default_middleware_connector(), + ); + let (client, polls, mut reports) = scripted_policy_gateway(); + polls.send(v1).unwrap(); + let handle = tokio::spawn(run_policy_poll_loop_with_client(ctx, client)); + + expect_policy_report(&mut reports, 1).await; + polls.send(v2).unwrap(); + expect_policy_report(&mut reports, 2).await; + assert_eq!( + engine.current_generation(), + 1, + "changed policy content must still reload OPA" + ); + handle.abort(); + } + + #[tokio::test] + async fn failed_external_startup_registry_build_preserves_installed_builtins() { + let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); + install_builtin_middleware_registry(&engine) + .await + .expect("install built-in middleware registry"); + let builtins_generation = engine.current_generation(); + assert_eq!(builtins_generation, 1); + + let invalid_external = openshell_core::proto::SupervisorMiddlewareService { + name: "unavailable-guard".into(), + grpc_endpoint: "http://127.0.0.1:1".into(), + max_payload_bytes: 1024, + ..Default::default() + }; + connect_middleware_registry(&[invalid_external], &MiddlewareAuthentication::default()) + .await + .expect_err("unavailable external service must not replace built-ins"); + + assert_eq!(engine.current_generation(), builtins_generation); + } + + #[tokio::test] + async fn unavailable_middleware_reload_keeps_last_known_good_runtime_active() { + let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); + install_builtin_middleware_registry(&engine) + .await + .expect("install built-in middleware registry"); + let active_generation = engine.current_generation(); + let unavailable_service = openshell_core::proto::SupervisorMiddlewareService { + name: "unavailable-guard".into(), + grpc_endpoint: "http://127.0.0.1:1".into(), + max_payload_bytes: 1024, + ..Default::default() + }; + + let failure = reload_gateway_policy_runtime( + &engine, + Some(&proto_policy_fixture()), + 0, + MiddlewareReloadContext { + desired_services: &[unavailable_service], + authentication: &MiddlewareAuthentication::default(), + registry_changed: true, + connector: &default_middleware_connector(), + }, + TransparentTcpReloadState::default(), + ) + .await + .expect_err("unavailable middleware must fail candidate preparation"); + let disposition = apply_gateway_runtime_reload_failure( + &engine, + failure, + PolicyValidationFailureMode::FailClosed, + true, + 2, + ) + .expect("middleware failure handling must succeed"); + + assert!(matches!( + disposition, + GatewayRuntimeFailureDisposition::MiddlewareUnavailable { .. } + )); + assert_eq!(engine.current_generation(), active_generation); + assert!(engine.fail_closed_reason().is_none()); + } + + #[tokio::test] + async fn tcp_policy_reload_without_startup_substrate_is_rejected_and_keeps_previous_policy() { + let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); + let active_generation = engine.current_generation(); + + let failure = reload_gateway_policy_runtime( + &engine, + Some(&proto_tcp_policy_fixture()), + 0, + MiddlewareReloadContext { + desired_services: &[], + authentication: &MiddlewareAuthentication::default(), + registry_changed: false, + connector: &default_middleware_connector(), + }, + TransparentTcpReloadState { + capable: true, + substrate_ready: false, + }, + ) + .await + .expect_err("TCP expansion must require startup substrate"); + let disposition = apply_gateway_runtime_reload_failure( + &engine, + failure, + PolicyValidationFailureMode::FailClosed, + true, + 2, + ) + .expect("runtime prerequisite failure handling must succeed"); + + assert!(matches!( + disposition, + GatewayRuntimeFailureDisposition::TransparentTcpExpansionRejected { + active_generation: generation, + .. + } if generation == active_generation + )); + assert_eq!(engine.current_generation(), active_generation); + assert!(engine.fail_closed_reason().is_none()); + } + + #[tokio::test] + async fn tcp_policy_reload_on_unsupported_runtime_is_rejected() { + let engine = OpaEngine::from_proto(&proto_policy_fixture()).expect("build OPA engine"); + + let failure = reload_gateway_policy_runtime( + &engine, + Some(&proto_tcp_policy_fixture()), + 0, + MiddlewareReloadContext { + desired_services: &[], + authentication: &MiddlewareAuthentication::default(), + registry_changed: false, + connector: &default_middleware_connector(), + }, + TransparentTcpReloadState::default(), + ) + .await + .expect_err("unsupported runtime must reject TCP expansion"); + + assert!(matches!( + failure, + GatewayRuntimeReloadError::TransparentTcpPrerequisite(_) + )); + assert_eq!(engine.current_generation(), 0); + } + + #[test] + fn policy_rejection_after_middleware_outage_is_not_deduplicated() { + let engine = OpaEngine::from_strings( + include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), + "network_policies: {}\n", + ) + .unwrap(); + let middleware_failure = GatewayRuntimeReloadError::MiddlewareRegistry(miette::miette!( + "middleware service unavailable" + )); + let first_failure = FailedRuntimeRevision::new(42, "sha256:candidate", &middleware_failure); + let middleware_disposition = apply_gateway_runtime_reload_failure( + &engine, + middleware_failure, + PolicyValidationFailureMode::FailClosed, + true, + 7, + ) + .unwrap(); + + assert!(matches!( + middleware_disposition, + GatewayRuntimeFailureDisposition::MiddlewareUnavailable { .. } + )); + assert!(engine.fail_closed_reason().is_none()); + + let policy_failure = GatewayRuntimeReloadError::PolicyValidation(miette::miette!( + "conflicting endpoint metadata" + )); + let second_failure = FailedRuntimeRevision::new(42, "sha256:candidate", &policy_failure); + assert_ne!( + first_failure, second_failure, + "a changed failure class for the same candidate must be handled" + ); + + let policy_disposition = apply_gateway_runtime_reload_failure( + &engine, + policy_failure, + PolicyValidationFailureMode::FailClosed, + true, + 7, + ) + .unwrap(); + assert!(matches!( + policy_disposition, + GatewayRuntimeFailureDisposition::PolicyRejected { .. } + )); + assert!(engine.fail_closed_reason().is_some()); + } + + #[test] + fn failed_gateway_runtime_snapshot_is_retried_without_revision_change() { + let services = Vec::new(); + + assert!(gateway_policy_runtime_needs_reconciliation( + true, + "hash-v1", + "hash-v1", + &services, + &services, + MiddlewareRegistryStatus::NeedsReconciliation, + )); + assert!(!gateway_policy_runtime_needs_reconciliation( + true, + "hash-v1", + "hash-v1", + &services, + &services, + MiddlewareRegistryStatus::Synchronized, + )); + } + + #[test] + fn gateway_runtime_reconciliation_tracks_policy_and_service_changes() { + let no_services = Vec::new(); + let desired_services = vec![openshell_core::proto::SupervisorMiddlewareService { + name: "guard".into(), + ..Default::default() + }]; + + assert!(gateway_policy_runtime_needs_reconciliation( + true, + "hash-v1", + "hash-v2", + &no_services, + &no_services, + MiddlewareRegistryStatus::Synchronized, + )); + assert!(gateway_policy_runtime_needs_reconciliation( + true, + "hash-v1", + "hash-v1", + &no_services, + &desired_services, + MiddlewareRegistryStatus::Synchronized, + )); + assert!(!gateway_policy_runtime_needs_reconciliation( + false, + "local-policy", + "hash-v2", + &no_services, + &desired_services, + MiddlewareRegistryStatus::NeedsReconciliation, + )); + } + + #[test] + fn policy_only_change_does_not_rebuild_middleware_registry() { + let services = vec![openshell_core::proto::SupervisorMiddlewareService { + name: "guard".into(), + ..Default::default() + }]; + + // The runtime must reconcile, but the registry (and therefore + // middleware reachability) is not part of that reconciliation. + assert!(gateway_policy_runtime_needs_reconciliation( + true, + "hash-v1", + "hash-v2", + &services, + &services, + MiddlewareRegistryStatus::Synchronized, + )); + assert!(!middleware_registry_needs_rebuild( + MiddlewareRegistryStatus::Synchronized, + &services, + &services, + )); + } + + #[test] + fn registry_rebuild_requires_service_set_change_or_degraded_registry() { + let no_services = Vec::new(); + let desired_services = vec![openshell_core::proto::SupervisorMiddlewareService { + name: "guard".into(), + ..Default::default() + }]; + + assert!(middleware_registry_needs_rebuild( + MiddlewareRegistryStatus::Synchronized, + &no_services, + &desired_services, + )); + assert!(middleware_registry_needs_rebuild( + MiddlewareRegistryStatus::NeedsReconciliation, + &desired_services, + &desired_services, + )); + assert!(!middleware_registry_needs_rebuild( + MiddlewareRegistryStatus::Synchronized, + &desired_services, + &desired_services, + )); + } + + #[test] + fn initial_ack_candidate_matches_sandbox_revision() { + let canonical = settings_poll_result( + Some(proto_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Sandbox, + ); + let loaded = LoadedPolicyRevision::from_snapshot(&canonical); + + let ack = initial_policy_ack_candidate(Some(&loaded), &canonical) + .expect("sandbox-sourced matching revision should be acknowledged"); + + assert_eq!(ack.version, 2); + assert_eq!(ack.policy_hash, "hash-v2"); + assert_eq!(ack.config_revision, 200); + } + + #[test] + fn initial_ack_candidate_ignores_global_policy() { + let canonical = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Global, + ); + let loaded = LoadedPolicyRevision::from_snapshot(&canonical); + + assert!(initial_policy_ack_candidate(Some(&loaded), &canonical).is_none()); + } + + #[test] + fn initial_ack_candidate_ignores_version_zero() { + let canonical = settings_poll_result( + Some(proto_policy_fixture()), + 0, + openshell_core::proto::PolicySource::Sandbox, + ); + let loaded = LoadedPolicyRevision::from_snapshot(&canonical); + + assert!(initial_policy_ack_candidate(Some(&loaded), &canonical).is_none()); + } + + #[test] + fn initial_ack_candidate_ignores_local_file_mode() { + // Local-file mode retains no proto policy, so there is nothing to + // acknowledge to the gateway. + let canonical = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + + assert!(initial_policy_ack_candidate(None, &canonical).is_none()); + } + + #[test] + fn initial_ack_candidate_rejects_mismatched_identity() { + let loaded_snapshot = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + let loaded = LoadedPolicyRevision::from_snapshot(&loaded_snapshot); + let canonical = settings_poll_result( + Some(proto_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Sandbox, + ); + + assert!(initial_policy_ack_candidate(Some(&loaded), &canonical).is_none()); + } + + #[test] + fn initial_poll_reconciles_provider_composition_that_was_not_loaded() { + let loaded_snapshot = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + let loaded = LoadedPolicyRevision::from_snapshot(&loaded_snapshot); + let mut newer = proto_policy_fixture(); + newer.network_policies.insert( + "_provider_work_github".to_string(), + openshell_core::proto::NetworkPolicyRule::default(), + ); + let canonical = + settings_poll_result(Some(newer), 1, openshell_core::proto::PolicySource::Sandbox); + let canonical = openshell_core::grpc_client::SettingsPollResult { + policy_hash: "hash-provider-change".to_string(), + config_revision: loaded.config_revision + 1, + ..canonical + }; + + assert_eq!( + initial_poll_disposition( + &LoadedPolicyOrigin::Gateway { + revision: Some(loaded), + has_last_valid_policy: true, + }, + &canonical, + ), + InitialPollDisposition::Reconcile + ); + } + + #[test] + fn initial_poll_tracks_local_override_without_reconciliation() { + let canonical = settings_poll_result( + Some(proto_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Sandbox, + ); + + assert_eq!( + initial_poll_disposition(&LoadedPolicyOrigin::LocalOverride, &canonical), + InitialPollDisposition::TrackOnly + ); + assert!(!LoadedPolicyOrigin::LocalOverride.allows_gateway_policy_reload()); + } + + #[test] + fn initial_poll_reconciles_unbound_gateway_policy() { + let canonical = settings_poll_result( + Some(proto_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Sandbox, + ); + let origin = LoadedPolicyOrigin::Gateway { + revision: None, + has_last_valid_policy: true, + }; + + assert_eq!( + initial_poll_disposition(&origin, &canonical), + InitialPollDisposition::Reconcile + ); + assert!(origin.allows_gateway_policy_reload()); + } + + #[test] + fn unchanged_sandbox_policy_revision_candidate_is_strictly_scoped() { + let sandbox_result = openshell_core::grpc_client::SettingsPollResult { + policy_hash: "same-policy".to_string(), + ..settings_poll_result( + Some(proto_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Sandbox, + ) + }; + + assert_eq!( + unchanged_policy_revision_candidate(true, false, 1, "same-policy", &sandbox_result), + Some(2) + ); + assert_eq!( + unchanged_policy_revision_candidate(true, false, 2, "same-policy", &sandbox_result), + None + ); + assert_eq!( + unchanged_policy_revision_candidate( + true, + false, + 1, + "different-policy", + &sandbox_result, + ), + None + ); + assert_eq!( + unchanged_policy_revision_candidate(false, false, 1, "same-policy", &sandbox_result), + None + ); + assert_eq!( + unchanged_policy_revision_candidate(true, false, 1, "", &sandbox_result), + None + ); + assert_eq!( + unchanged_policy_revision_candidate(true, true, 1, "same-policy", &sandbox_result), + None + ); + + let global_result = openshell_core::grpc_client::SettingsPollResult { + policy_hash: "same-policy".to_string(), + ..settings_poll_result( + Some(proto_policy_fixture()), + 2, + openshell_core::proto::PolicySource::Global, + ) + }; + assert_eq!( + unchanged_policy_revision_candidate(true, false, 1, "same-policy", &global_result), + None + ); + } + + #[test] + fn unchanged_policy_revision_waits_for_required_runtime_reconciliation() { + assert_eq!( + unchanged_policy_revision_ready_to_ack(Some(2), false, false), + Some(2), + "a same-hash revision needs no OPA reload" + ); + assert_eq!( + unchanged_policy_revision_ready_to_ack(Some(2), true, false), + None, + "failed runtime reconciliation must keep the revision pending" + ); + assert_eq!( + unchanged_policy_revision_ready_to_ack(Some(2), true, true), + Some(2), + "successful runtime reconciliation permits acknowledgement" + ); + assert_eq!( + unchanged_policy_revision_ready_to_ack(None, false, true), + None, + "runtime success cannot manufacture a revision candidate" + ); + } + + #[test] + fn credential_gating_unavailable_for_local_override_with_credentials() { + assert!(credential_gating_unavailable( + &LoadedPolicyOrigin::LocalOverride, + true, + true + )); + } + + #[test] + fn credential_gating_available_without_local_override_or_credentials() { + // A gateway policy is stamped with provenance, so the gates apply. + assert!(!credential_gating_unavailable( + &LoadedPolicyOrigin::Gateway { + revision: None, + has_last_valid_policy: true, + }, + true, + true + )); + // No provider credentials means there is nothing to leak. + assert!(!credential_gating_unavailable( + &LoadedPolicyOrigin::LocalOverride, + false, + true + )); + // Without networking the proxy never evaluates endpoint provenance. + assert!(!credential_gating_unavailable( + &LoadedPolicyOrigin::LocalOverride, + true, + false + )); + } + + #[test] + fn policy_status_outbox_preserves_all_revision_order() { + let (sender, mut receiver) = tokio::sync::mpsc::unbounded_channel(); + for version in 1..=128 { + enqueue_policy_status(&sender, PolicyStatusUpdate::loaded(version)); + } + + for version in 1..=128 { + assert_eq!( + receiver.try_recv().unwrap(), + PolicyStatusUpdate::loaded(version) + ); + } + } + + #[test] + fn settings_snapshot_carries_workspace_for_policy_sync() { + let mut snapshot = settings_poll_result( + Some(proto_policy_fixture()), + 1, + openshell_core::proto::PolicySource::Sandbox, + ); + snapshot.workspace = "beta".to_string(); + + let revision = LoadedPolicyRevision::from_snapshot(&snapshot); + assert_eq!(revision.version, 1); + assert_eq!( + snapshot.workspace, "beta", + "workspace must survive the snapshot so sync_policy_and_fetch_snapshot receives it" + ); + } + #[test] + fn fail_closed_validation_failure_deactivates_previous_generation() { + let engine = OpaEngine::from_strings( + include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), + "network_policies: {}\n", + ) + .unwrap(); + let previous_generation = engine.current_generation(); + + let disposition = apply_policy_validation_failure( + &engine, + PolicyValidationFailureMode::FailClosed, + true, + 7, + "conflicting tls metadata", + ) + .unwrap(); + + assert!(!disposition.previous_policy_active); + assert!(disposition.active_generation > previous_generation); + assert!( + engine + .fail_closed_reason() + .expect("quarantine reason") + .contains("candidate version 7 rejected") + ); + } + + #[test] + fn retain_validation_failure_keeps_previous_generation_active() { + let engine = OpaEngine::from_strings( + include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), + "network_policies: {}\n", + ) + .unwrap(); + let previous_generation = engine.current_generation(); + + let quarantined = apply_policy_validation_failure( + &engine, + PolicyValidationFailureMode::FailClosed, + true, + 6, + "conflicting tls metadata", + ) + .unwrap(); + assert!(!quarantined.previous_policy_active); + + let disposition = apply_policy_validation_failure( + &engine, + PolicyValidationFailureMode::RetainLastValid, + true, + 7, + "conflicting tls metadata", + ) + .unwrap(); + + assert!(disposition.previous_policy_active); + assert!(disposition.active_generation > quarantined.active_generation); + assert!(disposition.active_generation > previous_generation); + assert!(engine.fail_closed_reason().is_none()); + } + + #[test] + fn retain_validation_failure_without_last_valid_policy_stays_fail_closed() { + let engine = OpaEngine::from_strings( + include_str!("../../openshell-supervisor-network/data/sandbox-policy.rego"), + "network_policies: {}\n", + ) + .unwrap(); + + let disposition = apply_policy_validation_failure( + &engine, + PolicyValidationFailureMode::RetainLastValid, + false, + 1, + "conflicting tls metadata", + ) + .unwrap(); + + assert_eq!( + disposition.configured_mode, + PolicyValidationFailureMode::RetainLastValid + ); + assert_eq!(disposition.mode, PolicyValidationFailureMode::FailClosed); + assert!(!disposition.previous_policy_active); + assert!(engine.fail_closed_reason().is_some()); + + let [config, _] = policy_validation_failure_events( + &disposition, + 1, + "sha256:test", + "conflicting tls metadata", + ); + let config = config.to_json().unwrap(); + assert_eq!(config["unmapped"]["validation_failure_mode"], "fail_closed"); + assert_eq!( + config["unmapped"]["configured_validation_failure_mode"], + "retain_last_valid" + ); + assert!( + config["message"] + .as_str() + .unwrap() + .contains("previous policy IS NOT active") + ); + } + + #[test] + fn validation_failure_ocsf_states_whether_previous_policy_is_active() { + let fail_closed = PolicyValidationFailureDisposition { + configured_mode: PolicyValidationFailureMode::FailClosed, + mode: PolicyValidationFailureMode::FailClosed, + previous_policy_active: false, + active_generation: 9, + }; + let [config, finding] = policy_validation_failure_events( + &fail_closed, + 8, + "sha256:test", + "conflicting tls metadata", + ); + let config = config.to_json().unwrap(); + assert_eq!(config["class_uid"], 5019); + assert_eq!(config["status"], "Failure"); + assert_eq!(config["unmapped"]["validation_failure_mode"], "fail_closed"); + assert_eq!( + config["unmapped"]["configured_validation_failure_mode"], + "fail_closed" + ); + assert_eq!(config["unmapped"]["previous_policy_active"], false); + assert_eq!( + config["unmapped"]["validation_error"], + "conflicting tls metadata" + ); + assert!( + config["message"] + .as_str() + .unwrap() + .contains("previous policy IS NOT active") + ); + assert!( + config["message"] + .as_str() + .unwrap() + .contains("error:conflicting tls metadata") + ); + + let finding = finding.to_json().unwrap(); + assert_eq!(finding["class_uid"], 2004); + assert_eq!(finding["action"], "Denied"); + assert_eq!(finding["disposition"], "Blocked"); + + let retained = PolicyValidationFailureDisposition { + configured_mode: PolicyValidationFailureMode::RetainLastValid, + mode: PolicyValidationFailureMode::RetainLastValid, + previous_policy_active: true, + active_generation: 4, + }; + let [config, _] = policy_validation_failure_events( + &retained, + 8, + "sha256:test", + "conflicting tls metadata", + ); + let config = config.to_json().unwrap(); + assert_eq!(config["unmapped"]["previous_policy_active"], true); + assert!( + config["message"] + .as_str() + .unwrap() + .contains("previous policy IS active") + ); + } +} diff --git a/crates/openshell-supervisor/src/main.rs b/crates/openshell-supervisor/src/main.rs new file mode 100644 index 0000000000..515572b083 --- /dev/null +++ b/crates/openshell-supervisor/src/main.rs @@ -0,0 +1,559 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! `OpenShell` supervisor executable. + +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; + +use clap::{Parser, ValueEnum}; +use miette::{IntoDiagnostic, Result}; +use openshell_isolation_interface::contract::BackendDescriptor; +use openshell_ocsf::{OcsfJsonlLayer, OcsfShorthandLayer}; +use tracing::{info, warn}; +use tracing_subscriber::EnvFilter; +use tracing_subscriber::filter::LevelFilter; +use tracing_subscriber::{Layer as _, layer::SubscriberExt as _, util::SubscriberInitExt as _}; + +const DEBUG_RPC_SUBCOMMAND: &str = "debug-rpc"; +const HEALTH_SUBCOMMAND: &str = "health"; + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, ValueEnum)] +enum SupervisorRole { + /// Attach an Isolation Backend and supervise one sandbox generation. + #[default] + IsolationBackend, + /// Run only the explicit HTTP/CONNECT network proxy. + NetworkProxy, +} + +#[derive(Parser, Debug)] +#[command(name = "openshell-supervisor health")] +struct HealthArgs { + /// Private supervisor readiness socket. + #[arg(long, env = "OPENSHELL_HEALTH_SOCKET_PATH")] + socket: PathBuf, +} + +#[derive(Parser, Debug)] +#[command(name = "openshell-supervisor")] +#[command(version = openshell_core::VERSION)] +#[command(about = "OpenShell policy and workload supervisor")] +#[allow(clippy::struct_excessive_bools)] +struct Args { + /// Supervisor responsibility to run. + #[arg(long, value_enum, default_value_t)] + role: SupervisorRole, + + /// Command to execute as the canonical workload process. + #[arg(trailing_var_arg = true)] + command: Vec, + + #[arg(long, short)] + workdir: Option, + + #[arg(long, short, default_value = "0")] + timeout: u64, + + #[arg(long, short = 'i')] + interactive: bool, + + #[arg(long, env = openshell_core::sandbox_env::SANDBOX_ID)] + sandbox_id: Option, + + #[arg(long, env = openshell_core::sandbox_env::SANDBOX)] + sandbox: Option, + + #[arg(long, env = openshell_core::sandbox_env::ENDPOINT)] + openshell_endpoint: Option, + + #[arg(long, env = "OPENSHELL_POLICY_RULES")] + policy_rules: Option, + + #[arg(long, env = "OPENSHELL_POLICY_DATA")] + policy_data: Option, + + #[arg(long, env = openshell_core::sandbox_env::SSH_SOCKET_PATH)] + ssh_socket_path: Option, + + #[arg(long, default_value = "warn", env = openshell_core::sandbox_env::LOG_LEVEL)] + log_level: String, + + /// Create the private readiness socket after boundary and gateway attach. + #[arg(long, env = "OPENSHELL_HEALTH_SOCKET_PATH")] + health_socket_path: Option, + + #[arg(long)] + upstream_proxy: Option, + + /// Driver-pinned TCP dial address for the configured upstream proxy. + #[arg(long)] + upstream_proxy_dial_ip: Option, + + #[arg(long)] + upstream_no_proxy: Option, + + #[arg(long)] + upstream_proxy_auth_file: Option, + + #[arg(long)] + upstream_proxy_auth_allow_insecure: bool, + + #[arg(long)] + upstream_proxy_connect_by_hostname: bool, + + #[arg(long)] + upstream_proxy_ca_bundle: Option, + + #[arg(long)] + backend_descriptor_file: Option, + + /// Protected gateway-issued credentials for this exact sandbox launch. + #[arg(long)] + auth_bundle_file: Option, + + /// Loopback HTTP/CONNECT listener used by `--role=network-proxy`. + #[arg(long)] + listen: Option, + + /// Directory for the generated proxy CA certificate and trust bundle. + #[arg(long)] + tls_dir: Option, + + #[arg(long, hide = true)] + main_exit_marker: Option, + + /// Read end of a driver-owned pipe. EOF means the owning driver exited. + #[arg(long, hide = true)] + parent_liveness_fd: Option, +} + +#[cfg(unix)] +#[allow(unsafe_code)] +fn arm_parent_liveness(raw_fd: Option) -> Result<()> { + use std::io::Read as _; + use std::os::fd::{FromRawFd as _, OwnedFd}; + + let Some(raw_fd) = raw_fd else { + return Ok(()); + }; + if raw_fd <= 2 { + return Err(miette::miette!("parent liveness descriptor is invalid")); + } + nix::fcntl::fcntl(raw_fd, nix::fcntl::FcntlArg::F_GETFD) + .map_err(|error| miette::miette!("parent liveness descriptor is not open: {error}"))?; + // SAFETY: the driver transfers this inherited descriptor to the + // supervisor exactly once through the private command line. + let fd = unsafe { OwnedFd::from_raw_fd(raw_fd) }; + std::thread::Builder::new() + .name("supervisor-parent-liveness".to_string()) + .spawn(move || { + let mut stream = std::fs::File::from(fd); + let mut byte = [0_u8; 1]; + loop { + match stream.read(&mut byte) { + Ok(0) | Err(_) => std::process::exit(1), + Ok(_) => {} + } + } + }) + .map(|_| ()) + .into_diagnostic() +} + +#[cfg(not(unix))] +fn arm_parent_liveness(raw_fd: Option) -> Result<()> { + if raw_fd.is_some() { + return Err(miette::miette!( + "parent liveness descriptors are unsupported on this platform" + )); + } + Ok(()) +} + +fn backend_descriptor(args: &Args) -> Result { + let path = args.backend_descriptor_file.as_deref().ok_or_else(|| { + miette::miette!("--backend-descriptor-file is required for --role=isolation-backend") + })?; + let payload = std::fs::read(path) + .map_err(|error| miette::miette!("read backend descriptor {}: {error}", path.display()))?; + Ok(BackendDescriptor { + backend_name: openshell_sandbox_backend::BACKEND_NAME.to_string(), + payload, + }) +} + +fn auth_bundle(args: &Args) -> Result { + let path = args.auth_bundle_file.as_deref().ok_or_else(|| { + miette::miette!("--auth-bundle-file is required for --role=isolation-backend") + })?; + let bytes = std::fs::read(path).map_err(|error| { + miette::miette!( + "read supervisor authentication bundle {}: {error}", + path.display() + ) + })?; + let bundle = serde_json::from_slice::(&bytes) + .map_err(|error| miette::miette!("decode supervisor authentication bundle: {error}"))?; + bundle + .validate() + .map_err(|error| miette::miette!("validate supervisor authentication bundle: {error}"))?; + Ok(bundle) +} + +fn validate_role_arguments(args: &Args) -> Result<()> { + match args.role { + SupervisorRole::IsolationBackend => { + if args.backend_descriptor_file.is_none() { + return Err(miette::miette!( + "--backend-descriptor-file is required for --role=isolation-backend" + )); + } + if args.auth_bundle_file.is_none() { + return Err(miette::miette!( + "--auth-bundle-file is required for --role=isolation-backend" + )); + } + if args.listen.is_some() { + return Err(miette::miette!( + "--listen is only valid with --role=network-proxy" + )); + } + if args.tls_dir.is_some() { + return Err(miette::miette!( + "--tls-dir is only valid with --role=network-proxy" + )); + } + } + SupervisorRole::NetworkProxy => { + if args.backend_descriptor_file.is_some() + || args.auth_bundle_file.is_some() + || args.sandbox_id.is_some() + || args.sandbox.is_some() + || args.openshell_endpoint.is_some() + || args.ssh_socket_path.is_some() + || args.health_socket_path.is_some() + || args.main_exit_marker.is_some() + || args.parent_liveness_fd.is_some() + { + return Err(miette::miette!( + "--role=network-proxy does not use sandbox identity, gateway, runtime, or process-control arguments" + )); + } + if !args.command.is_empty() + || args.workdir.is_some() + || args.interactive + || args.timeout != 0 + { + return Err(miette::miette!( + "--role=network-proxy does not launch or manage a workload" + )); + } + if args.policy_rules.is_none() || args.policy_data.is_none() { + return Err(miette::miette!( + "--policy-rules and --policy-data are required for --role=network-proxy" + )); + } + } + } + Ok(()) +} + +fn validate_main_exit_marker(marker: Option<&Path>) -> Result<()> { + if let Some(marker) = marker + && !marker.is_absolute() + { + return Err(miette::miette!( + "--main-exit-marker must be an absolute path" + )); + } + Ok(()) +} + +fn main() -> Result<()> { + let raw_args = std::env::args().collect::>(); + if raw_args.get(1).map(String::as_str) == Some(DEBUG_RPC_SUBCOMMAND) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .into_diagnostic()?; + return runtime.block_on(async move { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let exit = openshell_supervisor_process::debug_rpc::run(&raw_args[2..]).await?; + std::process::exit(exit); + }); + } + if raw_args.get(1).map(String::as_str) == Some(HEALTH_SUBCOMMAND) { + let args = HealthArgs::parse_from(&raw_args[1..]); + return openshell_supervisor::check_control_readiness(&args.socket); + } + + let args = Args::parse(); + validate_role_arguments(&args)?; + arm_parent_liveness(args.parent_liveness_fd)?; + validate_main_exit_marker(args.main_exit_marker.as_deref())?; + let isolation_inputs = if args.role == SupervisorRole::IsolationBackend { + let descriptor = backend_descriptor(&args)?; + let auth = auth_bundle(&args)?; + // Install the driver-provisioned session before starting log push or + // any other gateway client. `run_sandbox` obtains the same Sandbox + // Protocol bearer slot after validating the descriptor binding. + let _ = openshell_core::grpc_client::install_supervisor_auth_bundle(&auth)?; + Some((descriptor, auth)) + } else { + None + }; + + let file_logging = tracing_appender::rolling::RollingFileAppender::builder() + .rotation(tracing_appender::rolling::Rotation::DAILY) + .filename_prefix("openshell") + .filename_suffix("log") + .max_log_files(3) + .build("/var/log") + .ok() + .map(|roller| { + let (writer, guard) = tracing_appender::non_blocking(roller); + (writer, guard) + }); + let console_filter = + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)); + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .into_diagnostic()?; + + let exit_code = runtime.block_on(async move { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + let log_push_state = if args.role == SupervisorRole::IsolationBackend + && let (Some(sandbox_id), Some(endpoint)) = (&args.sandbox_id, &args.openshell_endpoint) + { + let (tx, handle) = openshell_supervisor_process::log_push::spawn_log_push_task( + endpoint.clone(), + sandbox_id.clone(), + ); + let layer = + openshell_supervisor_process::log_push::LogPushLayer::new(sandbox_id.clone(), tx); + Some((layer, handle)) + } else { + None + }; + let push_layer = log_push_state.as_ref().map(|(layer, _)| layer.clone()); + let _log_push_handle = log_push_state.map(|(_, handle)| handle); + let ocsf_enabled = Arc::new(AtomicBool::new(false)); + + let (_file_guard, _jsonl_guard) = if let Some((file_writer, file_guard)) = file_logging { + let jsonl_logging = tracing_appender::rolling::RollingFileAppender::builder() + .rotation(tracing_appender::rolling::Rotation::DAILY) + .filename_prefix("openshell-ocsf") + .filename_suffix("log") + .max_log_files(3) + .build("/var/log") + .ok() + .map(|roller| { + let (writer, guard) = tracing_appender::non_blocking(roller); + let layer = OcsfJsonlLayer::new(writer).with_enabled_flag(ocsf_enabled.clone()); + (layer, guard) + }); + let (jsonl_layer, jsonl_guard) = + jsonl_logging.map_or((None, None), |(layer, guard)| (Some(layer), Some(guard))); + tracing_subscriber::registry() + .with( + OcsfShorthandLayer::new(std::io::stderr()) + .with_non_ocsf(true) + .with_filter(console_filter), + ) + .with( + OcsfShorthandLayer::new(file_writer) + .with_non_ocsf(true) + .with_filter(EnvFilter::new("info")), + ) + .with(jsonl_layer.with_filter(LevelFilter::INFO)) + .with(push_layer.clone()) + .init(); + (Some(file_guard), jsonl_guard) + } else { + tracing_subscriber::registry() + .with( + OcsfShorthandLayer::new(std::io::stderr()) + .with_non_ocsf(true) + .with_filter(console_filter), + ) + .with(push_layer) + .init(); + warn!("Could not open /var/log for log rotation; using stderr-only logging"); + (None, None) + }; + + let workdir = args.workdir.clone(); + let (command, interactive, await_main_process_attachment) = if !args.command.is_empty() { + (args.command, args.interactive, false) + } else if let Ok(json) = std::env::var(openshell_core::sandbox_env::MAIN_PROCESS_SPEC) { + let config = openshell_core::sandbox_env::MainProcessConfig::decode(&json) + .map_err(|error| miette::miette!("{error}"))?; + ( + config.command, + config.tty, + config.await_main_process_attachment, + ) + } else { + let config = openshell_core::sandbox_env::MainProcessConfig::scratch(); + ( + config.command, + config.tty, + config.await_main_process_attachment, + ) + }; + let upstream_proxy_args = openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs { + https_proxy: args.upstream_proxy, + proxy_dial_ip: args.upstream_proxy_dial_ip, + no_proxy: args.upstream_no_proxy, + proxy_auth_file: args.upstream_proxy_auth_file, + proxy_auth_allow_insecure: args.upstream_proxy_auth_allow_insecure, + proxy_connect_by_hostname: args.upstream_proxy_connect_by_hostname, + proxy_ca_bundle: args.upstream_proxy_ca_bundle, + }; + match args.role { + SupervisorRole::IsolationBackend => { + info!(command = ?command, "Starting sandbox supervision"); + let Some((backend_descriptor, auth_bundle)) = isolation_inputs else { + return Err(miette::miette!( + "isolation-backend role started without validated runtime inputs" + )); + }; + let admitted_isolation_backend = + std::env::var(openshell_core::sandbox_env::ADMITTED_ISOLATION_BACKEND).ok(); + openshell_supervisor::run_sandbox( + command, + workdir, + args.timeout, + interactive, + await_main_process_attachment, + args.sandbox_id, + args.sandbox, + args.openshell_endpoint, + args.policy_rules, + args.policy_data, + args.ssh_socket_path, + args.health_socket_path, + ocsf_enabled, + upstream_proxy_args, + backend_descriptor, + auth_bundle, + admitted_isolation_backend, + args.main_exit_marker, + ) + .await + } + SupervisorRole::NetworkProxy => { + let listen = args.listen.unwrap_or_else(|| ([127, 0, 0, 1], 3128).into()); + let (Some(policy_rules), Some(policy_data)) = (args.policy_rules, args.policy_data) + else { + return Err(miette::miette!( + "network-proxy role started without validated policy files" + )); + }; + openshell_supervisor::run_network_proxy( + listen, + policy_rules, + policy_data, + args.tls_dir, + upstream_proxy_args, + ) + .await + } + } + })?; + + std::process::exit(exit_code); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn isolation_backend_is_the_default_role() { + let directory = tempfile::tempdir().expect("temporary runtime descriptor directory"); + let descriptor_path = directory.path().join("runtime-descriptor.json"); + let auth_bundle_path = directory.path().join("auth-bundle.json"); + std::fs::write(&descriptor_path, [0]).expect("write runtime descriptor payload"); + std::fs::write(&auth_bundle_path, [0]).expect("write auth bundle payload"); + let args = Args::try_parse_from([ + "openshell-supervisor", + "--backend-descriptor-file", + descriptor_path + .to_str() + .expect("UTF-8 runtime descriptor path"), + "--auth-bundle-file", + auth_bundle_path.to_str().expect("UTF-8 auth bundle path"), + ]) + .expect("supervisor arguments"); + assert_eq!(args.role, SupervisorRole::IsolationBackend); + assert!(validate_role_arguments(&args).is_ok()); + assert_eq!( + backend_descriptor(&args) + .expect("runtime descriptor") + .payload, + vec![0] + ); + } + + #[test] + fn isolation_backend_inputs_are_mandatory() { + let args = Args::try_parse_from(["openshell-supervisor"]).expect("parse defaults"); + assert!(validate_role_arguments(&args).is_err()); + } + + #[test] + fn network_proxy_accepts_local_policy_files() { + let args = Args::try_parse_from([ + "openshell-supervisor", + "--role", + "network-proxy", + "--policy-rules", + "/tmp/policy.rego", + "--policy-data", + "/tmp/policy.yaml", + ]) + .expect("network-proxy arguments"); + assert_eq!(args.role, SupervisorRole::NetworkProxy); + assert!(validate_role_arguments(&args).is_ok()); + } + + #[test] + fn network_proxy_rejects_isolation_inputs() { + let args = Args::try_parse_from([ + "openshell-supervisor", + "--role", + "network-proxy", + "--policy-rules", + "/tmp/policy.rego", + "--policy-data", + "/tmp/policy.yaml", + "--backend-descriptor-file", + "/tmp/descriptor.json", + ]) + .expect("network-proxy arguments"); + assert!(validate_role_arguments(&args).is_err()); + } + + #[test] + fn network_proxy_requires_both_policy_files() { + let args = Args::try_parse_from([ + "openshell-supervisor", + "--role", + "network-proxy", + "--policy-rules", + "/tmp/policy.rego", + ]) + .expect("network-proxy arguments"); + assert!(validate_role_arguments(&args).is_err()); + } + + #[test] + fn completion_marker_must_be_absolute() { + assert!(validate_main_exit_marker(Some(Path::new("relative"))).is_err()); + assert!(validate_main_exit_marker(Some(Path::new("/run/openshell/main-exit"))).is_ok()); + } +} diff --git a/crates/openshell-supervisor/src/mechanistic_mapper.rs b/crates/openshell-supervisor/src/mechanistic_mapper.rs new file mode 100644 index 0000000000..260f4ec180 --- /dev/null +++ b/crates/openshell-supervisor/src/mechanistic_mapper.rs @@ -0,0 +1,778 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Supervisor policy mapper — deterministically converts denial summaries into +//! draft `NetworkPolicyRule` proposals. +//! +//! This is the "zero-LLM" baseline for policy recommendations. It inspects +//! denial patterns (host, port, binary, frequency) and generates concrete rules +//! that would allow the denied connections, annotated with confidence scores and +//! security notes. +//! +//! The LLM-powered `PolicyAdvisor` (issue #205) wraps and enriches these +//! mechanistic proposals with context-aware rationale and smarter grouping. + +use openshell_core::net::{is_always_blocked_ip, is_internal_ip, is_known_metadata_hostname}; +use openshell_core::proto::{ + DenialSummary, L7Allow, L7Rule, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, PolicyChunk, +}; +use std::collections::HashMap; +use std::net::IpAddr; + +/// Well-known ports that get higher confidence scores. +const WELL_KNOWN_PORTS: &[(u16, &str)] = &[ + (80, "HTTP"), + (443, "HTTPS"), + (8080, "HTTP-alt"), + (8443, "HTTPS-alt"), + (5432, "PostgreSQL"), + (3306, "MySQL"), + (6379, "Redis"), + (27017, "MongoDB"), + (9200, "Elasticsearch"), + (9092, "Kafka"), + (2181, "ZooKeeper"), + (11211, "Memcached"), + (5672, "RabbitMQ"), + (6443, "Kubernetes API"), + (53, "DNS"), + (587, "SMTP"), + (993, "IMAP"), + (995, "POP3"), +]; + +/// Generate draft `PolicyChunk` proposals from denial summaries. +/// +/// Groups denials by `(host, port, binary)`, then for each group generates a +/// `PolicyChunk` with a `NetworkPolicyRule` allowing that endpoint for that +/// single binary. This produces one proposal per binary so each +/// `(sandbox_id, host, port, binary)` maps to exactly one DB row. +/// +/// Proposals never include `allowed_ips`. If the user applies a proposed rule +/// and the host resolves to a private IP, the proxy's SSRF defense will deny +/// the connection. That SSRF denial flows back through the aggregator, and the +/// user can then explicitly add `allowed_ips` to their policy. This two-step +/// flow avoids DNS resolution in the mapper, which would leak the denied +/// hostname via DNS even though the connection was blocked. See #1169. +/// +/// Returns an empty vec if there are no actionable denials. +pub fn generate_proposals(summaries: &[DenialSummary]) -> Vec { + // Group denials by (host, port, binary). + let mut groups: HashMap<(String, u32, String), Vec<&DenialSummary>> = HashMap::new(); + + for summary in summaries { + let binary_key = if summary.binary.is_empty() { + String::new() + } else { + summary.binary.clone() + }; + groups + .entry((summary.host.clone(), summary.port, binary_key)) + .or_default() + .push(summary); + } + + let mut proposals = Vec::new(); + + for ((host, port, binary), denials) in &groups { + let rule_name = generate_rule_name(host, *port); + + let mut total_count: u32 = 0; + let mut first_seen_ms: i64 = i64::MAX; + let mut last_seen_ms: i64 = 0; + let mut is_ssrf = false; + + for denial in denials { + total_count += denial.count; + first_seen_ms = first_seen_ms.min(denial.first_seen_ms); + last_seen_ms = last_seen_ms.max(denial.last_seen_ms); + if denial.denial_stage == "ssrf" { + is_ssrf = true; + } + } + + // Collect L7 request samples across all denials in this group. + let mut l7_methods: HashMap<(String, String), u32> = HashMap::new(); + let mut has_l7 = false; + for denial in denials { + if denial.l7_inspection_active || !denial.l7_request_samples.is_empty() { + has_l7 = true; + } + for sample in &denial.l7_request_samples { + *l7_methods + .entry((sample.method.clone(), sample.path.clone())) + .or_insert(0) += sample.count; + } + } + + // Skip proposals for always-blocked destinations (loopback, + // link-local, unspecified, and known metadata hostnames). These would + // be denied at runtime regardless of policy, producing an infinite + // proposal loop in the TUI. + if is_always_blocked_destination(host) { + tracing::info!( + host, + port, + "Skipped proposal for always-blocked destination \ + (SSRF hardening — loopback/link-local/unspecified/metadata)" + ); + continue; + } + + // Build proposed NetworkPolicyRule. + let l7_rules = build_l7_rules(&l7_methods); + let endpoint = if has_l7 && !l7_rules.is_empty() { + NetworkEndpoint { + host: host.clone(), + port: *port, + ports: vec![*port], + protocol: "rest".to_string(), + enforcement: "enforce".to_string(), + rules: l7_rules, + advisor_proposed: true, + ..Default::default() + } + } else { + NetworkEndpoint { + host: host.clone(), + port: *port, + ports: vec![*port], + advisor_proposed: true, + ..Default::default() + } + }; + + let binaries: Vec = if binary.is_empty() { + vec![] + } else { + let proposal_binary = NetworkBinary { + path: binary.clone(), + }; + vec![proposal_binary] + }; + + let proposed_rule = NetworkPolicyRule { + name: rule_name.clone(), + endpoints: vec![endpoint], + binaries, + }; + + // Compute confidence. + #[allow(clippy::cast_possible_truncation)] + let confidence = compute_confidence(total_count, *port as u16, is_ssrf); + + // Generate rationale. + let binary_list = if binary.is_empty() { + "unknown binary".to_string() + } else { + short_binary_name(binary) + }; + + #[allow(clippy::cast_possible_truncation)] + let port_u16 = *port as u16; + let port_name = WELL_KNOWN_PORTS + .iter() + .find(|(p, _)| *p == port_u16) + .map(|(_, name)| format!(" ({name})")) + .unwrap_or_default(); + + // Note: hit_count in the DB accumulates across flush cycles, so we + // don't bake a denial count into the rationale text (it would go stale). + let rationale = if has_l7 && !l7_methods.is_empty() { + let paths: Vec = l7_methods.keys().map(|(m, p)| format!("{m} {p}")).collect(); + format!( + "Allow {binary_list} to connect to {host}:{port}{port_name} \ + with L7 inspection. \ + Allowed paths: {}.", + paths.join(", ") + ) + } else { + format!( + "Allow {binary_list} to connect to \ + {host}:{port}{port_name}." + ) + }; + + // Generate security notes. + #[allow(clippy::cast_possible_truncation)] + let security_notes = generate_security_notes(host, *port as u16, is_ssrf); + + // Determine stage based on denial source. + let stage = denials + .first() + .map_or_else(|| "connect".to_string(), |d| d.denial_stage.clone()); + + proposals.push(PolicyChunk { + id: String::new(), // Assigned by the gateway on persist + status: "pending".to_string(), + rule_name, + proposed_rule: Some(proposed_rule), + rationale, + security_notes, + confidence, + denial_summary_ids: vec![], + created_at_ms: 0, // Set by gateway on persist + decided_at_ms: 0, + stage, + supersedes_chunk_id: String::new(), + hit_count: total_count.cast_signed(), + first_seen_ms, + last_seen_ms, + binary: binary.clone(), + validation_result: String::new(), + rejection_reason: String::new(), + ..Default::default() + }); + } + + // Sort proposals by confidence (highest first). + proposals.sort_by(|a, b| { + b.confidence + .partial_cmp(&a.confidence) + .unwrap_or(std::cmp::Ordering::Equal) + }); + + proposals +} + +/// Generate a rule name that doesn't conflict with existing rules. +/// Generate a deterministic, idempotent rule name from host and port. +/// +/// The same `(host, port)` always produces the same name. DB-level dedup on +/// `(sandbox_id, host, port, binary)` handles collisions — no need to check +/// existing rule names. +fn generate_rule_name(host: &str, port: u32) -> String { + let sanitized = host + .replace(['.', '-'], "_") + .chars() + .filter(|c| c.is_alphanumeric() || *c == '_') + .collect::(); + + format!("allow_{sanitized}_{port}") +} + +/// Compute a confidence score (0.0 to 1.0) for a proposed rule. +fn compute_confidence(total_count: u32, port: u16, is_ssrf: bool) -> f32 { + let mut score: f32 = 0.5; + + // Higher count → higher confidence (the denial is repeatable). + if total_count >= 10 { + score += 0.2; + } else if total_count >= 3 { + score += 0.1; + } + + // Well-known port → higher confidence. + if WELL_KNOWN_PORTS.iter().any(|(p, _)| *p == port) { + score += 0.15; + } + + // SSRF denials are lower confidence (may be legitimate blocking). + if is_ssrf { + score -= 0.2; + } + + score.clamp(0.1, 0.95) +} + +/// Generate security notes for a proposed rule. +fn generate_security_notes(host: &str, port: u16, is_ssrf: bool) -> String { + let mut notes = Vec::new(); + + if is_ssrf { + notes.push( + "This connection was blocked by SSRF protection. \ + Private IP access requires an explicit `allowed_ips` policy entry." + .to_string(), + ); + } + + // Flag destinations that are an internal/private address. Parse the host as + // an IP literal and defer to the canonical RFC-accurate classifier + // (openshell-core net::is_internal_ip) rather than naive string prefixes: + // `starts_with("172.")` wrongly matched 172.0-15 / 172.32-255 (RFC 1918 is + // only 172.16.0.0/12) and missed CGNAT (100.64.0.0/10), IPv6 ULA, etc. The + // "localhost" hostname is not an IP literal, so it is checked separately. + // See #1777. + let resolves_internal = host.parse::().is_ok_and(is_internal_ip); + if resolves_internal || host == "localhost" { + notes.push(format!( + "Destination '{host}' appears to be an internal/private address." + )); + } + + // High port numbers may indicate ephemeral services. + if port > 49152 { + notes.push(format!( + "Port {port} is in the ephemeral range — \ + this may be a temporary service." + )); + } + + // Database ports get extra scrutiny. + let db_ports = [5432, 3306, 6379, 27017, 9200, 11211, 5672]; + if db_ports.contains(&port) { + notes.push(format!( + "Port {port} is a well-known database/service port. \ + Consider restricting with L7 rules or read-only access." + )); + } + + notes.join(" ") +} + +/// Build L7 allow-rules from observed (method, path) samples. +/// +/// Groups paths by HTTP method and generalises path patterns where possible: +/// - `/v1/models/abc123` → `/v1/models/**` (ID-like trailing segments) +/// - `/api/v2/users/42` → `/api/v2/users/*` (numeric trailing segment) +/// +/// Falls back to the exact observed path when no pattern applies. +fn build_l7_rules(samples: &HashMap<(String, String), u32>) -> Vec { + // Deduplicate after generalisation. + let mut seen: std::collections::HashSet<(String, String)> = std::collections::HashSet::new(); + let mut rules = Vec::new(); + + for (method, path) in samples.keys() { + let generalised = generalise_path(path); + let key = (method.clone(), generalised.clone()); + if !seen.insert(key) { + continue; + } + + rules.push(L7Rule { + allow: Some(L7Allow { + method: method.clone(), + path: generalised, + command: String::new(), + query: HashMap::new(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), + params: HashMap::new(), + }), + }); + } + + // Sort for deterministic output. + rules.sort_by(|a, b| { + let a = a.allow.as_ref().unwrap(); + let b = b.allow.as_ref().unwrap(); + (&a.method, &a.path).cmp(&(&b.method, &b.path)) + }); + + rules +} + +/// Generalise a URL path for policy rules. +/// +/// Heuristics: +/// - Strip query strings. +/// - If the last segment looks like an ID (hex, UUID, or numeric), replace +/// with `*`. +/// - Preserve all other segments verbatim. +fn generalise_path(raw: &str) -> String { + // Strip query string. + let path = raw.split('?').next().unwrap_or(raw); + + let segments: Vec<&str> = path.split('/').collect(); + if segments.len() <= 1 { + return path.to_string(); + } + + let last = segments.last().unwrap_or(&""); + + // Replace ID-like trailing segments with a wildcard. + if looks_like_id(last) { + let mut out = segments[..segments.len() - 1].join("/"); + out.push_str("/*"); + return out; + } + + path.to_string() +} + +/// Heuristic: does a path segment look like an opaque identifier? +fn looks_like_id(segment: &str) -> bool { + if segment.is_empty() { + return false; + } + // Pure numeric + if segment.chars().all(|c| c.is_ascii_digit()) && segment.len() >= 2 { + return true; + } + // UUID-ish (contains dashes, 32+ hex chars) + let hex_only: String = segment.chars().filter(char::is_ascii_hexdigit).collect(); + if hex_only.len() >= 24 && segment.contains('-') { + return true; + } + // Long hex string (hash, token) + if hex_only.len() >= 16 && segment.len() == hex_only.len() { + return true; + } + false +} + +/// Extract just the binary name from a full path. +fn short_binary_name(path: &str) -> String { + path.rsplit('/').next().unwrap_or(path).to_string() +} + +/// Check if a destination host is always-blocked. +/// +/// For literal IP hosts, checks against [`is_always_blocked_ip`]. +/// For hostnames, checks well-known loopback and cloud metadata names. +/// For other hostnames, returns false (DNS may resolve to anything). +fn is_always_blocked_destination(host: &str) -> bool { + // Check literal IP addresses + if let Ok(ip) = host.parse::() { + return is_always_blocked_ip(ip); + } + // Check well-known loopback hostnames + let host_lc = host.to_lowercase(); + host_lc == "localhost" || host_lc == "localhost." || is_known_metadata_hostname(host) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_generate_rule_name() { + let name = generate_rule_name("example.com", 443); + assert_eq!(name, "allow_example_com_443"); + } + + #[test] + fn test_generate_rule_name_subdomain() { + let name = generate_rule_name("api.github.com", 443); + assert_eq!(name, "allow_api_github_com_443"); + } + + #[test] + fn test_compute_confidence() { + // Well-known port + high count + let conf = compute_confidence(10, 443, false); + assert!(conf > 0.8); + + // SSRF + let conf = compute_confidence(5, 80, true); + assert!(conf < 0.6); + } + + #[test] + fn test_security_notes_ssrf() { + let notes = generate_security_notes("169.254.169.254", 80, true); + assert!(notes.contains("SSRF")); + } + + #[test] + fn test_security_notes_internal_ip_uses_canonical_classifier() { + // RFC 1918 is 172.16.0.0/12 only: the old starts_with("172.") prefix + // wrongly flagged 172.15/172.32 and missed CGNAT (100.64.0.0/10). #1777. + assert!(generate_security_notes("172.16.0.1", 80, false).contains("internal/private")); + assert!(!generate_security_notes("172.15.0.1", 80, false).contains("internal/private")); + assert!(!generate_security_notes("172.32.0.1", 80, false).contains("internal/private")); + assert!(generate_security_notes("100.64.0.1", 80, false).contains("internal/private")); + assert!(generate_security_notes("10.0.0.1", 80, false).contains("internal/private")); + assert!(generate_security_notes("192.168.1.1", 80, false).contains("internal/private")); + assert!(generate_security_notes("127.0.0.1", 80, false).contains("internal/private")); + assert!(generate_security_notes("localhost", 80, false).contains("internal/private")); + assert!(!generate_security_notes("8.8.8.8", 80, false).contains("internal/private")); + // Hostnames that merely start with a private-range prefix must NOT be + // flagged: classification parses an IP literal, not a string prefix. #1824. + assert!(!generate_security_notes("10.example.com", 80, false).contains("internal/private")); + assert!( + !generate_security_notes("172.example.com", 80, false).contains("internal/private") + ); + // IPv6 ULA (fc00::/7, RFC 4193) is internal/private. + assert!(generate_security_notes("fd00::1", 80, false).contains("internal/private")); + } + + #[test] + fn test_generate_proposals_empty() { + let proposals = generate_proposals(&[]); + assert!(proposals.is_empty()); + } + + #[test] + fn test_generate_proposals_basic() { + let summaries = vec![DenialSummary { + sandbox_id: "test".to_string(), + host: "api.example.com".to_string(), + port: 443, + binary: "/usr/bin/curl".to_string(), + ancestors: vec![], + deny_reason: "no matching policy".to_string(), + first_seen_ms: 1000, + last_seen_ms: 2000, + count: 5, + suppressed_count: 0, + total_count: 5, + sample_cmdlines: vec![], + binary_sha256: String::new(), + persistent: false, + denial_stage: "connect".to_string(), + l7_request_samples: vec![], + l7_inspection_active: false, + }]; + + let proposals = generate_proposals(&summaries); + assert_eq!(proposals.len(), 1); + assert_eq!(proposals[0].rule_name, "allow_api_example_com_443"); + assert!(proposals[0].proposed_rule.is_some()); + + let rule = proposals[0].proposed_rule.as_ref().unwrap(); + assert_eq!(rule.endpoints.len(), 1); + assert_eq!(rule.endpoints[0].host, "api.example.com"); + assert_eq!(rule.endpoints[0].port, 443); + assert_eq!(rule.binaries.len(), 1); + assert_eq!(rule.binaries[0].path, "/usr/bin/curl"); + // No L7 fields when no samples provided. + assert!(rule.endpoints[0].protocol.is_empty()); + assert!(rule.endpoints[0].rules.is_empty()); + + // Proposals never include allowed_ips (two-step approval flow). + assert!(rule.endpoints[0].allowed_ips.is_empty()); + } + + #[test] + fn test_generate_proposals_with_l7_samples() { + use openshell_core::proto::L7RequestSample; + + let summaries = vec![DenialSummary { + sandbox_id: "test".to_string(), + host: "icanhazdadjoke.com".to_string(), + port: 443, + binary: "/usr/bin/python3".to_string(), + ancestors: vec![], + deny_reason: "l7 deny".to_string(), + first_seen_ms: 1000, + last_seen_ms: 2000, + count: 3, + suppressed_count: 0, + total_count: 3, + sample_cmdlines: vec![], + binary_sha256: String::new(), + persistent: false, + denial_stage: "l7_deny".to_string(), + l7_request_samples: vec![ + L7RequestSample { + method: "GET".to_string(), + path: "/".to_string(), + decision: "deny".to_string(), + count: 2, + }, + L7RequestSample { + method: "GET".to_string(), + path: "/j/abc123def456abcd0099".to_string(), + decision: "deny".to_string(), + count: 1, + }, + ], + l7_inspection_active: true, + }]; + + let proposals = generate_proposals(&summaries); + assert_eq!(proposals.len(), 1); + + let rule = proposals[0].proposed_rule.as_ref().unwrap(); + let ep = &rule.endpoints[0]; + + // L7 fields should be set. + assert_eq!(ep.protocol, "rest"); + // tls field is no longer set (auto-detection handles it). + assert!(ep.tls.is_empty()); + assert_eq!(ep.enforcement, "enforce"); + + // Should have L7 rules. + assert!(!ep.rules.is_empty()); + + let paths: Vec<&str> = ep + .rules + .iter() + .filter_map(|r| r.allow.as_ref()) + .map(|a| a.path.as_str()) + .collect(); + assert!(paths.contains(&"/")); + // The /j/abc123def456 path should be generalised to /j/* + assert!(paths.contains(&"/j/*")); + + // Rationale should mention L7. + assert!(proposals[0].rationale.contains("L7")); + } + + // -- is_always_blocked_destination tests ------------------------------------ + + #[test] + fn test_always_blocked_destination_loopback_ip() { + assert!(is_always_blocked_destination("127.0.0.1")); + } + + #[test] + fn test_always_blocked_destination_link_local_ip() { + assert!(is_always_blocked_destination("169.254.169.254")); + } + + #[test] + fn test_always_blocked_destination_unspecified_ip() { + assert!(is_always_blocked_destination("0.0.0.0")); + } + + #[test] + fn test_always_blocked_destination_localhost_hostname() { + assert!(is_always_blocked_destination("localhost")); + assert!(is_always_blocked_destination("LOCALHOST")); + } + + #[test] + fn test_always_blocked_destination_known_metadata_hostname() { + assert!(is_always_blocked_destination("metadata.google.internal")); + assert!(is_always_blocked_destination("METADATA.GOOGLE.INTERNAL.")); + } + + #[test] + fn test_always_blocked_destination_allows_rfc1918() { + assert!(!is_always_blocked_destination("10.0.5.20")); + assert!(!is_always_blocked_destination("192.168.1.1")); + } + + #[test] + fn test_always_blocked_destination_allows_public_hostname() { + assert!(!is_always_blocked_destination("api.github.com")); + } + + // -- generate_proposals: always-blocked filtering tests -------------------- + + #[test] + fn test_generate_proposals_skips_loopback_destination() { + let summaries = vec![DenialSummary { + host: "127.0.0.1".to_string(), + port: 80, + binary: "/usr/bin/curl".to_string(), + count: 5, + first_seen_ms: 1000, + last_seen_ms: 2000, + denial_stage: "ssrf".to_string(), + ..Default::default() + }]; + + let proposals = generate_proposals(&summaries); + assert!( + proposals.is_empty(), + "should skip proposals for loopback: {proposals:?}" + ); + } + + #[test] + fn test_generate_proposals_skips_link_local_destination() { + let summaries = vec![DenialSummary { + host: "169.254.169.254".to_string(), + port: 80, + binary: "/usr/bin/curl".to_string(), + count: 5, + first_seen_ms: 1000, + last_seen_ms: 2000, + denial_stage: "ssrf".to_string(), + ..Default::default() + }]; + + let proposals = generate_proposals(&summaries); + assert!( + proposals.is_empty(), + "should skip proposals for link-local: {proposals:?}" + ); + } + + #[test] + fn test_generate_proposals_skips_known_metadata_hostname() { + let summaries = vec![DenialSummary { + host: "metadata.google.internal".to_string(), + port: 80, + binary: "/usr/bin/curl".to_string(), + count: 5, + first_seen_ms: 1000, + last_seen_ms: 2000, + denial_stage: "ssrf".to_string(), + ..Default::default() + }]; + + let proposals = generate_proposals(&summaries); + assert!( + proposals.is_empty(), + "should skip proposals for metadata hostname: {proposals:?}" + ); + } + + #[test] + fn test_generate_proposals_skips_localhost_hostname() { + let summaries = vec![DenialSummary { + host: "localhost".to_string(), + port: 8080, + binary: "/usr/bin/curl".to_string(), + count: 3, + first_seen_ms: 1000, + last_seen_ms: 2000, + denial_stage: "ssrf".to_string(), + ..Default::default() + }]; + + let proposals = generate_proposals(&summaries); + assert!( + proposals.is_empty(), + "should skip proposals for localhost: {proposals:?}" + ); + } + + #[test] + fn test_generate_proposals_keeps_public_destination() { + let summaries = vec![DenialSummary { + host: "api.github.com".to_string(), + port: 443, + binary: "/usr/bin/curl".to_string(), + count: 5, + first_seen_ms: 1000, + last_seen_ms: 2000, + denial_stage: "connect".to_string(), + ..Default::default() + }]; + + let proposals = generate_proposals(&summaries); + assert_eq!(proposals.len(), 1, "should keep proposals for public host"); + } + + #[test] + fn test_generalise_path() { + // Exact path preserved. + assert_eq!( + generalise_path("/api/breeds/image/random"), + "/api/breeds/image/random" + ); + + // Numeric ID replaced. + assert_eq!(generalise_path("/posts/42"), "/posts/*"); + + // UUID-ish replaced. + assert_eq!( + generalise_path("/chunks/550e8400-e29b-41d4-a716-446655440000"), + "/chunks/*" + ); + + // Query string stripped. + assert_eq!(generalise_path("/json/?fields=status,country"), "/json/"); + + // Short path preserved. + assert_eq!(generalise_path("/"), "/"); + } + + #[test] + fn test_looks_like_id() { + assert!(looks_like_id("42")); + assert!(looks_like_id("550e8400-e29b-41d4-a716-446655440000")); + assert!(looks_like_id("abc123def456abcd")); + assert!(!looks_like_id("random")); + assert!(!looks_like_id("get")); + assert!(!looks_like_id("")); + assert!(!looks_like_id("v1")); + } +} diff --git a/deny.toml b/deny.toml index 032cfcce3b..410a92e3b3 100644 --- a/deny.toml +++ b/deny.toml @@ -57,7 +57,7 @@ registries = [] # -- Bans ---------------------------------------------------------------------- [bans] deny = [ - { name = "ring", wrappers = ["rustls", "rustls-webpki"], reason = "Use the AWS-LC crypto backend instead. These wrappers are required only by z3-sys's prebuilt archive downloader." }, + { name = "ring", wrappers = ["rustls", "rustls-webpki", "x509-parser"], reason = "Use the AWS-LC crypto backend instead. These wrappers are required only by rcgen's CA certificate parser and z3-sys's prebuilt archive downloader." }, ] multiple-versions = "warn" wildcards = "allow" diff --git a/deploy/docker/Dockerfile.driver-vm-macos b/deploy/docker/Dockerfile.driver-vm-macos index 58317a52d8..438700eb09 100644 --- a/deploy/docker/Dockerfile.driver-vm-macos +++ b/deploy/docker/Dockerfile.driver-vm-macos @@ -8,7 +8,7 @@ # # openshell-driver-vm loads libkrun/libkrunfw at runtime via dlopen, so it # does NOT need Hypervisor.framework headers at build time. Pre-compressed -# runtime artifacts (libkrun, libkrunfw, gvproxy, bundled supervisor) are injected via +# runtime artifacts (libkrun, libkrunfw, bundled sandbox/supervisor) are injected via # the vm-runtime-compressed build context and embedded into the binary via # include_bytes!(). # diff --git a/deploy/docker/Dockerfile.sandbox b/deploy/docker/Dockerfile.sandbox new file mode 100644 index 0000000000..003c31bc74 --- /dev/null +++ b/deploy/docker/Dockerfile.sandbox @@ -0,0 +1,22 @@ +# syntax=docker/dockerfile:1.4 + +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# The static musl sandbox binary is staged under +# deploy/docker/.build/prebuilt-binaries//. Runtime-specific bootstrap +# tools belong to their compute drivers rather than this portable image. + +FROM scratch AS sandbox + +ARG TARGETARCH + +# Keep the binary root-owned for image-volume mounts and executable by the +# sandbox runtime's non-root UID. +COPY --chmod=0555 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /openshell-sandbox + +# Drivers may override this identity to match the admitted workload. Keep the +# standalone image non-root by default. +USER 65532:65532 + +ENTRYPOINT ["/openshell-sandbox"] diff --git a/deploy/docker/Dockerfile.supervisor b/deploy/docker/Dockerfile.supervisor index 7e7342ccb3..c25b5a90d6 100644 --- a/deploy/docker/Dockerfile.supervisor +++ b/deploy/docker/Dockerfile.supervisor @@ -3,21 +3,17 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# The static sandbox binary is staged at: -# deploy/docker/.build/prebuilt-binaries//openshell-sandbox -# -# Alpine supplies nftables and iptables for pod-namespace egress enforcement. +# The dynamically linked GNU supervisor binary is staged under +# deploy/docker/.build/prebuilt-binaries//. -FROM alpine:3.22 AS supervisor +FROM debian:bookworm-slim AS supervisor ARG TARGETARCH -# Refresh base packages too: adding firewall tools does not upgrade OpenSSL. -RUN apk upgrade --no-cache \ - && apk add --no-cache nftables iptables iptables-legacy +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates \ + && rm -rf /var/lib/apt/lists/* -# Keep the binary root-owned for Podman image-volume mounts and executable by -# the Kubernetes network sidecar's non-root proxy UID. -COPY --chmod=0555 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /openshell-sandbox +COPY --chmod=0555 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-supervisor /openshell-supervisor -ENTRYPOINT ["/openshell-sandbox"] +ENTRYPOINT ["/openshell-supervisor"] diff --git a/deploy/docker/gateway.toml b/deploy/docker/gateway.toml index 9fbd574035..1372bfc3fe 100644 --- a/deploy/docker/gateway.toml +++ b/deploy/docker/gateway.toml @@ -36,8 +36,9 @@ disable_tls = true [openshell.drivers.docker] # Default image pulled for `openshell sandbox create` without --from. default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" -# Supervisor image from which the openshell-sandbox binary is extracted on -# first start. The binary is cached to XDG_DATA_HOME and reused on restart. +# Sandbox runtime image from which the openshell-sandbox binary is extracted. +sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:latest" +# Image containing the external supervisor process. supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest" # Only pull images that are not already cached locally. image_pull_policy = "if_not_present" diff --git a/deploy/helm/openshell/Chart.yaml b/deploy/helm/openshell/Chart.yaml index 06608adb37..7b3a5c60d5 100644 --- a/deploy/helm/openshell/Chart.yaml +++ b/deploy/helm/openshell/Chart.yaml @@ -8,6 +8,6 @@ description: runtime environment for autonomous agents type: application # version and appVersion are patched to the release semver by CI before helm package. # appVersion doubles as the default image tag (image.tag defaults to appVersion when -# empty), so a released chart automatically pulls the matching gateway and supervisor images. +# empty), so a released chart automatically pulls matching gateway, sandbox, and supervisor images. version: 0.0.0 appVersion: "0.0.0" diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 3c32e24b67..aabcf426e6 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -35,7 +35,8 @@ where Helm cannot discover cluster APIs. ## Install on Kubernetes ```shell -helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true ``` ## Install on OpenShift @@ -43,14 +44,12 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version -n openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set server.disableTls=true \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null @@ -63,7 +62,7 @@ On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the | Tag | Source | Notes | | --- | --- | --- | -| `` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway and supervisor image versions. Recommended for production. | +| `` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway, sandbox, and supervisor image versions. Recommended for production. | | `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. | | `0.0.0-dev.` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. | @@ -110,6 +109,7 @@ Then install the chart pointing at that Secret: ```bash helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ -n openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workload.kind=deployment \ --set server.externalDbSecret=my-pg-credentials ``` @@ -231,6 +231,9 @@ discovery endpoint or its TLS CA. | probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. | | replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. | | resources | object | `{}` | Gateway pod resource requests and limits. | +| sandboxRuntime.image.pullPolicy | string | `""` | Sandbox runtime image pull policy. Defaults to the gateway image pull policy when empty. | +| sandboxRuntime.image.repository | string | `"ghcr.io/nvidia/openshell/sandbox"` | Sandbox runtime image repository. Changing it uses the effective gateway image tag unless tag is also set. | +| sandboxRuntime.image.tag | string | `""` | Sandbox runtime image tag override. Empty uses the version pinned into the gateway unless repository is changed. | | sandboxServiceAccount.annotations | object | `{}` | Annotations to add to the generated sandbox service account. | | sandboxServiceAccount.create | bool | `true` | Create a service account for sandbox pods. | | sandboxServiceAccount.name | string | `""` | Existing service account name for sandbox pods when sandboxServiceAccount.create is false. | @@ -238,7 +241,6 @@ discovery endpoint or its TLS CA. | securityContext.capabilities.drop | list | `["ALL"]` | Linux capabilities dropped from the gateway container. | | securityContext.runAsNonRoot | bool | `true` | Require the gateway container to run as a non-root user. | | securityContext.runAsUser | int | `1000` | UID assigned to the gateway container. | -| server.appArmorProfile | string | `"Unconfined"` | Kubernetes AppArmor profile requested for sandbox agent containers. Default Unconfined avoids runtime/default AppArmor blocking the supervisor's network namespace mount setup on AppArmor-enabled nodes. Set to "" to omit the field, "RuntimeDefault" to force the runtime default profile, or "Localhost/profile-name" for an operator-managed localhost profile. | | server.auth.allowUnauthenticatedUsers | bool | `false` | UNSAFE: accept unauthenticated CLI/user requests as a local developer principal. Intended only for trusted local Skaffold/k3d development or a fully trusted fronting proxy. Leave false for shared or production clusters. | | server.credentialDrivers.kubernetesSecrets.allowReferenceNamespace | bool | `false` | Deprecated compatibility field. Credential storage no longer supports user-authored namespace references. | | server.credentialDrivers.kubernetesSecrets.enabled | bool | `false` | Enable the in-tree Kubernetes Secret credential driver. WARNING: The RBAC Role grants read/write access to ALL Secrets in the configured namespace. Use a dedicated namespace to limit blast radius. | @@ -309,10 +311,8 @@ discovery endpoint or its TLS CA. | supervisor.image.pullPolicy | string | `nil` | Sandbox supervisor pull policy. Leave unset to use the Kubernetes image default. Prefer always, if_not_present, or never; the chart also accepts legacy Kubernetes spellings Always, IfNotPresent, and Never. | | supervisor.image.repository | string | `"ghcr.io/nvidia/openshell/supervisor"` | Supervisor image repository. Changing it uses the effective gateway image tag unless tag is also set. | | supervisor.image.tag | string | `""` | Supervisor image tag override. Empty uses the version pinned into the gateway unless repository is changed. | -| supervisor.sidecar.processBinaryAwareNetworkPolicy | bool | `true` | Keep process/binary-aware network policy enabled in sidecar topology. When false, the network sidecar runs as proxyUid, drops the extra /proc inspection capabilities, and enforces endpoint/L7 policy without matching policy.binaries. | -| supervisor.sidecar.proxyUid | int | `1337` | UID for relaxed long-running network sidecars in sidecar topology. Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants the required /proc inspection capabilities into the effective set. The network init container installs nftables rules that exempt the effective sidecar UID. | -| supervisor.sideloadMethod | string | `""` | How the supervisor binary is delivered into sandbox pods. Empty (default) = auto-detect from cluster version: K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) K8s < v1.35 -> "init-container" (copies via init container + emptyDir) On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, set this to "image-volume" explicitly. | -| supervisor.topology | string | `"combined"` | Supervisor pod topology for Kubernetes sandboxes. "combined" runs the current single supervisor container in the agent pod. "sidecar" runs network enforcement in a dedicated sidecar and the process supervisor as a low-capability wrapper in the agent container. | +| supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. | +| supervisor.sandboxRuntime.networkPolicyEnforced | bool | `false` | Required operator acknowledgement that the cluster CNI enforces NetworkPolicy. | | tolerations | list | `[]` | Tolerations for the gateway pod. | | upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. | | upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 6e6f2012fc..0adac0d070 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -35,7 +35,8 @@ where Helm cannot discover cluster APIs. ## Install on Kubernetes ```shell -helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true ``` ## Install on OpenShift @@ -43,14 +44,12 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version -n openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set server.disableTls=true \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null @@ -63,7 +62,7 @@ On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the | Tag | Source | Notes | | --- | --- | --- | -| `` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway and supervisor image versions. Recommended for production. | +| `` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway, sandbox, and supervisor image versions. Recommended for production. | | `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. | | `0.0.0-dev.` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. | @@ -110,6 +109,7 @@ Then install the chart pointing at that Secret: ```bash helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ -n openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workload.kind=deployment \ --set server.externalDbSecret=my-pg-credentials ``` diff --git a/deploy/helm/openshell/ci/values-corporate-proxy-e2e.yaml b/deploy/helm/openshell/ci/values-corporate-proxy-e2e.yaml index 35532440b2..70fdec8d7b 100644 --- a/deploy/helm/openshell/ci/values-corporate-proxy-e2e.yaml +++ b/deploy/helm/openshell/ci/values-corporate-proxy-e2e.yaml @@ -3,9 +3,6 @@ # The Kubernetes corporate-proxy e2e wrapper supplies the generated proxy URL # and creates `openshell-e2e-proxy-auth` before Helm installs the gateway. -supervisor: - topology: sidecar - upstreamProxy: authSecret: name: openshell-e2e-proxy-auth diff --git a/deploy/helm/openshell/ci/values-keycloak.yaml b/deploy/helm/openshell/ci/values-keycloak.yaml index cc6ca658bf..74e38b6043 100644 --- a/deploy/helm/openshell/ci/values-keycloak.yaml +++ b/deploy/helm/openshell/ci/values-keycloak.yaml @@ -8,7 +8,8 @@ # # Then layer this file on top of values.yaml when deploying: # helm upgrade --install openshell . \ -# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml +# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml \ +# --set supervisor.sandboxRuntime.networkPolicyEnforced=true # # Or add this file to skaffold.yaml valuesFiles for iterative dev. # diff --git a/deploy/helm/openshell/ci/values-openshift-scc.yaml b/deploy/helm/openshell/ci/values-openshift-scc.yaml index b7f37be6e0..8f1a8d07d6 100644 --- a/deploy/helm/openshell/ci/values-openshift-scc.yaml +++ b/deploy/helm/openshell/ci/values-openshift-scc.yaml @@ -4,7 +4,8 @@ # OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and # fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace- # assigned UID/GID range. Layer after values.yaml: -# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml +# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml \ +# --set supervisor.sandboxRuntime.networkPolicyEnforced=true # # The e2e Kubernetes harness applies this automatically when it detects an # OpenShift cluster (route.openshift.io API present). diff --git a/deploy/helm/openshell/ci/values-sidecar-kata.yaml b/deploy/helm/openshell/ci/values-sidecar-kata.yaml deleted file mode 100644 index 2e23a1009e..0000000000 --- a/deploy/helm/openshell/ci/values-sidecar-kata.yaml +++ /dev/null @@ -1,24 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# CI/dev overlay for exercising the Kubernetes supervisor sidecar topology under -# a Kata RuntimeClass. -# -# Use with e2e/with-kube-gateway.sh by setting: -# OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-sidecar-kata.yaml -# The e2e wrapper supplies the image repository and tag through OPENSHELL_REGISTRY -# and IMAGE_TAG for existing-cluster runs. - -supervisor: - # Use the sidecar topology under Kata so network enforcement runs in the - # sidecar and the sandbox agent container stays low-privilege. - topology: sidecar - sidecar: - # Keep strict process/binary-aware network policy enabled for the Kata - # validation path. Set this false only when intentionally validating the - # documented endpoint/L7-only downgrade mode. - processBinaryAwareNetworkPolicy: true - -# Kata validation clusters normally install this RuntimeClass. -server: - defaultRuntimeClassName: kata-qemu diff --git a/deploy/helm/openshell/ci/values-sidecar.yaml b/deploy/helm/openshell/ci/values-sidecar.yaml deleted file mode 100644 index ba8dc50ef1..0000000000 --- a/deploy/helm/openshell/ci/values-sidecar.yaml +++ /dev/null @@ -1,18 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# CI/dev overlay for exercising the Kubernetes supervisor sidecar topology. -# -# Merge after values.yaml and ci/values-skaffold.yaml: -# helm install ... -f values.yaml -f ci/values-skaffold.yaml -f ci/values-sidecar.yaml -# -# Or set: -# OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-sidecar.yaml -# before running `mise run e2e:kubernetes`. -supervisor: - topology: sidecar - sidecar: - # The strict sidecar default requires cross-container /proc identity access. - # CI/dev e2e uses the explicit downgraded mode so endpoint and L7 policy - # coverage remains runnable on local k3d while that path is hardened. - processBinaryAwareNetworkPolicy: false diff --git a/deploy/helm/openshell/ci/values-skaffold.yaml b/deploy/helm/openshell/ci/values-skaffold.yaml index 706df3eca5..6f72265039 100644 --- a/deploy/helm/openshell/ci/values-skaffold.yaml +++ b/deploy/helm/openshell/ci/values-skaffold.yaml @@ -14,3 +14,7 @@ server: supervisor: image: pullPolicy: if_not_present + # The local k3s cluster created by `mise run helm:k3s:create` enables its + # built-in NetworkPolicy controller for sandbox namespaces. + sandboxRuntime: + networkPolicyEnforced: true diff --git a/deploy/helm/openshell/skaffold.yaml b/deploy/helm/openshell/skaffold.yaml index ce32c72132..63037b075b 100644 --- a/deploy/helm/openshell/skaffold.yaml +++ b/deploy/helm/openshell/skaffold.yaml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Local dev: builds gateway + supervisor images via tasks/scripts/docker-build-image.sh, +# Local dev: builds gateway, sandbox, and supervisor images via tasks/scripts/docker-build-image.sh, # which first stages Rust binaries natively on the host (using cargo / cargo-zigbuild # when cross-compiling) and then builds the image from the prebuilt binary. This # mirrors CI and is faster than compiling inside Docker on every rebuild because @@ -56,6 +56,23 @@ build: - deploy/docker/Dockerfile.supervisor - tasks/scripts/docker-build-image.sh - tasks/scripts/stage-prebuilt-binaries.sh + - image: openshell/sandbox + context: ../../.. + custom: + buildCommand: | + CONTAINER_ENGINE_TARGET=local-k8s-cluster \ + IMAGE_NAME="${IMAGE%:*}" \ + IMAGE_TAG="${IMAGE##*:}" \ + tasks/scripts/docker-build-image.sh sandbox + dependencies: + paths: + - Cargo.toml + - Cargo.lock + - crates/** + - proto/** + - deploy/docker/Dockerfile.sandbox + - tasks/scripts/docker-build-image.sh + - tasks/scripts/stage-prebuilt-binaries.sh deploy: helm: releases: @@ -119,8 +136,6 @@ deploy: # To enable SPIFFE/SPIRE provider token grants (requires the # spire-crds and spire releases above): #- ci/values-spire.yaml - # To exercise the Kubernetes supervisor sidecar topology: - #- ci/values-sidecar.yaml # To test multi-replica external PostgreSQL behavior: #- ci/values-high-availability.yaml setValueTemplates: @@ -128,21 +143,9 @@ deploy: image.tag: '{{.IMAGE_TAG_openshell_gateway}}' supervisor.image.repository: '{{.IMAGE_REPO_openshell_supervisor}}' supervisor.image.tag: '{{.IMAGE_TAG_openshell_supervisor}}' + sandboxRuntime.image.repository: '{{.IMAGE_REPO_openshell_sandbox}}' + sandboxRuntime.image.tag: '{{.IMAGE_TAG_openshell_sandbox}}' profiles: - - name: sidecar - patches: - - op: add - path: /deploy/helm/releases/0/valuesFiles/- - value: ci/values-sidecar.yaml - - name: sidecar-mtls - patches: - - op: add - path: /deploy/helm/releases/0/valuesFiles/- - value: ci/values-sidecar.yaml - - op: add - path: /deploy/helm/releases/0/setValues - value: - server.disableTls: "false" - name: credential-driver-kubernetes-secrets patches: - op: add diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index a35dc96db5..8ce4c73656 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -91,6 +91,25 @@ so a released chart automatically pulls the matching image without extra overrid {{- printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) }} {{- end }} +{{/* Official sandbox runtime repository used by the gateway's built-in default. */}} +{{- define "openshell.defaultSandboxRuntimeRepository" -}} +ghcr.io/nvidia/openshell/sandbox +{{- end }} + +{{/* Whether Helm must propagate a sandbox runtime image override. */}} +{{- define "openshell.sandboxRuntimeImageOverrideEnabled" -}} +{{- $defaultRepository := include "openshell.defaultSandboxRuntimeRepository" . -}} +{{- $repository := .Values.sandboxRuntime.image.repository | default $defaultRepository -}} +{{- if or (ne $repository $defaultRepository) .Values.sandboxRuntime.image.tag -}}true{{- end -}} +{{- end }} + +{{/* Sandbox runtime image override. */}} +{{- define "openshell.sandboxRuntimeImage" -}} +{{- $repository := .Values.sandboxRuntime.image.repository | default (include "openshell.defaultSandboxRuntimeRepository" .) -}} +{{- $tag := .Values.sandboxRuntime.image.tag | default .Values.image.tag | default .Chart.AppVersion -}} +{{- printf "%s:%s" $repository $tag }} +{{- end }} + {{/* Official supervisor repository used by the gateway's built-in default. */}} {{- define "openshell.defaultSupervisorRepository" -}} ghcr.io/nvidia/openshell/supervisor @@ -195,25 +214,6 @@ the in-cluster Service DNS, release namespace, service port, and disableTls flag — so the default value works for any release name or namespace without override. */}} -{{/* -Supervisor sideload method. When supervisor.sideloadMethod is set, use it -verbatim. Otherwise auto-detect from the cluster version: the ImageVolume -feature gate is enabled by default starting in K8s v1.35 (GA in v1.36). -Clusters on v1.33-v1.34 can opt in by setting sideloadMethod explicitly -after enabling the feature gate. -*/}} -{{- define "openshell.supervisorSideloadMethod" -}} -{{- if .Values.supervisor.sideloadMethod -}} -{{- .Values.supervisor.sideloadMethod -}} -{{- else -}} -{{- if semverCompare ">=1.35-0" .Capabilities.KubeVersion.Version -}} -image-volume -{{- else -}} -init-container -{{- end -}} -{{- end -}} -{{- end }} - {{- define "openshell.grpcEndpoint" -}} {{- if .Values.server.grpcEndpoint -}} {{- .Values.server.grpcEndpoint -}} diff --git a/deploy/helm/openshell/templates/clusterrole.yaml b/deploy/helm/openshell/templates/clusterrole.yaml index 5d328a1261..02ace73a12 100644 --- a/deploy/helm/openshell/templates/clusterrole.yaml +++ b/deploy/helm/openshell/templates/clusterrole.yaml @@ -71,7 +71,12 @@ rules: resources: - pods verbs: + - create + - delete - get + - list + - patch + - watch {{- end }} {{- $copiedSecretNames := list }} {{- if and (ne $workspaceMode "shared") (not .Values.server.disableTls) }} @@ -84,6 +89,17 @@ rules: {{- end }} {{- end }} {{- end }} + {{- if ne $workspaceMode "shared" }} + - apiGroups: [""] + resources: ["services"] + verbs: ["create", "get"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["create"] + - apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + verbs: ["create", "get"] + {{- end }} {{- $copiedSecretNames = uniq $copiedSecretNames }} {{- if $copiedSecretNames }} # Copy only explicitly configured TLS and image-pull Secrets into workspace diff --git a/deploy/helm/openshell/templates/gateway-config.yaml b/deploy/helm/openshell/templates/gateway-config.yaml index 2c2b8216ee..5265f9b46b 100644 --- a/deploy/helm/openshell/templates/gateway-config.yaml +++ b/deploy/helm/openshell/templates/gateway-config.yaml @@ -138,6 +138,9 @@ data: [openshell.drivers.kubernetes] namespace = {{ include "openshell.sandboxNamespace" . | quote }} default_image = {{ .Values.server.sandboxImage | quote }} + {{- if include "openshell.sandboxRuntimeImageOverrideEnabled" . }} + sandbox_runtime_image = {{ include "openshell.sandboxRuntimeImage" . | quote }} + {{- end }} {{- if include "openshell.supervisorImageOverrideEnabled" . }} supervisor_image = {{ include "openshell.supervisorImage" . | quote }} {{- end }} @@ -160,8 +163,6 @@ data: {{- if .Values.server.drivers.kubernetes.operatorNamespaceFile }} operator_namespace_file = {{ .Values.server.drivers.kubernetes.operatorNamespaceFile | quote }} {{- end }} - supervisor_sideload_method = {{ include "openshell.supervisorSideloadMethod" . | quote }} - topology = {{ .Values.supervisor.topology | default "combined" | quote }} sa_token_ttl_secs = {{ .Values.server.sandboxJwt.k8sSaTokenTtlSecs | default 3600 }} {{- if .Values.upstreamProxy.url }} https_proxy = {{ .Values.upstreamProxy.url | quote }} @@ -205,21 +206,21 @@ data: {{- if .Values.server.defaultRuntimeClassName }} default_runtime_class_name = {{ .Values.server.defaultRuntimeClassName | quote }} {{- end }} - {{- if .Values.server.appArmorProfile }} - app_armor_profile = {{ .Values.server.appArmorProfile | quote }} - {{- end }} {{- if .Values.supervisor.image.pullPolicy }} supervisor_image_pull_policy = {{ include "openshell.canonicalImagePullPolicy" .Values.supervisor.image.pullPolicy | quote }} {{- end }} + {{- if .Values.sandboxRuntime.image.pullPolicy }} + sandbox_runtime_image_pull_policy = {{ include "openshell.canonicalImagePullPolicy" .Values.sandboxRuntime.image.pullPolicy | quote }} + {{- end }} [openshell.drivers.kubernetes.managed_ssh_ingress] enabled = {{ .Values.networkPolicy.enabled }} gateway_namespace = {{ .Release.Namespace | quote }} gateway_pod_selector = { "app.kubernetes.io/name" = {{ include "openshell.name" . | quote }}, "app.kubernetes.io/instance" = {{ .Release.Name | quote }} } - [openshell.drivers.kubernetes.sidecar] - proxy_uid = {{ .Values.supervisor.sidecar.proxyUid | default 1337 }} - process_binary_aware_network_policy = {{ .Values.supervisor.sidecar.processBinaryAwareNetworkPolicy }} + [openshell.drivers.kubernetes.sandbox_runtime] + network_policy_enforced = {{ .Values.supervisor.sandboxRuntime.networkPolicyEnforced }} + boundary_port = {{ .Values.supervisor.sandboxRuntime.boundaryPort | default 5500 }} {{- if not $credentialDrivers }} diff --git a/deploy/helm/openshell/templates/network-policy-ack.yaml b/deploy/helm/openshell/templates/network-policy-ack.yaml new file mode 100644 index 0000000000..780d4314c9 --- /dev/null +++ b/deploy/helm/openshell/templates/network-policy-ack.yaml @@ -0,0 +1,5 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +{{- if not .Values.supervisor.sandboxRuntime.networkPolicyEnforced }} +{{- fail "supervisor.sandboxRuntime.networkPolicyEnforced must be true after you verify that the cluster CNI enforces ingress and egress NetworkPolicy in every sandbox namespace" }} +{{- end }} diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index dfd6423615..dac863b805 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -33,14 +33,28 @@ rules: - watch # Per-sandbox identity: TokenReview authenticates the projected token from # the configured sandbox service account, then the gateway resolves the - # returned pod name and UID to the pod's `openshell.ai/sandbox-id` - # annotation. patch is intentionally NOT granted — the annotation is set - # once at pod create and must remain immutable for the lifetime of the - # sandbox. + # returned pod name and UID to its immutable sandbox labels. The driver also + # creates and deletes the directly managed supervisor Pod. PATCH removes the + # driver-owned bootstrap scheduling gate after immutable bootstrap material + # has been created. - apiGroups: - "" resources: - pods verbs: + - create + - delete - get + - list + - patch + - watch + - apiGroups: [""] + resources: ["services"] + verbs: ["create", "get"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["create"] + - apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + verbs: ["create", "get"] {{- end }} diff --git a/deploy/helm/openshell/tests/clusterrole_test.yaml b/deploy/helm/openshell/tests/clusterrole_test.yaml index 8f16aef584..ec2f42082c 100644 --- a/deploy/helm/openshell/tests/clusterrole_test.yaml +++ b/deploy/helm/openshell/tests/clusterrole_test.yaml @@ -26,17 +26,23 @@ tests: resources: ["networkpolicies"] verbs: ["get", "create", "patch", "update"] - - it: omits managed NetworkPolicy permissions when isolation is disabled + - it: preserves sandbox-runtime fence permissions when gateway isolation is disabled set: server.drivers.kubernetes.workspaceMode: managed networkPolicy.enabled: false asserts: + - contains: + path: rules + content: + apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + verbs: ["create", "get"] - notContains: path: rules content: apiGroups: ["networking.k8s.io"] resources: ["networkpolicies"] - any: true + verbs: ["get", "create", "patch", "update"] - it: grants broad secret access when credential driver is enabled (operator) set: @@ -63,17 +69,23 @@ tests: resourceNames: ["custom-client-tls"] verbs: ["get", "patch"] - - it: omits operator secret access when TLS and credential storage are disabled + - it: limits operator bootstrap secret access to create when TLS and credential storage are disabled set: server.drivers.kubernetes.workspaceMode: operator server.disableTls: true asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["secrets"] + verbs: ["create"] - notContains: path: rules content: apiGroups: [""] resources: ["secrets"] - any: true + verbs: ["get", "create", "patch", "delete"] - it: restricts managed copies to TLS and configured image-pull secrets set: @@ -128,3 +140,49 @@ tests: apiGroups: [""] resources: ["secrets"] any: true + + - it: grants managed sandbox-runtime companion permissions + set: + server.drivers.kubernetes.workspaceMode: managed + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["pods"] + verbs: ["create", "delete", "get", "list", "patch", "watch"] + - contains: + path: rules + content: + apiGroups: [""] + resources: ["services"] + verbs: ["create", "get"] + - contains: + path: rules + content: + apiGroups: [""] + resources: ["secrets"] + verbs: ["create"] + + - it: grants operator sandbox-runtime companion permissions + set: + server.drivers.kubernetes.workspaceMode: operator + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["pods"] + verbs: ["create", "delete", "get", "list", "patch", "watch"] + - contains: + path: rules + content: + apiGroups: [""] + resources: ["services"] + verbs: ["create", "get"] + - contains: + path: rules + content: + apiGroups: [""] + resources: ["secrets"] + verbs: ["create"] diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 2cf7705b8e..93d9fe6cde 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -237,16 +237,6 @@ tests: path: data["gateway.toml"] pattern: '(?ms)\[openshell\.gateway\][^\[]*?enable_user_namespaces' - - it: renders combined supervisor topology by default under [openshell.drivers.kubernetes] - template: templates/gateway-config.yaml - asserts: - - matchRegex: - path: data["gateway.toml"] - pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?topology\s*=\s*"combined"' - - notMatchRegex: - path: data["gateway.toml"] - pattern: 'supervisor[_]topology\s*=' - - it: renders operator-owned upstream proxy settings under the Kubernetes driver template: templates/gateway-config.yaml set: @@ -283,64 +273,60 @@ tests: path: data["gateway.toml"] pattern: 'supervisor_image\s*=' - - it: renders a supervisor tag override with the official repository + - it: uses the gateway built-in sandbox runtime image by default template: templates/gateway-config.yaml - set: - supervisor.image.tag: 1.2.3 asserts: - - matchRegex: + - notMatchRegex: path: data["gateway.toml"] - pattern: 'supervisor_image\s*=\s*"ghcr\.io/nvidia/openshell/supervisor:1\.2\.3"' + pattern: 'sandbox_runtime_image\s*=' - - it: renders a supervisor repository override with the effective gateway tag + - it: renders independent sandbox runtime and supervisor image overrides template: templates/gateway-config.yaml set: - image.tag: gateway-build + sandboxRuntime.image.repository: registry.example.com/openshell/sandbox + sandboxRuntime.image.tag: sandbox-build + sandboxRuntime.image.pullPolicy: Always supervisor.image.repository: registry.example.com/openshell/supervisor + supervisor.image.tag: supervisor-build asserts: - matchRegex: path: data["gateway.toml"] - pattern: 'supervisor_image\s*=\s*"registry\.example\.com/openshell/supervisor:gateway-build"' - - - it: renders complete supervisor repository and tag overrides - template: templates/gateway-config.yaml - set: - supervisor.image.repository: registry.example.com/openshell/supervisor - supervisor.image.tag: supervisor-build - asserts: + pattern: 'sandbox_runtime_image\s*=\s*"registry\.example\.com/openshell/sandbox:sandbox-build"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'sandbox_runtime_image_pull_policy\s*=\s*"always"' - matchRegex: path: data["gateway.toml"] pattern: 'supervisor_image\s*=\s*"registry\.example\.com/openshell/supervisor:supervisor-build"' - - it: renders sidecar supervisor topology under [openshell.drivers.kubernetes] + - it: renders a supervisor tag override with the official repository template: templates/gateway-config.yaml set: - supervisor.topology: sidecar + supervisor.image.tag: 1.2.3 asserts: - matchRegex: path: data["gateway.toml"] - pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?topology\s*=\s*"sidecar"' - - notMatchRegex: - path: data["gateway.toml"] - pattern: 'supervisor[_]topology\s*=' + pattern: 'supervisor_image\s*=\s*"ghcr\.io/nvidia/openshell/supervisor:1\.2\.3"' - - it: renders proxy uid under [openshell.drivers.kubernetes.sidecar] + - it: renders a supervisor repository override with the effective gateway tag template: templates/gateway-config.yaml set: - supervisor.sidecar.proxyUid: 2200 + image.tag: gateway-build + supervisor.image.repository: registry.example.com/openshell/supervisor asserts: - matchRegex: path: data["gateway.toml"] - pattern: '(?ms)\[openshell\.drivers\.kubernetes\.sidecar\].*?proxy_uid\s*=\s*2200' + pattern: 'supervisor_image\s*=\s*"registry\.example\.com/openshell/supervisor:gateway-build"' - - it: renders process binary aware network policy under [openshell.drivers.kubernetes.sidecar] + - it: renders complete supervisor repository and tag overrides template: templates/gateway-config.yaml set: - supervisor.sidecar.processBinaryAwareNetworkPolicy: false + supervisor.image.repository: registry.example.com/openshell/supervisor + supervisor.image.tag: supervisor-build asserts: - matchRegex: path: data["gateway.toml"] - pattern: '(?ms)\[openshell\.drivers\.kubernetes\.sidecar\].*?process_binary_aware_network_policy\s*=\s*false' + pattern: 'supervisor_image\s*=\s*"registry\.example\.com/openshell/supervisor:supervisor-build"' - it: configures managed SSH isolation with the gateway peer template: templates/gateway-config.yaml @@ -360,22 +346,6 @@ tests: path: data["gateway.toml"] pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?image_pull_secrets\s*=\s*\["regcred", "backup-regcred"\]' - - it: renders the default sandbox AppArmor profile under [openshell.drivers.kubernetes] - template: templates/gateway-config.yaml - asserts: - - matchRegex: - path: data["gateway.toml"] - pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?app_armor_profile\s*=\s*"Unconfined"' - - - it: omits sandbox AppArmor profile when disabled - template: templates/gateway-config.yaml - set: - server.appArmorProfile: "" - asserts: - - notMatchRegex: - path: data["gateway.toml"] - pattern: 'app_armor_profile\s*=' - - it: does not reuse gateway image pull secrets for sandbox pods template: templates/gateway-config.yaml set: diff --git a/deploy/helm/openshell/tests/network_policy_ack_test.yaml b/deploy/helm/openshell/tests/network_policy_ack_test.yaml new file mode 100644 index 0000000000..12ca399e73 --- /dev/null +++ b/deploy/helm/openshell/tests/network_policy_ack_test.yaml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: NetworkPolicy enforcement acknowledgement +templates: + - templates/network-policy-ack.yaml +tests: + - it: rejects an install without operator acknowledgement + set: + supervisor.sandboxRuntime.networkPolicyEnforced: false + asserts: + - failedTemplate: + errorMessage: supervisor.sandboxRuntime.networkPolicyEnforced must be true after you verify that the cluster CNI enforces ingress and egress NetworkPolicy in every sandbox namespace + + - it: accepts an acknowledged CNI + set: + supervisor.sandboxRuntime.networkPolicyEnforced: true + asserts: + - hasDocuments: + count: 0 diff --git a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml index 337c991961..d02300e428 100644 --- a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml +++ b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml @@ -62,6 +62,12 @@ tests: - equal: path: metadata.namespace value: other-ns + - contains: + path: rules + content: + apiGroups: [""] + resources: ["services"] + verbs: ["create", "get"] - it: uses explicit sandboxNamespace for sandbox RoleBinding template: templates/rolebinding.yaml diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 432af61195..ad50cd4b50 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -26,9 +26,17 @@ image: # -- Gateway image tag. Defaults to the chart appVersion when empty. tag: "" -# Supervisor image for the openshell-sandbox binary injected into sandbox pods. -# The default repository and empty tag use the version-pinned image built into -# the gateway. Changing the repository or setting a tag enables a Helm override. +# Trusted workload-side runtime image. +sandboxRuntime: + image: + # -- Sandbox runtime image repository. Changing it uses the effective gateway image tag unless tag is also set. + repository: ghcr.io/nvidia/openshell/sandbox + # -- Sandbox runtime image pull policy. Defaults to the gateway image pull policy when empty. + pullPolicy: "" + # -- Sandbox runtime image tag override. Empty uses the version pinned into the gateway unless repository is changed. + tag: "" + +# Trusted control-side runtime image. supervisor: image: # -- Supervisor image repository. Changing it uses the effective gateway image tag unless tag is also set. @@ -39,30 +47,11 @@ supervisor: pullPolicy: null # -- Supervisor image tag override. Empty uses the version pinned into the gateway unless repository is changed. tag: "" - # -- How the supervisor binary is delivered into sandbox pods. - # Empty (default) = auto-detect from cluster version: - # K8s >= v1.35 -> "image-volume" (ImageVolume enabled by default; GA in v1.36) - # K8s < v1.35 -> "init-container" (copies via init container + emptyDir) - # On K8s v1.33-v1.34 with the ImageVolume feature gate manually enabled, - # set this to "image-volume" explicitly. - sideloadMethod: "" - # -- Supervisor pod topology for Kubernetes sandboxes. - # "combined" runs the current single supervisor container in the agent pod. - # "sidecar" runs network enforcement in a dedicated sidecar and the process - # supervisor as a low-capability wrapper in the agent container. - topology: "combined" - sidecar: - # -- UID for relaxed long-running network sidecars in sidecar topology. - # Strict process/binary-aware sidecars run as UID 0 so Kubernetes grants - # the required /proc inspection capabilities into the effective set. The - # network init container installs nftables rules that exempt the effective - # sidecar UID. - proxyUid: 1337 - # -- Keep process/binary-aware network policy enabled in sidecar topology. - # When false, the network sidecar runs as proxyUid, drops the extra /proc - # inspection capabilities, and enforces endpoint/L7 policy without matching - # policy.binaries. - processBinaryAwareNetworkPolicy: true + sandboxRuntime: + # -- Required operator acknowledgement that the cluster CNI enforces NetworkPolicy. + networkPolicyEnforced: false + # -- Workload boundary TLS listener port. + boundaryPort: 5500 # -- Operator-owned corporate forward proxy for policy-approved TLS egress # from Kubernetes sandboxes. The workload cannot select or override it. @@ -269,12 +258,6 @@ server: # Linux 5.12+. When enabled, container UID 0 maps to an unprivileged host # UID and capabilities become namespaced. enableUserNamespaces: false - # -- Kubernetes AppArmor profile requested for sandbox agent containers. - # Default Unconfined avoids runtime/default AppArmor blocking the supervisor's - # network namespace mount setup on AppArmor-enabled nodes. Set to "" to omit - # the field, "RuntimeDefault" to force the runtime default profile, or - # "Localhost/profile-name" for an operator-managed localhost profile. - appArmorProfile: "Unconfined" # Kubernetes compute driver settings. drivers: kubernetes: diff --git a/deploy/helm/test-split-ownership.sh b/deploy/helm/test-split-ownership.sh index 30fd7360d4..a7a865363b 100755 --- a/deploy/helm/test-split-ownership.sh +++ b/deploy/helm/test-split-ownership.sh @@ -11,6 +11,7 @@ trap 'rm -rf "${work_dir}"' EXIT helm template openshell "${repo_root}/deploy/helm/openshell" \ --namespace openshell \ --set agentSandbox.preflight.enabled=false \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workspaceResources.enabled=false \ >"${work_dir}/gateway.yaml" @@ -40,6 +41,7 @@ fi helm template openshell "${repo_root}/deploy/helm/openshell" \ --namespace openshell \ --set agentSandbox.preflight.enabled=false \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set-json workspaceResources=null \ >"${work_dir}/legacy-reuse-values.yaml" diff --git a/deploy/rpm/CONFIGURATION.md b/deploy/rpm/CONFIGURATION.md index e5b16d00d0..af2e97a94f 100644 --- a/deploy/rpm/CONFIGURATION.md +++ b/deploy/rpm/CONFIGURATION.md @@ -222,7 +222,8 @@ overrides that persist across package upgrades. | `bind_address` | `127.0.0.1:17670` (gateway default) | Address for the primary gRPC/HTTP API listener. | | `compute_driver` | `"podman"` (RPM default) | When unset, the gateway auto-detects Kubernetes, then Podman, then Docker. The RPM default pins to Podman; legacy `compute_drivers` lists are rejected. | | `[openshell.drivers.podman].default_image` | `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` | Default sandbox image. | -| `[openshell.drivers.podman].supervisor_image` | `ghcr.io/nvidia/openshell/supervisor:latest` | Supervisor image mounted into Podman sandboxes. | +| `[openshell.drivers.podman].sandbox_runtime_image` | `ghcr.io/nvidia/openshell/sandbox:latest` | Static musl sandbox runtime image mounted into Podman workloads. | +| `[openshell.drivers.podman].supervisor_image` | `ghcr.io/nvidia/openshell/supervisor:latest` | Dynamic glibc supervisor image used outside the workload. | | `[openshell.gateway].guest_tls_ca`, `guest_tls_cert`, `guest_tls_key` | auto-generated paths | Gateway-owned client TLS material injected into the selected local driver and mounted into sandbox containers. | | `[openshell.gateway.tls]` paths | auto-generated paths | Server TLS certificate, key, and client CA. | | `disable_tls` | unset | Set to `true` to disable TLS. | @@ -270,6 +271,7 @@ To pin specific image versions instead of `:latest`, set these values in `[openshell.drivers.podman]`: ```toml +sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:v0.0.37" supervisor_image = "ghcr.io/nvidia/openshell/supervisor:v0.0.37" default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:v0.0.37" ``` diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 85d12fc8d9..5af0589f20 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -57,6 +57,7 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.create=true \ --set grpcRoute.gateway.className=eg @@ -113,6 +114,7 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.create=true \ --set grpcRoute.gateway.className=eg \ diff --git a/docs/kubernetes/managing-certificates.mdx b/docs/kubernetes/managing-certificates.mdx index d97c1e24be..efef147f0a 100644 --- a/docs/kubernetes/managing-certificates.mdx +++ b/docs/kubernetes/managing-certificates.mdx @@ -52,6 +52,7 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set certManager.enabled=true ``` @@ -73,6 +74,7 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set certManager.enabled=true \ --set certManager.serverIssuerRef.name=letsencrypt-prod \ --set certManager.serverIssuerRef.kind=ClusterIssuer \ diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index cfae001a38..aaad7f6c53 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -3,87 +3,62 @@ # SPDX-License-Identifier: Apache-2.0 title: "OpenShift" sidebar-title: "OpenShift" -description: "Install the OpenShell Helm chart on OpenShift, including the SCC binding and chart overrides required by OpenShift's Security Context Constraints." +description: "Install the OpenShell Helm chart on OpenShift with capability-free sandbox workloads." keywords: "Generative AI, Cybersecurity, Kubernetes, OpenShift, SCC, Security Context Constraints, Helm, Gateway, Installation" position: 6 --- +The Kubernetes driver resolves the UID range assigned to each OpenShift +namespace and renders the sandbox and supervisor with a numeric non-root +identity from that range. OpenShell does not require the `privileged` SCC or any +added Linux capability. + -The OpenShift install path is experimental. It currently requires running sandbox pods under the `privileged` SCC and installing the gateway with TLS disabled. Use only for evaluation on a private network. +Verify that the selected OpenShift runtime profile permits an unprivileged +process to install a nested seccomp user-notification filter and use Landlock. +OpenShell fails sandbox startup when either capability-free runtime probe fails. -OpenShift's [Security Context Constraints](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html) reject the chart's default pod security settings. Installing on OpenShift requires precreating the namespace, granting the `privileged` SCC to the sandbox service account, and overriding a few chart values so the cluster admission controller can assign UIDs and FS groups itself. - -OpenShell installs sandbox nftables rules as individual commands. On OpenShift -nodes where optional conntrack or packet log expressions are unavailable, those -optional rules can fail without rolling back the required proxy bypass reject -rules. - ## Prerequisites -- OpenShift 4.x cluster with `oc` configured -- Helm 3.x -- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs installed +- OpenShift 4.x cluster with `oc` configured. +- Helm 3.x. +- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs. +- A CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces. -## Install +## Install OpenShell - - -## Create the namespace - -Pre-create the namespace so the SCC binding can be applied before the chart installs: +Pre-create the namespace, then install the chart. Keep the default restricted +security posture and acknowledge NetworkPolicy only after validating the CNI. ```shell oc create ns openshell -``` - -## Grant the privileged SCC to sandbox pods - -Sandbox pods run under the `openshell-sandbox` service account in the `openshell` namespace and require the `privileged` SCC: - -```shell -oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell -``` - -## Install the chart with OpenShift overrides - -```shell helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set server.disableTls=true \ - --set podSecurityContext.fsGroup=null \ - --set securityContext.runAsUser=null + --set supervisor.sandboxRuntime.networkPolicyEnforced=true ``` -| Override | Reason | -|---|---| -| `server.disableTls=true` | Runs the gateway over plaintext HTTP for simpler evaluation. | -| `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Clear the chart's hardcoded UID and fsGroup so OpenShift's SCC admission can assign them. | +The driver reads the namespace's `openshift.io/sa.scc.uid-range` annotation and +uses the resulting UID/GID for the sandbox, agent, trusted init containers, and +supervisor. Each container sets `allowPrivilegeEscalation: false`, drops all +Linux capabilities, and uses `RuntimeDefault` seccomp. -## Wait for the gateway to be ready +Wait for the gateway: ```shell oc -n openshell rollout status statefulset/openshell ``` -If you set `workload.kind=deployment`, use -`oc -n openshell rollout status deployment/openshell` instead. +If you set `workload.kind=deployment`, wait for `deployment/openshell` instead. - +## Connect to the Gateway -## Connect to the gateway - -The gateway is now running over plaintext HTTP. Connect with `oc port-forward`: +Forward the gateway port for local evaluation: ```shell oc -n openshell port-forward svc/openshell 8080:8080 -``` - -Register the gateway with the CLI: - -```shell -openshell gateway add http://127.0.0.1:8080 --local --name openshift +openshell gateway add https://127.0.0.1:8080 --local --name openshift openshell status ``` diff --git a/docs/kubernetes/sandbox-runtime.mdx b/docs/kubernetes/sandbox-runtime.mdx new file mode 100644 index 0000000000..319619ee7f --- /dev/null +++ b/docs/kubernetes/sandbox-runtime.mdx @@ -0,0 +1,128 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: "Kubernetes Sandbox Runtime" +sidebar-title: "Sandbox runtime" +description: "Understand how Kubernetes places and protects the sandbox runtime and supervisor." +keywords: "Generative AI, Cybersecurity, Kubernetes, Sandboxing, Network Policy, Seccomp, Landlock" +position: 2 +--- + +OpenShell runs each Kubernetes sandbox as two separately scheduled workloads. +The sandbox owns the agent process. The supervisor owns gateway credentials, +policy decisions, and upstream connections. + +## Understand the Components + +The Kubernetes driver always uses this placement: + +```mermaid +flowchart LR + Gateway[OpenShell gateway] + Supervisor[Supervisor Pod] + Sandbox[Sandbox workload Pod] + Agent[Agent processes] + External[External services] + + Gateway <-->|JWT-authenticated session| Supervisor + Supervisor <-->|TLS and bootstrap token| Sandbox + Sandbox --> Agent + Supervisor --> External +``` + +`openshell-sandbox` runs as PID 1 in the workload container. It launches the +agent, applies Landlock and child seccomp filters, identifies the process behind +each network operation, and relays approved streams. `openshell-supervisor` runs +in a directly managed Pod. It authenticates to the gateway, evaluates policy, +handles L7 and provider transformations, and opens upstream connections. + +Both containers run as the same namespace-resolved non-root UID and GID. Their +Pod specs set `allowPrivilegeEscalation: false`, drop every Linux capability, +and use `RuntimeDefault` seccomp. The sandbox installs an additional nested +seccomp user-notification filter without requesting a capability. Startup fails +closed if the runtime blocks the required seccomp or Landlock operations. + +The supervisor Pod points directly to the Sandbox resource with a non-controller +owner reference. Kubernetes therefore removes it when the Sandbox is deleted, +while the Agent Sandbox controller remains the sole controller of the workload +Pod. + +## Enforce Network Isolation + +The driver creates one workload fence per sandbox namespace before it releases +any workload Pod: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +spec: + podSelector: + matchLabels: + openshell.ai/boundary-role: workload + policyTypes: [Ingress, Egress] + ingress: + - from: + - podSelector: + matchLabels: + openshell.ai/boundary-role: supervisor + ports: + - protocol: TCP + port: 5500 + egress: [] +``` + +The empty egress list blocks direct DNS, gateway, node, metadata, and Internet +connections from every OpenShell workload in the namespace. The ingress rule +allows OpenShell supervisor Pods to reach sandbox TLS listeners. TLS, JWT +claims, session generation, and recorded Pod UIDs—not the NetworkPolicy—bind a +supervisor to its exact sandbox. Supervisors can reach cluster DNS, the gateway, +and policy-approved upstream destinations unless another namespace policy +restricts them. + +Kubernetes policies are additive. Keep sandbox namespaces under administrative +control so another principal cannot add permissive policies, create Pods with +OpenShell labels, or read bootstrap Secrets. Set +`supervisor.sandboxRuntime.networkPolicyEnforced: true` only after you verify that the +cluster CNI enforces both ingress and egress policies for these namespaces. + +## Bootstrap a Sandbox + +The driver creates each sandbox generation in a fail-closed order: + +1. Create and validate the workload egress fence. +2. Create the Sandbox resource with a scheduling gate. +3. Inspect the admitted Pod identity, security context, DNS settings, and + generation-specific Secret reference. +4. Create a gated supervisor Pod and separate immutable Secrets for sandbox and + supervisor trust material. +5. Remove both scheduling gates. +6. Publish readiness only after the supervisor attaches, confirms enforcement, + and registers the gateway relay. + +The trusted sandbox init container copies its Secret into a memory-backed +volume. The main container never mounts the projected Secret and removes the +staged bootstrap before it launches untrusted code. The supervisor receives an +audience-bound Kubernetes token, exchanges it for a sandbox-scoped JWT, and +keeps gateway and provider credentials outside the workload Pod. + +Stopping a sandbox removes both the workload and supervisor Pods. Starting it +creates a new generation with new Secrets and a replacement supervisor Pod +while preserving the workspace PVC. The namespace-wide workload fence remains +in place across sandbox generations. + +## Check Cluster Requirements + +This architecture requires the following cluster behavior: + +- Linux nodes and a container runtime that permits an unprivileged process to + install a nested seccomp user-notification filter under `RuntimeDefault`. +- Landlock enabled and usable by the non-root sandbox process. +- A CNI that enforces `networking.k8s.io/v1` ingress and egress policies, + including node-local and metadata destinations. +- Support for Pod scheduling gates and the safe + `net.ipv4.ip_unprivileged_port_start=0` sysctl. +- Administrative control of sandbox namespaces and OpenShell role labels. + +OpenShell actively probes the Linux primitives and validates the admitted Pod +before starting the agent. Treat a failed probe or changed security posture as +an unsupported runtime, not a degraded mode. diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index fb7881af2d..c3f4e2416e 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -85,7 +85,8 @@ Install from the OCI registry on GHCR. Replace `` with the chart versio helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ - --namespace openshell + --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true ``` To use the latest development build instead of a stable release: @@ -94,7 +95,8 @@ To use the latest development build instead of a stable release: helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version 0.0.0-dev \ - --namespace openshell + --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true ``` The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required. @@ -109,6 +111,7 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workspaceResources.enabled=false \ --set server.sandboxNamespace=app-a @@ -199,14 +202,12 @@ The most commonly changed values are: | `server.sandboxImage` | Default sandbox image used when a sandbox does not specify one. | | `server.sandboxImagePullSecrets` | Image pull secrets attached to sandbox pods. Referenced Secrets must exist in the sandbox namespace. | | `server.grpcEndpoint` | Endpoint that sandbox supervisors use to call back to the gateway. Must be reachable from inside the cluster. | -| `server.appArmorProfile` | AppArmor profile requested for sandbox agent containers. Defaults to `Unconfined`. | | `server.disableTls` | Run the gateway over plaintext HTTP. Use only behind a trusted transport. | | `server.auth.allowUnauthenticatedUsers` | Accept user-facing calls without OIDC or mTLS credentials. Use only for trusted local development or a fully trusted access proxy. | | `server.enableLoopbackServiceHttp` | Enable local plaintext HTTP for loopback sandbox service URLs. Defaults to `true`. | | `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. | -| `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. | -| `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). | -| `supervisor.sidecar.proxyUid` | Non-root UID used when sidecar process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0 instead. The configured UID must not match the sandbox UID. | +| `supervisor.sandboxRuntime.networkPolicyEnforced` | Required acknowledgement that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. | +| `supervisor.sandboxRuntime.boundaryPort` | Non-privileged TLS port used between paired supervisor and sandbox Pods. | | `upstreamProxy` | Operator-owned corporate HTTP forward proxy for policy-approved TLS egress. Refer to [Configure a Corporate Upstream Proxy](#configure-a-corporate-upstream-proxy). | Use a values file for repeatable deployments: @@ -216,16 +217,10 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --values my-values.yaml ``` -The chart defaults `server.appArmorProfile` to `Unconfined` because -runtime/default AppArmor profiles can block the supervisor's network namespace -mount setup on AppArmor-enabled nodes. Set `server.appArmorProfile` to an empty -string to omit the field, `RuntimeDefault` to force the runtime default, or -`Localhost/` when you load and manage a localhost profile on each -node. - To use private sandbox images, create a `kubernetes.io/dockerconfigjson` Secret in the sandbox namespace and reference its name: @@ -265,13 +260,12 @@ upstreamProxy: key: credentials authAllowInsecure: true -supervisor: - topology: sidecar ``` Use `authAllowInsecure: true` only when you accept that Basic authentication is cleartext on the connection to an `http://` proxy. The initial release supports `http://` proxy endpoints and TLS CONNECT egress. It does not support HTTPS-to-proxy, custom corporate CA bundles, or forwarding plain HTTP egress through the proxy. -Proxy credentials require `sidecar` topology. It mounts the credential only into the dedicated network supervisor container. OpenShell rejects credential Secrets with `combined` topology because Kubernetes `fsGroup` volume permission handling can make a shared credential mount readable by the sandbox group. +The credential mounts only in the separately scheduled supervisor Pod. The +sandbox workload cannot read it through its environment or volumes. ## RBAC @@ -306,6 +300,7 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set serviceAccount.create=false \ --set serviceAccount.name=my-existing-sa ``` @@ -321,7 +316,7 @@ The gateway exposes `/healthz` for process liveness and `/readyz` for dependency ## Next Steps -- To choose between combined and sidecar sandbox pods, refer to [Topology](/kubernetes/topology). +- Kubernetes sandboxes use separate workload and directly managed supervisor Pods; refer to [Sandbox runtime](/kubernetes/sandbox-runtime). - To enable automatic certificate rotation with cert-manager, refer to [Managing Certificates](/kubernetes/managing-certificates). - To expose the gateway externally without port-forwarding, refer to [Ingress](/kubernetes/ingress). - To configure OIDC or reverse-proxy authentication, refer to [Access Control](/kubernetes/access-control). diff --git a/docs/kubernetes/topology.mdx b/docs/kubernetes/topology.mdx deleted file mode 100644 index 869fc07f1b..0000000000 --- a/docs/kubernetes/topology.mdx +++ /dev/null @@ -1,249 +0,0 @@ ---- -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -title: "Kubernetes Sandbox Topology" -sidebar-title: "Topology" -description: "Choose between combined and sidecar supervisor topology for Kubernetes sandbox pods." -keywords: "Generative AI, Cybersecurity, Kubernetes, Sandboxing, Sidecar, Network Policy, RuntimeClass" -position: 2 ---- - -Kubernetes sandbox pods can run the OpenShell supervisor in `combined` or -`sidecar` topology. Choose the topology based on which controls you need inside -the pod and how much privilege your cluster allows on the agent container. - -## Choose a Topology - -The default `combined` topology preserves the full OpenShell enforcement model. -Use `sidecar` only when you accept network-focused enforcement in exchange for a -lower-privilege agent container. - -| Topology | Use when | Main tradeoff | -|---|---|---| -| `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. | -| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. | - -## Privilege Model - -The long-running container permissions differ by topology: - -| Topology | Pod or container | UID/GID | Privilege escalation | Capabilities | Result | -|---|---|---|---|---|---| -| `combined` | Agent container, which also runs the supervisor | Not forced by topology | Not explicitly disabled by the driver | Adds `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, and `SYSLOG`; adds `SETUID`, `SETGID`, and `DAC_READ_SEARCH` when user namespaces are enabled | Full supervisor controls run in the agent container. | -| `sidecar` | Agent container, process-only supervisor (`network-only`) | `sandbox_uid:sandbox_gid` | `false` | Drops `ALL` | Agent and workload run without added Linux capabilities. | -| `sidecar` | Network supervisor sidecar, binary-aware mode (default) | `0:sandbox_gid` | `false` | Drops `ALL`; adds `SYS_PTRACE` and `DAC_READ_SEARCH` | Root sidecar inspects cross-UID workload `/proc` entries. The nftables fence exempts UID 0, so do not inject other root containers into these pods. | -| `sidecar` | Network supervisor sidecar, endpoint/L7-only mode | `proxyUid:sandbox_gid` | `false` | Drops `ALL` | Non-root sidecar enforces endpoint and L7 policy without matching `policy.binaries`. | - -Short-lived setup containers still have the permissions needed to prepare the -pod: - -| Topology | Setup container | UID/GID | Privilege escalation | Capabilities | Purpose | -|---|---|---|---|---|---| -| `combined` | Supervisor install init container | `0` | Not set | Not set | Copies the supervisor binary into the agent container volume. | -| `sidecar` | Network init container | `0` | `false` | Drops `ALL`; adds `NET_ADMIN`, `NET_RAW`, `CHOWN`, and `FOWNER` | Installs pod-local nftables rules and prepares shared sidecar state. | - -## Combined Topology - -Combined topology is the original Kubernetes mode and remains the default. The -agent container starts the OpenShell supervisor, and the supervisor launches the -workload after applying sandbox setup. - -```mermaid -flowchart TB - Sandbox["agents.x-k8s.io Sandbox"] - - subgraph Pod["Sandbox pod"] - subgraph Agent["agent container"] - Supervisor["OpenShell supervisor
network + process + filesystem"] - Workload["Agent workload"] - end - end - - Gateway["OpenShell Gateway"] - External["External services"] - - Sandbox --> Pod - Supervisor --> Workload - Supervisor -->|"gateway callback / SSH relay"| Gateway - Supervisor -->|"policy-enforced egress"| External -``` - -Combined topology keeps these controls in one supervisor path: - -- Network endpoint and L7 policy enforcement. -- Filesystem policy enforcement. -- Process and binary identity checks. -- Privilege drop into the sandbox user. -- Gateway relay, SSH sessions, exec, and file sync. - -Because the supervisor performs network namespace setup and process/filesystem -controls from the agent container, Kubernetes grants that container elevated -Linux capabilities. Use this mode when you need the complete OpenShell sandbox -contract and your cluster policy permits those capabilities. - -## Sidecar Topology - -Sidecar topology splits the supervisor into a network sidecar and a -low-privilege process supervisor in the agent container. - -```mermaid -flowchart TB - Sandbox["agents.x-k8s.io Sandbox"] - - subgraph Pod["Sandbox pod"] - Init["network init container
root setup capabilities"] - State["shared state + TLS volumes"] - NetNS["pod network namespace"] - - subgraph Agent["agent container"] - ProcessSupervisor["process supervisor
network-only"] - Workload["Agent workload"] - end - - NetworkSidecar["network supervisor sidecar
UID 0 by default"] - SshEndpoint["abstract SSH relay socket
peer-PID authenticated"] - end - - Gateway["OpenShell Gateway"] - External["External services"] - - Sandbox --> Pod - Init -->|"installs nftables rules"| NetNS - ProcessSupervisor --> Workload - Workload -->|"egress redirected on loopback"| NetworkSidecar - NetworkSidecar -->|"gateway session + relays"| Gateway - NetworkSidecar -->|"policy-enforced egress"| External - NetworkSidecar -->|"control socket + proxy TLS"| State - ProcessSupervisor -->|"bootstrap + updates"| State - ProcessSupervisor --> SshEndpoint - NetworkSidecar -->|"SSH relay"| SshEndpoint - NetworkSidecar --- State -``` - -The pod contains these OpenShell-managed pieces: - -| Component | Runs as | Purpose | -|---|---|---| -| Network init container | Root with setup capabilities | Installs pod-level nftables rules and prepares shared sidecar state. | -| Network sidecar | UID 0 by default; `supervisor.sidecar.proxyUid` when binary-aware policy is disabled | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and serves local policy/provider state over the sidecar control socket. | -| Agent container | Resolved sandbox UID/GID | Runs the process supervisor and launches the user workload. | - -In this topology, the agent container defaults to `runAsNonRoot: true`, -`allowPrivilegeEscalation: false`, and `capabilities.drop: ["ALL"]`. The -default binary-aware network sidecar runs as UID 0, drops default Linux -capabilities, and adds `SYS_PTRACE` plus `DAC_READ_SEARCH` for cross-UID workload -process identity resolution. Setting -`supervisor.sidecar.processBinaryAwareNetworkPolicy=false` runs the sidecar as -the configured non-root `proxyUid`, omits both capabilities, and downgrades -network policy to endpoint/L7 enforcement without binary matching. The root -init container keeps the setup capabilities needed to configure pod networking. - -Sidecar mode preserves gateway session behavior, including SSH connectivity, -because the network sidecar owns the gateway session and bridges relay requests -to a Linux abstract SSH socket owned by the process supervisor. The relay -verifies the socket peer PID against the authenticated control connection, so -the workload cannot replace the relay endpoint. The agent container does not get a -gateway endpoint, gateway TLS material, or the sandbox bootstrap token in the -default sidecar path. - - -Sidecar mode runs the process supervisor in `network-only` mode. OpenShell still -enforces network endpoint and L7 policy through the sidecar, and the process -supervisor applies Landlock filesystem policy and child seccomp filters where -the kernel/runtime supports them. The process supervisor does not perform -root-to-sandbox privilege dropping because Kubernetes starts the container as -the sandbox UID/GID, and it does not perform supervisor identity mount -isolation because gateway credentials are not mounted into the agent container. -Sidecar pods use `shareProcessNamespace: true` so the network sidecar can -resolve workload process and binary identity through `/proc/`. - - -## Credential Exposure - -Sidecar topology keeps gateway credentials in the network sidecar. The agent -container does not mount the projected ServiceAccount token used for sandbox -token bootstrap, does not mount the sandbox client TLS secret, and does not get -gateway callback environment variables. - -The network sidecar serves the policy and workload-facing provider environment -over a Unix control socket in the shared sidecar state volume. Before launching -the workload, the process supervisor establishes the only accepted connection. -The sidecar validates its UID, GID, and PID with peer credentials, unlinks the -listener, derives the SSH target from trusted configuration, and rejects later -clients. The connection receives bootstrap state and provider-environment -updates after settings polls. If it closes, the network sidecar exits so -Kubernetes recreates the one-client bootstrap listener, and the process -supervisor exits so Kubernetes terminates the workload and restarts the agent -container. This symmetric failure behavior prevents a surviving workload from -claiming the new control listener after an isolated sidecar restart. Future -child processes can see refreshed provider env without giving the agent -container gateway authentication material. This does not mutate the environment -of the already-running workload entrypoint. Use `combined` topology when you -need the full single-supervisor enforcement path; use additional runtime -isolation when you need a stronger container boundary around sidecar workloads. - -## RuntimeClass Isolation - -Sidecar topology has been validated with Kata Containers. It does not currently -support gVisor because sidecar mode requires pod-local nftables setup, which -gVisor does not provide to the init container. A supported sandboxed runtime -strengthens the container boundary while OpenShell focuses on network policy -enforcement from the sidecar. - -Runtime classes do not re-enable the OpenShell privilege-drop or supervisor -mount-isolation controls that sidecar mode relaxes. Use them as an additional -workload boundary, not as a replacement for the combined topology's full -supervisor controls. - -You can set a default runtime class in the Kubernetes driver configuration or -override it per sandbox with driver config: - -```shell -openshell sandbox create \ - --driver-config-json '{"kubernetes":{"pod":{"runtime_class_name":"kata-containers"}}}' \ - -- claude -``` - -## Enable Sidecar Mode - -For direct gateway TOML configuration, set the Kubernetes driver fields: - -```toml -[openshell.drivers.kubernetes] -topology = "sidecar" - -[openshell.drivers.kubernetes.sidecar] -proxy_uid = 1337 -``` - -`proxy_uid` configures only the relaxed endpoint/L7-only sidecar. It must be at -least `1000` and must not match the sandbox UID. The default binary-aware mode -runs the sidecar as UID 0 instead. The network init container exempts the -effective sidecar UID from proxy redirection so the sidecar can reach the -gateway. - -When the Helm chart renders `gateway.toml`, set the equivalent chart values: - -```yaml -supervisor: - topology: sidecar - sidecar: - proxyUid: 1337 - processBinaryAwareNetworkPolicy: true -``` - -Leave `topology` unset, or set it to `combined`, to keep the original -single-container supervisor path. For Helm installs, leave -`supervisor.topology` unset or set it to `combined`. - -Set `supervisor.sidecar.processBinaryAwareNetworkPolicy=false` only when you -accept downgrading sidecar network policy to endpoint/L7 enforcement without -matching `policy.binaries`. This changes the sidecar from UID 0 to `proxyUid` -and removes its `SYS_PTRACE` and `DAC_READ_SEARCH` capabilities, which are used -for cross-UID `/proc` inspection. - -## Next Steps - -- To install OpenShell on Kubernetes, refer to [Setup](/kubernetes/setup). -- To configure gateway authentication, refer to [Access Control](/kubernetes/access-control). -- To review the driver fields, refer to [Gateway Configuration File](/reference/gateway-config). diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 7c8e043e16..0a8ac76e09 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -94,7 +94,7 @@ future version. To migrate an existing file: omit Docker or Podman `sandbox_pids_limit` to use OpenShell's default limit of 2048, and omit Podman `health_check_interval_secs` to disable health checks. Explicit zero values are invalid. -7. Remove local Docker, Podman, or VM `grpc_endpoint` when the topology-derived +7. Remove local Docker, Podman, or VM `grpc_endpoint` when the driver-derived callback is correct, or retain it as an explicit override. Kubernetes raw TOML requires an explicit endpoint; Helm derives one from the release's gateway Service. New VM root filesystems use an image-provided `sandbox` @@ -546,16 +546,12 @@ default_image = "ghcr.io/nvidia/openshell-community/sandboxes/b image_pull_policy = "if_not_present" image_pull_secrets = ["regcred"] # Defaults to the gateway version; override to pin a specific build. +# sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" +sandbox_runtime_image_pull_policy = "if_not_present" +# Defaults to the gateway version; override to pin a specific build. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" supervisor_image_pull_policy = "if_not_present" -# Use the image volume on Kubernetes >= 1.35 (GA in 1.36); switch to "init-container" -# on older clusters or where the ImageVolume feature gate is off. -supervisor_sideload_method = "image-volume" -# "combined" runs the existing single supervisor container with full process, -# filesystem, and network enforcement in the agent container. "sidecar" moves -# pod-level network enforcement and gateway session handling into a network sidecar. -topology = "combined" # Optional corporate HTTP forward proxy for policy-approved TLS egress. The # sandbox workload cannot select or override these settings. Only http:// proxy # endpoints and TLS CONNECT traffic are supported; plain HTTP egress remains @@ -573,9 +569,7 @@ topology = "combined" # Secret prevents that Pod from starting; unreadable or malformed `user:pass` # content is validated fail-closed by the supervisor at startup and never # falls back to direct egress. -# Proxy credential Secrets require `topology = "sidecar"`. Combined topology -# shares its credential mount with the workload and can make it readable by the -# sandbox group through Kubernetes `fsGroup` volume permission handling. +# Proxy credentials mount only in the separately scheduled supervisor Pod. # Required with a credential Secret: Basic authentication to an http:// proxy # is cleartext on the connection to that proxy. # proxy_auth_allow_insecure = true @@ -590,7 +584,6 @@ ssh_socket_path = "/run/openshell/ssh.sock" client_tls_secret_name = "openshell-client-tls" host_gateway_ip = "10.0.0.1" enable_user_namespaces = false -app_armor_profile = "Unconfined" workspace_default_storage_size = "10Gi" # Kubernetes StorageClass for the workspace PVC. Empty (default) omits the # field, using the cluster's default StorageClass. Set this on clusters with no @@ -625,19 +618,12 @@ enabled = true gateway_namespace = "openshell" gateway_pod_selector = { "app.kubernetes.io/name" = "openshell", "app.kubernetes.io/instance" = "openshell" } -[openshell.drivers.kubernetes.sidecar] -# UID used by relaxed long-running network sidecars. Strict process/binary-aware -# sidecars run as UID 0 so Kubernetes grants the required /proc inspection -# capabilities into the effective set. In sidecar topology the network init -# container installs nftables rules that exempt the effective sidecar UID, so -# this dedicated infrastructure UID must remain at least 1000 and must not -# match the sandbox workload UID. -proxy_uid = 1337 -# Keep process/binary-aware network policy enabled in sidecar topology. Set -# false to run the sidecar as proxy_uid, drop the sidecar's extra /proc -# inspection capabilities, and enforce endpoint/L7 policy without matching -# policy.binaries. -process_binary_aware_network_policy = true +[openshell.drivers.kubernetes.sandbox_runtime] +# Required acknowledgement that the cluster CNI enforces NetworkPolicy and the +# sandbox namespaces prevent untrusted policy, pod, label, and Secret changes. +network_policy_enforced = true +# TLS-protected boundary listener reached only by the paired control pod. +boundary_port = 5500 ``` In managed workspace mode, the Kubernetes driver copies each explicitly named @@ -660,12 +646,12 @@ map Windows Sandboxing provider events into the gateway's OCSF stream. ```toml [openshell] -version = 1 +version = 2 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" -compute_drivers = ["mxc"] +compute_driver = "mxc" [openshell.drivers.mxc] wxc_exec_path = "C:\\mxc-kit\\bin\\wxc-exec.exe" @@ -673,10 +659,6 @@ backend = "process_container" default_configuration_id = "composable" pc_least_privilege = false pc_capabilities = [] -# Pattern-C governed egress. The address is a loopback seed; the driver -# allocates a unique ephemeral port for each sandbox. -egress_proxy = false -egress_proxy_addr = "" debug = false etw_audit = true ``` @@ -686,19 +668,6 @@ administrator or belong to the Windows Performance Log Users group. Workload commands and working directories remain sandbox-scoped and must be supplied in the `mxc` driver configuration when creating a sandbox. -Set `egress_proxy = true` with a loopback seed such as -`egress_proxy_addr = "127.0.0.1:18080"` to enable the Windows Pattern-C split. -MXC redirects sandbox traffic to a sandbox-unique port on the configured -loopback IP, where the host CONNECT proxy enforces the trimmed network policy. -The driver exposes the proxy's public CA bundle through a sandbox-unique -internal filesystem grant so HTTPS clients can read the injected trust path; -the CA private key remains in gateway memory. -Governed egress requires `process_container` until MXC supports `network.proxy` -for isolation sessions. -MXC rejects policies containing `network_middlewares` before launch because -this host-proxy path does not yet receive the gateway's built-in and remote -middleware registry. - The driver records executable identity in process audit events and omits raw command arguments because they can contain credentials or personal data. See [OCSF JSON Export](/observability/ocsf-json-export) for durable Windows audit @@ -729,16 +698,14 @@ image_pull_policy = "if_not_present" # Value assigned to the openshell.sandbox_namespace label on sandbox containers. sandbox_label = "docker-dev" # Optional override. When omitted, the gateway derives -# https://host.openshell.internal: for this topology. +# https://host.openshell.internal: for this driver. grpc_endpoint = "https://host.openshell.internal:17670" -# Skip the image-pull-and-extract step by pointing at a locally built binary. -supervisor_bin = "/usr/local/libexec/openshell/openshell-sandbox" -# When supervisor_bin is omitted, Docker extracts /openshell-sandbox from this image. -# Defaults to the gateway version; override to pin a specific build. +# Workload-side runtime. Defaults to the gateway version. +# sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" +# Supervisor runtime. Defaults to the gateway version. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" network_name = "openshell-docker" host_gateway_ip = "172.17.0.1" -ssh_socket_path = "/run/openshell/ssh.sock" # Unsafe operator override. Host bind mounts, including Docker local-driver # bind-backed volumes, expose gateway-host paths inside sandboxes and can # negate OpenShell isolation and filesystem controls. @@ -765,7 +732,7 @@ Use `sandbox_label` for Docker configurations. The legacy ### Podman -Sandboxes run as Podman containers on a user-mode bridge network. The supervisor image is mounted read-only via Podman's `type=image` mount. Configure guest mTLS paths once under `[openshell.gateway]`; the gateway validates and injects the bundle into the selected local driver. +Each Podman sandbox uses two containers. The workload container runs `openshell-sandbox` with `network=none`; the supervisor container runs on the host network and initiates policy-approved upstream connections. A private volume carries their authenticated Unix-domain socket. Configure guest mTLS paths once under `[openshell.gateway]`; the gateway validates and injects the bundle into the selected local driver. ```toml [openshell] @@ -798,7 +765,9 @@ network_name = "openshell" # host_gateway_ip = "192.168.127.254" ssh_socket_path = "/run/openshell/ssh.sock" stop_timeout_secs = 45 -# Defaults to the gateway version; override to pin a specific build. +# Statically linked workload-side runtime. Defaults to the gateway version. +# sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" +# Dynamically linked supervisor runtime. Defaults to the gateway version. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" # Unsafe operator override. Host bind mounts, including Podman local-driver # bind-backed volumes, expose gateway-host paths inside sandboxes and can @@ -901,18 +870,17 @@ health_check_interval_secs = 10 # explicit container-reachable TCP endpoint, for provider token exchange. # provider_spiffe_workload_api_socket = "/run/spire/agent.sock" # provider_spiffe_workload_api_socket = "tcp:169.254.1.2:8081" -# Omit app_armor_profile to preserve Podman's runtime-selected profile. -# Set Unconfined only when the supervisor's mount setup requires it. +# Omit app_armor_profile to preserve Podman's runtime-selected workload profile. # Explicit RuntimeDefault and Localhost/ require Podman to report # AppArmor support. -# app_armor_profile = "Unconfined" +# app_armor_profile = "RuntimeDefault" ``` Use `ssh_socket_path` for Podman configurations. The legacy `sandbox_ssh_socket_path` key is rejected. When `app_armor_profile` is omitted, OpenShell sends no override and Podman applies its runtime-selected profile. -Set `Unconfined` explicitly only when the deployment requires the supervisor's -mount setup to bypass that profile. +The setting applies to the workload container; the supervisor retains Podman's +runtime-selected profile. ### MicroVM @@ -938,7 +906,7 @@ state_dir = "/var/lib/openshell/vm" driver_dir = "/usr/local/libexec/openshell" default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" # Optional override. When omitted, the gateway derives -# https://host.openshell.internal: for the VM topology. +# https://host.openshell.internal: for the VM driver. grpc_endpoint = "https://host.openshell.internal:17670" # Empty falls back to default_image. bootstrap_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" @@ -961,35 +929,17 @@ overlay_disk_mib = 4096 # any present-but-invalid value is rejected at gateway startup rather than # degrading to a direct dial. proxy_auth_file is a path on the gateway host. # -# The sandbox cannot select or override these settings. They reach the guest -# supervisor through a protected per-sandbox argument file the driver writes -# into the overlay upperdir on every launch, so a sandbox image cannot supply -# its own values or disable the operator's by baking a file at that path. +# The sandbox cannot select or override these settings. The driver passes them +# only to the host supervisor. # # Reachability: a proxy on the corporate network needs no special address and -# works on every VM sandbox. The guest's callback to the gateway is unaffected -# and never traverses the proxy. +# works on every VM sandbox. # -# A proxy on the gateway host itself is reachable only from libkrun-backed -# (non-GPU) sandboxes: their egress leaves through gvproxy, which NATs -# 192.168.127.254 to the host's 127.0.0.1, so address it as -# http://host.openshell.internal: rather than http://127.0.0.1:. -# GPU sandboxes run on the QEMU/TAP backend, which has no such NAT — -# host.openshell.internal resolves to the TAP host address, and the driver's -# nftables rules let the guest reach only the gateway port on the host. A -# gateway-host proxy URL is therefore rejected when the sandbox launches on -# QEMU, rather than timing out on every CONNECT; give GPU sandboxes a proxy -# address routable from the guest's masqueraded egress. +# A proxy on the gateway host is addressed as +# http://host.openshell.internal:. The host supervisor normalizes that +# name to host loopback for both libkrun and QEMU sandboxes. # -# Because a microVM has no bind mounts or container secrets, the driver stages -# the credential and optional CA bundle into the per-sandbox overlay disk and -# removes them with the sandbox. The credential is root-only; the non-secret CA -# bundle is mode 0644. Both are referenced from the protected supervisor -# argument file by fixed guest paths, never their gateway-host paths. The -# credential is therefore at rest in that overlay image on the gateway host — -# the same delivery the per-sandbox gateway token already uses, and a difference -# from the Podman secret model worth noting when choosing where to keep proxy -# credentials. +# Credentials and private CA material remain host-side with the supervisor. # https_proxy = "http://host.openshell.internal:8080" # no_proxy = "10.0.0.0/8,.internal.example" # proxy_auth_file = "/etc/openshell/secrets/proxy-auth" diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index f328909ceb..76548f60c7 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -40,36 +40,9 @@ an exited canonical process remains a terminal sandbox result. Exit code zero produces `Completed`; a nonzero or signal-normalized exit produces `Error` with the exact exit code. Driver and supervisor failures remain `Error`. -## Build with Selected Compute Drivers - -Source builds of `openshell-gateway` can include any subset of the Docker, -Podman, Kubernetes, VM, and MXC drivers. Enable the corresponding -`compute-driver-docker`, `compute-driver-podman`, `compute-driver-kubernetes`, -`compute-driver-vm`, or `compute-driver-mxc` Cargo features. For example, build -a Docker-only gateway with telemetry support: - -```shell -cargo build --release -p openshell-gateway --no-default-features --features telemetry,compute-driver-docker -``` - -On Windows, select only MXC with: - -```shell -cargo build --release -p openshell-gateway --no-default-features --features telemetry,compute-driver-mxc,bundled-z3 -``` - -The default `in-tree-compute-drivers` feature retains the full platform driver -set, including MXC on Windows. MXC links only on Windows. The other four -features link drivers on non-Windows platforms; on Windows they install -registrations that report the driver as unsupported. A build with no default -features and no driver features connects to external drivers only. -Auto-detection probes only compiled registrations. -To select a driver omitted from a custom build, configure its external -`socket_path` as described below. - ## Configure a Compute Driver -Configure the compute driver on the gateway. Current releases accept one driver per gateway. Set the singular `compute_driver` key in the gateway TOML file: +Configure the compute driver on the gateway. Current releases accept one driver per gateway. Set `compute_driver` in the gateway TOML file: ```toml [openshell.gateway] @@ -83,15 +56,13 @@ Non-reserved names select an extension driver and require a When `compute_driver` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Docker must respond on a known API socket. Podman first probes known API sockets and then asks the `podman` CLI for the active native or machine-backed socket. The VM driver is never auto-detected; configure it explicitly with `compute_driver = "vm"` or set `OPENSHELL_COMPUTE_DRIVER=vm` in the launch environment. -`compute_driver` accepts exactly one scalar driver name. The legacy `compute_drivers` list is rejected by schema version 2. - Common gateway options: | Gateway TOML option | Description | |---|---| | `compute_driver = ""` | Select the compute driver. Built-in values are `docker`, `podman`, `kubernetes`, and `vm`; custom names require `[openshell.drivers.].socket_path`. | -Set driver-specific values such as sandbox images, callback endpoints, network names, and VM sizing in the gateway TOML file. A TLS-enabled gateway-managed Docker, Podman, or VM driver requires a complete `guest_tls_ca`, `guest_tls_cert`, and `guest_tls_key` bundle in `[openshell.gateway]`; package-managed local TLS supplies it automatically. Driver tables reject those gateway-owned fields. Kubernetes projects guest TLS through a Secret instead. See the [Gateway Configuration File](./gateway-config) reference for the full schema and migration steps. +Set driver-specific values such as sandbox images, callback endpoints, network names, TLS material, and VM sizing in the gateway TOML file. See the [Gateway Configuration File](./gateway-config) reference for the full `[openshell.drivers.]` schema. Extension drivers use the same `compute_driver.proto` gRPC surface as the managed VM driver. For an out-of-tree driver, choose a driver name and point @@ -110,8 +81,8 @@ socket path. The endpoint replaces normal driver construction for that name, including canonical built-in names: ```shell -openshell-gateway --compute-driver kyma --compute-driver-socket /run/openshell/kyma.sock -openshell-gateway --compute-driver docker --compute-driver-socket /run/openshell/docker.sock +openshell-gateway --drivers kyma --compute-driver-socket /run/openshell/kyma.sock +openshell-gateway --drivers docker --compute-driver-socket /run/openshell/docker.sock ``` The gateway connects to the operator-provided endpoint; it does not provision @@ -186,7 +157,7 @@ the gateway. If the primary listener covers that address, the gateway reuses it and sandbox JWT authentication restricts the supervisor to its callback RPC allowlist. If the primary listener is not reachable through that address, the gateway creates an additional callback-only listener. Use the primary endpoint -for CLI, administrator, health, reflection, provider management, and +for CLI, administrator, health, reflection, inference-route management, and HTTP requests. A `PermissionDenied` response from an additional callback-only listener is expected for those requests. Do not broaden the primary listener to `0.0.0.0` solely to make sandbox callbacks reachable. @@ -195,7 +166,7 @@ to `0.0.0.0` solely to make sandbox callbacks reachable. [Docker](https://www.docker.com/get-started/)-backed sandboxes run as containers on the gateway host. Use Docker for local development, single-machine gateways, and hosts that already use Docker Desktop or Docker Engine. -The gateway talks to the Docker daemon to create sandbox containers. +The gateway talks to the Docker daemon to create sandbox containers. Docker is also required for local image builds from directories or Dockerfiles. Docker Desktop and compatible macOS runtimes route `host.openshell.internal` through an IPv4 host-gateway alias. The gateway reuses an IPv4 primary listener @@ -204,7 +175,7 @@ that already covers loopback. Otherwise, the Docker driver requests a separate For maintainer-level implementation details, refer to the [Docker driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-docker/README.md). -Select Docker with `compute_driver = "docker"` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `sandbox_label`, `supervisor_bin`, `supervisor_image`, `image_pull_policy`, `ssh_socket_path`, and `sandbox_pids_limit` in `[openshell.drivers.docker]`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds. +Select Docker with `compute_driver = "docker"` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.docker]`. The sandbox runtime image contains `/openshell-sandbox`; the supervisor image contains `/openshell-supervisor`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds. When operating `openshell-driver-docker` as an external driver, set `OPENSHELL_OTLP_ENDPOINT` to export its spans. The driver continues W3C trace @@ -271,7 +242,7 @@ OpenShell rejects mount `source`, `target`, and Docker volume `subpath` values with surrounding whitespace. OpenShell also rejects mount targets that replace the workspace root or container root, or contain or are contained by the configured SSH socket or reserved `/opt/openshell`, `/etc/openshell`, -`/etc/openshell-tls`, `/run/openshell`, `/run/openshell-sidecar`, and network +`/etc/openshell-tls`, `/run/openshell`, and network namespace roots. These checks do not make host bind mounts safe. ## Podman Driver @@ -280,29 +251,11 @@ namespace roots. These checks do not make host bind mounts safe. The gateway talks to the Podman API socket. The Podman driver requires Podman 5.x, cgroups v2, rootless networking, and an active Podman user socket. When `socket_path` is not set, the driver probes known socket paths, then uses the `podman` CLI to resolve the active native or machine-backed connection. It fails to start if neither method finds a socket. -For maintainer-level implementation details, refer to the [Podman driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/README.md) and [Podman networking notes](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/NETWORKING.md). - -Select Podman with `compute_driver = "podman"` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `ssh_socket_path`, and `sandbox_pids_limit` in `[openshell.drivers.podman]`. - -### macOS Podman Socket Path +The agent workload uses `network=none`. Its trusted supervisor companion uses Podman's host network for gateway callbacks and policy-approved upstream connections. -On macOS, Homebrew-installed Podman does not create the default socket path -that the driver probes (`~/.local/share/containers/podman/machine/podman.sock`). -The actual API socket lives under `/var/folders/` in a path that macOS can -rotate after a reboot. - -If the gateway fails with `Podman socket not found; is podman machine running?` -while `podman machine list` shows a running machine, set the -`OPENSHELL_PODMAN_SOCKET` environment variable to the dynamic socket path: - -```shell -export OPENSHELL_PODMAN_SOCKET="$(podman machine inspect --format '{{.ConnectionInfo.PodmanSocket.Path}}')" -``` +For maintainer-level implementation details, refer to the [Podman driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/README.md) and [Podman networking notes](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/NETWORKING.md). -Add this to your shell profile or gateway launch environment so it resolves -correctly after each reboot. Alternatively, set `socket_path` in -`[openshell.drivers.podman]` to the current path, but note that the path may -change when macOS rotates `/var/folders/`. +Select Podman with `compute_driver = "podman"` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `sandbox_runtime_image`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.podman]`. Podman sandboxes default to a 45-second graceful stop window before Podman escalates from `SIGTERM` to `SIGKILL`. Set `stop_timeout_secs` in gateway config, or `OPENSHELL_STOP_TIMEOUT` for the standalone driver, when a local runtime needs a different teardown window. @@ -316,12 +269,6 @@ stopped sandboxes alone. For proxy-required networks, the Podman driver also accepts the corporate egress proxy keys `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, and `proxy_connect_by_hostname`. The supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior. -Podman preserves its runtime-selected AppArmor profile when -`app_armor_profile` is omitted. Set `Unconfined` explicitly only when the -supervisor's mount setup requires it. Explicit `RuntimeDefault` and -`Localhost/` selections fail startup when Podman reports that AppArmor -is unavailable. - On macOS with `podman machine`, the driver uses gvproxy's host-loopback IP, `192.168.127.254`, for sandbox host aliases by default. Set `host_gateway_ip` only when your Podman machine uses a non-standard host-loopback address. On Linux, an empty `host_gateway_ip` keeps Podman's `host-gateway` resolver behavior. Direct local callbacks from rootless Podman require Podman to report the pasta network helper. Slirp4netns, other helpers, and Podman versions that do not report their helper require an explicitly remote `grpc_endpoint`; otherwise the gateway fails startup rather than leaving sandbox callbacks unreachable. Rootful Podman continues to use the configured network's bridge gateway address. ### Podman Driver Config Mounts @@ -411,16 +358,10 @@ compute_driver = "vm" For a launch-time override, set `OPENSHELL_COMPUTE_DRIVER=vm` in the gateway environment and restart the service. -Configure VM driver values such as `grpc_endpoint`, `driver_dir`, `state_dir`, `default_image`, `bootstrap_image`, `vcpus`, `mem_mib`, `overlay_disk_mib`, and `krun_log_level` in `[openshell.drivers.vm]`. The VM `state_dir` stores overlay disks, console logs, runtime state, image-rootfs cache, and the private `run/compute-driver.sock` socket. The VM socket path is managed by the gateway and is not configurable through remote endpoint settings. +Configure VM driver values such as `grpc_endpoint`, `driver_dir`, `state_dir`, `default_image`, `bootstrap_image`, `vcpus`, `mem_mib`, `overlay_disk_mib`, `krun_log_level`, and `guest_tls_*` in `[openshell.drivers.vm]`. The VM `state_dir` stores overlay disks, console logs, runtime state, image-rootfs cache, and the private `run/compute-driver.sock` socket. The VM socket path is managed by the gateway and is not configurable through remote endpoint settings. The gateway starts `openshell-driver-vm` over a private Unix socket and passes its process ID so the driver can reject unexpected local clients. The driver's standalone TCP listener is disabled unless `--allow-unauthenticated-tcp` is set for local development. -Scripts that invoke the experimental standalone driver directly must use -`--grpc-endpoint` and the `--upstream-proxy*` option family. Schema v2 removes -the previous `--openshell-endpoint`, `--https-proxy`, `--no-proxy`, and -`--proxy-*` spellings; gateway-managed deployments do not use those options -directly. - ### Local image resolution The VM driver resolves sandbox images from a local container engine before falling back to registry pulls. It tries Docker first, then uses the same Podman socket discovery as the Podman driver. On Linux with Podman, enable the API socket so the driver can find local images: @@ -429,23 +370,28 @@ The VM driver resolves sandbox images from a local container engine before falli systemctl --user start podman.socket ``` -### Host Firewall - -The VM driver creates nftables rules on the host for each sandbox VM's TAP network interface. These rules provide NAT for VM connectivity and defense-in-depth isolation: unsolicited inbound connections to the VM are dropped, and the VM can only reach the gateway port on the host. Primary security enforcement (proxy-only egress and bypass detection) is handled by the sandbox supervisor inside the VM guest. +### Network isolation -On hosts with restrictive firewalls (e.g. firewalld), the host firewall may additionally block VM traffic that the driver's rules accept. If VM sandboxes cannot reach the network, verify that the host firewall allows forwarding and input for `vmtap-*` interfaces. See the [VM driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-vm/README.md#host-side-nftables-rules) for details. +VM sandboxes boot without a virtual NIC. `openshell-sandbox` intercepts workload +network syscalls inside the guest and carries mediated streams over virtio-vsock +to the host `openshell-supervisor`, which owns DNS, policy evaluation, and +external connections. The driver does not create TAP interfaces or host +nftables rules. ### Corporate Proxy Egress -For proxy-required networks, the VM driver accepts the same corporate egress proxy keys as the Podman driver: `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, and `proxy_ca_bundle`. The in-guest supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly. +For proxy-required networks, the VM driver accepts the same corporate egress proxy keys as the Podman driver: `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, and `proxy_ca_bundle`. The host supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly. -The settings reach the guest supervisor on its command line through a per-sandbox argument file the driver writes into the overlay upperdir on every launch, so a sandbox cannot select, alter, or disable the proxy from inside the guest — including through image `ENV`, the sandbox environment, or files baked into the image at the paths the driver uses. +The settings reach the host supervisor through driver-owned arguments, so a sandbox cannot select, alter, or disable the proxy from inside the guest. A proxy on the corporate network needs no special address and works on every VM sandbox. The guest's callback to the gateway never traverses the proxy. -A proxy on the gateway host itself works only for libkrun-backed (non-GPU) sandboxes, whose egress leaves through gvproxy: configure `https_proxy = "http://host.openshell.internal:"` rather than a `127.0.0.1` URL, because gvproxy NATs that alias to the host's `127.0.0.1`. GPU sandboxes use the QEMU/TAP backend, where `host.openshell.internal` resolves to the TAP host address and the driver's [host firewall rules](#host-firewall) allow the guest to reach only the gateway port on the host. The driver rejects a gateway-host proxy URL when a sandbox launches on QEMU instead of letting every CONNECT time out, so give GPU sandboxes a proxy address routable from the guest's masqueraded egress. +A proxy on the gateway host works for both libkrun and QEMU sandboxes. Configure +`https_proxy = "http://host.openshell.internal:"`; the host supervisor +normalizes that name to host loopback. -Because a microVM has no bind mounts or container secrets, the driver stages the credential (root-only) and the CA bundle into the per-sandbox overlay disk and removes them with the sandbox. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior. +The credential and private CA material stay with the host supervisor rather +than being staged into the guest. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior. ## Kubernetes Driver @@ -462,7 +408,7 @@ owner references or use the sandbox ServiceAccount. The operator namespace allowlist is a trust grant, not a tenant isolation mechanism. -Helm deployments set Kubernetes driver values through the chart. Canonical TOML places `namespace`, `service_account_name`, and `enable_user_namespaces` in `[openshell.drivers.kubernetes]`; schema version 2 rejects their historical `[openshell.gateway]` locations. +Helm deployments set Kubernetes driver values through the chart. For maintainer-level implementation details, refer to the [Kubernetes driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-kubernetes/README.md). @@ -470,27 +416,25 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA |---|---|---| | `compute_driver = "kubernetes"` | Not applicable | Select the Kubernetes compute driver. | | `[openshell.drivers.kubernetes].namespace` | `server.sandboxNamespace` | Set the namespace for sandbox resources. The Helm chart defaults to the release namespace when left empty. | -| `[openshell.drivers.kubernetes].service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the Kubernetes driver's TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. | -| `[openshell.drivers.kubernetes].enable_user_namespaces` | `server.enableUserNamespaces` | Enable Kubernetes user namespaces for sandbox pods. | +| `service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the Kubernetes driver's TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. | | `default_image` | `server.sandboxImage` | Set the default sandbox image. | -| `image_pull_policy` | `server.sandboxImagePullPolicy` | Set the canonical sandbox pull policy: `always`, `if_not_present`, or `never`. `newer` is Podman-only. | +| `image_pull_policy` | `server.sandboxImagePullPolicy` | Set the Kubernetes image pull policy for sandbox pods. | | `image_pull_secrets` | `server.sandboxImagePullSecrets` | Attach Kubernetes image-pull Secrets to sandbox pods. Managed mode copies these explicitly named Secrets from the configured source namespace into each workspace namespace. In shared and operator modes, the Secrets must already exist in the sandbox namespace. | | `[managed_ssh_ingress]` | `networkPolicy.enabled` | In managed mode, create an SSH ingress policy in every workspace namespace. Helm configures the gateway namespace and pod selector automatically. Operator mode leaves namespace policy management to the platform operator. | -| `grpc_endpoint` | `server.grpcEndpoint` | Set the gateway callback endpoint reachable from sandbox pods. Raw TOML and the standalone Kubernetes driver require an explicit endpoint because the sandbox namespace does not identify the gateway Service. Helm derives it from the release's gateway Service when the value is empty. | +| `grpc_endpoint` | `server.grpcEndpoint` | Set the gateway callback endpoint reachable from sandbox pods. | | `client_tls_secret_name` | `server.tls.clientTlsSecretName` | Mount sandbox client TLS materials from a Kubernetes secret. | -| `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Override the supervisor image that provides the `openshell-sandbox` binary. The default repository with an empty tag uses the version-pinned image built into the gateway. Changing the repository uses the effective gateway image tag, while setting a tag pins that version explicitly. | -| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the canonical supervisor pull policy: `always`, `if_not_present`, or `never`. `newer` is Podman-only. | -| `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. | -| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. | +| `sandbox_runtime_image` | `sandboxRuntime.image.repository` / `sandboxRuntime.image.tag` | Override the image that provides `openshell-sandbox`. The default repository with an empty tag uses the version pinned into the gateway. | +| `sandbox_runtime_image_pull_policy` | `sandboxRuntime.image.pullPolicy` | Set the Kubernetes image pull policy for the sandbox runtime image. | +| `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Override the image that provides `openshell-supervisor`. The default repository with an empty tag uses the version pinned into the gateway. | +| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. | +| `sandbox_runtime.network_policy_enforced` | `supervisor.sandboxRuntime.networkPolicyEnforced` | Acknowledge that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. This must be `true`. | +| `sandbox_runtime.boundary_port` | `supervisor.sandboxRuntime.boundaryPort` | Set the non-privileged TLS port used between the paired supervisor and sandbox Pods. | | `https_proxy` | `upstreamProxy.url` | Set the operator-owned `http://host:port` corporate forward proxy used for policy-approved TLS CONNECT egress. | | `no_proxy` | `upstreamProxy.noProxy` | Set destinations that bypass only the corporate proxy. OpenShell policy evaluation still applies. | -| `proxy_auth_secret_name` | `upstreamProxy.authSecret.name` | Set the existing Secret name in the sandbox namespace that contains the proxy credential. Requires `sidecar` topology. | -| `proxy_auth_secret_key` | `upstreamProxy.authSecret.key` | Set the Secret key containing the `user:pass` credential. Requires `sidecar` topology. | +| `proxy_auth_secret_name` | `upstreamProxy.authSecret.name` | Set the existing Secret name in the sandbox namespace that contains the proxy credential. The Secret mounts only in the supervisor Pod. | +| `proxy_auth_secret_key` | `upstreamProxy.authSecret.key` | Set the Secret key containing the `user:pass` credential. | | `proxy_auth_allow_insecure` | `upstreamProxy.authAllowInsecure` | Set `true` to acknowledge that Basic authentication to an HTTP proxy is cleartext. Required with a proxy credential Secret. | | `proxy_connect_by_hostname` | `upstreamProxy.connectByHostname` | Send hostnames rather than validated IPs in CONNECT requests. Use only when proxy ACLs require hostname targets. | -| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | Dedicated UID of at least `1000` used by the relaxed sidecar when process/binary-aware network policy is disabled. It must not match the workload UID. The default binary-aware sidecar runs as UID 0. The network init container exempts the effective sidecar UID from proxy redirection. | -| `sidecar.process_binary_aware_network_policy` | `supervisor.sidecar.processBinaryAwareNetworkPolicy` | Keep process/binary-aware network policy enabled in `sidecar` topology. The default runs the sidecar as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH`. Set false to run as `proxy_uid`, drop both capabilities, and enforce endpoint/L7 policy without matching `policy.binaries`. | -| `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/` for operator-managed profiles. | | `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. | | `workspace_storage_class` | `server.workspaceStorageClass` | Set the `StorageClass` for the workspace PVC. Empty (default) omits `storageClassName` and uses the cluster's default `StorageClass`. Set this on clusters with no default `StorageClass`, otherwise the workspace PVC stays `Pending` and the sandbox never starts. | | `sa_token_ttl_secs` | `server.sandboxJwt.k8sSaTokenTtlSecs` | Set the projected ServiceAccount token TTL used for the bootstrap token exchange. | @@ -502,30 +446,17 @@ remain limited to the explicitly configured TLS and image-pull Secret names. The driver creates copies only in gateway-owned managed namespaces. Do not reuse the gateway ServiceAccount for unrelated workloads. -In `combined` topology, the agent container carries the Linux capabilities -needed by the supervisor for network namespace setup, Landlock filesystem -policy, process privilege changes, and network policy enforcement. In `sidecar` -topology, the agent container runs as the resolved sandbox UID/GID with no added -Linux capabilities. A root init container performs the nftables setup, and the -long-running binary-aware sidecar runs as UID 0, drops default capabilities, -and adds `SYS_PTRACE` plus `DAC_READ_SEARCH` for workload process identity -resolution through shared `/proc`. The -`sidecar.process_binary_aware_network_policy = false` setting runs it as the -configured non-root `proxy_uid`, removes both capabilities, and relaxes network -policy to endpoint/L7 matching only. The -network sidecar owns gateway authentication and writes local policy/provider -state to the process supervisor over a local control socket, so the agent -container does not mount the sandbox bootstrap token or client TLS secret in -the default sidecar path. The provider environment is refreshed by the network -sidecar after settings polls and streamed to the process supervisor so future -child processes can see updated provider env without gateway access in the -agent container. -Sidecar mode keeps gateway session and SSH behavior. The process supervisor -applies Landlock filesystem policy and child seccomp filters where supported, -but it does not perform root-to-sandbox privilege dropping or supervisor -identity mount isolation. Network policy still runs in the sidecar, and sidecar -pods set `shareProcessNamespace: true` so the network sidecar can resolve -process/binary identity through `/proc/`. +The Kubernetes driver always places the sandbox runtime in the workload Pod and +the supervisor in a separate, directly managed Pod. The +workload Pod runs `openshell-sandbox` as the same non-root UID/GID as the agent +and requests no added Linux capabilities. The supervisor Pod runs +`openshell-supervisor`, authenticates to the gateway with a sandbox JWT, and +owns upstream connections. Both containers disable privilege escalation, drop +all capabilities, and use `RuntimeDefault` seccomp. The sandbox adds a nested +seccomp user-notification filter and Landlock restrictions before it launches +the agent. One namespace-wide, empty-egress `NetworkPolicy` is the mandatory +outer fence for all OpenShell workload Pods. It permits supervisor Pods to +reach sandbox listeners; TLS and JWT identity enforce the exact pairing. The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources from the Kubernetes SIG Apps [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`, so supported Agent Sandbox installations work without version-specific operator configuration. The Agent Sandbox controller turns those resources into sandbox pods and related storage. @@ -670,7 +601,7 @@ The resolved UID/GID appear in: ### VM Driver -The VM driver preserves an image-provided `sandbox` account when `sandbox_uid` and `sandbox_gid` are omitted. Images without that account use UID/GID `1000`. Explicit values in `[openshell.drivers.vm]` override the image account. Persisted overlays retain the UID/GID recorded when they were created. An unmarked overlay recovers identity from concrete overlay or prepared-image state, an explicit override, or the current image; the driver never assigns legacy `10001:10001` without persisted evidence. +The VM driver injects the sandbox UID into the rootfs guest's `/etc/passwd`, `/etc/group`, and `/etc/gshadow` during rootfs preparation. Default UID is `10001`; configure `sandbox_uid` in `[openshell.drivers.vm]` to use a different value. ### Custom Images diff --git a/docs/reference/support-matrix.mdx b/docs/reference/support-matrix.mdx index cfa5b69549..7671866deb 100644 --- a/docs/reference/support-matrix.mdx +++ b/docs/reference/support-matrix.mdx @@ -84,12 +84,18 @@ To override the default image references, use Helm values: ## Kernel Requirements -OpenShell enforces sandbox isolation through two Linux kernel security modules: +The sandbox boundary requires the following Linux kernel facilities, including +when it runs inside a container or microVM: | Module | Requirement | Details | | -------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| [Landlock LSM](https://docs.kernel.org/security/landlock.html) | Recommended | Enforces filesystem access restrictions at the kernel level. The `best_effort` compatibility mode uses the highest Landlock ABI the host kernel supports. The `hard_requirement` mode fails sandbox creation if the required ABI is unavailable. | -| seccomp | Required | Filters dangerous system calls. Available on all modern Linux kernels (3.17+). | +| [Landlock LSM](https://docs.kernel.org/security/landlock.html) | Required | ABI 3 or newer, introduced in Linux 6.2, with Landlock enabled. The mandatory baseline protects private channel and bootstrap files, including against truncation. A filesystem policy's `best_effort` setting never disables this baseline. | +| seccomp | Required | Nested user-notification filters and atomic `SECCOMP_IOCTL_NOTIF_ADDFD` with `SECCOMP_ADDFD_FLAG_SEND`, usable under the runtime's existing seccomp profile without added capabilities. The sandbox actively probes these operations before admitting the workload. | + +A kernel version alone does not establish support. A disabled Landlock LSM or a +runtime profile that blocks the required seccomp operations causes launch to +fail closed. An upstream Linux 6.2 or newer kernel provides the required +Landlock ABI; distribution backports must pass the same active qualification. On macOS, these kernel modules run inside the Docker Desktop Linux VM, not on the host kernel. diff --git a/docs/sandboxes/policies.mdx b/docs/sandboxes/policies.mdx index da2cb16eb4..f4202319ae 100644 --- a/docs/sandboxes/policies.mdx +++ b/docs/sandboxes/policies.mdx @@ -90,6 +90,12 @@ to forward conversation text. Malformed placeholder candidates and candidates longer than 4096 wire bytes are denied. Authentication headers and HTTP trailers retain their credential restrictions. +When `openshell-sandbox` and the supervisor run separately, the supervisor mediates DNS by +hostname. DNS sender identity is unavailable because native socket writes can +come from a process that inherited the socket or replaced its executable. +Resolving a name does not authorize a connection: the supervisor still checks +the destination and calling binary when the workload opens TCP traffic. + ## Supervisor Middleware Supervisor middleware can inspect, deny, or replace admitted HTTP request bodies and client WebSocket text messages before provider credentials are injected. Middleware selection is independent of the `network_policies` rule that admitted the traffic: each keyed `network_middlewares` entry matches the destination host through `endpoints.include` and `endpoints.exclude`. diff --git a/docs/security/best-practices.mdx b/docs/security/best-practices.mdx index bd7cd5ff61..a011677c23 100644 --- a/docs/security/best-practices.mdx +++ b/docs/security/best-practices.mdx @@ -159,14 +159,20 @@ Paths listed in `read_only` receive read-only access. Paths listed in `read_write` receive full access. All other paths are inaccessible. -Landlock setup runs in two phases. The parent supervisor probes the kernel ABI and opens the configured path file descriptors before forking. The child then applies the ruleset with `restrict_self()` after privilege drop. At startup, OpenShell emits the selected ABI version and the applied read-only and read-write rule counts so you can confirm what the kernel accepted. +The sandbox installs a mandatory Landlock baseline before running agent code. +It requires ABI 3 or newer, introduced in Linux 6.2, so denied writes include +file truncation. The baseline hides the private channel and bootstrap root from +the agent by granting access only to explicitly opened, permitted root children; +it never grants the filesystem root or follows a root-level symlink when +constructing that allowlist. Startup fails if this baseline cannot be enforced. +The configured filesystem policy adds restrictions on top of the baseline. | Aspect | Detail | |---|---| -| Default | `compatibility: best_effort`. Uses the highest kernel ABI available. Missing paths are skipped. If the kernel does not support Landlock or any configured path cannot be opened, the sandbox continues without those restrictions and emits a High-severity OCSF `DetectionFinding`. | -| What you can change | Set `compatibility: hard_requirement` to abort sandbox startup if Landlock is unavailable or any configured path cannot be opened. | -| Risk if relaxed | On kernels without Landlock (pre-5.13), or when all paths fail to open, the sandbox runs without kernel-level filesystem restrictions. The agent can access any file the process user can access. | -| Recommendation | Use `best_effort` for development. Use `hard_requirement` in environments where any gap in filesystem isolation is unacceptable. Treat High-severity Landlock findings as a signal to investigate the host kernel or the image. Run on Ubuntu 22.04+ or any kernel 5.13+ for Landlock support. | +| Default | The mandatory baseline always requires Landlock ABI 3 or newer. The additional filesystem policy defaults to `compatibility: best_effort`, which may skip unavailable policy paths and emits a High-severity OCSF `DetectionFinding` when that policy cannot be fully applied. | +| What you can change | Set `compatibility: hard_requirement` to reject startup when the additional filesystem policy cannot be applied. Neither setting relaxes the private-file baseline. | +| Risk if relaxed | A skipped additional policy can expose otherwise permitted image or workspace files to the agent. It cannot grant access to files excluded by the mandatory baseline. | +| Recommendation | Require Linux 6.2 or newer with Landlock enabled, or a qualified distribution backport providing ABI 3. Use `hard_requirement` where all configured filesystem restrictions must apply. Investigate High-severity Landlock findings rather than treating them as a supported degraded security mode. | ### Read-Only vs Read-Write Paths diff --git a/e2e/configs/gateway/docker.toml b/e2e/configs/gateway/docker.toml index 63e587ef62..3e872885e1 100644 --- a/e2e/configs/gateway/docker.toml +++ b/e2e/configs/gateway/docker.toml @@ -23,5 +23,6 @@ gateway_id = "openshell-e2e" default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" image_pull_policy = "if_not_present" sandbox_label = "openshell-e2e" +sandbox_runtime_image = "localhost/openshell/sandbox:e2e-vm" supervisor_image = "localhost/openshell/supervisor:e2e-vm" app_armor_profile = "Unconfined" diff --git a/e2e/configs/gateway/podman.toml b/e2e/configs/gateway/podman.toml index 6eff3b5615..a53708c61a 100644 --- a/e2e/configs/gateway/podman.toml +++ b/e2e/configs/gateway/podman.toml @@ -26,5 +26,6 @@ health_check_interval_secs = 10 network_name = "openshell-e2e" grpc_endpoint = "http://host.containers.internal:8080" ssh_socket_path = "/run/openshell/ssh.sock" +sandbox_runtime_image = "localhost/openshell/sandbox:e2e-vm" supervisor_image = "localhost/openshell/supervisor:e2e-vm" app_armor_profile = "Unconfined" diff --git a/e2e/docker/Dockerfile.external-kubernetes-gateway b/e2e/docker/Dockerfile.external-kubernetes-gateway index 4bda320440..20738a512a 100644 --- a/e2e/docker/Dockerfile.external-kubernetes-gateway +++ b/e2e/docker/Dockerfile.external-kubernetes-gateway @@ -8,6 +8,7 @@ FROM ${GATEWAY_BASE_IMAGE} ARG TARGETARCH ARG SUPERVISOR_IMAGE=ghcr.io/nvidia/openshell/supervisor:latest +ARG SANDBOX_RUNTIME_IMAGE=ghcr.io/nvidia/openshell/sandbox:latest COPY deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-gateway /usr/local/bin/openshell-gateway COPY deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-driver-kubernetes /usr/local/bin/openshell-driver-kubernetes @@ -21,8 +22,10 @@ ENV OPENSHELL_COMPUTE_DRIVER=kubernetes \ OPENSHELL_GRPC_ENDPOINT=http://openshell.openshell.svc.cluster.local:8080 \ OPENSHELL_SUPERVISOR_IMAGE=${SUPERVISOR_IMAGE} \ OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY=if_not_present \ + OPENSHELL_SANDBOX_RUNTIME_IMAGE=${SANDBOX_RUNTIME_IMAGE} \ + OPENSHELL_SANDBOX_RUNTIME_IMAGE_PULL_POLICY=if_not_present \ OPENSHELL_SUPERVISOR_SIDELOAD_METHOD=init-container \ - OPENSHELL_K8S_TOPOLOGY=combined + OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED=true USER 1000:1000 EXPOSE 8080 diff --git a/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh b/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh index 4d9d2eeff2..43ee52140a 100755 --- a/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh +++ b/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh @@ -3,7 +3,8 @@ # SPDX-License-Identifier: Apache-2.0 # Helm post-renderer for the external Kubernetes compute-driver smoke test. -# It keeps the test-only sidecar and Unix socket plumbing out of the chart. +# It keeps the test-only external-driver container and Unix socket plumbing out +# of the chart. set -euo pipefail diff --git a/e2e/parity/test.sh b/e2e/parity/test.sh index 93137a9860..45b9bf2097 100755 --- a/e2e/parity/test.sh +++ b/e2e/parity/test.sh @@ -215,6 +215,7 @@ if external: "external_driver_proxy": False, "external_driver_app_armor": False, "external_driver_environment": { + "XDG_DATA_HOME": f"/tmp/{variant}-driver-data", "OPENSHELL_COMPUTE_DRIVER_SOCKET": driver_socket, "OPENSHELL_PODMAN_SOCKET": podman_socket, "OPENSHELL_SANDBOX_IMAGE": sandbox_runtime, diff --git a/e2e/parity/verify-results.py b/e2e/parity/verify-results.py index 8f4942dda3..4e8a4619bb 100644 --- a/e2e/parity/verify-results.py +++ b/e2e/parity/verify-results.py @@ -262,6 +262,7 @@ def verify_variant( sandbox_id = launch.get("sandbox_image_id") sandbox_digest = launch.get("sandbox_image_digest") sandbox_runtime = launch.get("sandbox_runtime_image") + sandbox_boundary_image = launch.get("sandbox_boundary_image") sandbox_match = ( DIGEST_REFERENCE_RE.fullmatch(sandbox_runtime) if isinstance(sandbox_runtime, str) @@ -285,6 +286,10 @@ def verify_variant( sandbox_request == sandbox_runtime, f"{launch_path}: sandbox image request was not the resolved digest reference", ) + require( + isinstance(sandbox_boundary_image, str) and sandbox_boundary_image, + f"{launch_path}: sandbox boundary image is missing", + ) if not external: require( podman_config["default_image"] == sandbox_runtime @@ -369,6 +374,7 @@ def verify_variant( ) driver_environment = launch.get("external_driver_environment") expected_environment_keys = { + "XDG_DATA_HOME", "OPENSHELL_COMPUTE_DRIVER_SOCKET", "OPENSHELL_PODMAN_SOCKET", "OPENSHELL_SANDBOX_IMAGE", @@ -378,6 +384,7 @@ def verify_variant( "OPENSHELL_GATEWAY_PORT", "OPENSHELL_NETWORK_NAME", "OPENSHELL_STOP_TIMEOUT", + "OPENSHELL_SANDBOX_RUNTIME_IMAGE", "OPENSHELL_SUPERVISOR_IMAGE", "OPENSHELL_PODMAN_TLS_CA", "OPENSHELL_PODMAN_TLS_CERT", @@ -389,6 +396,10 @@ def verify_variant( and set(driver_environment) == expected_environment_keys, f"{launch_path}: external driver allowlisted environment is incomplete", ) + require( + Path(driver_environment["XDG_DATA_HOME"]).is_absolute(), + f"{launch_path}: external driver data directory is not absolute", + ) require( driver_environment["OPENSHELL_COMPUTE_DRIVER_SOCKET"] == podman_config["socket_path"], @@ -402,7 +413,7 @@ def verify_variant( f"{launch_path}: external driver Podman socket is not isolated", ) require( - driver_environment["OPENSHELL_SANDBOX_IMAGE"] == sandbox_runtime + driver_environment["OPENSHELL_SANDBOX_IMAGE"] == sandbox_request and driver_environment["OPENSHELL_SANDBOX_IMAGE_PULL_POLICY"] == expected_policy and driver_environment["OPENSHELL_HEALTH_CHECK_INTERVAL_SECS"] == 10 @@ -412,6 +423,8 @@ def verify_variant( and driver_environment["OPENSHELL_NETWORK_NAME"] and isinstance(driver_environment["OPENSHELL_STOP_TIMEOUT"], int) and driver_environment["OPENSHELL_STOP_TIMEOUT"] >= 0 + and driver_environment["OPENSHELL_SANDBOX_RUNTIME_IMAGE"] + == sandbox_boundary_image and driver_environment["OPENSHELL_SUPERVISOR_IMAGE"] == runtime_image and driver_environment["OPENSHELL_ENABLE_BIND_MOUNTS"] is True, f"{launch_path}: external driver allowlisted runtime inputs differ", diff --git a/e2e/python/test_sandbox_policy.py b/e2e/python/test_sandbox_policy.py index d2ce47e2e0..80d9963e02 100644 --- a/e2e/python/test_sandbox_policy.py +++ b/e2e/python/test_sandbox_policy.py @@ -1,9 +1,15 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 +"""Python SDK policy integration tests. + +The Rust E2E suites cover the complete transparent mediation pipeline. These +SDK-level checks retain the most important policy denials through normal +workload sockets, without relying on a workload-visible proxy endpoint. +""" + from __future__ import annotations -import json from typing import TYPE_CHECKING import grpc @@ -14,13 +20,9 @@ if TYPE_CHECKING: from collections.abc import Callable - from openshell import Sandbox, SandboxClient + from openshell import Sandbox -# ============================================================================= -# Policy helpers -# ============================================================================= - _BASE_FILESYSTEM = sandbox_pb2.FilesystemPolicy( include_workdir=True, read_only=["/usr", "/lib", "/etc", "/app", "/var/log", "/proc", "/dev/urandom"], @@ -28,20 +30,11 @@ ) _BASE_LANDLOCK = sandbox_pb2.LandlockPolicy(compatibility="best_effort") _BASE_PROCESS = sandbox_pb2.ProcessPolicy(run_as_user="sandbox", run_as_group="sandbox") -# Standard proxy address inside the sandbox network namespace -_PROXY_HOST = "10.200.0.1" -_PROXY_PORT = 3128 -# example.com keeps the wildcard test on public DNS while avoiding sslip.io -# rewrites that can resolve to internal ranges in CI. -_PUBLIC_WILDCARD_SUFFIX = "example.com" -_PUBLIC_WILDCARD_PATTERN = f"*.{_PUBLIC_WILDCARD_SUFFIX}" -_PUBLIC_WILDCARD_SUBDOMAIN = f"www.{_PUBLIC_WILDCARD_SUFFIX}" def _base_policy( network_policies: dict[str, sandbox_pb2.NetworkPolicyRule] | None = None, ) -> sandbox_pb2.SandboxPolicy: - """Build a sandbox policy with standard filesystem/process/landlock settings.""" return sandbox_pb2.SandboxPolicy( version=1, filesystem=_BASE_FILESYSTEM, @@ -51,316 +44,37 @@ def _base_policy( ) -def _policy_for_python_proxy_tests() -> sandbox_pb2.SandboxPolicy: - return _base_policy( - network_policies={ - "python": sandbox_pb2.NetworkPolicyRule( - name="python", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="api.openai.com", port=443) - ], - binaries=[ - sandbox_pb2.NetworkBinary(path="/sandbox/.uv/python/**/python*") - ], - ) - }, - ) - - -# ============================================================================= -# Shared test function factories -# -# cloudpickle serializes module-level functions by reference (module + name). -# The sandbox doesn't have this module, so deserialization fails. These -# factories return closures that cloudpickle serializes by value instead. -# ============================================================================= - - -def _proxy_connect(): - """Return a closure that sends a raw CONNECT and returns the status line.""" - - def fn(host, port): - import socket - - conn = socket.create_connection(("10.200.0.1", 3128), timeout=10) - try: - conn.sendall( - f"CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}\r\n\r\n".encode() - ) - return conn.recv(256).decode("latin1") - finally: - conn.close() - - return fn - - -def _proxy_connect_then_http(): - """Return a closure that CONNECTs, does TLS + HTTP, returns JSON string.""" - - def fn(host, port, method="GET", path="/"): - import json as _json - import socket - import ssl - - conn = socket.create_connection(("10.200.0.1", 3128), timeout=30) - try: - conn.sendall( - f"CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}\r\n\r\n".encode() - ) - connect_resp = conn.recv(256).decode("latin1") - if "200" not in connect_resp: - return _json.dumps( - {"connect_status": connect_resp.strip(), "http_status": 0} - ) - - sock = conn - if port == 443: - import os - - ctx = ssl.create_default_context() - ca_file = os.environ.get("SSL_CERT_FILE") - if ca_file: - ctx.load_verify_locations(ca_file) - sock = ctx.wrap_socket(conn, server_hostname=host) - - sock.settimeout(15) - - request = ( - f"{method} {path} HTTP/1.1\r\nHost: {host}\r\nConnection: close\r\n\r\n" - ) - sock.sendall(request.encode()) - - # Read response. The L7 relay loops back to parse the next - # request after relaying, so neither side closes — read until - # we have headers, then drain body with a short timeout. - data = b"" - while b"\r\n\r\n" not in data: - chunk = sock.recv(4096) - if not chunk: - break - data += chunk - - # Drain body with short timeout - sock.settimeout(2) - while len(data) < 65536: - try: - chunk = sock.recv(4096) - if not chunk: - break - data += chunk - except (socket.timeout, TimeoutError): - break - - response = data.decode("latin1", errors="replace") - status_line = response.split("\r\n")[0] if response else "" - status_code = ( - int(status_line.split()[1]) if len(status_line.split()) >= 2 else 0 - ) - - header_end = response.find("\r\n\r\n") - headers_raw = response[:header_end] if header_end > 0 else "" - body = response[header_end + 4 :] if header_end > 0 else "" - - return _json.dumps( - { - "connect_status": connect_resp.strip(), - "http_status": status_code, - "headers": headers_raw, - "body": body, - } - ) - finally: - conn.close() - - return fn - - -def _read_openshell_log(): - """Return a closure that reads the openshell log file(s). - - Since the sandbox uses a rolling file appender, logs are written to - date-stamped files like ``/var/log/openshell.YYYY-MM-DD.log`` instead - of a single ``/var/log/openshell.log``. This helper globs for all - matching files so tests work with both the legacy and rolling layouts. - """ - - def fn(): - import glob - - logs = [] - for path in sorted(glob.glob("/var/log/openshell*.log*")): - try: - with open(path) as f: - logs.append(f.read()) - except (FileNotFoundError, PermissionError): - pass - return "\n".join(logs) - - return fn - - -def _forward_proxy_with_server(): - """Return a closure that starts an HTTP server and sends a forward proxy request. - - The closure starts a minimal HTTP server on the given port inside the sandbox, - then sends a plain HTTP forward proxy request (non-CONNECT) through the sandbox - proxy and returns the raw response. - """ - - def fn(proxy_host, proxy_port, target_host, target_port): - import socket - import threading - import time - from http.server import BaseHTTPRequestHandler, HTTPServer - - class Handler(BaseHTTPRequestHandler): - def do_GET(self): - self.send_response(200) - body = b"forward-proxy-ok" - self.send_header("Content-Length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - def log_message(self, *args): - pass # suppress log output - - srv = HTTPServer(("0.0.0.0", int(target_port)), Handler) - threading.Thread(target=srv.handle_request, daemon=True).start() - time.sleep(0.5) - - conn = socket.create_connection((proxy_host, int(proxy_port)), timeout=10) - try: - req = ( - f"GET http://{target_host}:{target_port}/test HTTP/1.1\r\n" - f"Host: {target_host}:{target_port}\r\n\r\n" - ) - conn.sendall(req.encode()) - data = b"" - conn.settimeout(5) - try: - while True: - chunk = conn.recv(4096) - if not chunk: - break - data += chunk - except socket.timeout: - pass - return data.decode("latin1") - finally: - conn.close() - srv.server_close() - - return fn - - -def _forward_proxy_raw(): - """Return a closure that sends a forward proxy request (no server needed). - - For testing deny cases — sends the request and returns whatever the proxy - responds with. - """ - - def fn(proxy_host, proxy_port, target_url): - import socket - from urllib.parse import urlparse - - conn = socket.create_connection((proxy_host, int(proxy_port)), timeout=10) - try: - parsed = urlparse(target_url) - host_header = parsed.netloc or parsed.hostname - req = f"GET {target_url} HTTP/1.1\r\nHost: {host_header}\r\n\r\n" - conn.sendall(req.encode()) - return conn.recv(4096).decode("latin1") - finally: - conn.close() - - return fn - - -def _proxy_connect_then_http_with_server(): - """Return a closure that starts a local HTTP server and sends CONNECT+HTTP.""" - - def fn(proxy_host, proxy_port, target_host, target_port, method="GET", path="/"): - import json as _json +def _tcp_connect_errno(): + def connect(host: str, port: int) -> int: import socket - import threading - import time - from http.server import BaseHTTPRequestHandler, HTTPServer - - class Handler(BaseHTTPRequestHandler): - def do_GET(self): - self.send_response(200) - body = b"connect-server-ok" - self.send_header("Content-Length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - def do_POST(self): - self.send_response(200) - body = b"connect-server-ok" - self.send_header("Content-Length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - def log_message(self, *args): - pass - - srv = HTTPServer(("0.0.0.0", int(target_port)), Handler) - threading.Thread(target=srv.handle_request, daemon=True).start() - time.sleep(0.5) - conn = socket.create_connection((proxy_host, int(proxy_port)), timeout=10) try: - conn.sendall( - f"CONNECT {target_host}:{target_port} HTTP/1.1\r\nHost: {target_host}\r\n\r\n".encode() - ) - connect_resp = conn.recv(256).decode("latin1") - if "200" not in connect_resp: - return _json.dumps( - {"connect_status": connect_resp.strip(), "http_status": 0} - ) - - request = ( - f"{method} {path} HTTP/1.1\r\n" - f"Host: {target_host}:{target_port}\r\n" - "Connection: close\r\n\r\n" - ) - conn.sendall(request.encode()) - - data = b"" - conn.settimeout(5) - try: - while True: - chunk = conn.recv(4096) - if not chunk: - break - data += chunk - except socket.timeout: - pass - - response = data.decode("latin1", errors="replace") - status_line = response.split("\r\n")[0] if response else "" - status_code = ( - int(status_line.split()[1]) if len(status_line.split()) >= 2 else 0 - ) - - header_end = response.find("\r\n\r\n") - headers_raw = response[:header_end] if header_end > 0 else "" - body = response[header_end + 4 :] if header_end > 0 else "" - - return _json.dumps( - { - "connect_status": connect_resp.strip(), - "http_status": status_code, - "headers": headers_raw, - "body": body, - } + with socket.create_connection((host, port), timeout=5): + return 0 + except OSError as error: + return error.errno or -1 + + return connect + + +def _network_rule( + host: str, + port: int, + *, + binary: str = "/**", + allowed_ips: list[str] | None = None, +) -> sandbox_pb2.NetworkPolicyRule: + return sandbox_pb2.NetworkPolicyRule( + name="test_rule", + endpoints=[ + sandbox_pb2.NetworkEndpoint( + host=host, + port=port, + allowed_ips=allowed_ips or [], ) - finally: - conn.close() - srv.server_close() - - return fn + ], + binaries=[sandbox_pb2.NetworkBinary(path=binary)], + ) def test_policy_applies_to_exec_commands( @@ -379,8 +93,7 @@ def write_allowed_files() -> str: Path("/tmp/allowed.txt").write_text("ok") return "ok" - spec = datamodel_pb2.SandboxSpec(policy=_policy_for_python_proxy_tests()) - + spec = datamodel_pb2.SandboxSpec(policy=_base_policy()) with sandbox(spec=spec, delete_on_exit=True) as policy_sandbox: user_result = policy_sandbox.exec_python(current_user) assert user_result.exit_code == 0, user_result.stderr @@ -391,1661 +104,80 @@ def write_allowed_files() -> str: assert file_result.stdout.strip() == "ok" -def test_policy_blocks_unauthorized_proxy_connect( - sandbox: Callable[..., Sandbox], -) -> None: - spec = datamodel_pb2.SandboxSpec(policy=_policy_for_python_proxy_tests()) - with sandbox(spec=spec, delete_on_exit=True) as policy_sandbox: - proxy_result = policy_sandbox.exec_python( - _proxy_connect(), args=("example.com", 443) - ) - assert proxy_result.exit_code == 0, proxy_result.stderr - assert "403" in proxy_result.stdout - - -# ============================================================================= -# L4 Tests -- Connection-level OPA policy (host:port + binary identity) -# ============================================================================= -# -# L4-1: No network policies -> all CONNECT requests denied -# L4-2: Wildcard binary (/**) + specific endpoint -> any binary can connect -# but non-listed endpoints still denied -# L4-3: Binary-restricted policy -> matched binary allowed, others denied -# L4-4: Correct endpoint, wrong port -> denied -# L4-5: Multiple disjoint policies -> cross-policy access denied -# L4-6: Non-CONNECT HTTP method -> rejected with 405 -# L4-7: Log fields are structured correctly (action, binary, policy, engine) -# ============================================================================= - - -def test_l4_no_policy_denies_all( - sandbox: Callable[..., Sandbox], -) -> None: - """L4-1: No matching endpoint in any network policy -> CONNECT denied. - - We need at least one network policy so the proxy and network namespace - start (empty network_policies disables networking entirely, including - socket syscalls). The policy allows python->example.com:443 but - api.anthropic.com:443 should still be denied. - """ - policy = _base_policy( - network_policies={ - "other": sandbox_pb2.NetworkPolicyRule( - name="other", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="example.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_l4_wildcard_binary_allows_any_binary( - sandbox: Callable[..., Sandbox], -) -> None: - """L4-2: Wildcard binary glob allows python (and anything else) to connect.""" - policy = _base_policy( - network_policies={ - "wildcard": sandbox_pb2.NetworkPolicyRule( - name="wildcard", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="api.anthropic.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Python can reach the allowed endpoint - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout - - # Non-listed endpoint is still denied - result = sb.exec_python(_proxy_connect(), args=("example.com", 443)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_l4_binary_restricted_denies_wrong_binary( - sandbox: Callable[..., Sandbox], -) -> None: - """L4-3: Policy restricted to specific binary denies others. - - Policy allows /usr/bin/curl -> api.anthropic.com:443. - Python (exec_python uses python) should be denied. - """ - policy = _base_policy( - network_policies={ - "curl_only": sandbox_pb2.NetworkPolicyRule( - name="curl_only", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="api.anthropic.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/usr/bin/curl")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Python is NOT the allowed binary -> denied - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_l4_wrong_port_denied( - sandbox: Callable[..., Sandbox], -) -> None: - """L4-4: Correct host but wrong port -> denied.""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="api.anthropic.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Port 443 -> allowed - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout - - # Port 80 -> denied - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 80)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_l4_cross_policy_denied( - sandbox: Callable[..., Sandbox], -) -> None: - """L4-5: Multiple disjoint policies -> cross-policy access denied. - - Policy A: python -> api.anthropic.com:443 - Policy B: curl -> example.com:443 - Python should NOT reach example.com (that's curl's policy). - """ - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="api.anthropic.com", port=443), - ], - binaries=[ - sandbox_pb2.NetworkBinary(path="/sandbox/.uv/python/**/python*") - ], - ), - "other": sandbox_pb2.NetworkPolicyRule( - name="other", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="example.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/usr/bin/curl")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Python -> its own policy endpoint: allowed - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout - - # Python -> curl's policy endpoint: denied - result = sb.exec_python(_proxy_connect(), args=("example.com", 443)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_l4_non_connect_method_rejected( - sandbox: Callable[..., Sandbox], -) -> None: - """L4-6: Non-CONNECT HTTP method -> rejected with 403.""" - - def send_get_to_proxy() -> str: - import socket - - conn = socket.create_connection(("10.200.0.1", 3128), timeout=10) - try: - conn.sendall( - b"GET http://example.com/ HTTP/1.1\r\nHost: example.com\r\n\r\n" - ) - return conn.recv(256).decode("latin1") - finally: - conn.close() - - policy = _base_policy( - network_policies={ - "any": sandbox_pb2.NetworkPolicyRule( - name="any", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="example.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(send_get_to_proxy) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_l4_log_fields( - sandbox: Callable[..., Sandbox], -) -> None: - """L4-7: CONNECT log contains structured fields for allow and deny.""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="api.anthropic.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Generate an allow - sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - # Generate a deny - sb.exec_python(_proxy_connect(), args=("example.com", 443)) - - log_result = sb.exec_python(_read_openshell_log()) - assert log_result.exit_code == 0, log_result.stderr - log = log_result.stdout - - # Verify OCSF shorthand fields in allow line - assert "ALLOWED" in log, "Expected ALLOWED in OCSF shorthand" - assert "api.anthropic.com" in log, "Expected destination host in log" - assert "engine:opa" in log, "Expected engine:opa in log context" - - # Verify deny line exists - assert "DENIED" in log, "Expected DENIED in OCSF shorthand" - - -# ============================================================================= -# SSRF Tests -- Internal IP rejection (defense-in-depth) -# -# The proxy resolves DNS before connecting and rejects any destination that -# resolves to a loopback, RFC1918 private, or link-local address. These -# tests verify the check works even when OPA policy explicitly allows the -# internal endpoint. -# -# SSRF-1: Loopback (127.0.0.1) blocked despite OPA allow -# SSRF-2: Cloud metadata (169.254.169.254) blocked despite OPA allow -# SSRF-3: Log shows "internal address" block reason -# ============================================================================= - - -def test_ssrf_blocks_loopback_despite_policy_allow( - sandbox: Callable[..., Sandbox], -) -> None: - """SSRF-1: CONNECT to 127.0.0.1 blocked even with explicit OPA allow.""" - policy = _base_policy( - network_policies={ - "internal": sandbox_pb2.NetworkPolicyRule( - name="internal", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="127.0.0.1", port=80), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], +@pytest.mark.parametrize( + ("policy", "host", "port"), + [ + (_base_policy(), "1.1.1.1", 443), + ( + _base_policy( + {"test_rule": _network_rule("1.1.1.1", 80)}, ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=("127.0.0.1", 80)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_ssrf_blocks_metadata_endpoint_despite_policy_allow( - sandbox: Callable[..., Sandbox], -) -> None: - """SSRF-2: CONNECT to 169.254.169.254 blocked even with explicit OPA allow.""" - policy = _base_policy( - network_policies={ - "metadata": sandbox_pb2.NetworkPolicyRule( - name="metadata", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="169.254.169.254", port=80), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], + "1.1.1.1", + 443, + ), + ( + _base_policy( + {"test_rule": _network_rule("1.1.1.1", 443, binary="/bin/false")}, ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=("169.254.169.254", 80)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -def test_ssrf_log_shows_blocked_address( + "1.1.1.1", + 443, + ), + ], + ids=["no-policy", "wrong-port", "wrong-binary"], +) +def test_transparent_tcp_policy_denies_unauthorized_connections( sandbox: Callable[..., Sandbox], + policy: sandbox_pb2.SandboxPolicy, + host: str, + port: int, ) -> None: - """SSRF-3: Proxy log includes block reason when SSRF check fires. + import errno - Loopback addresses are always-blocked. Since implicit_allowed_ips_for_ip_host - now skips always-blocked hosts, 127.0.0.1 falls through to the default - resolve_and_reject_internal path which blocks it as an internal address. - The shorthand log should include 'ssrf' and a '[reason:' tag for denied events. - """ - policy = _base_policy( - network_policies={ - "internal": sandbox_pb2.NetworkPolicyRule( - name="internal", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="127.0.0.1", port=80), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - sb.exec_python(_proxy_connect(), args=("127.0.0.1", 80)) - - log_result = sb.exec_python(_read_openshell_log()) - assert log_result.exit_code == 0, log_result.stderr - log = log_result.stdout - # OCSF shorthand uses "engine:ssrf" for SSRF blocks - assert "engine:ssrf" in log.lower() or "ssrf" in log.lower(), ( - f"Expected SSRF block indicator in proxy log, got:\n{log}" - ) - # Shorthand for denied events should include [reason:...] tag - assert "[reason:" in log.lower(), ( - f"Expected [reason:] tag in denied event shorthand, got:\n{log}" + with sandbox(spec=spec, delete_on_exit=True) as policy_sandbox: + result = policy_sandbox.exec_python( + _tcp_connect_errno(), + args=(host, port), ) - -# ============================================================================= -# SSRF Tests -- allowed_ips (CIDR-based private IP access) -# -# When an endpoint has `allowed_ips`, the proxy validates resolved IPs against -# the CIDR allowlist instead of blanket-blocking all private IPs. -# Loopback and link-local remain always-blocked regardless. -# -# SSRF-4: Private IP allowed with allowed_ips (mode 2: host + IPs) -# SSRF-5: Private IP allowed with allowed_ips (mode 3: IPs only, no host) -# SSRF-6: Private IP still blocked without allowed_ips (default behavior) -# SSRF-7: Loopback always blocked even with allowed_ips covering 127.0.0.0/8 -# ============================================================================= + assert result.exit_code == 0, result.stderr + assert int(result.stdout.strip()) in {errno.EACCES, errno.EPERM} -def test_ssrf_allowed_ips_permits_private_ip( +def test_conflicting_destination_metadata_is_rejected( sandbox: Callable[..., Sandbox], ) -> None: - """SSRF-4: CONNECT to private IP succeeds when allowed_ips covers it. - - Uses 10.200.0.1 (the proxy's own host-side veth IP) as the target. - The connection attempt will fail at the TCP level (nothing listening on - port 19999) but the proxy should return 200 Connection Established - instead of 403, proving the SSRF check passed. - """ + """The gateway rejects ambiguous endpoint pinning before launch.""" + target = "10.200.0.2" + port = 19876 policy = _base_policy( network_policies={ - "internal": sandbox_pb2.NetworkPolicyRule( - name="internal", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="10.200.0.1", - port=19999, - allowed_ips=["10.200.0.0/24"], - ), - ], + "user_rule": sandbox_pb2.NetworkPolicyRule( + name="user_rule", + endpoints=[sandbox_pb2.NetworkEndpoint(host=target, port=port)], binaries=[sandbox_pb2.NetworkBinary(path="/**")], ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=("10.200.0.1", 19999)) - assert result.exit_code == 0, result.stderr - # Should get 200 (connection established) — not 403. - # The actual TCP connection may fail but the SSRF check passed. - assert "403" not in result.stdout, ( - "Expected SSRF check to pass with allowed_ips, but got 403" - ) - - -def test_ssrf_allowed_ips_hostless_permits_private_ip( - sandbox: Callable[..., Sandbox], -) -> None: - """SSRF-5: CONNECT to private IP succeeds with hostless allowed_ips (mode 3). - - An endpoint with no host but with allowed_ips matches any hostname on the - given port. The resolved IP must be in the allowlist. - """ - policy = _base_policy( - network_policies={ - "private_net": sandbox_pb2.NetworkPolicyRule( - name="private_net", + "approved_rule": sandbox_pb2.NetworkPolicyRule( + name="approved_rule", endpoints=[ sandbox_pb2.NetworkEndpoint( - # No host — matches any hostname on this port - port=19999, + host=target, + port=port, allowed_ips=["10.200.0.0/24"], - ), + ) ], binaries=[sandbox_pb2.NetworkBinary(path="/**")], ), - }, + } ) spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=("10.200.0.1", 19999)) - assert result.exit_code == 0, result.stderr - assert "403" not in result.stdout, ( - "Expected SSRF check to pass with hostless allowed_ips, but got 403" - ) - - -def test_ssrf_private_ip_allowed_with_literal_ip_host( - sandbox: Callable[..., Sandbox], -) -> None: - """SSRF-6: Private IP allowed when policy host is a literal IP address. - - When the policy endpoint host is a literal IP, the user has explicitly - declared intent. The proxy synthesizes an implicit allowed_ips entry, - so the CONNECT succeeds (200) even without explicit allowed_ips. - """ - policy = _base_policy( - network_policies={ - "internal": sandbox_pb2.NetworkPolicyRule( - name="internal", - endpoints=[ - # No allowed_ips — but host is a literal IP, so implicit - sandbox_pb2.NetworkEndpoint(host="10.200.0.1", port=19999), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=("10.200.0.1", 19999)) - assert result.exit_code == 0, result.stderr - # Should not get 403 — the SSRF check should pass. - # The actual TCP connection may fail (nothing listening on 19999) - # so recv() might return empty, but 403 must not appear. - assert "403" not in result.stdout, ( - "Expected SSRF check to pass for literal IP host, but got 403" - ) - - -def test_ssrf_loopback_blocked_even_with_allowed_ips( - sandbox: Callable[..., Sandbox], -) -> None: - """SSRF-7: Loopback always blocked even when allowed_ips covers 127.0.0.0/8. - - With always-blocked validation, parse_allowed_ips rejects 127.0.0.0/8 at - connection time (returns Err), so the proxy treats this as "invalid - allowed_ips in policy" and returns 403. The end result is the same: - loopback is never reachable. - """ - policy = _base_policy( - network_policies={ - "internal": sandbox_pb2.NetworkPolicyRule( - name="internal", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="127.0.0.1", - port=80, - allowed_ips=["127.0.0.0/8"], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=("127.0.0.1", 80)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout, ( - "Expected loopback to be blocked even with allowed_ips" - ) - - -# ============================================================================= -# L7 Tests -- TLS termination HTTPS inspection (Phase 2: tls=terminate) -# -# These tests use api.anthropic.com:443 as a real HTTPS endpoint since the -# sandbox already has proxy connectivity. The ephemeral CA is trusted via -# SSL_CERT_FILE injected into the sandbox environment. -# -# L7-T1: TLS terminate + access=full allows HTTPS requests through -# L7-T2: TLS terminate + access=read-only denies HTTPS POST (enforce) -# L7-T3: TLS terminate + enforcement=audit logs but allows HTTPS POST -# L7-T4: TLS terminate with explicit path rules -# L7-T5: CA trust store is injected (SSL_CERT_FILE, NODE_EXTRA_CA_CERTS) -# L7-T6: L7 deny response is valid JSON with expected fields -# L7-T7: L7 request logging includes structured fields -# L7-T8: Port 443 + protocol=rest without tls=terminate warns (L7 not evaluated) -# L7-T9: Query matcher glob/any allows and denies as expected -# L7-T10: Rule without query matcher allows any query params -# ============================================================================= - - -def test_l7_tls_full_access_allows_all( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T1: TLS terminate + access=full allows HTTPS GET through.""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - tls="terminate", - enforcement="enforce", - access="full", - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "GET", "/v1/models"), - ) - assert result.exit_code == 0, result.stderr - resp = json.loads(result.stdout) - assert "200" in resp["connect_status"] - # Upstream returns a real response (likely 401 without auth, but not 403 from proxy) - assert resp["http_status"] != 0 - assert resp["http_status"] != 403 # Not a proxy deny - - -def test_l7_tls_read_only_denies_post( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T2: TLS terminate + access=read-only denies HTTPS POST (enforce).""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - tls="terminate", - enforcement="enforce", - access="read-only", - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # GET should be allowed through (read-only permits GET) - get_result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "GET", "/v1/models"), - ) - assert get_result.exit_code == 0, get_result.stderr - get_resp = json.loads(get_result.stdout) - assert get_resp["http_status"] != 403 # Not proxy denied - - # POST should be denied by the proxy with 403 - post_result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "POST", "/v1/messages"), - ) - assert post_result.exit_code == 0, post_result.stderr - post_resp = json.loads(post_result.stdout) - assert post_resp["http_status"] == 403 - assert "policy_denied" in post_resp["body"] - - -def test_l7_tls_audit_mode_allows_but_logs( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T3: TLS terminate + enforcement=audit logs but allows HTTPS POST.""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - tls="terminate", - enforcement="audit", - access="read-only", - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # POST goes through in audit mode (not denied) - post_result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "POST", "/v1/messages"), - ) - assert post_result.exit_code == 0, post_result.stderr - post_resp = json.loads(post_result.stdout) - # Should NOT be 403 from proxy -- traffic is forwarded - assert post_resp["http_status"] != 403 - - # Log should contain audit decision - log_result = sb.exec_python(_read_openshell_log()) - assert log_result.exit_code == 0, log_result.stderr - log = log_result.stdout - # OCSF shorthand: audit decisions show as ALLOWED (audit mode allows through) - assert "HTTP:" in log, "Expected OCSF HTTP activity event in log" - assert "ALLOWED" in log, "Expected ALLOWED for audit-mode decision" - - -def test_l7_tls_explicit_path_rules( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T4: TLS terminate with explicit path rules.""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - tls="terminate", - enforcement="enforce", - rules=[ - sandbox_pb2.L7Rule( - allow=sandbox_pb2.L7Allow(method="GET", path="/v1/**"), - ), - ], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # GET /v1/models -> allowed (matches /v1/**) - get_result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "GET", "/v1/models"), - ) - assert get_result.exit_code == 0, get_result.stderr - get_resp = json.loads(get_result.stdout) - assert get_resp["http_status"] != 403 - - # POST /v1/messages -> denied (no POST rule) - post_result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "POST", "/v1/messages"), - ) - assert post_result.exit_code == 0, post_result.stderr - post_resp = json.loads(post_result.stdout) - assert post_resp["http_status"] == 403 - - # GET /v2/anything -> denied (path doesn't match /v1/**) - v2_result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "GET", "/v2/anything"), - ) - assert v2_result.exit_code == 0, v2_result.stderr - v2_resp = json.loads(v2_result.stdout) - assert v2_resp["http_status"] == 403 - - -def test_l7_tls_ca_trust_store_injected( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T5: Sandbox CA is injected into trust store environment variables.""" - - def check_ca_env() -> str: - import json as _json - import os - - return _json.dumps( - { - "SSL_CERT_FILE": os.environ.get("SSL_CERT_FILE", ""), - "NODE_EXTRA_CA_CERTS": os.environ.get("NODE_EXTRA_CA_CERTS", ""), - "REQUESTS_CA_BUNDLE": os.environ.get("REQUESTS_CA_BUNDLE", ""), - "CURL_CA_BUNDLE": os.environ.get("CURL_CA_BUNDLE", ""), - "ca_cert_exists": os.path.exists("/etc/openshell-tls/openshell-ca.pem"), - "bundle_exists": os.path.exists("/etc/openshell-tls/ca-bundle.pem"), - } - ) - - policy = _base_policy( - network_policies={ - "any": sandbox_pb2.NetworkPolicyRule( - name="any", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="example.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(check_ca_env) - assert result.exit_code == 0, result.stderr - env = json.loads(result.stdout) - assert env["ca_cert_exists"], "openshell-ca.pem should exist" - assert env["bundle_exists"], "ca-bundle.pem should exist" - assert "openshell-tls" in env["SSL_CERT_FILE"] - assert "openshell-tls" in env["NODE_EXTRA_CA_CERTS"] - - -def test_l7_tls_deny_response_format( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T6: L7 deny response is valid JSON with expected fields.""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - tls="terminate", - enforcement="enforce", - access="read-only", - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "DELETE", "/v1/anything"), - ) - assert result.exit_code == 0, result.stderr - resp = json.loads(result.stdout) - assert resp["http_status"] == 403 - - # Verify response headers - assert "X-OpenShell-Policy" in resp["headers"] - assert "application/json" in resp["headers"] - - # Verify JSON body structure - body = json.loads(resp["body"]) - assert body["error"] == "policy_denied" - assert "policy" in body - assert "rule" in body - assert "detail" in body - - -def test_l7_tls_log_fields( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T7: L7 request logging includes structured fields.""" - policy = _base_policy( - network_policies={ - "anthropic": sandbox_pb2.NetworkPolicyRule( - name="anthropic", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - tls="terminate", - enforcement="enforce", - access="full", - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - sb.exec_python( - _proxy_connect_then_http(), - args=("api.anthropic.com", 443, "GET", "/v1/models"), - ) - - log_result = sb.exec_python(_read_openshell_log()) - assert log_result.exit_code == 0, log_result.stderr - log = log_result.stdout - - # OCSF shorthand: L7 requests show as HTTP:method events - assert "HTTP:" in log, "Expected OCSF HTTP activity event in log" - assert "ALLOWED" in log or "DENIED" in log, "Expected L7 decision in log" - assert "policy:" in log, "Expected policy context in log" - - -def test_l7_query_matchers_enforced( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T9: Query matcher glob/any allows and denies as expected.""" - policy = _base_policy( - network_policies={ - "query_api": sandbox_pb2.NetworkPolicyRule( - name="query_api", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - protocol="rest", - enforcement="enforce", - allowed_ips=["10.200.0.0/24"], - rules=[ - sandbox_pb2.L7Rule( - allow=sandbox_pb2.L7Allow( - method="GET", - path="/download", - query={ - "tag": sandbox_pb2.L7QueryMatcher(glob="foo-*"), - }, - ), - ), - sandbox_pb2.L7Rule( - allow=sandbox_pb2.L7Allow( - method="GET", - path="/search", - query={ - "tag": sandbox_pb2.L7QueryMatcher( - any=["foo-*", "bar-*"] - ), - }, - ), - ), - ], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - allowed = sb.exec_python( - _proxy_connect_then_http_with_server(), - args=( - _PROXY_HOST, - _PROXY_PORT, - _SANDBOX_IP, - _FORWARD_PROXY_PORT, - "GET", - "/download?tag=foo-a&tag=foo-b", - ), - ) - assert allowed.exit_code == 0, allowed.stderr - allowed_resp = json.loads(allowed.stdout) - assert "200" in allowed_resp["connect_status"] - assert allowed_resp["http_status"] == 200 - assert "connect-server-ok" in allowed_resp["body"] - - denied = sb.exec_python( - _proxy_connect_then_http_with_server(), - args=( - _PROXY_HOST, - _PROXY_PORT, - _SANDBOX_IP, - _FORWARD_PROXY_PORT, - "GET", - "/download?tag=foo-a&tag=evil", - ), - ) - assert denied.exit_code == 0, denied.stderr - denied_resp = json.loads(denied.stdout) - assert denied_resp["http_status"] == 403 - assert "policy_denied" in denied_resp["body"] - - any_allowed = sb.exec_python( - _proxy_connect_then_http_with_server(), - args=( - _PROXY_HOST, - _PROXY_PORT, - _SANDBOX_IP, - _FORWARD_PROXY_PORT, - "GET", - "/search?tag=foo-a&tag=bar-b", - ), - ) - assert any_allowed.exit_code == 0, any_allowed.stderr - any_resp = json.loads(any_allowed.stdout) - assert any_resp["http_status"] == 200 - assert "connect-server-ok" in any_resp["body"] - - missing_required = sb.exec_python( - _proxy_connect_then_http_with_server(), - args=( - _PROXY_HOST, - _PROXY_PORT, - _SANDBOX_IP, - _FORWARD_PROXY_PORT, - "GET", - "/download?slug=skill-1", - ), - ) - assert missing_required.exit_code == 0, missing_required.stderr - missing_resp = json.loads(missing_required.stdout) - assert missing_resp["http_status"] == 403 - assert "policy_denied" in missing_resp["body"] - - -def test_l7_rule_without_query_matcher_allows_any_query_params( - sandbox: Callable[..., Sandbox], -) -> None: - """L7-T10: Rule without query matcher allows any query params.""" - policy = _base_policy( - network_policies={ - "query_optional": sandbox_pb2.NetworkPolicyRule( - name="query_optional", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - protocol="rest", - enforcement="enforce", - allowed_ips=["10.200.0.0/24"], - rules=[ - sandbox_pb2.L7Rule( - allow=sandbox_pb2.L7Allow( - method="GET", - path="/download", - ), - ), - ], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _proxy_connect_then_http_with_server(), - args=( - _PROXY_HOST, - _PROXY_PORT, - _SANDBOX_IP, - _FORWARD_PROXY_PORT, - "GET", - "/download?tag=anything&slug=any-value", - ), - ) - assert result.exit_code == 0, result.stderr - resp = json.loads(result.stdout) - assert "200" in resp["connect_status"] - assert resp["http_status"] == 200 - assert "connect-server-ok" in resp["body"] - - -# ============================================================================= -# Forward proxy tests (plain HTTP, non-CONNECT) -# ============================================================================= - -# The sandbox's own IP within the network namespace -_SANDBOX_IP = "10.200.0.2" -_FORWARD_PROXY_PORT = 19876 - - -def test_forward_proxy_allows_private_ip_with_allowed_ips( - sandbox: Callable[..., Sandbox], -) -> None: - """FWD-1: Forward proxy GET to private IP with allowed_ips succeeds. - - Starts an HTTP server inside the sandbox, sends a plain forward proxy - request through the sandbox proxy, and verifies the response is relayed. - """ - policy = _base_policy( - network_policies={ - "internal_http": sandbox_pb2.NetworkPolicyRule( - name="internal_http", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - allowed_ips=["10.200.0.0/24"], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _forward_proxy_with_server(), - args=(_PROXY_HOST, _PROXY_PORT, _SANDBOX_IP, _FORWARD_PROXY_PORT), - ) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout, ( - f"Expected 200 in forward proxy response, got: {result.stdout}" - ) - assert "forward-proxy-ok" in result.stdout, ( - f"Expected response body relayed, got: {result.stdout}" - ) - - -def test_forward_proxy_allows_private_ip_host_without_allowed_ips( - sandbox: Callable[..., Sandbox], -) -> None: - """FWD-2: Forward proxy to literal IP host without allowed_ips -> 200. - - When the policy host field is a literal IP address, the user has explicitly - declared intent to allow that destination. The SSRF guard synthesizes an - implicit allowed_ips entry, so explicit allowed_ips is not required. - """ - policy = _base_policy( - network_policies={ - "internal_http": sandbox_pb2.NetworkPolicyRule( - name="internal_http", - endpoints=[ - # No allowed_ips — but host is a literal IP, so implicit - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _forward_proxy_with_server(), - args=(_PROXY_HOST, _PROXY_PORT, _SANDBOX_IP, _FORWARD_PROXY_PORT), - ) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout, ( - f"Expected 200 for literal IP host, got: {result.stdout}" - ) - assert "forward-proxy-ok" in result.stdout, ( - f"Expected response body relayed, got: {result.stdout}" - ) - - -def test_forward_proxy_rejects_https_scheme( - sandbox: Callable[..., Sandbox], -) -> None: - """FWD-3: Forward proxy with https:// scheme -> 400. - - HTTPS must use CONNECT tunneling, not forward proxy. - """ - policy = _base_policy( - network_policies={ - "internal_http": sandbox_pb2.NetworkPolicyRule( - name="internal_http", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - allowed_ips=["10.200.0.0/24"], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _forward_proxy_raw(), - args=( - _PROXY_HOST, - _PROXY_PORT, - f"https://{_SANDBOX_IP}:{_FORWARD_PROXY_PORT}/test", - ), - ) - assert result.exit_code == 0, result.stderr - assert "400" in result.stdout, ( - f"Expected 400 for HTTPS forward proxy, got: {result.stdout}" - ) - - -def test_forward_proxy_denied_no_policy_match( - sandbox: Callable[..., Sandbox], -) -> None: - """FWD-4: Forward proxy to unmatched host:port -> 403.""" - policy = _base_policy( - network_policies={ - "other": sandbox_pb2.NetworkPolicyRule( - name="other", - endpoints=[ - # Policy for a different host/port - sandbox_pb2.NetworkEndpoint( - host="10.200.0.1", - port=9999, - allowed_ips=["10.200.0.0/24"], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _forward_proxy_raw(), - args=( - _PROXY_HOST, - _PROXY_PORT, - f"http://{_SANDBOX_IP}:{_FORWARD_PROXY_PORT}/test", - ), - ) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout, ( - f"Expected 403 for unmatched policy, got: {result.stdout}" - ) - - -def test_forward_proxy_public_ip_denied( - sandbox: Callable[..., Sandbox], -) -> None: - """FWD-5: Forward proxy to public IP -> 403. - - Even with allowed_ips, forward proxy is restricted to private IPs. - Plain HTTP should never traverse the public internet. - """ - policy = _base_policy( - network_policies={ - "public": sandbox_pb2.NetworkPolicyRule( - name="public", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="example.com", - port=80, - allowed_ips=["93.184.0.0/16"], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _forward_proxy_raw(), - args=(_PROXY_HOST, _PROXY_PORT, "http://example.com/"), - ) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout, ( - f"Expected 403 for public IP forward proxy, got: {result.stdout}" - ) - - -def test_forward_proxy_log_fields( - sandbox: Callable[..., Sandbox], -) -> None: - """FWD-6: Forward proxy requests produce structured FORWARD log lines.""" - policy = _base_policy( - network_policies={ - "internal_http": sandbox_pb2.NetworkPolicyRule( - name="internal_http", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - allowed_ips=["10.200.0.0/24"], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Trigger an allowed forward proxy request (with server) - sb.exec_python( - _forward_proxy_with_server(), - args=(_PROXY_HOST, _PROXY_PORT, _SANDBOX_IP, _FORWARD_PROXY_PORT), - ) - # Trigger a denied forward proxy request (no allowed_ips match) - sb.exec_python( - _forward_proxy_raw(), - args=( - _PROXY_HOST, - _PROXY_PORT, - "http://example.com/", - ), - ) - # Read the log - result = sb.exec_python(_read_openshell_log()) - assert result.exit_code == 0, result.stderr - log = result.stdout - - # OCSF shorthand: FORWARD requests show as HTTP:method events - assert "HTTP:" in log, "Expected OCSF HTTP activity event for FORWARD request" - assert "ALLOWED" in log, "Expected ALLOWED for forward proxy allow" - assert f"{_SANDBOX_IP}" in log, "Expected destination IP in FORWARD log" - - -# ============================================================================= -# Baseline filesystem path enrichment tests (BFS-*) -# ============================================================================= - - -def _verify_sandbox_functional(): - """Return a closure that verifies basic sandbox functionality.""" - - def fn(): - import json - import os - import sys - - checks = {} - # Can resolve DNS config - checks["resolv_conf"] = os.path.exists("/etc/resolv.conf") - # Can access shared libraries - checks["lib_exists"] = os.path.isdir("/usr/lib") - # Python interpreter works - checks["python_version"] = sys.version - # Can write to /tmp - tmp_path = "/tmp/enrichment_test.txt" - try: - with open(tmp_path, "w") as f: - f.write("ok") - with open(tmp_path) as f: - checks["tmp_write"] = f.read() == "ok" - os.unlink(tmp_path) - except Exception as e: - checks["tmp_write"] = str(e) - # Can write to /sandbox - sb_path = "/sandbox/enrichment_test.txt" - try: - with open(sb_path, "w") as f: - f.write("ok") - with open(sb_path) as f: - checks["sandbox_write"] = f.read() == "ok" - os.unlink(sb_path) - except Exception as e: - checks["sandbox_write"] = str(e) - # Can read openshell log (rolling appender writes date-stamped files) - import glob - - checks["var_log"] = len(glob.glob("/var/log/openshell*.log*")) > 0 - return json.dumps(checks) - - return fn - - -def test_baseline_enrichment_missing_filesystem_policy( - sandbox: Callable[..., Sandbox], -) -> None: - """BFS-1: Sandbox with network_policies but NO filesystem_policy should - come up and function correctly thanks to baseline path enrichment.""" - # Intentionally omit filesystem, landlock, and process fields — - # only provide network_policies. - spec = datamodel_pb2.SandboxSpec( - policy=sandbox_pb2.SandboxPolicy( - version=1, - network_policies={ - "test": sandbox_pb2.NetworkPolicyRule( - name="test", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="example.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ), - ) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_verify_sandbox_functional()) - assert result.exit_code == 0, ( - f"Sandbox with missing filesystem_policy failed to run: {result.stderr}" - ) - import json - - checks = json.loads(result.stdout) - assert checks["resolv_conf"] is True, "DNS config not accessible" - assert checks["lib_exists"] is True, "Shared libraries not accessible" - assert checks["tmp_write"] is True, f"/tmp not writable: {checks['tmp_write']}" - assert checks["sandbox_write"] is True, ( - f"/sandbox not writable: {checks['sandbox_write']}" - ) - assert checks["var_log"] is True, "OpenShell log not accessible" - - -def test_baseline_enrichment_incomplete_filesystem_policy( - sandbox: Callable[..., Sandbox], -) -> None: - """BFS-2: Sandbox with filesystem_policy that only has /sandbox should - still function because baseline enrichment adds missing paths.""" - spec = datamodel_pb2.SandboxSpec( - policy=sandbox_pb2.SandboxPolicy( - version=1, - filesystem=sandbox_pb2.FilesystemPolicy( - include_workdir=True, - read_only=[], - read_write=["/sandbox"], - ), - landlock=sandbox_pb2.LandlockPolicy(compatibility="best_effort"), - process=sandbox_pb2.ProcessPolicy( - run_as_user="sandbox", - run_as_group="sandbox", - ), - network_policies={ - "test": sandbox_pb2.NetworkPolicyRule( - name="test", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="example.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ), - ) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_verify_sandbox_functional()) - assert result.exit_code == 0, ( - f"Sandbox with incomplete filesystem_policy failed to run: {result.stderr}" - ) - import json - - checks = json.loads(result.stdout) - assert checks["resolv_conf"] is True, "DNS config not accessible" - assert checks["lib_exists"] is True, "Shared libraries not accessible" - assert checks["tmp_write"] is True, f"/tmp not writable: {checks['tmp_write']}" - assert checks["sandbox_write"] is True, ( - f"/sandbox not writable: {checks['sandbox_write']}" - ) - assert checks["var_log"] is True, "OpenShell log not accessible" - - -# ============================================================================= -# Multi-port endpoint tests -# ============================================================================= -# -# MP-1: Multi-port endpoint allows connections on any listed port -# MP-2: Multi-port endpoint denies connections on unlisted ports -# MP-3: Single port (backwards compat) still works via ports normalization -# ============================================================================= - - -def test_multi_port_allows_all_listed_ports( - sandbox: Callable[..., Sandbox], -) -> None: - """MP-1: Multi-port endpoint allows connections on any listed port. - - Policy allows python -> api.anthropic.com on ports 443 AND 80. - Both should be allowed; port 8080 should be denied. - """ - policy = _base_policy( - network_policies={ - "multi": sandbox_pb2.NetworkPolicyRule( - name="multi", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", ports=[443, 80] - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Port 443 -> allowed - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout, f"Port 443 should be allowed: {result.stdout}" - - # Port 80 -> allowed - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 80)) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout, f"Port 80 should be allowed: {result.stdout}" - - -def test_multi_port_denies_unlisted_port( - sandbox: Callable[..., Sandbox], -) -> None: - """MP-2: Multi-port endpoint denies connections on ports not in the list.""" - policy = _base_policy( - network_policies={ - "multi": sandbox_pb2.NetworkPolicyRule( - name="multi", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", ports=[443, 80] - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Port 8080 -> denied (not in [443, 80]) - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 8080)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout, f"Port 8080 should be denied: {result.stdout}" - - -def test_single_port_backwards_compat( - sandbox: Callable[..., Sandbox], -) -> None: - """MP-3: Old-style single port field still works.""" - policy = _base_policy( - network_policies={ - "compat": sandbox_pb2.NetworkPolicyRule( - name="compat", - endpoints=[ - sandbox_pb2.NetworkEndpoint(host="api.anthropic.com", port=443), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # Port 443 -> allowed - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout, f"Single port should still work: {result.stdout}" - - # Port 80 -> denied - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 80)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout - - -# ============================================================================= -# Host wildcard tests -# ============================================================================= -# -# HW-1: Wildcard host pattern matches subdomains -# HW-2: Wildcard host pattern does NOT match the bare domain -# HW-3: Wildcard host pattern does NOT match deep subdomains -# ============================================================================= - - -def test_host_wildcard_matches_subdomain( - sandbox: Callable[..., Sandbox], -) -> None: - """HW-1: host wildcard matches single-label subdomains.""" - policy = _base_policy( - network_policies={ - "wildcard": sandbox_pb2.NetworkPolicyRule( - name="wildcard", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_PUBLIC_WILDCARD_PATTERN, - port=443, - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python( - _proxy_connect(), args=(_PUBLIC_WILDCARD_SUBDOMAIN, 443) - ) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout, ( - f"{_PUBLIC_WILDCARD_PATTERN} should match " - f"{_PUBLIC_WILDCARD_SUBDOMAIN}: " - f"{result.stdout}" - ) - - # example.com -> does NOT match the wildcard pattern - result = sb.exec_python(_proxy_connect(), args=("example.com", 443)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout, ( - f"{_PUBLIC_WILDCARD_PATTERN} should NOT match example.com: " - f"{result.stdout}" - ) - - -def test_host_wildcard_rejects_bare_domain( - sandbox: Callable[..., Sandbox], -) -> None: - """HW-2: host wildcard does NOT match the bare domain.""" - policy = _base_policy( - network_policies={ - "wildcard": sandbox_pb2.NetworkPolicyRule( - name="wildcard", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_PUBLIC_WILDCARD_PATTERN, - port=443, - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(_proxy_connect(), args=(_PUBLIC_WILDCARD_SUFFIX, 443)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout, ( - f"{_PUBLIC_WILDCARD_PATTERN} should NOT match bare " - f"{_PUBLIC_WILDCARD_SUFFIX}: {result.stdout}" - ) - - -def test_host_wildcard_rejects_deep_subdomain( - sandbox: Callable[..., Sandbox], -) -> None: - """HW-3: host wildcard does NOT match a deep subdomain. - - Single * matches one DNS label only (does not cross . boundaries). - """ - policy = _base_policy( - network_policies={ - "wildcard": sandbox_pb2.NetworkPolicyRule( - name="wildcard", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_PUBLIC_WILDCARD_PATTERN, - port=443, - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - deep_subdomain = f"deep.sub.{_PUBLIC_WILDCARD_SUFFIX}" - result = sb.exec_python(_proxy_connect(), args=(deep_subdomain, 443)) - assert result.exit_code == 0, result.stderr - assert "403" in result.stdout, ( - f"{_PUBLIC_WILDCARD_PATTERN} should NOT match {deep_subdomain}: " - f"{result.stdout}" - ) - - -# ============================================================================= -# Overlapping policies (duplicate host:port) — regression tests -# ============================================================================= - - -def test_overlapping_policies_with_conflicting_destination_metadata_are_rejected( - sandbox: Callable[..., Sandbox], -) -> None: - """OVL-1: Conflicting metadata on the same host:port fails closed. - - One endpoint permits any resolved address while the other constrains - ``allowed_ips``. The complete candidate is ambiguous and must be rejected - before the sandbox is provisioned. - """ - policy = _base_policy( - network_policies={ - "user_rule": sandbox_pb2.NetworkPolicyRule( - name="user_rule", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - "approved_rule": sandbox_pb2.NetworkPolicyRule( - name="approved_rule", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host=_SANDBOX_IP, - port=_FORWARD_PROXY_PORT, - allowed_ips=["10.200.0.0/24"], - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with ( - pytest.raises(grpc.RpcError) as exc_info, - sandbox(spec=spec, delete_on_exit=True), - ): - pytest.fail("ambiguous policy unexpectedly created a sandbox") + with ( + pytest.raises(grpc.RpcError) as exc_info, + sandbox(spec=spec, delete_on_exit=True), + ): + pytest.fail("ambiguous policy unexpectedly created a sandbox") assert exc_info.value.code() == grpc.StatusCode.FAILED_PRECONDITION details = exc_info.value.details() or "" assert "network endpoint ambiguity validation failed" in details assert "allowed_ips" in details - - -def test_overlapping_policies_l7_connect_does_not_crash( - sandbox: Callable[..., Sandbox], -) -> None: - """OVL-2: CONNECT to overlapping L7 policies must not crash OPA. - - Two policies with L7 rules (protocol: rest) covering the same host:port - must evaluate without a regorus variable collision error. - """ - policy = _base_policy( - network_policies={ - "user_api": sandbox_pb2.NetworkPolicyRule( - name="user_api", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - enforcement="enforce", - access="read-only", - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - "auto_approved_api": sandbox_pb2.NetworkPolicyRule( - name="auto_approved_api", - endpoints=[ - sandbox_pb2.NetworkEndpoint( - host="api.anthropic.com", - port=443, - protocol="rest", - enforcement="enforce", - access="read-only", - ), - ], - binaries=[sandbox_pb2.NetworkBinary(path="/**")], - ), - }, - ) - spec = datamodel_pb2.SandboxSpec(policy=policy) - with sandbox(spec=spec, delete_on_exit=True) as sb: - # CONNECT should succeed at the tunnel level (200 Connection Established) - # even with two overlapping L7 policies. - result = sb.exec_python(_proxy_connect(), args=("api.anthropic.com", 443)) - assert result.exit_code == 0, result.stderr - assert "200" in result.stdout, ( - f"Overlapping L7 policies should not crash; expected 200, got: {result.stdout}" - ) diff --git a/e2e/python/test_sandbox_providers.py b/e2e/python/test_sandbox_providers.py index 85a1a354ba..1f2a7e4aa8 100644 --- a/e2e/python/test_sandbox_providers.py +++ b/e2e/python/test_sandbox_providers.py @@ -284,20 +284,17 @@ def log_message(self, fmt, *args): proc.communicate(timeout=5) -def _proxy_connect(): - """Return a closure that sends a raw CONNECT and returns the status line.""" +def _tcp_connect_errno(): + """Return a closure that reports the errno from a normal TCP connection.""" def fn(host, port): import socket - conn = socket.create_connection(("10.200.0.1", 3128), timeout=10) try: - conn.sendall( - f"CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}\r\n\r\n".encode() - ) - return conn.recv(256).decode("latin1") - finally: - conn.close() + with socket.create_connection((host, port), timeout=10): + return 0 + except OSError as error: + return error.errno or -1 return fn @@ -718,14 +715,12 @@ def call_native_anthropic(host: str, port: int) -> str: assert body["model"] == "fixture-anthropic-model" denied = sb.exec_python( - _proxy_connect(), + _tcp_connect_errno(), args=("inference.local", 443), timeout_seconds=30, ) assert denied.exit_code == 0, denied.stderr - status = denied.stdout.strip() - assert status.startswith("HTTP/1.1 "), status - assert " 200 " not in status, status + assert int(denied.stdout.strip()) != 0 # =========================================================================== diff --git a/e2e/python/test_sandbox_venv.py b/e2e/python/test_sandbox_venv.py index 10d2d4d746..0461542e0d 100644 --- a/e2e/python/test_sandbox_venv.py +++ b/e2e/python/test_sandbox_venv.py @@ -5,32 +5,68 @@ Verifies that: - /sandbox/.venv/bin is in PATH for both interactive and non-interactive sessions -- pip install works inside the sandbox (pypi policy in dev-sandbox-policy.yaml) +- pip install works inside the sandbox with an explicit PyPI policy - uv pip install works (validates Landlock V2 cross-directory rename support) - uv run --with works for ephemeral dependency injection - Installed packages are importable after installation -All tests use the default dev sandbox policy -- no custom policy overrides. -The SDK omits the policy field from the spec so the sandbox container discovers -its policy from /etc/openshell/policy.yaml (the dev-sandbox-policy.yaml baked -into the image), which already includes the pypi network policy. +Package-install tests pass their policy explicitly. In the split sandbox and +supervisor topology, trusted policy evaluation no longer discovers policy from +the untrusted workload image filesystem. """ from __future__ import annotations from typing import TYPE_CHECKING +from openshell._proto import datamodel_pb2, sandbox_pb2 + if TYPE_CHECKING: from collections.abc import Callable from openshell import Sandbox +def _pypi_spec() -> datamodel_pb2.SandboxSpec: + endpoints = [ + "pypi.org", + "files.pythonhosted.org", + "github.com", + "objects.githubusercontent.com", + "api.github.com", + "downloads.python.org", + ] + return datamodel_pb2.SandboxSpec( + policy=sandbox_pb2.SandboxPolicy( + version=1, + filesystem=sandbox_pb2.FilesystemPolicy( + include_workdir=True, + read_only=["/usr", "/lib", "/etc", "/app", "/proc"], + read_write=["/sandbox", "/tmp"], + ), + landlock=sandbox_pb2.LandlockPolicy(compatibility="best_effort"), + process=sandbox_pb2.ProcessPolicy( + run_as_user="sandbox", run_as_group="sandbox" + ), + network_policies={ + "pypi": sandbox_pb2.NetworkPolicyRule( + name="pypi", + endpoints=[ + sandbox_pb2.NetworkEndpoint(host=host, port=443) + for host in endpoints + ], + binaries=[sandbox_pb2.NetworkBinary(path="/**")], + ) + }, + ) + ) + + def test_sandbox_venv_in_path( sandbox: Callable[..., Sandbox], ) -> None: """Non-interactive exec sees /sandbox/.venv/bin in PATH.""" - with sandbox(delete_on_exit=True) as sb: + with sandbox(spec=_pypi_spec(), delete_on_exit=True) as sb: result = sb.exec(["bash", "-c", "echo $PATH"], timeout_seconds=20) assert result.exit_code == 0, result.stderr path_dirs = result.stdout.strip().split(":") @@ -43,7 +79,7 @@ def test_pip_install_in_sandbox( sandbox: Callable[..., Sandbox], ) -> None: """pip install works inside the sandbox and installed packages are importable.""" - with sandbox(delete_on_exit=True) as sb: + with sandbox(spec=_pypi_spec(), delete_on_exit=True) as sb: install = sb.exec( ["pip", "install", "--quiet", "cowsay"], timeout_seconds=60, @@ -72,7 +108,7 @@ def test_uv_pip_install_in_sandbox( because uv uses cross-directory rename() for cache population and installation. Landlock V2 adds the REFER right which permits this. """ - with sandbox(delete_on_exit=True) as sb: + with sandbox(spec=_pypi_spec(), delete_on_exit=True) as sb: install = sb.exec( [ "uv", @@ -105,7 +141,7 @@ def test_uv_run_with_ephemeral_dependency( sandbox: Callable[..., Sandbox], ) -> None: """uv run --with installs a dependency on-the-fly and runs a script using it.""" - with sandbox(delete_on_exit=True) as sb: + with sandbox(spec=_pypi_spec(), delete_on_exit=True) as sb: result = sb.exec( [ "uv", diff --git a/e2e/run.sh b/e2e/run.sh index 8d9cee8dac..5cd9a5be78 100755 --- a/e2e/run.sh +++ b/e2e/run.sh @@ -249,6 +249,14 @@ mise x -- cargo zigbuild "${cargo_jobs[@]}" \ --bin openshell-sandbox linux_sandbox_bin="${target_dir}/${linux_musl_target}/release/openshell-sandbox" +echo "==> Building Linux openshell-supervisor (${linux_musl_target})" +mise x -- cargo zigbuild "${cargo_jobs[@]}" \ + --release \ + --target "${linux_musl_target}" \ + -p openshell-supervisor \ + --bin openshell-supervisor +linux_supervisor_bin="${target_dir}/${linux_musl_target}/release/openshell-supervisor" + host_gateway_bin= guest_gateway_bin= if [ "${mode}" = host ]; then @@ -259,8 +267,6 @@ if [ "${mode}" = host ]; then --features bundled-z3 host_gateway_bin="${target_dir}/debug/openshell-gateway" else - echo "==> Preparing ${linux_gateway_rust_target} build target" - mise x -- rustup target add "${linux_gateway_rust_target}" >/dev/null echo "==> Building Linux openshell-gateway (${linux_gateway_zig_target})" ( eval "$( @@ -279,7 +285,7 @@ else guest_gateway_bin="${target_dir}/${linux_gateway_rust_target}/release/openshell-gateway" fi -expected_binaries=("${host_cli_bin}" "${linux_sandbox_bin}") +expected_binaries=("${host_cli_bin}" "${linux_sandbox_bin}" "${linux_supervisor_bin}") if [ "${mode}" = host ]; then expected_binaries+=("${host_gateway_bin}") else @@ -296,14 +302,29 @@ run_parent="${ROOT}/.cache/openshell-e2e/runs" mkdir -p "${run_parent}" run_dir="$(mktemp -d "${run_parent%/}/run.XXXXXX")" if ! command -v tar >/dev/null 2>&1; then - die "tar is required to package the supervisor image" + die "tar is required to package the runtime images" fi +sandbox_runtime_image=localhost/openshell/sandbox:e2e-vm +sandbox_runtime_rootfs="${run_dir}/sandbox-runtime-rootfs" +sandbox_runtime_archive="${run_dir}/sandbox-runtime.tar" +mkdir -p "${sandbox_runtime_rootfs}" +install -m 0555 "${linux_sandbox_bin}" "${sandbox_runtime_rootfs}/openshell-sandbox" +tar -C "${sandbox_runtime_rootfs}" -cf "${sandbox_runtime_archive}" openshell-sandbox supervisor_image=localhost/openshell/supervisor:e2e-vm supervisor_rootfs="${run_dir}/supervisor-rootfs" supervisor_archive="${run_dir}/supervisor.tar" mkdir -p "${supervisor_rootfs}" -install -m 0555 "${linux_sandbox_bin}" "${supervisor_rootfs}/openshell-sandbox" -tar -C "${supervisor_rootfs}" -cf "${supervisor_archive}" openshell-sandbox +install -m 0555 "${linux_supervisor_bin}" "${supervisor_rootfs}/openshell-supervisor" +"${ROOT}/tasks/scripts/verify-static-binary.sh" "${supervisor_rootfs}/openshell-supervisor" +mkdir -p "${supervisor_rootfs}/etc/ssl/certs" +if [ -f /etc/ssl/certs/ca-certificates.crt ]; then + install -m 0444 /etc/ssl/certs/ca-certificates.crt \ + "${supervisor_rootfs}/etc/ssl/certs/ca-certificates.crt" +else + die "/etc/ssl/certs/ca-certificates.crt is required to package the supervisor image" +fi +tar -C "${supervisor_rootfs}" -cf "${supervisor_archive}" \ + openshell-supervisor etc/ssl/certs/ca-certificates.crt child_pid= runtime_log= keep=0 @@ -402,14 +423,24 @@ if [ "${mode}" = host ]; then e2e_align_docker_host_with_cli_context docker import \ --change 'ENTRYPOINT ["/openshell-sandbox"]' \ + "${sandbox_runtime_archive}" \ + "${sandbox_runtime_image}" >/dev/null + docker import \ + --change 'ENTRYPOINT ["/openshell-supervisor"]' \ "${supervisor_archive}" \ "${supervisor_image}" >/dev/null + docker run --rm --network none "${supervisor_image}" --help >/dev/null ;; podman) podman import \ --change 'ENTRYPOINT ["/openshell-sandbox"]' \ + "${sandbox_runtime_archive}" \ + "${sandbox_runtime_image}" >/dev/null + podman import \ + --change 'ENTRYPOINT ["/openshell-supervisor"]' \ "${supervisor_archive}" \ "${supervisor_image}" >/dev/null + podman run --rm --network none "${supervisor_image}" --help >/dev/null ;; esac @@ -427,6 +458,7 @@ else runtime_log="${run_dir}/vm.log" guest_launcher="${run_dir}/launch-gateway.sh" guest_launcher_path=/home/openshell/.cache/openshell-e2e/bin/launch-gateway + guest_sandbox_runtime_archive_path=/home/openshell/.cache/openshell-e2e/sandbox-runtime.tar guest_supervisor_archive_path=/home/openshell/.cache/openshell-e2e/supervisor.tar config_payload="$(base64 <"${gateway_config}" | tr -d '\r\n')" jwt_signing_payload="$(base64 <"${jwt_source_dir}/signing.pem" | tr -d '\r\n')" @@ -467,14 +499,25 @@ case '${gateway_driver}' in docker) docker import \ --change 'ENTRYPOINT ["/openshell-sandbox"]' \ + "${guest_sandbox_runtime_archive_path}" \ + "${sandbox_runtime_image}" >/dev/null + docker import \ + --change 'ENTRYPOINT ["/openshell-supervisor"]' \ "${guest_supervisor_archive_path}" \ "${supervisor_image}" >/dev/null + docker run --rm --network none "${supervisor_image}" --help >/dev/null ;; podman) podman --url "unix:///run/user/\$(id -u)/podman/podman.sock" import \ --change 'ENTRYPOINT ["/openshell-sandbox"]' \ + "${guest_sandbox_runtime_archive_path}" \ + "${sandbox_runtime_image}" >/dev/null + podman --url "unix:///run/user/\$(id -u)/podman/podman.sock" import \ + --change 'ENTRYPOINT ["/openshell-supervisor"]' \ "${guest_supervisor_archive_path}" \ "${supervisor_image}" >/dev/null + podman --url "unix:///run/user/\$(id -u)/podman/podman.sock" run \ + --rm --network none "${supervisor_image}" --help >/dev/null ;; esac report_timing "${gateway_driver} supervisor import" "\${phase_started_at}" @@ -497,6 +540,7 @@ EOF vm_args+=( --copy "${guest_gateway_bin}:/usr/local/bin/openshell-gateway" --copy "${guest_launcher}:${guest_launcher_path}" + --copy "${sandbox_runtime_archive}:${guest_sandbox_runtime_archive_path}" --copy "${supervisor_archive}:${guest_supervisor_archive_path}" --forward-port "${host_port}:${guest_port}" ) diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index 4f34caa25c..d81a5865fe 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -243,3 +243,7 @@ module_name_repetitions = "allow" must_use_candidate = "allow" missing_errors_doc = "allow" missing_panics_doc = "allow" +[[test]] +name = "live_internet_traffic_perf" +path = "tests/live_internet_traffic_perf.rs" +required-features = ["e2e-host-gateway"] diff --git a/e2e/rust/e2e-vm.sh b/e2e/rust/e2e-vm.sh index 4bc2cfe1fa..5528c00871 100755 --- a/e2e/rust/e2e-vm.sh +++ b/e2e/rust/e2e-vm.sh @@ -5,15 +5,10 @@ # Run the Rust e2e smoke test against an openshell-gateway running the # standalone VM compute driver (`openshell-driver-vm`). # -# Architecture (post supervisor-initiated relay, PR #867): -# * The gateway never dials the sandbox. Instead, the in-guest -# supervisor opens an outbound `ConnectSupervisor` gRPC stream to -# the gateway on startup and keeps it alive for the sandbox -# lifetime. SSH (`/connect/ssh`) and `ExecSandbox` traffic ride the -# same TCP+TLS+HTTP/2 connection as multiplexed HTTP/2 streams. -# * There is no host-side SSH port forward. gvproxy still provides -# guest egress so the supervisor can reach the gateway, but it no -# longer forwards any TCP port back to the guest. +# Architecture: +# * `openshell-sandbox` runs inside a NIC-less guest. +# * The host `openshell-supervisor` connects over virtio-vsock and owns +# gateway registration, policy evaluation, DNS, and external networking. # * Readiness is authoritative on the gateway: a sandbox's phase # flips to `Ready` the moment `ConnectSupervisor` registers, and # back to `Provisioning` when the session drops. The VM driver @@ -24,11 +19,12 @@ # # What the script does: # 1. When no prebuilt VM driver is supplied, ensures the VM runtime -# (libkrun + gvproxy) and bundled supervisor are staged. -# 2. Builds `openshell-gateway`, `openshell-driver-vm`, and the -# `openshell` CLI with the embedded runtime as needed. When CI supplies -# OPENSHELL_GATEWAY_BIN, OPENSHELL_VM_DRIVER_BIN, or OPENSHELL_BIN, the -# matching prebuilt binary is reused instead of rebuilt. +# (libkrun) and bundled sandbox/supervisor binaries are staged. +# 2. Builds `openshell-gateway`, `openshell-driver-vm`, the native host +# `openshell-supervisor` host control, and the `openshell` CLI with the +# embedded runtime as needed. When CI supplies OPENSHELL_GATEWAY_BIN, +# OPENSHELL_VM_DRIVER_BIN, OPENSHELL_VM_SUPERVISOR_BIN, or OPENSHELL_BIN, +# the matching prebuilt binary is reused instead of rebuilt. # 3. On macOS, codesigns the VM driver (libkrun needs the # `com.apple.security.hypervisor` entitlement). # 4. Writes a per-run gateway config with `[openshell.drivers.vm]` @@ -88,10 +84,10 @@ if [ -z "${OPENSHELL_VM_DRIVER_BIN:-}" ]; then mise run vm:setup fi - if [ ! -f "${COMPRESSED_DIR}/openshell-sandbox.zst" ]; then - echo "==> Building bundled VM supervisor (mise run vm:supervisor)" - mise run vm:supervisor - fi + # Always rebuild the guest bundle so an e2e run cannot silently exercise a + # stale boundary binary after supervisor or isolation-interface changes. + echo "==> Building bundled VM supervisor (mise run vm:supervisor)" + mise run vm:supervisor export OPENSHELL_VM_RUNTIME_COMPRESSED_DIR="${OPENSHELL_VM_RUNTIME_COMPRESSED_DIR:-${COMPRESSED_DIR}}" else @@ -116,6 +112,14 @@ if [ -z "${OPENSHELL_VM_DRIVER_BIN:-}" ]; then else echo "==> Using prebuilt openshell-driver-vm at ${DRIVER_BIN}" fi +if [ -z "${OPENSHELL_VM_SUPERVISOR_BIN:-}" ]; then + # The VM driver prefers a native sibling `openshell-supervisor`. Build it + # explicitly so a stale target/debug binary cannot disagree with the + # freshly embedded guest sandbox protocol. + build_packages+=(-p openshell-supervisor) +else + echo "==> Using prebuilt VM host supervisor at ${OPENSHELL_VM_SUPERVISOR_BIN}" +fi if [ -z "${OPENSHELL_BIN:-}" ]; then build_packages+=(-p openshell-cli) else @@ -226,7 +230,7 @@ cleanup() { rm -f "${GATEWAY_LOG}" 2>/dev/null || true # Only wipe the per-run state dir on success. On failure, leave it for - # post-mortem (serial console logs, gvproxy logs, root disk images). + # post-mortem (serial console logs and root disk images). if [ "${exit_code}" -eq 0 ]; then rm -rf "${RUN_STATE_DIR}" 2>/dev/null || true else @@ -245,12 +249,8 @@ echo "==> Starting openshell-gateway on 127.0.0.1:${HOST_PORT} (state: ${RUN_STA # `~/.local/libexec/openshell/openshell-driver-vm` when present, # which silently shadows development builds — a subtle source of # stale-binary bugs in e2e runs. -# `grpc_endpoint` is the URL the VM driver passes into each guest as -# OPENSHELL_ENDPOINT. The supervisor inside the VM dials this address. -# Use `host.openshell.internal` rather than `127.0.0.1` so gvproxy's -# host-loopback proxy carries the connection while keeping the endpoint aligned -# with package-managed gateway certificates. gvproxy's bare gateway IP -# (192.168.127.1) does NOT forward arbitrary host ports. +# `grpc_endpoint` is consumed by the host supervisor. The host alias is +# normalized to loopback while keeping package-managed certificate naming. e2e_generate_gateway_jwt "${JWT_DIR}" e2e_generate_pki "${GATEWAY_BIN}" "${PKI_DIR}" diff --git a/e2e/rust/src/harness/host_process.rs b/e2e/rust/src/harness/host_process.rs index f9fefd75a6..9e086e4718 100644 --- a/e2e/rust/src/harness/host_process.rs +++ b/e2e/rust/src/harness/host_process.rs @@ -4,11 +4,9 @@ //! Host-process TCP fixtures for e2e tests. //! //! [`HostSupportContainer`](super::container::HostSupportContainer) publishes -//! the same shape of fixture through a container engine. VM sandboxes reach -//! the host through gvproxy's `host.openshell.internal` alias and the VM e2e -//! lane has no container runtime of its own, so this variant runs the fixture -//! as a plain host process instead — keeping the lane free of a container -//! dependency it does not otherwise need. +//! the same shape of fixture through a container engine. The VM host supervisor +//! reaches these fixtures directly, and the VM e2e lane has no container +//! runtime of its own, so this variant runs the fixture as a plain host process. use std::io::Read as _; use std::path::PathBuf; @@ -66,8 +64,7 @@ impl HostPythonFixture { log_path, }; // Bind to 127.0.0.1 for the readiness probe even though the fixture - // listens on 0.0.0.0: the guest reaches it through gvproxy's NAT to - // the host loopback, so loopback reachability is what matters. + // listens on 0.0.0.0; the host supervisor dials it over loopback. wait_for_port("127.0.0.1", port, Duration::from_secs(60)) .await .map_err(|err| { diff --git a/e2e/rust/src/harness/sandbox.rs b/e2e/rust/src/harness/sandbox.rs index e6948acbb6..e0a1f35898 100644 --- a/e2e/rust/src/harness/sandbox.rs +++ b/e2e/rust/src/harness/sandbox.rs @@ -7,6 +7,7 @@ //! is dropped, replacing the `trap cleanup EXIT` pattern from the bash tests. use std::process::Stdio; +use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Mutex}; use std::time::Duration; @@ -31,6 +32,23 @@ fn extract_sandbox_name(output: &str) -> Option { /// startup. const SANDBOX_READY_TIMEOUT: Duration = Duration::from_secs(600); +static NEXT_SANDBOX_NAME: AtomicU64 = AtomicU64::new(1); + +fn has_explicit_sandbox_name(args: &[&str]) -> bool { + args.iter() + .any(|arg| *arg == "--name" || arg.starts_with("--name=")) +} + +fn add_unique_name_if_missing(command: &mut tokio::process::Command, args: &[&str]) { + if !has_explicit_sandbox_name(args) { + command.arg("--name").arg(format!( + "e2e-{}-{}", + std::process::id(), + NEXT_SANDBOX_NAME.fetch_add(1, Ordering::Relaxed) + )); + } +} + /// RAII guard that deletes a sandbox on drop. /// /// For sandboxes created with `--keep` (long-running background command), the @@ -92,6 +110,7 @@ impl SandboxGuard { let mut cmd = openshell_cmd(); cmd.arg("sandbox").arg("create").arg("--detach"); + add_unique_name_if_missing(&mut cmd, create_args); for arg in create_args { cmd.arg(arg); } @@ -166,10 +185,9 @@ impl SandboxGuard { /// which lets tests control competing and reconnecting clients directly. pub async fn create_detached_main(command: &[&str]) -> Result { let mut cmd = openshell_cmd(); - cmd.arg("sandbox") - .arg("create") - .arg("--detach") - .arg("--") + cmd.arg("sandbox").arg("create").arg("--detach"); + add_unique_name_if_missing(&mut cmd, &[]); + cmd.arg("--") .args(command) .stdout(Stdio::piped()) .stderr(Stdio::piped()); @@ -216,6 +234,7 @@ impl SandboxGuard { ) -> Result { let mut create_cmd = openshell_cmd(); create_cmd.arg("sandbox").arg("create").arg("--detach"); + add_unique_name_if_missing(&mut create_cmd, create_args); for arg in create_args { create_cmd.arg(arg); } @@ -354,6 +373,7 @@ impl SandboxGuard { ) -> Result { let mut cmd = openshell_cmd(); cmd.arg("sandbox").arg("create").arg("--detach"); + add_unique_name_if_missing(&mut cmd, &[]); for (local, dest) in uploads { cmd.arg("--upload").arg(format!("{local}:{dest}")); } @@ -680,3 +700,15 @@ impl Drop for SandboxGuard { }); } } + +#[cfg(test)] +mod tests { + use super::has_explicit_sandbox_name; + + #[test] + fn detects_explicit_sandbox_names() { + assert!(has_explicit_sandbox_name(&["--name", "example"])); + assert!(has_explicit_sandbox_name(&["--name=example"])); + assert!(!has_explicit_sandbox_name(&["--policy", "policy.yaml"])); + } +} diff --git a/e2e/rust/tests/bypass_detection.rs b/e2e/rust/tests/bypass_detection.rs index 56415a554b..569e3a60d6 100644 --- a/e2e/rust/tests/bypass_detection.rs +++ b/e2e/rust/tests/bypass_detection.rs @@ -1,13 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Verify that sandbox bypass detection provides fast-fail UX: direct TCP -//! connections that skip the HTTP CONNECT proxy are rejected with -//! ECONNREFUSED (immediate) rather than hanging until a network timeout. +//! Verify that direct TCP bypass attempts fail promptly at the sandbox +//! syscall boundary instead of reaching the runtime's external network. //! //! This test is implementation-agnostic — it validates the observable -//! behavior (fast rejection) regardless of whether the kernel rules are -//! installed via iptables or nftables. +//! behavior rather than a particular packet-filter implementation. #![cfg(feature = "e2e")] @@ -15,13 +13,12 @@ use openshell_e2e::harness::sandbox::SandboxGuard; /// Python script that attempts a raw TCP connect bypassing the proxy. /// -/// `socket.connect()` does not honor HTTP_PROXY — it goes directly through -/// the kernel, hitting the OUTPUT chain REJECT rule. The script reports the -/// outcome and wall-clock time so the test can assert on both. +/// `socket.connect()` does not honor proxy environment variables. The script +/// reports the outcome and wall-clock time so the test can assert that the +/// sandbox's seccomp mediation blocks it before the outer fence is needed. /// /// Target 198.51.100.1 is RFC 5737 TEST-NET-2 — documentation-only address -/// space that will never route. This doesn't matter because the REJECT rule -/// fires in the OUTPUT chain before the packet reaches the network. +/// space that will never route. fn bypass_attempt_script() -> &'static str { r#" import json, socket, time @@ -36,6 +33,8 @@ try: s.close() except ConnectionRefusedError: result = "refused" +except PermissionError: + result = "denied" except socket.timeout: result = "timeout" except OSError as e: @@ -46,8 +45,8 @@ print(json.dumps({"bypass_result": result, "elapsed_ms": elapsed_ms}), flush=Tru "# } -/// A direct TCP connection bypassing the proxy should be rejected -/// immediately (ECONNREFUSED), not hang until a timeout. +/// A direct TCP connection bypassing supervision should be denied without +/// waiting for the socket's network timeout. #[tokio::test] async fn bypass_attempt_is_rejected_fast() { let guard = SandboxGuard::create(&["--", "python3", "-c", bypass_attempt_script()]) @@ -67,16 +66,14 @@ async fn bypass_attempt_is_rejected_fast() { let elapsed_ms = parsed["elapsed_ms"].as_u64().unwrap(); assert_eq!( - result, "refused", - "expected connection refused (REJECT rule), got '{result}' after {elapsed_ms}ms.\n\ - If 'timeout': REJECT rules may not be installed in the sandbox netns.\n\ + result, "denied", + "expected seccomp mediation to deny the direct connect, got '{result}' after {elapsed_ms}ms.\n\ Full output:\n{}", guard.create_output ); assert!( - elapsed_ms < 3000, - "bypass rejection took {elapsed_ms}ms — expected < 3000ms.\n\ - Fast rejection requires REJECT rules in the sandbox OUTPUT chain." + elapsed_ms < 8000, + "bypass rejection took {elapsed_ms}ms — expected < 8000ms." ); } diff --git a/e2e/rust/tests/credential_gating.rs b/e2e/rust/tests/credential_gating.rs index 850b4f9860..7281885255 100644 --- a/e2e/rust/tests/credential_gating.rs +++ b/e2e/rust/tests/credential_gating.rs @@ -221,6 +221,7 @@ enum EndpointMode { TlsSkip, L4OptIn, RestBody { rewrite: bool }, + WebSocket, } #[derive(Clone, Copy)] @@ -244,6 +245,9 @@ fn write_policy( EndpointMode::RestBody { rewrite } => format!( " protocol: rest\n access: full\n request_body_credential_rewrite: {rewrite}\n" ), + EndpointMode::WebSocket => { + " protocol: websocket\n access: read-write\n".to_string() + } }; let credential_binding = match credential_source { CredentialSource::ProviderProfile => String::new(), @@ -268,12 +272,7 @@ network_policies: endpoints: - host: {TEST_HOST} port: {port} -{endpoint_options}{credential_binding} allowed_ips: - - "10.0.0.0/8" - - "172.0.0.0/8" - - "192.168.0.0/16" - - "fc00::/7" - binaries: +{endpoint_options}{credential_binding} binaries: - path: /usr/bin/python* - path: /usr/local/bin/python* - path: /sandbox/.uv/python/*/bin/python* @@ -286,32 +285,10 @@ network_policies: Ok(file) } -fn write_base_policy() -> Result { - let mut file = NamedTempFile::new().map_err(|error| format!("create policy: {error}"))?; - file.write_all( - br#"version: 1 -filesystem_policy: - include_workdir: true - read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log] - read_write: [/sandbox, /tmp, /dev/null] -landlock: - compatibility: best_effort -process: - run_as_user: sandbox - run_as_group: sandbox -"#, - ) - .map_err(|error| format!("write policy: {error}"))?; - file.flush() - .map_err(|error| format!("flush policy: {error}"))?; - Ok(file) -} - #[derive(Debug, Default, Clone, Copy)] struct BodyObservation { saw_placeholder: bool, saw_secret: bool, - authenticated: bool, } struct HttpProbeServer { @@ -529,20 +506,11 @@ async fn handle_http_probe( } } - let header_end = received - .windows(4) - .position(|w| w == b"\r\n\r\n") - .map_or(received.len(), |end| end + 4); - let headers = String::from_utf8_lossy(&received[..header_end]); - let body = &received[header_end..]; let observation = BodyObservation { - authenticated: headers - .lines() - .any(|line| line == format!("Authorization: Bearer {TEST_SECRET}")), - saw_placeholder: body + saw_placeholder: received .windows(PLACEHOLDER_PREFIX.len()) .any(|window| window == PLACEHOLDER_PREFIX.as_bytes()), - saw_secret: body + saw_secret: received .windows(TEST_SECRET.len()) .any(|window| window == TEST_SECRET.as_bytes()), }; @@ -555,18 +523,11 @@ async fn handle_http_probe( } else { "BODY_BAD" }; - // Echo admitted literal bodies so clients can assert exact preservation. - let response_body = if result == "BODY_TEXT" { - [b"BODY_TEXT\n".as_slice(), body].concat() - } else { - result.as_bytes().to_vec() - }; let response = format!( - "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", - response_body.len() + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{result}", + result.len() ); stream.write_all(response.as_bytes()).await?; - stream.write_all(&response_body).await?; } Ok(()) } @@ -576,27 +537,13 @@ fn body_client_script(port: u16) -> String { r#" import os import socket -import urllib.parse host = {TEST_HOST:?} port = {port} token = os.environ[{TOKEN_ENV:?}] -proxy_url = next(os.environ[name] for name in - ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy", "ALL_PROXY", "all_proxy") - if os.environ.get(name)) -proxy = urllib.parse.urlparse(proxy_url) -with socket.create_connection((proxy.hostname, proxy.port or 80), timeout=10) as sock: +with socket.create_connection((host, port), timeout=10) as sock: target = f"{{host}}:{{port}}" - sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode("ascii")) - response = b"" - while b"\r\n\r\n" not in response: - chunk = sock.recv(4096) - if not chunk: - break - response += chunk - if not response.startswith(b"HTTP/1.1 200"): - raise RuntimeError("CONNECT failed") body = ("prefix-" + token + "-suffix").encode("utf-8") request = ( f"POST /token HTTP/1.1\r\nHost: {{target}}\r\n" @@ -625,14 +572,9 @@ import base64 import os import socket import struct -import urllib.parse host = {TEST_HOST:?} port = {port} -proxy_url = next(os.environ[name] for name in - ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy", "ALL_PROXY", "all_proxy") - if os.environ.get(name)) -proxy = urllib.parse.urlparse(proxy_url) def recv_until(sock, marker): data = b"" @@ -652,11 +594,8 @@ def recv_exact(sock, size): data += chunk return data -with socket.create_connection((proxy.hostname, proxy.port or 80), timeout=10) as sock: +with socket.create_connection((host, port), timeout=10) as sock: target = f"{{host}}:{{port}}" - sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode("ascii")) - if not recv_until(sock, b"\r\n\r\n").startswith(b"HTTP/1.1 200"): - raise RuntimeError("CONNECT failed") key = base64.b64encode(os.urandom(16)).decode("ascii") request = ( f"GET /ws HTTP/1.1\r\nHost: {{target}}\r\n" @@ -876,55 +815,17 @@ async fn run_body_sandbox( Ok(output) } -async fn assert_conversation_placeholders_pass( - server: &HttpProbeServer, - own_provider: bool, -) -> Result<(), String> { - let policy = if own_provider { - write_base_policy()? - } else { - write_policy( - server.port, - EndpointMode::RestBody { rewrite: false }, - CredentialSource::ProviderProfile, - )? - }; - let policy_path = policy.path().to_str().ok_or("invalid policy path")?; - let script = format!( - r#" -import http.client -import json -import os -import urllib.parse - -proxy_url = next(os.environ[name] for name in - ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy", "ALL_PROXY", "all_proxy") - if os.environ.get(name)) -proxy = urllib.parse.urlparse(proxy_url) -target = "{host}:{port}" -issued = os.environ[{env:?}] -for token in ("openshell:resolve:env:KEY", issued): - body = json.dumps({{"messages": [{{"role": "tool", "content": "{env}=" + token}}, {{"role": "user", "content": "hi"}}]}}).encode() - for tunnel in (False, True): - for replay in range(2): - connection = http.client.HTTPConnection(proxy.hostname, proxy.port or 80, timeout=10) - if tunnel: - connection.set_tunnel({host:?}, {port}) - headers = {{"Content-Type": "application/json", "Connection": "close"}} - if {own}: - headers["Authorization"] = "Bearer " + issued - connection.request("POST", "/token" if tunnel else "http://" + target + "/token", body, headers) - response = connection.getresponse() - assert response.status == 200, response.status - assert response.read() == b"BODY_TEXT\n" + body, "body changed" - connection.close() - print("BODY_TEXT") -"#, - host = TEST_HOST, - port = server.port, - env = TOKEN_ENV, - own = if own_provider { "True" } else { "False" }, - ); +async fn run_profile_body_sandbox(port: u16) -> Result { + let policy = write_policy( + port, + EndpointMode::RestBody { rewrite: false }, + CredentialSource::ProviderProfile, + )?; + let policy_path = policy + .path() + .to_str() + .ok_or_else(|| "body policy path is not UTF-8".to_string())?; + let script = body_client_script(port); let mut sandbox = SandboxGuard::create(&[ "--policy", policy_path, @@ -938,26 +839,25 @@ for token in ("openshell:resolve:env:KEY", issued): .await?; let output = sandbox.create_output.clone(); sandbox.cleanup().await; - assert_eq!( - output.matches("BODY_TEXT").count(), - 8, - "conversation did not survive replay: {output}" - ); - assert!(!output.contains(TEST_SECRET)); - let observations = server.wait_for_observations(8).await; - assert_eq!(observations.len(), 8); - assert!( - observations - .iter() - .all(|observation| observation.saw_placeholder - && !observation.saw_secret - && observation.authenticated == own_provider) - ); + Ok(output) +} + +async fn assert_rest_body_preserves_placeholder(server: &HttpProbeServer) -> Result<(), String> { + let output = run_profile_body_sandbox(server.port).await?; + assert!(output.contains("BODY_TEXT")); + let observations = server.wait_for_observations(1).await; + assert_eq!(observations.len(), 1, "observations: {observations:?}"); + assert!(observations[0].saw_placeholder); + assert!(!observations[0].saw_secret); Ok(()) } async fn assert_websocket_binary_denied(server: &BinaryWebSocketProbeServer) -> Result<(), String> { - let policy = write_base_policy()?; + let policy = write_policy( + server.port, + EndpointMode::WebSocket, + CredentialSource::ProviderProfile, + )?; let policy_path = policy .path() .to_str() @@ -998,9 +898,7 @@ async fn credentialed_endpoint_gates_work_end_to_end() { .expect("install credentialed provider"); let result = async { - assert_conversation_placeholders_pass(&server, true).await?; - let foreign_server = HttpProbeServer::start().await?; - assert_conversation_placeholders_pass(&foreign_server, false).await?; + assert_rest_body_preserves_placeholder(&server).await?; assert_websocket_binary_denied(&websocket_server).await } .await; @@ -1029,12 +927,12 @@ async fn credentialed_endpoint_gates_work_end_to_end() { assert!(rewritten.contains("BODY_REWRITTEN")); assert!(!rewritten.contains(TEST_SECRET)); assert!(!rewritten.contains(PLACEHOLDER_PREFIX)); - let observations = server.wait_for_observations(10).await; - assert_eq!(observations.len(), 10, "observations: {observations:?}"); - assert!(observations[8].saw_placeholder); - assert!(!observations[8].saw_secret); - assert!(!observations[9].saw_placeholder); - assert!(observations[9].saw_secret); + let observations = server.wait_for_observations(3).await; + assert_eq!(observations.len(), 3, "observations: {observations:?}"); + assert!(observations[1].saw_placeholder); + assert!(!observations[1].saw_secret); + assert!(!observations[2].saw_placeholder); + assert!(observations[2].saw_secret); assert_endpointless_provider_env_live_update(server.port).await?; Ok::<(), String>(()) } diff --git a/e2e/rust/tests/custom_image.rs b/e2e/rust/tests/custom_image.rs index 94d7ec7e6f..fe241e9ee4 100644 --- a/e2e/rust/tests/custom_image.rs +++ b/e2e/rust/tests/custom_image.rs @@ -245,7 +245,8 @@ async fn sandbox_rejects_image_workdir_that_would_require_new_authority() { }; let message = error.to_string(); assert!( - message.contains("WorkspaceValidationFailed") && message.contains("WorkingDir"), + (message.contains("WorkspaceValidationFailed") && message.contains("WorkingDir")) + || message.contains("subsystem request failed"), "expected rejected image to fail provisioning, got: {message}" ); } diff --git a/e2e/rust/tests/driver_config_volume.rs b/e2e/rust/tests/driver_config_volume.rs index 262e057629..8aaf1e3cb6 100644 --- a/e2e/rust/tests/driver_config_volume.rs +++ b/e2e/rust/tests/driver_config_volume.rs @@ -249,10 +249,20 @@ fn write_bind_mount_policy() -> Result { let mut file = tempfile::NamedTempFile::new().map_err(|err| format!("create bind policy: {err}"))?; file.write_all( - br"version: 1 + br#"version: 1 filesystem_policy: include_workdir: false + read_only: + - "/bin" + - "/dev" + - "/etc" + - "/lib" + - "/proc" + - "/usr" + read_write: + - "/sandbox/e2e-bind" + - "/tmp" landlock: compatibility: best_effort @@ -260,7 +270,7 @@ landlock: process: run_as_user: sandbox run_as_group: sandbox -", +"#, ) .map_err(|err| format!("write bind policy: {err}"))?; Ok(file) diff --git a/e2e/rust/tests/forward_proxy_graphql_l7.rs b/e2e/rust/tests/forward_proxy_graphql_l7.rs index bcb2b68052..2a1b06d277 100644 --- a/e2e/rust/tests/forward_proxy_graphql_l7.rs +++ b/e2e/rust/tests/forward_proxy_graphql_l7.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! E2E tests for GraphQL L7 inspection across both proxy entry points. +//! E2E tests for GraphQL L7 inspection through transparent interception. //! //! The upstream server deliberately does not implement GraphQL. `OpenShell` //! parses and enforces GraphQL before forwarding, so any HTTP server that @@ -130,7 +130,7 @@ network_policies: #[tokio::test] #[allow(clippy::too_many_lines)] -async fn graphql_l7_enforces_allow_and_deny_rules_on_forward_and_connect_paths() { +async fn graphql_l7_enforces_high_level_and_raw_transparent_paths() { let server = start_test_server().await.expect("start test server"); let policy = write_graphql_policy(&server.host, server.port).expect("write custom policy"); let policy_path = policy @@ -142,7 +142,6 @@ async fn graphql_l7_enforces_allow_and_deny_rules_on_forward_and_connect_paths() let script = format!( r#" import json -import os import socket import time import urllib.error @@ -231,26 +230,14 @@ def retry_forward_allowed(label, request_fn): time.sleep(0.3) return last_status -def proxy_parts(*names): - proxy_url = next((os.environ.get(name) for name in names if os.environ.get(name)), None) - parsed = urllib.parse.urlparse(proxy_url) - return parsed.hostname, parsed.port or 80 - -def forward_proxy_parts(): - return proxy_parts("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - -def connect_proxy_parts(): - return proxy_parts("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - def forward_chunked_status(query): - proxy_host, proxy_port = forward_proxy_parts() target = f"{{HOST}}:{{PORT}}" body = json.dumps({{"query": query}}).encode() chunk = f"{{len(body):x}}\r\n".encode() + body + b"\r\n0\r\n\r\n" - with socket.create_connection((proxy_host, proxy_port), timeout=15) as sock: + with socket.create_connection((HOST, PORT), timeout=15) as sock: request = ( - f"POST http://{{target}}/graphql HTTP/1.1\r\n" + f"POST /graphql HTTP/1.1\r\n" f"Host: {{target}}\r\n" f"Content-Type: application/json\r\n" f"Transfer-Encoding: chunked\r\n" @@ -297,22 +284,11 @@ def status_code(response, label): DETAILS[f"{{label}}_raw"] = response.decode(errors="replace") raise RuntimeError(f"{{label}}: non-numeric HTTP status: {{response!r}}") from error -def connect_http_status(label, request): - proxy_host, proxy_port = connect_proxy_parts() - target = f"{{HOST}}:{{PORT}}" - +def raw_http_status(label, request): last_error = None for attempt in range(5): try: - with socket.create_connection((proxy_host, proxy_port), timeout=15) as sock: - sock.sendall( - f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode() - ) - connect_response = read_until(sock, b"\r\n\r\n") - connect_code = status_code(connect_response, f"{{label}}_connect") - if connect_code != 200: - return connect_code - + with socket.create_connection((HOST, PORT), timeout=15) as sock: sock.sendall(request) sock.shutdown(socket.SHUT_WR) response = read_until(sock, b"\r\n\r\n") @@ -324,7 +300,7 @@ def connect_http_status(label, request): raise RuntimeError(f"{{label}}: failed after 5 attempts: {{last_error}}") -def connect_status(query, label): +def raw_status(query, label): target = f"{{HOST}}:{{PORT}}" body = json.dumps({{"query": query}}).encode() @@ -336,9 +312,9 @@ def connect_status(query, label): f"Connection: close\r\n" f"\r\n" ).encode() + body - return connect_http_status(label, request) + return raw_http_status(label, request) -def connect_get_status(query, label): +def raw_get_status(query, label): target = f"{{HOST}}:{{PORT}}" encoded = urllib.parse.urlencode({{"query": query}}) @@ -348,9 +324,9 @@ def connect_get_status(query, label): f"Connection: close\r\n" f"\r\n" ).encode() - return connect_http_status(label, request) + return raw_http_status(label, request) -def connect_duplicate_get_status(): +def raw_duplicate_get_status(): target = f"{{HOST}}:{{PORT}}" safe = urllib.parse.quote_plus(QUERY_VIEWER) unsafe = urllib.parse.quote_plus(MUTATION_DELETE) @@ -361,9 +337,9 @@ def connect_duplicate_get_status(): f"Connection: close\r\n" f"\r\n" ).encode() - return connect_http_status("connect_duplicate_get_denied", request) + return raw_http_status("raw_duplicate_get_denied", request) -def connect_persisted_get_status(hash_value, label): +def raw_persisted_get_status(hash_value, label): target = f"{{HOST}}:{{PORT}}" extensions = json.dumps({{"persistedQuery": {{"version": 1, "sha256Hash": hash_value}}}}) encoded = urllib.parse.urlencode({{"operationName": "Viewer", "extensions": extensions}}) @@ -374,9 +350,9 @@ def connect_persisted_get_status(hash_value, label): f"Connection: close\r\n" f"\r\n" ).encode() - return connect_http_status(label, request) + return raw_http_status(label, request) -def connect_chunked_status(query): +def raw_chunked_status(query): target = f"{{HOST}}:{{PORT}}" body = json.dumps({{"query": query}}).encode() chunk = f"{{len(body):x}}\r\n".encode() + body + b"\r\n0\r\n\r\n" @@ -389,7 +365,7 @@ def connect_chunked_status(query): f"Connection: close\r\n" f"\r\n" ).encode() + chunk - return connect_http_status("connect_chunked_query_allowed", request) + return raw_http_status("raw_chunked_query_allowed", request) results = {{ "forward_query_allowed": retry_forward_allowed("forward_query_allowed", lambda: forward_status(QUERY_VIEWER)), @@ -401,15 +377,15 @@ results = {{ "forward_unlisted_field_denied": forward_status(QUERY_REPOSITORY), "forward_mutation_allowed": retry_forward_allowed("forward_mutation_allowed", lambda: forward_status(MUTATION_CREATE)), "forward_deny_rule_denied": forward_status(MUTATION_DELETE), - "connect_query_allowed": connect_status(QUERY_VIEWER, "connect_query_allowed"), - "connect_get_query_allowed": connect_get_status(QUERY_VIEWER, "connect_get_query_allowed"), - "connect_duplicate_get_denied": connect_duplicate_get_status(), - "connect_persisted_get_allowed": connect_persisted_get_status("abc123", "connect_persisted_get_allowed"), - "connect_unregistered_persisted_get_denied": connect_persisted_get_status("missing", "connect_unregistered_persisted_get_denied"), - "connect_chunked_query_allowed": connect_chunked_status(QUERY_VIEWER), - "connect_unlisted_field_denied": connect_status(QUERY_REPOSITORY, "connect_unlisted_field_denied"), - "connect_mutation_allowed": connect_status(MUTATION_CREATE, "connect_mutation_allowed"), - "connect_deny_rule_denied": connect_status(MUTATION_DELETE, "connect_deny_rule_denied"), + "raw_query_allowed": raw_status(QUERY_VIEWER, "raw_query_allowed"), + "raw_get_query_allowed": raw_get_status(QUERY_VIEWER, "raw_get_query_allowed"), + "raw_duplicate_get_denied": raw_duplicate_get_status(), + "raw_persisted_get_allowed": raw_persisted_get_status("abc123", "raw_persisted_get_allowed"), + "raw_unregistered_persisted_get_denied": raw_persisted_get_status("missing", "raw_unregistered_persisted_get_denied"), + "raw_chunked_query_allowed": raw_chunked_status(QUERY_VIEWER), + "raw_unlisted_field_denied": raw_status(QUERY_REPOSITORY, "raw_unlisted_field_denied"), + "raw_mutation_allowed": raw_status(MUTATION_CREATE, "raw_mutation_allowed"), + "raw_deny_rule_denied": raw_status(MUTATION_DELETE, "raw_deny_rule_denied"), }} results.update(DETAILS) print(json.dumps(results, sort_keys=True)) @@ -432,15 +408,15 @@ print(json.dumps(results, sort_keys=True)) ("forward_unlisted_field_denied", 403), ("forward_mutation_allowed", 200), ("forward_deny_rule_denied", 403), - ("connect_query_allowed", 200), - ("connect_get_query_allowed", 200), - ("connect_duplicate_get_denied", 403), - ("connect_persisted_get_allowed", 200), - ("connect_unregistered_persisted_get_denied", 403), - ("connect_chunked_query_allowed", 200), - ("connect_unlisted_field_denied", 403), - ("connect_mutation_allowed", 200), - ("connect_deny_rule_denied", 403), + ("raw_query_allowed", 200), + ("raw_get_query_allowed", 200), + ("raw_duplicate_get_denied", 403), + ("raw_persisted_get_allowed", 200), + ("raw_unregistered_persisted_get_denied", 403), + ("raw_chunked_query_allowed", 200), + ("raw_unlisted_field_denied", 403), + ("raw_mutation_allowed", 200), + ("raw_deny_rule_denied", 403), ] { let expected_fragment = format!(r#""{key}": {expected}"#); assert!( diff --git a/e2e/rust/tests/forward_proxy_jsonrpc_l7.rs b/e2e/rust/tests/forward_proxy_jsonrpc_l7.rs index 174e3b6db9..b46dac1313 100644 --- a/e2e/rust/tests/forward_proxy_jsonrpc_l7.rs +++ b/e2e/rust/tests/forward_proxy_jsonrpc_l7.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! E2E tests for JSON-RPC L7 inspection across both proxy entry points. +//! E2E tests for JSON-RPC L7 inspection through transparent interception. //! //! The upstream server deliberately does not implement JSON-RPC. `OpenShell` //! parses and enforces JSON-RPC before forwarding, so any HTTP server that @@ -187,7 +187,7 @@ network_policies: #[tokio::test] #[allow(clippy::too_many_lines)] -async fn jsonrpc_l7_enforces_method_rules_on_forward_and_connect_paths() { +async fn jsonrpc_l7_enforces_high_level_and_raw_transparent_paths() { let server = start_test_server(RULES_TEST_SERVER_ALIAS) .await .expect("start test server"); @@ -201,25 +201,15 @@ async fn jsonrpc_l7_enforces_method_rules_on_forward_and_connect_paths() { let script = format!( r#" import json -import os import socket import time import urllib.error -import urllib.parse import urllib.request HOST = {host:?} PORT = {port} DETAILS = {{ "debug_target": {{"host": HOST, "port": PORT}}, - "debug_proxy_env": {{ - "http_proxy": os.environ.get("http_proxy"), - "https_proxy": os.environ.get("https_proxy"), - "HTTP_PROXY": os.environ.get("HTTP_PROXY"), - "HTTPS_PROXY": os.environ.get("HTTPS_PROXY"), - "NO_PROXY": os.environ.get("NO_PROXY"), - "no_proxy": os.environ.get("no_proxy"), - }}, }} def text(data): @@ -291,11 +281,6 @@ def post_invalid_json(label): except urllib.error.HTTPError as error: return record_http_error(label, error, text(encoded)) -def proxy_parts(*names): - proxy_url = next((os.environ.get(name) for name in names if os.environ.get(name)), None) - parsed = urllib.parse.urlparse(proxy_url) - return parsed.hostname, parsed.port or 80 - def read_until(sock, marker): data = b"" while marker not in data: @@ -339,21 +324,11 @@ def record_raw_response(label, response, body=b""): DETAILS[f"{{label}}_body"] = text(body) return code -def connect_http_status(label, request): - proxy_host, proxy_port = proxy_parts("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - target = f"{{HOST}}:{{PORT}}" - +def raw_http_status(label, request): last_error = None for attempt in range(5): try: - with socket.create_connection((proxy_host, proxy_port), timeout=15) as sock: - sock.sendall( - f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode() - ) - connect_response = read_until(sock, b"\r\n\r\n") - connect_code = record_raw_response(f"{{label}}_connect", connect_response) - if connect_code != 200: - return connect_code + with socket.create_connection((HOST, PORT), timeout=15) as sock: sock.sendall(request) sock.shutdown(socket.SHUT_WR) response, body = read_response(sock) @@ -365,7 +340,7 @@ def connect_http_status(label, request): raise RuntimeError(f"{{label}}: failed after 5 attempts: {{last_error}}") -def connect_jsonrpc_status(method, params, label): +def raw_jsonrpc_status(method, params, label): target = f"{{HOST}}:{{PORT}}" body = {{"jsonrpc": "2.0", "id": 1, "method": method}} if params is not None: @@ -379,7 +354,7 @@ def connect_jsonrpc_status(method, params, label): f"Connection: close\r\n" f"\r\n" ).encode() + encoded - return connect_http_status(label, request) + return raw_http_status(label, request) results = {{ # forward proxy — method-only allow rules @@ -406,12 +381,12 @@ results = {{ # forward proxy — invalid JSON body fails closed before generic rules apply "forward_invalid_json_denied": post_invalid_json("forward_invalid_json_denied"), - # CONNECT path — representative allowed and denied cases - "connect_method_initialize_allowed": connect_jsonrpc_status("initialize", {{"protocolVersion": "2025-11-25", "capabilities": {{}}}}, "connect_method_initialize_allowed"), - "connect_method_tools_list_allowed": connect_jsonrpc_status("tools/list", None, "connect_method_tools_list_allowed"), - "connect_method_tools_call_allowed": connect_jsonrpc_status("tools/call", {{"name": "read_status"}}, "connect_method_tools_call_allowed"), - "connect_method_tools_call_with_unmatched_params_allowed": connect_jsonrpc_status("tools/call", {{"name": "blocked_action", "arguments": {{"scope": "ignored"}}}}, "connect_method_tools_call_with_unmatched_params_allowed"), - "connect_method_tools_delete_denied": connect_jsonrpc_status("tools/delete", {{"name": "purge_cache"}}, "connect_method_tools_delete_denied"), + # raw socket path — representative allowed and denied cases + "raw_method_initialize_allowed": raw_jsonrpc_status("initialize", {{"protocolVersion": "2025-11-25", "capabilities": {{}}}}, "raw_method_initialize_allowed"), + "raw_method_tools_list_allowed": raw_jsonrpc_status("tools/list", None, "raw_method_tools_list_allowed"), + "raw_method_tools_call_allowed": raw_jsonrpc_status("tools/call", {{"name": "read_status"}}, "raw_method_tools_call_allowed"), + "raw_method_tools_call_with_unmatched_params_allowed": raw_jsonrpc_status("tools/call", {{"name": "blocked_action", "arguments": {{"scope": "ignored"}}}}, "raw_method_tools_call_with_unmatched_params_allowed"), + "raw_method_tools_delete_denied": raw_jsonrpc_status("tools/delete", {{"name": "purge_cache"}}, "raw_method_tools_delete_denied"), }} results.update(DETAILS) print(json.dumps(results, sort_keys=True)) @@ -440,16 +415,13 @@ print(json.dumps(results, sort_keys=True)) ("forward_batch_one_denied", 403), // forward proxy — parse error ("forward_invalid_json_denied", 403), - // CONNECT path — allowed - ("connect_method_initialize_allowed", 200), - ("connect_method_tools_list_allowed", 200), - ("connect_method_tools_call_allowed", 200), - ( - "connect_method_tools_call_with_unmatched_params_allowed", - 200, - ), - // CONNECT path — method denied - ("connect_method_tools_delete_denied", 403), + // raw socket path — allowed + ("raw_method_initialize_allowed", 200), + ("raw_method_tools_list_allowed", 200), + ("raw_method_tools_call_allowed", 200), + ("raw_method_tools_call_with_unmatched_params_allowed", 200), + // raw socket path — method denied + ("raw_method_tools_delete_denied", 403), ] { let expected_fragment = format!(r#""{key}": {expected}"#); assert!( diff --git a/e2e/rust/tests/forward_proxy_l7_bypass.rs b/e2e/rust/tests/forward_proxy_l7_bypass.rs index f5df4f53e3..e346ac169a 100644 --- a/e2e/rust/tests/forward_proxy_l7_bypass.rs +++ b/e2e/rust/tests/forward_proxy_l7_bypass.rs @@ -14,9 +14,7 @@ use openshell_e2e::harness::container::ContainerHttpServer; use openshell_e2e::harness::sandbox::SandboxGuard; use tempfile::NamedTempFile; -const TEST_SERVER_ALIAS: &str = "rest-l7.openshell.test"; - -async fn start_test_server() -> Result { +async fn start_test_server(alias: &str) -> Result { let script = r#"from http.server import BaseHTTPRequestHandler, HTTPServer class Handler(BaseHTTPRequestHandler): @@ -34,7 +32,7 @@ class Handler(BaseHTTPRequestHandler): HTTPServer(("0.0.0.0", 8000), Handler).serve_forever() "#; - ContainerHttpServer::start_python(TEST_SERVER_ALIAS, script).await + ContainerHttpServer::start_python(alias, script).await } fn write_policy_with_l7_rules(host: &str, port: u16) -> Result { @@ -98,7 +96,9 @@ network_policies: /// GET /allowed should succeed — the L7 policy explicitly allows it. #[tokio::test] async fn forward_proxy_allows_l7_permitted_request() { - let server = start_test_server().await.expect("start test server"); + let server = start_test_server("rest-l7-allow.openshell.test") + .await + .expect("start test server"); let policy = write_policy_with_l7_rules(&server.host, server.port).expect("write custom policy"); let policy_path = policy @@ -148,7 +148,9 @@ print(json.dumps(last)) /// POST /allowed should be denied — the L7 policy only allows GET. #[tokio::test] async fn forward_proxy_denies_l7_blocked_request() { - let server = start_test_server().await.expect("start test server"); + let server = start_test_server("rest-l7-deny.openshell.test") + .await + .expect("start test server"); let policy = write_policy_with_l7_rules(&server.host, server.port).expect("write custom policy"); let policy_path = policy diff --git a/e2e/rust/tests/gateway_start.rs b/e2e/rust/tests/gateway_start.rs index cca35e3d59..31ffabb003 100644 --- a/e2e/rust/tests/gateway_start.rs +++ b/e2e/rust/tests/gateway_start.rs @@ -26,12 +26,21 @@ const STOPPED_READY_MARKER: &str = "gateway-start-stopped-ready"; const START_FILE: &str = "/sandbox/gateway-start-state"; const SANDBOX_NAMESPACE_LABEL: &str = "openshell.ai/sandbox-namespace"; const SANDBOX_NAME_LABEL: &str = "openshell.ai/sandbox-name"; +const SANDBOX_ROLE_LABEL_FILTER: &str = "label=openshell.ai/isolation-role=sandbox"; fn sandbox_container_id(namespace: &str, sandbox_name: &str) -> Result { let namespace_filter = format!("label={SANDBOX_NAMESPACE_LABEL}={namespace}"); let sandbox_name_filter = format!("label={SANDBOX_NAME_LABEL}={sandbox_name}"); let output = Command::new("docker") - .args(["ps", "-aq", "--filter", MANAGED_BY_LABEL_FILTER, "--filter"]) + .args([ + "ps", + "-aq", + "--filter", + MANAGED_BY_LABEL_FILTER, + "--filter", + SANDBOX_ROLE_LABEL_FILTER, + "--filter", + ]) .arg(namespace_filter) .args(["--filter"]) .arg(sandbox_name_filter) diff --git a/e2e/rust/tests/internet_network_perf.rs b/e2e/rust/tests/internet_network_perf.rs new file mode 100644 index 0000000000..70c0f1af25 --- /dev/null +++ b/e2e/rust/tests/internet_network_perf.rs @@ -0,0 +1,170 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Opt-in Internet benchmark for the complete sandbox-to-supervisor data path. + +#![cfg(feature = "e2e-host-gateway")] + +use std::io::Write as _; +use std::process::Stdio; + +use openshell_e2e::harness::sandbox::SandboxGuard; +use tempfile::NamedTempFile; + +const BENCHMARK: &str = r#" +import http.client +import json +import os +import socket +import ssl +import statistics +import time + +HOST = "example.com" +ITERATIONS = int(os.environ.get("OPENSHELL_INET_PERF_ITERATIONS", "20")) + +def percentile(values, fraction): + values = sorted(values) + return values[min(len(values) - 1, int((len(values) - 1) * fraction))] + +def measure(name, operation, iterations=ITERATIONS): + operation() + samples = [] + started = time.perf_counter_ns() + for _ in range(iterations): + before = time.perf_counter_ns() + operation() + samples.append((time.perf_counter_ns() - before) / 1_000_000) + elapsed = (time.perf_counter_ns() - started) / 1_000_000_000 + return { + "name": name, + "iterations": iterations, + "mean_ms": statistics.fmean(samples), + "p50_ms": percentile(samples, 0.50), + "p95_ms": percentile(samples, 0.95), + "ops_per_second": iterations / elapsed, + } + +def dns_lookup(): + result = socket.getaddrinfo(HOST, 443, socket.AF_INET, socket.SOCK_STREAM) + if not result: + raise RuntimeError("DNS returned no IPv4 addresses") + +def tcp_connect(): + with socket.create_connection((HOST, 443), timeout=10): + pass + +tls_context = ssl.create_default_context() + +def https_cold(): + connection = http.client.HTTPSConnection(HOST, 443, timeout=10, context=tls_context) + try: + connection.request("HEAD", "/", headers={"Connection": "close"}) + response = connection.getresponse() + response.read() + if response.status != 200: + raise RuntimeError(f"unexpected HTTP status {response.status}") + finally: + connection.close() + +warm_connection = http.client.HTTPSConnection(HOST, 443, timeout=10, context=tls_context) + +def https_reuse(): + warm_connection.request("HEAD", "/", headers={"Connection": "keep-alive"}) + response = warm_connection.getresponse() + response.read() + if response.status != 200: + raise RuntimeError(f"unexpected HTTP status {response.status}") + +try: + metrics = [ + measure("dns_lookup", dns_lookup), + measure("tcp_connect", tcp_connect), + measure("https_cold", https_cold, max(5, ITERATIONS // 2)), + measure("https_reuse", https_reuse), + ] +finally: + warm_connection.close() + +print(json.dumps({"metrics": metrics}, separators=(",", ":"))) +"#; + +fn write_policy() -> Result { + let mut file = NamedTempFile::new().map_err(|error| format!("create policy: {error}"))?; + write!( + file, + r#"version: 1 + +filesystem_policy: + include_workdir: true + read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log] + read_write: [/sandbox, /tmp, /dev/null] + +landlock: + compatibility: best_effort + +network_policies: + internet_performance: + name: internet_performance + endpoints: + - host: example.com + port: 80 + protocol: tcp + - host: example.com + port: 443 + protocol: tcp + binaries: + - path: "/**" +"# + ) + .map_err(|error| format!("write policy: {error}"))?; + file.flush() + .map_err(|error| format!("flush policy: {error}"))?; + Ok(file) +} + +async fn run_host_benchmark() -> Result { + let output = tokio::process::Command::new("python3") + .args(["-c", BENCHMARK]) + .env("OPENSHELL_INET_PERF_ITERATIONS", "20") + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .await + .map_err(|error| format!("run host benchmark: {error}"))?; + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + if !output.status.success() { + return Err(format!("host benchmark failed: {combined}")); + } + Ok(combined) +} + +#[tokio::test] +#[ignore = "manual Internet performance benchmark"] +async fn benchmark_complete_internet_path() { + let policy = write_policy().expect("write Internet benchmark policy"); + let policy_path = policy.path().to_str().expect("UTF-8 policy path"); + let sandbox = SandboxGuard::create(&["--policy", policy_path]) + .await + .expect("create benchmark sandbox"); + + for round in 1..=3 { + let host = run_host_benchmark().await.expect("host benchmark"); + println!("INTERNET_PERF host round={round} {}", host.trim()); + let mediated = sandbox + .exec(&[ + "sh", + "-c", + "OPENSHELL_INET_PERF_ITERATIONS=20 python3 -c \"$1\"", + "openshell-internet-perf", + BENCHMARK, + ]) + .await + .expect("sandbox benchmark"); + println!("INTERNET_PERF sandbox round={round} {}", mediated.trim()); + } +} diff --git a/e2e/rust/tests/kubernetes_corporate_proxy.rs b/e2e/rust/tests/kubernetes_corporate_proxy.rs index 0e83a1a799..3fc1c12a30 100644 --- a/e2e/rust/tests/kubernetes_corporate_proxy.rs +++ b/e2e/rust/tests/kubernetes_corporate_proxy.rs @@ -6,7 +6,7 @@ //! Kubernetes wiring coverage for authenticated corporate forward proxies. //! //! The shell wrapper configures the gateway before it starts, creates the -//! credential Secret, and forces sidecar topology. This test starts a proxy +//! credential Secret, and enables the Kubernetes sandbox runtime. This test starts a proxy //! and an HTTPS upstream on the host visible to sandbox pods, then proves that //! a permitted request uses authenticated CONNECT while a policy-denied port //! never reaches the proxy. diff --git a/e2e/rust/tests/live_internet_traffic_perf.rs b/e2e/rust/tests/live_internet_traffic_perf.rs new file mode 100644 index 0000000000..e8727bbce3 --- /dev/null +++ b/e2e/rust/tests/live_internet_traffic_perf.rs @@ -0,0 +1,337 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Opt-in live-Internet benchmark shaped like a short coding-agent session. + +#![cfg(feature = "e2e-host-gateway")] + +use std::io::Write as _; +use std::process::Stdio; +use std::time::Instant; + +use openshell_e2e::harness::sandbox::SandboxGuard; +use tempfile::NamedTempFile; + +const DEFAULT_ROUNDS: usize = 10; + +const BENCHMARK: &str = r#" +import concurrent.futures +import http.client +import json +import os +import shutil +import socket +import ssl +import statistics +import subprocess +import tempfile +import time +import urllib.parse + +MODE = os.environ.get("OPENSHELL_LIVE_PERF_MODE", "direct") +TLS = ssl.create_default_context() +USER_AGENT = "OpenShell-live-Internet-benchmark/1" + +METADATA_URLS = [ + "https://raw.githubusercontent.com/NVIDIA/OpenShell/main/README.md", + "https://pypi.org/pypi/requests/json", + "https://registry.npmjs.org/typescript/latest", + "https://docs.python.org/3/", +] +DNS_HOSTS = [ + "github.com", + "raw.githubusercontent.com", + "pypi.org", + "registry.npmjs.org", + "crates.io", + "docs.python.org", + "speed.cloudflare.com", +] + +def percentile(values, fraction): + values = sorted(values) + return values[min(len(values) - 1, int((len(values) - 1) * fraction))] + +def request(url, method="GET", max_bytes=None): + parsed = urllib.parse.urlsplit(url) + connection = http.client.HTTPSConnection(parsed.hostname, parsed.port or 443, timeout=20, context=TLS) + path = parsed.path or "/" + if parsed.query: + path += "?" + parsed.query + started = time.perf_counter_ns() + connection.request(method, path, headers={"User-Agent": USER_AGENT, "Connection": "close"}) + response = connection.getresponse() + first = response.read(1) + first_byte_ms = (time.perf_counter_ns() - started) / 1_000_000 + body_bytes = len(first) + while max_bytes is None or body_bytes < max_bytes: + remaining = None if max_bytes is None else max_bytes - body_bytes + chunk = response.read(65536 if remaining is None else min(65536, remaining)) + if not chunk: + break + body_bytes += len(chunk) + total_ms = (time.perf_counter_ns() - started) / 1_000_000 + status = response.status + connection.close() + if status < 200 or status >= 400: + raise RuntimeError(f"{url} returned HTTP {status}") + return {"status": status, "bytes": body_bytes, "first_byte_ms": first_byte_ms, "total_ms": total_ms} + +def metric(name, operation): + started = time.perf_counter_ns() + try: + detail = operation() + return { + "name": name, + "ok": True, + "elapsed_ms": (time.perf_counter_ns() - started) / 1_000_000, + "detail": detail, + } + except Exception as error: + return { + "name": name, + "ok": False, + "elapsed_ms": (time.perf_counter_ns() - started) / 1_000_000, + "error": f"{type(error).__name__}: {error}", + } + +def dns_set(): + samples = [] + for host in DNS_HOSTS: + started = time.perf_counter_ns() + addresses = socket.getaddrinfo(host, 443, socket.AF_UNSPEC, socket.SOCK_STREAM) + samples.append({ + "host": host, + "elapsed_ms": (time.perf_counter_ns() - started) / 1_000_000, + "addresses": len(addresses), + }) + timings = [sample["elapsed_ms"] for sample in samples] + return { + "lookups": samples, + "p50_ms": percentile(timings, 0.50), + "p95_ms": percentile(timings, 0.95), + } + +def metadata_serial(): + return {"requests": [request(url, max_bytes=2_000_000) for url in METADATA_URLS]} + +def metadata_concurrent(): + urls = METADATA_URLS + METADATA_URLS + with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool: + responses = list(pool.map(lambda url: request(url, max_bytes=2_000_000), urls)) + return {"requests": responses} + +def https_reuse(): + connection = http.client.HTTPSConnection("example.com", 443, timeout=20, context=TLS) + samples = [] + try: + for _ in range(20): + started = time.perf_counter_ns() + connection.request("HEAD", "/", headers={"User-Agent": USER_AGENT, "Connection": "keep-alive"}) + response = connection.getresponse() + response.read() + if response.status != 200: + raise RuntimeError(f"example.com returned HTTP {response.status}") + samples.append((time.perf_counter_ns() - started) / 1_000_000) + finally: + connection.close() + return { + "requests": len(samples), + "p50_ms": percentile(samples, 0.50), + "p95_ms": percentile(samples, 0.95), + "mean_ms": statistics.fmean(samples), + } + +def git_clone(): + if shutil.which("git") is None: + return {"skipped": "git is not installed in the workload image"} + with tempfile.TemporaryDirectory(prefix="openshell-live-git-") as directory: + checkout = os.path.join(directory, "sampleproject") + command = [ + "git", "-c", "advice.detachedHead=false", "clone", "--quiet", + "--depth", "1", "--filter=blob:none", + "https://github.com/pypa/sampleproject.git", checkout, + ] + output = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=60, check=False) + if output.returncode != 0: + raise RuntimeError(output.stderr.decode(errors="replace")[-500:]) + files = sum(len(names) for _, _, names in os.walk(checkout)) + size = sum(os.path.getsize(os.path.join(root, name)) for root, _, names in os.walk(checkout) for name in names) + return {"files": files, "checkout_bytes": size} + +def package_download(): + metadata = request("https://pypi.org/pypi/idna/json", max_bytes=2_000_000) + connection = http.client.HTTPSConnection("pypi.org", 443, timeout=20, context=TLS) + connection.request("GET", "/pypi/idna/json", headers={"User-Agent": USER_AGENT}) + response = connection.getresponse() + document = json.loads(response.read()) + connection.close() + wheels = [entry for entry in document["urls"] if entry["packagetype"] == "bdist_wheel"] + if not wheels: + raise RuntimeError("PyPI returned no idna wheel") + artifact = request(wheels[0]["url"], max_bytes=5_000_000) + return {"metadata": metadata, "artifact": artifact, "filename": wheels[0]["filename"]} + +def bulk_download(): + return request("https://speed.cloudflare.com/__down?bytes=5242880", max_bytes=5242880) + +def denied_destination(): + if MODE != "sandbox": + return {"skipped": "policy denial applies only to sandbox mode"} + started = time.perf_counter_ns() + try: + socket.create_connection(("example.net", 443), timeout=5).close() + except OSError as error: + return { + "denied": True, + "latency_ms": (time.perf_counter_ns() - started) / 1_000_000, + "error": str(error), + } + raise RuntimeError("destination omitted from policy was reachable") + +started = time.perf_counter_ns() +metrics = [ + metric("dns_service_set", dns_set), + metric("https_metadata_serial", metadata_serial), + metric("https_metadata_concurrent", metadata_concurrent), + metric("https_reuse", https_reuse), + metric("git_clone", git_clone), + metric("package_download", package_download), + metric("bulk_download_5mib", bulk_download), + metric("policy_denial", denied_destination), +] +document = { + "schema": "openshell.live-internet-perf.v1", + "mode": MODE, + "total_ms": (time.perf_counter_ns() - started) / 1_000_000, + "metrics": metrics, +} +print(json.dumps(document, separators=(",", ":"))) +if any(item["name"] == "policy_denial" and not item["ok"] for item in metrics): + raise SystemExit(1) +"#; + +fn rounds() -> usize { + std::env::var("OPENSHELL_LIVE_PERF_ROUNDS") + .ok() + .and_then(|value| value.parse().ok()) + .filter(|value| *value > 0) + .unwrap_or(DEFAULT_ROUNDS) +} + +fn write_policy() -> Result { + let mut file = NamedTempFile::new().map_err(|error| format!("create policy: {error}"))?; + write!( + file, + r#"version: 1 + +filesystem_policy: + include_workdir: true + read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log] + read_write: [/sandbox, /tmp, /dev/null] + +landlock: + compatibility: best_effort + +network_policies: + live_internet_performance: + name: live_internet_performance + endpoints: + - {{ host: example.com, port: 443, protocol: tcp }} + - {{ host: github.com, port: 443, protocol: tcp }} + - {{ host: raw.githubusercontent.com, port: 443, protocol: tcp }} + - {{ host: pypi.org, port: 443, protocol: tcp }} + - {{ host: files.pythonhosted.org, port: 443, protocol: tcp }} + - {{ host: registry.npmjs.org, port: 443, protocol: tcp }} + - {{ host: crates.io, port: 443, protocol: tcp }} + - {{ host: docs.python.org, port: 443, protocol: tcp }} + - {{ host: speed.cloudflare.com, port: 443, protocol: tcp }} + binaries: + - path: "/**" +"# + ) + .map_err(|error| format!("write policy: {error}"))?; + file.flush() + .map_err(|error| format!("flush policy: {error}"))?; + Ok(file) +} + +async fn run_direct() -> Result { + run_python("direct").await +} + +async fn run_python(mode: &str) -> Result { + let output = tokio::process::Command::new("python3") + .args(["-c", BENCHMARK]) + .env("OPENSHELL_LIVE_PERF_MODE", mode) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .await + .map_err(|error| format!("run {mode} live Internet benchmark: {error}"))?; + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + if !output.status.success() { + return Err(format!( + "{mode} live Internet benchmark failed (exit {:?}): {combined}", + output.status.code() + )); + } + Ok(combined) +} + +async fn run_sandbox(sandbox: &SandboxGuard) -> Result { + sandbox + .exec(&[ + "sh", + "-c", + "OPENSHELL_LIVE_PERF_MODE=sandbox python3 -c \"$1\"", + "openshell-live-internet-perf", + BENCHMARK, + ]) + .await +} + +#[tokio::test] +#[ignore = "manual live-Internet performance benchmark"] +async fn benchmark_live_internet_agent_traffic() { + let policy = write_policy().expect("write live Internet benchmark policy"); + let policy_path = policy.path().to_str().expect("UTF-8 policy path"); + let create_started = Instant::now(); + let mut sandbox = SandboxGuard::create(&["--policy", policy_path]) + .await + .expect("create live Internet benchmark sandbox"); + println!( + "LIVE_INTERNET_PERF create {{\"elapsed_ms\":{}}}", + create_started.elapsed().as_secs_f64() * 1000.0 + ); + + for round in 1..=rounds() { + if round % 2 == 1 { + let direct = run_direct().await.expect("direct live Internet benchmark"); + println!("LIVE_INTERNET_PERF direct round={round} {}", direct.trim()); + let mediated = run_sandbox(&sandbox) + .await + .expect("sandbox live Internet benchmark"); + println!( + "LIVE_INTERNET_PERF sandbox round={round} {}", + mediated.trim() + ); + } else { + let mediated = run_sandbox(&sandbox) + .await + .expect("sandbox live Internet benchmark"); + println!( + "LIVE_INTERNET_PERF sandbox round={round} {}", + mediated.trim() + ); + let direct = run_direct().await.expect("direct live Internet benchmark"); + println!("LIVE_INTERNET_PERF direct round={round} {}", direct.trim()); + } + } + + sandbox.cleanup().await; +} diff --git a/e2e/rust/tests/live_policy_update.rs b/e2e/rust/tests/live_policy_update.rs index 7618696c62..ce47d3c8b7 100644 --- a/e2e/rust/tests/live_policy_update.rs +++ b/e2e/rust/tests/live_policy_update.rs @@ -32,30 +32,6 @@ use openshell_e2e::harness::output::{extract_field, strip_ansi}; use openshell_e2e::harness::sandbox::SandboxGuard; use tempfile::NamedTempFile; -#[cfg(feature = "e2e-docker")] -const LOCAL_OVERRIDE_REGO: &str = include_str!(concat!( - env!("CARGO_MANIFEST_DIR"), - "/../../crates/openshell-supervisor-network/data/sandbox-policy.rego" -)); - -#[cfg(feature = "e2e-docker")] -const LOCAL_OVERRIDE_DOCKERFILE: &str = r#"FROM public.ecr.aws/docker/library/python:3.13-slim - -RUN apt-get update && apt-get install -y --no-install-recommends iproute2 \ - && rm -rf /var/lib/apt/lists/* -RUN groupadd -g 1000660000 sandbox && \ - useradd -m -u 1000660000 -g sandbox sandbox - -COPY local-policy.rego /etc/openshell/local-policy.rego -COPY local-policy.yaml /etc/openshell/local-policy.yaml - -ENV OPENSHELL_POLICY_RULES=/etc/openshell/local-policy.rego -ENV OPENSHELL_POLICY_DATA=/etc/openshell/local-policy.yaml -ENV OPENSHELL_POLICY_POLL_INTERVAL_SECS=1 - -CMD ["sleep", "infinity"] -"#; - // --------------------------------------------------------------------------- // Policy YAML builders // --------------------------------------------------------------------------- @@ -146,56 +122,6 @@ landlock: Ok(file) } -#[cfg(feature = "e2e-docker")] -fn write_local_override_image() -> Result< - ( - tempfile::TempDir, - openshell_e2e::harness::container::ImageGuard, - ), - String, -> { - let dir = tempfile::tempdir().map_err(|e| format!("create image context: {e}"))?; - let dockerfile = dir.path().join("Dockerfile"); - std::fs::write(&dockerfile, LOCAL_OVERRIDE_DOCKERFILE) - .map_err(|e| format!("write local override Dockerfile: {e}"))?; - std::fs::write(dir.path().join("local-policy.rego"), LOCAL_OVERRIDE_REGO) - .map_err(|e| format!("write local override Rego policy: {e}"))?; - std::fs::write( - dir.path().join("local-policy.yaml"), - r"version: 1 - -filesystem_policy: - include_workdir: true - read_only: - - /usr - - /lib - - /proc - - /dev/urandom - - /etc - read_write: - - /sandbox - - /tmp - - /dev/null - -landlock: - compatibility: best_effort - -process: - run_as_user: sandbox - run_as_group: sandbox - -network_policies: {} -", - ) - .map_err(|e| format!("write local override policy data: {e}"))?; - let image = openshell_e2e::harness::container::ImageGuard::build( - "local-override", - &dockerfile, - dir.path(), - )?; - Ok((dir, image)) -} - // --------------------------------------------------------------------------- // CLI helpers // --------------------------------------------------------------------------- @@ -592,115 +518,3 @@ async fn initial_sparse_policy_is_acknowledged_as_loaded() { guard.cleanup().await; } - -/// An explicit local Rego/data override remains authoritative even when the -/// sandbox has a gateway policy and that policy changes while it is running. -/// Gateway polling must continue for settings and providers without replacing -/// the locally loaded OPA engine. -#[cfg(feature = "e2e-docker")] -#[tokio::test] -async fn local_policy_override_survives_gateway_policy_polls() { - let (_image_context, image) = write_local_override_image().expect("write local override image"); - - let gateway_policy_a_file = write_policy(&["example.com"]).expect("write gateway policy A"); - let gateway_policy_a_path = gateway_policy_a_file - .path() - .to_str() - .expect("gateway policy A path should be utf-8") - .to_string(); - let gateway_policy_b_file = - write_policy(&["example.com", "api.anthropic.com"]).expect("write gateway policy B"); - let gateway_policy_b_path = gateway_policy_b_file - .path() - .to_str() - .expect("gateway policy B path should be utf-8") - .to_string(); - - let mut guard = SandboxGuard::create_keep_with_args( - &[ - "--name", - "e2e-lcl-pol-ovrd", - "--from", - image.tag(), - "--policy", - &gateway_policy_a_path, - "--no-tty", - ], - &["sh", "-c", "echo Ready && sleep infinity"], - "Ready", - ) - .await - .expect("create sandbox with local policy override"); - - // Allow several one-second poll intervals. Before the fix, the first poll - // immediately reloaded gateway policy A over the local override. - tokio::time::sleep(std::time::Duration::from_secs(4)).await; - let initial_logs = run_cli(&[ - "logs", - &guard.name, - "-n", - "500", - "--since", - "1m", - "--source", - "sandbox", - ]) - .await; - assert!( - initial_logs.success, - "fetch initial sandbox logs:\n{}", - initial_logs.output - ); - assert!( - initial_logs - .output - .contains("Loading OPA policy engine from local files"), - "sandbox should load the explicit local policy:\n{}", - initial_logs.output - ); - assert!( - !initial_logs.output.contains("Policy reloaded successfully"), - "the first gateway poll must not replace the local policy:\n{}", - initial_logs.output - ); - - let update = run_cli(&[ - "policy", - "set", - &guard.name, - "--policy", - &gateway_policy_b_path, - ]) - .await; - assert!( - update.success, - "publish gateway policy B:\n{}", - update.output - ); - - // A later gateway revision must also remain observational in local mode. - tokio::time::sleep(std::time::Duration::from_secs(4)).await; - let updated_logs = run_cli(&[ - "logs", - &guard.name, - "-n", - "500", - "--since", - "1m", - "--source", - "sandbox", - ]) - .await; - assert!( - updated_logs.success, - "fetch updated sandbox logs:\n{}", - updated_logs.output - ); - assert!( - !updated_logs.output.contains("Policy reloaded successfully"), - "gateway policy updates must not replace the local override:\n{}", - updated_logs.output - ); - - guard.cleanup().await; -} diff --git a/e2e/rust/tests/local_driver_token_restart.rs b/e2e/rust/tests/local_driver_token_restart.rs index 5223e3a704..5c9661d89c 100644 --- a/e2e/rust/tests/local_driver_token_restart.rs +++ b/e2e/rust/tests/local_driver_token_restart.rs @@ -67,6 +67,7 @@ impl LocalDriver { match self { Self::Docker => vec![ "label=openshell.ai/managed-by=openshell".to_string(), + "label=openshell.ai/isolation-role=sandbox".to_string(), format!("label=openshell.ai/sandbox-namespace={namespace}"), format!("label=openshell.ai/sandbox-name={sandbox_name}"), ], @@ -274,8 +275,24 @@ async fn stop_container_sandbox( sandbox_name: &str, ) -> Result<(), String> { let container_id = sandbox_container_id(engine, driver, namespace, sandbox_name)?; - let token = read_bootstrap_token(engine, &container_id)?; - require_non_expiring_token(&token, "local-driver bootstrap JWT")?; + if driver == LocalDriver::Docker { + run_engine( + engine, + &[ + "exec".to_string(), + container_id.clone(), + "sh".to_string(), + "-c".to_string(), + format!("test ! -r {CONTAINER_TOKEN_MOUNT_PATH}"), + ], + ) + .map_err(|error| { + format!("Docker sandbox workload must not be able to read its bootstrap JWT: {error}") + })?; + } else { + let token = read_bootstrap_token(engine, &container_id)?; + require_non_expiring_token(&token, "local-driver bootstrap JWT")?; + } run_engine(engine, &["stop".to_string(), container_id.clone()])?; wait_for_container_running(engine, &container_id, false, Duration::from_secs(60)).await diff --git a/e2e/rust/tests/no_proxy.rs b/e2e/rust/tests/no_proxy.rs index ced4d02d5f..447c408f8b 100644 --- a/e2e/rust/tests/no_proxy.rs +++ b/e2e/rust/tests/no_proxy.rs @@ -5,7 +5,7 @@ use openshell_e2e::harness::sandbox::SandboxGuard; -fn localhost_bypass_script() -> &'static str { +fn localhost_transparent_script() -> &'static str { r#" import json import os @@ -13,11 +13,8 @@ import threading import urllib.request from http.server import BaseHTTPRequestHandler, HTTPServer -expected_no_proxy = '127.0.0.1,localhost,::1' -assert os.environ['HTTP_PROXY'].startswith('http://') -assert os.environ['HTTPS_PROXY'].startswith('http://') -assert os.environ['NO_PROXY'] == expected_no_proxy -assert os.environ['no_proxy'] == expected_no_proxy +for name in ('HTTP_PROXY', 'HTTPS_PROXY', 'NO_PROXY', 'http_proxy', 'https_proxy', 'no_proxy'): + assert name not in os.environ, f'unexpected proxy environment variable: {name}' class Handler(BaseHTTPRequestHandler): def log_message(self, format, *args): @@ -36,7 +33,7 @@ thread.start() try: with urllib.request.urlopen(f'http://127.0.0.1:{server.server_port}', timeout=10) as response: print(json.dumps({ - 'no_proxy': os.environ['NO_PROXY'], + 'proxy_env_absent': True, 'payload': json.loads(response.read().decode()), }), flush=True) finally: @@ -47,16 +44,16 @@ finally: } #[tokio::test] -async fn sandbox_bypasses_proxy_for_localhost_http() { - let guard = SandboxGuard::create(&["--", "python3", "-c", localhost_bypass_script()]) +async fn sandbox_reaches_localhost_without_proxy_environment() { + let guard = SandboxGuard::create(&["--", "python3", "-c", localhost_transparent_script()]) .await - .expect("sandbox create with localhost proxy bypass check"); + .expect("sandbox create with transparent localhost check"); assert!( - guard.create_output.contains( - r#"{"no_proxy": "127.0.0.1,localhost,::1", "payload": {"message": "hello"}}"# - ), - "expected localhost HTTP request to bypass proxy and succeed:\n{}", + guard + .create_output + .contains(r#"{"proxy_env_absent": true, "payload": {"message": "hello"}}"#), + "expected localhost HTTP request to stay local and succeed:\n{}", guard.create_output ); } diff --git a/e2e/rust/tests/podman_gateway_start.rs b/e2e/rust/tests/podman_gateway_start.rs index 28a9ceb15c..7e159ffa78 100644 --- a/e2e/rust/tests/podman_gateway_start.rs +++ b/e2e/rust/tests/podman_gateway_start.rs @@ -38,10 +38,10 @@ const SANDBOX_NAME_LABEL: &str = "openshell.ai/sandbox-name"; /// harness), fall back to plain `podman`, leaving Linux behavior unchanged. fn podman_command() -> Command { let mut command = Command::new("podman"); - if let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") { - if !socket.is_empty() { - command.arg("--url").arg(format!("unix://{socket}")); - } + if let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") + && !socket.is_empty() + { + command.arg("--url").arg(format!("unix://{socket}")); } command } @@ -49,7 +49,15 @@ fn podman_command() -> Command { fn sandbox_container_running(sandbox_name: &str) -> Result { let sandbox_name_filter = format!("label={SANDBOX_NAME_LABEL}={sandbox_name}"); let output = podman_command() - .args(["ps", "-aq", "--filter", MANAGED_BY_LABEL_FILTER, "--filter"]) + .args([ + "ps", + "-aq", + "--filter", + MANAGED_BY_LABEL_FILTER, + "--filter", + "label=openshell.io/isolation-role=sandbox", + "--filter", + ]) .arg(sandbox_name_filter) .stdout(Stdio::piped()) .stderr(Stdio::piped()) diff --git a/e2e/rust/tests/podman_oci_identity.rs b/e2e/rust/tests/podman_oci_identity.rs index e30516bf09..e74888c84c 100644 --- a/e2e/rust/tests/podman_oci_identity.rs +++ b/e2e/rust/tests/podman_oci_identity.rs @@ -22,7 +22,7 @@ const BASE_IMAGE: &str = "ghcr.io/nvidia/openshell-community/sandboxes/base:late const READY_MARKER: &str = "podman-oci-identity-ready"; const OCI_UID: &str = "2345"; const OCI_GID: &str = "2346"; -const OCI_FALLBACK_POLICY: &str = r#"version: 1 +const OCI_FALLBACK_POLICY: &str = r"version: 1 filesystem_policy: include_workdir: true @@ -32,7 +32,7 @@ landlock: compatibility: best_effort network_policies: {} -"#; +"; struct ImageGuard { engine: ContainerEngine, @@ -128,7 +128,16 @@ fn run_engine(engine: &ContainerEngine, args: &[&str]) -> Result } fn sandbox_container_id(engine: &ContainerEngine, sandbox_name: &str) -> Result { + container_id_for_role(engine, sandbox_name, "sandbox") +} + +fn container_id_for_role( + engine: &ContainerEngine, + sandbox_name: &str, + role: &str, +) -> Result { let name_filter = format!("label=openshell.ai/sandbox-name={sandbox_name}"); + let role_filter = format!("label=openshell.io/isolation-role={role}"); let stdout = run_engine( engine, &[ @@ -138,6 +147,8 @@ fn sandbox_container_id(engine: &ContainerEngine, sandbox_name: &str) -> Result< "label=openshell.managed=true", "--filter", &name_filter, + "--filter", + &role_filter, ], )?; let ids = stdout @@ -233,5 +244,54 @@ async fn podman_uses_oci_identity_and_inspected_image_id() { "Podman sandbox must launch the immutable image ID inspected before creation" ); + assert_isolated_pair(&image, &sandbox, &container_id).await; sandbox.cleanup().await; } + +async fn assert_isolated_pair(image: &ImageGuard, sandbox: &SandboxGuard, container_id: &str) { + let supervisor_id = container_id_for_role(&image.engine, &sandbox.name, "supervisor") + .expect("find separate supervisor companion"); + assert_ne!(supervisor_id, container_id); + for id in [container_id, &supervisor_id] { + let user = run_engine( + &image.engine, + &["inspect", "--format", "{{.Config.User}}", id], + ) + .unwrap(); + assert_eq!(user, format!("{OCI_UID}:{OCI_GID}")); + let caps = run_engine( + &image.engine, + &["inspect", "--format", "{{.EffectiveCaps}}", id], + ) + .unwrap(); + assert_eq!( + caps, "[]", + "neither container may have effective capabilities" + ); + } + let network = run_engine( + &image.engine, + &[ + "inspect", + "--format", + "{{.HostConfig.NetworkMode}}", + container_id, + ], + ) + .unwrap(); + assert_eq!(network, "none"); + let mounts = run_engine( + &image.engine, + &[ + "inspect", + "--format", + "{{range .Mounts}}{{println .Destination}}{{end}}", + container_id, + ], + ) + .unwrap(); + assert!(!mounts.contains("/etc/openshell/tls")); + assert!(!mounts.contains("/.openshell/supervisor")); + let posture = sandbox.exec(&["sh", "-c", "set -eu; awk '/^CapEff:|^CapBnd:|^NoNewPrivs:/ {print}' /proc/self/status; test ! -r /.openshell/channel/sandbox/server.key; test ! -r /.openshell/supervisor/runtime-descriptor.json"]).await.expect("workload cannot read either control credential set"); + assert!(posture.contains("0000000000000000")); +} diff --git a/e2e/rust/tests/proxy_egress_pipeline.rs b/e2e/rust/tests/proxy_egress_pipeline.rs index a2b9c49a3f..f841e509c1 100644 --- a/e2e/rust/tests/proxy_egress_pipeline.rs +++ b/e2e/rust/tests/proxy_egress_pipeline.rs @@ -3,16 +3,16 @@ #![cfg(feature = "e2e")] -//! E2E coverage for the shared explicit-proxy egress pipeline. +//! E2E coverage for the transparent sandbox egress pipeline. //! -//! These tests exercise behavior that must remain identical while CONNECT and -//! forward HTTP converge on shared authorization, destination, and relay -//! primitives: -//! - live policy reloads affect new requests through both adapters and close a -//! pre-existing CONNECT HTTP stream before its next request is forwarded; +//! Workloads connect directly to their requested destinations. Seccomp +//! notification diverts those sockets to the supervisor without proxy +//! environment variables or explicit CONNECT requests. These tests cover: +//! - live policy reloads affect new requests and close a pre-existing HTTP +//! stream before its next request is forwarded; //! - `tls: skip` selects a byte-transparent TCP relay; //! - provider placeholders in HTTP headers and opted-in REST bodies are -//! resolved through both adapters without appearing in test output. +//! resolved without appearing in test output. use std::io::{self, Error, ErrorKind, Write}; use std::process::Stdio; @@ -22,6 +22,7 @@ use std::sync::{ }; use openshell_e2e::harness::binary::openshell_cmd; +use openshell_e2e::harness::container::{SupportContainer, e2e_network_name}; use openshell_e2e::harness::sandbox::SandboxGuard; use serde_json::Value; use tempfile::{Builder as TempFileBuilder, NamedTempFile}; @@ -359,9 +360,9 @@ network_policies: } fn write_ip_literal_success_policy( - ip: &str, - explicit_port: u16, - implicit_port: u16, + explicit_ip: &str, + implicit_ip: &str, + port: u16, ) -> Result { let mut file = NamedTempFile::new().map_err(|error| format!("create policy: {error}"))?; let policy = format!( @@ -384,13 +385,14 @@ network_policies: name: destination_successes endpoints: - host: {ip} - port: {explicit_port} - allowed_ips: ["{ip}/32"] - - host: {ip} - port: {implicit_port} + port: {port} + allowed_ips: ["{explicit_ip}/32"] + - host: {implicit_ip} + port: {port} binaries: - path: "/**" -"# +"#, + ip = explicit_ip, ); file.write_all(policy.as_bytes()) .map_err(|error| format!("write policy: {error}"))?; @@ -749,26 +751,15 @@ async fn handle_credential_probe(mut stream: TcpStream) -> io::Result<()> { stream.write_all(response.as_bytes()).await } -fn proxy_status_script(host: &str, port: u16) -> String { +fn transparent_status_script(host: &str, port: u16) -> String { format!( r#" import json -import os import socket -import urllib.parse HOST = {host:?} PORT = {port} -def proxy_parts(): - proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) - ) - parsed = urllib.parse.urlparse(proxy_url) - return parsed.hostname, parsed.port or 80 - def read_headers(sock): data = b"" while b"\r\n\r\n" not in data: @@ -782,44 +773,35 @@ def status(response): parts = response.split(None, 2) return int(parts[1]) if len(parts) > 1 else 0 -def forward_status(): - proxy_host, proxy_port = proxy_parts() - target = f"{{HOST}}:{{PORT}}" - with socket.create_connection((proxy_host, proxy_port), timeout=10) as sock: - sock.sendall( - f"GET http://{{target}}/forward HTTP/1.1\r\n" - f"Host: {{target}}\r\nConnection: close\r\n\r\n".encode() - ) - return status(read_headers(sock)) - -def connect_status(): - proxy_host, proxy_port = proxy_parts() +def request_status(path): target = f"{{HOST}}:{{PORT}}" - with socket.create_connection((proxy_host, proxy_port), timeout=10) as sock: - sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode()) - code = status(read_headers(sock)) - if code != 200: - return code - sock.sendall( - f"GET /connect HTTP/1.1\r\nHost: {{target}}\r\nConnection: close\r\n\r\n".encode() - ) - return status(read_headers(sock)) - -print(json.dumps({{"connect": connect_status(), "forward": forward_status()}}, sort_keys=True)) + try: + with socket.create_connection((HOST, PORT), timeout=10) as sock: + sock.sendall( + f"GET {{path}} HTTP/1.1\r\n" + f"Host: {{target}}\r\nConnection: close\r\n\r\n".encode() + ) + return status(read_headers(sock)) + except OSError as error: + return {{"errno": error.errno, "error": repr(error)}} + +print(json.dumps({{ + "first": request_status("/first"), + "second": request_status("/second"), +}}, sort_keys=True)) "#, host = host, port = port, ) } -fn persistent_connect_script(host: &str, port: u16) -> String { +fn persistent_transparent_script(host: &str, port: u16) -> String { format!( r#" import json import os import socket import time -import urllib.parse HOST = {host:?} PORT = {port} @@ -827,15 +809,6 @@ READY = "/tmp/proxy-reload-ready" GO = "/tmp/proxy-reload-go" RESULT = "/tmp/proxy-reload-result" -def proxy_parts(): - proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) - ) - parsed = urllib.parse.urlparse(proxy_url) - return parsed.hostname, parsed.port or 80 - def read_response(sock): data = b"" while b"\r\n\r\n" not in data: @@ -855,15 +828,11 @@ def read_response(sock): body += chunk return int(headers.split(None, 2)[1]) -proxy_host, proxy_port = proxy_parts() target = f"{{HOST}}:{{PORT}}" failed_closed = False second_status = 0 try: - with socket.create_connection((proxy_host, proxy_port), timeout=10) as sock: - sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode()) - if read_response(sock) != 200: - raise RuntimeError("initial CONNECT was denied") + with socket.create_connection((HOST, PORT), timeout=10) as sock: sock.sendall( f"GET /before-reload HTTP/1.1\r\nHost: {{target}}\r\nConnection: keep-alive\r\n\r\n".encode() ) @@ -920,7 +889,7 @@ fn parse_json_line(output: &str) -> Value { } #[tokio::test] -async fn policy_reload_updates_both_adapters_and_closes_existing_http_tunnel() { +async fn policy_reload_updates_transparent_requests_and_closes_existing_http_stream() { let server = KeepAliveHttpServer::start() .await .expect("start keep-alive HTTP server"); @@ -950,7 +919,7 @@ async fn policy_reload_updates_both_adapters_and_closes_existing_http_tunnel() { .await .expect("wait for policy A"); - let persistent_script = persistent_connect_script(TEST_SERVER_HOST, server.port); + let persistent_script = persistent_transparent_script(TEST_SERVER_HOST, server.port); guard .exec(&[ "sh", @@ -960,7 +929,7 @@ async fn policy_reload_updates_both_adapters_and_closes_existing_http_tunnel() { &persistent_script, ]) .await - .expect("start persistent CONNECT client"); + .expect("start persistent transparent client"); wait_for_sandbox_file( &guard, "/tmp/proxy-reload-ready", @@ -968,16 +937,19 @@ async fn policy_reload_updates_both_adapters_and_closes_existing_http_tunnel() { ) .await; - let status_script = proxy_status_script(TEST_SERVER_HOST, server.port); + let status_script = transparent_status_script(TEST_SERVER_HOST, server.port); let before = guard .exec(&["python3", "-c", &status_script]) .await - .expect("exercise both adapters before reload"); + .expect("exercise transparent requests before reload"); let before = parse_json_line(&before); - assert_eq!(before["connect"], 200, "CONNECT before reload: {before}"); assert_eq!( - before["forward"], 200, - "forward HTTP before reload: {before}" + before["first"], 200, + "first request before reload: {before}" + ); + assert_eq!( + before["second"], 200, + "second request before reload: {before}" ); run_cli(&[ @@ -996,7 +968,7 @@ async fn policy_reload_updates_both_adapters_and_closes_existing_http_tunnel() { guard .exec(&["sh", "-c", "touch /tmp/proxy-reload-go"]) .await - .expect("release persistent CONNECT client"); + .expect("release persistent transparent client"); let stale_tunnel = wait_for_sandbox_file( &guard, "/tmp/proxy-reload-result", @@ -1006,16 +978,16 @@ async fn policy_reload_updates_both_adapters_and_closes_existing_http_tunnel() { let stale_tunnel = parse_json_line(&stale_tunnel); assert_eq!( stale_tunnel["failed_closed"], true, - "existing CONNECT HTTP stream forwarded after policy reload: {stale_tunnel}" + "existing transparent HTTP stream forwarded after policy reload: {stale_tunnel}" ); let after = guard .exec(&["python3", "-c", &status_script]) .await - .expect("exercise both adapters after reload"); + .expect("exercise transparent requests after reload"); let after = parse_json_line(&after); - assert_eq!(after["connect"], 403, "CONNECT after reload: {after}"); - assert_eq!(after["forward"], 403, "forward HTTP after reload: {after}"); + assert_ne!(after["first"], 200, "first request after reload: {after}"); + assert_ne!(after["second"], 200, "second request after reload: {after}"); guard.cleanup().await; } @@ -1052,14 +1024,20 @@ async fn ambiguous_policy_update_is_rejected_without_replacing_active_policy() { .await .expect("wait for valid policy"); - let status_script = proxy_status_script(TEST_SERVER_HOST, server.port); + let status_script = transparent_status_script(TEST_SERVER_HOST, server.port); let before = guard .exec(&["python3", "-c", &status_script]) .await - .expect("exercise both adapters before invalid update"); + .expect("exercise transparent requests before invalid update"); let before = parse_json_line(&before); - assert_eq!(before["connect"], 200, "CONNECT before update: {before}"); - assert_eq!(before["forward"], 200, "forward before update: {before}"); + assert_eq!( + before["first"], 200, + "first request before update: {before}" + ); + assert_eq!( + before["second"], 200, + "second request before update: {before}" + ); let history_before = run_cli(&["policy", "list", &guard.name]) .await .expect("list policy history before rejected update"); @@ -1093,15 +1071,15 @@ async fn ambiguous_policy_update_is_rejected_without_replacing_active_policy() { let after_rejection = guard .exec(&["python3", "-c", &status_script]) .await - .expect("exercise both adapters after rejected update"); + .expect("exercise transparent requests after rejected update"); let after_rejection = parse_json_line(&after_rejection); assert_eq!( - after_rejection["connect"], 200, - "CONNECT should keep using the active valid policy: {after_rejection}" + after_rejection["first"], 200, + "first request should keep using the active valid policy: {after_rejection}" ); assert_eq!( - after_rejection["forward"], 200, - "forward HTTP should keep using the active valid policy: {after_rejection}" + after_rejection["second"], 200, + "second request should keep using the active valid policy: {after_rejection}" ); assert!( server.connection_count() > connections_before_rejection, @@ -1112,69 +1090,25 @@ async fn ambiguous_policy_update_is_rejected_without_replacing_active_policy() { } #[tokio::test] -async fn destination_denial_modes_match_across_connect_and_forward_adapters() { +async fn transparent_destination_denials_fail_connect_with_eacces() { let policy = write_destination_denial_policy().expect("write destination denial policy"); let policy_path = policy_path(&policy); let script = r#" import json -import os import socket -import urllib.parse - -proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) -) -parsed = urllib.parse.urlparse(proxy_url) - -def read_response(sock): - data = b"" - while b"\r\n\r\n" not in data: - chunk = sock.recv(4096) - if not chunk: - break - data += chunk - headers, _, body = data.partition(b"\r\n\r\n") - length = 0 - for line in headers.split(b"\r\n")[1:]: - if line.lower().startswith(b"content-length:"): - length = int(line.split(b":", 1)[1].strip()) - while len(body) < length: - chunk = sock.recv(4096) - if not chunk: - break - body += chunk - status = int(headers.split(None, 2)[1]) - return {"status": status, "body": json.loads(body.decode())} - -def connect_result(host, port): - with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as sock: - target = f"{host}:{port}" - sock.sendall(f"CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n\r\n".encode()) - return read_response(sock) - -def forward_result(host, port): - with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as sock: - target = f"{host}:{port}" - sock.sendall( - f"GET http://{target}/probe HTTP/1.1\r\n" - f"Host: {target}\r\nConnection: close\r\n\r\n".encode() - ) - return read_response(sock) targets = { "metadata": ("169.254.169.254", 80), - "loopback": ("127.0.0.1", 80), "control_plane": ("203.0.113.10", 6443), "outside_allowed_ips": ("203.0.113.10", 8080), } result = {} for name, target in targets.items(): - result[name] = { - "connect": connect_result(*target), - "forward": forward_result(*target), - } + try: + with socket.create_connection(target, timeout=10): + result[name] = 0 + except OSError as error: + result[name] = error.errno print(json.dumps(result, sort_keys=True)) "#; @@ -1182,81 +1116,46 @@ print(json.dumps(result, sort_keys=True)) .await .expect("sandbox create"); let result = parse_json_line(&guard.create_output); - for name in [ - "metadata", - "loopback", - "control_plane", - "outside_allowed_ips", - ] { - for adapter in ["connect", "forward"] { - assert_eq!( - result[name][adapter]["status"], 403, - "{name} {adapter}: {result}" - ); - assert_eq!( - result[name][adapter]["body"]["error"], "ssrf_denied", - "{name} {adapter}: {result}" - ); - } + for name in ["metadata", "control_plane", "outside_allowed_ips"] { + assert_eq!(result[name], 13, "{name} should fail with EACCES: {result}"); } - assert_eq!( - result["metadata"]["connect"]["body"]["detail"], - "CONNECT 169.254.169.254:80 blocked: declared endpoint check failed" - ); - assert_eq!( - result["metadata"]["forward"]["body"]["detail"], - "GET 169.254.169.254:80 blocked: declared endpoint check failed" - ); - assert_eq!( - result["control_plane"]["connect"]["body"]["detail"], - "CONNECT 203.0.113.10:6443 blocked: allowed_ips check failed" - ); - assert_eq!( - result["outside_allowed_ips"]["forward"]["body"]["detail"], - "GET 203.0.113.10:8080 blocked: allowed_ips check failed" - ); } #[tokio::test] -async fn explicit_allowed_ips_and_implicit_ip_literals_succeed_through_both_adapters() { - let resolver = SandboxGuard::create(&[ - "--", - "python3", - "-c", - "import socket; print('GATEWAY_IP=' + socket.gethostbyname('host.openshell.internal'))", - ]) - .await - .expect("resolve host gateway inside sandbox"); - let gateway_ip = resolver - .create_output - .lines() - .find_map(|line| line.trim().strip_prefix("GATEWAY_IP=")) - .expect("sandbox gateway IPv4 output") - .parse::() - .expect("host gateway must resolve to IPv4 for this e2e"); - - // Rootless Podman with pasta exposes its trusted host-gateway alias as a - // link-local address. The hostname receives a narrow runtime exemption, - // but the equivalent raw IP literal must remain hard-blocked. Other - // drivers still exercise the successful IP-literal path below. - if gateway_ip.is_loopback() || gateway_ip.is_link_local() || gateway_ip.is_unspecified() { - eprintln!( - "skipping IP-literal success assertions: host gateway {gateway_ip} is always blocked" - ); +async fn explicit_allowed_ips_and_implicit_ip_literals_succeed_transparently() { + if e2e_network_name().is_none() { + eprintln!("skipping IP-literal success assertions without a shared container network"); return; } - let gateway_ip = gateway_ip.to_string(); + const HTTP_SERVER: &str = r#" +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer - let explicit_server = KeepAliveHttpServer::start() - .await - .expect("start explicit allowed_ips server"); - let implicit_server = KeepAliveHttpServer::start() - .await - .expect("start implicit IP-literal server"); - let policy = - write_ip_literal_success_policy(&gateway_ip, explicit_server.port, implicit_server.port) - .expect("write IP literal policy"); +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + body = b"ok" + self.send_response(200) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, format, *args): + pass + +ThreadingHTTPServer(("0.0.0.0", 8000), Handler).serve_forever() +"#; + let explicit_server = + SupportContainer::start_python("explicit-ip.openshell.test", HTTP_SERVER, 8000) + .await + .expect("start explicit allowed_ips support container"); + let implicit_server = + SupportContainer::start_python("implicit-ip.openshell.test", HTTP_SERVER, 8000) + .await + .expect("start implicit IP-literal support container"); + let explicit_ip = explicit_server.ip().expect("explicit support container IP"); + let implicit_ip = implicit_server.ip().expect("implicit support container IP"); + let policy = write_ip_literal_success_policy(&explicit_ip, &implicit_ip, 8000) + .expect("write IP literal policy"); let policy_path = policy_path(&policy); let mut guard = SandboxGuard::create_keep_with_args( &["--policy", &policy_path], @@ -1266,17 +1165,21 @@ async fn explicit_allowed_ips_and_implicit_ip_literals_succeed_through_both_adap .await .expect("create keep sandbox"); - for (mode, port) in [ - ("explicit_allowed_ips", explicit_server.port), - ("implicit_ip_literal", implicit_server.port), + for (mode, destination) in [ + ("explicit_allowed_ips", explicit_ip.as_str()), + ("implicit_ip_literal", implicit_ip.as_str()), ] { let output = guard - .exec(&["python3", "-c", &proxy_status_script(&gateway_ip, port)]) + .exec(&[ + "python3", + "-c", + &transparent_status_script(destination, 8000), + ]) .await .unwrap_or_else(|error| panic!("exercise {mode}: {error}")); let statuses = parse_json_line(&output); - assert_eq!(statuses["connect"], 200, "{mode} CONNECT: {statuses}"); - assert_eq!(statuses["forward"], 200, "{mode} forward: {statuses}"); + assert_eq!(statuses["first"], 200, "{mode} first request: {statuses}"); + assert_eq!(statuses["second"], 200, "{mode} second request: {statuses}"); } guard.cleanup().await; @@ -1290,28 +1193,13 @@ async fn tls_skip_connect_relays_opaque_bytes_bidirectionally() { let policy_path = policy_path(&policy); let script = format!( r#" -import os import socket -import urllib.parse HOST = {host:?} PORT = {port} PAYLOAD = bytes([0x00, 0xff, 0x13, 0x37, 0x80, 0x0a]) + b"not-http-or-tls" + bytes(range(64)) -proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) -) -parsed = urllib.parse.urlparse(proxy_url) -with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as sock: - target = f"{{HOST}}:{{PORT}}" - sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode()) - response = b"" - while b"\r\n\r\n" not in response: - response += sock.recv(4096) - if int(response.split(None, 2)[1]) != 200: - raise RuntimeError("CONNECT was denied") +with socket.create_connection((HOST, PORT), timeout=10) as sock: sock.sendall(PAYLOAD) echoed = b"" while len(echoed) < len(PAYLOAD): @@ -1338,7 +1226,7 @@ print("RAW_RELAY_OK") } #[tokio::test] -async fn middleware_redacts_request_bodies_through_both_adapters() { +async fn middleware_redacts_transparent_request_bodies() { let server = RequestBodyEchoServer::start() .await .expect("start request body echo server"); @@ -1348,21 +1236,12 @@ async fn middleware_redacts_request_bodies_through_both_adapters() { let script = format!( r#" import json -import os import socket -import urllib.parse HOST = {host:?} PORT = {port} SECRET = "sk-1234567890abcdef" -proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) -) -parsed = urllib.parse.urlparse(proxy_url) - def read_response(sock): data = b"" while b"\r\n\r\n" not in data: @@ -1395,22 +1274,12 @@ def request_bytes(target): "Connection: close\r\n\r\n" ).encode() + body -target = f"{{HOST}}:{{PORT}}" -with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as forward_sock: - forward_sock.sendall(request_bytes(f"http://{{target}}/middleware")) - forward = read_response(forward_sock) - -with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as connect_sock: - connect_sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode()) - connect_response = b"" - while b"\r\n\r\n" not in connect_response: - connect_response += connect_sock.recv(4096) - if int(connect_response.split(None, 2)[1]) != 200: - raise RuntimeError("CONNECT was denied") - connect_sock.sendall(request_bytes("/middleware")) - connect = read_response(connect_sock) - -print(json.dumps({{"connect": connect, "forward": forward}}, sort_keys=True)) +def request_once(): + with socket.create_connection((HOST, PORT), timeout=10) as sock: + sock.sendall(request_bytes("/middleware")) + return read_response(sock) + +print(json.dumps({{"first": request_once(), "second": request_once()}}, sort_keys=True)) "#, host = TEST_SERVER_HOST, port = server.port, @@ -1420,44 +1289,29 @@ print(json.dumps({{"connect": connect, "forward": forward}}, sort_keys=True)) .await .expect("sandbox create"); let result = parse_json_line(&guard.create_output); - for adapter in ["connect", "forward"] { + for request in ["first", "second"] { assert_eq!( - result[adapter]["api_key"], "[REDACTED]", - "{adapter} did not deliver the middleware-transformed body: {result}" + result[request]["api_key"], "[REDACTED]", + "{request} did not deliver the middleware-transformed body: {result}" ); } } #[tokio::test] -async fn fail_closed_middleware_blocks_uninspectable_connect_payload_before_upstream() { +async fn fail_closed_middleware_blocks_uninspectable_transparent_payload_before_upstream() { let server = EchoServer::start().await.expect("start TCP echo server"); let policy = write_middleware_policy(TEST_SERVER_HOST, server.port, "", "fail_closed") .expect("write fail-closed middleware policy"); let policy_path = policy_path(&policy); let script = format!( r#" -import os import socket -import urllib.parse HOST = {host:?} PORT = {port} PAYLOAD = bytes([0x00, 0xff, 0x13, 0x37]) + b"not-http-or-tls" -proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) -) -parsed = urllib.parse.urlparse(proxy_url) -with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as sock: - target = f"{{HOST}}:{{PORT}}" - sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode()) - response = b"" - while b"\r\n\r\n" not in response: - response += sock.recv(4096) - if int(response.split(None, 2)[1]) != 200: - raise RuntimeError("CONNECT was denied before tunnel establishment") +with socket.create_connection((HOST, PORT), timeout=10) as sock: sock.sendall(PAYLOAD) denial = b"" while True: @@ -1518,7 +1372,7 @@ print("UNINSPECTABLE_MIDDLEWARE_BLOCKED") } #[tokio::test] -async fn fail_open_middleware_bypasses_uninspectable_tls_skip_connect() { +async fn fail_open_middleware_bypasses_uninspectable_transparent_tls_skip() { let server = EchoServer::start().await.expect("start TCP echo server"); let policy = write_middleware_policy( TEST_SERVER_HOST, @@ -1530,28 +1384,13 @@ async fn fail_open_middleware_bypasses_uninspectable_tls_skip_connect() { let policy_path = policy_path(&policy); let script = format!( r#" -import os import socket -import urllib.parse HOST = {host:?} PORT = {port} PAYLOAD = bytes([0x00, 0xff, 0x13, 0x37, 0x80]) + b"middleware-bypass" -proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) -) -parsed = urllib.parse.urlparse(proxy_url) -with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as sock: - target = f"{{HOST}}:{{PORT}}" - sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode()) - response = b"" - while b"\r\n\r\n" not in response: - response += sock.recv(4096) - if int(response.split(None, 2)[1]) != 200: - raise RuntimeError("CONNECT was denied") +with socket.create_connection((HOST, PORT), timeout=10) as sock: sock.sendall(PAYLOAD) echoed = b"" while len(echoed) < len(PAYLOAD): @@ -1588,7 +1427,7 @@ print("UNINSPECTABLE_MIDDLEWARE_BYPASSED") } #[tokio::test] -async fn forward_pipeline_never_reaches_upstream_as_first_request_overflow() { +async fn transparent_pipeline_never_reaches_upstream_as_first_request_overflow() { let server = PipelineProbeServer::start() .await .expect("start pipeline probe server"); @@ -1603,26 +1442,18 @@ async fn forward_pipeline_never_reaches_upstream_as_first_request_overflow() { let policy_path = policy_path(&policy); let script = format!( r#" -import os import socket -import urllib.parse -proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) -) -parsed = urllib.parse.urlparse(proxy_url) target = "{host}:{port}" first = ( - f"GET http://{{target}}/allowed HTTP/1.1\r\n" + f"GET /allowed HTTP/1.1\r\n" f"Host: {{target}}\r\nConnection: keep-alive\r\n\r\n" ) second = ( - f"POST http://{{target}}/blocked HTTP/1.1\r\n" + f"POST /blocked HTTP/1.1\r\n" f"Host: {{target}}\r\nContent-Length: 0\r\n\r\n" ) -with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) as sock: +with socket.create_connection(({host:?}, {port}), timeout=10) as sock: sock.sendall((first + second).encode()) response = b"" while True: @@ -1630,9 +1461,11 @@ with socket.create_connection((parsed.hostname, parsed.port or 80), timeout=10) if not chunk: break response += chunk -if response.count(b"HTTP/1.1 ") != 1 or b" 200 " not in response.split(b"\r\n", 1)[0]: +responses = response.count(b"HTTP/1.1 ") +first_status = response.split(b"\r\n", 1)[0] +if responses != 2 or b" 200 " not in first_status or b"HTTP/1.1 403 Forbidden" not in response: raise RuntimeError(f"unexpected pipelined response: {{response!r}}") -print("FORWARD_PIPELINE_CLOSED") +print("TRANSPARENT_PIPELINE_DENIED") "#, host = TEST_SERVER_HOST, port = server.port, @@ -1642,8 +1475,8 @@ print("FORWARD_PIPELINE_CLOSED") .await .expect("sandbox create"); assert!( - guard.create_output.contains("FORWARD_PIPELINE_CLOSED"), - "forward proxy did not close after one response:\n{}", + guard.create_output.contains("TRANSPARENT_PIPELINE_DENIED"), + "transparent HTTP stream did not deny the disallowed pipelined request:\n{}", guard.create_output ); @@ -1657,7 +1490,7 @@ print("FORWARD_PIPELINE_CLOSED") } #[tokio::test] -async fn http_credentials_are_rewritten_in_headers_and_bodies_for_both_adapters() { +async fn http_credentials_are_rewritten_in_transparent_headers_and_bodies() { let _provider_lock = PROVIDER_LOCK .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -1682,21 +1515,11 @@ async fn http_credentials_are_rewritten_in_headers_and_bodies_for_both_adapters( import json import os import socket -import urllib.parse HOST = {host:?} PORT = {port} TOKEN = os.environ[{token_env:?}] -def proxy_parts(): - proxy_url = next( - os.environ[name] - for name in ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy") - if os.environ.get(name) - ) - parsed = urllib.parse.urlparse(proxy_url) - return parsed.hostname, parsed.port or 80 - def read_response(sock): data = b"" while b"\r\n\r\n" not in data: @@ -1730,23 +1553,12 @@ def request_bytes(target): "Connection: close\r\n\r\n" ).encode() + body -proxy_host, proxy_port = proxy_parts() -target = f"{{HOST}}:{{PORT}}" -with socket.create_connection((proxy_host, proxy_port), timeout=10) as forward_sock: - forward_sock.sendall(request_bytes(f"http://{{target}}/probe")) - forward = read_response(forward_sock) - -with socket.create_connection((proxy_host, proxy_port), timeout=10) as connect_sock: - connect_sock.sendall(f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n".encode()) - connect_response = b"" - while b"\r\n\r\n" not in connect_response: - connect_response += connect_sock.recv(4096) - if int(connect_response.split(None, 2)[1]) != 200: - raise RuntimeError("CONNECT was denied") - connect_sock.sendall(request_bytes("/probe")) - connect = read_response(connect_sock) - -print(json.dumps({{"connect": connect, "forward": forward}}, sort_keys=True)) +def request_once(): + with socket.create_connection((HOST, PORT), timeout=10) as sock: + sock.sendall(request_bytes("/probe")) + return read_response(sock) + +print(json.dumps({{"first": request_once(), "second": request_once()}}, sort_keys=True)) "#, host = TEST_SERVER_HOST, port = server.port, @@ -1772,18 +1584,18 @@ print(json.dumps({{"connect": connect, "forward": forward}}, sort_keys=True)) let guard = result.expect("sandbox create"); let result = parse_json_line(&guard.create_output); - for adapter in ["connect", "forward"] { + for request in ["first", "second"] { assert_eq!( - result[adapter]["header_resolved"], true, - "{adapter} header placeholder was not resolved: {result}" + result[request]["header_resolved"], true, + "{request} header placeholder was not resolved: {result}" ); assert_eq!( - result[adapter]["body_resolved"], true, - "{adapter} body placeholder was not resolved: {result}" + result[request]["body_resolved"], true, + "{request} body placeholder was not resolved: {result}" ); assert_eq!( - result[adapter]["saw_placeholder"], false, - "{adapter} leaked an unresolved placeholder upstream: {result}" + result[request]["saw_placeholder"], false, + "{request} leaked an unresolved placeholder upstream: {result}" ); } assert!( diff --git a/e2e/rust/tests/sandbox_labels.rs b/e2e/rust/tests/sandbox_labels.rs index 4de545d7bb..90351266c5 100644 --- a/e2e/rust/tests/sandbox_labels.rs +++ b/e2e/rust/tests/sandbox_labels.rs @@ -115,23 +115,26 @@ async fn delete_sandbox(name: &str) { #[tokio::test] #[allow(clippy::too_many_lines)] // end-to-end test exercises full label lifecycle async fn sandbox_labels_are_stored_and_filterable() { + // Keep the unique names below the public 19-character routable-name + // limit. Hex keeps the per-process suffix compact on busy CI hosts. + let suffix = format!("{:x}", std::process::id()); + let dev_backend = format!("lbl-db-{suffix}"); + let staging_backend = format!("lbl-sb-{suffix}"); + let prod_frontend = format!("lbl-pf-{suffix}"); + let dev_data = format!("lbl-dd-{suffix}"); + // Create sandboxes with different labels let name1 = - create_sandbox_with_labels("e2e-lbl-dev-back", &[("env", "dev"), ("team", "backend")]) - .await; + create_sandbox_with_labels(&dev_backend, &[("env", "dev"), ("team", "backend")]).await; - let name2 = create_sandbox_with_labels( - "e2e-lbl-stg-back", - &[("env", "staging"), ("team", "backend")], - ) - .await; + let name2 = + create_sandbox_with_labels(&staging_backend, &[("env", "staging"), ("team", "backend")]) + .await; let name3 = - create_sandbox_with_labels("e2e-lbl-prd-frnt", &[("env", "prod"), ("team", "frontend")]) - .await; + create_sandbox_with_labels(&prod_frontend, &[("env", "prod"), ("team", "frontend")]).await; - let name4 = - create_sandbox_with_labels("e2e-lbl-dev-data", &[("env", "dev"), ("team", "data")]).await; + let name4 = create_sandbox_with_labels(&dev_data, &[("env", "dev"), ("team", "data")]).await; // Test 1: Verify labels are stored in sandbox metadata let details = get_sandbox_details(&name1).await; diff --git a/e2e/rust/tests/transparent_tcp.rs b/e2e/rust/tests/transparent_tcp.rs index 1027654641..6f3f751193 100644 --- a/e2e/rust/tests/transparent_tcp.rs +++ b/e2e/rust/tests/transparent_tcp.rs @@ -356,7 +356,7 @@ print('transparent-tcp-e2e-ok') let logs = wait_for_sandbox_logs(&sandbox.name, |logs| { logs.contains(&format!("-> {FIXTURE_ALIAS}:{FIXTURE_PORT}")) - && logs.contains("transparent_tcp_port_mismatch") + && logs.contains("Denied staged transparent connection") }) .await .expect("wait for sandbox logs"); @@ -364,7 +364,10 @@ print('transparent-tcp-e2e-ok') logs.contains(&format!("-> {FIXTURE_ALIAS}:{FIXTURE_PORT}")), "{logs}" ); - assert!(logs.contains("transparent_tcp_port_mismatch"), "{logs}"); + assert!( + logs.contains("Denied staged transparent connection"), + "{logs}" + ); sandbox.cleanup().await; } diff --git a/e2e/rust/tests/user_namespaces.rs b/e2e/rust/tests/user_namespaces.rs index 53f2c6ec1f..abe29fbc9f 100644 --- a/e2e/rust/tests/user_namespaces.rs +++ b/e2e/rust/tests/user_namespaces.rs @@ -8,8 +8,8 @@ //! Enables `OPENSHELL_ENABLE_USER_NAMESPACES` on the gateway, triggers sandbox //! creation, and inspects the resulting pod spec to confirm: //! 1. `spec.hostUsers` is `false` -//! 2. The container security context includes the extra capabilities -//! (SETUID, SETGID, DAC_READ_SEARCH) required for user namespace operation +//! 2. The container security context requests no added capabilities and +//! drops every capability //! //! The sandbox pod may fail to start in Docker-in-Docker dev clusters where the //! filesystem does not support ID-mapped mounts. The test inspects the pod spec @@ -211,7 +211,7 @@ async fn sandbox_pod_spec_has_user_namespace_fields() { .await; // Inspect capabilities on the agent container. - let caps = kubectl(&[ + let cap_add = kubectl(&[ "get", "pod", &sandbox_name, @@ -221,6 +221,16 @@ async fn sandbox_pod_spec_has_user_namespace_fields() { "jsonpath={.spec.containers[?(@.name=='agent')].securityContext.capabilities.add}", ]) .await; + let cap_drop = kubectl(&[ + "get", + "pod", + &sandbox_name, + "-n", + "openshell", + "-o", + "jsonpath={.spec.containers[?(@.name=='agent')].securityContext.capabilities.drop}", + ]) + .await; // Clean up. stop_child(&mut child).await; @@ -235,18 +245,14 @@ async fn sandbox_pod_spec_has_user_namespace_fields() { "sandbox pod must have spec.hostUsers=false when user namespaces are enabled" ); - // Assert extra capabilities are present. - let caps_val = caps.expect("failed to get capabilities from pod spec"); - for cap in ["SETUID", "SETGID", "DAC_READ_SEARCH"] { - assert!( - caps_val.contains(cap), - "sandbox pod must include {cap} in capabilities when user namespaces are enabled, got: {caps_val}" - ); - } - for cap in ["SYS_ADMIN", "NET_ADMIN", "SYS_PTRACE", "SYSLOG"] { - assert!( - caps_val.contains(cap), - "sandbox pod must include {cap} in capabilities, got: {caps_val}" - ); - } + let cap_add = cap_add.expect("failed to get added capabilities from pod spec"); + assert!( + cap_add.trim().is_empty(), + "sandbox pod must not request added capabilities, got: {cap_add}" + ); + let cap_drop = cap_drop.expect("failed to get dropped capabilities from pod spec"); + assert!( + cap_drop.contains("ALL"), + "sandbox pod must drop every capability, got: {cap_drop}" + ); } diff --git a/e2e/rust/tests/vm_corporate_proxy.rs b/e2e/rust/tests/vm_corporate_proxy.rs index 71c74c272c..1e34e71d0d 100644 --- a/e2e/rust/tests/vm_corporate_proxy.rs +++ b/e2e/rust/tests/vm_corporate_proxy.rs @@ -10,8 +10,7 @@ //! chain end to end: //! //! gateway TOML → VM driver config → driver subprocess argv → per-sandbox -//! overlay staging (credential, CA, supervisor argument list) → guest init -//! script → supervisor CLI parsing → policy evaluation → proxied CONNECT +//! host supervisor configuration → policy evaluation → proxied CONNECT //! //! and asserts the properties only a real run can establish: //! @@ -25,12 +24,9 @@ //! 6. An incoherent setting is fatal at gateway startup rather than //! degrading to a direct dial. //! -//! Fixtures run as host processes and are reached from the guest through -//! gvproxy's `host.openshell.internal` alias, which is also what proves the -//! documented host-loopback reachability rule for the libkrun backend. That -//! rule is libkrun-specific: QEMU/TAP sandboxes (GPU) cannot reach a -//! gateway-host proxy at all, and the driver rejects such a configuration at -//! launch — see `qemu_backend_rejects_a_gateway_host_proxy` in the driver. +//! Fixtures run as host processes and are reached by the host supervisor. +//! `host.openshell.internal` is normalized to host loopback for both libkrun +//! and QEMU guests because no workload networking leaves the VM. use std::fmt::Write as _; use std::io::Write as _; @@ -45,12 +41,9 @@ use openshell_e2e::harness::sandbox::SandboxGuard; use serial_test::serial; use tempfile::NamedTempFile; -/// The gvproxy host alias seeded into every guest's `/etc/hosts`. A host-bound -/// fixture is reachable from the guest only through this name. +/// The OpenShell alias for the gateway host. const HOST_ALIAS: &str = "host.openshell.internal"; -/// The address `HOST_ALIAS` resolves to inside the guest, and therefore the -/// CONNECT target the supervisor validates and sends to the proxy. -const GVPROXY_HOST_LOOPBACK_IP: &str = "192.168.127.254"; +const HOST_LOOPBACK_IP: &str = "127.0.0.1"; const PROXY_USER: &str = "proxyuser"; const PROXY_PASS: &str = "proxypass"; @@ -79,18 +72,10 @@ impl FixturePorts { } } -/// Python that maps the guest-side host address back to the host loopback. -/// -/// The supervisor sends the *validated* CONNECT target, which inside the guest -/// is gvproxy's host-loopback address. That address means nothing on the host -/// where the proxy fixture runs, so the fixture rewrites it to `127.0.0.1` -/// when dialing — standing in for the corporate network's own routing to the -/// destination. The target the fixture *logs* is left untouched, so the -/// validated-IP assertion still tests the supervisor's behaviour rather than -/// the fixture's. +/// Python helper retained by the shared proxy fixture shape. const HOST_REWRITE: &str = " def dial_host(host): - return '127.0.0.1' if host == '192.168.127.254' else host + return host "; /// A forward proxy that requires Basic auth and logs every CONNECT it sees. @@ -671,7 +656,7 @@ fn assert_proxied_egress(output: &str, proxy_logs: &str, ports: &FixturePorts) { // staged into the overlay reached the supervisor. assert!( proxy_logs.contains(&format!( - "CONNECT {GVPROXY_HOST_LOOPBACK_IP}:{} auth=ok", + "CONNECT {HOST_LOOPBACK_IP}:{} auth=ok", ports.allowed )), "proxy should have seen an authenticated validated-IP CONNECT to the approved upstream:\n{proxy_logs}" @@ -713,7 +698,7 @@ async fn vm_corporate_proxy_routes_approved_tls_egress() { let ports = FixturePorts::pick(); - // ── Host fixtures, reached from the guest through the gvproxy alias ── + // ── Host fixtures, reached by the host supervisor ── let proxy = HostPythonFixture::start(&proxy_script(ports.proxy), ports.proxy) .await .expect("start fake corporate proxy"); @@ -832,10 +817,7 @@ async fn vm_corporate_proxy_trusts_ca_bundle_for_https_proxy() { sandbox.create_output ); assert!( - proxy_logs.contains(&format!( - "CONNECT {GVPROXY_HOST_LOOPBACK_IP}:{} ok", - ports.allowed - )), + proxy_logs.contains(&format!("CONNECT {HOST_LOOPBACK_IP}:{} ok", ports.allowed)), "https proxy should have seen a validated-IP CONNECT to the approved upstream:\n{proxy_logs}" ); assert!( diff --git a/e2e/rust/tests/websocket_conformance.rs b/e2e/rust/tests/websocket_conformance.rs index 4ba4dbd046..d95841d07f 100644 --- a/e2e/rust/tests/websocket_conformance.rs +++ b/e2e/rust/tests/websocket_conformance.rs @@ -364,7 +364,6 @@ import os import socket import struct import time -import urllib.parse HOST = {host:?} PORT = {port} @@ -413,34 +412,15 @@ def read_frame(sock): payload = bytes(byte ^ mask[index % 4] for index, byte in enumerate(payload)) return first, payload -def proxy_parts(): - names = ("HTTP_PROXY", "http_proxy", "HTTPS_PROXY", "https_proxy", "ALL_PROXY", "all_proxy") - proxy_url = next((os.environ.get(name) for name in names if os.environ.get(name)), None) - if not proxy_url: - raise RuntimeError("proxy environment is not configured") - parsed = urllib.parse.urlparse(proxy_url) - if not parsed.hostname: - raise RuntimeError(f"invalid proxy URL: {{proxy_url!r}}") - return parsed.hostname, parsed.port or 80 - -def proxy_socket_with_retry(host, port, mode, timeout_seconds=20): - proxy_host, proxy_port = proxy_parts() - target = f"{{host}}:{{port}}" +def transparent_socket_with_retry(host, port, timeout_seconds=20): deadline = time.monotonic() + timeout_seconds last_error = None while time.monotonic() < deadline: sock = None try: - sock = socket.create_connection((proxy_host, proxy_port), timeout=5) - if mode == "connect": - request = f"CONNECT {{target}} HTTP/1.1\r\nHost: {{target}}\r\n\r\n" - sock.sendall(request.encode("ascii")) - response = recv_until(sock, b"\r\n\r\n").decode("iso-8859-1", "replace") - if not (response.startswith("HTTP/1.1 200") or response.startswith("HTTP/1.0 200")): - first_line = response.splitlines()[0] if response else "" - raise RuntimeError(f"proxy CONNECT failed: {{first_line}}") + sock = socket.create_connection((host, port), timeout=5) return sock - except (OSError, RuntimeError) as error: + except OSError as error: if sock is not None: sock.close() last_error = error @@ -449,27 +429,25 @@ def proxy_socket_with_retry(host, port, mode, timeout_seconds=20): token = os.environ[TOKEN_ENV] payload = json.dumps({{"authorization": "Bearer " + token}}, sort_keys=True) -results = {{}} -for mode in ("connect", "forward"): - key = base64.b64encode(os.urandom(16)).decode("ascii") - with proxy_socket_with_retry(HOST, PORT, mode) as sock: - request_target = "/ws" if mode == "connect" else f"http://{{HOST}}:{{PORT}}/ws" - request = ( - f"GET {{request_target}} HTTP/1.1\r\n" - f"Host: {{HOST}}:{{PORT}}\r\n" - "Upgrade: websocket\r\n" - "Connection: Upgrade\r\n" - f"Sec-WebSocket-Key: {{key}}\r\n" - "Sec-WebSocket-Version: 13\r\n" - "\r\n" - ) - sock.sendall(request.encode("ascii")) - response = recv_until(sock, b"\r\n\r\n").decode("iso-8859-1", "replace") - if not response.startswith("HTTP/1.1 101"): - raise RuntimeError(f"{{mode}} websocket upgrade failed: {{response!r}}") - sock.sendall(masked_text_frame(payload)) - _, response_payload = read_frame(sock) - results[mode] = json.loads(response_payload.decode("utf-8")) +key = base64.b64encode(os.urandom(16)).decode("ascii") +with transparent_socket_with_retry(HOST, PORT) as sock: + request = ( + "GET /ws HTTP/1.1\r\n" + f"Host: {{HOST}}:{{PORT}}\r\n" + "Upgrade: websocket\r\n" + "Connection: Upgrade\r\n" + f"Sec-WebSocket-Key: {{key}}\r\n" + "Sec-WebSocket-Version: 13\r\n" + "\r\n" + ) + sock.sendall(request.encode("ascii")) + response = recv_until(sock, b"\r\n\r\n").decode("iso-8859-1", "replace") + if not response.startswith("HTTP/1.1 101"): + raise RuntimeError(f"websocket upgrade failed: {{response!r}}") + sock.sendall(masked_text_frame(payload)) + _, response_payload = read_frame(sock) + result = json.loads(response_payload.decode("utf-8")) +results = {{"transparent": result}} print(json.dumps(results, sort_keys=True)) "#, host = host, @@ -479,7 +457,7 @@ print(json.dumps(results, sort_keys=True)) } #[tokio::test] -async fn websocket_text_placeholder_is_rewritten_through_both_adapters() { +async fn websocket_text_placeholder_is_rewritten_transparently() { let _provider_lock = PROVIDER_LOCK .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -522,14 +500,7 @@ async fn websocket_text_placeholder_is_rewritten_through_both_adapters() { assert!( guard .create_output - .contains(r#""connect": {"saw_placeholder": false, "saw_secret": true}"#), - "expected CONNECT upstream to see only the resolved secret marker:\n{}", - guard.create_output - ); - assert!( - guard - .create_output - .contains(r#""forward": {"saw_placeholder": false, "saw_secret": true}"#), + .contains(r#""transparent": {"saw_placeholder": false, "saw_secret": true}"#), "expected upstream to see only the resolved secret marker:\n{}", guard.create_output ); diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index 0f64247731..c700801f6f 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -337,6 +337,31 @@ resolve_docker_supervisor_image() { printf '%s\n' "openshell/supervisor:dev" } +resolve_docker_sandbox_runtime_image() { + if [ -n "${OPENSHELL_DOCKER_SANDBOX_RUNTIME_IMAGE:-}" ]; then + printf '%s\n' "${OPENSHELL_DOCKER_SANDBOX_RUNTIME_IMAGE}" + return 0 + fi + + if [ -n "${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-}" ]; then + printf '%s\n' "${OPENSHELL_SANDBOX_RUNTIME_IMAGE}" + return 0 + fi + + if [ -n "${CI:-}" ]; then + if [ -z "${IMAGE_TAG:-}" ]; then + echo "ERROR: IMAGE_TAG must be set in CI when no Docker sandbox runtime image override is provided." >&2 + exit 2 + fi + + local registry="${OPENSHELL_REGISTRY:-ghcr.io/nvidia/openshell}" + printf '%s/sandbox:%s\n' "${registry%/}" "${IMAGE_TAG}" + return 0 + fi + + printf '%s\n' "openshell/sandbox:dev" +} + docker_pull_with_retry() { local image=$1 local attempts=4 @@ -384,6 +409,27 @@ build_local_docker_supervisor_image_if_required() { exit 2 } +build_local_docker_sandbox_runtime_image_if_required() { + local image=$1 + + if [ "${image}" != "openshell/sandbox:dev" ]; then + return 0 + fi + + local daemon_arch + daemon_arch="$(ce_info_arch)" + + echo "Building local Docker sandbox runtime image ${image} for linux/${daemon_arch}..." + CONTAINER_ENGINE=docker DOCKER_PLATFORM="linux/${daemon_arch}" IMAGE_TAG=dev \ + bash "${ROOT}/tasks/scripts/docker-build-image.sh" sandbox + if docker image inspect "${image}" >/dev/null 2>&1; then + return 0 + fi + + echo "ERROR: expected sandbox runtime image '${image}' after local build." >&2 + exit 2 +} + ensure_docker_supervisor_image() { local image=$1 @@ -401,6 +447,23 @@ ensure_docker_supervisor_image() { exit 2 } +ensure_docker_sandbox_runtime_image() { + local image=$1 + + if docker image inspect "${image}" >/dev/null 2>&1; then + return 0 + fi + + echo "Pulling Docker sandbox runtime image ${image}..." + if docker_pull_with_retry "${image}"; then + return 0 + fi + + echo "ERROR: sandbox runtime image '${image}' is not available." >&2 + echo " Build it, push it, or set OPENSHELL_SANDBOX_RUNTIME_IMAGE to a pullable image." >&2 + exit 2 +} + image_uses_latest_tag() { local image=$1 local last_component @@ -448,6 +511,11 @@ build_local_docker_supervisor_image_if_required "${SUPERVISOR_IMAGE}" ensure_docker_supervisor_image "${SUPERVISOR_IMAGE}" echo "Using Docker supervisor image: ${SUPERVISOR_IMAGE}" +SANDBOX_RUNTIME_IMAGE="$(resolve_docker_sandbox_runtime_image)" +build_local_docker_sandbox_runtime_image_if_required "${SANDBOX_RUNTIME_IMAGE}" +ensure_docker_sandbox_runtime_image "${SANDBOX_RUNTIME_IMAGE}" +echo "Using Docker sandbox runtime image: ${SANDBOX_RUNTIME_IMAGE}" + DEFAULT_SANDBOX_IMAGE="ghcr.io/nvidia/openshell-community/sandboxes/base:latest" SANDBOX_IMAGE="${OPENSHELL_E2E_DOCKER_SANDBOX_IMAGE:-${OPENSHELL_SANDBOX_IMAGE:-${DEFAULT_SANDBOX_IMAGE}}}" SANDBOX_IMAGE_PULL_POLICY="${OPENSHELL_E2E_DOCKER_SANDBOX_IMAGE_PULL_POLICY:-${OPENSHELL_SANDBOX_IMAGE_PULL_POLICY:-if_not_present}}" @@ -523,6 +591,7 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")" printf 'enable_bind_mounts = true\n' + printf 'sandbox_runtime_image = %s\n' "$(toml_string "${SANDBOX_RUNTIME_IMAGE}")" printf 'supervisor_image = %s\n' "$(toml_string "${SUPERVISOR_IMAGE}")" if [ -n "${GATEWAY_HOST_ALIAS_IP}" ]; then printf 'host_gateway_ip = %s\n' "$(toml_string "${GATEWAY_HOST_ALIAS_IP}")" @@ -541,6 +610,7 @@ if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then printf 'guest_tls_cert = %s\n' "$(toml_string "${PKI_DIR}/client/tls.crt")" printf 'guest_tls_key = %s\n' "$(toml_string "${PKI_DIR}/client/tls.key")" printf 'enable_bind_mounts = true\n' + printf 'sandbox_runtime_image = %s\n' "$(toml_string "${SANDBOX_RUNTIME_IMAGE}")" printf 'supervisor_image = %s\n' "$(toml_string "${SUPERVISOR_IMAGE}")" if [ -n "${GATEWAY_HOST_ALIAS_IP}" ]; then printf 'host_gateway_ip = %s\n' "$(toml_string "${GATEWAY_HOST_ALIAS_IP}")" diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh index 069acc0f01..e9c5db7a15 100755 --- a/e2e/with-kube-gateway.sh +++ b/e2e/with-kube-gateway.sh @@ -33,10 +33,12 @@ # configuration on top of ci/values-skaffold.yaml. # # Image source: -# - Ephemeral k3d mode builds local `openshell/{gateway,supervisor}:${IMAGE_TAG}` +# - Ephemeral k3d mode builds local +# `openshell/{gateway,sandbox,supervisor}:${IMAGE_TAG}` # images by default, imports them into k3d, then installs the chart. This # mirrors the Skaffold local-dev path. -# - Existing-context mode pulls from ${OPENSHELL_REGISTRY}/{gateway,supervisor}:${IMAGE_TAG} +# - Existing-context mode pulls from +# ${OPENSHELL_REGISTRY}/{gateway,sandbox,supervisor}:${IMAGE_TAG} # (defaults: ghcr.io/nvidia/openshell, latest). CI sets IMAGE_TAG to the # commit SHA and preloads or publishes the images before running this script. # @@ -315,9 +317,34 @@ cleanup() { && kctl get namespace "${NAMESPACE}" >/dev/null 2>&1; then echo "=== gateway pod state (preserved for debugging) ===" kctl -n "${NAMESPACE}" get pods -o wide 2>&1 || true + echo "=== Agent Sandbox resources ===" + kctl -n "${NAMESPACE}" get sandboxes.agents.x-k8s.io -o yaml 2>&1 || true + echo "=== gateway sandbox records ===" + "${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}" \ + sandbox list --all-workspaces --output json 2>&1 || true + echo "=== sandbox-runtime supervisor Pods ===" + kctl -n "${NAMESPACE}" get pods \ + -l "openshell.ai/boundary-role=supervisor" -o yaml 2>&1 || true + echo "=== sandbox-runtime supervisor logs (last 200 lines each) ===" + while IFS= read -r supervisor_pod; do + [ -n "${supervisor_pod}" ] || continue + echo "--- ${supervisor_pod} ---" + kctl -n "${NAMESPACE}" logs "${supervisor_pod}" \ + --all-containers --prefix --tail=200 2>&1 || true + echo "--- ${supervisor_pod} (previous containers) ---" + kctl -n "${NAMESPACE}" logs "${supervisor_pod}" --previous \ + --all-containers --prefix --tail=200 2>&1 || true + done < <(kctl -n "${NAMESPACE}" get pods \ + -l "openshell.ai/boundary-role=supervisor" -o name 2>/dev/null || true) echo "=== gateway events ===" kctl -n "${NAMESPACE}" get events --sort-by=.lastTimestamp 2>&1 \ | tail -n 80 || true + echo "=== gateway lifecycle and supervisor-session logs ===" + kctl -n "${NAMESPACE}" logs "$(kube_workload_ref "${RELEASE_NAME}")" \ + --since=20m \ + --all-containers --prefix 2>&1 \ + | grep -Ei "sandbox phase changed|start_sandbox|stop_sandbox|supervisor session|sandbox-runtime|bootstrap" \ + || true echo "=== gateway logs (last 200 lines) ===" kctl -n "${NAMESPACE}" logs \ -l "app.kubernetes.io/instance=${RELEASE_NAME}" --tail=200 \ @@ -474,6 +501,8 @@ run_scenario() { --set "fullnameOverride=openshell" \ --set "image.repository=${REGISTRY_VALUE}/gateway" \ --set "image.tag=${IMAGE_TAG_VALUE}" \ + --set "sandboxRuntime.image.repository=${REGISTRY_VALUE}/sandbox" \ + --set "sandboxRuntime.image.tag=${IMAGE_TAG_VALUE}" \ --set "supervisor.image.repository=${REGISTRY_VALUE}/supervisor" \ --set "supervisor.image.tag=${IMAGE_TAG_VALUE}" \ "${helm_post_renderer_args[@]}" \ @@ -742,6 +771,7 @@ if [ -z "${OPENSHELL_E2E_KUBE_BUILD_IMAGES+x}" ]; then fi fi +reuse_sandbox_image=0 reuse_supervisor_image=0 if [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" = "1" ]; then REGISTRY_VALUE="${OPENSHELL_REGISTRY:-openshell}" @@ -827,7 +857,7 @@ if [ -z "${HOST_GATEWAY_IP}" ]; then echo " Set OPENSHELL_E2E_HOST_GATEWAY_IP to override." >&2 fi -# Import locally-available gateway/supervisor images into the k3d cluster so +# Import locally available gateway, sandbox, and supervisor images into the k3d cluster so # devs working off local builds don't depend on the configured registry. For # kind clusters (used by CI), images must be loaded before this script runs — # the workflow handles that via `kind load docker-image`. Best-effort: when an @@ -843,7 +873,7 @@ elif [[ "${KUBE_CONTEXT}" == k3d-* ]] && command -v k3d >/dev/null 2>&1; then fi if [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" = "1" ]; then require_cmd docker - echo "Building local Kubernetes e2e images (${REGISTRY_VALUE}/{gateway,supervisor}:${IMAGE_TAG_VALUE})..." + echo "Building local Kubernetes e2e images (${REGISTRY_VALUE}/{gateway,sandbox,supervisor}:${IMAGE_TAG_VALUE})..." if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then if [ "$(uname -s)" != "Linux" ]; then echo "ERROR: external Kubernetes driver image composition currently requires a Linux build host." >&2 @@ -873,6 +903,7 @@ if [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" = "1" ]; then docker build \ --build-arg "TARGETARCH=${external_arch}" \ --build-arg "SUPERVISOR_IMAGE=${REGISTRY_VALUE}/supervisor:${IMAGE_TAG_VALUE}" \ + --build-arg "SANDBOX_RUNTIME_IMAGE=${REGISTRY_VALUE}/sandbox:${IMAGE_TAG_VALUE}" \ --tag "${REGISTRY_VALUE}/gateway:${IMAGE_TAG_VALUE}" \ --file "${ROOT}/e2e/docker/Dockerfile.external-kubernetes-gateway" \ "${ROOT}" @@ -880,7 +911,16 @@ if [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" = "1" ]; then CONTAINER_ENGINE=docker IMAGE_REGISTRY="${REGISTRY_VALUE}" IMAGE_TAG="${IMAGE_TAG_VALUE}" \ bash "${ROOT}/tasks/scripts/docker-build-image.sh" gateway fi + sandbox_image="${REGISTRY_VALUE}/sandbox:${IMAGE_TAG_VALUE}" supervisor_image="${REGISTRY_VALUE}/supervisor:${IMAGE_TAG_VALUE}" + if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" != "1" ] \ + || ! docker image inspect "${sandbox_image}" >/dev/null 2>&1; then + CONTAINER_ENGINE=docker IMAGE_REGISTRY="${REGISTRY_VALUE}" IMAGE_TAG="${IMAGE_TAG_VALUE}" \ + bash "${ROOT}/tasks/scripts/docker-build-image.sh" sandbox + else + reuse_sandbox_image=1 + echo "Reusing existing sandbox image ${sandbox_image}" + fi if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" != "1" ] \ || ! docker image inspect "${supervisor_image}" >/dev/null 2>&1; then CONTAINER_ENGINE=docker IMAGE_REGISTRY="${REGISTRY_VALUE}" IMAGE_TAG="${IMAGE_TAG_VALUE}" \ @@ -894,6 +934,7 @@ fi if [ -n "${import_cluster_name}" ]; then for image in \ "${REGISTRY_VALUE}/gateway:${IMAGE_TAG_VALUE}" \ + "${REGISTRY_VALUE}/sandbox:${IMAGE_TAG_VALUE}" \ "${REGISTRY_VALUE}/supervisor:${IMAGE_TAG_VALUE}"; do if docker image inspect "${image}" >/dev/null 2>&1; then echo "Importing ${image} into k3d cluster ${import_cluster_name}..." @@ -906,6 +947,11 @@ elif [ "${OPENSHELL_E2E_KUBE_BUILD_IMAGES}" = "1" ] \ && command -v kind >/dev/null 2>&1; then kind_cluster_name="${KUBE_CONTEXT#kind-}" kind_images=("${REGISTRY_VALUE}/gateway:${IMAGE_TAG_VALUE}") + # The CI workflow loads its published sandbox archive before invoking this + # wrapper. Only load a sandbox image here when this script rebuilt it. + if [ "${reuse_sandbox_image}" != "1" ]; then + kind_images+=("${REGISTRY_VALUE}/sandbox:${IMAGE_TAG_VALUE}") + fi # The CI workflow loads its published supervisor archive before invoking this # wrapper. Only load a supervisor image here when this script rebuilt it. if [ "${reuse_supervisor_image}" != "1" ]; then @@ -1099,6 +1145,8 @@ else --set "fullnameOverride=openshell" \ --set "image.repository=${REGISTRY_VALUE}/gateway" \ --set "image.tag=${IMAGE_TAG_VALUE}" \ + --set "sandboxRuntime.image.repository=${REGISTRY_VALUE}/sandbox" \ + --set "sandboxRuntime.image.tag=${IMAGE_TAG_VALUE}" \ --set "supervisor.image.repository=${REGISTRY_VALUE}/supervisor" \ --set "supervisor.image.tag=${IMAGE_TAG_VALUE}" \ "${helm_extra_args[@]}" \ diff --git a/e2e/with-podman-gateway.sh b/e2e/with-podman-gateway.sh index e7c47a251f..de34d6c10c 100755 --- a/e2e/with-podman-gateway.sh +++ b/e2e/with-podman-gateway.sh @@ -128,6 +128,8 @@ DRIVER_PID="" DRIVER_LOG="${OPENSHELL_PARITY_EXTERNAL_DRIVER_LOG_CAPTURE:-${WORKDIR}/podman-driver.log}" mkdir -p "$(dirname "${DRIVER_LOG}")" DRIVER_SOCKET="${WORKDIR}/compute-driver.sock" +DRIVER_DATA_HOME="${WORKDIR}/driver-data" +mkdir -p "${DRIVER_DATA_HOME}" E2E_NAMESPACE="" PODMAN_NETWORK_NAME="" PODMAN_NETWORK_MANAGED=0 @@ -182,9 +184,24 @@ cleanup() { for id in ${sandbox_ids}; do local sandbox_id sandbox_id="$(podman_cmd inspect --format '{{ index .Config.Labels "openshell.ai/sandbox-id" }}' "${id}" 2>/dev/null || true)" - podman_cmd rm -f "${id}" >/dev/null 2>&1 || true if [ -n "${sandbox_id}" ] && [ "${sandbox_id}" != "" ]; then + # Only the companion is attached to the test network. Remove it first + # (it depends on the workload user namespace), then locate the isolated + # network=none workload by this test sandbox's immutable label. + podman_cmd rm -f "openshell-supervisor-${sandbox_id}" >/dev/null 2>&1 || true + local workload_ids workload_id + workload_ids="$(podman_cmd ps -aq --filter "label=openshell.managed=true" \ + --filter "label=openshell.ai/sandbox-id=${sandbox_id}" \ + --filter "label=openshell.io/isolation-role=sandbox" 2>/dev/null || true)" + for workload_id in ${workload_ids}; do + podman_cmd rm -f "${workload_id}" >/dev/null 2>&1 || true + done + podman_cmd volume rm "openshell-channel-${sandbox_id}" >/dev/null 2>&1 || true podman_cmd volume rm -f "openshell-sandbox-${sandbox_id}-workspace" >/dev/null 2>&1 || true + local secret_prefix + for secret_prefix in openshell-token openshell-proxy-auth openshell-tls-ca openshell-tls-cert openshell-tls-key; do + podman_cmd secret rm "${secret_prefix}-${sandbox_id}" >/dev/null 2>&1 || true + done fi done fi @@ -339,6 +356,26 @@ resolve_podman_supervisor_image() { printf '%s\n' "openshell/supervisor:dev" } +resolve_podman_sandbox_runtime_image() { + if [ -n "${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-}" ]; then + printf '%s\n' "${OPENSHELL_SANDBOX_RUNTIME_IMAGE}" + return 0 + fi + + if [ -n "${CI:-}" ]; then + if [ -z "${IMAGE_TAG:-}" ]; then + echo "ERROR: IMAGE_TAG must be set in CI when no Podman sandbox runtime image override is provided." >&2 + exit 2 + fi + + local registry="${OPENSHELL_REGISTRY:-ghcr.io/nvidia/openshell}" + printf '%s/sandbox:%s\n' "${registry%/}" "${IMAGE_TAG}" + return 0 + fi + + printf '%s\n' "openshell/sandbox:dev" +} + ensure_podman_supervisor_image() { local image=$1 @@ -433,6 +470,37 @@ ensure_podman_supervisor_image() { exit 2 } +ensure_podman_sandbox_runtime_image() { + local image=$1 + + if [ "${image}" = "openshell/sandbox:dev" ] \ + && [ -z "${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-}" ] \ + && [ -z "${CI:-}" ]; then + echo "Building local Podman sandbox runtime image ${image}..." + with_podman_config env CONTAINER_ENGINE=podman IMAGE_TAG=dev \ + bash "${ROOT}/tasks/scripts/docker-build-image.sh" sandbox + if podman_cmd image exists "${image}" 2>/dev/null; then + return 0 + fi + + echo "ERROR: expected sandbox runtime image '${image}' after local build." >&2 + exit 2 + fi + + if podman_cmd image exists "${image}" 2>/dev/null; then + return 0 + fi + + echo "Pulling Podman sandbox runtime image ${image}..." + if podman_cmd pull "${image}"; then + return 0 + fi + + echo "ERROR: sandbox runtime image '${image}' is not available." >&2 + echo " Build it, push it, or set OPENSHELL_SANDBOX_RUNTIME_IMAGE to a pullable image." >&2 + exit 2 +} + if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ]; then case "${OPENSHELL_GATEWAY_ENDPOINT}" in http://*) ;; @@ -512,6 +580,10 @@ if ! [[ "${SUPERVISOR_RUNTIME_IMAGE}" =~ ^[^@]+@sha256:[0-9a-f]{64}$ ]]; then exit 2 fi SUPERVISOR_BASE_IMAGE="$(awk '$1 == "FROM" { print $2; exit }' "${OPENSHELL_E2E_SUPERVISOR_DOCKERFILE:-${ROOT}/deploy/docker/Dockerfile.supervisor}")" +if ! podman_cmd image exists "${SUPERVISOR_BASE_IMAGE}" 2>/dev/null; then + echo "Pulling Podman supervisor base image ${SUPERVISOR_BASE_IMAGE}..." + podman_cmd pull "${SUPERVISOR_BASE_IMAGE}" +fi SUPERVISOR_BASE_IMAGE_ID="$(podman_cmd image inspect --format '{{.Id}}' "${SUPERVISOR_BASE_IMAGE}")" SUPERVISOR_BASE_IMAGE_ID="${SUPERVISOR_BASE_IMAGE_ID#sha256:}" SUPERVISOR_BASE_IMAGE_DIGEST="$(podman_cmd image inspect --format '{{.Digest}}' "${SUPERVISOR_BASE_IMAGE}")" @@ -522,11 +594,16 @@ if ! [[ "${SUPERVISOR_BASE_IMAGE_ID}" =~ ^[0-9a-f]{64}$ ]] \ fi SUPERVISOR_PACKAGE_MANIFEST="${OPENSHELL_PARITY_SUPERVISOR_PACKAGE_CAPTURE:-${WORKDIR}/supervisor.packages.txt}" mkdir -p "$(dirname "${SUPERVISOR_PACKAGE_MANIFEST}")" -podman_cmd run --rm --network none --entrypoint /sbin/apk \ - "${SUPERVISOR_RUNTIME_IMAGE}" info -v | LC_ALL=C sort >"${SUPERVISOR_PACKAGE_MANIFEST}" +podman_cmd run --rm --network none --entrypoint /usr/bin/dpkg-query \ + "${SUPERVISOR_RUNTIME_IMAGE}" -W '-f=${binary:Package}=${Version}\n' \ + | LC_ALL=C sort >"${SUPERVISOR_PACKAGE_MANIFEST}" SUPERVISOR_PACKAGE_MANIFEST_SHA256="$(sha256sum "${SUPERVISOR_PACKAGE_MANIFEST}" | cut -d' ' -f1)" echo "Using Podman supervisor image: ${SUPERVISOR_RUNTIME_IMAGE} (ID ${SUPERVISOR_IMAGE_ID}, digest ${SUPERVISOR_IMAGE_DIGEST}, base ${SUPERVISOR_BASE_IMAGE} ID ${SUPERVISOR_BASE_IMAGE_ID} digest ${SUPERVISOR_BASE_IMAGE_DIGEST}, packages ${SUPERVISOR_PACKAGE_MANIFEST_SHA256})" +SANDBOX_BOUNDARY_IMAGE="$(resolve_podman_sandbox_runtime_image)" +ensure_podman_sandbox_runtime_image "${SANDBOX_BOUNDARY_IMAGE}" +echo "Using Podman sandbox runtime image: ${SANDBOX_BOUNDARY_IMAGE}" + DEFAULT_SANDBOX_IMAGE="ghcr.io/nvidia/openshell-community/sandboxes/base:latest" SANDBOX_IMAGE_REQUEST="${OPENSHELL_E2E_PODMAN_SANDBOX_IMAGE:-${OPENSHELL_SANDBOX_IMAGE:-${DEFAULT_SANDBOX_IMAGE}}}" if [ "${OPENSHELL_E2E_REQUIRE_DIGEST_PINNED_SANDBOX_IMAGE:-0}" = "1" ] \ @@ -651,16 +728,18 @@ if [ -n "${OPENSHELL_PARITY_LAUNCH_MANIFEST_CAPTURE:-}" ]; then driver_tls_ca_sha256="$(sha256sum "${EXTERNAL_DRIVER_TLS_CA}" | cut -d' ' -f1)" driver_tls_cert_sha256="$(sha256sum "${EXTERNAL_DRIVER_TLS_CERT}" | cut -d' ' -f1)" driver_tls_key_sha256="$(sha256sum "${EXTERNAL_DRIVER_TLS_KEY}" | cut -d' ' -f1)" - external_driver_environment="$(printf '{\"OPENSHELL_COMPUTE_DRIVER_SOCKET\":\"%s\",\"OPENSHELL_PODMAN_SOCKET\":\"%s\",\"OPENSHELL_SANDBOX_IMAGE\":\"%s\",\"OPENSHELL_SANDBOX_IMAGE_PULL_POLICY\":\"%s\",\"OPENSHELL_HEALTH_CHECK_INTERVAL_SECS\":%s,\"OPENSHELL_GRPC_ENDPOINT\":\"%s\",\"OPENSHELL_GATEWAY_PORT\":%s,\"OPENSHELL_NETWORK_NAME\":\"%s\",\"OPENSHELL_STOP_TIMEOUT\":%s,\"OPENSHELL_SUPERVISOR_IMAGE\":\"%s\",\"OPENSHELL_PODMAN_TLS_CA\":{\"path\":\"%s\",\"sha256\":\"%s\"},\"OPENSHELL_PODMAN_TLS_CERT\":{\"path\":\"%s\",\"sha256\":\"%s\"},\"OPENSHELL_PODMAN_TLS_KEY\":{\"path\":\"%s\",\"sha256\":\"%s\"},\"OPENSHELL_ENABLE_BIND_MOUNTS\":%s}' \ + external_driver_environment="$(printf '{\"XDG_DATA_HOME\":\"%s\",\"OPENSHELL_COMPUTE_DRIVER_SOCKET\":\"%s\",\"OPENSHELL_PODMAN_SOCKET\":\"%s\",\"OPENSHELL_SANDBOX_IMAGE\":\"%s\",\"OPENSHELL_SANDBOX_IMAGE_PULL_POLICY\":\"%s\",\"OPENSHELL_HEALTH_CHECK_INTERVAL_SECS\":%s,\"OPENSHELL_GRPC_ENDPOINT\":\"%s\",\"OPENSHELL_GATEWAY_PORT\":%s,\"OPENSHELL_NETWORK_NAME\":\"%s\",\"OPENSHELL_STOP_TIMEOUT\":%s,\"OPENSHELL_SANDBOX_RUNTIME_IMAGE\":\"%s\",\"OPENSHELL_SUPERVISOR_IMAGE\":\"%s\",\"OPENSHELL_PODMAN_TLS_CA\":{\"path\":\"%s\",\"sha256\":\"%s\"},\"OPENSHELL_PODMAN_TLS_CERT\":{\"path\":\"%s\",\"sha256\":\"%s\"},\"OPENSHELL_PODMAN_TLS_KEY\":{\"path\":\"%s\",\"sha256\":\"%s\"},\"OPENSHELL_ENABLE_BIND_MOUNTS\":%s}' \ + "${DRIVER_DATA_HOME}" \ "${DRIVER_SOCKET}" \ "${OPENSHELL_PODMAN_SOCKET:-}" \ - "${SANDBOX_RUNTIME_IMAGE}" \ + "${SANDBOX_IMAGE_REQUEST}" \ "${EXTERNAL_DRIVER_PULL_POLICY}" \ "${EXTERNAL_DRIVER_HEALTH_CHECK_INTERVAL_SECS}" \ "${EXTERNAL_DRIVER_CALLBACK_ENDPOINT}" \ "${HOST_PORT}" \ "${PODMAN_NETWORK_NAME}" \ "${PODMAN_STOP_TIMEOUT_SECS}" \ + "${SANDBOX_BOUNDARY_IMAGE}" \ "${SUPERVISOR_RUNTIME_IMAGE}" \ "${EXTERNAL_DRIVER_TLS_CA}" \ "${driver_tls_ca_sha256}" \ @@ -670,7 +749,7 @@ if [ -n "${OPENSHELL_PARITY_LAUNCH_MANIFEST_CAPTURE:-}" ]; then "${driver_tls_key_sha256}" \ "${EXTERNAL_DRIVER_ENABLE_BIND_MOUNTS}")" fi - printf '{"schema_version":%s,"gateway_port":%s,"external_compute_driver":%s,"compute_driver_transport":"%s","external_driver_pull_policy":"%s","supervisor_image":"%s","supervisor_image_id":"%s","supervisor_image_digest":"%s","supervisor_runtime_image":"%s","supervisor_base_image":"%s","supervisor_base_image_id":"%s","supervisor_base_image_digest":"%s","supervisor_base_runtime_image":"%s","supervisor_package_manifest_sha256":"%s","sandbox_image_request":"%s","sandbox_image_id":"%s","sandbox_image_digest":"%s","sandbox_runtime_image":"%s","sandbox_client_image_alias":"%s","sandbox_client_image_alias_id":"%s","gateway_sha256_before_execution":"%s","cli_sha256_before_execution":"%s","conformance_sha256_before_execution":"%s","external_driver_sha256_before_execution":"%s","supervisor_sha256_before_execution":"%s","supervisor_dockerfile_sha256_before_execution":"%s","cli_trace_wrapper_sha256_before_execution":"%s","external_driver_grpc_endpoint":%s,"external_driver_host_gateway_ip":%s,"external_driver_userns":%s,"external_driver_spiffe":%s,"external_driver_proxy":%s,"external_driver_app_armor":%s,"external_driver_environment":%s}\n' \ + printf '{"schema_version":%s,"gateway_port":%s,"external_compute_driver":%s,"compute_driver_transport":"%s","external_driver_pull_policy":"%s","supervisor_image":"%s","supervisor_image_id":"%s","supervisor_image_digest":"%s","supervisor_runtime_image":"%s","supervisor_base_image":"%s","supervisor_base_image_id":"%s","supervisor_base_image_digest":"%s","supervisor_base_runtime_image":"%s","supervisor_package_manifest_sha256":"%s","sandbox_image_request":"%s","sandbox_image_id":"%s","sandbox_image_digest":"%s","sandbox_runtime_image":"%s","sandbox_boundary_image":"%s","sandbox_client_image_alias":"%s","sandbox_client_image_alias_id":"%s","gateway_sha256_before_execution":"%s","cli_sha256_before_execution":"%s","conformance_sha256_before_execution":"%s","external_driver_sha256_before_execution":"%s","supervisor_sha256_before_execution":"%s","supervisor_dockerfile_sha256_before_execution":"%s","cli_trace_wrapper_sha256_before_execution":"%s","external_driver_grpc_endpoint":%s,"external_driver_host_gateway_ip":%s,"external_driver_userns":%s,"external_driver_spiffe":%s,"external_driver_proxy":%s,"external_driver_app_armor":%s,"external_driver_environment":%s}\n' \ "${CONFIG_SCHEMA_VERSION}" \ "${HOST_PORT}" \ "$([ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ] && printf true || printf false)" \ @@ -689,6 +768,7 @@ if [ -n "${OPENSHELL_PARITY_LAUNCH_MANIFEST_CAPTURE:-}" ]; then "${SANDBOX_IMAGE_ID}" \ "${SANDBOX_IMAGE_DIGEST}" \ "${SANDBOX_RUNTIME_IMAGE}" \ + "${SANDBOX_BOUNDARY_IMAGE}" \ "${SANDBOX_CLIENT_IMAGE_ALIAS}" \ "${SANDBOX_CLIENT_IMAGE_ALIAS_ID}" \ "${OPENSHELL_E2E_EXPECTED_GATEWAY_SHA256:-}" \ @@ -712,15 +792,17 @@ if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then require_expected_sha256 "external compute driver" "${DRIVER_BIN}" \ "${OPENSHELL_E2E_EXPECTED_EXTERNAL_DRIVER_SHA256:-}" env -i \ + XDG_DATA_HOME="${DRIVER_DATA_HOME}" \ OPENSHELL_COMPUTE_DRIVER_SOCKET="${DRIVER_SOCKET}" \ OPENSHELL_PODMAN_SOCKET="${OPENSHELL_PODMAN_SOCKET:-}" \ - OPENSHELL_SANDBOX_IMAGE="${SANDBOX_RUNTIME_IMAGE}" \ + OPENSHELL_SANDBOX_IMAGE="${SANDBOX_IMAGE_REQUEST}" \ OPENSHELL_SANDBOX_IMAGE_PULL_POLICY="${EXTERNAL_DRIVER_PULL_POLICY}" \ OPENSHELL_HEALTH_CHECK_INTERVAL_SECS="${EXTERNAL_DRIVER_HEALTH_CHECK_INTERVAL_SECS}" \ OPENSHELL_GRPC_ENDPOINT="${EXTERNAL_DRIVER_CALLBACK_ENDPOINT}" \ OPENSHELL_GATEWAY_PORT="${HOST_PORT}" \ OPENSHELL_NETWORK_NAME="${PODMAN_NETWORK_NAME}" \ OPENSHELL_STOP_TIMEOUT="${PODMAN_STOP_TIMEOUT_SECS}" \ + OPENSHELL_SANDBOX_RUNTIME_IMAGE="${SANDBOX_BOUNDARY_IMAGE}" \ OPENSHELL_SUPERVISOR_IMAGE="${SUPERVISOR_RUNTIME_IMAGE}" \ OPENSHELL_PODMAN_TLS_CA="${EXTERNAL_DRIVER_TLS_CA}" \ OPENSHELL_PODMAN_TLS_CERT="${EXTERNAL_DRIVER_TLS_CERT}" \ diff --git a/examples/gateway-deploy-connect.md b/examples/gateway-deploy-connect.md index 37ed37bf2e..2861444d1c 100644 --- a/examples/gateway-deploy-connect.md +++ b/examples/gateway-deploy-connect.md @@ -15,6 +15,7 @@ Install the gateway into a Kubernetes cluster you manage: kubectl create namespace openshell helm upgrade --install openshell deploy/helm/openshell \ --namespace openshell \ + --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set server.disableTls=true \ --set service.type=ClusterIP ``` diff --git a/examples/governance-interceptor/Cargo.lock b/examples/governance-interceptor/Cargo.lock index b0df0429d4..1a0410f8ae 100644 --- a/examples/governance-interceptor/Cargo.lock +++ b/examples/governance-interceptor/Cargo.lock @@ -1083,6 +1083,7 @@ dependencies = [ "tonic-prost-build", "tracing", "url", + "uuid", ] [[package]] @@ -2267,6 +2268,17 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" +[[package]] +name = "uuid" +version = "1.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ef6dac1e96601b4fb3acccccff2139741fcb757cb9a36089bf5be91cfb285ce" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "wasm-bindgen", +] + [[package]] name = "version_check" version = "0.9.5" diff --git a/examples/supervisor-middleware-content-guard/Cargo.lock b/examples/supervisor-middleware-content-guard/Cargo.lock index f19951981b..41532a8980 100644 --- a/examples/supervisor-middleware-content-guard/Cargo.lock +++ b/examples/supervisor-middleware-content-guard/Cargo.lock @@ -19,9 +19,9 @@ checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" [[package]] name = "aho-corasick" -version = "1.1.4" +version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" dependencies = [ "memchr", ] @@ -78,19 +78,19 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.103" +version = "1.0.104" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a4385e2e34eb35d6b3efe798b9eb88096925d87726c0798709bf56d9ed84af3" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" [[package]] name = "async-trait" -version = "0.1.89" +version = "0.1.92" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 3.0.5", ] [[package]] @@ -197,21 +197,27 @@ checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "bitflags" -version = "2.13.0" +version = "2.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" + +[[package]] +name = "bumpalo" +version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" [[package]] name = "bytes" -version = "1.12.0" +version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ae3f5d315924270530207e2a68396c3cc547f6dca3fbdca317cfb1a51edb593" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" [[package]] name = "cc" -version = "1.2.65" +version = "1.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96" +checksum = "005ec2760ca554fae18df7a11195552ec576cd665632a881bc011d5bb2fd4d80" dependencies = [ "find-msvc-tools", "jobserver", @@ -233,9 +239,9 @@ checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" [[package]] name = "clap" -version = "4.6.1" +version = "4.6.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" +checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" dependencies = [ "clap_builder", "clap_derive", @@ -243,9 +249,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.6.0" +version = "4.6.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" +checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" dependencies = [ "anstream", "anstyle", @@ -255,14 +261,14 @@ dependencies = [ [[package]] name = "clap_derive" -version = "4.6.1" +version = "4.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" +checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" dependencies = [ "heck", "proc-macro2", "quote", - "syn", + "syn 3.0.5", ] [[package]] @@ -304,13 +310,13 @@ checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] name = "displaydoc" -version = "0.2.6" +version = "0.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 3.0.5", ] [[package]] @@ -321,9 +327,9 @@ checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" [[package]] name = "either" -version = "1.16.0" +version = "1.18.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" +checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34" [[package]] name = "equivalent" @@ -343,15 +349,15 @@ dependencies = [ [[package]] name = "fastrand" -version = "2.4.1" +version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" [[package]] name = "find-msvc-tools" -version = "0.1.9" +version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" +checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d" [[package]] name = "fixedbitset" @@ -388,36 +394,36 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" [[package]] name = "futures-channel" -version = "0.3.32" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4" dependencies = [ "futures-core", ] [[package]] name = "futures-core" -version = "0.3.32" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" +checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" [[package]] name = "futures-sink" -version = "0.3.32" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" +checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d" [[package]] name = "futures-task" -version = "0.3.32" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" [[package]] name = "futures-util" -version = "0.3.32" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" dependencies = [ "futures-core", "futures-task", @@ -455,15 +461,15 @@ checksum = "e629b9b98ef3dd8afe6ca2bd0f89306cec16d43d907889945bc5d6687f2f13c7" [[package]] name = "glob" -version = "0.3.3" +version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" +checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b" [[package]] name = "h2" -version = "0.4.15" +version = "0.4.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155" +checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16" dependencies = [ "atomic-waker", "bytes", @@ -501,9 +507,9 @@ checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" [[package]] name = "http" -version = "1.4.2" +version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" dependencies = [ "bytes", "itoa", @@ -511,9 +517,9 @@ dependencies = [ [[package]] name = "http-body" -version = "1.0.1" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" +checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" dependencies = [ "bytes", "http", @@ -521,9 +527,9 @@ dependencies = [ [[package]] name = "http-body-util" -version = "0.1.3" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" +checksum = "23169fe34a5fbcdd3f3862e78fb9b6fccd5f02a6dc6f732547005d45631ce71c" dependencies = [ "bytes", "futures-core", @@ -546,9 +552,9 @@ checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" [[package]] name = "hyper" -version = "1.10.1" +version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" +checksum = "27b501faa50e7a26c3d3560ca625132f4078a17771f4810baf70475ae48cbe43" dependencies = [ "atomic-waker", "bytes", @@ -601,9 +607,9 @@ dependencies = [ [[package]] name = "icu_collections" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513" dependencies = [ "displaydoc", "potential_utf", @@ -615,22 +621,43 @@ dependencies = [ [[package]] name = "icu_locale_core" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb" dependencies = [ "displaydoc", "litemap", + "serde", "tinystr", "writeable", "zerovec", ] +[[package]] +name = "icu_locale_fallback" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "251af8e57c9400e3eb58242fe5b8b1152b2a64fdf4cf632f923c38ccee6f2fa9" +dependencies = [ + "icu_locale_core", + "icu_locale_fallback_data", + "icu_provider", + "potential_utf", + "tinystr", + "zerovec", +] + +[[package]] +name = "icu_locale_fallback_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "decf2a22ec8fa68f1a0c1129a3f8583f8f8bc24e8b9ccbe98ead99f62a4dc3a8" + [[package]] name = "icu_normalizer" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f" dependencies = [ "icu_collections", "icu_normalizer_data", @@ -642,16 +669,17 @@ dependencies = [ [[package]] name = "icu_normalizer_data" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" +checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0" [[package]] name = "icu_properties" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148" dependencies = [ + "displaydoc", "icu_collections", "icu_locale_core", "icu_properties_data", @@ -662,18 +690,20 @@ dependencies = [ [[package]] name = "icu_properties_data" -version = "2.2.0" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" +checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa" [[package]] name = "icu_provider" -version = "2.2.0" +version = "2.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +checksum = "d27bbb9d3abbefac45d55f647c9de1d44aafcd1186eb91879afef17c396c3e73" dependencies = [ "displaydoc", "icu_locale_core", + "serde", + "stable_deref_trait", "writeable", "yoke", "zerofrom", @@ -681,6 +711,28 @@ dependencies = [ "zerovec", ] +[[package]] +name = "icu_segmenter" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82d07aafccd67af15d02512a6adf5896fbc5ed00f2e99b471d2efa14016db3db" +dependencies = [ + "icu_collections", + "icu_locale_fallback", + "icu_provider", + "icu_segmenter_data", + "potential_utf", + "smallvec", + "utf8_iter", + "zerovec", +] + +[[package]] +name = "icu_segmenter_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae293c039020f9ec10710af98d29ce6aa2051486638b49c9a6409f3b4a9e98ad" + [[package]] name = "idna" version = "1.1.0" @@ -704,9 +756,9 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.14.0" +version = "2.14.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855" dependencies = [ "equivalent", "hashbrown 0.17.1", @@ -714,9 +766,9 @@ dependencies = [ [[package]] name = "ipnet" -version = "2.12.0" +version = "2.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" +checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0" [[package]] name = "is_ci" @@ -755,11 +807,22 @@ dependencies = [ "libc", ] +[[package]] +name = "js-sys" +version = "0.3.105" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce57d20d1ea864ce2ac172ab472d409214f4fd359f0b2a2775abdf522e2af99e" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + [[package]] name = "libc" -version = "0.2.186" +version = "0.2.189" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" [[package]] name = "linux-raw-sys" @@ -769,9 +832,9 @@ checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" [[package]] name = "litemap" -version = "0.8.2" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" +checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae" [[package]] name = "lock_api" @@ -784,9 +847,9 @@ dependencies = [ [[package]] name = "log" -version = "0.4.33" +version = "0.4.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" +checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" [[package]] name = "matchit" @@ -796,9 +859,9 @@ checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" [[package]] name = "memchr" -version = "2.8.2" +version = "2.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" [[package]] name = "miette" @@ -827,7 +890,7 @@ checksum = "db5b29714e950dbb20d5e6f74f9dcec4edbcc1067bb7f8ed198c097b8c1a818b" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.119", ] [[package]] @@ -847,9 +910,9 @@ dependencies = [ [[package]] name = "mio" -version = "1.2.1" +version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda" +checksum = "4b18443e9c262bfe8fa82f51666e2642c53393f7e5c27b3e1aeab922cff5b9d8" dependencies = [ "libc", "wasi", @@ -922,6 +985,7 @@ dependencies = [ "tonic-prost-build", "tracing", "url", + "uuid", ] [[package]] @@ -956,9 +1020,9 @@ checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" [[package]] name = "owo-colors" -version = "4.3.0" +version = "4.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d211803b9b6b570f68772237e415a029d5a50c65d382910b879fb19d3271f94d" +checksum = "13c45bb4a6ae1280ec0803b1ef9d3455eb50f01efbbe1447ab020f1d54fba9d8" [[package]] name = "parking_lot" @@ -1017,7 +1081,7 @@ checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.119", ] [[package]] @@ -1034,10 +1098,12 @@ checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548" [[package]] name = "potential_utf" -version = "0.1.5" +version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +checksum = "d83eb9bc6d8e5cf568e7a1101d60ee05e81ed50ea106026f3d18deeb046d7661" dependencies = [ + "serde_core", + "writeable", "zerovec", ] @@ -1048,14 +1114,14 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn", + "syn 2.0.119", ] [[package]] name = "proc-macro2" -version = "1.0.106" +version = "1.0.107" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" dependencies = [ "unicode-ident", ] @@ -1087,7 +1153,7 @@ dependencies = [ "pulldown-cmark", "pulldown-cmark-to-cmark", "regex", - "syn", + "syn 2.0.119", "tempfile", ] @@ -1101,7 +1167,7 @@ dependencies = [ "itertools", "proc-macro2", "quote", - "syn", + "syn 2.0.119", ] [[package]] @@ -1190,18 +1256,18 @@ dependencies = [ [[package]] name = "pulldown-cmark-to-cmark" -version = "22.0.0" +version = "22.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50793def1b900256624a709439404384204a5dc3a6ec580281bfaac35e882e90" +checksum = "ab1ad36992cead65f02aa399a373a42730922f1525d988172634fdefdecb8a60" dependencies = [ "pulldown-cmark", ] [[package]] name = "quote" -version = "1.0.46" +version = "1.0.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" dependencies = [ "proc-macro2", ] @@ -1223,9 +1289,9 @@ dependencies = [ [[package]] name = "regex" -version = "1.12.4" +version = "1.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba" +checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" dependencies = [ "aho-corasick", "memchr", @@ -1235,9 +1301,9 @@ dependencies = [ [[package]] name = "regex-automata" -version = "0.4.14" +version = "0.4.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" dependencies = [ "aho-corasick", "memchr", @@ -1266,9 +1332,9 @@ dependencies = [ [[package]] name = "rustc-demangle" -version = "0.1.27" +version = "0.1.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b50b8869d9fc858ce7266cce0194bd74df58b9d0e3f6df3a9fc8eb470d95c09d" +checksum = "b74b56ffa8bb2830709a538c2cbcae9aa062db0d2a42563bfb09bdaae44020eb" [[package]] name = "rustix" @@ -1285,9 +1351,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.41" +version = "0.23.44" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b92b125634d9b795e7beca796cc790df15a7fb38323bf3196fda83292d06b1f" +checksum = "6725596c3f2c3a0aef021139e145d4eafe314a6623e4680ca83852b2c67ab2ba" dependencies = [ "aws-lc-rs", "log", @@ -1321,18 +1387,18 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.14.1" +version = "1.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9" +checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96" dependencies = [ "zeroize", ] [[package]] name = "rustls-webpki" -version = "0.103.13" +version = "0.103.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2" dependencies = [ "aws-lc-rs", "ring", @@ -1340,6 +1406,12 @@ dependencies = [ "untrusted", ] +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + [[package]] name = "schannel" version = "0.1.29" @@ -1380,9 +1452,9 @@ dependencies = [ [[package]] name = "serde" -version = "1.0.228" +version = "1.0.229" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" dependencies = [ "serde_core", "serde_derive", @@ -1390,29 +1462,29 @@ dependencies = [ [[package]] name = "serde_core" -version = "1.0.228" +version = "1.0.229" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.228" +version = "1.0.229" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 3.0.5", ] [[package]] name = "serde_json" -version = "1.0.150" +version = "1.0.151" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" dependencies = [ "itoa", "memchr", @@ -1445,15 +1517,15 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "smallvec" -version = "1.15.2" +version = "1.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" +checksum = "ba467056f1b547ed52077911161fc86985becbc60e8e1857c8a144dab0def891" [[package]] name = "socket2" -version = "0.6.4" +version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ "libc", "windows-sys 0.61.2", @@ -1500,9 +1572,20 @@ checksum = "b7401a30af6cb5818bb64852270bb722533397edcfc7344954a38f420819ece2" [[package]] name = "syn" -version = "2.0.118" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9" dependencies = [ "proc-macro2", "quote", @@ -1523,7 +1606,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.119", ] [[package]] @@ -1551,49 +1634,50 @@ dependencies = [ [[package]] name = "textwrap" -version = "0.16.2" +version = "0.16.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c13547615a44dc9c452a8a534638acdf07120d4b6847c8178705da06306a3057" +checksum = "b81c0cb5fce14f53e49c1d4da0c508334ff12040221bb8ab01b2dabd91d04b6e" dependencies = [ - "unicode-linebreak", + "icu_segmenter", "unicode-width 0.2.2", ] [[package]] name = "thiserror" -version = "2.0.18" +version = "2.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" dependencies = [ "thiserror-impl", ] [[package]] name = "thiserror-impl" -version = "2.0.18" +version = "2.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 3.0.5", ] [[package]] name = "tinystr" -version = "0.8.3" +version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +checksum = "b1e27c91459209c2986af3dcf603a5a74a4368754ce37414f59acc971167f643" dependencies = [ "displaydoc", + "serde_core", "zerovec", ] [[package]] name = "tokio" -version = "1.52.3" +version = "1.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" dependencies = [ "bytes", "libc", @@ -1608,20 +1692,20 @@ dependencies = [ [[package]] name = "tokio-macros" -version = "2.7.0" +version = "2.7.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 3.0.5", ] [[package]] name = "tokio-rustls" -version = "0.26.4" +version = "0.26.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67" dependencies = [ "rustls", "tokio", @@ -1629,9 +1713,9 @@ dependencies = [ [[package]] name = "tokio-stream" -version = "0.1.18" +version = "0.1.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" +checksum = "a3d06f0b082ba57c26b79407372e57cf2a1e28124f78e9479fe80322cf53420b" dependencies = [ "futures-core", "pin-project-lite", @@ -1640,13 +1724,14 @@ dependencies = [ [[package]] name = "tokio-util" -version = "0.7.18" +version = "0.7.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" +checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52" dependencies = [ "bytes", "futures-core", "futures-sink", + "libc", "pin-project-lite", "tokio", ] @@ -1691,7 +1776,7 @@ dependencies = [ "prettyplease", "proc-macro2", "quote", - "syn", + "syn 2.0.119", ] [[package]] @@ -1716,7 +1801,7 @@ dependencies = [ "prost-build", "prost-types", "quote", - "syn", + "syn 2.0.119", "tempfile", "tonic-build", ] @@ -1771,7 +1856,7 @@ checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.119", ] [[package]] @@ -1801,12 +1886,6 @@ version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" -[[package]] -name = "unicode-linebreak" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b09c83c3c29d37506a3e260c08c03743a6bb66a9cd432c6934ab501a190571f" - [[package]] name = "unicode-width" version = "0.1.14" @@ -1849,6 +1928,17 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" +[[package]] +name = "uuid" +version = "1.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ef6dac1e96601b4fb3acccccff2139741fcb757cb9a36089bf5be91cfb285ce" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "wasm-bindgen", +] + [[package]] name = "want" version = "0.3.1" @@ -1864,6 +1954,51 @@ version = "0.11.1+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" +[[package]] +name = "wasm-bindgen" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aecb87a33d3b0c5e3b7aa46336eaf486cffafbd281b195e4c8b80d50df2351bf" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a690d511e3c1a8b3a55e33511e3c2c00c78415cd23650f32b808627f5696b9ed" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "411e4887f0071ef2d2164a9d5fdf2d20efbef78fccd3a78b0c10a1dc5295e48a" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 3.0.5", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81941cd78d0c92026c33e5e01312845a4cb1e9af3407f9134b100dd03144103e" +dependencies = [ + "unicode-ident", +] + [[package]] name = "windows-link" version = "0.2.1" @@ -1954,9 +2089,9 @@ checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" [[package]] name = "writeable" -version = "0.6.3" +version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" +checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc" [[package]] name = "yoke" @@ -1977,7 +2112,7 @@ checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.119", "synstructure", ] @@ -1998,7 +2133,7 @@ checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.119", "synstructure", ] @@ -2010,21 +2145,23 @@ checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" [[package]] name = "zerotrie" -version = "0.2.4" +version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +checksum = "4ea269c3bd32f0a32c321907a2ae912ba6f4649bb0fc764a15627e99a7095a3f" dependencies = [ "displaydoc", "yoke", "zerofrom", + "zerovec", ] [[package]] name = "zerovec" -version = "0.11.6" +version = "0.11.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +checksum = "bb0464e17806c1d976d5cba29399c7f08e516e279e2ba493f63123b5fca67dd8" dependencies = [ + "serde", "yoke", "zerofrom", "zerovec-derive", @@ -2032,17 +2169,17 @@ dependencies = [ [[package]] name = "zerovec-derive" -version = "0.11.3" +version = "0.11.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 3.0.5", ] [[package]] name = "zmij" -version = "1.0.21" +version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/nix/pkgs/vm-runtime.nix b/nix/pkgs/vm-runtime.nix index 963209c8a6..eb3819c09f 100644 --- a/nix/pkgs/vm-runtime.nix +++ b/nix/pkgs/vm-runtime.nix @@ -13,31 +13,28 @@ let { x86_64-linux = { platform = "linux-x86_64"; - hash = "sha256-dw3Lc7IapCyNeE7j6dnlgd/b8Yc91/7IOi3XJORyILQ="; + hash = "sha256-kA6ZQz53geBp2zHx8tv+D8VcGkSrnC74Op/60H4fmck="; artifacts = [ "libkrun.so" "libkrunfw.so.5" - "gvproxy" "umoci" ]; }; aarch64-linux = { platform = "linux-aarch64"; - hash = "sha256-aJDuDb7AsuH9R+AyXA/JIxE9fJmZ5kP0Lkhg6F0Ot5A="; + hash = "sha256-7FNqUtC6ixfw202EyoYCgZOLdyxJYfg/YzdlkTT/+sM="; artifacts = [ "libkrun.so" "libkrunfw.so.5" - "gvproxy" "umoci" ]; }; aarch64-darwin = { platform = "darwin-aarch64"; - hash = "sha256-BDSeY5XGDozaBZzHTiQQX90jzsSc6shJZs5zdzludX0="; + hash = "sha256-cPfogd7QiPFk76kR1JJeKLaWwFksAZ9BeW/LsmDgsjo="; artifacts = [ "libkrun.dylib" "libkrunfw.5.dylib" - "gvproxy" "umoci" ]; }; @@ -59,7 +56,6 @@ stdenv.mkDerivation { mkdir -p "$out" tar --extract --file ${archive} --directory "$out" - mkdir -p "$out/compressed" for artifact in ${lib.escapeShellArgs runtime.artifacts}; do zstd -19 -T1 "$out/$artifact" -o "$out/compressed/$artifact.zst" diff --git a/proto/compute_driver.proto b/proto/compute_driver.proto index f9a19589f9..45ddd63d55 100644 --- a/proto/compute_driver.proto +++ b/proto/compute_driver.proto @@ -216,6 +216,44 @@ message DriverSandboxSpec { // One-shot launch hint forwarded by the gateway when the creating client // will attach to the canonical main process. bool await_main_process_attachment = 14; + // Admitted identity selectors that the driver must resolve before creating + // an immutable workload. Empty selectors mean the pinned image/rootfs + // defaults. Resolution is mandatory for supported isolation backends. + WorkloadIdentityRequest workload_identity = 15; + // Opaque, gateway-created launch credentials. The driver must split this + // material between the host supervisor and workload-side sandbox runtime; + // it must never expose the supervisor bearer tokens to the workload. + bytes launch_authentication = 16 [(openshell.options.v1.secret) = true]; +} + +// Identity inputs admitted by the gateway before workload provisioning. +message WorkloadIdentityRequest { + // User selector from policy or driver configuration (numeric or symbolic). + string user = 1; + // Group selector from policy or driver configuration (numeric or symbolic). + string group = 2; +} + +// Exact immutable identity selected by the driver from pinned runtime +// metadata. UID/GID zero are invalid for the capability-free sandbox. +message ResolvedWorkloadIdentity { + uint32 uid = 1; + uint32 gid = 2; + // Sorted, unique supplementary groups. GID zero is invalid. + repeated uint32 supplementary_gids = 3; + // Driver-defined resolution source such as policy, template, or image. + string source = 4; + // Immutable image/rootfs/config digest used during resolution. + string resource_digest = 5; +} + +// Driver-owned proof that the immutable workload and its outer network fence +// match the sandbox generation. The gateway joins this with sandbox and +// supervisor evidence before permitting the first untrusted instruction. +message DriverFenceEvidence { + string generation = 1; + string evidence_digest = 2; + map resource_claims = 3; } message ResourceRequirements { @@ -300,6 +338,10 @@ message DriverSandboxStatus { repeated DriverCondition conditions = 5; // True when the compute platform has begun deleting this sandbox. bool deleting = 6; + // Exact process identity used to create the workload. + ResolvedWorkloadIdentity resolved_identity = 7; + // Immutable backend and outer-fence evidence for this generation. + DriverFenceEvidence fence_evidence = 8; } // Raw compute-platform condition. @@ -379,6 +421,12 @@ message StartSandboxRequest { string sandbox_id = 1; // Compute-runtime name used by the driver. string sandbox_name = 2; + // Fresh launch credentials for start-from-stopped. Empty only for drivers + // that do not implement the OpenShell Sandbox Protocol. + bytes launch_authentication = 3 [(openshell.options.v1.secret) = true]; + // Stable identity of this gateway start transition. Retries of the same + // transition carry the same generation ID; a later start uses a new ID. + string generation_id = 4; } message StartSandboxResponse {} diff --git a/proto/openshell.proto b/proto/openshell.proto index def6d7c473..33e331115e 100644 --- a/proto/openshell.proto +++ b/proto/openshell.proto @@ -785,6 +785,14 @@ message RefreshSandboxTokenResponse { // Fresh credentials for the requested, policy-authorized extension // services. These remain in supervisor memory and are never persisted. repeated ExtensionServiceCredential extension_credentials = 3; + // Fresh Sandbox Protocol bearer token from the same atomic refresh. + string sandbox_token = 4 [(openshell.options.v1.secret) = true]; + // Absolute Sandbox Protocol token expiry, milliseconds since the epoch. + int64 sandbox_expires_at_ms = 5; + // Launch generation to which both refreshed credentials are bound. + string session_id = 6; + // Durable authorization epoch shared by the gateway and Sandbox Runtime. + uint64 credential_epoch = 7; } diff --git a/python/openshell/gateway_schema_v2_compute_boundary_verifier_test.py b/python/openshell/gateway_schema_v2_compute_boundary_verifier_test.py index 6e8009b759..43b3b4aa2c 100644 --- a/python/openshell/gateway_schema_v2_compute_boundary_verifier_test.py +++ b/python/openshell/gateway_schema_v2_compute_boundary_verifier_test.py @@ -22,6 +22,7 @@ IMAGE_ID = "3" * 64 IMAGE_DIGEST = f"sha256:{'4' * 64}" RUNTIME_IMAGE = f"localhost/openshell/supervisor@{IMAGE_DIGEST}" +BOUNDARY_IMAGE = f"localhost/openshell/sandbox@{IMAGE_DIGEST}" BASE_RUNTIME_IMAGE = f"docker.io/library/alpine@{IMAGE_DIGEST}" @@ -126,6 +127,7 @@ def create_variant( "sandbox_image_id": IMAGE_ID, "sandbox_image_digest": IMAGE_DIGEST, "sandbox_runtime_image": "example.invalid/sandbox@" + IMAGE_DIGEST, + "sandbox_boundary_image": BOUNDARY_IMAGE, "sandbox_client_image_alias": "example.invalid/sandbox:latest", "sandbox_client_image_alias_id": IMAGE_ID, "gateway_sha256_before_execution": result["gateway_sha256"], @@ -146,6 +148,7 @@ def create_variant( "external_driver_proxy": False, "external_driver_app_armor": False, "external_driver_environment": { + "XDG_DATA_HOME": f"/tmp/{variant}-driver-data", "OPENSHELL_COMPUTE_DRIVER_SOCKET": f"/tmp/{variant}.sock", "OPENSHELL_PODMAN_SOCKET": f"/tmp/{variant}-podman.sock", "OPENSHELL_SANDBOX_IMAGE": "example.invalid/sandbox@" + IMAGE_DIGEST, @@ -155,6 +158,7 @@ def create_variant( "OPENSHELL_GATEWAY_PORT": 18181, "OPENSHELL_NETWORK_NAME": f"{variant}-network", "OPENSHELL_STOP_TIMEOUT": 15, + "OPENSHELL_SANDBOX_RUNTIME_IMAGE": BOUNDARY_IMAGE, "OPENSHELL_SUPERVISOR_IMAGE": RUNTIME_IMAGE, "OPENSHELL_PODMAN_TLS_CA": { "path": f"/tmp/{variant}-pki/ca.crt", @@ -178,7 +182,7 @@ def create_variant( (results_dir / f"{variant}.log").write_text( f"CLI conformance run ID: fixture\n" f"gateway preflight connected: gateway=fixture, authentication=authenticated\n" - f"{lifecycle}\n{RUNTIME_IMAGE} {BASE_RUNTIME_IMAGE} example.invalid/sandbox@{IMAGE_DIGEST} example.invalid/sandbox:latest " + f"{lifecycle}\n{RUNTIME_IMAGE} {BOUNDARY_IMAGE} {BASE_RUNTIME_IMAGE} example.invalid/sandbox@{IMAGE_DIGEST} example.invalid/sandbox:latest " f'{IMAGE_ID} {IMAGE_DIGEST} {package_hash}\n"passed": true\n', encoding="utf-8", ) diff --git a/rfc/0012-isolation-backend/README.md b/rfc/0012-isolation-backend/README.md index a074292794..321df6d6bc 100644 --- a/rfc/0012-isolation-backend/README.md +++ b/rfc/0012-isolation-backend/README.md @@ -19,7 +19,7 @@ links: Today the supervisor both builds the workload's isolation boundary and applies its network policy. Because the supervisor runs inside the agent container, the privilege needed to build that boundary sits beside the code it confines. This RFC moves boundary construction and process operations behind a pluggable **Isolation Backend**. The supervisor continues to apply approved network policy through network mediation. -The compute driver provisions the workload and trusted components. The logical supervisor is the trusted bridge between the gateway and the workload: it maintains the gateway connection, handles authorized requests, and drives the backend. The backend establishes the isolation controls, manages workload processes, and routes egress to network mediation. The same lifecycle supports today's in-pod implementation and future delegated implementations without topology-specific supervisor paths. +The compute driver orchestrates the workload, supervisor, sandbox runtime, protected channel, and outer network fence. The supervisor is the trusted bridge between the gateway and the workload: it maintains the gateway connection, handles authorized requests, and drives the backend. The backend establishes the isolation controls, manages workload processes, and routes egress to network mediation. The same lifecycle supports different compute drivers and backend implementations without driver-specific supervisor paths. ## Motivation @@ -33,7 +33,7 @@ All three come from coupling boundary construction to boundary operation. A comm ## Non-goals -- **Implementing a delegated backend.** Each topology requires its own design and implementation. +- **Implementing a delegated backend.** Each backend implementation and compute-driver placement requires its own design and implementation. - **Changing authorization.** [RFC 0001](../0001-core-architecture/README.md) owns control-plane and sandbox identity. A delegated backend must still authenticate callers and scope them to one boundary. - **Standardizing backend-internal component coordination.** A backend may coordinate helper, sidecar, or interception processes behind one lifecycle; how those components cooperate is backend-specific, not contract surface. - **Changing gateway lifecycle or public status.** This RFC adds no gateway activation operation, public phase, or status API, and it does not define how a boundary's effective isolation model is surfaced to operators. @@ -42,23 +42,23 @@ All three come from coupling boundary construction to boundary operation. A comm The mental model has three roles: -- The **compute driver** provisions the sandbox instance according to the selected placement of the workload and trusted isolation components. That placement is the **topology**. -- The **Isolation Backend** establishes and operates the topology-specific controls around the workload. It also routes workload egress to network mediation and provides process operations. +- The **compute driver** orchestrates the sandbox instance: it places the workload, supervisor, and sandbox runtime; creates their protected channel; and applies the outer network fence. +- The **Isolation Backend** establishes and operates the controls around the workload. It also routes workload egress to network mediation and provides process operations. - The **logical supervisor** is the trusted control-plane bridge between the gateway and the workload. It drives the backend, handles authorized gateway requests, and applies approved network policy through network mediation. Together, network policy, filesystem isolation, syscall filtering, and sandbox identity form the workload's isolation boundary. The roles above enforce that boundary and may run in one process or across several trusted components. Their placement does not change the contract. -Each active boundary has at most one logical supervisor, which may span multiple coupled processes. The backend routes all workload egress through a per-boundary source, and the supervisor consumes that source. Internal delegation and transport remain topology-private. +Each active boundary has at most one logical supervisor, which may span multiple coupled processes. The backend routes all workload egress through a per-boundary source, and the supervisor consumes that source. Internal delegation and transport remain backend-private. [RFC 0001](../0001-core-architecture/README.md) continues to own sandbox authentication and authorization. In this contract, sandbox identity means binding the authenticated sandbox context to the isolation boundary. -Admission selects the sandbox's topology and determines its trusted context. The compute driver sets up the topology and gives the logical supervisor a `TopologyDescriptor` describing what it provisioned. The supervisor uses the descriptor to attach the matching Isolation Backend. The backend prepares the required controls before the agent starts. +Admission selects the sandbox's Isolation Backend and determines its trusted context. The compute driver orchestrates the required resources and gives the supervisor a `BackendDescriptor` describing the selected backend and provisioned runtime. The supervisor uses the descriptor to attach the matching Isolation Backend. The backend prepares the required controls before the agent starts. ```mermaid flowchart TB Gateway["Gateway"] -->|"create sandbox"| Driver["Compute driver"] - subgraph Topology["Driver-provisioned topology (placement varies)"] + subgraph Runtime["Driver-orchestrated sandbox"] Supervisor["Supervisor"] Backend["Isolation Backend (may coordinate components)"] subgraph Boundary["Isolation boundary"] @@ -76,13 +76,13 @@ flowchart TB Workload ==>|"only egress"| Mediator end - Driver -->|"resources + TopologyDescriptor"| Supervisor + Driver -->|"resources + BackendDescriptor"| Supervisor Mediator -->|"allowed egress"| Egress["Egress"] ``` -In the in-pod topology, the supervisor drives a backend implemented in the same process. Other topologies may delegate backend operations without changing the supervisor lifecycle. +The initial implementation is `OpenShellRuntimeBackend`: a supervisor-side client that delegates operations over the authenticated OpenShell Sandbox Protocol to `openshell-sandbox` beside the workload. A future runtime-native backend could implement the same interface without using that protocol or binary. -A boundary is active from successful `attach` until normal backend cleanup releases the binding or the topology's trusted cleanup path invalidates it. A backend may coordinate multiple trusted helper or interception processes for that boundary. The backend owns the active-boundary binding; the compute driver owns the sandbox instance and topology lifecycle. +A boundary is active from successful `attach` until normal backend cleanup releases the binding or the compute driver's trusted cleanup path invalidates it. A backend may coordinate multiple trusted helper or interception processes for that boundary. The backend owns the active-boundary binding; the compute driver owns the sandbox instance and infrastructure lifecycle. ### Contract invariants @@ -93,56 +93,63 @@ Six invariants hold for every boundary: 3. An operation is authorized only when the complete effective policy permits it; network operations are decided through network mediation. There is no silent weakening. 4. Agent startup, `exec`, and forwarding occur only through the active backend, and every workload process remains in the compute driver's provisioned execution environment. 5. Shared infrastructure preserves strict per-boundary lifecycle, policy, identity, enforcement, and cleanup isolation. -6. If the logical supervisor is lost, the boundary remains under its last confirmed enforcement state while supervisor-dependent operations fail closed. Loss of required enforcement ends `Running` and terminates all workload processes within a documented bound; detection and termination may be performed by a trusted node or control-plane actor. Network-mediation unavailability denies outbound connections and never enables direct egress. +6. If the logical supervisor is lost, the boundary remains under its last confirmed enforcement state while supervisor-dependent operations fail closed. Loss of required enforcement ends `Running` and terminates all workload processes within a documented bound; detection and termination may be performed by a trusted node or control-plane actor. Network-mediation unavailability denies outbound connections and never enables direct egress. Exit of the Sandbox Runtime that owns the boundary terminates its workload through the compute driver's containment primitive; it never leaves an unmanaged agent process tree. Each backend states its termination bound in its implementation documentation. Loss of the logical supervisor means loss of the components holding the backend lifecycle, not loss of the gateway connection; gateway disconnection follows RFC 0001's reconnection semantics. +Transport recovery is scoped to the same logical supervisor process. A backend +may reconnect that process and re-confirm its binding, but a replacement +supervisor process cannot resume an existing runtime generation with launch +credentials. Planned upgrades stop the sandbox and start a fresh generation. + ### Provisioning -Provisioning runs on the control plane, and three rules hold in every topology: +Provisioning runs on the control plane, and three rules hold for every sandbox: -1. **Admission selects the topology** from trusted deployment configuration, not `SandboxPolicy`, and records its required backend. The `TopologyDescriptor` supplied by the compute driver must name that backend, and resolution never falls back to another backend. -2. **The compute driver provisions the topology** and anything the selected backend needs. +1. **Admission selects the Isolation Backend** from trusted deployment configuration, not `SandboxPolicy`. The `BackendDescriptor` supplied by the compute driver must name that backend, and resolution never falls back to another backend. +2. **The compute driver orchestrates the sandbox** and anything the selected backend needs. 3. **The backend establishes standing enforcement before untrusted code runs**, during provisioning or `attach`, depending on the backend. -If a topology depends on cluster-scoped coverage or registration, admission verifies that the prerequisite covers the boundary's placement before untrusted code runs. +If a driver depends on cluster-scoped coverage or registration, admission verifies that the prerequisite covers the boundary's placement before untrusted code runs. + +Every compute driver provides a trusted cleanup path that does not depend on supervisor availability. -Every topology provides a trusted cleanup path that does not depend on logical-supervisor availability. +A compute driver may provision a resource and `BackendDescriptor` before the control plane assigns it to a sandbox. No untrusted workload runs while the resource is unassigned. After claim or assignment produces a trusted `SandboxContext`, the supervisor calls `attach`; the backend either binds that context to the prepared resource and returns `Bound`, or rejects it as incompatible. Pool creation, claim, reset, release, and recycling remain outside this contract. -A compute driver may provision a resource and `TopologyDescriptor` before the control plane assigns it to a sandbox. No untrusted workload runs while the resource is unassigned. After claim or assignment produces a trusted `SandboxContext`, the supervisor calls `attach`; the backend either binds that context to the prepared resource and returns `Bound`, or rejects it as incompatible. Pool creation, claim, reset, release, and recycling remain outside this contract. +Every create or start-from-stopped operation receives a fresh `SandboxSessionId`. Retries of the same durable launch transition reuse that identity, while a later launch receives a new one even when the compute platform reuses its outer resource. The backend binds `attach` to that exact session and rejects descriptors or runtime sessions left over from an earlier launch. -### The topology descriptor +### The backend descriptor -The driver supplies a descriptor for every provisioned topology, including in-pod and resources prepared before assignment. The common envelope names the backend and carries an opaque payload. +The driver supplies a descriptor for every provisioned sandbox, including resources prepared before assignment. The common envelope names the backend and carries an opaque payload. ```rust -struct TopologyDescriptor { +struct BackendDescriptor { backend_name: String, version: u32, payload: Vec, } ``` -`version` is the Isolation Backend interface version. Backend name and version match exactly; this contract does not negotiate compatibility ranges. The descriptor is transport-neutral. Provisioning supplies it to the supervisor before `attach`; how it is transported is topology-specific and outside this contract, and every transport preserves one property: workload-controlled input cannot select or modify the descriptor. +`version` is the Isolation Backend interface version. Backend name and version match exactly; this contract does not negotiate compatibility ranges. The descriptor is transport-neutral. Provisioning supplies it to the supervisor before `attach`; how it is transported is compute-driver-specific and outside this contract, and every transport preserves one property: workload-controlled input cannot select or modify the descriptor. -The opaque payload identifies, or gives the backend enough information to resolve, the exact driver-provisioned resource. It may also carry topology-specific endpoint or helper-role information; there are no common topology or role fields. +The opaque payload identifies, or gives the backend enough information to resolve, the exact driver-provisioned resource. It may also carry backend-specific endpoint or helper-role information; there are no common placement or role fields. Common verification requires: -- the descriptor's `backend_name` matches the backend required by the admitted topology; +- the descriptor's `backend_name` matches the backend selected during admission; - the descriptor's version is one the supervisor supports, and the resolved backend reports that same version; and - `SandboxContext` is constructed after the control plane assigns the resource to the admitted sandbox, using authenticated control-plane and trusted supervisor state. -The supervisor validates the descriptor's common fields and produces a `VerifiedTopologyDescriptor`, then resolves its `backend_name` and version without fallback. Verification does not imply that the opaque payload is valid; the selected backend validates it and atomically binds the provisioned resource to the trusted `SandboxContext` during `attach`. Any failure rejects the sandbox. +The supervisor validates the descriptor's common fields and produces a `VerifiedBackendDescriptor`, then resolves its `backend_name` and version without fallback. Verification does not imply that the opaque payload is valid; the selected backend validates it and atomically binds the provisioned resource to the trusted `SandboxContext` during `attach`. Any failure rejects the sandbox. ### The lifecycle The contract does not prescribe enforcement mechanisms; it standardizes how the supervisor drives whichever backend a deployment admits. -A backend registers under a `backend_name` and version. The supervisor attaches to the admitted topology and drives the boundary through a fixed sequence of states. Each transition consumes the prior state, so the supervisor cannot skip a stage or invoke a later transition through an earlier handle. The Rust names are illustrative; the states and their semantics are normative. +A backend registers under a `backend_name` and version. The supervisor attaches to the admitted sandbox and drives the boundary through a fixed sequence of states. Each transition consumes the prior state, so the supervisor cannot skip a stage or invoke a later transition through an earlier handle. The Rust names are illustrative; the states and their semantics are normative. ```text -attach topology + sandbox context -> Bound -> confirm -> Ready -> start_agent -> Running +attach backend descriptor + sandbox context -> Bound -> confirm -> Ready -> start_agent -> Running ``` ```rust @@ -153,13 +160,14 @@ trait IsolationBackend: Send + Sync { async fn attach( &self, - descriptor: VerifiedTopologyDescriptor, + descriptor: VerifiedBackendDescriptor, sandbox: SandboxContext, ) -> Result, BackendError>; } struct SandboxContext { sandbox_id: SandboxId, + session_id: SandboxSessionId, policy: SandboxPolicy, agent: AgentSpec, } @@ -190,11 +198,11 @@ trait RunningBoundary: Send + Sync { `AgentSpec` carries the complete admitted agent launch specification, including command, arguments, working directory, timeout, and interactive mode. -`SandboxContext` carries the admitted create-time policy. [RFC 0002](../0002-agent-driven-policy-management/README.md) defines how network-policy revisions are proposed and approved. Approved revisions reach the supervisor through the existing [`GetSandboxConfig`](../../proto/sandbox.proto) gateway-supervisor contract, described in the [gateway](../../architecture/gateway.md) and [sandbox](../../architecture/sandbox.md#policy-revision-acknowledgement) architecture. The supervisor makes approved network-policy revisions effective through network mediation. If an approved network-policy revision cannot be loaded, it never becomes effective; the configured rejection posture retains the last valid generation or denies network access until a valid generation is loaded. +`SandboxContext` carries the admitted create-time policy and the identity of this launch. [RFC 0002](../0002-agent-driven-policy-management/README.md) defines how network-policy revisions are proposed and approved. Approved revisions reach the supervisor through the existing [`GetSandboxConfig`](../../proto/sandbox.proto) gateway-supervisor contract, described in the [gateway](../../architecture/gateway.md) and [sandbox](../../architecture/sandbox.md#policy-revision-acknowledgement) architecture. The supervisor makes approved network-policy revisions effective through network mediation. If an approved network-policy revision cannot be loaded, it never becomes effective; the configured rejection posture retains the last valid generation or denies network access until a valid generation is loaded. The states have normative meanings: -- **Bound:** the topology descriptor and trusted sandbox context are bound to the same resource, and the network-mediation source is available. No untrusted workload code is running. +- **Bound:** the backend descriptor and trusted sandbox context are bound to the same resource, and the network-mediation source is available. No untrusted workload code is running. - **Ready:** the backend has confirmed standing enforcement for this concrete boundary and is prepared to apply the admitted launch-time controls before untrusted execution. - **Running:** `start_agent` has made the admitted agent runnable and returned `RunningBoundary`. Every applicable launch-time control was in force before the first untrusted instruction. Whether the backend creates the agent process or releases a held, driver-provisioned execution object is backend-specific; the contract fixes the ordering, not the mechanism. @@ -251,18 +259,30 @@ trait BoundaryTerminal: Send + Sync { ```rust #[async_trait] trait NetworkMediationSource: Send + Sync { - async fn accept(&self) -> Result; + async fn accept_tcp(&self) -> Result; + async fn accept_dns(&self) -> Result; } -struct MediatedConnection { +struct PendingTcpOpen { stream: BoundaryDuplexStream, binary_identity: Result, + destination: SocketAddr, + socket: NetworkSocketMetadata, + policy_generation: u64, + decision: oneshot::Sender, +} + +struct PendingDnsQuery { + message: Vec, // exactly one DNS message; no TCP length prefix + transport: DnsTransport, + binary_identity: Result, + response: oneshot::Sender, BackendError>>, } ``` -`NetworkMediationSource` supplies outbound connections from one boundary to supervisor-owned network mediation. The backend routes all workload egress through that source and authoritatively associates each connection with the boundary without relying solely on workload-provided data. Capture, transport, placement, and coordination are backend-private. +`NetworkMediationSource` supplies typed TCP opens and DNS queries from one boundary to supervisor-owned network mediation. The two accepts are independent so the supervisor can consume them concurrently without coupling their ordering or backpressure. The backend routes all workload egress through these sources and authoritatively associates each request with the boundary without relying solely on workload-provided data. Capture, transport, framing, placement, and coordination are backend-private. General UDP mediation is outside this contract. -Every topology may use the same supervisor-owned mediation libraries or services; the source does not require a backend-specific policy engine. +Every backend may use the same supervisor-owned mediation libraries or services; the source does not require a backend-specific policy engine. Shared implementations isolate each boundary's state and enforcement. Failure or teardown of one boundary cannot weaken another. Network-mediation unavailability never enables direct egress. @@ -293,84 +313,84 @@ The logical supervisor resolves `backend_name` and version through a trusted imp The supervisor runs the same sequence for every backend: -1. Obtain the `TopologyDescriptor` and trusted `SandboxContext`. +1. Obtain the `BackendDescriptor` and trusted `SandboxContext`. 2. Verify the descriptor and resolve its `backend_name` and version without fallback. 3. Call `attach` to obtain `Bound`. 4. Connect the boundary's `NetworkMediationSource` to network mediation. 5. Call `confirm` to obtain `Ready`, then `start_agent` to obtain `Running`. 6. Use the returned runtime handles for agent wait, `exec`, and port forwarding while network mediation consumes outbound connections. -This RFC supersedes RFC 0001's fixed in-sandbox supervisor placement and its assignment of topology-specific isolation controls to that process, generalizing the supervisor into a logical role. RFC 0001's authentication, sandbox-identity, outbound-connection, session, and reconnection requirements continue to apply. The component hosting the logical supervisor holds the required outbound gateway connection. A driver-hosted or shared supervisor routes gateway `exec`, SSH, and forwarding requests through the backend; the gateway does not initiate a connection to the boundary. +This RFC supersedes RFC 0001's fixed in-sandbox supervisor placement and its assignment of driver-specific isolation controls to that process, generalizing the supervisor into a logical role. RFC 0001's authentication, sandbox-identity, outbound-connection, session, and reconnection requirements continue to apply. The component hosting the logical supervisor holds the required outbound gateway connection. A driver-hosted or shared supervisor routes gateway `exec`, SSH, and forwarding requests through the backend; the gateway does not initiate a connection to the boundary. ### Failure semantics Every failure carries a machine-readable kind for supervisor status mapping: ```rust -enum BackendErrorKind { Invalid, Denied, Unavailable, Failed, Terminated } +enum BackendErrorKind { Invalid, Denied, Unavailable, Unsupported, Failed, Terminated } ``` -`Invalid` covers descriptor, version, and backend mismatches; `Denied` covers authenticated attachment rejection; `Unavailable` covers transient inability to serve an operation; `Failed` covers other backend faults; and `Terminated` reports boundary or workload termination, or an operation against an inactive boundary. An error never advances the lifecycle or authorizes an operation, and backend selection never falls back. +`Invalid` covers descriptor, version, and backend mismatches; `Denied` covers authenticated attachment rejection; `Unavailable` covers transient inability to serve an operation; `Unsupported` identifies an optional operation the selected backend does not implement; `Failed` covers other backend faults; and `Terminated` reports boundary or workload termination, or an operation against an inactive boundary. An error never advances the lifecycle or authorizes an operation, and backend selection never falls back. -A backend may retry backend-private work within one `attach` call. The supervisor calls `attach` at most once per provisioned topology. If it does not return `Bound`, the topology is reclaimed rather than reused. +A backend may retry backend-private work within one `attach` call. The supervisor calls `attach` at most once per provisioned sandbox. If it does not return `Bound`, the sandbox is reclaimed rather than reused. Failures resolve as follows: -- an `attach` or `confirm` failure, or network-mediation initialization failure while `Bound`, prevents untrusted workload execution and causes the driver to reclaim the topology; -- if `start_agent` does not return `Running`, no untrusted process from that attempt remains, and the driver reclaims the topology; +- an `attach` or `confirm` failure, or network-mediation initialization failure while `Bound`, prevents untrusted workload execution and causes the driver to reclaim the sandbox; +- if `start_agent` does not return `Running`, no untrusted process from that attempt remains, and the driver reclaims the sandbox; - if `exec` or loopback `connect` fails, the backend terminates any process or closes any connection created by that attempt while the boundary otherwise remains active; - after `Running`, supervisor or enforcement loss follows invariant 6; when enforcement loss ends the agent, `BoundaryProcess::wait` fails with `BackendErrorKind::Terminated` where process-exit observation survives; - network-mediation errors yield no authorized connection and do not by themselves end `Running`; and - retained runtime handles and the network-mediation source reject new operations whenever the boundary ends, except `BoundaryProcess::wait` where the backend can still return its stable result. -Whenever a boundary ends, the backend terminates remaining workload processes and releases the active-boundary binding before the compute driver reclaims or deprovisions the topology. If normal backend cleanup is unavailable, the compute driver uses the topology's trusted cleanup path to terminate the execution environment and invalidate the binding before reclaim or reuse. On normal agent exit, `BoundaryProcess::wait` returns the stable exit status. A retained `wait` result may outlive teardown. +Whenever a boundary ends, the backend terminates remaining workload processes and releases the active-boundary binding before the compute driver reclaims or deprovisions the sandbox. If normal backend cleanup is unavailable, the compute driver uses its trusted cleanup path to terminate the execution environment and invalidate the binding before reclaim or reuse. On normal agent exit, `BoundaryProcess::wait` returns the stable exit status. A retained `wait` result may outlive teardown. -### Topologies +### Compute-driver placement -The contract fixes the roles; a topology fixes their placement. Components may be co-located with the workload or hosted in trusted services, and one component may implement multiple roles. Every arrangement preserves the same lifecycle, interfaces, and invariants. Actual containment depends on the workload's kernel relationship to the trusted components. The non-normative [topology matrix](./topology-matrix.md) catalogs representative placements. +The contract fixes the roles; each compute driver decides where they run. Components may be co-located with the workload or hosted in trusted services, and one component may implement multiple roles. Every arrangement preserves the same lifecycle, interfaces, and invariants. Actual containment depends on the workload's kernel relationship to the trusted components. The non-normative [driver placement guide](./driver-placement.md) describes the supported mappings. ## Conformance -Every admitted backend and topology must pass tests for the six contract invariants plus descriptor verification, lifecycle ordering, runtime operations, and failure semantics. +Every admitted backend and compute driver must pass tests for the six contract invariants plus descriptor verification, lifecycle ordering, runtime operations, and failure semantics. -The conformance suite exercises each backend in its deployed topology, including delegated components and their transport. It verifies observable contract behavior while transport details remain backend-private. +The conformance suite exercises each backend in its driver-orchestrated placement, including delegated components and their transport. It verifies observable contract behavior while transport details remain backend-private. ## Risks | Risk | Mitigation | |---|---| -| The Isolation Backend could duplicate compute-driver responsibilities or allow topology-specific behavior to leak back into the supervisor. | Keep the responsibility boundary explicit: the compute driver owns, provisions, and deprovisions the topology; the backend binds and operates the active boundary. The same component may implement both roles. | -| Contract conformance could be mistaken for equivalent isolation across topologies. | Treat conformance as behavioral, not as a security-strength rating. Document and validate each topology's actual containment and reject policy it cannot enforce. | +| The Isolation Backend could duplicate compute-driver responsibilities or allow driver-specific behavior to leak back into the supervisor. | Keep the responsibility boundary explicit: the compute driver orchestrates and deprovisions the sandbox infrastructure; the backend binds and operates the active boundary. The same component may implement both roles. | +| Contract conformance could be mistaken for equivalent isolation across compute drivers. | Treat conformance as behavioral, not as a security-strength rating. Document and validate each driver's actual containment and reject policy it cannot enforce. | | Shared backend or network-mediation components concentrate privilege and failure impact. | Isolate state, connection attribution, enforcement, and control authority per boundary. Failure of one boundary must not weaken another or enable direct egress. | | The mandatory contract may exclude otherwise useful but incomplete backends. | Keep the network-mediation source, binary identity, process control, `exec`, and port forwarding mandatory. An incomplete backend does not claim conformance or silently degrade. | -| A future topology may not fit the lifecycle or interfaces. | Keep placement and coordination backend-private. Add versioned contract surface only when a concrete implementation requires new common semantics. | +| A future runtime may not fit the lifecycle or interfaces. | Keep placement and coordination backend-private. Add versioned contract surface only when a concrete implementation requires new common semantics. | | A component restart may interrupt boundary operation. | Keep the last confirmed enforcement state in force and deny supervisor-dependent operations. | ## Alternatives ### Keep isolation embedded in the supervisor -OpenShell could keep the current in-pod design and add topology-specific supervisor and compute-driver paths as new requirements arise. +OpenShell could keep the current in-pod design and add driver-specific supervisor paths as new requirements arise. -Doing nothing avoids a new interface, but retains privileged boundary construction beside the workload. Implementing each delegated topology as a one-off supervisor change moves that privilege for one placement but accretes topology-specific supervisor behavior. The proposed contract instead keeps one supervisor lifecycle while allowing the topology to change. +Doing nothing avoids a new interface, but retains privileged boundary construction beside the workload. Implementing each runtime placement as a one-off supervisor change moves that privilege for one driver but adds driver-specific supervisor behavior. The proposed contract instead keeps one supervisor lifecycle while allowing runtime implementations and placements to change. ### Extend the compute-driver contract The compute driver could own both provisioning and active-boundary operation. -This is natural for topologies such as MXC, and the same component may implement both responsibilities. The interfaces remain distinct because they serve different callers and lifecycles: the gateway uses the compute driver to provision and deprovision resources, while the supervisor uses the Isolation Backend to operate an active boundary. Combining them would couple runtime policy, identity, network mediation, and process operations to the gateway-facing driver API. +This is natural for drivers such as MXC, and the same component may implement both responsibilities. The interfaces remain distinct because they serve different callers and lifecycles: the gateway uses the compute driver to provision and deprovision resources, while the supervisor uses the Isolation Backend to operate an active boundary. Combining them would couple runtime policy, identity, network mediation, and process operations to the gateway-facing driver API. ### Start with a remote backend service The contract could be expressed as a gRPC service or plugin ABI rather than an in-process Rust contract. [RFC 0001](../0001-core-architecture/README.md) chose gRPC for its gateway-facing drivers, so the question applies here. -The callers differ. A gateway driver is a control-plane peer with its own release cycle, while the Isolation Backend is driven by the supervisor that operates the boundary, and the co-located topology needs no transport at all. Starting in-process serves that case directly and lets delegated implementations carry their own transport behind the same interface. A transport-bearing surface is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it. +The callers differ. A gateway driver is a control-plane peer with its own release cycle, while the Isolation Backend is driven by the supervisor that operates the boundary, and a co-located implementation needs no transport at all. Starting in-process serves that case directly and lets delegated implementations carry their own transport behind the same interface. A transport-bearing surface is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it. -### Standardize topology and capabilities +### Standardize placement and capabilities -The contract could expose common topology roles, placement fields, capability flags, and recovery behavior so the supervisor can compose backend components. +The contract could expose common placement roles, capability flags, and recovery behavior so the supervisor can compose backend components. -That would make known deployments explicit, but it would also encode current topology assumptions and introduce capability-dependent supervisor paths. The proposal keeps placement and coordination in the opaque descriptor, requires one baseline contract, and uses the non-normative topology matrix to document representative arrangements. +That would make known deployments explicit, but it would also encode current placement assumptions and introduce capability-dependent supervisor paths. The proposal keeps placement and coordination in the opaque descriptor, requires one baseline contract, and uses the non-normative driver placement guide to document supported arrangements. ## Prior art diff --git a/rfc/0012-isolation-backend/topology-matrix.md b/rfc/0012-isolation-backend/topology-matrix.md deleted file mode 100644 index 8dc14d84ba..0000000000 --- a/rfc/0012-isolation-backend/topology-matrix.md +++ /dev/null @@ -1,36 +0,0 @@ -# Topology matrix - -This non-normative matrix compares representative mappings of RFC 0012's logical -roles. It records role placement, sharing, and relationship to the workload -kernel; it does not select a deployment or establish conformance. - -## Representative placements - -| Pattern | Logical supervisor and network-mediation placement | Backend placement | Workload-kernel relationship | Topology status | -|---|---|---|---|---| -| **Co-located/in-pod** | With the workload | In the supervisor process | Trusted components share the workload's host, guest, or application kernel, depending on the runtime | Placement implemented (original topology) | -| **Same-pod composite** | Spans the workload-local supervisor process and, when used, a network-mediation sidecar | In the workload-local supervisor process | Components share the workload's kernel | Placement implemented (#2076) | -| **Delegated backend components** | With the workload and any delegated mediation component | A node or remote helper establishes some controls behind a workload-local backend | Depends on which trusted components remain with the workload | Placement proposed (#2606) | -| **Driver-hosted/shared service** | With the compute driver or another trusted service; no in-sandbox supervisor process is required | May be co-located with the logical supervisor; one host may operate many isolated boundaries | Depends on the workload runtime | Placement proposed | - -## Durable rules - -- Every active boundary has one verified descriptor, one trusted - `SandboxContext`, and at most one logical supervisor, which may span multiple - coupled processes. -- Physical processes and listeners may be shared, but lifecycle state, policy, - binary identity, enforcement, and cleanup remain isolated per boundary. -- Moving a privileged component does not itself provide kernel separation. - -## Kernel relationships - -| Relationship | Meaning | -|---|---| -| **Shared host kernel** | The workload and the trusted components relied on for containment run on the host's kernel. | -| **Shared guest or application kernel** | The workload and those trusted components share one isolated kernel: a VM guest kernel or a userspace application kernel. | -| **Kernel-separated** | The trusted components relied on for containment run outside the workload's kernel. | - -## Status - -This matrix is non-normative. It illustrates implementations of RFC 0012; it -does not extend the contract. diff --git a/scripts/baseline_workflow_metrics.py b/scripts/baseline_workflow_metrics.py index ffd5054679..430fc93e23 100755 --- a/scripts/baseline_workflow_metrics.py +++ b/scripts/baseline_workflow_metrics.py @@ -43,7 +43,7 @@ "branch-e2e.yml", "build-cli-binaries.yml", "build-gateway-binaries.yml", - "build-sandbox-binaries.yml", + "build-runtime-binaries.yml", "ci-image.yml", "docker-build.yml", "e2e-docker-test.yml", @@ -63,7 +63,7 @@ REUSABLE_WORKFLOWS: set[str] = { "build-cli-binaries.yml", "build-gateway-binaries.yml", - "build-sandbox-binaries.yml", + "build-runtime-binaries.yml", "docker-build.yml", "e2e-docker-test.yml", "e2e-podman-test.yml", diff --git a/sdk/go/proto/openshellv1/openshell.pb.go b/sdk/go/proto/openshellv1/openshell.pb.go index 735c17bc88..78e44d3987 100644 --- a/sdk/go/proto/openshellv1/openshell.pb.go +++ b/sdk/go/proto/openshellv1/openshell.pb.go @@ -665,8 +665,16 @@ type RefreshSandboxTokenResponse struct { // Fresh credentials for the requested, policy-authorized extension // services. These remain in supervisor memory and are never persisted. ExtensionCredentials []*ExtensionServiceCredential `protobuf:"bytes,3,rep,name=extension_credentials,json=extensionCredentials,proto3" json:"extension_credentials,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // Fresh Sandbox Protocol bearer token from the same atomic refresh. + SandboxToken string `protobuf:"bytes,4,opt,name=sandbox_token,json=sandboxToken,proto3" json:"sandbox_token,omitempty"` + // Absolute Sandbox Protocol token expiry, milliseconds since the epoch. + SandboxExpiresAtMs int64 `protobuf:"varint,5,opt,name=sandbox_expires_at_ms,json=sandboxExpiresAtMs,proto3" json:"sandbox_expires_at_ms,omitempty"` + // Launch generation to which both refreshed credentials are bound. + SessionId string `protobuf:"bytes,6,opt,name=session_id,json=sessionId,proto3" json:"session_id,omitempty"` + // Monotonic Sandbox Protocol connection replacement order. + CredentialEpoch uint64 `protobuf:"varint,7,opt,name=credential_epoch,json=credentialEpoch,proto3" json:"credential_epoch,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *RefreshSandboxTokenResponse) Reset() { @@ -720,6 +728,34 @@ func (x *RefreshSandboxTokenResponse) GetExtensionCredentials() []*ExtensionServ return nil } +func (x *RefreshSandboxTokenResponse) GetSandboxToken() string { + if x != nil { + return x.SandboxToken + } + return "" +} + +func (x *RefreshSandboxTokenResponse) GetSandboxExpiresAtMs() int64 { + if x != nil { + return x.SandboxExpiresAtMs + } + return 0 +} + +func (x *RefreshSandboxTokenResponse) GetSessionId() string { + if x != nil { + return x.SessionId + } + return "" +} + +func (x *RefreshSandboxTokenResponse) GetCredentialEpoch() uint64 { + if x != nil { + return x.CredentialEpoch + } + return 0 +} + // Health check request. type HealthRequest struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -14322,11 +14358,16 @@ const file_openshell_proto_rawDesc = "" + "\x05token\x18\x01 \x01(\tB\x04\x88\xb5\x18\x01R\x05token\x12\"\n" + "\rexpires_at_ms\x18\x02 \x01(\x03R\vexpiresAtMs\"T\n" + "\x1aRefreshSandboxTokenRequest\x126\n" + - "\x17extension_service_names\x18\x01 \x03(\tR\x15extensionServiceNames\"\xbc\x01\n" + + "\x17extension_service_names\x18\x01 \x03(\tR\x15extensionServiceNames\"\xe4\x02\n" + "\x1bRefreshSandboxTokenResponse\x12\x1a\n" + "\x05token\x18\x01 \x01(\tB\x04\x88\xb5\x18\x01R\x05token\x12\"\n" + "\rexpires_at_ms\x18\x02 \x01(\x03R\vexpiresAtMs\x12]\n" + - "\x15extension_credentials\x18\x03 \x03(\v2(.openshell.v1.ExtensionServiceCredentialR\x14extensionCredentials\"\x0f\n" + + "\x15extension_credentials\x18\x03 \x03(\v2(.openshell.v1.ExtensionServiceCredentialR\x14extensionCredentials\x12)\n" + + "\rsandbox_token\x18\x04 \x01(\tB\x04\x88\xb5\x18\x01R\fsandboxToken\x121\n" + + "\x15sandbox_expires_at_ms\x18\x05 \x01(\x03R\x12sandboxExpiresAtMs\x12\x1d\n" + + "\n" + + "session_id\x18\x06 \x01(\tR\tsessionId\x12)\n" + + "\x10credential_epoch\x18\a \x01(\x04R\x0fcredentialEpoch\"\x0f\n" + "\rHealthRequest\"_\n" + "\x0eHealthResponse\x123\n" + "\x06status\x18\x01 \x01(\x0e2\x1b.openshell.v1.ServiceStatusR\x06status\x12\x18\n" + diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 7530fc5c7c..7688a3351f 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -173,7 +173,7 @@ rationale, configured and effective modes, active generation, and the explicit docker info docker ps --filter name=openshell docker logs --tail=200 -docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version +docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-ghcr.io/nvidia/openshell/sandbox:latest}" --version openshell status ``` @@ -213,7 +213,7 @@ Common findings: - Sandbox fails before readiness with an OCI workspace validation error: inspect the image's `WorkingDir` using the immutable image ID reported by the gateway. Empty, `/`, and explicit `/sandbox` use the managed `/sandbox` compatibility workspace. Any other workdir must be an absolute normalized directory with no symlink components; the final policy UID, primary GID, and supplementary groups must pass the kernel's effective traverse/write checks, including POSIX ACL and LSM decisions. OpenShell does not create, chown, or chmod a non-default image workdir. - Docker also rejects an image `VOLUME` that covers the workdir or one of its parents because the runtime would mask the immutable path before validation. Move the `VOLUME` below the workspace or remove the declaration. - A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error. -- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`. +- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify the sibling binary next to `openshell-gateway`, or that the configured `sandbox_runtime_image` contains `/openshell-sandbox`. - Sandbox never registers: check gateway logs and supervisor callback endpoint. - On macOS, repeated `Policy fetch failed after 5 attempts` messages with a Homebrew gateway bound to `[::1]:17670` indicate that the Docker @@ -222,8 +222,10 @@ Common findings: `127.0.0.1:17670` primary listener, and reuse it for authenticated sandbox callbacks. On an older release, set `bind_address = "127.0.0.1:17670"` or upgrade. -- Supervisor image exits before printing `openshell-sandbox --version`: verify the configured supervisor image contains a static executable at `/openshell-sandbox`. -- A sandbox with explicit `protocol: tcp` endpoints fails before workload readiness: confirm the Docker or Podman driver supplied the `policy-dns-transparent-tcp` runtime capability and inspect supervisor logs for missing `nft`, synthetic-route overlap, or namespace-local DNS/TCP listener bind failures. Kubernetes, VM, sidecar, and out-of-tree drivers must reject this policy until they provide the complete substrate; use omitted protocol with an explicit proxy on those runtimes. +- Sandbox runtime image exits before printing `openshell-sandbox --version`: verify the configured image contains a static executable at `/openshell-sandbox`. +- A sandbox with explicit `protocol: tcp` endpoints fails before workload readiness: confirm the selected isolation backend advertises TCP mediation, then inspect the sandbox and supervisor logs for protected-channel setup or listener failures. A driver that cannot supply the required outer egress fence and authenticated runtime channel must reject the policy before starting the agent. +- Supervisor runtime validation fails: verify `supervisor_image` contains a static `/openshell-supervisor` executable from the same release as the sandbox runtime. +- The sandbox fails its enforcement probe: inspect the sandbox log for the exact nested seccomp user-notification, task-memory, Landlock, loopback DNS, or socket-injection check that failed. Do not add capabilities or switch to an unconfined seccomp profile; use a runtime whose default profile permits the unprivileged probe. - A GPU sandbox fails because Docker reports no discovered NVIDIA CDI devices: verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. Restart `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs, then restart or reload Docker and re-check `docker info`. During a graceful gateway restart, Docker, Podman, and VM sandboxes with @@ -251,10 +253,14 @@ Common findings: - Sandbox image missing or pull denied: verify image reference and registry credentials. - Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Numeric workload identities `1` through `4294967294` are accepted; root, the invalid identity sentinel, and missing identities are rejected. - Supervisor cannot call back: check callback endpoint and gateway logs. -- A sandbox with explicit `protocol: tcp` endpoints fails before readiness: - inspect supervisor logs for policy DNS port-53 binding, synthetic-route, or - nftables redirect failures. Rootless Podman must provide these primitives - inside the supervisor-owned nested network namespace; setup fails closed. +- Inspect both Podman containers for the sandbox: the `sandbox` isolation role + must have network mode `none`; the `supervisor` role owns gateway callbacks + and egress. Both run non-root with all capabilities dropped. Check the private + channel volume and shared user-namespace mapping if authentication fails. +- If a sandbox fails before readiness, inspect its unprivileged enforcement + probe and the companion supervisor's private health check. Do not add + capabilities, attach a workload network, or disable the runtime seccomp + profile. There is no sandbox nftables or nested-network setup to repair. - Gateway exits before becoming healthy with a callback-listener discovery error: inspect `podman info --debug`, the configured Podman network, and the host's IPv4 default route. Rootless pasta uses the private source address @@ -269,12 +275,12 @@ Common findings: When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`): -- Supervisor delivery uses bind-mount fallback instead of image volumes because +- Sandbox runtime delivery uses bind-mount fallback instead of image volumes because overlay mounts do not support `idmapped` mounts. The supervisor binary is - extracted from the supervisor image and cached at + extracted from the sandbox runtime image and cached at `$XDG_DATA_HOME/openshell/podman-supervisor/` (typically `~/.local/share/openshell/podman-supervisor/`). -- Stale cache: if the supervisor image is updated but the cached binary is not +- Stale cache: if the sandbox runtime image is updated but the cached binary is not refreshed, sandbox creation may fail with an ELF validation error or version mismatch. Remove the cache directory and retry. - `auto` mode requires subuid/subgid ranges for the current user in @@ -426,7 +432,7 @@ If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret. SPIRE is used by both the gateway and sandbox supervisors for dynamic provider token grants. The gateway pod must mount the `spiffe-workload-api` CSI volume -and set `OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`; sandbox pods must +and set `OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`; supervisor Pods must receive the matching Workload API socket from the Kubernetes driver config. The gateway verifies supervisor JWT-SVIDs from JWT bundles fetched through this Workload API socket, not from the SPIRE OIDC discovery endpoint. @@ -479,7 +485,7 @@ kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload' ``` -The gateway and supervisor images should use the same release tag. A stale supervisor image can make sandbox behavior lag behind gateway policy or protocol changes. +The gateway, sandbox, and supervisor images should use the same release tag. A stale runtime image can make sandbox behavior lag behind gateway policy or protocol changes. For vulnerability reports, record the running image digest and scan that exact artifact. The gateway includes a pinned Distroless base; the supervisor includes @@ -574,58 +580,49 @@ kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\ kubectl -n get sandbox -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}' ``` -If `topology = "sidecar"` is rendered under `[openshell.drivers.kubernetes]`, -sandbox pods should have an `openshell-network-init` init container running -`--mode=network-init`, an `agent` container running -`openshell-sandbox --mode=process`, and an `openshell-supervisor-network` -container running `--mode=network`. The init container owns nftables setup and -should be the only sidecar topology container with `NET_ADMIN`. It also needs -`CHOWN`/`FOWNER` to hand shared emptyDir state to the effective sidecar UID. The -default binary-aware network sidecar runs as UID 0 with primary GID -`sandbox_gid` and adds `SYS_PTRACE` plus `DAC_READ_SEARCH`. When -`process_binary_aware_network_policy = false`, it runs as the configured -non-root `proxy_uid` without those inspection capabilities. That dedicated -proxy UID must remain at least `1000` and must not match the workload UID -because the pod egress fence exempts its traffic. The pod `fsGroup` is set to -`sandbox_gid` in both modes. - -In sidecar topology only the network sidecar should mount the gateway bootstrap -credentials (`openshell-sa-token` and `openshell-client-tls`). The process -container should not receive `OPENSHELL_ENDPOINT`, gateway TLS env vars, the -sandbox token file, or those credential mounts. Instead, the network sidecar -serves policy and provider environment state over the Unix control socket from -`OPENSHELL_SIDECAR_CONTROL_SOCKET` (`/run/openshell-sidecar/control.sock` by -default). The process supervisor must be the first and only client. After -validating its peer UID, GID, and PID, the sidecar unlinks the listener. If the -connection later closes, the network sidecar exits non-zero so Kubernetes can -restart it with a fresh listener. If the process supervisor fails before -launching the workload, -inspect both containers for control-socket bind, connect, bootstrap, or update -errors. If new SSH/exec sessions do not pick up refreshed provider environment, -inspect the network sidecar settings-poll logs and the process container logs -for provider environment update handling; the process container should consume -newer provider-env revisions without receiving gateway credentials. - -The process container reports the workload entrypoint PID over the same control -socket, and the network sidecar uses that PID for binary-scoped policy -decisions through `/proc`. If rules with `policy.binaries` are unexpectedly -denied, inspect the sidecar control logs and confirm the pod has -`shareProcessNamespace: true`. -The shared state directory should preserve `sandbox_gid` inheritance -(`02775`). Sidecar SSH uses the Linux abstract socket -`@openshell-sidecar-ssh`; the network sidecar verifies its peer PID before -bridging gateway relay requests. No `ssh.sock` file should appear in the shared -state directory. -Inspect all three when sandbox registration or egress enforcement fails: - -```bash -kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*=' -kubectl -n get pod -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}' -kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' -kubectl -n logs -c openshell-network-init --tail=200 -kubectl -n logs -c openshell-supervisor-network --tail=200 -kubectl -n logs -c agent --tail=200 -``` +The Kubernetes driver creates a sandbox workload Pod and a separate, directly +managed supervisor Pod. Helm must render +`network_policy_enforced = true`. This is an explicit operator assertion that +the cluster CNI enforces Kubernetes NetworkPolicy; the Kubernetes API cannot +attest enforcement. Run sandboxes only in a trusted namespace +where tenants cannot create Pods, copy OpenShell role labels, or read the +bootstrap Secret. + +The workload Pod runs `/openshell-sandbox`. It has no gateway credentials and +no direct egress. One namespace-wide workload NetworkPolicy is created before +the suspended Sandbox resource. It denies all workload egress and allows +supervisor Pods to reach sandbox TLS listeners. The driver then creates a +per-sandbox Service, split immutable bootstrap Secrets, and a gated supervisor +Pod before releasing either Pod. The supervisor Pod runs +`/openshell-supervisor`. Both Pods use the +same resolved non-root identity, request no capabilities, drop `ALL`, disable +privilege escalation, and use `RuntimeDefault` seccomp. The supervisor reaches +the sandbox over per-sandbox TLS with server-certificate verification plus +bootstrap-token client authentication, and owns gateway policy, provider +credentials, DNS, and mediated upstream connections. + +Inspect all driver-managed resources when a Kubernetes sandbox remains Starting +or loses readiness: + +```bash +kubectl -n get sandbox,pod,service,secret -l openshell.ai/sandbox-id= +kubectl -n get networkpolicy openshell-sandbox-workloads -o yaml +kubectl -n describe pod -l openshell.ai/sandbox-id=,openshell.ai/boundary-role=supervisor +kubectl -n logs pod/ --tail=200 +kubectl -n get pod -l openshell.ai/sandbox-id=,openshell.ai/boundary-role=workload -o yaml +kubectl -n get networkpolicy -l openshell.ai/sandbox-id= -o yaml +``` + +Creation and recovery fail closed. A missing Secret leaves both pods inert; a +missing or unobserved workload fence must prevent the driver from releasing the +Sandbox; and readiness requires both Agent Sandbox readiness and an Available +supervisor Pod. Its exec readiness check succeeds only after the +supervisor has attached, confirmed enforcement, started or resumed the +workload, and registered the gateway access plane. Use both Pod logs for +bootstrap errors. An `EPERM` during enforcement setup means the runtime blocked +a required unprivileged seccomp, task-memory, or Landlock operation. Do not add +capabilities, gateway egress, or credentials to the workload Pod as a +workaround. #### Corporate upstream proxy @@ -642,26 +639,21 @@ helm -n openshell get values openshell | grep -A8 upstreamProxy ``` Only `http://host:port` forward proxies are supported; `https://` proxy URLs and -plain-HTTP egress are out of scope and rejected. Proxy credentials require -`topology = "sidecar"` — combined topology shares the credential mount with the -workload, so the gateway rejects credentials there. The credential Secret named -by `proxy_auth_secret_name` must exist in the sandbox namespace with the key -named by `proxy_auth_secret_key`, and Kubernetes will not create keys longer -than 253 bytes or named `.`/`..`. - -The proxy arguments and credential mount are injected only into the container -that runs network supervision (the `agent` container in combined topology, the -`openshell-supervisor-network` sidecar in sidecar topology). The one-shot -`openshell-network-init` container and the process `agent` container in sidecar -topology must never receive them. The credential is projected read-only as the -`openshell-upstream-proxy-auth` volume at `/run/openshell/upstream-proxy-auth` -and passed as `--upstream-proxy-auth-file`; it must never appear in env, -annotations, or command arguments. +plain-HTTP egress are out of scope and rejected. The credential Secret named by +`proxy_auth_secret_name` must exist in the sandbox namespace with the key named +by `proxy_auth_secret_key`, and Kubernetes will not create keys longer than 253 +bytes or named `.`/`..`. + +The proxy arguments and credential mount belong only to the separate supervisor +Pod. The workload Pod must never receive them. The credential is projected +read-only as `openshell-upstream-proxy-auth` at +`/run/openshell/upstream-proxy-auth` and passed by file path; it must never +appear in environment variables, annotations, or command arguments. ```bash kubectl -n get secret -o jsonpath='{.data}' >/dev/null && echo "secret present" -kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' | grep -- '--upstream-' -kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{": "}{range .volumeMounts[*]}{.name}{" "}{end}{"\n"}{end}' | grep upstream-proxy-auth +kubectl -n get pod -o jsonpath='{.spec.containers[0].command}' | grep -- '--upstream-' +kubectl -n get pod -o jsonpath='{.spec.containers[0].volumeMounts}' | grep upstream-proxy-auth kubectl -n get events --sort-by=.lastTimestamp | grep -Ei 'secret|MountVolume' | tail -n 20 ``` @@ -674,7 +666,7 @@ destination that should be direct is missing from `no_proxy`. Inspect the network supervisor logs for CONNECT and upstream-proxy decisions: ```bash -kubectl -n logs -c openshell-supervisor-network --tail=200 | grep -Ei 'upstream|connect|proxy' +kubectl -n logs pod/ --tail=200 | grep -Ei 'upstream|connect|proxy' ``` ### Step 7: Check VM-Backed Gateways @@ -715,41 +707,12 @@ grep -A20 '^\[openshell.drivers.vm\]' | grep -E 'https_proxy|no_p ps -o args= -p "$(pgrep -f openshell-driver-vm | head -n1)" | tr ' ' '\n' | grep -A1 -- '--upstream-proxy\|--upstream-no-proxy' ``` -Reachability is the most common failure, and it depends on the VM backend. -On libkrun (non-GPU sandboxes) guest egress leaves through gvproxy, so a proxy -bound to the gateway host's loopback is **not** reachable at `127.0.0.1` from -inside the guest: it must be addressed as -`http://host.openshell.internal:`, which gvproxy NATs from -`192.168.127.254` to the host's `127.0.0.1`. A `https_proxy` pointing at a -loopback URL produces policy-approved CONNECT attempts that time out while -public destinations still work. - -GPU sandboxes run on QEMU/TAP, where no gateway-host proxy is reachable at -all: `host.openshell.internal` resolves to the TAP host address, and the -driver's nftables `input` chain accepts only the gateway port from the guest. -The driver rejects such a configuration at launch — a create failing with -`https_proxy ... addresses the gateway host, which a QEMU/TAP sandbox ... -cannot reach` means the proxy must move to an address routable from the -guest's masqueraded egress (or the sandbox must run without a GPU). - -The settings reach the supervisor through a driver-written argument file in -the per-sandbox overlay, not through the guest environment. The credential and -CA bundle are staged into the same overlay at fixed guest paths. Inspect the -guest side from the VM console log, which records how many driver-supplied -arguments the init script read: - -```bash -grep -E 'supervisor arguments from driver|supervisor argument list' /sandboxes//rootfs-console.log -grep -Ei 'upstream|connect|proxy' /sandboxes//rootfs-console.log | tail -n 40 -``` - -`FATAL: supervisor argument list ... is not readable` or `FATAL: empty entry in -supervisor argument list` means the overlay is broken or was tampered with, and -the guest deliberately aborts rather than starting a supervisor with a -truncated egress configuration. If the guest logs no driver arguments at all -while `gateway.toml` sets `https_proxy`, the running driver predates the -configuration — check that the gateway spawned the driver binary you expect -(`[openshell.drivers.vm].driver_dir`). +Both libkrun and QEMU guests are NIC-less. Proxy settings, credentials, and +private CA material stay with the host `openshell-supervisor`. For a proxy on +the gateway host, use `http://host.openshell.internal:`; the supervisor +normalizes that name to host loopback. Inspect `supervisor.log` and +`supervisor.err.log` under the sandbox state directory for connection or +credential failures. ## Common Failure Patterns @@ -758,7 +721,7 @@ configuration — check that the gateway spawned the driver binary you expect | `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs | | `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart `gateway:docker`, `gateway:podman`, or `gateway:vm` so it re-detects the listener; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` | | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs | -| Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route | +| Gateway exits while resolving compute-driver listener requirements | The callback hostname is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route | | Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected | | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs | | Docker GPU sandbox fails before startup | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` | diff --git a/tasks/docker.toml b/tasks/docker.toml index 502b2363c5..ec585c2de9 100644 --- a/tasks/docker.toml +++ b/tasks/docker.toml @@ -7,6 +7,7 @@ description = "Build all Docker images" depends = [ "build:docker:gateway", + "build:docker:sandbox", "build:docker:supervisor", ] hide = true @@ -26,6 +27,11 @@ description = "Build the gateway Docker image" run = "tasks/scripts/docker-build-image.sh gateway" hide = true +["build:docker:sandbox"] +description = "Build the sandbox runtime image" +run = "tasks/scripts/docker-build-image.sh sandbox" +hide = true + ["build:docker:supervisor"] description = "Build the supervisor image" run = "tasks/scripts/docker-build-image.sh supervisor" @@ -36,13 +42,18 @@ description = "Alias for build:docker:gateway" depends = ["build:docker:gateway"] hide = true +["docker:build:sandbox"] +description = "Alias for build:docker:sandbox" +depends = ["build:docker:sandbox"] +hide = true + ["docker:build:supervisor"] description = "Alias for build:docker:supervisor" depends = ["build:docker:supervisor"] hide = true ["build:docker:multiarch"] -description = "Build multi-arch gateway and supervisor images and push to a registry" +description = "Build multi-arch gateway, sandbox, and supervisor images and push to a registry" run = "tasks/scripts/docker-publish-multiarch.sh" hide = true diff --git a/tasks/helm.toml b/tasks/helm.toml index 525e9c2e51..33b3550dff 100644 --- a/tasks/helm.toml +++ b/tasks/helm.toml @@ -39,12 +39,12 @@ run = """ helm dependency build deploy/helm/openshell echo "--- helm lint: defaults ---" echo "values files: deploy/helm/openshell/values.yaml" - helm lint deploy/helm/openshell --set agentSandbox.preflight.enabled=false + helm lint deploy/helm/openshell --set agentSandbox.preflight.enabled=false --set supervisor.sandboxRuntime.networkPolicyEnforced=true for f in deploy/helm/openshell/ci/values-*.yaml; do variant=$(basename "$f" .yaml | sed 's/values-//') echo "--- helm lint: $variant ---" echo "values files: deploy/helm/openshell/values.yaml, $f" - helm lint deploy/helm/openshell -f "$f" --set agentSandbox.preflight.enabled=false + helm lint deploy/helm/openshell -f "$f" --set agentSandbox.preflight.enabled=false --set supervisor.sandboxRuntime.networkPolicyEnforced=true done echo "--- helm lint: workspace defaults ---" helm lint deploy/helm/openshell-workspace @@ -71,46 +71,16 @@ description = "Run skaffold dev for deploy/helm/openshell (iterative deploy)" dir = "deploy/helm/openshell" run = "skaffold dev" -["helm:skaffold:dev:sidecar"] -description = "Run skaffold dev with the Kubernetes supervisor sidecar topology" -dir = "deploy/helm/openshell" -run = "skaffold dev -p sidecar" - -["helm:skaffold:dev:sidecar-mtls"] -description = "Run skaffold dev with the Kubernetes supervisor sidecar topology and TLS/mTLS enabled" -dir = "deploy/helm/openshell" -run = "skaffold dev -p sidecar-mtls" - ["helm:skaffold:run"] description = "Run a one-shot Skaffold deploy and register its local plaintext gateway" dir = "deploy/helm/openshell" run = "skaffold run && ../../../tasks/scripts/helm-k3s-local.sh register" -["helm:skaffold:run:sidecar"] -description = "Run a sidecar Skaffold deploy and register its local plaintext gateway" -dir = "deploy/helm/openshell" -run = "skaffold run -p sidecar && ../../../tasks/scripts/helm-k3s-local.sh register" - -["helm:skaffold:run:sidecar-mtls"] -description = "Run skaffold run with the Kubernetes supervisor sidecar topology and TLS/mTLS enabled" -dir = "deploy/helm/openshell" -run = "skaffold run -p sidecar-mtls" - ["helm:skaffold:delete"] description = "Run skaffold delete for deploy/helm/openshell" dir = "deploy/helm/openshell" run = "skaffold delete" -["helm:skaffold:delete:sidecar"] -description = "Run skaffold delete for the Kubernetes supervisor sidecar topology" -dir = "deploy/helm/openshell" -run = "skaffold delete -p sidecar" - -["helm:skaffold:delete:sidecar-mtls"] -description = "Run skaffold delete for the Kubernetes supervisor sidecar topology with TLS/mTLS enabled" -dir = "deploy/helm/openshell" -run = "skaffold delete -p sidecar-mtls" - ["helm:skaffold:diagnose"] description = "Run skaffold diagnose for deploy/helm/openshell" dir = "deploy/helm/openshell" diff --git a/tasks/rust.toml b/tasks/rust.toml index 548e302f12..742b882316 100644 --- a/tasks/rust.toml +++ b/tasks/rust.toml @@ -3,10 +3,17 @@ # Rust check, lint, and format tasks +["perf:seccomp"] +description = "Compare native and sandbox-filtered TCP socket performance" +run = "cargo run --release -p openshell-sandbox --features perf-harness --bin openshell-seccomp-perf --" + ["rust:check"] description = "Check all Rust crates for errors" depends = ["rust:lockfiles:check"] -run = "cargo check --workspace" +run = [ + "cargo check --workspace", + "cargo check -p openshell-sandbox --all-targets --features perf-harness", +] run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 check native" hide = true @@ -21,6 +28,7 @@ description = "Lint Rust code with Clippy (deny warnings)" depends = ["rust:lockfiles:check"] run = [ "cargo clippy --workspace --all-targets -- -D warnings", + "cargo clippy -p openshell-sandbox --all-targets --features perf-harness -- -D warnings", "cargo clippy --manifest-path e2e/rust/Cargo.toml --all-targets -- -D warnings", "cargo clippy --manifest-path examples/governance-interceptor/Cargo.toml --all-targets -- -D warnings", "cargo clippy --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all-targets -- -D warnings", @@ -69,8 +77,8 @@ run = [ # operators to produce these artifacts. "cargo build -p openshell-gateway --bin openshell-gateway --no-default-features --features defaults-without-telemetry", "tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-gateway", - "cargo build -p openshell-sandbox --bin openshell-sandbox --no-default-features --features defaults-without-telemetry", - "tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-sandbox", + "cargo build -p openshell-supervisor --bin openshell-supervisor --no-default-features --features defaults-without-telemetry", + "tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-supervisor", ] ["rust:verify:defaults-without-telemetry"] @@ -80,11 +88,11 @@ run = "tasks/scripts/verify-defaults-without-telemetry.sh" ["rust:verify:system-ca-roots"] description = "Verify system CA roots build mode compiles and excludes bundled Mozilla root crates" run = [ - # Check that the sandbox compiles cleanly in system CA roots mode (all + # Check that the supervisor compiles cleanly in system CA roots mode (all # defaults except bundled-ca-roots). - "cargo check -p openshell-sandbox --all-targets --no-default-features --features system-ca-roots", + "cargo check -p openshell-supervisor --all-targets --no-default-features --features system-ca-roots", # Guard: webpki-roots must not appear in the dependency graph. - "bash -c 'if cargo tree -p openshell-sandbox -i webpki-roots --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-roots; then echo \"ERROR: webpki-roots found in system CA roots build\" >&2; exit 1; fi'", + "bash -c 'if cargo tree -p openshell-supervisor -i webpki-roots --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-roots; then echo \"ERROR: webpki-roots found in system CA roots build\" >&2; exit 1; fi'", # Guard: webpki-root-certs must not appear either (webpki-roots re-exports it). - "bash -c 'if cargo tree -p openshell-sandbox -i webpki-root-certs --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-root-certs; then echo \"ERROR: webpki-root-certs found in system CA roots build\" >&2; exit 1; fi'", + "bash -c 'if cargo tree -p openshell-supervisor -i webpki-root-certs --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-root-certs; then echo \"ERROR: webpki-root-certs found in system CA roots build\" >&2; exit 1; fi'", ] diff --git a/tasks/scripts/docker-build-image.sh b/tasks/scripts/docker-build-image.sh index 08ba00e066..06102fc649 100755 --- a/tasks/scripts/docker-build-image.sh +++ b/tasks/scripts/docker-build-image.sh @@ -43,9 +43,12 @@ required_prebuilt_binaries() { gateway) echo "openshell-gateway" ;; - supervisor|supervisor-sideload|supervisor-output) + sandbox) echo "openshell-sandbox" ;; + supervisor|supervisor-sideload|supervisor-output) + echo "openshell-supervisor" + ;; esac } @@ -92,7 +95,7 @@ ensure_prebuilt_binaries() { fi } -TARGET=${1:?"Usage: docker-build-image.sh [extra-args...]"} +TARGET=${1:?"Usage: docker-build-image.sh [extra-args...]"} shift IS_FINAL_IMAGE=0 @@ -106,6 +109,12 @@ case "${TARGET}" in DOCKER_TARGET="gateway" DOCKERFILE="deploy/docker/Dockerfile.gateway" ;; + sandbox) + IS_FINAL_IMAGE=1 + IMAGE_NAME="openshell/sandbox" + DOCKER_TARGET="sandbox" + DOCKERFILE="deploy/docker/Dockerfile.sandbox" + ;; supervisor) IS_FINAL_IMAGE=1 IMAGE_NAME="openshell/supervisor" diff --git a/tasks/scripts/docker-publish-multiarch.sh b/tasks/scripts/docker-publish-multiarch.sh index 2f23c5106f..430bfa2b7d 100755 --- a/tasks/scripts/docker-publish-multiarch.sh +++ b/tasks/scripts/docker-publish-multiarch.sh @@ -3,7 +3,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Build multi-arch gateway + supervisor images and push to a container registry. +# Build multi-arch gateway, sandbox, and supervisor images and push them. # Requires DOCKER_REGISTRY to be set (e.g. ghcr.io/myorg). set -euo pipefail @@ -46,6 +46,10 @@ _publish_multiarch_docker() { echo "Building multi-arch gateway image..." tasks/scripts/docker-build-image.sh gateway + echo + echo "Building multi-arch sandbox image..." + tasks/scripts/docker-build-image.sh sandbox + echo echo "Building multi-arch supervisor image..." tasks/scripts/docker-build-image.sh supervisor @@ -56,7 +60,7 @@ _publish_multiarch_docker() { fi if [[ ${#TAGS_TO_APPLY[@]} -gt 0 ]]; then - for component in gateway supervisor; do + for component in gateway sandbox supervisor; do full_image="${REGISTRY}/${component}" for tag in "${TAGS_TO_APPLY[@]}"; do [[ "${tag}" == "${IMAGE_TAG}" ]] && continue @@ -79,7 +83,7 @@ _publish_multiarch_podman() { # Split comma-separated platforms into an array. IFS=',' read -ra PLATFORM_LIST <<< "${PLATFORMS}" - for component in gateway supervisor; do + for component in gateway sandbox supervisor; do local full_image="${REGISTRY}/${component}" local manifest_name="${full_image}:${IMAGE_TAG}" @@ -145,6 +149,7 @@ fi echo echo "Done! Multi-arch images pushed to ${REGISTRY}:" echo " ${REGISTRY}/gateway:${IMAGE_TAG}" +echo " ${REGISTRY}/sandbox:${IMAGE_TAG}" echo " ${REGISTRY}/supervisor:${IMAGE_TAG}" if [[ "${TAG_LATEST}" == "true" ]]; then echo " (all also tagged :latest)" diff --git a/tasks/scripts/gateway-docker.sh b/tasks/scripts/gateway-docker.sh index 8771728a89..0b3edca729 100644 --- a/tasks/scripts/gateway-docker.sh +++ b/tasks/scripts/gateway-docker.sh @@ -16,6 +16,7 @@ # OPENSHELL_DOCKER_GATEWAY_NAME=my-docker-gateway mise run gateway:docker # OPENSHELL_SANDBOX_NAMESPACE=my-ns mise run gateway:docker # OPENSHELL_SANDBOX_IMAGE=ghcr.io/... mise run gateway:docker +# OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/... mise run gateway:docker # # After the gateway is running, point the CLI at it with either: # openshell --gateway docker-dev @@ -33,6 +34,7 @@ GATEWAY_NAME="${OPENSHELL_DOCKER_GATEWAY_NAME:-docker-dev}" STATE_DIR="${OPENSHELL_DOCKER_GATEWAY_STATE_DIR:-${ROOT}/.cache/gateway-docker}" SANDBOX_NAMESPACE="${OPENSHELL_SANDBOX_NAMESPACE:-docker-dev}" SANDBOX_IMAGE="${OPENSHELL_SANDBOX_IMAGE:-ghcr.io/nvidia/openshell-community/sandboxes/base:latest}" +SUPERVISOR_IMAGE="${OPENSHELL_SUPERVISOR_IMAGE:-openshell/supervisor:dev}" SANDBOX_IMAGE_PULL_POLICY="$(normalize_image_pull_policy "${OPENSHELL_SANDBOX_IMAGE_PULL_POLICY:-if_not_present}")" LOG_LEVEL="${OPENSHELL_LOG_LEVEL:-info}" GATEWAY_BIN="${ROOT}/target/debug/openshell-gateway" @@ -124,6 +126,13 @@ if port_is_in_use "${PORT}"; then exit 2 fi +if [[ -z "${OPENSHELL_SUPERVISOR_IMAGE:-}" ]]; then + # Keep the default development image aligned with this checkout. Cargo and + # BuildKit caches make an unchanged rebuild incremental. + echo "Building local supervisor image (${SUPERVISOR_IMAGE})..." + CONTAINER_ENGINE=docker IMAGE_TAG=dev mise run build:docker:supervisor +fi + GRPC_ENDPOINT="${OPENSHELL_GRPC_ENDPOINT:-http://host.openshell.internal:${PORT}}" DAEMON_ARCH="$(normalize_arch "$(docker info --format '{{.Architecture}}' 2>/dev/null || true)")" @@ -233,6 +242,7 @@ gateway_id = "${GATEWAY_NAME}" [openshell.drivers.docker] default_image = "${SANDBOX_IMAGE}" +supervisor_image = "${SUPERVISOR_IMAGE}" image_pull_policy = "${SANDBOX_IMAGE_PULL_POLICY}" sandbox_label = "${SANDBOX_NAMESPACE}" grpc_endpoint = "${GRPC_ENDPOINT}" diff --git a/tasks/scripts/gateway-podman.sh b/tasks/scripts/gateway-podman.sh index ccf30ca104..a45c1d35d4 100644 --- a/tasks/scripts/gateway-podman.sh +++ b/tasks/scripts/gateway-podman.sh @@ -17,6 +17,8 @@ # OPENSHELL_PODMAN_GATEWAY_NAME=my-podman-gateway mise run gateway:podman # OPENSHELL_SANDBOX_NAMESPACE=my-ns mise run gateway:podman # OPENSHELL_SANDBOX_IMAGE=ghcr.io/... mise run gateway:podman +# OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/... mise run gateway:podman +# OPENSHELL_SANDBOX_RUNTIME_IMAGE=ghcr.io/... mise run gateway:podman set -euo pipefail @@ -69,27 +71,30 @@ require_podman_service() { fi } -ensure_podman_supervisor_image() { - local supervisor_image=$1 +ensure_podman_runtime_image() { + local image=$1 + local configured_image=$2 + local build_target=$3 + local role=$4 - if [[ -n "${OPENSHELL_SUPERVISOR_IMAGE:-}" ]]; then - if podman image exists "${supervisor_image}" >/dev/null 2>&1; then + if [[ -n "${configured_image}" ]]; then + if podman image exists "${image}" >/dev/null 2>&1; then return fi - echo "ERROR: supervisor image '${supervisor_image}' not found locally." >&2 - echo " Build it with Podman or unset OPENSHELL_SUPERVISOR_IMAGE to build openshell/supervisor:dev." >&2 + echo "ERROR: ${role} image '${image}' not found locally." >&2 + echo " Build it with Podman or unset its image override to build the local :dev image." >&2 exit 1 fi # Always run the build pipeline for the default development image so source - # changes cannot leave the fixed :dev tag pointing at a stale supervisor. + # changes cannot leave the fixed :dev tag pointing at a stale runtime. # Cargo and BuildKit caches keep unchanged rebuilds incremental. - echo "Refreshing Podman supervisor sideload image (${supervisor_image})..." + echo "Refreshing Podman ${role} image (${image})..." require_mise - CONTAINER_ENGINE=podman IMAGE_TAG=dev mise run build:docker:supervisor + CONTAINER_ENGINE=podman IMAGE_TAG=dev mise run "build:docker:${build_target}" - if ! podman image exists "${supervisor_image}" >/dev/null 2>&1; then - echo "ERROR: expected supervisor image '${supervisor_image}' after build" >&2 + if ! podman image exists "${image}" >/dev/null 2>&1; then + echo "ERROR: expected ${role} image '${image}' after build" >&2 exit 1 fi } @@ -166,7 +171,17 @@ if port_is_in_use "${PORT}"; then fi SUPERVISOR_IMAGE="${OPENSHELL_SUPERVISOR_IMAGE:-openshell/supervisor:dev}" -ensure_podman_supervisor_image "${SUPERVISOR_IMAGE}" +SANDBOX_RUNTIME_IMAGE="${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-openshell/sandbox:dev}" +ensure_podman_runtime_image \ + "${SUPERVISOR_IMAGE}" \ + "${OPENSHELL_SUPERVISOR_IMAGE:-}" \ + supervisor \ + supervisor +ensure_podman_runtime_image \ + "${SANDBOX_RUNTIME_IMAGE}" \ + "${OPENSHELL_SANDBOX_RUNTIME_IMAGE:-}" \ + sandbox \ + "sandbox runtime" export OPENSHELL_SUPERVISOR_IMAGE="${SUPERVISOR_IMAGE}" echo "Building openshell-gateway..." @@ -213,6 +228,7 @@ ttl_secs = 3600 [openshell.drivers.podman] default_image = "${SANDBOX_IMAGE}" supervisor_image = "${SUPERVISOR_IMAGE}" +sandbox_runtime_image = "${SANDBOX_RUNTIME_IMAGE}" image_pull_policy = "${SANDBOX_IMAGE_PULL_POLICY}" # Local development requires supervisor mount setup that Podman's runtime # profile may deny. Production configs preserve Podman's default when omitted. diff --git a/tasks/scripts/gateway-vm.sh b/tasks/scripts/gateway-vm.sh index 7c38541e77..ff3f1e7dff 100755 --- a/tasks/scripts/gateway-vm.sh +++ b/tasks/scripts/gateway-vm.sh @@ -213,7 +213,7 @@ check_supervisor_cross_toolchain() { fi local missing=0 if ! command -v cargo-zigbuild >/dev/null 2>&1; then - echo "ERROR: cargo-zigbuild not found (required to cross-compile the guest supervisor)." >&2 + echo "ERROR: cargo-zigbuild not found (required to cross-compile the guest sandbox)." >&2 echo " Install: cargo install --locked cargo-zigbuild && brew install zig" >&2 missing=1 fi @@ -291,16 +291,15 @@ VM_DRIVER_STATE_DIR="${OPENSHELL_VM_DRIVER_STATE_DIR:-${VM_DRIVER_STATE_DIR_DEFA DISABLE_TLS="$(normalize_bool "${OPENSHELL_DISABLE_TLS:-true}")" -# Build prerequisites: VM runtime artifacts + bundled supervisor. +# Build prerequisites: VM runtime artifacts + bundled sandbox/supervisor. if [ ! -d "${COMPRESSED_DIR}" ] \ || ! find "${COMPRESSED_DIR}" -maxdepth 1 -name 'libkrun*.zst' | grep -q . \ - || [ ! -f "${COMPRESSED_DIR}/gvproxy.zst" ] \ || [ ! -f "${COMPRESSED_DIR}/umoci.zst" ]; then echo "==> Preparing embedded VM runtime (mise run vm:setup)" mise run vm:setup fi -if [ ! -f "${COMPRESSED_DIR}/openshell-sandbox.zst" ]; then +if [ ! -f "${COMPRESSED_DIR}/openshell-sandbox.zst" ] || [ ! -f "${COMPRESSED_DIR}/openshell-supervisor.zst" ]; then check_supervisor_cross_toolchain echo "==> Building bundled VM supervisor (mise run vm:supervisor)" mise run vm:supervisor @@ -313,9 +312,9 @@ if [[ -n "${CARGO_BUILD_JOBS:-}" ]]; then CARGO_BUILD_JOBS_ARG=(-j "${CARGO_BUILD_JOBS}") fi -echo "==> Building openshell-gateway and openshell-driver-vm" +echo "==> Building openshell-gateway, openshell-driver-vm, and native control supervisor" cargo build ${CARGO_BUILD_JOBS_ARG[@]+"${CARGO_BUILD_JOBS_ARG[@]}"} \ - -p openshell-gateway -p openshell-driver-vm + -p openshell-gateway -p openshell-driver-vm -p openshell-supervisor if [ "$(uname -s)" = "Darwin" ]; then echo "==> Codesigning openshell-driver-vm (Hypervisor entitlement)" diff --git a/tasks/scripts/stage-prebuilt-binaries.sh b/tasks/scripts/stage-prebuilt-binaries.sh index 9687e24fd5..ca03b42612 100755 --- a/tasks/scripts/stage-prebuilt-binaries.sh +++ b/tasks/scripts/stage-prebuilt-binaries.sh @@ -46,25 +46,6 @@ target_triple() { esac } -# Resolve the supervisor libc variant. Both options produce a fully static -# binary because the supervisor is executed from inside arbitrary sandbox -# images; see verify-static-binary.sh. -# -# Scope: this selects the libc for the supervisor *image* binary. The VM driver -# bundles its own supervisor build (tasks/scripts/vm/build-supervisor-bundle.sh) -# and is not affected by this setting. -supervisor_libc() { - local selection=${SUPERVISOR_LIBC:-musl} - case "$selection" in - musl) echo "musl" ;; - glibc-static) echo "gnu-static" ;; - *) - echo "unsupported SUPERVISOR_LIBC: ${selection} (expected musl or glibc-static)" >&2 - exit 1 - ;; - esac -} - host_arch() { normalize_arch "$(uname -m)" } @@ -108,11 +89,14 @@ components_for_target() { gateway) echo "gateway" ;; - sandbox|supervisor|supervisor-output) + sandbox) + echo "sandbox" + ;; + supervisor|supervisor-output) echo "supervisor" ;; all) - echo "gateway supervisor" + echo "gateway sandbox supervisor" ;; *) usage @@ -128,10 +112,15 @@ resolve_component() { binary=openshell-gateway target_libc=gnu ;; - supervisor) + sandbox) crate=openshell-sandbox binary=openshell-sandbox - target_libc=$(supervisor_libc) + target_libc=musl + ;; + supervisor) + crate=openshell-supervisor + binary=openshell-supervisor + target_libc=gnu ;; *) echo "unsupported binary component: $1" >&2 @@ -191,7 +180,7 @@ build_component_for_arch() { build_target="$target" build_rustflags="${RUSTFLAGS:-}" - if [[ "$component" == "gateway" ]]; then + if [[ "$target_libc" == "gnu" ]]; then if has_cargo_zigbuild; then cargo_subcommand=(cargo zigbuild) build_target="${target}.2.28" @@ -199,20 +188,6 @@ build_component_for_arch() { echo "Error: cargo-zigbuild + zig are required to build ${binary} with the glibc 2.28 floor." >&2 exit 1 fi - elif [[ "$target_libc" == "gnu-static" ]]; then - # `zig cc` accepts `-static` for *-linux-gnu and emits a dynamically linked - # binary anyway, so cargo-zigbuild cannot produce this variant and there is - # no cross-compile fallback. Require a native toolchain that can link glibc - # statically (Fedora/RHEL: glibc-static, Debian/Ubuntu: libc6-dev). - build_rustflags="${build_rustflags} -C target-feature=+crt-static" - if [[ "$current_host_os" != "Linux" || "$current_host_arch" != "$arch" ]]; then - echo "Error: SUPERVISOR_LIBC=glibc-static cannot build ${binary} for linux/${arch} on ${current_host_os}/${current_host_arch}." >&2 - echo "cargo-zigbuild cannot statically link glibc, so this variant has no cross-compile path." >&2 - echo "Build on a linux/${arch} host with glibc static libraries installed, use SUPERVISOR_LIBC=musl," >&2 - echo "or provide prebuilt binaries in:" >&2 - echo " deploy/docker/.build/prebuilt-binaries/${arch}/" >&2 - exit 1 - fi elif [[ "$target_libc" == "musl" ]] && has_cargo_zigbuild; then cargo_subcommand=(cargo zigbuild) elif [[ "$current_host_os" != "Linux" || "$current_host_arch" != "$arch" ]]; then @@ -248,7 +223,7 @@ build_component_for_arch() { ( cd "$ROOT" - if [[ "$component" == "gateway" ]]; then + if [[ "$target_libc" == "gnu" ]]; then eval "$("$SCRIPT_DIR/setup-zig-cc-wrapper.sh" "$build_target" "$build_target" "$ROOT/target/zig-gnu-wrapper/$arch")" fi if [[ -n "${OPENSHELL_CARGO_VERSION:-}" ]]; then @@ -262,9 +237,9 @@ build_component_for_arch() { cargo_output_dir="$(cd "$ROOT" && mise x -- cargo metadata --format-version=1 --no-deps | jq -er '.target_directory')" binary_path="${cargo_output_dir}/${target}/release/${binary}" - if [[ "$component" == "gateway" ]]; then + if [[ "$target_libc" == "gnu" ]]; then "$SCRIPT_DIR/verify-glibc-symbols.sh" 2.28 "$binary_path" - elif [[ "$component" == "supervisor" ]]; then + else "$SCRIPT_DIR/verify-static-binary.sh" "$binary_path" fi diff --git a/tasks/scripts/trivy-scan-test.sh b/tasks/scripts/trivy-scan-test.sh index 8ce49e544b..764a8d1a62 100755 --- a/tasks/scripts/trivy-scan-test.sh +++ b/tasks/scripts/trivy-scan-test.sh @@ -212,10 +212,10 @@ expect_status 0 "scan selected profiles and retain distinct packaged chart versi --chart-ref oci://registry-a.example/charts/helm-chart:1.1.0 \ --chart-ref oci://registry-b.example/charts/helm-chart:1.0.0 jq -se ' - length == 21 + length == 20 and ([.[] | select(.[-1] == "deploy")] | length) == 1 and ([.[] | select(.[-1] == "deploy/helm")] | length) == 1 - and ([.[] | select(.[-1] == "deploy/helm/openshell")] | length) == 16 + and ([.[] | select(.[-1] == "deploy/helm/openshell")] | length) == 15 and all(.[]; (join(" ") | contains("values-spire-stack.yaml")) | not) and all(.[]; index("UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL") != null) ' "${TRIVY_TEST_CALLS}" >/dev/null diff --git a/tasks/scripts/trivy-scan.sh b/tasks/scripts/trivy-scan.sh index 3dd58ebdce..78862a5574 100755 --- a/tasks/scripts/trivy-scan.sh +++ b/tasks/scripts/trivy-scan.sh @@ -33,7 +33,7 @@ HELM_PROFILES=( cert-manager credential-driver-kubernetes-secrets credential-driver-vault gateway gateway-tls high-availability openshift-route-cert-manager spire tls-disabled workspace-managed workspace-operator - corporate-proxy-e2e keycloak sidecar sidecar-kata skaffold + corporate-proxy-e2e keycloak sandbox-runtime skaffold ) # Reject ignore entries broader than one concrete basename. diff --git a/tasks/scripts/verify-defaults-without-telemetry.sh b/tasks/scripts/verify-defaults-without-telemetry.sh index 1fd7e67dff..104523b04b 100755 --- a/tasks/scripts/verify-defaults-without-telemetry.sh +++ b/tasks/scripts/verify-defaults-without-telemetry.sh @@ -23,7 +23,7 @@ set -euo pipefail # `defaults-without-telemetry`. CRATES=( openshell-gateway - openshell-sandbox + openshell-supervisor openshell-driver-vm ) diff --git a/tasks/scripts/vm/build-libkrun.sh b/tasks/scripts/vm/build-libkrun.sh index 580f6bf4e5..e204dc0829 100755 --- a/tasks/scripts/vm/build-libkrun.sh +++ b/tasks/scripts/vm/build-libkrun.sh @@ -5,7 +5,7 @@ # Build libkrun and libkrunfw from source on Linux. # # This script builds libkrun (VMM) and libkrunfw (kernel firmware) from source -# with OpenShell's custom kernel configuration for bridge/netfilter support. +# with OpenShell's custom kernel configuration for sandbox enforcement. # # In addition to the platform's native .so artifacts, this script exports # kernel.c and ABI_VERSION metadata so that other platforms (e.g. macOS) can @@ -212,7 +212,11 @@ if [ -f openshell.kconfig ]; then # Verify that the key options were actually applied. all_ok=true - for opt in CONFIG_BRIDGE CONFIG_NETFILTER CONFIG_NF_NAT; do + for opt in \ + CONFIG_SECURITY \ + CONFIG_SECURITY_LANDLOCK \ + CONFIG_SECCOMP \ + CONFIG_SECCOMP_FILTER; do val="$(grep "^${opt}=" "${KERNEL_SOURCES}/.config" 2>/dev/null || true)" if [ -n "$val" ]; then echo " ${opt}: ${val#*=}" @@ -221,6 +225,13 @@ if [ -f openshell.kconfig ]; then all_ok=false fi done + lsm_order="$(grep '^CONFIG_LSM=' "${KERNEL_SOURCES}/.config" 2>/dev/null || true)" + if [[ "$lsm_order" == *landlock* ]]; then + echo " CONFIG_LSM: ${lsm_order#*=}" + else + echo " WARNING: CONFIG_LSM does not activate Landlock: ${lsm_order:-unset}" >&2 + all_ok=false + fi if [ "$all_ok" = false ]; then echo "ERROR: kernel config fragment merge failed — required options missing" >&2 exit 1 diff --git a/tasks/scripts/vm/build-supervisor-bundle.sh b/tasks/scripts/vm/build-supervisor-bundle.sh index 0085c0619d..ced808fff3 100755 --- a/tasks/scripts/vm/build-supervisor-bundle.sh +++ b/tasks/scripts/vm/build-supervisor-bundle.sh @@ -46,10 +46,10 @@ fi case "${GUEST_ARCH}" in aarch64|arm64) - RUST_TARGET="aarch64-unknown-linux-gnu" + SANDBOX_RUST_TARGET="aarch64-unknown-linux-musl" ;; x86_64|amd64) - RUST_TARGET="x86_64-unknown-linux-gnu" + SANDBOX_RUST_TARGET="x86_64-unknown-linux-musl" ;; *) echo "ERROR: Unsupported guest architecture: ${GUEST_ARCH}" >&2 @@ -58,12 +58,17 @@ case "${GUEST_ARCH}" in ;; esac -SUPERVISOR_BIN="${ROOT}/target/${RUST_TARGET}/release/openshell-sandbox" +SUPERVISOR_BIN="${ROOT}/target/${SANDBOX_RUST_TARGET}/release/openshell-sandbox" SUPERVISOR_OUTPUT="${OUTPUT_DIR}/openshell-sandbox.zst" +VM_INIT_BIN="${ROOT}/target/${SANDBOX_RUST_TARGET}/release/openshell-vm-init" +VM_INIT_OUTPUT="${OUTPUT_DIR}/openshell-vm-init.zst" +HOST_SUPERVISOR_BIN="${ROOT}/target/release/openshell-supervisor" +HOST_SUPERVISOR_OUTPUT="${OUTPUT_DIR}/openshell-supervisor.zst" echo "==> Building openshell-sandbox supervisor bundle" echo " Guest arch: ${GUEST_ARCH}" -echo " Rust target: ${RUST_TARGET}" +echo " Sandbox target: ${SANDBOX_RUST_TARGET} (static musl)" +echo " Host supervisor target: native" echo " Output: ${SUPERVISOR_OUTPUT}" mkdir -p "${OUTPUT_DIR}" @@ -79,13 +84,19 @@ run_supervisor_build() { fi if command -v cargo-zigbuild >/dev/null 2>&1; then - ${cargo_prefix[@]+"${cargo_prefix[@]}"} cargo zigbuild --release -p openshell-sandbox --target "${RUST_TARGET}" \ + ${cargo_prefix[@]+"${cargo_prefix[@]}"} cargo zigbuild --release -p openshell-sandbox --target "${SANDBOX_RUST_TARGET}" \ + --manifest-path "${ROOT}/Cargo.toml" + ${cargo_prefix[@]+"${cargo_prefix[@]}"} cargo zigbuild --release -p openshell-driver-vm --bin openshell-vm-init --no-default-features --target "${SANDBOX_RUST_TARGET}" \ --manifest-path "${ROOT}/Cargo.toml" else echo " cargo-zigbuild not found, falling back to cargo build..." - ${cargo_prefix[@]+"${cargo_prefix[@]}"} cargo build --release -p openshell-sandbox --target "${RUST_TARGET}" \ + ${cargo_prefix[@]+"${cargo_prefix[@]}"} cargo build --release -p openshell-sandbox --target "${SANDBOX_RUST_TARGET}" \ + --manifest-path "${ROOT}/Cargo.toml" + ${cargo_prefix[@]+"${cargo_prefix[@]}"} cargo build --release -p openshell-driver-vm --bin openshell-vm-init --no-default-features --target "${SANDBOX_RUST_TARGET}" \ --manifest-path "${ROOT}/Cargo.toml" fi + ${cargo_prefix[@]+"${cargo_prefix[@]}"} cargo build --release -p openshell-supervisor \ + --manifest-path "${ROOT}/Cargo.toml" } print_build_failure() { @@ -116,13 +127,25 @@ else fi fi -if [ ! -f "${SUPERVISOR_BIN}" ]; then - echo "ERROR: supervisor binary not found at ${SUPERVISOR_BIN}" >&2 +if [ ! -f "${SUPERVISOR_BIN}" ] || [ ! -f "${VM_INIT_BIN}" ] || [ ! -f "${HOST_SUPERVISOR_BIN}" ]; then + echo "ERROR: sandbox, VM init, or supervisor binary not found after build" >&2 + exit 1 +fi + +if readelf -l "${SUPERVISOR_BIN}" 2>/dev/null | grep -q 'Requesting program interpreter'; then + echo "ERROR: VM guest openshell-sandbox must be statically linked" >&2 + exit 1 +fi +if readelf -l "${VM_INIT_BIN}" 2>/dev/null | grep -q 'Requesting program interpreter'; then + echo "ERROR: openshell-vm-init must be statically linked" >&2 exit 1 fi zstd -19 -T0 -f "${SUPERVISOR_BIN}" -o "${SUPERVISOR_OUTPUT}" +zstd -19 -T0 -f "${VM_INIT_BIN}" -o "${VM_INIT_OUTPUT}" +zstd -19 -T0 -f "${HOST_SUPERVISOR_BIN}" -o "${HOST_SUPERVISOR_OUTPUT}" echo "==> Bundled supervisor ready" echo " Binary: $(du -sh "${SUPERVISOR_BIN}" | cut -f1)" echo " Compressed: $(du -sh "${SUPERVISOR_OUTPUT}" | cut -f1)" +echo " VM init: $(du -sh "${VM_INIT_OUTPUT}" | cut -f1)" diff --git a/tasks/scripts/vm/compress-vm-runtime.sh b/tasks/scripts/vm/compress-vm-runtime.sh index 598dc5505d..d2d5b5878b 100755 --- a/tasks/scripts/vm/compress-vm-runtime.sh +++ b/tasks/scripts/vm/compress-vm-runtime.sh @@ -4,7 +4,7 @@ # Gather VM runtime artifacts from local sources and compress for embedding. # -# This script collects libkrun, libkrunfw, gvproxy, and the guest OCI unpacker +# This script collects libkrun, libkrunfw, and the guest OCI unpacker # from local sources or pinned releases and compresses them with zstd for # embedding into the openshell-driver-vm binary. # @@ -28,7 +28,6 @@ ROOT="$(vm_lib_root)" # Source pins for runtime tool versions. source "${ROOT}/crates/openshell-driver-vm/runtime/pins.env" 2>/dev/null || true -GVPROXY_VERSION="${GVPROXY_VERSION:-v0.8.8}" UMOCI_VERSION="${UMOCI_VERSION:-v0.6.0}" # ── macOS dylib portability helpers ───────────────────────────────────── @@ -72,12 +71,12 @@ _check_compressed_artifacts() { platform="$(uname -s)-$(uname -m)" case "$platform" in Darwin-arm64) - for f in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst; do + for f in libkrun.dylib.zst libkrunfw.5.dylib.zst umoci.zst; do [ -f "${dir}/${f}" ] || return 1 done ;; Linux-*) - for f in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst; do + for f in libkrun.so.zst libkrunfw.so.5.zst umoci.zst; do [ -f "${dir}/${f}" ] || return 1 done ;; @@ -224,25 +223,13 @@ case "$(uname -s)-$(uname -m)" in cp "$WORK_DIR/libkrunfw.dylib" "$WORK_DIR/libkrunfw.5.dylib" fi - # gvproxy - prefer Podman, fall back to Homebrew - if [ -x /opt/podman/bin/gvproxy ]; then - cp /opt/podman/bin/gvproxy "$WORK_DIR/" - echo " Using gvproxy from Podman" - elif [ -x "${BREW_PREFIX}/bin/gvproxy" ]; then - cp "${BREW_PREFIX}/bin/gvproxy" "$WORK_DIR/" - echo " Using gvproxy from Homebrew" - else - echo "Error: gvproxy not found. Install Podman Desktop or run: brew install gvproxy" >&2 - exit 1 - fi download_umoci_for_guest "$WORK_DIR/umoci" "arm64" ;; Linux-*) ARCH="$(uname -m)" case "$ARCH" in - aarch64) GVPROXY_ARCH="arm64" ;; - x86_64) GVPROXY_ARCH="amd64" ;; + aarch64|x86_64) ;; *) echo "Error: Unsupported Linux architecture: ${ARCH}" >&2 exit 1 @@ -274,13 +261,6 @@ case "$(uname -s)-$(uname -m)" in fi fi - # Download gvproxy if not present - if [ ! -f "$WORK_DIR/gvproxy" ]; then - echo " Downloading gvproxy for linux-${GVPROXY_ARCH}..." - curl -fsSL -o "$WORK_DIR/gvproxy" \ - "https://github.com/containers/gvisor-tap-vsock/releases/download/${GVPROXY_VERSION}/gvproxy-linux-${GVPROXY_ARCH}" - chmod +x "$WORK_DIR/gvproxy" - fi download_umoci_for_guest "$WORK_DIR/umoci" "$ARCH" ;; diff --git a/tasks/scripts/vm/download-kernel-runtime.sh b/tasks/scripts/vm/download-kernel-runtime.sh index 2e83d0fb3d..5b778cc94d 100755 --- a/tasks/scripts/vm/download-kernel-runtime.sh +++ b/tasks/scripts/vm/download-kernel-runtime.sh @@ -2,7 +2,8 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Download pre-built VM kernel runtime artifacts from the vm-runtime GitHub +# Download pre-built VM kernel runtime artifacts from the capability-free +# vm-runtime GitHub # Release and stage them for the openshell-driver-vm cargo build. # # This script is used by driver release CI and can also be used locally diff --git a/tasks/scripts/vm/package-vm-runtime.sh b/tasks/scripts/vm/package-vm-runtime.sh index e9371bb509..fbbd482e55 100755 --- a/tasks/scripts/vm/package-vm-runtime.sh +++ b/tasks/scripts/vm/package-vm-runtime.sh @@ -4,8 +4,8 @@ # Package VM runtime artifacts into a release tarball. # -# Used by CI (release-vm-kernel.yml) to bundle libkrun, libkrunfw, gvproxy, -# and the guest OCI unpacker into a platform-specific tarball for the +# Used by CI (release-vm-kernel.yml) to bundle libkrun, libkrunfw, and the +# guest OCI unpacker into a platform-specific tarball for the # vm-runtime GitHub Release. Handles tool downloads, provenance metadata # generation, and tarball creation. # @@ -31,7 +31,6 @@ ROOT="$(vm_lib_root)" # Source pins for runtime tool versions. source "${ROOT}/crates/openshell-driver-vm/runtime/pins.env" 2>/dev/null || true -GVPROXY_VERSION="${GVPROXY_VERSION:-v0.8.8}" UMOCI_VERSION="${UMOCI_VERSION:-v0.6.0}" PLATFORM="" @@ -106,18 +105,6 @@ case "$PLATFORM" in ;; esac -# ── Download gvproxy ──────────────────────────────────────────────────── - -echo "==> Downloading gvproxy ${GVPROXY_VERSION} for ${PLATFORM}..." -case "$PLATFORM" in - linux-aarch64) GVPROXY_SUFFIX="linux-arm64" ;; - linux-x86_64) GVPROXY_SUFFIX="linux-amd64" ;; - darwin-aarch64) GVPROXY_SUFFIX="darwin" ;; -esac - -curl -fsSL -o "${PACKAGE_DIR}/gvproxy" \ - "https://github.com/containers/gvisor-tap-vsock/releases/download/${GVPROXY_VERSION}/gvproxy-${GVPROXY_SUFFIX}" -chmod +x "${PACKAGE_DIR}/gvproxy" # ── Download umoci for the Linux guest ─────────────────────────────────── @@ -162,11 +149,10 @@ jq -n \ --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg kfw_commit "$LIBKRUNFW_COMMIT" \ --arg kver "$KERNEL_VERSION" \ - --arg gvproxy "$GVPROXY_VERSION" \ --arg umoci "$UMOCI_VERSION" \ --arg sha "${GITHUB_SHA:-unknown}" \ --arg run "${GITHUB_RUN_ID:-unknown}" \ - '{artifact: $artifact, platform: $platform, build_timestamp: $ts, libkrunfw_commit: $kfw_commit, kernel_version: $kver, gvproxy_version: $gvproxy, umoci_version: $umoci, github_sha: $sha, github_run_id: $run}' \ + '{artifact: $artifact, platform: $platform, build_timestamp: $ts, libkrunfw_commit: $kfw_commit, kernel_version: $kver, umoci_version: $umoci, github_sha: $sha, github_run_id: $run}' \ > "${PACKAGE_DIR}/provenance.json" # ── Create tarball ────────────────────────────────────────────────────── diff --git a/tasks/scripts/vm/smoke-orphan-cleanup.sh b/tasks/scripts/vm/smoke-orphan-cleanup.sh index 6d66fbb5d4..f747b195f7 100755 --- a/tasks/scripts/vm/smoke-orphan-cleanup.sh +++ b/tasks/scripts/vm/smoke-orphan-cleanup.sh @@ -4,7 +4,7 @@ # # Smoke test: start the gateway with the VM driver, create a sandbox, then # signal the gateway (SIGTERM then SIGKILL) and verify that no driver, -# launcher, gvproxy, or libkrun worker processes survive. +# launcher or libkrun worker processes survive. # # Exit codes: # 0 — both SIGTERM and SIGKILL cleanup passed @@ -108,10 +108,8 @@ create_sandbox() { for _ in $(seq 1 60); do if pgrep -f "openshell-vm-orphan-$$|$STATE_DIR/sandboxes/" >/dev/null 2>&1; then - if pgrep -f gvproxy >/dev/null 2>&1; then - echo "sandbox came up (cli pid=$CLI_PID)" - return 0 - fi + echo "sandbox came up (cli pid=$CLI_PID)" + return 0 fi sleep 2 done @@ -122,17 +120,14 @@ create_sandbox() { snapshot_kids() { # Return all PIDs whose --state-dir or --vm-rootfs references our - # per-run directory, plus any gvproxy that mentions our socket base. + # per-run directory. pgrep -fl "state-dir $STATE_DIR|$STATE_DIR/sandboxes" 2>/dev/null || true - pgrep -fl "gvproxy" 2>/dev/null | grep "osd-gv" || true } count_alive() { local alive alive=$(pgrep -f "state-dir $STATE_DIR|$STATE_DIR/sandboxes" 2>/dev/null | wc -l | tr -d ' ') - local gv - gv=$(pgrep -f 'gvproxy' 2>/dev/null | xargs -r ps -o pid=,command= -p 2>/dev/null | grep -c 'osd-gv' || true) - echo $((alive + gv)) + echo "$alive" } verify_cleanup() { @@ -176,8 +171,7 @@ run_scenario() { # Belt-and-braces teardown between scenarios. pkill -9 -f "$STATE_DIR/sandboxes|$STATE_DIR " 2>/dev/null || true - pkill -9 -f 'gvproxy.*osd-gv' 2>/dev/null || true - rm -rf "$STATE_DIR" /tmp/osd-gv "$XDG" 2>/dev/null || true + rm -rf "$STATE_DIR" "$XDG" 2>/dev/null || true # CLI may still be running; reap it. kill "${CLI_PID:-0}" 2>/dev/null || true sleep 1 @@ -191,7 +185,6 @@ main() { # Clean starting state. pkill -9 -f 'openshell-gateway|openshell-driver-vm' 2>/dev/null || true - pkill -9 -f 'gvproxy.*osd-gv' 2>/dev/null || true sleep 1 if ! run_scenario TERM "graceful SIGTERM"; then diff --git a/tasks/scripts/vm/vm-setup.sh b/tasks/scripts/vm/vm-setup.sh index de568d8783..4b6bb297b3 100755 --- a/tasks/scripts/vm/vm-setup.sh +++ b/tasks/scripts/vm/vm-setup.sh @@ -4,7 +4,7 @@ # One-time setup for the openshell-driver-vm runtime. # -# Downloads pre-built runtime artifacts (libkrun, libkrunfw, gvproxy, umoci) +# Downloads pre-built runtime artifacts (libkrun, libkrunfw, umoci) # from the vm-runtime GitHub Release, or builds them from source when # --from-source is set. # After obtaining the runtime, compresses the artifacts for embedding into the @@ -35,7 +35,7 @@ while [[ $# -gt 0 ]]; do --help|-h) echo "Usage: $0 [--from-source]" echo "" - echo "Set up the openshell-driver-vm runtime (libkrun, libkrunfw, gvproxy, umoci)." + echo "Set up the openshell-driver-vm runtime (libkrun, libkrunfw, umoci)." echo "" echo "Options:" echo " --from-source Build runtime from source instead of downloading (~15-45min)" @@ -101,7 +101,7 @@ OUTPUT_DIR="${OPENSHELL_VM_RUNTIME_COMPRESSED_DIR:-${ROOT}/target/vm-runtime-com missing=0 case "$PLATFORM" in darwin-aarch64) - for f in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst; do + for f in libkrun.dylib.zst libkrunfw.5.dylib.zst umoci.zst; do if [ ! -f "${OUTPUT_DIR}/${f}" ]; then echo "ERROR: Missing ${OUTPUT_DIR}/${f}" >&2 missing=1 @@ -109,7 +109,7 @@ case "$PLATFORM" in done ;; linux-aarch64|linux-x86_64) - for f in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst; do + for f in libkrun.so.zst libkrunfw.so.5.zst umoci.zst; do if [ ! -f "${OUTPUT_DIR}/${f}" ]; then echo "ERROR: Missing ${OUTPUT_DIR}/${f}" >&2 missing=1 diff --git a/tasks/test.toml b/tasks/test.toml index 8c41e70025..f26950ce51 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -190,9 +190,8 @@ run = [ "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" AGENT_SANDBOX_VERSION=v0.4.6 e2e/rust/e2e-kubernetes.sh", ] -["e2e:kubernetes:sidecar"] -description = "Run Kubernetes e2e with the supervisor sidecar topology overlay" -env = { OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-sidecar.yaml" } +["e2e:kubernetes:isolation"] +description = "Run Kubernetes e2e with the workload network fence and separate supervisor" depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-kubernetes.sh" @@ -248,7 +247,7 @@ depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-vm.sh" ["e2e:kubernetes:external-driver"] -description = "Run Kubernetes conformance with a driver-free gateway and external Kubernetes driver sidecar" +description = "Run Kubernetes conformance with a driver-free gateway and external Kubernetes driver" env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_KUBE_BUILD_IMAGES = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "" } depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-kubernetes.sh"