feat(kubernetes): support corporate proxy CA bundle for https:// and TLS-intercepting proxies

[feloy]

Follow-up to #1792, tracking the Kubernetes driver the way #3088 tracked the VM driver.

User Story

As a platform operator running OpenShell on Kubernetes behind a corporate egress proxy, I want to supply the corporate CA bundle to sandbox supervisors, so that agents can reach policy-approved external APIs through an https:// forward proxy or a TLS-intercepting proxy whose certificates are signed by a private CA.

Problem Statement

#1792 added corporate forward-proxy egress. The Podman (#2512) and VM (#3090) drivers both expose proxy_ca_bundle; the Kubernetes driver does not.

[openshell.drivers.kubernetes] accepts https_proxy, no_proxy, proxy_auth_secret_name, proxy_auth_secret_key, proxy_auth_allow_insecure, and proxy_connect_by_hostname (crates/openshell-driver-kubernetes/src/config.rs:201-219), and the supervisor command it builds never includes --upstream-proxy-ca-bundle (crates/openshell-driver-kubernetes/src/sandbox_runtime.rs:283-298). The Helm chart has no matching value (deploy/helm/openshell/values.yaml:58).

An https:// proxy URL is accepted by Kubernetes-driver validation — parse_upstream_proxy_url accepts both schemes (crates/openshell-core/src/driver_utils.rs:289) — and reaches the supervisor, which does TLS-wrap the connection. Verification then uses only the built-in Mozilla roots plus the supervisor image's system bundle, so a proxy certificate issued by a private corporate CA fails the handshake with no way to supply the anchor. The same gap applies to a TLS-intercepting proxy reached over http://, which re-signs tunneled upstream certificates with its own CA.

Two secondary defects in the same code path:

• The Kubernetes driver hand-rolls its proxy validation (config.rs:379) instead of calling the shared validate_upstream_proxy_settings (driver_utils.rs:651), so the two can drift.
• As a result it requires proxy_auth_allow_insecure = true whenever a credential Secret is configured, including for an https:// proxy (config.rs:447), where the credential travels inside a verified TLS session. The shared validator correctly waives the acknowledgement in that case (driver_utils.rs:691).

Impact / Why This Matters

Operators on Kubernetes whose proxy presents a private-CA certificate, or who sit behind a TLS-intercepting proxy, have no working configuration. Setting https_proxy = "https://..." passes startup validation and then fails at sandbox egress, which reads as a runtime networking fault rather than a missing capability.

There is no workaround. The Kubernetes driver exposes no pod-template or volume override, so an operator cannot mount a CA over the supervisor container's system trust store, and sandbox pods run with automountServiceAccountToken: false (sandbox_runtime.rs:401). The only remaining option is forking the supervisor image per corporate environment.

The documentation is currently accurate — "Only http:// proxy endpoints and TLS CONNECT traffic are supported" (docs/reference/gateway-config.mdx:563) — so this is a documented capability gap rather than a silent bug. But it blocks exactly the enterprise Kubernetes deployments #1792 was opened for, and it was explicitly requested there: "If the server address uses https, we need option to pass custom CA."

Proposed Design

Add proxy_ca_bundle to [openshell.drivers.kubernetes], keeping the operator-facing key identical to the Podman and VM drivers so the corporate-proxy configuration surface stays uniform across compute backends.

The value is a path the gateway Pod can read — supplied by the operator the same way any other gateway file is, not a reference to an object in the sandbox namespace. The gateway is responsible for getting those bytes to each sandbox supervisor.

Observable behavior:

• With proxy_ca_bundle set, the supervisor trusts the bundle for the TLS handshake with an https:// proxy, for upstream certificates re-signed by a TLS-intercepting proxy, and in the trust bundle exposed to sandbox processes — matching the Podman and VM drivers exactly.
• Configuration is fail-closed and consistent with the existing drivers: an empty value, a path that cannot be read, a non-regular file, a bundle over the size bound, a bundle contributing no usable trust anchor, or proxy_ca_bundle without https_proxy is a startup or sandbox-create error naming the offending key — never a silent fall-back to direct dialing or to the built-in roots.
• The trust anchor a sandbox starts with does not change underneath it for that sandbox's lifetime.
• The setting works in all three workspace modes (shared, managed, operator), including namespaces the gateway creates itself.
• The Helm chart exposes the bundle under upstreamProxy, mounts it into the gateway Pod, and renders the resulting proxy_ca_bundle path into gateway.toml.
• The Kubernetes driver's proxy validation is replaced by the shared validate_upstream_proxy_settings, which also drops the incorrect proxy_auth_allow_insecure requirement for an https:// proxy.

Security note for the implementation: the corporate CA is folded into three trust stores (crates/openshell-supervisor-network/src/run.rs:354-381) — the proxy-listener handshake, the L7 upstream verification store, and the sandbox trust bundle. Whoever controls those bytes can transparently intercept all sandbox egress, and OpenShell's L7 enforcement can no longer distinguish a genuine upstream certificate from a re-signed one. That is inherent to TLS interception and already accepted for Podman and VM, but it is the reason the bundle must be anchored in the gateway's own trust domain rather than in the workload namespace.

Acceptance Criteria

• [openshell.drivers.kubernetes] proxy_ca_bundle is accepted and documented alongside the other corporate-proxy keys.
• A sandbox whose gateway sets https_proxy = "https://..." with a private-CA proxy completes policy-approved TLS egress; without proxy_ca_bundle the same configuration fails closed with a clear error.
• A TLS-intercepting proxy reached over http:// works when its re-signing CA is supplied via proxy_ca_bundle.
• Invalid proxy_ca_bundle values (empty, unreadable, non-regular file, oversized, certificate-free, or set without https_proxy) fail closed with an error naming the key.
• The setting works in shared, managed, and operator workspace modes.
• The Helm chart exposes the CA bundle under upstreamProxy and renders proxy_ca_bundle; chart tests cover the rendered output.
• The Kubernetes driver validates corporate-proxy settings through openshell_core::driver_utils::validate_upstream_proxy_settings, and a credential Secret with an https:// proxy no longer requires proxy_auth_allow_insecure.
• docs/reference/gateway-config.mdx and docs/reference/sandbox-compute-drivers.mdx no longer state that only http:// proxy endpoints are supported for Kubernetes.
• Architecture docs and the relevant crate READMEs reflect the new setting.

Alternatives Considered

Reference a pre-existing ConfigMap in the sandbox namespace by name/key, mirroring how proxy_auth_secret_name handles proxy credentials. Rejected on three grounds:

1. It widens the trust boundary from the gateway's trust domain to anyone holding update on ConfigMaps in the workload namespace. Given the three-store fold described above, that is control over interception of all sandbox egress.
2. Kubelet live-updates mounted ConfigMaps, so the trust anchor is not pinned.
3. It cannot work in managed or operator workspace mode: the gateway auto-creates per-workspace namespaces (crates/openshell-driver-kubernetes/src/driver.rs:961) and deploy/helm/openshell/templates/role.yaml grants it no ConfigMap verbs at all, so no operator can pre-provision an object into a namespace that does not yet exist.

The credential precedent does not fully transfer. In shared mode the gateway holds no get on Secrets at all (role.yaml:54-58 grants create, list, delete). In multi-namespace modes it does hold name-restricted get + patch on explicitly configured Secrets (clusterrole.yaml:105-119), which is how ensure_tls_secret copies TLS and image-pull Secrets into workspace namespaces. That mechanism could in principle carry a credential too, but a CA certificate is not secret, so reading it from the gateway's own filesystem is simpler and avoids widening what the gateway may read.

Inline the PEM in gateway.toml. Works, and needs no new RBAC, but bloats the rendered config, diverges from the proxy_ca_bundle path semantics the other drivers use, and gives up the existing bounded-read and regular-file checks.

Document the gap and require a forked supervisor image with the corporate CA baked into its system trust store. Rejected: it pushes image maintenance onto every affected operator, ties trust-anchor rotation to an image rebuild, and would still leave the https:// proxy URL misleadingly accepted by validation.

Agent Investigation

Supervisor-side support is already complete and driver-independent. upstream_proxy.rs TLS-wraps the proxy connection for a secure URL and builds its trust store from the built-in Mozilla roots, the system bundle, and the optional corporate bundle (upstream_proxy.rs:644-660). Validation is shared between host and guest via read_upstream_proxy_ca_bundle_file (driver_utils.rs:484), which bounds the read at 1 MiB, rejects non-regular files, and requires rustls to accept at least one anchor rather than merely parsing PEM framing (validate_upstream_proxy_ca_bundle_pem, driver_utils.rs:520). The bundle is validated even for an http:// proxy, because an intercepting proxy reached in plaintext still re-signs tunneled certificates (upstream_proxy.rs:530). Only the Kubernetes driver's plumbing is missing.

Related pre-existing gap, same feature, tracked separately: proxy_auth_secret_name is documented as "an existing Secret in the sandbox namespace," which no operator can pre-provision in managed or operator workspace mode for the same auto-created-namespace reason described above. Scoped out of this issue rather than addressed here; the ensure_tls_secret copying mechanism (driver.rs:1121) that already carries TLS and image-pull Secrets into workspace namespaces looks directly reusable for it.

[feloy]

🏗️ build-plan

Implementation Plan

Issue type: feat (carries one fix: the proxy_auth_allow_insecure requirement for https:// proxies)
Complexity: Medium
Confidence: High — every claim in the issue body was re-verified against main @ c108c31. Two corrections are folded in below.

Summary

Add proxy_ca_bundle to [openshell.drivers.kubernetes] as a gateway-Pod filesystem path, stage the PEM into the existing per-generation supervisor bootstrap Secret, and pass --upstream-proxy-ca-bundle <mounted path> on the supervisor argv. This needs no new object kind and no RBAC change: secrets: create,list,delete already exists in role.yaml:55 and the clusterrole.yaml managed/operator block, and that Secret is already immutable: true, owner-referenced, and explicitly garbage-collected. Supervisor-side support is already complete; only driver plumbing, Helm wiring, validator consolidation, and docs are missing.

Corrections to the issue's analysis

1. A blanket switch to the shared validator would be a silent regression. config.rs:403 rejects only proxy_auth_allow_insecure = Some(true) without credentials; an explicit = false without credentials is accepted today and pinned by upstream_proxy_config_allows_explicit_false_acknowledgement_without_credentials (config.rs:1307-1314). driver_utils.rs:698-703 rejects any Some(_) without auth_file. Mapping the Kubernetes fields straight onto auth_file/auth_allow_insecure would break existing hand-written gateway.toml. The shared validator is adopted for the URL / no_proxy / connect_by_hostname / ca_bundle rules only, with the Secret-specific block kept locally.
2. Kubernetes caps a Secret at ~1 MiB, and MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES is exactly 1 MiB (driver_utils.rs:461). A bundle that is legal to the shared reader could fail with an opaque apiserver Secret "…" is invalid: data: Too long on every sandbox create. This change owns that, so it adds a Kubernetes-specific bound.

Design decisions

How the PEM reaches the supervisor Pod — extend the existing supervisor bootstrap Secret. supervisor_bootstrap_secret (sandbox_runtime.rs:461-501) already exists per sandbox generation and already carries trust material of exactly this kind (proxy-ca.crt, the generated interception CA, likewise non-secret). Adding an optional upstream-proxy-ca.pem key needs zero new RBAC, works identically in shared/managed/operator including gateway-created namespaces, is immutable: true so kubelet cannot live-update the anchor, and inherits GC for free (owner reference at driver.rs:2401-2419 plus explicit label-selected deletion in delete_sandbox_runtime_generation_secrets, driver.rs:3204-3236).

Mount path — reuse the existing bootstrap volume. It already mounts the whole Secret read-only at /.openshell/supervisor with no items filter (sandbox_runtime.rs:227,233), so a new key appears at /.openshell/supervisor/upstream-proxy-ca.pem with no new volume or mount. PROXY_CA_MOUNT_PATH is deliberately not reused: driver_utils.rs:185-194 defines it as where drivers bind-mount a host file, a local-container convention with no Kubernetes analogue.

Workload Pod — no change. Only the supervisor consumes the bundle; run.rs:354-381 folds it into the system bundle, which reaches sandbox processes over the boundary protocol as Request::StartAgent { ca_bundle, .. } (boundary_protocol.rs:536), not via a shared volume.

Helm — mirror vault.caConfigMapName (values.yaml:332), which is already mounted into the gateway Pod through _gateway-workload.tpl:204-211 and :122-126. That template is shared by Deployment and StatefulSet, so one edit covers both.

Scope

File	Change
crates/openshell-core/src/driver_utils.rs:617-668	Add auth_setting_label: Option<&'a str> to UpstreamProxySettings, used only in the InlineCredentials message so it can name the Secret keys instead of proxy_auth_file. No behavior change for Podman/VM.
crates/openshell-driver-kubernetes/src/config.rs:201-219	Add pub proxy_ca_bundle: Option<String>, rustdoc matching the Podman field.
crates/openshell-driver-kubernetes/src/config.rs:379-465	Rewrite validate_upstream_proxy_config: shared validator + local Secret block; drop the https:// acknowledgement requirement at line 447.
crates/openshell-driver-kubernetes/src/config.rs	New MAX_STAGED_PROXY_CA_BUNDLE_BYTES = 256 KiB, citing the apiserver Secret limit.
crates/openshell-driver-kubernetes/src/sandbox_runtime.rs:33-42	UPSTREAM_PROXY_CA_BUNDLE_KEY / ..._PATH constants.
crates/openshell-driver-kubernetes/src/sandbox_runtime.rs:165-190, 271-299	New proxy_ca_bundle_staged: bool; emit --upstream-proxy-ca-bundle. No volume change.
crates/openshell-driver-kubernetes/src/sandbox_runtime.rs:461-501	supervisor_bootstrap_secret takes upstream_proxy_ca_bundle: Option<Vec<u8>>.
crates/openshell-driver-kubernetes/src/driver.rs:672-690	Startup preflight read via spawn_blocking, so a bad path fails gateway startup rather than the first sandbox create.
crates/openshell-driver-kubernetes/src/driver.rs:2134, 2465	Read+validate the PEM at the top of both generation-install paths, before any Service/Pod is created.
crates/openshell-driver-kubernetes/src/driver.rs:2198-2225, 2401, 2639, 2934-2960	Thread the flag and the bytes through the four call sites.
crates/openshell-driver-kubernetes/src/main.rs:138-161, 260-265	--proxy-ca-bundle / OPENSHELL_UPSTREAM_PROXY_CA_BUNDLE for the standalone driver binary.
deploy/helm/openshell/values.yaml:56-71	upstreamProxy.caBundle.{configMapName,key}.
deploy/helm/openshell/templates/_gateway-workload.tpl	Read-only mount + configMap volume with items, guarded on configMapName.
deploy/helm/openshell/templates/gateway-config.yaml:184-186	Render proxy_ca_bundle when configMapName is set.

No RBAC template changes. No workload-Pod changes. No openshell-supervisor* changes.

Implementation Steps

1. driver_utils.rs label field — add auth_setting_label, thread into the InlineCredentials arm. Independently mergeable.
2. Consolidate Kubernetes proxy validation (the fix half) — shared validator + local Secret block; https:// + Secret credentials no longer requires the acknowledgement. User-visible on its own.
3. Config key + size bound — add proxy_ca_bundle; the shared validator then enforces empty-value and requires-https_proxy for free.
4. Staging — constants, Secret parameter, supervisor_pod flag, driver helper (read_upstream_proxy_ca_bundle_file in spawn_blocking, then the 256 KiB check), startup preflight, call-site updates.
5. Helm — values + mount + volume + gateway.toml rendering, with chart tests.
6. Docs + debug-openshell-cluster skill.
7. E2E https-ca mode — sequenced last, separately mergeable.

Test Plan

• Unit: driver_utils.rs tests for the new label in both default and overridden form. config.rs tests (~1137-1320): proxy_ca_bundle round-trips through TOML; without https_proxy → error naming the key; empty → error naming the key; https:// + Secret without acknowledgement → now ok; http:// + Secret without acknowledgement → still an error; and ..._allows_explicit_false_acknowledgement_without_credentials must stay green as the regression guard for the validator switch. sandbox_runtime.rs tests (604-828): argv pair present when staged and absent when not, no new volume/mount introduced, and the bootstrap Secret stays immutable: true with the key present/absent. Plus a driver-level test for the 256 KiB bound and for errors naming proxy_ca_bundle.
• Integration (Helm): extend deploy/helm/openshell/tests/gateway_config_test.yaml:254-281 to assert the rendered proxy_ca_bundle, plus a case asserting the key is absent when configMapName is empty; and a mount/volume assertion covering both workload.kind values, since _gateway-workload.tpl is shared.
• E2E: e2e/rust/tests/kubernetes_corporate_proxy.rs and e2e/with-kube-gateway.sh:1056-1099 already dispatch on OPENSHELL_E2E_KUBE_CORPORATE_PROXY_MODE. Add an https-ca mode. One structural difference from the Podman test: the Kubernetes gateway is Helm-configured before the test runs, so the wrapper must mint the CA and listener leaf with openssl (leaf must be CA:FALSE), kubectl create configmap it in the release namespace, and --set upstreamProxy.caBundle.configMapName. Per AGENTS.md "Sandbox Infra Changes", the supervisor Pod spec changes here, so the Kubernetes sandbox e2e path must be exercised regardless.

Risks & Open Questions

• Secret size (owned by this change). ~1 MiB apiserver limit vs. the shared 1 MiB read bound, mitigated by a 256 KiB staging check. Open question for maintainers: is 256 KiB right, or should the driver just surface the apiserver error?
• Boundary control-frame size (pre-existing, not owned here). MAX_CONTROL_FRAME_BYTES = 1 MiB (boundary_protocol.rs:35) bounds StartAgent, which carries policy plus the combined system+corporate bundle. A very large corporate bundle could push it over on any driver. The 256 KiB staging bound keeps Kubernetes well clear; worth a separate issue for a clearer diagnostic.
• Rotation semantics. The PEM is read per sandbox create, so rotating the ConfigMap affects new sandboxes only; running sandboxes keep their pinned anchor. This is the intended reading of the "does not change underneath it" acceptance criterion — confirm it matches operator expectations.
• Trust concentration. Whoever can write the gateway's CA ConfigMap can intercept all sandbox egress (run.rs:354-381 folds it into three stores). Anchoring in the gateway's release namespace rather than the workload namespace is the mitigation; the docs must say so.
• Missing ConfigMap leaves the gateway Pod in ContainerCreating, same as oidc-ca/vault-ca.
• Out of scope, flagged in the issue: proxy_auth_secret_name still cannot be pre-provisioned in managed/operator mode. Same root cause, different mechanism — file separately rather than expanding this PR.
• LSM: N/A, confirmed. No /proc/<pid>/exe resolution, no fork+exec identity, no Landlock rule changes.

Documentation Impact

• docs/reference/gateway-config.mdx:562-585 — required. Remove "Only http:// proxy endpoints ... are supported"; document proxy_ca_bundle and the relaxed acknowledgement rule.
• docs/reference/sandbox-compute-drivers.mdx:437-442 — add the proxy_ca_bundle → upstreamProxy.caBundle.configMapName row and correct the https_proxy row.
• skills/debug-openshell-cluster/SKILL.md:705-742 — required (Helm values/templates change). Line 719 currently asserts https:// proxy URLs are unsupported.
• crates/openshell-driver-kubernetes/README.md — the bootstrap-Secret section should mention the optional key.
• architecture/sandbox.md — one line if the corporate-proxy trust fold is described there. No new architecture doc.
• TESTING.md:399-406 — the new e2e mode env var, if step 7 lands.

---

Revision 1 — initial plan