bug: Kubernetes bootstrap retry limits can leave sandboxes terminal during gateway startup

[jtatum]

User Story

As a Kubernetes operator, I want sandbox bootstrap to tolerate temporary gateway unavailability, so that dependency startup ordering does not require destructive recovery.

Problem Statement

This is a source-based report against public OpenShell v0.0.116 code. The deterministic reproduction below is proposed, not a claim of a completed isolated upstream test. No private deployment logs or configuration are included.

Three public implementation details appear to make delayed gateway startup a terminal sandbox failure:

1. grpc_retry in the sandbox runtime makes five attempts, with 1/2/4/4-second waits. When initial policy fetching cannot connect, it returns an error after exhausting those attempts.
2. The same file defines a separate SIDECAR_READY_TIMEOUT_SECS = 120 for the process supervisor waiting on the network supervisor. Extending the policy retry window alone would not remove this second deadline.
3. The Kubernetes driver deliberately renders restartPolicy: Never, with a comment explaining that kubelet must not replace the canonical main-process generation after supervisor exit.

Consequently, a transient dependency outage before workload activation can exhaust bootstrap retries, terminate the pod, and leave no automatic retry when the gateway becomes available. The same retry limits remain in inspected main commit 02b664bb0d978ac0baec9aaa0bf06a2a4f67e83d.

Impact / Why This Matters

Kubernetes does not guarantee that the gateway is ready before sandbox pods start during cluster recovery. A short-lived startup outage should not require deleting a persistent sandbox.

The v0.0.116 gateway start handler rejects terminal Error phases; it allows Ready as an idempotent return and Stopped/Starting for start. This makes a documented, data-preserving recovery path important in addition to bootstrap retry behavior. Blanket automatic deletion/recreation would be inappropriate for persistent workspaces.

Acceptance Criteria

• Before workload activation, a sandbox can tolerate transient gateway unavailability and become usable once the gateway returns without manual deletion.
• A regression test delays gateway availability beyond both the current policy-fetch retry window and the 120-second process-sidecar wait.
• No workload starts before authenticated policy and required enforcement bootstrap succeed. Permanent authentication/configuration/policy errors remain explicit failures, and TLS validation is preserved.
• Individual attempts are bounded, retries use capped backoff, and waiting is cancellable and diagnosable.
• Existing workspace data survives the tested recovery.
• Explicit stops and completed workloads remain terminal; fixing bootstrap does not silently replay completed commands.
• Document a supported, data-preserving recovery path for sandboxes already in terminal bootstrap error.

Reproduction Steps

Proposed deterministic test, not yet executed as an isolated upstream reproduction:

1. Use a disposable Kubernetes deployment with the v0.0.116 gateway, supervisor, and sidecar topology.
2. Create a persistent sandbox and write a workspace sentinel.
3. Simulate dependency recovery so a replacement sandbox pod starts while its gateway callback is unavailable. Keep the gateway unavailable for more than 120 seconds.
4. Restore gateway availability.
5. Check pod/Sandbox/gateway state, whether a shell becomes usable without resource deletion, and whether the workspace sentinel is retained.

The public code predicts an exhausted Policy fetch retry followed by the separate sidecar-control-socket timeout. A regression test should establish the exact lifecycle transitions as well as successful recovery after the fix. Fault injection should use an isolated test deployment, not a shared production gateway.

Environment

Source versions and configuration relevant to the proposed test, not private deployment inventory:

• OpenShell: public tag v0.0.116, latest published release at filing.
• Additional source inspection: main commit 02b664b; no runtime test of that commit is claimed.
• Compute driver: Kubernetes.
• Supervisor topology: sidecar.
• Persistent workspace; driver-generated restartPolicy: Never.

Related Work

#2179 and #2269 concern Docker/Podman Error-phase recovery. #2417 explicitly keeps Kubernetes lifecycle cluster-owned. They provide useful lifecycle context but do not appear to resolve the Kubernetes bootstrap retry limits above.

Changing pod restartPolicy globally to Always would alter the deliberately terminal main-process contract. This report requests bootstrap-specific resilience and a supported recovery path, not automatic replay or deletion of arbitrary failed workloads.