VM driver (v0.0.116, Linux x86_64, nested KVM): sandbox never reaches Ready - PushSandboxLogs stream dropped with UnexpectedEof during boot

[gdraper-fs]

User Story

As an operator running the OpenShell gateway with the experimental vm compute driver on a Linux VM with nested KVM, I want openshell sandbox create to bring a sandbox to Ready so that I can run per-task workloads in microVM isolation instead of falling back to the podman driver.

Problem Statement

On a fresh 0.0.116 deb install, every sandbox create with the vm driver hangs in Provisioning and times out after 300 s ("Last reported status: Starting: VM is starting"). Gateway debug logs show the guest completes the mTLS handshake (TLS 1.3, ALPN h2, client-cert auth) and several unary RPCs (ReportPolicyStatus, GetSandboxConfig, UpdateConfig, GetSandboxProviderEnvironment, GetInferenceBundle) cleanly, but the one client-streaming call, PushSandboxLogs, has its connection dropped mid-stream with a raw TCP EOF (UnexpectedEof, no TLS close_notify) and returns gRPC status 13. After that the guest supervisor makes no further progress; it does not crash, does not retry, and the openshell-driver-vm --internal-run-vm process stays alive indefinitely. The failure is identical for a sub-second main process and for sleep infinity left running 14+ minutes, so it is not the short-lived-main-process case in #3117. Ruled out: gateway hostname (host.openshell.internal is set and the handshake succeeds), driver_dir (corrected to /usr/libexec/openshell, driver connects with in_tree=true), TLS/cert config, permissions/sudo, state_dir. Secondary: the guest logs two [HIGH] "Landlock Filesystem Sandbox Unavailable" findings on every boot, so landlock.compatibility: hard_requirement could not be satisfied under this driver/image even if boot completed.

Impact / Why This Matters

The vm driver is unusable on this platform class (Linux guest under a hypervisor with nested KVM), which is a common lab and homelab shape. Operators who need microVM isolation for untrusted agent workloads must fall back to container isolation (podman driver), which also means losing the guest-kernel boundary. Because the guest reports Landlock unavailable, the fallback is currently the only way to get hard_requirement enforcement at all on this host.

Acceptance Criteria

• With the configuration below, openshell sandbox create --name hello --detach -- sleep infinity reaches Ready within normal image-pull plus boot time, and openshell sandbox exec -n hello -- uname -r returns the guest kernel version.
• PushSandboxLogs completes (gRPC status 0) during boot, or a failure of that stream is retried/non-fatal so provisioning can continue.
• Either Landlock is available inside the community VM-driver guest kernel, or the documentation states that hard_requirement is not satisfiable under the VM driver.

Reproduction Steps

1. Install openshell_0.0.116-1_amd64.deb (checksum-verified) on the host described below; run the gateway as a systemd user service with the config in Environment.
2. openshell gateway add https://127.0.0.1:17670 --local
3. openshell sandbox create --no-keep -- sh -c "echo hello-from-microvm; uname -r" → times out after 300 s in Provisioning.
4. To rule out a short-lived main process: openshell sandbox create --name hello --detach -- sleep infinity, then poll openshell sandbox get hello -o json every 15 s for 14+ minutes → phase stays Provisioning; openshell sandbox exec -n hello -- sh -c "uname -r" → "sandbox 'hello' is not ready (phase: Provisioning)".
5. ps shows openshell-driver-vm --internal-run-vm … alive throughout; the sandbox's sleep infinity never exited.
6. Switch the same gateway to compute_drivers = ["podman"] (rootless socket, grpc_endpoint = "https://host.containers.internal:17670"), no other change → sandboxes reach Ready immediately and exec works.

Environment

• OpenShell 0.0.116 (official GitHub release deb); driver binaries at /usr/libexec/openshell.
• Host: Ubuntu 26.04.1, kernel 7.0.0-30-generic, x86_64, a guest VM under Proxmox VE with nested KVM enabled (kvm_amd loaded, /dev/kvm usable by the operator, member of group kvm); 8 vCPU, 22 GB RAM.
• Rootless Podman 5.7.0 (used for image pulls; the podman driver works on the same host).
• openshell-gateway as a systemd user service, mTLS, bind_address = "127.0.0.1:17670".
• Gateway config:

[openshell.gateway]
bind_address    = "127.0.0.1:17670"
compute_drivers = ["vm"]

[openshell.drivers.vm]
driver_dir       = "/usr/libexec/openshell"
default_image    = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
bootstrap_image  = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
grpc_endpoint    = "https://host.openshell.internal:17670"
vcpus            = 2
mem_mib          = 3072
overlay_disk_mib = 4096
krun_log_level   = 1

Logs

Guest console, last lines before it goes silent:


2026-09-16T05:08:41.730Z OCSF CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded [version:1] ...
2026-09-16T05:08:41.745Z OCSF FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" (x2)
2026-09-16T05:08:41.789Z OCSF CONFIG:CLEANED_UP [INFO] Network namespace cleaned up [ns:sandbox-2194cf63]


Gateway (debug) for the failing call; the handshake for this same connection had already completed with client auth:


2026-09-16T05:11:54.194Z DEBUG log: Attempting client auth
2026-09-16T05:11:54.951095Z DEBUG Connection{peer=Server}: h2::proto::connection: Connection::poll; IO error error=UnexpectedEof
2026-09-16T05:11:54.951209Z DEBUG request{...path="/openshell.v1.OpenShell/PushSandboxLogs"...}: tonic::codec::decode: decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: UnexpectedEof, error: "peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof" }) })) }
2026-09-16T05:11:54.951359Z  INFO request{...rpc.method="PushSandboxLogs" http.response.status_code=200 rpc.grpc.status_code=13 otel.status_code="ERROR"}: response status=200 latency_ms=615


CLI:


Error:   × sandbox provisioning timed out after 300s. Last reported status: Starting: VM is starting


Happy to provide the full debug-level gateway journal, the complete guest `rootfs-console.log`, and a run with `krun_log_level` raised.