bug: Podman driver picks the wrong host IP for the supervisor callback on multi-homed Linux hosts (regression from #2942)

[jgarciao]

User Story

As a user running the local Podman driver on a multi-homed Linux laptop (wired dock + Wi-Fi + Tailscale), I want openshell sandbox create to work without hand-editing the gateway config or bringing interfaces down.

Problem Statement

On a host with multiple network interfaces, the Podman driver and Podman disagree on which host IP host.containers.internal points to, so the in-container supervisor can't reach the gateway and provisioning fails.

• The gateway binds its callback listener to the default-route interface plus loopback: 192.168.1.27:17670 and 127.0.0.1:17670.
• The driver tells the supervisor to dial host.containers.internal:17670 and injects --add-host host.containers.internal:host-gateway.
• Podman resolves host-gateway to a different interface — here the Tailscale address 100.64.0.2 (wt0, CGNAT 100.64.0.0/10), where nothing listens. Bringing Wi-Fi down didn't help; Podman then used the Tailscale IP instead of the wired one.
• The supervisor's policy fetch fails, it exits 1, the sandbox container follows, and the sandbox enters Error.

The reported error is also misleading: ContainerExited: Container exited with code 0 refers to the sandbox container reacting to its supervisor dying. The real failure is only in the openshell-supervisor-<id> container logs.

Impact / Why This Matters

• Consequence: sandbox creation fails outright on multi-homed hosts (dock + Wi-Fi + VPN/mesh laptops are common), and the surfaced error points at the wrong container, so it's hard to diagnose.

• Workaround: set host_gateway_ip = "127.0.0.1" under [openshell.drivers.podman] in ~/.config/openshell/gateway.toml, then systemctl --user restart openshell-gateway. This works because the supervisor uses --network host, so loopback reaches the gateway's 127.0.0.1:17670 listener, and both host.containers.internal and 127.0.0.1 are cert SANs. The sandbox then reaches Ready.

  [openshell.drivers.podman]
  host_gateway_ip = "127.0.0.1"

• Why insufficient: it relies on an undocumented field and non-obvious reasoning about host-network loopback; it isn't discoverable from the error. The default should just work on multi-homed hosts.

Confirmed Regression (#2942)

Confirmed by version bisection: this is a regression introduced by PR #2942 (RFC 0012 sandbox architecture).

• 0.0.116 (does not contain feat(isolation): implement the RFC 0012 sandbox architecture #2942; tagged 2026-08-28): openshell sandbox create reaches a working sandbox shell with default config (no host_gateway_ip) on this multi-homed host. ✅
• 0.0.117-dev / current main (contains feat(isolation): implement the RFC 0012 sandbox architecture #2942, merged 2026-09-16 as c1f2e7189, first released in v0.1.0-pre.2): the same default config fails with Error, and only the host_gateway_ip = "127.0.0.1" workaround makes it succeed. ❌
• The gateway binds the same two listeners (192.168.1.27 + 127.0.0.1) on both versions, so the DefaultRouteInterface listener is not the trigger — the change is the callback path.

Mechanism:

• feat(isolation): implement the RFC 0012 sandbox architecture #2942 split the sandbox into two containers and moved the gateway callback onto a host-networked supervisor (crates/openshell-driver-podman/src/container.rs: supervisor.netns.nsmode = "host"). Previously a single container made the callback.

• The callback machinery predates feat(isolation): implement the RFC 0012 sandbox architecture #2942 and assumes bridge/pasta semantics: the DefaultRouteInterface listener negotiation is from feat(compute): negotiate gateway callback listeners #2492, and the host-gateway --add-host injection from fix(podman): avoid host-gateway on macOS machines #1637.

• Under the same --add-host host.containers.internal:host-gateway, the network mode changes what the alias resolves to:

  Network mode	host.containers.internal resolves to
  Bridge (--network openshell) — pre-feat(isolation): implement the RFC 0012 sandbox architecture #2942 style	169.254.1.2 (link-local host-gateway)
  Host netns (--network host) — feat(isolation): implement the RFC 0012 sandbox architecture #2942 supervisor	non-link-local host IP (here, the Tailscale interface)

• The supervisor even warns about this: host.openshell.internal maps to a non-link-local IP; trusted-gateway SSRF exemption disabled. The callback path was designed around a link-local host-gateway (bridge/pasta), which host networking does not provide. On a single-homed host the host-netns resolution still lands on a reachable IP; on a multi-homed host it selects the wrong interface, where nothing listens.

• Note: the host_gateway_ip = "127.0.0.1" workaround only works because the supervisor is now host-networked (an artifact of the feat(isolation): implement the RFC 0012 sandbox architecture #2942 change) and therefore shares the host loopback.

Acceptance Criteria

• A multi-homed Linux host reaches Ready with the Podman driver using default config (no manual host_gateway_ip).
• The supervisor reaches the gateway regardless of which interface Podman picks for host-gateway.
• When the supervisor can't reach the gateway, the phase error reports its failure reason, not ContainerExited: code 0.
• Covered by a test, or the multi-homed host_gateway_ip guidance is documented.

Reproduction Steps

1. On a Linux host whose interfaces differ from Podman's host-gateway resolution (e.g. wired dock + Wi-Fi + Tailscale), configure the Podman driver.
2. Run openshell sandbox create --provider <any>.
3. The sandbox enters Error during provisioning.

Confirm the mismatch:

ss -ltn | grep 17670
#   LISTEN ... 192.168.1.27:17670   (default-route iface)
#   LISTEN ...    127.0.0.1:17670

podman run --rm --network host --add-host host.containers.internal:host-gateway \
  registry.fedoraproject.org/fedora-minimal:latest getent hosts host.containers.internal
#   100.64.0.2  host.containers.internal   <-- Tailscale wt0, nothing listening there

Environment

• OpenShell: 0.0.117-dev.167+g7e7a8d561 (development build; regression absent in 0.0.116), installed with:

  curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | OPENSHELL_VERSION=dev sh

• OS / kernel: Fedora Linux 44 (Workstation) / 7.2.5-200.fc44.x86_64
• Podman: 5.8.4 (netavark)
• Driver: podman (rootless); gateway runs as the openshell-gateway systemd user service (native install, not containerized)
• Network: multi-homed — wired dock 192.168.1.27, Wi-Fi 192.168.1.75 (same subnet), Tailscale wt0 in 100.64.0.0/10

Related

Related: #2540, #1952, and the macOS cases #1519 / #1634. Distinct from #1909 (containerized gateway on Fedora 44).

Logs

Supervisor container (openshell-supervisor-<id>) — the actual failure:

WARN openshell_supervisor: Policy fetch failed, retrying
Error: × Policy fetch failed after 5 attempts: failed to connect to OpenShell server

[jgarciao]

CC: @elezar

[elezar]

📋 triage-agent

Triage Assessment

Classification: validated-bug

Summary

Issue #3412 is a confirmed regression in the local Linux Podman callback path. The report provides a before/after version bisection, concrete listener and resolver addresses, supervisor logs, and a discriminating loopback workaround. Current main retains the reported mismatch, so the evidence is sufficient to validate the bug without requesting more information.

Investigation

PR #2942 moved the callback-capable supervisor into the host network namespace. However, the default Podman endpoint still uses host.containers.internal, an empty host_gateway_ip still injects Podman's host-gateway mapping, and rootless listener negotiation still selects the host default-route interface. On a multi-homed host, Podman's independently selected host-gateway address can therefore differ from the gateway listener address, exactly as reported.

The host_gateway_ip = "127.0.0.1" workaround is consistent with the current architecture: the supervisor shares the host network namespace, the gateway already listens on loopback, and the generated certificate covers the alias and loopback address. The workaround is not an adequate default because it depends on non-obvious topology details and the surfaced workload exit does not identify the supervisor callback failure.

PR #2874 is related but does not address this regression. It handles a rootful Podman startup failure where an exact managed-bridge address is not yet locally bindable. Issue #3412 occurs after listeners bind successfully, in rootless mode, because the host-networked supervisor resolves host-gateway to a different interface. PR #2874 does not change callback endpoint selection, host entries, or supervisor failure reporting.

No duplicate was found. Issue #2540 is adjacent rootless slirp4netns relay work based on a callback originating in an isolated rootless network namespace; #3412 concerns the post-#2942 host-networked supervisor. Issues #1952, #1519, #1634, and #1909 cover different ownership, macOS, or containerized-gateway cases.

The misleading ContainerExited: code 0 result is also consistent with current watcher behavior: when the supervisor exits, reconciliation stops and re-inspects the workload, then reports the workload's exit state rather than the supervisor's policy-fetch failure.

Impact Signals

• Affected users/scope: Linux users of the local Podman driver on multi-homed hosts; rootless Fedora 44 with Podman 5.8.4 is confirmed. The rootful breadth is not yet measured.
• Regression: Yes; reported absent in 0.0.116 and introduced by PR feat(isolation): implement the RFC 0012 sandbox architecture #2942. Current main retains the relevant behavior.
• Workaround: Available by setting host_gateway_ip = "127.0.0.1", but topology-specific, non-obvious, and obscured by misleading diagnostics.
• Evidence quality: High. The report includes a version bisection, exact reproduction commands and addresses, logs, a working discriminating override, and code-level causal alignment. The triage did not independently recreate the multi-homed Fedora environment.

Human Decision Required

Decide whether OpenShell should address this issue. If yes, apply
state:accepted, associate it with a roadmap item, or do both, and decide
whether the work remains human-owned. Either action records acceptance;
roadmap placement additionally records sequencing.
To queue investigation or planning for an unattended agent, also apply
agent:plan-requested. You can instead directly ask an agent to use
create-spike or build-from-issue on this issue; the agent will warn about
missing expected workflow labels and continue without changing them. If no,
close it as not planned and record the rationale.

[elezar]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: Medium
Confidence: High — the callback topology after PR #2942 is clear from the current code

Summary

Update the existing PR #2874 implementation so callback routing follows the callback producer's network namespace rather than Podman's rootful/rootless mode. Native Linux supervisors use the host network namespace, so automatic callbacks will use the existing gateway aliases mapped to host loopback and request a loopback listener; Podman Machine and explicit remote endpoints keep their platform-specific routes.

Scope

• crates/openshell-driver-podman/src/driver.rs: select loopback for automatic Linux callbacks, remove bridge-gateway discovery and the rootless network-helper gate from this path, preserve explicit remote endpoints and explicit host-gateway overrides.
• crates/openshell-driver-podman/src/container.rs: map the supervisor's driver-owned aliases to 127.0.0.1 for the automatic Linux host-network route.
• crates/openshell-driver-podman/src/config.rs: document the updated empty/default host_gateway_ip behavior.
• proto/compute_driver.proto, crates/openshell-server/, and Docker driver compatibility changes: remove PR fix(gateway)!: handle delayed Podman bridge listeners #2874's delayed-bind protocol/server machinery because the managed bridge is no longer the callback origin after PR feat(isolation): implement the RFC 0012 sandbox architecture #2942.
• architecture/gateway.md, crates/openshell-driver-podman/NETWORKING.md, docs/reference/gateway-config.mdx, docs/reference/sandbox-compute-drivers.mdx, and skills/debug-openshell-cluster/SKILL.md: replace delayed-bind guidance with the callback-origin invariant and deployment boundaries.

Implementation Steps

1. Remove the delayed-bind protobuf/server implementation and restore the existing exact-address API.
2. Make automatic Linux Podman callbacks request loopback for both rootful and rootless operation; retain explicit remote and Podman Machine behavior.
3. Map automatic Linux supervisor aliases to loopback and add serialized-spec regression coverage.
4. Update architecture, configuration, compute-driver, and troubleshooting documentation.
5. Run focused Podman driver and gateway-listener tests, repository pre-commit checks, and the existing Fedora rootless/rootful validation paths where available.

Test Plan

• Unit tests: Cover rootful/rootless/slirp automatic loopback selection, explicit host_gateway_ip, explicit remote endpoints, loopback host entries, and absence of managed-network gateway inspection.
• Integration tests: Run existing gateway listener tests for primary-listener reuse and callback-only loopback binding.
• E2E tests: Reuse the existing Fedora rootless and rootful conformance paths without adding CI or conformance tooling to PR fix(gateway)!: handle delayed Podman bridge listeners #2874.

Risks & Open Questions

• Automatic Linux loopback assumes a native co-located gateway and Podman host. Containerized or remote gateways must configure an explicitly routable grpc_endpoint and appropriate gateway exposure.
• Podman Machine must retain its gvproxy route because supervisor loopback is inside the VM.
• The misleading ContainerExited: code 0 diagnostic requires an explicit supervisor-derived terminal condition and possibly structured failure propagation. Keep that lifecycle/diagnostics work separate from PR fix(gateway)!: handle delayed Podman bridge listeners #2874; the updated PR fixes the callback regression but does not claim that diagnostic criterion.

Documentation Impact

• Update docs/reference/gateway-config.mdx because the effective default semantics of empty host_gateway_ip change on Linux.
• Update the Podman compute-driver, architecture, networking, and troubleshooting documentation listed above.
• No SELinux or AppArmor-specific behavior is introduced.

---

Revision 1 — initial plan for updating PR #2874

[politerealism]

I'm picking this up.

[jgarciao]

Verified fixed by #3365 ✅

Built the gateway + runtime images from #3365 and re-ran sandbox create with default config (no host_gateway_ip) on a genuinely multi-homed host — wired dock (enp0s13f0u2, default route metric 100), Wi-Fi (wlp9s0f0, metric 600), and Tailscale (wt0) all up. The sandbox reaches Ready.

Why #3365 fixes it

The fix changes how the Podman driver picks the supervisor's gateway endpoint. Since RFC 0012 the supervisor runs host-networked (--network host), so it shares the host's loopback. #3365 adds select_grpc_endpoint(), which on a Linux host returns http://127.0.0.1:<gateway-port> instead of the host.containers.internal alias:

// PodmanEndpointEnvironment
LinuxHost     => "127.0.0.1",
PodmanMachine => "host.containers.internal",

Because the endpoint is now loopback, Podman's host-gateway resolution is off the callback path entirely — it no longer matters which interface Podman would have mapped host-gateway to (the wired IP, or the Tailscale IP as in the original report). That's the root cause removed, not worked around.

The old callback machinery this issue described — GetGatewayListenerRequirements negotiation, the DefaultRouteInterface listener, and nonlocal bind support — is deleted in the same PR.

Verification evidence

# gateway (driver) log
INFO openshell_driver_podman::driver: Auto-detected gRPC endpoint grpc_endpoint=http://127.0.0.1:18080 tls=false

# supervisor container env
OPENSHELL_ENDPOINT=http://127.0.0.1:18080

# supervisor container: Health=healthy, no "Policy fetch failed" lines

For contrast, a main build on the same host hands the supervisor OPENSHELL_ENDPOINT=http://host.containers.internal:18080; that only works when Podman's host-gateway happens to resolve to the interface the gateway binds, which is exactly the fragile behavior this issue is about.

Acceptance criteria

• Multi-homed Linux host reaches Ready with default config (no manual host_gateway_ip).
• Supervisor reaches the gateway regardless of Podman's host-gateway resolution (loopback removes that dependency).
• Covered by a test — grpc_endpoint_uses_loopback_on_linux asserts the Linux endpoint selection.