Skip to content

feat(podman): isolate workloads behind a separate supervisor - #3230

Open
drew wants to merge 19 commits into
codex/kubernetes-proxy-pod-isolationfrom
codex/podman-host-supervisor
Open

feat(podman): isolate workloads behind a separate supervisor#3230
drew wants to merge 19 commits into
codex/kubernetes-proxy-pod-isolationfrom
codex/podman-host-supervisor

Conversation

@drew

@drew drew commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add the Podman implementation of RFC 0012. The driver launches separate workload and supervisor containers, connects them through a private Unix socket, and denies direct workload egress.

The OpenShell Sandbox Protocol uses a pinned TLS server identity and launch-scoped sandbox JWT over the socket. Restart preserves the admitted workload configuration while replacing the session and protected authentication state.

Related Issue

Part of #1737.

Changes

  • Implement the shared isolation path in the existing Podman driver crate.
  • Provision the workload, supervisor, Unix socket, bootstrap material, and outer fence.
  • Keep supervisor tokens, client TLS material, and proxy credentials out of the workload container.
  • Preserve template and request environment variables while protecting OPENSHELL_* control values.
  • Apply native OCI PID and AppArmor settings.
  • Rotate generation and authentication material on restart.

Testing

  • mise run pre-commit
  • cargo test -p openshell-driver-podman --lib (226 passed)
  • Full Podman E2E delegated to remote CI

Checklist

  • Follows Conventional Commits
  • Authored commits are signed off (DCO)

Stack

  1. feat(isolation): implement the RFC 0012 sandbox architecture #2942 — RFC 0012 core architecture
  2. feat(vm): run the supervisor outside the guest workload #2945 — VM driver
  3. feat(docker): isolate workloads behind a companion supervisor #2965 — Docker driver
  4. feat(kubernetes): isolate workloads behind a dedicated supervisor #3144 — Kubernetes driver
  5. feat(podman): isolate workloads behind a separate supervisor #3230 — Podman driver (this PR)
  6. perf(isolation): add TCP and DNS benchmark harnesses #3229 — performance harnesses

@drew
drew requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 9, 2026 04:40
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

Label test:e2e applied for 355a410. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@drew
drew force-pushed the codex/podman-host-supervisor branch from 355a410 to 4ddf65a Compare September 9, 2026 05:03
@drew
drew force-pushed the codex/podman-host-supervisor branch from 4ddf65a to 6179a23 Compare September 9, 2026 05:14
@drew
drew force-pushed the codex/podman-host-supervisor branch 2 times, most recently from 342de37 to c009765 Compare September 9, 2026 05:38
@drew
drew force-pushed the codex/podman-host-supervisor branch from c009765 to a820116 Compare September 9, 2026 05:50
@drew
drew force-pushed the codex/podman-host-supervisor branch 3 times, most recently from c85c320 to 8e4ea44 Compare September 11, 2026 05:57

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s latest Podman compatibility fix replaces libpod-rejected tmpfs ownership options with a runtime-created material directory. The bounded critical-only delta review found no newly introduced Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: Fern gateway and compute-driver guidance remains at schema v2 with the actual Podman health-check contract.
  • Checks: Current-head Branch Checks and required E2E workflow are running; Helm Lint and Trivy Changes are green.
  • E2E: test:e2e is applied; current-head Branch E2E Checks run 34805732214 is in progress, so no /ok to test or rerun is needed.
  • Head SHA: 6bf038bc8f0de0bbe2397c125c09e5f97d50550b
  • Base SHA: d8f5390436d81c4a062e9aeb054361c245bc487a
  • Merge base SHA: d8f5390436d81c4a062e9aeb054361c245bc487a
  • Patch ID: 776bd8dce0d3e9f51c047e6456517894a468878c
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: 31c23cb2ff94de72db490b3c2068625732f70a64
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s latest Podman update replaces the libpod-dependent image-volume path with an extracted, verified runtime binary that is mounted read-only. I checked the single new author commit after separating the rebase-equivalent commits and upstream base changes; the bounded critical-only review found no newly introduced Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: Fern gateway and compute-driver guidance covers the current Podman configuration and isolation behavior.
  • Checks: Current-head Branch Checks are in progress; Helm Lint and Trivy Changes are green.
  • E2E: test:e2e is applied; current-head Branch E2E Checks run 34807045419 is in progress, so no /ok to test or rerun is needed.
  • Head SHA: 60ab7be71764b2b459a529d5a8f82d6bac132cc1
  • Base SHA: 83bbd7534fed5d9bc00f4957e8fd2807c75ea74c
  • Merge base SHA: 83bbd7534fed5d9bc00f4957e8fd2807c75ea74c
  • Patch ID: 536e57d721a1d09e0c1cd67d562dc051e5b46e88
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: 6bf038bc8f0de0bbe2397c125c09e5f97d50550b
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s latest update gives the external Podman driver an isolated data directory and carries that value through the parity manifest checks. I reviewed only this new author commit under the exhausted-budget critical-only policy; it introduces no new Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: No direct user-facing UX change in this delta; existing Fern coverage remains applicable.
  • Checks: Current-head Branch Checks and required E2E are active. Helm Lint and Trivy Changes are green; one E2E supervisor-image build job has failed, but the run is still active and its failed log is not yet available for triage.
  • E2E: test:e2e is applied and current-head Branch E2E Checks run 34808503037 is queued/active, so no /ok to test or rerun is currently needed.
  • Head SHA: 2524bee186a3e7fa8f320ac4d1cc94298a2c9e46
  • Base SHA: c6c3fccb3b666cc31ef8f0b266c1d38b5d4c9e84
  • Merge base SHA: c6c3fccb3b666cc31ef8f0b266c1d38b5d4c9e84
  • Patch ID: 6419fdbb31e093cea9545428c9a2f9c1f3721d6b
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: 60ab7be71764b2b459a529d5a8f82d6bac132cc1
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s latest parity correction passes the requested sandbox image to the external Podman driver while retaining the separately pinned runtime artifact. I reviewed only this author delta under the exhausted-budget critical-only policy; it introduces no new Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: No direct user-facing UX change in this delta; existing Fern coverage remains applicable.
  • Checks: Current-head Branch Checks and required E2E are active; Helm Lint and Trivy Changes are green.
  • E2E: test:e2e is applied and current-head Branch E2E Checks run 34809381625 is queued/active, so no /ok to test or rerun is needed.
  • Head SHA: 7223892f7dfa25dd263e454fac956ba95fd33ee2
  • Base SHA: c6c3fccb3b666cc31ef8f0b266c1d38b5d4c9e84
  • Merge base SHA: c6c3fccb3b666cc31ef8f0b266c1d38b5d4c9e84
  • Patch ID: 193c751e286d9429f29dfca7f8842e71d49a2bad
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: 2524bee186a3e7fa8f320ac4d1cc94298a2c9e46
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s current delta separates the requested workload image from the pinned sandbox-boundary runtime image and records both in parity evidence. I reviewed only this author delta under the exhausted-budget critical-only policy; it introduces no new Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: No direct user-facing UX change in this delta; existing Fern coverage remains applicable.
  • Checks: Current-head Branch Checks and required E2E are queued/running; Helm Lint and Trivy Changes are green.
  • E2E: test:e2e is applied and current-head Branch E2E Checks run 34810900154 is active, so no /ok to test or rerun is needed.
  • Head SHA: 93466075cc6e5d3c51c024c301fa9e8de59d55d1
  • Base SHA: d4d4ea74df654fdeb3770fed3281cab4e4d9606c
  • Merge base SHA: d4d4ea74df654fdeb3770fed3281cab4e4d9606c
  • Patch ID: d7987268c45575403a8dd7c3f6238415f6314c9e
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: 7223892f7dfa25dd263e454fac956ba95fd33ee2
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s current delta starts the fenced workload before creating the supervisor that joins its live user namespace. I reviewed only this author delta under the exhausted-budget critical-only policy; it introduces no new Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: No direct user-facing UX change in this delta; existing Fern coverage remains applicable.
  • Checks: Current-head Branch Checks and required E2E are active; Helm Lint and Trivy Changes are green. One Podman external-driver E2E job has failed, but the workflow is still active and GitHub has not made its log available for triage.
  • E2E: test:e2e is applied and current-head Branch E2E Checks run 34811906328 is queued/active, so no /ok to test or rerun is currently needed.
  • Head SHA: b69287ac826e1999061a2dfc67e03024f41aef9a
  • Base SHA: d4d4ea74df654fdeb3770fed3281cab4e4d9606c
  • Merge base SHA: d4d4ea74df654fdeb3770fed3281cab4e4d9606c
  • Patch ID: e376ef0f333d60d1751fc8fd915c77dedfac6c72
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: 93466075cc6e5d3c51c024c301fa9e8de59d55d1
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s current delta keeps the trusted supervisor in the caller’s user namespace for rootless Podman host networking and restores safe startup ordering after both containers receive their bootstrap material. I reviewed only this author delta under the exhausted-budget critical-only policy; it introduces no new Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: No direct user-facing UX change in this delta; existing Fern coverage remains applicable.
  • Checks: Current-head Branch Checks and required E2E are running; Helm Lint and Trivy Changes are green.
  • E2E: test:e2e is applied and current-head Branch E2E Checks run 34812916156 is active; both Podman E2E jobs are green, so no /ok to test or rerun is needed.
  • Head SHA: ff2213294d49257fcb3f877507688a1942157dbf
  • Base SHA: d4d4ea74df654fdeb3770fed3281cab4e4d9606c
  • Merge base SHA: d4d4ea74df654fdeb3770fed3281cab4e4d9606c
  • Patch ID: f0acc45190f2b661c2fe770ce8c80e488fa4b35b
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: b69287ac826e1999061a2dfc67e03024f41aef9a
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s current delta makes Podman start retries use the gateway’s durable generation identity and rejects a different generation when the workload and supervisor are already running. I reviewed only this author delta under the exhausted-budget critical-only policy; it introduces no new Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: No direct user-facing UX change in this delta; existing Fern coverage remains applicable.
  • Checks: Current-head Branch Checks and required E2E are running; Helm Lint, Trivy Changes, and GPU E2E are green.
  • E2E: test:e2e is applied and current-head Branch E2E Checks run 34814065912 is active; both Podman E2E jobs are green, so no /ok to test or rerun is needed.
  • Head SHA: 0c9ba2a7185937438c91e0062fccba5b3ccb1854
  • Base SHA: ff7f535a25df8e27964cadca75ab5c5c53e27dc5
  • Merge base SHA: ff7f535a25df8e27964cadca75ab5c5c53e27dc5
  • Patch ID: 23be403709fb9733c464222c209d6502f1bb5775
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: ff2213294d49257fcb3f877507688a1942157dbf
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s latest restart-recovery update rotates the running Podman workload and supervisor together when the recovered gateway supplies a fresh launch-authentication bundle for the existing generation. The bounded critical-only delta review found no newly introduced Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: Fern gateway and compute-driver guidance covers the current Podman configuration and isolation behavior.
  • Checks: Current-head Branch Checks and required E2E workflow are running; Helm Lint and Trivy Changes are green.
  • E2E: test:e2e is applied; current-head Branch E2E Checks run 34816526243 is in progress, so no /ok to test or rerun is needed.
  • Head SHA: 5bf6e66596eee4c04ab4c7262897b2f9e59cd874
  • Base SHA: df5c17f258f742f21fdcaeb66a49b2f805738229
  • Merge base SHA: df5c17f258f742f21fdcaeb66a49b2f805738229
  • Patch ID: 5a81f4d518fd816171d46c8bdf8a1d525cd95be9
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: 0c9ba2a7185937438c91e0062fccba5b3ccb1854
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The current head is a rebase-equivalent copy of the previously reviewed Podman patch (patch ID 5a81f4d518fd816171d46c8bdf8a1d525cd95be9), so no duplicate reviewer run was performed. The five durable findings remain resolved, and the required current-head Branch Checks and E2E workflows are queued.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: Fern gateway and compute-driver guidance remains at schema v2 with the actual Podman health-check contract.
  • Checks: Current-head Branch Checks and required E2E workflow are queued; Helm Lint and Trivy Changes are green.
  • E2E: test:e2e is applied; current-head Branch E2E Checks run 34864852928 is pending, so no /ok to test or rerun is needed.
  • Head SHA: af59258799e2df7821e2424872f9b774fe775236
  • Base SHA: 9c5031a40ee773a775acc5d517c2cc032e91c00c
  • Merge base SHA: 9c5031a40ee773a775acc5d517c2cc032e91c00c
  • Patch ID: 5a81f4d518fd816171d46c8bdf8a1d525cd95be9
  • Gator payload: 8
  • Review mode: already_reviewed (rebase-equivalent)
  • Previous reviewed SHA: 5bf6e66596eee4c04ab4c7262897b2f9e59cd874
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

drew added 19 commits September 14, 2026 09:29
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@drew’s latest update binds the Podman boundary configuration and supervisor runtime descriptor to the launch session’s rotation lineage. I reviewed only this new author commit under the exhausted-budget critical-only policy; it introduces no new Critical defect, and the five durable findings remain resolved.

Blocking findings:

  • No new blocking findings.

Carried findings:

  • None; GATOR-a83bc000-01 through GATOR-a83bc000-05 remain resolved.
Gator metadata
  • Validation: Maintainer-authored Podman implementation of the actively reviewed RFC 0012 architecture, scoped as layer 5 of 6.
  • Docs: No direct user-facing UX change in this delta; existing Fern coverage remains applicable.
  • Checks: Current-head Branch Checks and required E2E are active; Helm Lint and Trivy Changes are green. Both Podman E2E jobs are green; one Docker OIDC E2E job has failed while the workflow is still active.
  • E2E: test:e2e is applied and current-head Branch E2E Checks run 34869061921 is active, so no /ok to test or rerun is currently needed.
  • Head SHA: 238196fa56d5dab23133290c655c71ba717bf4e2
  • Base SHA: dc47f155324e6ae42b02392b13bd12aa62ea7b2b
  • Merge base SHA: dc47f155324e6ae42b02392b13bd12aa62ea7b2b
  • Patch ID: efd663098bba5719c57574a8ec67e29efcdc66cb
  • Gator payload: 8
  • Review mode: critical_only
  • Previous reviewed SHA: af59258799e2df7821e2424872f9b774fe775236
  • Review budget exhausted: yes
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:approval-needed Gator completed review; maintainer approval needed test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants