Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
| `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers |
| `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers |
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) |
| `crates/openshell-nft-ruleset/` | nftables rulesets | Shared per-sandbox NAT + default-deny nftables ruleset generation used by compute drivers |
| `crates/openshell-rootfs/` | Rootfs providers | Rootfs provider boundary for compute drivers; containerd-backed provider for image pull/unpack and snapshots |
| `crates/openshell-prover/` | Policy prover | Policy verification and proof generation |
| `crates/openshell-server-macros/` | Server macros | Compile-time helpers for gateway RPC authorization |
| `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution |
Expand Down
81 changes: 80 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions architecture/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ collect telemetry: `openshell-server` (gateway), `openshell-sandbox`
(supervisor), and `openshell-driver-vm`. Every crate depends on
`openshell-core` with `default-features = false`, so the binary crate's feature
is the single switch that enables `openshell-core/telemetry` for its build
graph. In-process drivers (`docker`, `kubernetes`, `podman`) inherit the
gateway's setting through feature unification and carry no passthrough.
graph. In-process drivers (`docker`, `kubernetes`, `podman`, `oci`) inherit
the gateway's setting through feature unification and carry no passthrough.

Building a binary with `--no-default-features` compiles out telemetry entirely:
no endpoint, no telemetry HTTP client, and no emission code. With telemetry
Expand Down
27 changes: 25 additions & 2 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,25 @@ The gateway records driver identity and version from the startup capability
response. Elevated gateway info reports that initialized driver snapshot instead
of re-querying drivers on each request.

## Rootfs Provisioning Boundary

Rootfs provisioning — resolving an OCI image reference into a root filesystem
a sandbox can be launched from — is a provider concern, separate from sandbox
provisioning (namespaces, cgroups, VM boot). `openshell-rootfs` defines that
boundary: a provider produces provider-neutral `PreparedRootfs` mounts keyed
by a per-sandbox key, and owns image resolution, acquisition, unpacking, and
the lifetime of whatever backend resources realize the root filesystem.
Backend concepts (containerd snapshots, leases, chain IDs) must stay behind
the provider and out of the compute-driver contract, so a daemon-backed
provider can be replaced by a daemonless one without redesigning the
provisioner on top of it.

`ContainerdRootfsProvider` is the first implementation, backed by a
system-provided containerd used only for image pull/unpack and snapshot
management. The OCI driver consumes it; the VM driver's image acquisition
and ext4 materialization pipeline is the same boundary shape, and converging
it onto the shared types is tracked follow-up work.

## Runtime Summary

| Runtime | Best fit | Sandbox boundary | Notes |
Expand All @@ -38,12 +57,14 @@ of re-querying drivers on each request.
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
| OCI | Early/opt-in kernel-primitive isolation against an existing system `containerd`. | Container built from Linux namespaces + cgroups v2. | In-process driver, like Docker/Podman/Kubernetes. The driver itself — not containerd — spawns the configured `runtime_binary` (`runc` by default, `crun`, or another OCI-runtime-spec-compatible binary) directly through its `create`/`start`/`state`/`delete` CLI contract; containerd is used only for image pull/unpack and snapshot management — consumed through the `openshell-rootfs` provider boundary (see Rootfs Provisioning Boundary above), which protects each sandbox's snapshot from containerd's GC via a lease since no containerd `Container`/`Task` is ever created — and never bundled. Sandboxes are tracked by scanning the driver's own state directory rather than querying containerd or the runtime for a global list. Per-sandbox network namespace + veth pair + nftables ruleset (shared `openshell-nft-ruleset` crate, also used by the VM driver). GPU support is kernel-device-node passthrough only. Rootless mode is implemented but not yet functional (defaults off) — see `crates/openshell-driver-oci/README.md`. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, `kubernetes`, and `oci` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

Per-sandbox CPU and memory values currently enter the driver layer through
template resource limits. Docker and Podman apply them as runtime limits.
Kubernetes mirrors each limit into the matching request. VM accepts the fields
but currently ignores them.
but currently ignores them. The OCI driver applies them as cgroup v2 CPU quota/period
and memory limits in the generated OCI spec.

Docker and Podman also accept per-sandbox driver-config mounts for existing
runtime-managed named volumes and tmpfs mounts. Podman additionally accepts
Expand Down Expand Up @@ -76,6 +97,7 @@ Runtime-specific implementation notes belong in the driver crate README:
- `crates/openshell-driver-podman/README.md`
- `crates/openshell-driver-kubernetes/README.md`
- `crates/openshell-driver-vm/README.md`
- `crates/openshell-driver-oci/README.md`

The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun
executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves
Expand All @@ -91,6 +113,7 @@ The supervisor must be available inside each sandbox workload:
| Podman | Read-only OCI image volume containing the supervisor binary. |
| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
| VM | Embedded in the guest rootfs bundle. |
| OCI | Bind-mounted from a host path (`supervisor_binary_path`), not sourced from an image like the other in-tree drivers — see the driver README's Known Gaps. |
| Extension | Defined by the out-of-tree driver. |

Driver-controlled environment variables must override sandbox image or template
Expand Down
9 changes: 8 additions & 1 deletion crates/openshell-core/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ pub enum ComputeDriverKind {
Vm,
Docker,
Podman,
Oci,
}

impl ComputeDriverKind {
Expand All @@ -97,6 +98,7 @@ impl ComputeDriverKind {
Self::Vm => "vm",
Self::Docker => "docker",
Self::Podman => "podman",
Self::Oci => "oci",
}
}
}
Expand Down Expand Up @@ -137,8 +139,9 @@ impl FromStr for ComputeDriverKind {
"vm" => Ok(Self::Vm),
"docker" => Ok(Self::Docker),
"podman" => Ok(Self::Podman),
"oci" => Ok(Self::Oci),
other => Err(format!(
"unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman"
"unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman, oci"
)),
}
}
Expand Down Expand Up @@ -1001,6 +1004,10 @@ mod tests {
"docker".parse::<ComputeDriverKind>().unwrap(),
ComputeDriverKind::Docker
);
assert_eq!(
"oci".parse::<ComputeDriverKind>().unwrap(),
ComputeDriverKind::Oci
);
}

#[test]
Expand Down
4 changes: 4 additions & 0 deletions crates/openshell-core/src/telemetry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ pub enum TelemetryComputeDriver {
Kubernetes,
Podman,
Vm,
Oci,
Unknown,
}

Expand All @@ -172,6 +173,7 @@ impl TelemetryComputeDriver {
Self::Kubernetes => "kubernetes",
Self::Podman => "podman",
Self::Vm => "vm",
Self::Oci => "oci",
Self::Unknown => "unknown",
}
}
Expand All @@ -183,6 +185,7 @@ impl TelemetryComputeDriver {
"k8s" | "kubernetes" => Self::Kubernetes,
"podman" => Self::Podman,
"vm" => Self::Vm,
"oci" => Self::Oci,
_ => Self::Unknown,
}
}
Expand All @@ -194,6 +197,7 @@ impl TelemetryComputeDriver {
Some(crate::ComputeDriverKind::Kubernetes) => Self::Kubernetes,
Some(crate::ComputeDriverKind::Podman) => Self::Podman,
Some(crate::ComputeDriverKind::Vm) => Self::Vm,
Some(crate::ComputeDriverKind::Oci) => Self::Oci,
None => Self::Unknown,
}
}
Expand Down
58 changes: 58 additions & 0 deletions crates/openshell-driver-oci/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

[package]
name = "openshell-driver-oci"
description = "OCI/runc-based compute driver for OpenShell, backed by a system-provided containerd"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true

[lib]
name = "openshell_driver_oci"
path = "src/lib.rs"

[[bin]]
name = "openshell-driver-oci"
path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-nft-ruleset = { path = "../openshell-nft-ruleset" }
openshell-policy = { path = "../openshell-policy" }
# Rootfs provider boundary. The containerd-backed provider lives there:
# this driver never talks to containerd directly (no containerd-client
# dependency here) and consumes only provider-neutral PreparedRootfs
# mounts. The driver itself, not containerd, spawns the configured
# low-level OCI runtime (runc/crun/etc.) directly (see src/runtime.rs).
openshell-rootfs = { path = "../openshell-rootfs" }

# OCI runtime-spec types (namespaces, cgroups v2 resources, seccomp).
oci-spec = { version = "0.10", default-features = false, features = [
"runtime",
] }

tokio = { workspace = true }
tokio-stream = { workspace = true }
tonic = { workspace = true, features = ["transport"] }
prost = { workspace = true }
prost-types = { workspace = true }
futures = { workspace = true }
clap = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
miette = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
nix = { workspace = true }
rand = { workspace = true }
thiserror = { workspace = true }
sha2 = "0.10"

[dev-dependencies]
tempfile = "3"

[lints]
workspace = true
Loading
Loading