storage: add automatic disk discovery

[Jordan Olshevski (jveski)]

• discover safe NVMe namespaces when disks are not explicitly configured
• exclude mounted, swap-backed, held, denied, and unverifiable devices
• fall back to a configurable file-backed disk when no eligible device exists
• preserve explicit and supervisor-allocated disk configurations
• document automatic disk selection and fallback behavior

[Copilot]

Pull request overview

This PR introduces automatic disk selection for unbounded-storage when no explicit disks are configured, shifting the “default disk” behavior from the Go supervisor into the Rust daemon so the daemon can safely discover unused NVMe namespaces and fall back to a file-backed disk when none are eligible.

Changes:

• Add a new disk_discovery configuration surface (protobuf + Rust config defaults/validation) and implement safe NVMe namespace discovery with exclusions (mounted, swap, holders/device-mapper, denied, unverifiable).
• Update the daemon startup + reload path to resolve disks before projecting runtime state, and adjust CPU core planning to pin storage cores per resolved concrete block disks.
• Update the Go storage supervisor to preserve explicit/allocated disks, otherwise pass an empty disk list (and optionally only a fallback size override) through to the daemon; update deployment/docs accordingly.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
internal/storagesupervisor/render.go	Updates documentation/comments to reflect that empty disk lists are now intentional for daemon-side discovery.
internal/storagesupervisor/render_test.go	Adjusts tests to expect empty disks[] and (when annotated) disk_discovery.fallback sizing instead of supervisor-injected file disks.
internal/storagesupervisor/disks.go	Stops injecting a default file disk; preserves explicit/allocated disks and only overlays fallback size annotation onto disk_discovery when applicable.
deploy/unbounded-storage-supervisor/02-configmap.yaml.tmpl	Documents automatic disk discovery, denied paths, and fallback behavior (including legacy size annotation behavior).
cmd/unbounded-storage/src/topology/cores.rs	Adds for_host_with_storage to size storage cores based on concrete resolved block disks (not just NVMe controllers).
cmd/unbounded-storage/src/main.rs	Resolves disk set (explicit/automatic/fallback) before runtime projection and on reload; logs resolution; plans storage cores using resolved disk NUMA hints.
cmd/unbounded-storage/src/lib.rs	Exposes the new disk_discovery module.
cmd/unbounded-storage/src/disk_discovery.rs	Implements safe NVMe namespace discovery, disk materialization, caching, and related unit tests.
cmd/unbounded-storage/src/config/schema.rs	Adds defaults for disk_discovery fallback path/size and exposes a disk_discovery() accessor.
cmd/unbounded-storage/src/config/mod.rs	Re-exports DiskDiscoveryCfg from the config schema module.
cmd/unbounded-storage/src/config/load.rs	Validates disk_discovery denied paths and fallback file disk shape; adds related loader tests.
cmd/unbounded-storage/README.md	Documents automatic disk discovery configuration and runtime behavior (caching, pinning implications).
cmd/unbounded-storage/build.rs	Includes DiskDiscoveryCfg in schema generation.
cmd/unbounded-storage/ARCHITECTURE.md	Updates architecture docs to describe disk resolution/discovery in startup and core planning.
api/unbounded-storage/config.proto	Adds DiskDiscoveryCfg message and Config.disk_discovery field.
api/unbounded-storage/config.pb.go	Regenerates Go protobuf bindings for the new disk_discovery field/message.

Files not reviewed (1)

• api/unbounded-storage/config.pb.go: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.