operator: per-Site image registry override for site-node workloads

[Philip Lombardi (plombardi89)]

Problem

Some Sites sit on networks that cannot reach the operator's default container
registry (e.g. ghcr.io/azure). Today every Site pulls the operator-managed
component images from that one registry, so a network-isolated Site cannot start
the workloads that run on its nodes. We need a way, per Site, to pull those images
from a registry the Site's network can reach.

Solution

Add spec.imageRegistry to the Site CR: a full image-repository prefix (host
plus any org/namespace path), identical in meaning to the operator-wide
UNBOUNDED_IMAGE_REGISTRY. When set, the operator-managed workloads that run on
that Site's nodes resolve their images from that prefix instead of the operator
default. Empty = operator default (byte-for-byte unchanged from today).

apiVersion: unbounded-cloud.io/v1alpha3
kind: Site
metadata:
  name: isolated-rack
spec:
  nodeCidrs: ["10.20.0.0/16"]
  imageRegistry: registry.corp.internal/unbounded

Architecture: base + per-Site DaemonSet split

The override applies to the four operator-managed workloads that land on a Site's
nodes: unbounded-net-node, gantry, metalman, and the
unbounded-storage-supervisor. Control-plane components (net controller, machina)
are unaffected.

• metalman and unbounded-storage-supervisor already run per-Site; they now
  resolve their image through component.ConfigForSite.
• unbounded-net-node and gantry were single cluster-wide DaemonSets. Each is
  split into a base DaemonSet that runs only on un-Sited nodes (control
  plane, etc.) using the operator-wide registry, plus a per-Site DaemonSet
  (<component>-<site>) node-affined to the Site, owner-referenced to the Site,
  using the Site's registry.

Node affinity partitions scheduling so every node runs exactly one copy. Nodes are
matched to a Site by the existing net-controller labeling
(unbounded-cloud.io/site), and the existing nodeCidrs overlap guards keep a
node in at most one Site, so registry selection is unambiguous.

Safety / correctness hardening

This split of host-network, single-owner dataplane agents surfaced several
correctness concerns (raised in review); the bulk of the PR addresses them:

• Canonical-authoritative site affinity. SiteNodeAffinity now matches the
  canonical unbounded-cloud.io/site label, falling back to the deprecated label
  only when the canonical one is absent. A node briefly carrying conflicting
  values (canonical=A, deprecated=B) can no longer match two Sites and
  double-schedule two privileged agents.
• Per-Site net config that tracks shared unless customized. Each
  unbounded-net-config-<site> records the shared-config hash it was seeded from;
  while unchanged it re-seeds when the shared config changes, and once a Site edits
  it the copy is preserved. This closes the window where a per-Site config seeded
  from an embedded default before the intended (or migrated) shared config lands
  would stay stale forever.
• Fail-safe fleet cutover. net and gantry apply their per-Site DaemonSets
  before narrowing the base to un-Sited nodes, so a per-Site apply failure
  leaves the blanket base covering the fleet rather than stranding Sited nodes.
  net-node is host-network / single-owner, so the handoff is break-before-make;
  because net-node does not tear down its dataplane on shutdown, established
  traffic survives the swap and only new-pod networking briefly stalls (equivalent
  to a normal net-node rollout, which already runs at maxUnavailable: 100%).
• Gantry per-Site config preserved on opt-out. A temporary
  gantry.enabled=false deletes only the DaemonSet; the Site-owned config
  (upstream_registries, credentials) is kept and GC'd only when the Site is
  deleted.
• Legacy reaper gate. The migration reaper now requires every Site's per-Site
  unbounded-net-node-<site> DaemonSet to be present and current before it reaps
  the legacy blanket net-node, mirroring the storage gate.

Caveats / known limitations

• The referenced registry must already host copies of the operator's component
  images at the operator's version; the operator only repoints references, it does
  not mirror images.
• Pulls are anonymous (per-Site image pull secrets are a follow-up).
• gantry's third-party busybox init image (mcr.microsoft.com/...) is not
  repointed.
• unbounded-net-node uses imagePullPolicy: Always, so the Site registry must
  be reachable on every node (re)start.

Follow-ups

• Per-Site gantry mesh scoping (membership + DHT) and workload image-naming
  convention - hack/discussion/gantry-per-site.md.
• Per-Site image pull secrets.
• Optional paced (batched) cutover if very large single-Site fleets need the
  new-pod-networking gap bounded to a rolling subset of nodes.

[Copilot]

Pull request overview

Adds a per-Site container image registry override (spec.imageRegistry) so node-scoped operator-managed workloads can be pulled from a site-local/mirrored registry when the default registry is unreachable.

Changes:

• Introduces SiteSpec.ImageRegistry and component.ConfigForSite to resolve component images per Site with fallback to the operator-wide registry.
• Splits unbounded-net-node and gantry into a base DaemonSet for un-Sited nodes plus per-Site DaemonSets and per-Site seeded ConfigMaps.
• Updates docs and the Site CRD to document/define spec.imageRegistry.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/operator/components/storage/storage.go	Resolves storage supervisor images via per-Site config.
internal/operator/components/storage/storage_test.go	Adds coverage for Site-level registry override behavior in storage mutation.
internal/operator/components/net/net.go	Splits net node DaemonSet into base + per-Site, adds per-Site config seeding/ownership, and applies un-Sited affinity to base.
internal/operator/components/net/net_test.go	Adds tests for base un-Sited affinity, per-Site DaemonSet/config behavior, and reconcile fan-out.
internal/operator/components/metalman/metalman.go	Resolves metalman image via per-Site config.
internal/operator/components/metalman/metalman_test.go	Adds coverage for Site-level registry override behavior in metalman deployment.
internal/operator/components/gantry/gantry.go	Splits gantry DaemonSet into base + per-Site, adds per-Site config seeding/cleanup, and applies un-Sited affinity to base.
internal/operator/components/gantry/gantry_test.go	Adds tests for base un-Sited affinity, per-Site DaemonSet/config behavior, and opted-out cleanup.
internal/operator/component/env.go	Adds ConfigForSite and UnsitedNodeAffinity helpers.
internal/operator/component/env_test.go	Adds tests for ConfigForSite and UnsitedNodeAffinity.
docs/net/custom-resources.md	Documents spec.imageRegistry semantics, behavior, and caveats.
docs/content/reference/networking/custom-resources.md	Adds spec.imageRegistry reference entry.
deploy/machina/crd/unbounded-cloud.io_sites.yaml	Regenerates/updates the Site CRD schema to include imageRegistry.
api/machina/v1alpha3/site_types.go	Adds ImageRegistry field to the Site API type with detailed docs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (5)

internal/operator/components/net/net.go:533

• The per-Site net ConfigMap created in code has no labels/annotations (only name/namespace/ownerRefs). If ManagedConfigPredicate(...) relies on management labels/annotations (common for operator self-heal/watch filtering), changes to these per-Site ConfigMaps may not trigger reconciles. Consider copying the shared ConfigMap’s management metadata (e.g., labels/annotations) onto the per-Site ConfigMap (or explicitly setting whatever managed markers ManagedConfigPredicate expects) when seeding/creating.

	cm := &corev1.ConfigMap{
		TypeMeta:   metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"},
		ObjectMeta: metav1.ObjectMeta{Name: SiteConfigName(site.Name), Namespace: env.Namespace, OwnerReferences: []metav1.OwnerReference{component.SiteOwnerReference(site)}},
		Data:       payload.Data,
		BinaryData: payload.BinaryData,
	}

internal/operator/components/gantry/gantry.go:563

• Same issue as net: per-Site gantry ConfigMaps are created without any management labels/annotations. Since SetupWatches uses ManagedConfigPredicate for per-Site configs, ensure the generated ConfigMaps carry the metadata required for them to be treated as managed/watched (e.g., copy labels/annotations from the shared gantry-config, or set explicit managed markers).

	cm := &corev1.ConfigMap{
		TypeMeta:   metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"},
		ObjectMeta: metav1.ObjectMeta{Name: SiteConfigName(site.Name), Namespace: env.Namespace, OwnerReferences: []metav1.OwnerReference{component.SiteOwnerReference(site)}},
		Data:       payload.Data,
		BinaryData: payload.BinaryData,
	}

internal/operator/components/net/net.go:252

• This reads and decodes the embedded YAML on every reconcile. Since the base DaemonSet is static, consider caching the decoded base object (e.g., package-level var + sync.Once) and returning a DeepCopy() from the cached value. This reduces allocations/CPU in frequently-triggered reconciles (especially now that reconciles also fan out per Site).

func baseNodeDaemonSet() (*unstructured.Unstructured, error) {
	data, err := fs.ReadFile(netmanifests.Manifests, nodeDaemonSetManifest)
	if err != nil {
		return nil, fmt.Errorf("read base net node manifest: %w", err)
	}

	decoder := utilyaml.NewYAMLOrJSONDecoder(bytes.NewReader(data), 4096)

	for {
		obj := &unstructured.Unstructured{}
		if err := decoder.Decode(obj); err != nil {
			if errors.Is(err, io.EOF) {
				break
			}

			return nil, fmt.Errorf("decode base net node manifest: %w", err)
		}

		if obj.GetKind() == "DaemonSet" && obj.GetName() == nodeName {
			return obj, nil
		}
	}

	return nil, fmt.Errorf("base net node DaemonSet %q not found in manifest", nodeName)
}

internal/operator/components/storage/storage_test.go:149

• This test ignores NestedSlice errors/ok and then type-asserts/indexes directly, which will panic (and obscure the real failure) if mutation doesn’t produce the expected structure. Prefer checking err, ok, and len(containers) and failing the test with a clear message before indexing/type asserting.

	containers, _, _ := unstructured.NestedSlice(obj.Object, "spec", "template", "spec", "containers")
	if got := containers[0].(map[string]any)["image"]; got != "registry.corp.internal/unbounded/unbounded-storage-supervisor:v1.2.3" {
		t.Fatalf("image = %q, want site-registry storage supervisor", got)
	}

internal/operator/components/net/net_test.go:471

• Similar to the storage test: ignoring NestedSlice’s ok/err and indexing/type-asserting can cause panics that hide why the test failed. It’s more robust to assert err == nil, ok == true, len(containers) > 0, and that the element is a map[string]any before reading image.

	containers, _, _ := unstructured.NestedSlice(perSite.Object, "spec", "template", "spec", "containers")
	if got := containers[0].(map[string]any)["image"]; got != "registry.corp.internal/unbounded/unbounded-net-node:v1.2.3" {
		t.Fatalf("per-site node image = %q, want site registry", got)
	}

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Philip Lombardi (plombardi89)]

Abandoned after team discussion about alternatives.