Skip to content

Add local image support for kind clusters#43

Open
GrimmiMeloni wants to merge 35 commits into
mainfrom
mh_roxie-41
Open

Add local image support for kind clusters#43
GrimmiMeloni wants to merge 35 commits into
mainfrom
mh_roxie-41

Conversation

@GrimmiMeloni
Copy link
Copy Markdown
Collaborator

@GrimmiMeloni GrimmiMeloni commented Jan 26, 2026 •
edited
Loading

Summary

  • Automatically detects when deploying to kind clusters and checks for locally-built images in podman
  • Loads local images into kind before deployment, eliminating the need to push to quay.io during local development
  • Patches operator CSV to reference local image tags, handling dual-branding support (RHACS_BRANDING vs STACKROX_BRANDING)
  • Supports partial local image scenarios with graceful fallback to quay.io for missing images

Test Plan

  • Build stackrox images locally with podman
  • Set MAIN_IMAGE_TAG to match locally-built image tag
  • Deploy to kind cluster using roxie
  • Verify local images are detected (9/9 expected)
  • Verify images are loaded into kind cluster
  • Verify CSV is patched with correct image references
  • Verify deployment succeeds using local images
  • Test with collector (stackrox-io branding) when ROX_PRODUCT_BRANDING=RHACS_BRANDING to verify dual-branding fallback
  • Test with ROXIE_SKIP_LOCAL_IMAGES=true to verify feature can be disabled

Fixes #41

🤖 Generated with Claude Code

@GrimmiMeloni GrimmiMeloni self-assigned this Jan 26, 2026
@GrimmiMeloni GrimmiMeloni added the enhancement New feature or request label Jan 27, 2026
@GrimmiMeloni
Copy link
Copy Markdown
Collaborator Author

GrimmiMeloni commented Jan 27, 2026

@mclasmeier I have a couple of remarks:

  1. I cannot get this to greeen, as the "Docker Build and Push" workflow is generally broken. Maybe you can check and fix it? I had a look at the failure. It looks like the download of helm fails, but the URL seems valid. Not sure what is going on there.
  2. The docs folder contains the feature design description. While AI generated, it should still serve as a good starting point prior to reviewing the code.
  3. While AI assisted, I have done a full manual review of the changes in this PR to hopefully find any slop before reviewers.
  4. This change currently makes roxie try to use locally built images by default, with the option to skip local images via env var. In a sense this is a change in behavior, so we could also turn this around and make the use of local images an optional feature that has to explicitly be enabled. FWIW, even when local images are considered, roxie still transparently falls back to fetching from quay.io when it cannot find the images locally
  5. The code currently loads up to 4 images in parallel from the local system into the kind cluster. I am a bit torn on this optimization, as it makes the respective code more complex. I have tested things locally and the parallelism saves ~30s (1:55 vs 1:25) during the loading (at least on my 2021 MBP). In relative terms it is a 25% improvement. I kept it in for now, but would also be OK if we just serialize the loading of the images to keep it simple.

@mclasmeier
Copy link
Copy Markdown
Collaborator

mclasmeier commented Jan 28, 2026

Wow, thanks. I will look into this.
Regarding the image building suddenly being broken, there's a PR underway. As far as I understand it, this breakage is due to some changes on the side of the GitHub action build environments related to the filesystem mount attributes...

GrimmiMeloni and others added 25 commits March 4, 2026 16:21
@GrimmiMeloni @claude
Add design document for local image support in kind clusters
96dfa12 
Documents the design for automatically detecting and loading locally-built
container images into kind clusters during roxie deployments. This eliminates
the need to push images to quay.io during local development.

Key features:
- Automatic detection of kind clusters
- Branding-aware local image detection in podman
- Parallel image loading into kind
- Graceful fallback to quay.io when images aren't local
- Zero configuration required

Relates to #41

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
Add implementation plan for local image support
f44b0da 
Detailed step-by-step plan for implementing automatic local image
detection and loading for kind clusters. Includes:
- 9 tasks with TDD approach
- Unit tests for each component
- Integration into deployer workflow
- Comprehensive manual testing checklist

Relates to #41

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
Add branding support for local image detection
3c5998c 
Implements ROX_PRODUCT_BRANDING environment variable support to determine
which registry organization to use (rhacs-eng vs stackrox-io). Defaults
to RHACS branding for typical development workflows.

Relates to #41
@GrimmiMeloni @claude
Fix formatting in localimages package
4ea9d43 
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
Add kind cluster detection utilities
f466e8f 
Implements detection of kind clusters based on kubectl context name.
Extracts cluster name for use with 'kind load' commands.

Relates to #41
@GrimmiMeloni @claude
Add edge case tests for kind cluster detection
0802cf8 
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
Add local image detection for podman
98afd1f 
Implements image existence checking in podman with branding-aware paths.
Checks localhost/stackrox and quay.io registry prefixes in priority order.
Supports all ACS images (main, scanner, operator, etc.).

Relates to #41
@GrimmiMeloni @claude
Fix code quality issues in local image detection
59ba0bc 
- Remove unused ImageSet struct
- Add context to error messages in CheckImages batch operations

Relates to #41

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
Add kind image loading functionality
f2b73be 
Implements parallel loading of container images from podman into kind
clusters using 'kind load docker-image'. Uses 4 concurrent workers
for performance.

Relates to #41
@GrimmiMeloni @claude
Fix error handling in parallel image loading
42ea345 
- Collect all errors from workers instead of just first one
- Add test for empty image map
- Document fail-fast error behavior

Relates to #41

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
Integrate local image detection into deployment flow
d8f9738 
Adds automatic detection and loading of local images for kind clusters.
Checks podman for images before deployment and loads them into kind.
Skips credential verification when all images are available locally.

Supports ROXIE_SKIP_LOCAL_IMAGES environment variable to disable.

Relates to #41
@GrimmiMeloni @claude
Improve code quality in deployer integration
539cec0 
- Extract magic numbers to named constants
- Add comprehensive GoDoc comments
- Improve error message context
- Extract environment variable name to constant

Relates to #41

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
Skip image pull secrets when all images are local
8f437ab 
When all required images are loaded locally into kind, skip creating
image pull secrets since they're not needed. Still creates secrets
for partial local image scenarios.

Relates to #41
@GrimmiMeloni
Add documentation for local image support
6470c13 
Documents the automatic local image detection and loading feature
for kind clusters, including requirements and usage examples.

Relates to #41
@GrimmiMeloni @claude
Fix missing stackrox-operator image in detection
e953afe 
The operator image list was missing the stackrox-operator controller
image, only checking for bundle and index. This caused the operator
image to not be detected and loaded into kind.

Total expected images updated from 9 to 10 (7 main + 3 operator).

Relates to #41

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
Fix stackrox-operator image tag detection
7bbdb62 
The stackrox-operator image uses the main tag (no v prefix), while
operator-bundle and operator-index use the operator tag (with v prefix).
Split the detection to handle both tagging conventions.

Relates to #41
@GrimmiMeloni @claude
feat: patch CSV with local image refs for kind clusters
3db4c68 
Adds functionality to patch ClusterServiceVersion (CSV) files with local
image references when deploying the operator to kind clusters.

Changes:
- Added patchCSVWithLocalImages() function in operator.go that:
  - Patches operator image reference if stackrox-operator is in localImages
  - Patches RELATED_IMAGE_* environment variables with local image tags
  - Only patches images that exist in localImages map
  - Preserves all other CSV content unchanged

- Integrated CSV patching into deployOperatorFromCSV():
  - Calls patchCSVWithLocalImages() after reading CSV but before parsing
  - Only executes if d.usingLocalImages is true
  - Logs the patching action

- Added comprehensive test suite in operator_test.go:
  - Tests patching with all local images
  - Tests patching with partial local images
  - Tests skipping when no local images
  - Tests error handling for malformed CSV and missing files
  - Tests preservation of all other CSV content

This ensures the operator uses locally loaded images instead of attempting
to pull from remote registries, enabling fully offline kind deployments.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: construct quay.io paths when patching CSV
06e0855 
When patching the CSV with local images, construct the proper
quay.io/<branding-org>/<image>:<tag> path instead of using the raw
reference from podman (which could be localhost/stackrox/...).

This prevents Kubernetes from prepending docker.io/ to localhost/ refs,
which would result in invalid paths like:
docker.io/localhost/stackrox/main:4.10.0-local

Now correctly uses:
quay.io/rhacs-eng/main:4.10.0-local

Kind recognizes this image already exists (loaded earlier) and uses
the local copy without actually pulling from the network.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: load images into kind with quay.io paths
860de87 
Load images into kind using quay.io/<branding-org>/<image>:<tag> paths
instead of localhost/stackrox/... paths.

This ensures the image tags in kind match what the operator CSV
references, preventing kubelet from trying to pull from the network.

Previously:
- Detected: localhost/stackrox/main:4.10.0-local (in podman)
- Loaded: kind load docker-image localhost/stackrox/main:4.10.0-local
- CSV referenced: quay.io/rhacs-eng/main:4.10.0-local
- Result: Kind doesn't have that tag → pulls from network → fails

Now:
- Detected: localhost/stackrox/main:4.10.0-local (in podman)
- Loaded: kind load docker-image quay.io/rhacs-eng/main:4.10.0-local
- CSV referenced: quay.io/rhacs-eng/main:4.10.0-local
- Result: Kind has that tag → uses local image → success!

Since podman dual-tags images, both localhost/stackrox/... and
quay.io/rhacs-eng/... point to the same image. Kind can pull from
podman using the quay.io tag.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: dynamically patch all RELATED_IMAGE_* env vars
470231e 
Replace hardcoded RELATED_IMAGE_* mapping with dynamic detection.
Now automatically patches ALL RELATED_IMAGE_* environment variables
based on what images exist in localImages, instead of just a
hardcoded subset.

Changes:
- Added envVarToImageName() helper to convert env var names to
  image names (e.g., RELATED_IMAGE_SCANNER_V4_DB → scanner-v4-db)
- Build reverse map from image names to their keys for fast lookup
- Iterate through all env vars, check if they start with
  RELATED_IMAGE_, convert to image name, and patch if found locally
- Extract tag from imageKey instead of assuming mainImageTag

This ensures ALL images we have locally get patched into the CSV,
including scanner-v4-indexer and any future images we add, without
needing to update a hardcoded mapping.

Fixes issue where some RELATED_IMAGE_* vars kept their original
hardcoded tags from the operator bundle, causing kubelet to try
pulling non-existent tags from quay.io.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
refactor: remove operator-index detection and loading
3bf37c4 
Roxie doesn't use operator-index in default (non-OLM) deployment mode.
The operator-index is only used when deploying via OLM with the --olm
flag, which is rare on kind clusters.

Changes:
- Removed stackrox-operator-index from detection in CheckImages()
- Removed stackrox-operator-index from loading in LoadImagesToKind()
- Updated totalRequiredImages from 10 to 9 (7 main + 2 operator images)
- Added comments explaining why operator-index is excluded

This saves time by not detecting and loading an unused image during
the common local development workflow.

If users need OLM mode on kind, they can add operator-index support
later, but it's not needed for the default operator deployment path.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: add scanner-v4 to detected and loaded images
a934687 
Add scanner-v4 (separate from scanner-v4-matcher) to the list of
images to detect and load into kind clusters.

Changes:
- Added "scanner-v4" to detection list in CheckImages()
- Added "scanner-v4" to loading list in LoadImagesToKind()
- Updated totalRequiredImages from 9 to 10 (8 main + 2 operator)
- Updated comments to list all main images for clarity

Complete list of main images now:
1. main
2. scanner
3. scanner-db
4. scanner-v4
5. scanner-v4-db
6. scanner-v4-indexer
7. scanner-v4-matcher
8. central-db

This fixes detection of the scanner-v4 image which was previously
being missed.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: remove non-existent scanner-v4-indexer and scanner-v4-matcher im…
333effd 
…ages

Scanner v4 uses a single image (scanner-v4) that runs in different
modes (indexer or matcher) based on runtime configuration. There are
no separate scanner-v4-indexer or scanner-v4-matcher images.

Changes:
- Removed "scanner-v4-indexer" from detection list
- Removed "scanner-v4-matcher" from detection list
- Removed both from loading list
- Updated totalRequiredImages from 10 to 8 (6 main + 2 operator)
- Added explanatory comments about scanner-v4 architecture

Complete image list (8 total):
Main images (6):
  1. main
  2. scanner
  3. scanner-db
  4. scanner-v4 (runs as indexer or matcher based on config)
  5. scanner-v4-db
  6. central-db

Operator images (2):
  7. stackrox-operator
  8. stackrox-operator-bundle

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: map scanner-v4-indexer and scanner-v4-matcher to scanner-v4 in C…
6da5765 
…SV patching

The CSV contains RELATED_IMAGE_SCANNER_V4_INDEXER and
RELATED_IMAGE_SCANNER_V4_MATCHER environment variables, but both
should reference the same scanner-v4 image (which runs in different
modes based on runtime configuration).

Without this fix:
- envVarToImageName("RELATED_IMAGE_SCANNER_V4_INDEXER") → "scanner-v4-indexer"
- Look for "scanner-v4-indexer:tag" in localImages → NOT FOUND
- Env var doesn't get patched → keeps hardcoded bundle tag

With this fix:
- Convert "scanner-v4-indexer" → "scanner-v4"
- Convert "scanner-v4-matcher" → "scanner-v4"
- Look for "scanner-v4:tag" in localImages → FOUND
- Patch both env vars with local scanner-v4 tag

Changes:
- Added special case handling in patchCSVWithLocalImages() to map
  scanner-v4-indexer and scanner-v4-matcher to scanner-v4
- Added test TestPatchCSVWithLocalImages_ScannerV4Mapping to verify
  all three RELATED_IMAGE_SCANNER_V4* env vars get patched correctly
- Updated TestEnvVarToImageName to include indexer/matcher variants

This ensures RELATED_IMAGE_SCANNER_V4, RELATED_IMAGE_SCANNER_V4_INDEXER,
and RELATED_IMAGE_SCANNER_V4_MATCHER all reference the local scanner-v4
image with MAIN_IMAGE_TAG.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: add collector image to detection and loading
7f71d27 
The collector image was accidentally removed during previous changes.
Collector is a required image for secured clusters.

Changes:
- Added "collector" to detection list in CheckImages()
- Added "collector" to loading list in LoadImagesToKind()
- Updated totalRequiredImages from 8 to 9 (7 main + 2 operator)
- Updated comments to include collector in main images list

Complete image list (9 total):
Main images (7):
  1. main
  2. scanner
  3. scanner-db
  4. scanner-v4
  5. scanner-v4-db
  6. central-db
  7. collector

Operator images (2):
  8. stackrox-operator
  9. stackrox-operator-bundle

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
GrimmiMeloni and others added 10 commits March 4, 2026 16:21
@GrimmiMeloni @claude
feat: check both branding organizations when detecting images
d526a53 
When detecting local images in podman, now check both rhacs-eng and
stackrox-io registry paths, regardless of ROX_PRODUCT_BRANDING setting.

This handles cases where images don't support ROX_PRODUCT_BRANDING:
- Collector currently only builds with stackrox-io branding
- Future images may have similar limitations

Detection order (tries all three, returns first found):
1. localhost/stackrox/<image>:<tag>
2. quay.io/<current-branding-org>/<image>:<tag>
3. quay.io/<fallback-branding-org>/<image>:<tag>

Benefits:
- No special cases needed for specific images
- Maintains "just works" philosophy - automatic detection
- Future-proof for any branding inconsistencies
- More robust - handles edge cases we haven't anticipated
- Minimal performance impact (podman checks are fast)

Example with ROX_PRODUCT_BRANDING=RHACS_BRANDING:
- Most images found at quay.io/rhacs-eng/<image>:tag (2nd check)
- Collector found at quay.io/stackrox-io/collector:tag (3rd check)
- Both get detected and loaded automatically

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: use actual image references when loading into kind
c9489f1 
When loading images into kind, use the actual image reference found
during detection instead of reconstructing it based on branding.

The Problem:
- Detection finds: quay.io/stackrox-io/collector:4.10.0-local (fallback)
- Loading reconstructed: quay.io/rhacs-eng/collector:4.10.0-local (wrong!)
- kind fails: image not present locally

The Fix:
- Use the value from the images map directly (the actual found reference)
- This preserves whichever branding path was found during detection

Example:
images["collector:4.10.0-local"] = "quay.io/stackrox-io/collector:4.10.0-local"
                                      ^^^^^^^^^^^^^^^^^ Use this actual ref

Now when collector is found via fallback branding (stackrox-io), that
exact path is used when loading into kind, ensuring the image exists.

Changes:
- Changed from reconstructing paths to using map values directly
- Removed unused brandingOrg variable
- Updated comment to clarify we use actual references from detection

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: prioritize quay.io paths over localhost paths in detection
c3c2747 
Change detection order to check quay.io paths BEFORE localhost paths.
This ensures the image references used match what the CSV will contain.

The Problem:
- Detection found: localhost/stackrox/main:4.10.0-local (checked first)
- Loaded into kind: localhost/stackrox/main:4.10.0-local
- CSV referenced: quay.io/rhacs-eng/main:4.10.0-local
- Kind doesn't know these are the same image → pull fails!

The Fix:
New detection order:
1. quay.io/<current-branding-org>/<image>:<tag>
2. quay.io/<fallback-branding-org>/<image>:<tag>
3. localhost/stackrox/<image>:<tag>

Now:
- Detection finds: quay.io/rhacs-eng/main:4.10.0-local (checked first)
- Loaded into kind: quay.io/rhacs-eng/main:4.10.0-local
- CSV references: quay.io/rhacs-eng/main:4.10.0-local
- Tags match → kind uses local image! ✓

Podman's dual-tagging means the same image has both localhost and
quay.io tags, so kind can pull using the quay.io tag and store it
with that tag, matching what the CSV will reference.

Changes:
- Reordered buildImageReferences() to check quay.io first
- Updated comment explaining why quay.io comes before localhost
- Updated tests to reflect new priority order

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
Remove localhost path support from local image detection
afd0408 
Podman dual-tags images with both localhost/stackrox and quay.io paths
pointing to the same image. Since CSV references require quay.io paths,
and kind needs to store images with tags matching the CSV, the localhost
path check adds complexity without value. Only checking quay.io paths
simplifies the logic while achieving the same result.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
Use actual image references from detection in CSV patching
093dbc2 
Previously, CSV patching reconstructed image paths using the current
branding organization. This broke for collector, which only builds
with stackrox-io but would be patched with rhacs-eng when that was
the current branding.

Now, CSV patching uses the actual image reference from the localImages
map that was detected. This handles fallback branding correctly - if
collector was detected at quay.io/stackrox-io, that's the exact path
used in the CSV.

Also updated tests to use quay.io paths instead of localhost paths,
matching the removal of localhost support in detection.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
fix: address code review findings
80e1819 
- Remove misleading "Exported for testing" comment from unexported function
- Remove redundant length check and magic number in ExtractKindClusterName
- Fix stale comments about image counts (9 total, not 10)
- Update tests to use realistic scanner-v4 image references
- Change test to use quay.io paths instead of localhost

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
format code
f90b107
@GrimmiMeloni @claude
refactor: unify local image handling and fix operator tag usage
e5a66f7 
Consolidates duplicate code and improves type safety in local image detection and CSV patching:

- Extract shared image list logic into getExpectedImages() helper to eliminate duplication between detection and loading
- Fix bug where stackrox-operator incorrectly used mainTag instead of operatorTag
- Replace generic map navigation in CSV patching with type-safe clusterServiceVersion struct
- Unify shouldSkipCredentialVerification() and shouldSkipImagePullSecrets() into hasAllImagesLocally()
- Change CheckLocalImage to idiomatic (value, bool, error) return pattern
- Make ExtractKindClusterName and helper functions private
- Remove unused test cases and helper functions

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni @claude
docs: update local image support documentation to match implementation
5d0cf5f 
Updates design documents and README to reflect the actual implemented
behavior:

README.md:
- Fix supported images list (add collector, scanner-v4, stackrox-operator)
- Remove non-existent images (scanner-v4-indexer, scanner-v4-matcher,
  stackrox-operator-index)
- Organize images by category (main vs operator)

Design documents:
- Add status notes indicating implementation is complete with variations
- Update image inventory to match actual code (9 images total)
- Document key implementation differences from original design:
  * Both branding orgs are checked (not just ROX_PRODUCT_BRANDING)
  * Only quay.io paths used (localhost paths removed)
  * CSV patching added for operator deployments
- Reference relevant commits for implementation evolution

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@GrimmiMeloni
fix formatting
047630b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mclasmeier mclasmeier Awaiting requested review from mclasmeier

At least 1 approving review is required to merge this pull request.

Assignees

@GrimmiMeloni GrimmiMeloni

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add support for locally built stackrox images

2 participants

@GrimmiMeloni @mclasmeier