feat(storagebox): add Gateway API routing and replace MinIO with Garage#111
Open
adamancini wants to merge 4 commits intomainfrom
Open
feat(storagebox): add Gateway API routing and replace MinIO with Garage#111adamancini wants to merge 4 commits intomainfrom
adamancini wants to merge 4 commits intomainfrom
Conversation
Replace ingress-nginx with Envoy Gateway as the Gateway API controller, installed as an EC extension via OCI chart. Each application gets its own Gateway resource with an independent Envoy proxy instance: - Garage S3: HTTP Gateway + HTTPRoute (port 3900) - PostgreSQL: TCP Gateway + TCPRoute (port 5432) - Cassandra: TCP Gateway + TCPRoute (port 9042) - rqlite: HTTP Gateway + HTTPRoute (port 4001) - NFS: stays on NodePort (Gateway API does not support UDP) Replace MinIO operator + Tenant subchart with Garage v1.3.1, a lightweight S3-compatible object storage that runs as a single StatefulSet with no operator dependency. A post-install/post-upgrade Helm hook Job handles cluster layout assignment, bucket creation, and S3 credential provisioning via the Garage admin API. An init container copies secrets to an emptyDir with mode 0600 to satisfy Garage's strict file permission requirements. Also includes: - Per-service gateway and TLS settings in KOTS admin console config - Helm test for Garage connectivity and S3 round-trip verification - Support bundle collectors and deployment health analyzers for all infrastructure (cert-manager, CNPG, Envoy Gateway, K8ssandra) - Status informers for infrastructure deployments - Builder key for air-gap image discovery - NFS kernel module preflight upgraded to hard fail - Consolidated all utility images to alpine:3.21 (removed busybox) - vm-kubectl Makefile target for remote kubectl on EC VMs - Updated CI workflow and smoke tests for Garage
Covers per-application Gateway pattern with Envoy Gateway, HTTPRoute for S3/HTTP services, TCPRoute for databases, GatewayClass/EnvoyProxy infrastructure, TLS termination, and KOTS config integration. All examples drawn from the storagebox application. Notes that TCPRoute's experimental status is point-in-time (February 2026) and that Traefik supports TCPRoute when experimental CRDs are installed separately.
adamancini
force-pushed
the
adamancini/gateway-api
branch
from
7ffd5b6 to
afc198b
Compare
scottrigby
reviewed
Feb 27, 2026
Member
scottrigby
left a comment
scottrigby
left a comment
There was a problem hiding this comment.
Glad to see a Gateway API pattern!
This PR looks great, except for one question (below)
applications/storagebox/charts/storagebox/templates/tests/test-garage.yaml
Show resolved
Hide resolved
…ctly Remove Kubernetes API calls from the helm test pod. Instead of fetching the S3 credentials Secret via the K8s API with SA token + CA cert, mount it directly as a volume. This eliminates the serviceAccountName, KUBE_API, SA_TOKEN, and CA_CERT plumbing that was confusing two auth contexts (Garage app-level auth vs K8s API auth).
Member
|
@adamancini did you mean to have a chef-360 reference in the backup.yaml? It's commented out but still worth asking. 
|
Member
|
@adamancini should we use the latest Replicated SDK 1.17.0 ? apiVersion: v2
I believe the ~ only allows patch level updates - z release. |
Member
|
@adamancini I'm having issues deploying to EC and I'm not sure where these minio references are coming from. I have switched to your PR branch and created a new release. It could be something my side but just wanted to check? {"level":"info","ts":"2026-03-11T02:45:50Z","msg":"stdout (helm install) = Release \"storagebox\" does not exist. Installing it now.\n"}
{"level":"info","ts":"2026-03-11T02:45:50Z","msg":"stderr (helm install) = Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: \"myminio\" namespace: \"\" from \"\": no matches for kind \"Tenant\" in version \"minio.min.io/v2\"\nensure CRDs are installed first\n"}All the infra Helm chart deploys are fine. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace ingress-nginx with Envoy Gateway as the Gateway API controller and replace MinIO with Garage for S3-compatible object storage.
Gateway API (Envoy Gateway)
Each application gets its own Gateway resource. Envoy Gateway provisions an independent Envoy proxy Deployment + NodePort Service per Gateway, providing full isolation.
GatewayClass+EnvoyProxyresource configures NodePort for EC environmentsoci://docker.io/envoyproxy/gateway-helmv1.7.0), bundles all Gateway API CRDs including experimental TCPRouteGarage S3 Storage (replaces MinIO)
chmod 0600(KubernetesfsGroupadds group-read bits to secret volume mounts, but Garage requires exactly mode 0600)alpine:3.21+curl+jqfor Garage admin API calls:Operational improvements
deploymentStatus/statefulsetStatusanalyzers for all infrastructure (cert-manager, CNPG, Envoy Gateway, K8ssandra, cass-operator) and application componentshelm templatealpine:3.21(removed busybox)helmUpgradeFlagsvm-kubectltarget for remote kubectl on EC VMs; removed minio-operator from test-install-operatorsPatterns doc
New
patterns/gateway-api/README.mdcovering per-application Gateway pattern, HTTPRoute/TCPRoute examples, EnvoyProxy/GatewayClass infrastructure, TLS termination, and KOTS integration. Notes TCPRoute experimental status is point-in-time.Test plan
helm lintpasseshelm templaterenders all resources with all components enabledmake validate-configfour-way contract passeshelm-install-test(pending with Garage v1.3.1 fixes)