Skip to content

Claude/kubernetes migration plan kq jw d #358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
t0mdavid-m merged 16 commits into from
Apr 4, 2026
Merged

Claude/kubernetes migration plan kq jw d #358

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3bbcc2a
Add Kubernetes manifests and CI workflows for de.NBI migration
claude Mar 5, 2026
233484e
Fix kubeconform validation to skip kustomization.yaml
claude Mar 5, 2026
aa28e91
Add matrix strategy to test both Dockerfiles in integration tests
claude Mar 5, 2026
9aa798b
Adapt K8s base manifests for de.NBI Cinder CSI storage
claude Mar 11, 2026
0d90cf7
Add pod affinity rules to co-locate all workspace pods on same node
claude Mar 11, 2026
e8d4671
Merge remote-tracking branch 'origin/main' into claude/kubernetes-mig…
claude Mar 11, 2026
ed3075b
Fix CI: wait for ingress-nginx admission webhook before deploying
claude Mar 11, 2026
6036d14
Fix CI: add -n openms namespace to integration test steps
claude Mar 11, 2026
e7cac75
Fix CI: retry kustomize deploy for webhook readiness
claude Apr 2, 2026
173f3b8
Fix REDIS_URL to use prefixed service name in overlay
claude Apr 2, 2026
d144aeb
Merge branch 'main' into claude/kubernetes-migration-plan-KQJwD
t0mdavid-m Apr 2, 2026
443ae24
Add Traefik IngressRoute for direct LB IP access
claude Apr 2, 2026
9c0fa08
Fix CI: skip Traefik IngressRoute CRD in validation and integration t…
claude Apr 2, 2026
b783ff7
Fix IngressRoute service name for kustomize namePrefix
claude Apr 4, 2026
5f8a5b2
fix: use ConfigMap as settings override instead of full replacement
claude Apr 4, 2026
b913b64
fix: add set -euo pipefail to fail fast on settings merge error
claude Apr 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/k8s-manifests-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@ jobs:

- name: Validate K8s manifests (base)
run: |
kubeconform -summary -strict -kubernetes-version 1.28.0 -ignore-filename-pattern 'kustomization.yaml' k8s/base/*.yaml
kubeconform -summary -strict -kubernetes-version 1.28.0 \
-ignore-filename-pattern 'kustomization.yaml' \
-ignore-filename-pattern 'traefik-ingressroute.yaml' \
k8s/base/*.yaml
Comment on lines +24 to +27
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Inspect CI workflow handling of Traefik/IngressRoute:"
rg -n -C3 'traefik-ingressroute|IngressRoute|kubeconform|kustomize|yq' .github/workflows

echo
echo "Find Traefik-related manifests in k8s tree:"
fd -i 'traefik|ingressroute' k8s

Repository: OpenMS/streamlit-template

Length of output: 3169

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== K8s directory structure ==="
find k8s -type f -name "*.yaml" | head -20

echo
echo "=== Check if traefik-ingressroute.yaml is in base or overlay ==="
find k8s -name "traefik-ingressroute.yaml" -exec ls -la {} \;

echo
echo "=== Look for all IngressRoute definitions across k8s ==="
rg -l "kind: IngressRoute" k8s

echo
echo "=== Check kustomization.yaml in base and overlay for resources ==="
echo "--- Base kustomization.yaml ---"
cat k8s/base/kustomization.yaml 2>/dev/null | head -30

echo
echo "--- Overlay kustomization.yaml ---"
cat k8s/overlays/template-app/kustomization.yaml 2>/dev/null | head -30

Repository: OpenMS/streamlit-template

Length of output: 1785

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Content of traefik-ingressroute.yaml ==="
cat k8s/base/traefik-ingressroute.yaml

echo
echo "=== Overlay kustomization.yaml (full) ==="
cat k8s/overlays/template-app/kustomization.yaml

echo
echo "=== Verify what kustomize outputs (check if IngressRoute is there) ==="
kubectl kustomize k8s/overlays/template-app/ 2>/dev/null | grep -A5 "kind: IngressRoute" || echo "No IngressRoute found in kustomize output"

Repository: OpenMS/streamlit-template

Length of output: 1640

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Full kustomize output (raw) ==="
kubectl kustomize k8s/overlays/template-app/ 2>/dev/null > /tmp/kustomize-output.yaml
wc -l /tmp/kustomize-output.yaml

echo
echo "=== Check for IngressRoute or traefik references in output ==="
grep -i "ingressroute\|traefik" /tmp/kustomize-output.yaml || echo "No matches found"

echo
echo "=== Show kinds present in output ==="
grep "^kind:" /tmp/kustomize-output.yaml | sort | uniq

echo
echo "=== Check if CRD definitions are in the file ==="
head -100 /tmp/kustomize-output.yaml

Repository: OpenMS/streamlit-template

Length of output: 106

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Verify which tools are available ==="
which kubectl kustomize yq jq python3 2>&1 | head -10 || echo "Checking alternatives..."

echo
echo "=== Check if we can use Python to parse YAML ==="
python3 --version

echo
echo "=== Try to understand the impact by reading the manifests directly ==="
echo "Base includes traefik-ingressroute.yaml in resources list"
echo "Overlay includes base via ../../base"
echo "Therefore overlay kustomize output SHOULD include IngressRoute"

echo
echo "=== Let's check the actual CI workflow logic ==="
echo "Line 24-27: kubeconform validates k8s/base/*.yaml directly (ignoring traefik-ingressroute.yaml by filename)"
echo "Line 39: pipes kustomize output through kubeconform with -skip IngressRoute"

echo
echo "=== Check if the ignore pattern on line 26 is actually effective ==="
echo "The pattern 'traefik-ingressroute.yaml' should match k8s/base/traefik-ingressroute.yaml"
python3 << 'EOF'
import re

# Test the ignore pattern
pattern = 'traefik-ingressroute.yaml'
filename = 'traefik-ingressroute.yaml'

# kubeconform uses filename pattern matching
if pattern in filename:
    print(f"Pattern '{pattern}' matches '{filename}': YES")
else:
    print(f"Pattern '{pattern}' matches '{filename}': NO")
EOF

Repository: OpenMS/streamlit-template

Length of output: 941

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Check kubeconform capabilities regarding CRDs ==="
echo "Searching for kubeconform documentation or usage info in repo..."
rg -i "kubeconform" .github -A2 -B2

echo
echo "=== Research: kubeconform is designed to validate against Kubernetes OpenAPI schemas ==="
echo "The -skip flag is meant for resource kinds that aren't in the core K8s schema"
echo "IngressRoute is a Traefik CRD, not part of Kubernetes core API"

echo
echo "=== Verify: Does kubeconform have CRD schemas built-in? ==="
echo "kubeconform validates against official K8s API versions specified by -kubernetes-version"
echo "CRDs like Traefik IngressRoute require custom schema validation setup"

echo
echo "=== Analysis of the CI configuration ==="
echo "1. Base validation ignores traefik-ingressroute.yaml by filename"
echo "2. Overlay validation skips IngressRoute kind entirely"
echo "3. Integration test (line 91) also filters out IngressRoute"
echo ""
echo "This appears INTENTIONAL, not accidental:"
echo "- kubeconform validates core K8s resources only"
echo "- CRDs cannot be validated without custom schema providers"
echo "- -skip is the documented way to exclude CRDs from validation"

echo
echo "=== Check integration test filtering ==="
grep -A10 "kind cluster uses nginx" .github/workflows/k8s-manifests-ci.yml

Repository: OpenMS/streamlit-template

Length of output: 3044

IngressRoute resources are not validated by kubeconform in either stage

The CI skips validation of Traefik IngressRoute manifests at line 26 (base validation ignores by filename) and line 39 (overlay validation skips by kind). This appears intentional—kubeconform validates only core Kubernetes API resources using official OpenAPI schemas; CRDs like IngressRoute require custom schema providers or manual dry-run validation.

However, this creates an unmonitored path for IngressRoute manifest changes. If validation of Traefik CRD syntax is desired, consider adding a schema-based validation step (e.g., using Traefik's CRD schemas) or enabling CRD validation via a custom kubeconform schema provider, rather than silently skipping the resource kind.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/k8s-manifests-ci.yml around lines 24 - 27, The CI is
skipping Traefik IngressRoute validation via kubeconform due to the
-ignore-filename-pattern 'traefik-ingressroute.yaml' and the -ignore-kind
IngressRoute usage; remove those ignore rules and add a dedicated validation
step (e.g., a job/step named "validate-traefik-ingressroutes") that runs
kubeconform or an equivalent validator against IngressRoute manifests using
Traefik's CRD schema (or a custom kubeconform schema provider) — keep the
existing kubeconform calls for core resources (the kubeconform invocation) and
add the new invocation that points to the Traefik CRD schema or uses
--schema-location/--schema-provider so IngressRoute is validated instead of
being silently skipped.


- name: Install kubectl
uses: azure/setup-kubectl@v3
Expand All @@ -33,7 +36,7 @@ jobs:

- name: Validate kustomized output
run: |
kubectl kustomize k8s/overlays/template-app/ | kubeconform -summary -strict -kubernetes-version 1.28.0
kubectl kustomize k8s/overlays/template-app/ | kubeconform -summary -strict -kubernetes-version 1.28.0 -skip IngressRoute

integration-test:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -83,7 +86,9 @@ jobs:
- name: Deploy with Kustomize
if: steps.check.outputs.exists == 'true'
run: |
# Filter out Traefik CRDs (kind cluster uses nginx, not Traefik)
kubectl kustomize k8s/overlays/template-app/ | \
yq 'select(.kind != "IngressRoute")' | \
sed 's|imagePullPolicy: IfNotPresent|imagePullPolicy: Never|g' > /tmp/manifests.yaml
for i in 1 2 3 4 5; do
if kubectl apply -f /tmp/manifests.yaml; then
Expand Down
34 changes: 2 additions & 32 deletions k8s/base/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,37 +3,7 @@ kind: ConfigMap
metadata:
name: streamlit-config
data:
settings.json: |
settings-overrides.json: |
{
"app-name": "OpenMS WebApp Template",
"online_deployment": true,
"enable_workspaces": true,
"workspaces_dir": "..",
"queue_settings": {
"default_timeout": 7200,
"result_ttl": 86400
},
"demo_workspaces": {
"enabled": true,
"source_dirs": ["example-data/workspaces"]
},
"max_threads": {
"local": 4,
"online": 2
},
"analytics": {
"matomo": {
"enabled": true,
"url": "https://cdn.matomo.cloud/openms.matomo.cloud",
"tag": "yDGK8bfY"
},
"google-analytics": {
"enabled": false,
"tag": ""
},
"piwik-pro": {
"enabled": false,
"tag": ""
}
}
"online_deployment": true
}
1 change: 1 addition & 0 deletions k8s/base/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ resources:
- streamlit-service.yaml
- rq-worker-deployment.yaml
- ingress.yaml
- traefik-ingressroute.yaml
- cleanup-cronjob.yaml
5 changes: 3 additions & 2 deletions k8s/base/rq-worker-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ spec:
args:
- |
source /root/miniforge3/bin/activate streamlit-env
jq -s '.[0] * .[1]' /app/settings.json /app/settings-overrides.json > /tmp/settings-merged.json && mv /tmp/settings-merged.json /app/settings.json
exec rq worker openms-workflows --url $REDIS_URL
Comment thread
coderabbitai[bot] marked this conversation as resolved.
env:
- name: REDIS_URL
Expand All @@ -41,8 +42,8 @@ spec:
- name: workspaces
mountPath: /workspaces-streamlit-template
- name: config
mountPath: /app/settings.json
subPath: settings.json
mountPath: /app/settings-overrides.json
subPath: settings-overrides.json
readOnly: true
resources:
requests:
Expand Down
5 changes: 3 additions & 2 deletions k8s/base/streamlit-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ spec:
args:
- |
source /root/miniforge3/bin/activate streamlit-env
jq -s '.[0] * .[1]' /app/settings.json /app/settings-overrides.json > /tmp/settings-merged.json && mv /tmp/settings-merged.json /app/settings.json
exec streamlit run app.py --server.address 0.0.0.0
Comment thread
coderabbitai[bot] marked this conversation as resolved.
ports:
- containerPort: 8501
Expand All @@ -43,8 +44,8 @@ spec:
- name: workspaces
mountPath: /workspaces-streamlit-template
- name: config
mountPath: /app/settings.json
subPath: settings.json
mountPath: /app/settings-overrides.json
subPath: settings-overrides.json
readOnly: true
readinessProbe:
httpGet:
Expand Down
18 changes: 18 additions & 0 deletions k8s/base/traefik-ingressroute.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: streamlit-traefik
spec:
entryPoints:
- web
routes:
- match: PathPrefix(`/`)
kind: Rule
services:
- name: streamlit
port: 8501
sticky:
cookie:
name: stroute
httpOnly: true
sameSite: lax
21 changes: 21 additions & 0 deletions k8s/overlays/template-app/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,24 @@ patches:
- op: replace
path: /spec/rules/0/host
value: template.openms.example.de
- target:
kind: Deployment
name: streamlit
patch: |
- op: replace
path: /spec/template/spec/containers/0/env/0/value
value: "redis://template-app-redis:6379/0"
- target:
kind: Deployment
name: rq-worker
patch: |
- op: replace
path: /spec/template/spec/containers/0/env/0/value
value: "redis://template-app-redis:6379/0"
- target:
kind: IngressRoute
name: streamlit-traefik
patch: |
- op: replace
path: /spec/routes/0/services/0/name
value: "template-app-streamlit"
Loading