Add Azure integration test backend by Jakob-Naucke · Pull Request #170 · trusted-execution-clusters/operator

Jakob-Naucke · 2026-01-27T14:54:53Z

Update attestation policy
Make KubeVirt one of multiple $VIRT_PROVIDER backends
Introduce environment variables for images, as well as a namespace prefix
Use Azure CLI for new Azure backend

Depends on #159, #179

openshift-ci · 2026-01-27T14:54:57Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

register-server/src/main.rs

alicefr · 2026-01-29T14:18:56Z

Makefile


 integration-tests: generate trusted-cluster-gen crds-rs
 	RUST_LOG=info REGISTRY=$(REGISTRY) TAG=$(TAG) \
+		TRUSTEE_IMAGE=$(TRUSTEE_IMAGE) APPROVED_IMAGE=$(APPROVED_IMAGE) TEST_IMAGE=$(TEST_IMAGE) \


nit: can you also update the README for the integrations tests

Jakob-Naucke · 2026-02-02T14:37:21Z

update: first passing test

Jakob-Naucke · 2026-02-04T21:12:02Z

Moving out of draft

NB this requires a trustee-attester without tpm-attester support until confidential-containers/guest-components#1277 is resolved. Even when az-snp-vtpm is registered, tpm-attester registers as composite evidence and then fails because the AK cannot be set up right on Azure. The Trustee community has some understandable concerns on silently not providing failed evidence, as well as disabling features on the command line because that might create misleading evidence. I've pushed one to quay.io/trusted-execution-clusters/trustee-attester:v0.17.0-az-only.

alicefr · 2026-02-05T11:55:50Z

tests/README.md

+### Running the tests
+
+The Azure VMs must be able to reach cluster services like the registration server and Trustee.
+If you are on OpenShift, export `PLATFORM=openshift` and `oc expose` will make the required services available.


can you provide an example for 'oc expose`, it requires the service to be exposed.

The test code included in this PR handles this, so there should be no need to manually oc expose. Or what do you mean?

This part of the docs isn't very clean on what is required to be expose. You mention oc expose, and it isn't very clear what needs to be exposed. That's why I asked an example

alicefr · 2026-02-05T12:28:39Z

@Jakob-Naucke I think we need to randmize the name of the VM in azure. In case of kubevirt, the VM is namespaced, but for Azure that's not the case. This might introduce some conflicts when the test are run in parallel and if the VM they have the same name. Can you add a randmon 5 string char after the name of each VM?

alicefr · 2026-02-05T13:09:02Z

As a follow up PR, we also require a clean-up script for the VMs of the namespace that have failed. In case of KubeVirt, it isn't that bad because they got destroyed with the kind cluster, but with Azure that's not the case

alicefr · 2026-02-05T13:17:43Z

tests/README.md

+export TEST_IMAGE=/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$resource_group/providers/Microsoft.Compute/galleries/$compute_gallery/images/$image_definition/versions/$image_version
+```
+
+### Running the tests


It is also missing how you build and push the images in the openshift cluster

Because I wasn't. But internal registry is probably possible, and I could add some general remarks on making them available.

Re both your comments on this file, I'm thinking if we should keep the areas "testing on OpenShift" and "testing on Azure" a little more separate, even if our CI will use them in tandem.

Yes, probably we should mention how to test on openshift and our suggestions. Since one could think of using kind and Azure as well, but they need to expose the services if they want to test in this way

alicefr · 2026-02-05T13:33:53Z

test_utils/src/virt/azure.rs

+        let out = ["--output", "json"];
+        let output = Command::new("az").args(args).args(out).output().await?;


can we check if the command is really installed and present in the system?

First line of create_vm

nit: why not here? instead of in create_vm?

Moved for clarity. The origin of it being there is that virtctl is checked to be there early for KubeVirt, because there's a lot of sunk time until virtctl is used

Jakob-Naucke · 2026-02-05T13:37:07Z

test_utils/src/virt/azure.rs

+        let out = ["--output", "json"];
+        let output = Command::new("az").args(args).args(out).output().await?;


First line of create_vm

test_utils/src/virt/azure.rs

Jakob-Naucke · 2026-02-05T13:43:30Z

tests/README.md

+export TEST_IMAGE=/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$resource_group/providers/Microsoft.Compute/galleries/$compute_gallery/images/$image_definition/versions/$image_version
+```
+
+### Running the tests


Because I wasn't. But internal registry is probably possible, and I could add some general remarks on making them available.

Re both your comments on this file, I'm thinking if we should keep the areas "testing on OpenShift" and "testing on Azure" a little more separate, even if our CI will use them in tandem.

Jakob-Naucke · 2026-02-05T13:44:10Z

tests/README.md

+### Running the tests
+
+The Azure VMs must be able to reach cluster services like the registration server and Trustee.
+If you are on OpenShift, export `PLATFORM=openshift` and `oc expose` will make the required services available.


The test code included in this PR handles this, so there should be no need to manually oc expose. Or what do you mean?

alicefr · 2026-02-05T14:46:00Z

The tests passed for me. Good job! We need to improve the docs and there is a refactoring for the openshift vs kind trait, but if you prefer we can address them in a follow up PR.

This commit updates the test utils to support running trusted execution cluster tests on the OpenShift platform. To execute these tests on OpenShift, the following environment variables must be configured: - REGISTRY: The repository location of the container image. - TAG: The specific tag of the container image. - CLUSTER_URL: The API URL of the target cluster. - PLATFORM: Set this to 'openshift'. Signed-off-by: Roy Kaufman <rkaufman@redhat.com> Co-authored-by: Jakob Naucke <jnaucke@redhat.com>

- Field is now `az-snp-vtpm` (which requires quoting due to the dashes) instead of `azsnpvtpm` - No lowercasing required Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

Split test_utils::virt into a module with separate virt::kubevirt. Introduce the VIRT_PROVIDER variable to select. - Reformat a bit. - Make root_key an Option for environments where machine IDs cannot be correlated to IPs. Signed-off-by: Jakob Naucke <jnaucke@redhat.com> Assisted-by: Claude

Less double-tracking and adds flexibility for testing on cloud platforms. Environment variables: - {TEST,TRUSTEE,APPROVED}_IMAGE - TEST_NAMESPACE_PREFIX Other changes: - OpenShift service exposure - Get cluster URL from OpenShift directly - Reduce some lines & indentation in TestContext::apply_operator_manifests and break it up some - Move endpoint constants to lib - Log error upon poll timeout Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

Add integration tests backend for Azure (confidential VMs, no cluster join). Use Azure CLI to create a resource group per test, same name as the namespace (which can be prefixed). Auto-shutdown VMs after one hour. Signed-off-by: Jakob Naucke <jnaucke@redhat.com> Co-authored-by: Uri Lublin <uril@redhat.com> Assisted-by: Gemini, Claude

Jakob-Naucke

@alicefr Thanks for your review! Everything should be addressed. Please take another look if you can.

Jakob-Naucke · 2026-02-06T12:34:50Z

test_utils/src/virt/azure.rs

+        let out = ["--output", "json"];
+        let output = Command::new("az").args(args).args(out).output().await?;


Moved for clarity. The origin of it being there is that virtctl is checked to be there early for KubeVirt, because there's a lot of sunk time until virtctl is used

openshift-ci · 2026-02-06T16:18:05Z

@Jakob-Naucke: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/operator-lifecycle-verify	`84dc744`	link	true	`/test operator-lifecycle-verify`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci · 2026-02-09T13:07:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alicefr, Jakob-Naucke

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label Jan 27, 2026

alicefr reviewed Jan 29, 2026

View reviewed changes

register-server/src/main.rs Outdated Show resolved Hide resolved

alicefr reviewed Jan 29, 2026

View reviewed changes

Jakob-Naucke force-pushed the azure-test branch from a72a201 to fece91b Compare January 30, 2026 08:20

Jakob-Naucke mentioned this pull request Jan 30, 2026

Enable trusted_execution_cluster tests on OpenShift #159

Closed

openshift-merge-robot added the needs-rebase label Jan 30, 2026

Jakob-Naucke mentioned this pull request Feb 2, 2026

reg: Do not delete Machines based on IP #179

Merged

Jakob-Naucke force-pushed the azure-test branch from fece91b to d93e5c9 Compare February 2, 2026 14:36

Jakob-Naucke force-pushed the azure-test branch from d93e5c9 to 85d41d2 Compare February 4, 2026 21:10

openshift-merge-robot removed the needs-rebase label Feb 4, 2026

Jakob-Naucke force-pushed the azure-test branch from 85d41d2 to 8f0139d Compare February 4, 2026 21:11

Jakob-Naucke marked this pull request as ready for review February 4, 2026 21:12

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 4, 2026

alicefr reviewed Feb 5, 2026

View reviewed changes

Jakob-Naucke commented Feb 5, 2026

View reviewed changes

alicefr approved these changes Feb 5, 2026

View reviewed changes

openshift-ci bot assigned alicefr Feb 5, 2026

openshift-ci bot added the lgtm label Feb 5, 2026

iroykaufman and others added 4 commits February 6, 2026 12:52

rego: Update Azure attestation policy

d74654b

- Field is now `az-snp-vtpm` (which requires quoting due to the dashes) instead of `azsnpvtpm` - No lowercasing required Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

Jakob-Naucke force-pushed the azure-test branch from 8f0139d to 84dc744 Compare February 6, 2026 14:06

openshift-ci bot removed the lgtm label Feb 6, 2026

Jakob-Naucke commented Feb 6, 2026

View reviewed changes

Jakob-Naucke mentioned this pull request Feb 9, 2026

Delete the libvirt VM related code openshift/release#73940

Merged

alicefr approved these changes Feb 9, 2026

View reviewed changes

openshift-ci bot added the lgtm label Feb 9, 2026

Jakob-Naucke merged commit e103e68 into trusted-execution-clusters:main Feb 9, 2026
8 of 9 checks passed

Jakob-Naucke deleted the azure-test branch February 9, 2026 13:12

		let out = ["--output", "json"];
		let output = Command::new("az").args(args).args(out).output().await?;

Conversation

Jakob-Naucke commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Jan 27, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jakob-Naucke commented Feb 2, 2026

Uh oh!

Jakob-Naucke commented Feb 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alicefr commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alicefr commented Feb 5, 2026

Uh oh!

alicefr Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jakob-Naucke Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alicefr Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jakob-Naucke Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alicefr commented Feb 5, 2026

Uh oh!

Jakob-Naucke left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Feb 6, 2026

Uh oh!

openshift-ci bot commented Feb 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jakob-Naucke commented Jan 27, 2026 •

edited

Loading

alicefr commented Feb 5, 2026 •

edited

Loading

alicefr Feb 5, 2026 •

edited

Loading

Jakob-Naucke Feb 5, 2026 •

edited

Loading

alicefr Feb 5, 2026 •

edited

Loading

Jakob-Naucke Feb 5, 2026 •

edited

Loading