Reproducer single node openshift (SNO)

[danpawlik]

This commit allows reproducer to create OpenShift cluster using Single Node OpenShift (SNO) feature.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[softwarefactory-project-zuul]

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

[softwarefactory-project-zuul]

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/5dc22fa6d6bf4c44bb15fb3bb7940af0

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 35m 59s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 09m 23s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 20m 19s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 7m 38s
❌ cifmw-pod-pre-commit FAILURE in 7m 51s
✔️ build-push-container-cifmw-client SUCCESS in 21m 07s
❌ cifmw-molecule-devscripts FAILURE in 10m 41s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 00s

[danpawlik]

Still WIP, there is an issue with ingress that raises few issues later.
Even export NODES_PLATFORM="none" does not help. Very helpful would be to make reverse engineering of a CI job that uses SNO_IPV4 scenario, but can not find it.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7e909c7819764fc3a6ece0e75bd495f0

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 41m 20s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 13m 20s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 26m 00s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 7m 55s
❌ cifmw-pod-pre-commit FAILURE in 7m 55s
✔️ build-push-container-cifmw-client SUCCESS in 15m 36s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 05s
✔️ cifmw-molecule-reproducer SUCCESS in 13m 42s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/58af74d994354186bc474744505368bc

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 02s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 09m 32s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 23m 18s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 13s
❌ cifmw-pod-pre-commit FAILURE in 7m 53s
✔️ build-push-container-cifmw-client SUCCESS in 16m 41s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 43s
✔️ cifmw-molecule-reproducer SUCCESS in 14m 04s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7b7fc5d03ff34d2cb09473f40f04bdaf

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 45m 19s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 10m 25s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 23m 43s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 7m 54s
❌ cifmw-pod-pre-commit FAILURE in 7m 57s
✔️ build-push-container-cifmw-client SUCCESS in 20m 16s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 03s
✔️ cifmw-molecule-reproducer SUCCESS in 14m 37s

[github-actions]

This PR is stale because it has been for over 15 days with no activity.
Remove stale label or comment or this will be closed in 7 days.

[danpawlik]

Seems PoC of SNO is almost working, without using reproducer or devscript role.
Example execution using nightly build:

cat << 'EOF' > pre.sh
sudo sed -i 's@metalink=https://mirrors.centos.org/metalink?repo=centos-baseos-$stream&arch=$basearch&protocol=https,http@baseurl=http://mirror.stream.centos.org/9-stream/
BaseOS/$basearch/os/@g' /etc/yum.repos.d/centos.repo

sudo sed -i 's@metalink=https://mirrors.centos.org/metalink?repo=centos-appstream-$stream&arch=$basearch&protocol=https,http@baseurl=http://mirror.stream.centos.org/9-stre
am/AppStream/$basearch/os/@g' /etc/yum.repos.d/centos.repo

sudo dnf update -y
sudo dnf install -y git make wget jq libvirt qemu-kvm tar
# https://github.com/openshift-metal3/dev-scripts/blob/master/agent/README.md
sudo setfacl -m u:qemu:rx /home/zuul
echo 'user = "+0"' | sudo tee -a /etc/libvirt/qemu.conf

# enable nested virt
sudo sed -i 's/#options kvm_intel nested=1/options kvm_intel nested=1/g' /etc/modprobe.d/kvm.conf

echo "Now reboot the host..."
sudo reboot
EOF

cat << 'EOF' > execute.sh
sudo swapoff -a

git clone https://github.com/openshift-metal3/dev-scripts /home/zuul/src/github.com/openshift-metal3/dev-scripts

# workaround for only 8 vcpus
sed -i 's/MASTER_VCPU=8/MASTER_VCPU=20/g' /home/zuul/src/github.com/openshift-metal3/dev-scripts/common.sh

# it should have: ci_token  pull_secret.json
cp -a ~/secrets/* /home/zuul/src/github.com/openshift-metal3/dev-scripts/


cat << 'EOL' > /home/zuul/src/github.com/openshift-metal3/dev-scripts/config_$(whoami).sh


#!/bin/bash

# OpenShift Container Platform configuration template
# Refer https://github.com/openshift-metal3/dev-scripts/blob/master/config_example.sh
#
set +x
export CI_TOKEN=$(cat /home/zuul/src/github.com/openshift-metal3/dev-scripts/ci_token)
set -x

if [ -z $CI_TOKEN ]; then
    echo "no ci token"
    exit 1
fi

export OPENSHIFT_RELEASE_TYPE="ga"
export OPENSHIFT_VERSION="4.18.3"
export CLUSTER_NAME="ocp"
export BASE_DOMAIN="openstack.lab"
export NTP_SERVERS="ntp.nask.pl"
export AGENT_E2E_TEST_SCENARIO=SNO_IPV4
export NUM_MASTERS=1
export NUM_WORKERS=0

export MASTER_VCPU=20
export MASTER_MEMORY=32000
EOL

curl -LO https://downloads-openshift-console.apps.ci.l2s4.p1.openshiftapps.com/amd64/linux/oc.tar
tar xaf oc.tar
sudo mv oc /usr/local/bin/

cd /home/zuul/src/github.com/openshift-metal3/dev-scripts/

echo "Now run `make agent`"
make agent
EOF

then:

pre.sh
# reboot

execute.sh

NOTE: if you are doing that on VM, make sure that the disk is not Virtio, but SCSI virtio. It will give you a lot more performance.

[github-actions]

This PR is stale because it has been for over 15 days with no activity.
Remove stale label or comment or this will be closed in 7 days.

[bogdando]

NOTE: if you are doing that on VM, make sure that the disk is not Virtio, but SCSI virtio. It will give you a lot more performance.

note, I drafted this #3271

btw, regarding export NUM_MASTERS=3 setting provided in the above snippet, should I use

cifmw_devscripts_config_overrides:
   num_masters: 3

for this SNO setup to function?

[danpawlik]

I suggest not to use this for now. Temporary postponed working on it.
Probably @tosky have another PR/solution that can be used for single node.

[danpawlik]

@bogdando I updated script for deploying SNO with devscript. TBH, something changed on Intel CPU, that I have so big difficulties to stop at the same point as 3 months earlier. Totally impossible.
Could you try on the VM, how it works please?

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/1bdadf10d3ab4e3883f87a789691fe14

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 43m 12s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 28m 33s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 26m 08s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 54s
❌ cifmw-pod-pre-commit FAILURE in 10m 18s
✔️ build-push-container-cifmw-client SUCCESS in 22m 26s
✔️ cifmw-molecule-devscripts SUCCESS in 11m 14s
✔️ cifmw-molecule-dnsmasq SUCCESS in 4m 39s
✔️ cifmw-molecule-libvirt_manager SUCCESS in 35m 05s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 12s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ccf3b99d74dd46eb80b839eb0fee2954

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 35m 35s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 21m 47s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 19m 57s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 47s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 24m 33s
❌ cifmw-pod-pre-commit FAILURE in 8m 24s
✔️ build-push-container-cifmw-client SUCCESS in 19m 05s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 55s
✔️ cifmw-molecule-dnsmasq SUCCESS in 5m 14s
✔️ cifmw-molecule-libvirt_manager SUCCESS in 34m 38s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 05s

[github-actions]

This PR is stale because it has been for over 15 days with no activity.
Remove stale label or comment or this will be closed in 7 days.

[danpawlik]

Closing PR, due it is not critical in prios and don't have time to work on that now.

[bogdando]

@danpawlik we are going to use this SNO for a single host BM hybrid setup (BM SNO + BM EDPM). So I will circle back to you after the testing is done on my side

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danpawlik for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[softwarefactory-project-zuul]

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/ci-framework for 3129,717e839d30b1121fd114ac7e09bebd78b28563c9

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/fdddb7708e5e4dfd810c292392722293

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 14m 19s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 20s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 29m 00s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 01m 18s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 56s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 51s
❌ cifmw-pod-pre-commit FAILURE in 8m 08s
✔️ cifmw-molecule-devscripts SUCCESS in 11m 00s
✔️ cifmw-molecule-dnsmasq SUCCESS in 4m 44s
✔️ cifmw-molecule-libvirt_manager SUCCESS in 39m 29s
✔️ cifmw-molecule-reproducer SUCCESS in 14m 28s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d18a922993814190b7b06ab242578905

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 14s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 06s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 32m 41s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 56m 28s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 51s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 58s
❌ cifmw-pod-pre-commit FAILURE in 8m 39s
✔️ cifmw-molecule-devscripts SUCCESS in 11m 20s
✔️ cifmw-molecule-dnsmasq SUCCESS in 4m 54s
✔️ cifmw-molecule-libvirt_manager SUCCESS in 41m 17s
✔️ cifmw-molecule-reproducer SUCCESS in 14m 48s