CentOS CI migration completed

CentOS CI (which we rely on for our pipelines) has been making changes. Their legacy bare metal machines were phased out. This meant we had to change our approach.

Taking foreman-nightly-rpm-pipeline [Jenkins] as an example, we have a number of operating systems (CentOS 8 Stream & AlmaLinux 8 at the moment). and for each we do both an install and upgrade tests. This results in 4 pipelines that we run to verify things.

Where we previously provisioned one bare metal machine per pipe (meaning 4 machines), in the new approach getting multiple physical machines is slower, but each machine individually is (much) faster. It has the capacity to run all pipelines on a single machine, so that’s now what we do.

There were various obstacles along the way, but starting today all our pipelines are green again.

For comparison, foreman-nightly-rpm-pipeline went from about 70 minutes to about 37 minutes. luna-nightly-rpm-pipeline went from abour 4 hours to about 2 hours. So roughly twice as fast.

@evgeni has been doing all the hard work.