Another thought. Since we, at least in nightly, split up the deb and rpm nightly pipelines, perhaps plugins should be split off too? It does mean that for release pipelines you need to trigger 2 jobs, but I think that’s ok.
Any chance we could get a couple more machines on centos ci? we’re always hitting the same issue on releases as well.
Another option is not to start 8 machines at once, but rather first run 4 install tests and only once they pass run 4 upgrade tests - it would mean the full pipeline run will take about 3 hours instead of ~1.5 it does now, but considering these are mostly nightly jobs no-one is waiting for the results anyways (the exception being release jobs). That would also mean that if the build is totally broken we only run 4 machines instead 8.
Note that currently we get a single (physical) machine for this test and run all VMs on that. That’s at limit. Splitting it into two jobs works around that by getting 2 machines as hypervisors. Technically we could do it as part of a single job as well, but given the packaging pipelines are already split, it makes sense to do the testing separate as separate jobs as well.