The two failed runs are because we can’t run that many VMs at once on CentOS-CI.
My suggestion would be we disable one debian-install and ubuntu-upgrade, pretending that if ubuntu-install worked, so would debian, and if debian-upgrade worked so would ubuntu?
Another thought. Since we, at least in nightly, split up the deb and rpm nightly pipelines, perhaps plugins should be split off too? It does mean that for release pipelines you need to trigger 2 jobs, but I think that’s ok.
Any chance we could get a couple more machines on centos ci? we’re always hitting the same issue on releases as well.
Another option is not to start 8 machines at once, but rather first run 4 install tests and only once they pass run 4 upgrade tests - it would mean the full pipeline run will take about 3 hours instead of ~1.5 it does now, but considering these are mostly nightly jobs no-one is waiting for the results anyways (the exception being release jobs). That would also mean that if the build is totally broken we only run 4 machines instead 8.
CentOS CI is rather at-limit, so probably not (but we didn’t ask specifically).
I am not a huge fan of the fact that we sometimes have rpm/deb split and sometimes not. Maybe we should just always split? (But yes, making those two independent would help here)
Note that currently we get a single (physical) machine for this test and run all VMs on that. That’s at limit. Splitting it into two jobs works around that by getting 2 machines as hypervisors. Technically we could do it as part of a single job as well, but given the packaging pipelines are already split, it makes sense to do the testing separate as separate jobs as well.
(But yes, making those two independent would help here)