There’s a lot to unpack here. First of all, I want to be explicit in that I consider working nightlies as crucial to the project. Any critique I have is about implementation.
On the technical level, I’m afraid our current pipelines wouldn’t handle the concurrency. The way we call jobs is based on kicking off a job and waiting for one to start. If two projects start the same pipeline (foreman + foreman-installer), it’s hard to track which one started which.
Another is who picks it up when something fails. There was Proposal - nightly pipeline monitoring rotation but it hasn’t been implemented yet. Adding more things that might fail might just make it worse.
My suggestion would be to start small. In the Build Package phase, I’d suggest we add more checks on dependencies. I use a script to wrap update-requirements. My suggestion would be integrate that into CI and if the RPM spec file differs from what the package requires, the nightly build should fail rather than attempting to build something and possible generate garbage.
The benefits is that it helps the quality in a way that’s easy to solve. It can allow us to build a process where we investigate the build failures.
I also believe this would catch more issues than actual end to end tests. Our coverage is very low (at least in Foreman, Katello is in a better position - which is why it’s red for longer). I’ve tried with smoker but increase coverage but got stuck on UI and received no help.
Regarding gating Foreman on Katello passing, I think we’re too soon on that. I’d actually consider moving the Katello GIthub organization into theforeman and treating it like any plugin. Right now the groups of people with access on both is limited which limits our ability to fix things, even if we know the solution.
tl;dr: good long term vision, but let’s start small and iterate.