Proposal: Merging foreman pipelines with pipelines of other plugins

[Justin_Sherrill]

I love all those ideas! Lets do those

@ekohl

On the other side: Foreman runs the Katello unit tests as well for every core PR but they break all the time and personally I’m ignoring them more often than not.

From my experience among the failures, i typically see about ~60% CI issues and ~40% random test failures. If we see random test failures, we typically try to fix them, but they still pop up from time to time. I’ve also noticed that sometimes the katello failures on the foreman PRs seem to be different than the failures on the katello prs (which is strange). Anyways, i think if we as katello developers take it even more seriously (investigate and fix each and every one), we can knock down that 40%. Lets start filling issues for breakages (or at least bringing them up on #theforeman-dev) and get them knocked out. As for the CI issues, that’s a bit out of my wheelhouse, maybe we can adjust some slave sizes, or reduce the number of simultaneous jobs?

We may end up in a similar situation where Katello is blocking things and nobody knows how to solve them. Timezones are an additional problem: most Katello developers are in the US and if you’re blocked until halfway through the day it quickly becomes very annoying.

So in short: I’m advocating for making Katello less special and more of a generic plugin. However, there are serious concerns over availability/reliability, both in people and technical.

I don’t fully have answers to this, as a global project we already have these issues. Having a bit of delay with stable nighties is better than moving faster with broken nightlies (and i get that foreman nightlies are stable in these cases).

I’m a bit reminded of the story around toyota vs the american automakers in the late 1980s, early 1990s. At the time American carmakers would fight hard to not stop the assembly line. Any issue that popped up would just be pushed on through and be fixed at the end. Cars came out of the factory half assembled and would have to be fixed. It was seen as an individual failure if the assembly line ‘stopped’ due to an issue at your station. As the Japanese automakers rose in popularity they used a different approach. Any issue on the line caused the line to immediately stop and the problem was seen as an opportunity for process improvement. Cars came out completed and done, and there were fewer one offs that had to be fixed after the fact. Because everyone was so focused on improving the process and quality, their quality levels far outpaced american cars at the time.

This isn’t a one to one match for our situation by any means (please don’t read too much into and try to align any particular aspect with our situation), but i think goes to what would help, which is a more singular focus around the nightlies.