We are coming up on branching for Katello 3.16 (Alongside foreman 2.1). The general plan had been:
3.16: Introduce yum with pulp3
4.0: Drop pulp2
As we approach branching, we are still about 4-5 weeks away from feature complete with yum support in pulp3. There are still around 8 issues until feature complete, but most of the larger ones are nearing completion. Given that we are not going to make branching in 1 week, we have a few choices:
a) Push the timeline for 4.0 3 more months
b) do a 3.17 release a month or so after 3.16
c) hold 3.16 for ~4-5 weeks
My proposal is c).
we still branch as normal and start to produce rc builds
the rc period will last the additional 4-5 weeks
we will backport weekly to the 3.16 branch and produce a new rc
Also, our current 3.16 tracker has ~34 issues assigned to it (Issues - Katello - Foreman) but we will work on trimming that down this week at our triage.
Thoughts on choosing c) as well as the details around that choice?
If the eventual GA is released more than a couple of weeks after Foreman GA - We saw many users who attempted to upgrade in the two weeks between Foreman 2.0 GA and Katello 3.15 GA and hit issues due to that. We can try to communicate better that Katello users shouldn’t upgrade yet, but the longer the gap grows the more users will hit this because not everyone reads the release announcements carefully or at all.
Changes merged into core after branching could require changes in katello, which would make rebasing 3.16-stable on master impossible to achieve. Unlike master, the stable branch only gets tested when a release is made, meaning we might end up with multiple DOA RCs due to incompatibility with Foreman stable.
Even assuming all of the pulp changes are indeed done in 4-5 weeks, that means katello GA will ship code that hasn’t even gone through nightly testing, which could result in a very unstable release. Any delays on the pulp side would cause even more delays in Katello.
Now, given that we haven’t had green katello nightlies in a long time, there could be more issues we aren’t even aware of. This also means that we aren’t even capable of building an RC right now, which worries me a lot.
I believe that perhaps option B would be more prudent. We already have a precedent of doing a Katello release mid-stream with 3.10. This will allow both properly focusing our efforts right now on getting nightlies green (hopefully) prior to branching, as well as properly handling the needed changes wrt pulp3 support for yum. If the changes in pulp require more time then expected, we can always fall back to option A instead of having a very long period with no up to date katello release.
Packaging wise this is becoming more difficult. As we’re merging more and more into basic Foreman workflows (foreman-packaging, foreman-installer), we’re becoming more reliant on Foreman. This means that if there’s a change needed in the installer that doesn’t work with 3.16, it becomes more difficult. For 3.10 this was no problem because there was no installer difference. With 3.16/3.17 this is more likely due to Pulp differences.
This brings the potential solution, option D) which is an additional Foreman 2.2 release. I haven’t fully thought this through, but since there’s no more separate katello-{packaging,installer} repositories this might make things a lot easier on the platform and delivery side.
This sounds like a broader conversation worth having if its becoming more common place. As it implies to me that our users perceive our releases and community differently than how we are approaching and documenting it. We can try harder, make it more bold to note this. Foreman and Katello are tightly coupled and we have a significant user base for whom this affects their environments and upgrading. Perhaps we need to consider making changes that help meet our users expectations.
Long term, this will require the installer work being discussed and Katello on Foreman. In the case of any plugin, this can at times actually be a problem. With Rails 6, for example, users who have any plugin enabled will require that plugin to be released to upgrade their entire installation.
This is always a risk. I don’t forsee this workflow being any worse. It often takes 4-5 weeks for us to get a final release out anyway.
That’s not exactly true. The changes will merge to master, flow through nightly and be pulled into the stable branch. That would be the exact same path code targeted for a 3.17 would follow. 3.17 would have to be released against Foreman 2.1 anyway, so this isn’t gaining much. If a 3.16 provides value to users, then I believe that should determine the path.
I think a 3.16 followed shortly by a 3.17 will just lead to more churn for everyone.
This is shades of grey. Failing pipelines do not imply everything is broken. Currently, the state is that Pulp 3 paths are not being tested due to an ordering bug in the installer that is persisting. @ekohl has looked into this and got blocked on Puppet dependency chaining. I believe he has a proposal for how to fix it we should need to prioritize it. The alternative is a hacky solution @Justin_Sherrill is proposing for a dynflow task to try and “fix” the problem.
