Fast forward to today and Katello 3.15 still hasn’t been released which means that the Katello nightly upgrade tests are failing (since that installs 3.14, upgrades to 3.15 before upgrading to nightly). This means the installer work can’t is getting blocked since packages like pulpcore-selinux are not promoting to the repositories.
This has been the messiest branching + release in the past 2 years and not the rock solid 2.0 release we want to provide. This concerns me and personally I expect that we’ll hit RC3 or RC4 just to fix all the common issues users run into.
What have we changed comparing to previous branching/releases? What do you suggest should be improved in order to avoid this? Is is the capacity issue? Is it that we did bigger drops of deprecated things or new features going in too late?
We certainly pushed a lot of changes, especially around the installer and packaging. These always have some fallout because they’re hard to properly review and test due to the many edge cases. Within our team we’ve been talking about ways to improve testing but a lack of time means that few are implemented. Classically we’ve always dealt with this by stabilizing in nightly.
I believe that we’ve tried to push a lot of things into 2.0 because it was a major version and suddenly we could make bigger changes that we otherwise didn’t. This is why I’ve been advocating for doing major releases more often. If we do a yearly major release, we can also get closer to semantic versioning. Things like any Foreman 2.x can talk to any Foreman Proxy 2.x (or show clear errors where an incompatibility has been detected) would become reasonable.
Another compounding factor is that we were delayed. Normally a delay isn’t that bad, but now it pushed stabilization week into the conference week. I’ve seen little to no work done on making sure that nightly was in a shippable state.
We should evaluate the release schedule and see if it makes sense to shift things around to avoid the conference season. In the future I’d certainly avoid doing a major release at the start of the year.
Lastly there was poor communication and synchronization between the Foreman and Katello process. I believe that we should improve procedures to indicate where one has to wait for something. IMHO Katello should also post the procedures as checklists on the releases sub-forum like Foreman does. This makes the process more transparent and collaboration easier.