We’ve had a very rocky past month where a lot of changes were pushed in. Rebuilding plugins to the new Javascript packages proved very painful and slow. Also the hammer bump to 2.0 caused some issues. This meant that during January nightly was usually broken to the point that for the conferences the demo system was using a developer setup rather than packages as we’ve always done. During the same time we branched for Foreman 2.0 and only stablized some things in the stable branches.
Fast forward to today and Katello 3.15 still hasn’t been released which means that the Katello nightly upgrade tests are failing (since that installs 3.14, upgrades to 3.15 before upgrading to nightly). This means the installer work can’t is getting blocked since packages like pulpcore-selinux are not promoting to the repositories.
At the same time the manual hasn’t been updated yet to reflect the changes (leading to posts like Foreman 2.0rc1 install problems).
This has been the messiest branching + release in the past 2 years and not the rock solid 2.0 release we want to provide. This concerns me and personally I expect that we’ll hit RC3 or RC4 just to fix all the common issues users run into.
I think this should allow Katello 3.15-rc1 to be released. After that, the nightly pipeline should complete which should allow regular development to continue.
What have we changed comparing to previous branching/releases? What do you suggest should be improved in order to avoid this? Is is the capacity issue? Is it that we did bigger drops of deprecated things or new features going in too late?
We certainly pushed a lot of changes, especially around the installer and packaging. These always have some fallout because they’re hard to properly review and test due to the many edge cases. Within our team we’ve been talking about ways to improve testing but a lack of time means that few are implemented. Classically we’ve always dealt with this by stabilizing in nightly.
I believe that we’ve tried to push a lot of things into 2.0 because it was a major version and suddenly we could make bigger changes that we otherwise didn’t. This is why I’ve been advocating for doing major releases more often. If we do a yearly major release, we can also get closer to semantic versioning. Things like any Foreman 2.x can talk to any Foreman Proxy 2.x (or show clear errors where an incompatibility has been detected) would become reasonable.
Another compounding factor is that we were delayed. Normally a delay isn’t that bad, but now it pushed stabilization week into the conference week. I’ve seen little to no work done on making sure that nightly was in a shippable state.
We should evaluate the release schedule and see if it makes sense to shift things around to avoid the conference season. In the future I’d certainly avoid doing a major release at the start of the year.
Lastly there was poor communication and synchronization between the Foreman and Katello process. I believe that we should improve procedures to indicate where one has to wait for something. IMHO Katello should also post the procedures as checklists on the releases sub-forum like Foreman does. This makes the process more transparent and collaboration easier.
Thanks for the original update. Any recent updates on the state of things, specifically on getting a katello nightly box up? I noticed the PRs in the original message are merged.
It would be nice to do some pulp3 migration testing on a nightly box.
