This is a tracking thread to cover the activities being under taken to get our RPM deployments moved over to Ruby 2.7. The biggest hurdle is the mass rebuilds of RPMs that has to occur whenever we undertake this kind of change. I am creating this tracking thread to bring awareness to the various efforts required to inform this as well to get the communities help in tackling tangential but required PRs.
There is a Redmine Tracker to track the code and/or testing changes needed for each project. If you are a plugin maintainer who runs your testing not through Jenkins and are not already testing against Ruby 2.7, please either update your plugin to do so or ping this thread and I will help look into.
There is a Packaging Milestone where I am collecting a broken out set of PRs that will be needed to in order to rebuild all of the RPMs. My goal here is to get PRs open with reasonable chunks of packages for review (should largely be release bumps) to then find a 2-3 day window to orchestrate rebuilding everything.
There are a few Jenkins job related PRs being tracked as well to get projects not currently testing against Ruby 2.7 to do so:
There will be at least one additional infrastructure change in Forklift to switch to rh-ruby27 as the default within the necessary roles, playbooks and devel puppet module.
A tangential blocker is that the following list of plugins will not currently RPM build due to dependencies that cannot be currently satisfied within the packaging environment.
foreman_leapp
foreman_statistics
foreman_templates
foreman_puppet
foreman_webhooks
foreman_discovery
In all cases they are requiring a version of @theforeman/builder less than 7 and the current released version (at the time of this post) of @theforeman/builder is 8.4.1 in nightly. For example:
Getting requirements for tfm-rubygem-foreman_webhooks-0.0.1-2.el7.src
--> Already installed : scl-utils-build-20130529-19.el7.x86_64
--> foreman-assets-2.5.0-0.3.develop.el7.noarch
--> foreman-plugin-2.5.0-0.3.develop.el7.noarch
--> rh-ruby27-ruby-libs-2.7.1-128.el7.x86_64
--> rh-ruby27-ruby-2.7.1-128.el7.x86_64
--> rh-ruby27-rubygems-devel-3.1.2-128.el7.noarch
--> tfm-nodejs-babel-core-7.7.4-2.el7.noarch
--> tfm-nodejs-babel-core-7.7.4-2.el7.noarch
--> tfm-nodejs-theforeman-builder-8.3.3-1.el7.noarch
Error: No Package found for tfm-npm(@theforeman/builder) < 7.0.0
Given this dependency is controlled by each plugins package.json requirements. This will require each plugin to make the appropriate update and perform a release w/ corresponding packaging update (I can help with this).
Would it make sense to drop the higher version limit for @theforeman/builder in packaging or in each package.json?
I am not sure how much had actually changed comparing to old versions of the builder, as we use lerna in theforeman/foreman-js and sometimes it just bumps all of the project’s packages to keep them aligned, any thoughts on this @sharvit@evgeni ?
Yeah, Avi has been saying that builder/vendor has been rather stable in terms of API and we’ve not seen plugins fail to build when they’d specify ^5 or similar as dep but being built with a much newer version.
That said, it’d be nice if the upper bound would be defined in the plugin more correctly (so >=5, if we don’t know which version will break, but are rather sure 6 or 8 isn’t) or, even better, if builder/vendor wouldn’t bump major version so often. My understanding is that you currently bump all packages simultaneously, and that’s what trigger major bumps even if the breaking changes don’t affect builder/vendor.
It seems that the fact we use Lerna to do releases of all modules concurrently actually breaks sevmer for the specific modules - e.g. builder didn’t have any breaking changes but bumps major version due to breaking change in vendor. This causes such issues since we can’t actually rely on the version numbers so either we bump versions for no reason or remove the version limits and risk breaking when an actual breaking change is merged. Perhaps something that should be discussed at the next @ui_ux interest group meeting.
I had a feeling we’ve been in this discussion already @John_Mitsch did we ever do anything about switching lerna to independent mode instead of fixed mode?
Yesterday on the foreman-js maintainers meeting, we discussed about that too,
and thought that it’s worth to explore if we can use lerna with different packages major versions ,
also for builder, it’s worth checking if we can import it once somewhere in Foreman core and remove the duplication across plugins… but we can talk all about that after @sharvit deep dive about the builder in the next UX interested group
For now, I think that doing something like @theforeman/builder: >= 8.3.3 in plugins package.json should resolve our current issue
There are a few outstanding plugins that need updating to be able to build but given that condition exists even prior to Ruby 2.7 I would like to move forward with building against Ruby 2.7. All package updates that are the result of testing and were stand-alone have been merged and I am down to the rebuild point. You can see that by what is linked to the milestone.
The rebuild is a large effort, 2-4 days of building, and then testing and getting pipelines passing. This is unfortunately a disruptive change for building and pipelines but I aim to minimize it as much as possible by maximizing the build throughput across those days.
I would like to target this week (29/3/2021 - 2/4/2021) or next week (5/4/2021 - 9/4/2021) to iterate through the building.
I am planning to start this process on Monday the 4th of April working in conjunction with @pcreech. Given the holidays in some areas on Monday, I am hoping to use this quieter period to get through the bulk of builds and ultimately reduce disruptions. This will lead to outages in our pipelines as we churn through the build process. Between the two of us the hope is to limit this to 2-3 days max.
Building of Ruby 2.7 packages is nearly complete. I should be able to resolve everything on Wednesday as I am down to just re-building Foreman plugin packages. There are still some plugin packages that are unbuildable as noted previously in the thread.
The Foreman pipeline has passed on EL7 thus far, with a change needed for EL8 to handle the switch in modularity streams for our pipelines:
RPM builds have “completed” for everything that is known. A reminder the following plugins could not be built and will remain out of date until they are updated:
foreman_azure_rm
foreman_statistics
foreman_templates
foreman_puppet
foreman_discovery
Testing of the builds continues with passing install tests for Foreman on EL7 and EL8 now. Upgrades for EL7 are being worked on as we currently test that no rh-ruby25 packages are needed and we need to update our pipelines to remove those packages as a user would and ensure things continue to work.
