Rails 7.0.1 was at least the first one to support Ruby 3.1 (Ruby on Rails — Rails 7.0.1 has been released), which is included in Debian 12. It’s also the first one to say Ruby 3.0+ preferred. So perhaps it’s based on an assumption on my side.
As promised, I’m summarizing our weekly sync up here.
We briefly discussed the possibility to run Rails 6.1 with Ruby 3. @ekohl will look at introducing expanding the Foreman Jenkins job to start testing on Ruby 3.
@ekohl will rebase the PRs to resolve merge conflicts, @ofedoren will look at reviewing
@ofedoren looked into Zeitwerk. We’ll need to enable eager loading in our test environment (which we already do for dev & prod). The previous time we tried that, some tests started to fail. This may also help with the compute resource loading problems we saw.
Priority wise the focus is no longer on Rails 7.0 & Zeitwerk, since it should be possible to deliver Rails 6.1 (which we use today) on Ruby 3.0. That should bring us to EL9 faster. Rails 7 should still happen, but it’s more important to start shipping on EL9. Given the Foreman 3.9 development cycle ends in about 2 weeks, we should aim at Foreman 3.10 instead.
Due to people having time off we didn’t have the regular sync up, so it’s been a bit quiet on the updates. That doesn’t mean no work has been done.
@ofedoren has been working on testing Foreman with Ruby 3 in CI (Fixes #36849 - Set up CI with Ruby 3.0 by ofedoren · Pull Request #9871 · theforeman/foreman · GitHub). We’ve decided to split this up into two parts: one part that adds GitHub Actions based testing with the current status (Ruby 2.7). The follow up will add Ruby 3.0 to the testing matrix. @ekohl will write up a separate RFC on this topic so it’s also clear what the plan for plugins is. This RFC is already big enough and it would get lost here.
@akumari has also started to look at updating GitHub - theforeman/theforeman-rubocop: Foreman RuboCop basic rules to the latest version and making it the recommended Foreman & Foreman plugin RuboCop configuration. This is useful because there are various cops that help with the Ruby 3 upgrade (Lint/DeprecatedClassMethods being the most obvious, but there are more).
First of all, @evgeni and @ehelms will try to assist in the effort. We took a look at the overall structure of the project to try and divide tasks.
Taking a high level overview, we can identify a few subsystems (assuming a full Katello installation):
Foreman
Foreman Proxy
Hammer
Pulp
Candlepin
For simplicity, I consider plugins part of the subsystem.
Broadly speaking we consider some subsystems ready for EL9, but sometimes untested. Foreman Proxy, Hammer, Pulp & Candlepin all should work but most of the readers will know that theory and practice don’t always agree.
For Foreman we still have quite a bit of work to do and that’s what we’ll continue to focus on. As mentioned previously, the whole CI setup needs an overhaul. Foreman has long used Jenkins, but it needs an update. I’m preparing a separate RFC for that, but the summary is that we want to use GitHub Actions.
The PR to build packaging on EL9 (Build Rails for EL9 by evgeni · Pull Request #10015 · theforeman/foreman-packaging · GitHub) has been discussed. The approach is to create a new config in COPR that has both EL8 & EL9 chroots, so it can build them in parallel. We can then bootstrap the platform by moving groups over, in the correct order. It will be good to keep notes since we’ll apply the same thing once EL10 comes around the corner.
Next week’s meeting is cancelled because most people have other obligations.
@evgeni has made a lot of progress packaging large parts on EL9. Next steps are to build the Smart Proxy plugins and publish the packages on yum.theforeman.org so we can start updating our Puppet modules to also test it.
@akumari has also been traveling and had time off.
As every year, Ruby has released a new version over Christmas. Ruby 3.3 appears to be small in terms of breaking changes. While reading up on it, I noticed that Ruby 3.0 is EOL in March 2024, so in 3 months. I’d still go forward with releasing it on EL9 on 3.0, even if it is EOL because Ruby 2.7 is already EOL since March 2023 anyway and that hasn’t held us back.
The dependency on foreman-debug and foreman & foreman-proxy has been dropped (Fixes #37022 - Drop requirement on foreman-debug · theforeman/foreman-packaging@acccb9a · GitHub) since it’s not really needed to run and simplifies publishing a minimal set. We’ve already largely switched over to sosreport anyway and for end users it also reduces the required dependencies on minimal installations.
How to install test dependencies? Today Jenkins for regular plugins uses gemfile.d/*.rb and our reusable actions do the same, but Katello has a special job in Jenkins that installs it as a gemspec, which installs all development dependencies. This came up in https://github.com/Katello/katello/pull/10780.
Today Katello’s tests run on every Foreman PR using Jenkins. Should this be ported to GitHub Actions?
One remaining issue that shows up in most plugins is running of tests from Foreman core in a plugin (always execute access_permissions_test from Foreman core by evgeni · Pull Request #16 · theforeman/actions · GitHub). This isn’t an issue in Jenkins, because that runs the entire Foreman test suite (to prevent regressions). GitHub actions only runs the plugin’s but some do run some tests from core. This needs a resolution for plugins using GitHub Actions today.
Packaging wise all EL9 RPM workflows are present: Foreman, Plugins & Katello. The list of packages is incomplete, but a content proxy now passes tests. Remaining is Foreman & its plugins. That’s blocked on Ruby 3 & Webpack 5.
For Webpack 5 the current proposal is to pull the trigger this Friday (2024-01-26). A separate announcement for that will be posted.
Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub has been extended with all the plugins from Foreman landscape that matter. To make this easier, the landscape has also been updated with the testing overview (Jenkins, GitHub Actions, Puppet). This will be even more important because after we merge Webpack 5 we will start to depend on NodeJS 14 and many current GitHub Action implementations still use NodeJS 12. So implicitly there’s also a deadline for this.
In some repositories (at least foreman_openscap) we’ve seen that RuboCop didn’t ran before. For those cases it’s OK to just convert the existing tests, but long term that should be figured out. It’s confusing when there is a RuboCop configuration, but it doesn’t run.
@evgeni has also done some initial work on Debian 12 & Ubuntu 22.04. Debian 12 is Ruby 3.1, but that requires Rails 7 (including Zeitwerk). Ubuntu 22.04 has switched compression, which doesn’t work with our own mirrors right now. This for sure won’t happen (not even experimental) for Foreman 3.10.
The big thing is that we are indeed going to merge webpack 5 tomorrow (Webpack 5 merge this Friday (2024-01-26)). To facilitate this the continued focus has been on the GitHub Actions conversions for plugins. By now for almost all the plugins there’s a PR and various PRs have been merged. While doing this, it was noticed that foreman_setup would need work, but it’s rather unmaintained so I’ve proposed to drop it: Retire foreman_setup plugin
Like with webpack we want to set a date where it’s merged. This date will be Tuesday (2024-01-30). That way we don’t (plan to) overlap with the webpack while also providing sufficient time to fix things before branching and before FOSDEM and cfgmgmtcamp.
Something I now notice about last week’s update: we are talking about delivering EL9 support in Foreman 3.10 as experimental. Mostly because this means we can deliver it without any EL8 → EL9 migration support (and postpone that to Foreman 3.11). Of course users can do this on their own, but they’re properly warned.
In the past week the focus has been less on Ruby and more on NodeJS. As announced we merged webpack 5 and as expected, there was some fall out. Nothing major, but it occupied us nonetheless. For all the details, see Webpack 5 merge this Friday (2024-01-26).
One uncomfortable thing that showed up NodeJS 16. This is needed for EL9. It looks like we still need legacy peer dependencies handling with NPM. Another is that on EL9 the development packages for NodeJS 16 are missing. Because of that the current investigation is looking at NodeJS 18. That version also ships NPM 8 and we expect few problems. Fedora 38 ships it, which means you can finally just use the system packages in your development environment. Fedora 39 ships NodeJS 20, so that support is also considered. Active work is in Fixes #37134 - upgrade to node 18 and npm 8 by MariaAga · Pull Request #10016 · theforeman/foreman · GitHub.
If we look at the Foreman 3.10 schedule then it’s getting rather close. Next week we have CfgMgmtCamp 2024 where both @evgeni and I will attend. The week after that it’s already stabilization week.
I see 3 options:
@ehelms has time to work on the NodeJS > 14 upgrade next week and can get it done
We postpone Foreman 3.10 branching for this
We postpone Foreman on EL9 to 3.11
@evgeni notes that it’s very useful to have a non-official Foreman on EL9 release (as 3.10 was intended to be) because that makes upgrade testing possible.
@ekohl will discuss with @ehelms and the @releases team what’s the best course of action here.
Good news! Thanks to @MariaAga in Node 18 PR I had the thought we should be able to use the same --legacy-peer-deps in our NodeJS packages that are bundled and achieve the same result. That is, a package that works on 14 and 18, easing the pain of transition by allowing us to rebuild on 14 and then the packages continue to work on 18.
I undertook and tested the following:
Building all packages on NodeJS 14 with the --legacy-peer-deps flag
Building all packages on NodeJS 18 with the --legacy-peer-deps flag
I have opened a PR to npm2rpm to add support so that not only will the spec files get generated by our update script properly, but also so that the caches are generated with legacy peer deps.
The meeting yesterday didn’t happen, as several people were either still traveling or recovering from travel/conferences.
Nevertheless, progress happened, so here is a few highlights from me:
thanks to the work of @MariaAga and @ehelms on NodeJS 18 / NPM 8+, we now have rebuilt our RPM packages with --legacy-peer-deps, which allowed us to also build them on NodeJS 18 on EL9 (EL8 remains on NodeJS 14 for now)
this allowed us to build Foreman and almost all Plugins in the “plugins” repository for EL9 (using Ruby 3 and NodeJS 18). notable exceptions are:
when deface and rex packaging changes were applied, and using the patched selinux policy, the “foreman” and “plugins” pipelines pass fine (well, plugins do if we disable discovery), which is a HUGE milestone! (those tests include REX/Ansible, which also validates that dynflow and friends are working)
the next steps will be tackling Katello and the plugins that depend on it
They aren’t blockers for RC1, but we should aim to get both resolved for RC2.
To be explicit: the Foreman 3.10 goal is experimental support. We won’t provide a migration path from EL8. We expect it to largely work, but it’s a large upgrade and we may have missed things. Foreman 3.11 aims to complete the work. This means:
Upgrade all platforms to NodeJS 18 (instead of just EL9)
For both I’m tempted to split them off to a separate project and rename the existing one to EL9 support, just to keep things clear.
Other follow ups for 3.11 include:
Start building for Ubuntu 22.04, which includes Ruby 3.0 so it should just work
Start the Zeitwerk migration to get us on a path to Rails 7, opening Ruby 3.1+
Ruby 3.1+ is important, because Debian 12 and Ubuntu 24.04 both ship Ruby 3.1. Right now CentOS Stream 10 is also being branched. Depending on which Fedora version they base it on, I’d expect Ruby 3.2 or 3.3 (with the latter being more likely).
As a last note we’ve started to discuss EL8 server support deprecation. We want to give users a smooth, well tested path so we’re currently looking at 3.13 to drop EL8. This will be communicated in the 3.10+ release notes.