Path to Ruby 3.0, 3.1, EL9 and Ubuntu 22.04

I tried to look for official support, but these guides don’t explicitly say “now it runs with Ruby x.y”. I looked at:

Rails 7.0.1 was at least the first one to support Ruby 3.1 (Ruby on Rails — Rails 7.0.1 has been released), which is included in Debian 12. It’s also the first one to say Ruby 3.0+ preferred. So perhaps it’s based on an assumption on my side.

But you’re right. Ruby & Rails Compatibility Table - | Rails Upgrade Service states that is recommended to run on Ruby 3.0.Z. I suppose I when I started, I was very focused on getting it to run on my Fedora which included Ruby 3.1.

That does change the dependencies a bit and allows for more parallelization.


As promised, I’m summarizing our weekly sync up here.

We briefly discussed the possibility to run Rails 6.1 with Ruby 3. @ekohl will look at introducing expanding the Foreman Jenkins job to start testing on Ruby 3.

@ofedoren noticed that openscap_parser started to require Ruby 3, though that may have been a mistake (fix: RHICOMPL-3900 disable required ruby version rubocop check by marleystipich2 · Pull Request #56 · OpenSCAP/openscap_parser · GitHub).

@ofedoren is wrapping up some other work before he can start investigating the Zeitwerk status

@akumari is investigating the update to fast_gettext.


And another weekly sync up.


Due to people having time off we didn’t have the regular sync up, so it’s been a bit quiet on the updates. That doesn’t mean no work has been done.

@ofedoren has been working on testing Foreman with Ruby 3 in CI (Fixes #36849 - Set up CI with Ruby 3.0 by ofedoren · Pull Request #9871 · theforeman/foreman · GitHub). We’ve decided to split this up into two parts: one part that adds GitHub Actions based testing with the current status (Ruby 2.7). The follow up will add Ruby 3.0 to the testing matrix. @ekohl will write up a separate RFC on this topic so it’s also clear what the plan for plugins is. This RFC is already big enough and it would get lost here.

While testing @ofedoren ran into Fix possible nil volumes for libvirt by ofedoren · Pull Request #133 · fog/fog-libvirt · GitHub and @ekohl will do a release of fog-libvirt. That’s useful anyway because there are unreleased changes that bring compatibility with modern libvirt as included in EL9 (Set the default machine type to q35 on i686 and x86_64 by ekohl · Pull Request #127 · fog/fog-libvirt · GitHub & Fix disk type for block devices on QEMU >= 6.0 by ekohl · Pull Request #129 · fog/fog-libvirt · GitHub).

@akumari has been moving forward with the apipie-dsl translations. While no longer a blocker to upgrading, is still a nice step forward.

@akumari has also started to look at updating GitHub - theforeman/theforeman-rubocop: Foreman RuboCop basic rules to the latest version and making it the recommended Foreman & Foreman plugin RuboCop configuration. This is useful because there are various cops that help with the Ruby 3 upgrade (Lint/DeprecatedClassMethods being the most obvious, but there are more).


Another week, another update.

First of all, @evgeni and @ehelms will try to assist in the effort. We took a look at the overall structure of the project to try and divide tasks.

Taking a high level overview, we can identify a few subsystems (assuming a full Katello installation):

  • Foreman
  • Foreman Proxy
  • Hammer
  • Pulp
  • Candlepin

For simplicity, I consider plugins part of the subsystem.

Broadly speaking we consider some subsystems ready for EL9, but sometimes untested. Foreman Proxy, Hammer, Pulp & Candlepin all should work but most of the readers will know that theory and practice don’t always agree.

@evgeni wants to get started on packaging effort for those already (and Pulp is already done). So EL9 yolo by evgeni · Pull Request #9990 · theforeman/foreman-packaging · GitHub is the first kick off for that.

For Foreman we still have quite a bit of work to do and that’s what we’ll continue to focus on. As mentioned previously, the whole CI setup needs an overhaul. Foreman has long used Jenkins, but it needs an update. I’m preparing a separate RFC for that, but the summary is that we want to use GitHub Actions.

We also made sure Ruby 3 support · GitHub was updated.


Today’s meeting wasn’t very eventful and we only made sure Ruby 3 support · GitHub was updated and everyone was in sync again.

@akumari is picking up an update of theforeman-rubocop. This is not just updating the gem versions, but also looking at cops which make sense to enable. In addition to that, verifying Kafo works on Ruby 3 & Puppet 8.

@ofedoren was on PTO, but we have merged Fixes #36913 - Set up GHA with matrix to run test on Ruby 2.7 by ofedoren · Pull Request #9900 · theforeman/foreman · GitHub.

@evgeni is moving forward with packaging: Build Rails for EL9 by evgeni · Pull Request #10015 · theforeman/foreman-packaging · GitHub

@ekohl finally finished the RFC he promised: Convert Foreman & plugins continuous integration to GitHub Actions


Since last week there’s been a lot of progress for GitHub Actions in foreman.git (History for .github/workflows/foreman.yml - theforeman/foreman · GitHub) to the point that once also execute assets:precompile as part of GHA tests by evgeni · Pull Request #9940 · theforeman/foreman · GitHub is merged, the coverage should be the same as in Jenkins. An issue has been opened to get the reusable workflow to the same level of coverage: Bring back changes from Foreman core's workflow to foreman_plugins · Issue #23 · theforeman/actions · GitHub

Reviews on safemode PRs (Support safe call by ekohl · Pull Request #46 · theforeman/safemode · GitHub & were done. Due to travel and PTO @ofedoren may take over the effort from @ekohl.

The PR to build packaging on EL9 (Build Rails for EL9 by evgeni · Pull Request #10015 · theforeman/foreman-packaging · GitHub) has been discussed. The approach is to create a new config in COPR that has both EL8 & EL9 chroots, so it can build them in parallel. We can then bootstrap the platform by moving groups over, in the correct order. It will be good to keep notes since we’ll apply the same thing once EL10 comes around the corner.

Next week’s meeting is cancelled because most people have other obligations.


Due to travel and time off I haven’t been here for the past 2 weeks, but today we synced up again.

@ofedoren has released a new version of safemode that is compatible with Ruby 3.0. Foreman still needs to be updated (included in

@evgeni has made a lot of progress packaging large parts on EL9. Next steps are to build the Smart Proxy plugins and publish the packages on so we can start updating our Puppet modules to also test it.

@akumari has also been traveling and had time off.


As every year, Ruby has released a new version over Christmas. Ruby 3.3 appears to be small in terms of breaking changes. While reading up on it, I noticed that Ruby 3.0 is EOL in March 2024, so in 3 months. I’d still go forward with releasing it on EL9 on 3.0, even if it is EOL because Ruby 2.7 is already EOL since March 2023 anyway and that hasn’t held us back.


Today we synced up again after we’ve all been out for the holidays. Some updates since last time.


Foreman has been updated to use safemode 1.4.0, including tests that verify the new safe navigator operator in templates (Fixes #37010 - Support safe navigation operator in safemode by ekohl · Pull Request #9967 · theforeman/foreman · GitHub). Some safe changes for Ruby 3 compatibility have been merged (Refs #36849 - Ruby 3 preparations by ekohl · Pull Request #9975 · theforeman/foreman · GitHub).

This leaves the Ruby 3 enablement with a more controversial change. Both Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9871 · theforeman/foreman · GitHub & Fixes #36849 - Run GHA on Ruby 3.0 (alternative) by ekohl · Pull Request #9923 · theforeman/foreman · GitHub try to deal with the method_missing change. @ofedoren will look further into this. @ekohl has submitted an update to clarify the Ruby docs as well (Improve method_missing method documentation by ekohl · Pull Request #9412 · ruby/ruby · GitHub).

@ofedoren will start looking at testing plugins. test with ruby3-compatible foreman by evgeni · Pull Request #24 · RedHatSatellite/foreman_theme_satellite · GitHub is the way for plugins that use our reusable GitHub Action.


On the packaging side some progress has been made:

With all of these changes we’re close to publishing el9 nightly repositories (enable nightly el9 repo generation, closure and publishing by evgeni · Pull Request #395 · theforeman/jenkins-jobs · GitHub). Just need add plugins el9 copr config and comps by evgeni · Pull Request #10200 · theforeman/foreman-packaging · GitHub. Then el9 should show up on Index of /foreman/nightly & Index of /plugins/nightly. Support EL9 by ekohl · Pull Request #823 · theforeman/puppet-foreman_proxy · GitHub was opened to test this, but we quickly found that the repository layout of stagingyum and yum differs.


To support testing Katello (which relies on postgresql-evr) the PostgreSQL container image has been made a parameter (allow configuring postgres container image · theforeman/actions@b663142 · GitHub & also allow custom DB container for DB job · theforeman/actions@e4e8fb7 · GitHub). uses this with @evgeni’s fork. also release container to GH registry · Katello/postgresql-evr@b79100d · GitHub has been merged, but it hasn’t ran yet.

Install NPM packages & compile webpack by ekohl · Pull Request #18 · theforeman/actions · GitHub has been updated and needs a review. @ekohl will take a look.

Add Ruby 3.0 + Puppet 7 & 3.2 + Puppet 8 to CI by archanaserver · Pull Request #369 · theforeman/kafo · GitHub was merged.

New discussion items for next time:

  • How to install test dependencies? Today Jenkins for regular plugins uses gemfile.d/*.rb and our reusable actions do the same, but Katello has a special job in Jenkins that installs it as a gemspec, which installs all development dependencies. This came up in
  • Today Katello’s tests run on every Foreman PR using Jenkins. Should this be ported to GitHub Actions?

Updating RuboCop is in progress. PRs to various repositories are being opened to prepare for an update:


Another week, another update.

Major milestones are around packaging. @evgeni has started to push out Smart Proxy & plugin packages. The Puppet module for foreman_proxy already passes its test suite (Support EL9 by ekohl · Pull Request #823 · theforeman/puppet-foreman_proxy · GitHub) so we can at least say that various packages install and the service starts. Work continues by doing the same for content proxies (Support EL9 by evgeni · Pull Request #469 · theforeman/puppet-foreman_proxy_content · GitHub). That’s green, so once the Katello packages have been published (add el9 to katello repository generation/publishing by evgeni · Pull Request #403 · theforeman/jenkins-jobs · GitHub) it should be good to go. Those test suites are limited in testing the actual functionality, but are a solid foundation to build on. allow overriding the server/proxy OS by evgeni · Pull Request #1752 · theforeman/forklift · GitHub allows spinning up an EL8 Foreman server with an EL9 content proxy. There @evgeni has found some functionality that doesn’t work (like SELinux issues), but things are coming together.

In doing this work it was a problem because foreman-debug and katello-debug need to Foreman to be built. Instead we’ve made those optional (Drop requirement on foreman-debug by evgeni · Pull Request #10187 · theforeman/foreman-packaging · GitHub Don't require katello-debug by evgeni · Pull Request #10257 · theforeman/foreman-packaging · GitHub do not install katello-debug by default by evgeni · Pull Request #468 · theforeman/puppet-foreman_proxy_content · GitHub). The functionality should be in sosreport so this also allows slightly smaller installations for end users.

On the Foreman side @ofedoren has started to properly analyze the issue we had. Run GHA on Ruby 3.0 (alternative 2.0) by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub uses the same constructor method signature that Rails’ activerecord has so it’s promising for a proper fix.

Other than that Install NPM packages & compile webpack by ekohl · Pull Request #18 · theforeman/actions · GitHub was merged so now Foreman plugins should get good test coverage when they use the reusable action. That is now a significant milestone in Convert Foreman & plugins continuous integration to GitHub Actions - #4 by ekohl. @akumari has started to expand our investigation in Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub, but even that is still incomplete. @ekohl will put out a call to plugin maintainers to collaborate on this.

One remaining issue that shows up in most plugins is running of tests from Foreman core in a plugin (always execute access_permissions_test from Foreman core by evgeni · Pull Request #16 · theforeman/actions · GitHub). This isn’t an issue in Jenkins, because that runs the entire Foreman test suite (to prevent regressions). GitHub actions only runs the plugin’s but some do run some tests from core. This needs a resolution for plugins using GitHub Actions today.

Another that came up is installation of chromedriver (Rewrite GHA workflow using reusable actions by archanaserver · Pull Request #381 · theforeman/foreman_puppet · GitHub). It’s unknown how Foreman itself does this today (it doesn’t install Chrome, so does it even run at all?). There’s also a PR open for a remote webdriver (Fixes #36978 - Add possibility to use remote webdriver by dosas · Pull Request #9952 · theforeman/foreman · GitHub) which may allow us to use a webdriver container and connect to that instead.

To know how far we are in the migration it would be good to add CI descriptions to Foreman landscape and @evgeni will take a look.

It also came up that we may want a reusable action for Smart Proxy plugins (add proxy plugin workflow by evgeni · Pull Request #27 · theforeman/actions · GitHub) but that’s considered low priority now.

Lastly, it’s noted that we will need to deal with the webpack update ( if we want to run on EL9. @ekohl will reach out to @MariaAga how we can get that merged as soon as possible.

1 Like

Should we undertake deprecation of these in the upcoming release?

Is this being asked a technical question or philosophical one?

The Debian situation is complicated. For example, there are fixes like [foreman] Correct Apache package on Debian · sosreport/sos@9a6279f · GitHub which are only in 4.3+ and only 4.0 is included in Debian 11 (Debian -- Details of package sosreport in bullseye) and even Debian 12 (Debian -- Details of package sosreport in bookworm).

I don’t know how much foreman-debug is used today but I’d advocate for a separate discussion if we really want to deprecate it.

Yes. I think it’s a case of “both” even. First, what do we want and then how do we implement it.

Last weeks’ update, with a bit of delay.

Packaging wise all EL9 RPM workflows are present: Foreman, Plugins & Katello. The list of packages is incomplete, but a content proxy now passes tests. Remaining is Foreman & its plugins. That’s blocked on Ruby 3 & Webpack 5.

Foreman & Ruby 3.0 is being verified using @ofedoren is going over the various plugins to see it breaks anything.

For Webpack 5 the current proposal is to pull the trigger this Friday (2024-01-26). A separate announcement for that will be posted.

Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub has been extended with all the plugins from Foreman landscape that matter. To make this easier, the landscape has also been updated with the testing overview (Jenkins, GitHub Actions, Puppet). This will be even more important because after we merge Webpack 5 we will start to depend on NodeJS 14 and many current GitHub Action implementations still use NodeJS 12. So implicitly there’s also a deadline for this.

In some repositories (at least foreman_openscap) we’ve seen that RuboCop didn’t ran before. For those cases it’s OK to just convert the existing tests, but long term that should be figured out. It’s confusing when there is a RuboCop configuration, but it doesn’t run.

On the part of running tests in core there’s no input from plugin maintainers. Run permissions tests from Foreman core by ekohl · Pull Request #51 · theforeman/foreman_plugin_template · GitHub has some ideas.

@evgeni has also done some initial work on Debian 12 & Ubuntu 22.04. Debian 12 is Ruby 3.1, but that requires Rails 7 (including Zeitwerk). Ubuntu 22.04 has switched compression, which doesn’t work with our own mirrors right now. This for sure won’t happen (not even experimental) for Foreman 3.10.

There was some discussion about Rails 7. Ruby on Rails | and Maintenance Policy for Ruby on Rails — Ruby on Rails Guides don’t mention an EOL date for Rails 6.1, but it’s good to work on this nonetheless. This will be evaluated for Foreman 3.11.


Today’s meeting was rather short.

The big thing is that we are indeed going to merge webpack 5 tomorrow (Webpack 5 merge this Friday (2024-01-26)). To facilitate this the continued focus has been on the GitHub Actions conversions for plugins. By now for almost all the plugins there’s a PR and various PRs have been merged. While doing this, it was noticed that foreman_setup would need work, but it’s rather unmaintained so I’ve proposed to drop it: Retire foreman_setup plugin

Note mentioned during the meeting, but Fixes #37092 - Use minitest_reporters_github in GHA by ekohl · Pull Request #10010 · theforeman/foreman · GitHub makes reading errors in GitHub Actions easier.

As for Ruby 3, Fixes #37087 - STI preparations for Ruby 3.0 by ofedoren · Pull Request #10006 · theforeman/foreman · GitHub has been merged. Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub now contains the update to matrix.json and one where mocha will become stricter. @ofedoren is verifying this on the various plugin repositories.

Like with webpack we want to set a date where it’s merged. This date will be Tuesday (2024-01-30). That way we don’t (plan to) overlap with the webpack while also providing sufficient time to fix things before branching and before FOSDEM and cfgmgmtcamp.

Something I now notice about last week’s update: we are talking about delivering EL9 support in Foreman 3.10 as experimental. Mostly because this means we can deliver it without any EL8 → EL9 migration support (and postpone that to Foreman 3.11). Of course users can do this on their own, but they’re properly warned.


GitHub Actions

Related: I’ve posted the GitHub Actions in the relevant RFC as a comment Convert Foreman & plugins continuous integration to GitHub Actions - #6 by ekohl

@evgeni raised that various plugins have JS tests and those don’t respect the matrix.json file. Reusable Javascript tests · Issue #42 · theforeman/actions · GitHub has been filed to implement a reusable action for this.


In the past week the focus has been less on Ruby and more on NodeJS. As announced we merged webpack 5 and as expected, there was some fall out. Nothing major, but it occupied us nonetheless. For all the details, see Webpack 5 merge this Friday (2024-01-26).

One uncomfortable thing that showed up NodeJS 16. This is needed for EL9. It looks like we still need legacy peer dependencies handling with NPM. Another is that on EL9 the development packages for NodeJS 16 are missing. Because of that the current investigation is looking at NodeJS 18. That version also ships NPM 8 and we expect few problems. Fedora 38 ships it, which means you can finally just use the system packages in your development environment. Fedora 39 ships NodeJS 20, so that support is also considered. Active work is in Fixes #37134 - upgrade to node 18 and npm 8 by MariaAga · Pull Request #10016 · theforeman/foreman · GitHub.

If we look at the Foreman 3.10 schedule then it’s getting rather close. Next week we have CfgMgmtCamp 2024 where both @evgeni and I will attend. The week after that it’s already stabilization week.

I see 3 options:

  • @ehelms has time to work on the NodeJS > 14 upgrade next week and can get it done
  • We postpone Foreman 3.10 branching for this
  • We postpone Foreman on EL9 to 3.11

@evgeni notes that it’s very useful to have a non-official Foreman on EL9 release (as 3.10 was intended to be) because that makes upgrade testing possible.

@ekohl will discuss with @ehelms and the @releases team what’s the best course of action here.

Ruby 3

We set a target to merge Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub on Tuesday. Due to webpack 5 that didn’t happen, but @ofedoren has continued testing (Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub)

Foreman tasks / Dynflow need work:

Katello needs this and other changes. virt_who_configure depends on Katello and may be fixed once Katello works.

foreman_ansible also has some error that I don’t understand.

We agree to merge it today, even if that breaks some test suites. Maintainers can ignore the Ruby 3.0 result for now.


Good news! Thanks to @MariaAga in Node 18 PR I had the thought we should be able to use the same --legacy-peer-deps in our NodeJS packages that are bundled and achieve the same result. That is, a package that works on 14 and 18, easing the pain of transition by allowing us to rebuild on 14 and then the packages continue to work on 18.

I undertook and tested the following:

  • Building all packages on NodeJS 14 with the --legacy-peer-deps flag
  • Building all packages on NodeJS 18 with the --legacy-peer-deps flag
  • Building Foreman against the NodeJS 18 packages
  • Testing Foreman on a nightly install (Builds for ehelms/foreman-nightly-staging)

Given all of my testing yielded successful results, I have opened a PR to add --legacy-peer-deps to those bundled RPM packages: Use --legacy-peer-deps in bundled NPM packages by ehelms · Pull Request #10389 · theforeman/foreman-packaging · GitHub

I have opened a PR to npm2rpm to add support so that not only will the spec files get generated by our update script properly, but also so that the caches are generated with legacy peer deps.


The meeting yesterday didn’t happen, as several people were either still traveling or recovering from travel/conferences.

Nevertheless, progress happened, so here is a few highlights from me:


Yesterday we met again and went over the updates since last week.

We now have built almost everything. At the time of the meeting foreman_discovery was still blocked on Move to inline settings validations by ekohl · Pull Request #622 · theforeman/foreman_discovery · GitHub but that has since been merged and released (including another fix: Put back webpacked_plugins_js_for, this is required for webpack by evgeni · Pull Request #627 · theforeman/foreman_discovery · GitHub). Ensure glibc-langpack-en is always installed by evgeni · Pull Request #329 · theforeman/puppet-pulpcore · GitHub was needed on more minimal content proxies.

With that our pipelines are passing. That doesn’t mean everything is bug free, but a significant milestone.

The last big issue is Fixes #37134 - upgrade to node 18 and npm 8 by MariaAga · Pull Request #10016 · theforeman/foreman · GitHub.

Some things aren’t covered in our pipelines. There is still Use specific `Host` class #26 by oneiros · Pull Request #27 · betadots/foreman_hdm · GitHub and Fixes #37065 - Call report_row with kwargs in Host Statuses report by jeremylenz · Pull Request #10047 · theforeman/foreman · GitHub (which may also affect users) but overall we should we ready for RC1 next week.


Foreman 3.10 has (mostly) branched and 3.10.0 RC1 is being released. See Release Team Meeting 2024-02-21 - #2 by ekohl as well. There are 2 remaining issues:

They aren’t blockers for RC1, but we should aim to get both resolved for RC2.

To be explicit: the Foreman 3.10 goal is experimental support. We won’t provide a migration path from EL8. We expect it to largely work, but it’s a large upgrade and we may have missed things. Foreman 3.11 aims to complete the work. This means:

Then within the project there are 2 types of issues remaining: GitHub Actions migration and RuboCop changes. For the former Support setting arbitrary environment variables in foreman_plugin · Issue #44 · theforeman/actions · GitHub is an issue that popped up.

For both I’m tempted to split them off to a separate project and rename the existing one to EL9 support, just to keep things clear.

Other follow ups for 3.11 include:

  • Start building for Ubuntu 22.04, which includes Ruby 3.0 so it should just work
  • Start the Zeitwerk migration to get us on a path to Rails 7, opening Ruby 3.1+

Ruby 3.1+ is important, because Debian 12 and Ubuntu 24.04 both ship Ruby 3.1. Right now CentOS Stream 10 is also being branched. Depending on which Fedora version they base it on, I’d expect Ruby 3.2 or 3.3 (with the latter being more likely).

As a last note we’ve started to discuss EL8 server support deprecation. We want to give users a smooth, well tested path so we’re currently looking at 3.13 to drop EL8. This will be communicated in the 3.10+ release notes.