Path to Ruby 3.0, 3.1, EL9 and Ubuntu 22.04

As every year, Ruby has released a new version over Christmas. Ruby 3.3 appears to be small in terms of breaking changes. While reading up on it, I noticed that Ruby 3.0 is EOL in March 2024, so in 3 months. I’d still go forward with releasing it on EL9 on 3.0, even if it is EOL because Ruby 2.7 is already EOL since March 2023 anyway and that hasn’t held us back.


Today we synced up again after we’ve all been out for the holidays. Some updates since last time.


Foreman has been updated to use safemode 1.4.0, including tests that verify the new safe navigator operator in templates (Fixes #37010 - Support safe navigation operator in safemode by ekohl · Pull Request #9967 · theforeman/foreman · GitHub). Some safe changes for Ruby 3 compatibility have been merged (Refs #36849 - Ruby 3 preparations by ekohl · Pull Request #9975 · theforeman/foreman · GitHub).

This leaves the Ruby 3 enablement with a more controversial change. Both Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9871 · theforeman/foreman · GitHub & Fixes #36849 - Run GHA on Ruby 3.0 (alternative) by ekohl · Pull Request #9923 · theforeman/foreman · GitHub try to deal with the method_missing change. @ofedoren will look further into this. @ekohl has submitted an update to clarify the Ruby docs as well (Improve method_missing method documentation by ekohl · Pull Request #9412 · ruby/ruby · GitHub).

@ofedoren will start looking at testing plugins. test with ruby3-compatible foreman by evgeni · Pull Request #24 · RedHatSatellite/foreman_theme_satellite · GitHub is the way for plugins that use our reusable GitHub Action.


On the packaging side some progress has been made:

With all of these changes we’re close to publishing el9 nightly repositories (enable nightly el9 repo generation, closure and publishing by evgeni · Pull Request #395 · theforeman/jenkins-jobs · GitHub). Just need add plugins el9 copr config and comps by evgeni · Pull Request #10200 · theforeman/foreman-packaging · GitHub. Then el9 should show up on Index of /foreman/nightly & Index of /plugins/nightly. Support EL9 by ekohl · Pull Request #823 · theforeman/puppet-foreman_proxy · GitHub was opened to test this, but we quickly found that the repository layout of stagingyum and yum differs.


To support testing Katello (which relies on postgresql-evr) the PostgreSQL container image has been made a parameter (allow configuring postgres container image · theforeman/actions@b663142 · GitHub & also allow custom DB container for DB job · theforeman/actions@e4e8fb7 · GitHub). uses this with @evgeni’s fork. also release container to GH registry · Katello/postgresql-evr@b79100d · GitHub has been merged, but it hasn’t ran yet.

Install NPM packages & compile webpack by ekohl · Pull Request #18 · theforeman/actions · GitHub has been updated and needs a review. @ekohl will take a look.

Add Ruby 3.0 + Puppet 7 & 3.2 + Puppet 8 to CI by archanaserver · Pull Request #369 · theforeman/kafo · GitHub was merged.

New discussion items for next time:

  • How to install test dependencies? Today Jenkins for regular plugins uses gemfile.d/*.rb and our reusable actions do the same, but Katello has a special job in Jenkins that installs it as a gemspec, which installs all development dependencies. This came up in
  • Today Katello’s tests run on every Foreman PR using Jenkins. Should this be ported to GitHub Actions?

Updating RuboCop is in progress. PRs to various repositories are being opened to prepare for an update:


Another week, another update.

Major milestones are around packaging. @evgeni has started to push out Smart Proxy & plugin packages. The Puppet module for foreman_proxy already passes its test suite (Support EL9 by ekohl · Pull Request #823 · theforeman/puppet-foreman_proxy · GitHub) so we can at least say that various packages install and the service starts. Work continues by doing the same for content proxies (Support EL9 by evgeni · Pull Request #469 · theforeman/puppet-foreman_proxy_content · GitHub). That’s green, so once the Katello packages have been published (add el9 to katello repository generation/publishing by evgeni · Pull Request #403 · theforeman/jenkins-jobs · GitHub) it should be good to go. Those test suites are limited in testing the actual functionality, but are a solid foundation to build on. allow overriding the server/proxy OS by evgeni · Pull Request #1752 · theforeman/forklift · GitHub allows spinning up an EL8 Foreman server with an EL9 content proxy. There @evgeni has found some functionality that doesn’t work (like SELinux issues), but things are coming together.

In doing this work it was a problem because foreman-debug and katello-debug need to Foreman to be built. Instead we’ve made those optional (Drop requirement on foreman-debug by evgeni · Pull Request #10187 · theforeman/foreman-packaging · GitHub Don't require katello-debug by evgeni · Pull Request #10257 · theforeman/foreman-packaging · GitHub do not install katello-debug by default by evgeni · Pull Request #468 · theforeman/puppet-foreman_proxy_content · GitHub). The functionality should be in sosreport so this also allows slightly smaller installations for end users.

On the Foreman side @ofedoren has started to properly analyze the issue we had. Run GHA on Ruby 3.0 (alternative 2.0) by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub uses the same constructor method signature that Rails’ activerecord has so it’s promising for a proper fix.

Other than that Install NPM packages & compile webpack by ekohl · Pull Request #18 · theforeman/actions · GitHub was merged so now Foreman plugins should get good test coverage when they use the reusable action. That is now a significant milestone in Convert Foreman & plugins continuous integration to GitHub Actions - #4 by ekohl. @akumari has started to expand our investigation in Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub, but even that is still incomplete. @ekohl will put out a call to plugin maintainers to collaborate on this.

One remaining issue that shows up in most plugins is running of tests from Foreman core in a plugin (always execute access_permissions_test from Foreman core by evgeni · Pull Request #16 · theforeman/actions · GitHub). This isn’t an issue in Jenkins, because that runs the entire Foreman test suite (to prevent regressions). GitHub actions only runs the plugin’s but some do run some tests from core. This needs a resolution for plugins using GitHub Actions today.

Another that came up is installation of chromedriver (Rewrite GHA workflow using reusable actions by archanaserver · Pull Request #381 · theforeman/foreman_puppet · GitHub). It’s unknown how Foreman itself does this today (it doesn’t install Chrome, so does it even run at all?). There’s also a PR open for a remote webdriver (Fixes #36978 - Add possibility to use remote webdriver by dosas · Pull Request #9952 · theforeman/foreman · GitHub) which may allow us to use a webdriver container and connect to that instead.

To know how far we are in the migration it would be good to add CI descriptions to Foreman landscape and @evgeni will take a look.

It also came up that we may want a reusable action for Smart Proxy plugins (add proxy plugin workflow by evgeni · Pull Request #27 · theforeman/actions · GitHub) but that’s considered low priority now.

Lastly, it’s noted that we will need to deal with the webpack update ( if we want to run on EL9. @ekohl will reach out to @MariaAga how we can get that merged as soon as possible.

1 Like

Should we undertake deprecation of these in the upcoming release?

Is this being asked a technical question or philosophical one?

The Debian situation is complicated. For example, there are fixes like [foreman] Correct Apache package on Debian · sosreport/sos@9a6279f · GitHub which are only in 4.3+ and only 4.0 is included in Debian 11 (Debian -- Details of package sosreport in bullseye) and even Debian 12 (Debian -- Details of package sosreport in bookworm).

I don’t know how much foreman-debug is used today but I’d advocate for a separate discussion if we really want to deprecate it.

Yes. I think it’s a case of “both” even. First, what do we want and then how do we implement it.

Last weeks’ update, with a bit of delay.

Packaging wise all EL9 RPM workflows are present: Foreman, Plugins & Katello. The list of packages is incomplete, but a content proxy now passes tests. Remaining is Foreman & its plugins. That’s blocked on Ruby 3 & Webpack 5.

Foreman & Ruby 3.0 is being verified using @ofedoren is going over the various plugins to see it breaks anything.

For Webpack 5 the current proposal is to pull the trigger this Friday (2024-01-26). A separate announcement for that will be posted.

Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub has been extended with all the plugins from Foreman landscape that matter. To make this easier, the landscape has also been updated with the testing overview (Jenkins, GitHub Actions, Puppet). This will be even more important because after we merge Webpack 5 we will start to depend on NodeJS 14 and many current GitHub Action implementations still use NodeJS 12. So implicitly there’s also a deadline for this.

In some repositories (at least foreman_openscap) we’ve seen that RuboCop didn’t ran before. For those cases it’s OK to just convert the existing tests, but long term that should be figured out. It’s confusing when there is a RuboCop configuration, but it doesn’t run.

On the part of running tests in core there’s no input from plugin maintainers. Run permissions tests from Foreman core by ekohl · Pull Request #51 · theforeman/foreman_plugin_template · GitHub has some ideas.

@evgeni has also done some initial work on Debian 12 & Ubuntu 22.04. Debian 12 is Ruby 3.1, but that requires Rails 7 (including Zeitwerk). Ubuntu 22.04 has switched compression, which doesn’t work with our own mirrors right now. This for sure won’t happen (not even experimental) for Foreman 3.10.

There was some discussion about Rails 7. Ruby on Rails | and Maintenance Policy for Ruby on Rails — Ruby on Rails Guides don’t mention an EOL date for Rails 6.1, but it’s good to work on this nonetheless. This will be evaluated for Foreman 3.11.


Today’s meeting was rather short.

The big thing is that we are indeed going to merge webpack 5 tomorrow (Webpack 5 merge this Friday (2024-01-26)). To facilitate this the continued focus has been on the GitHub Actions conversions for plugins. By now for almost all the plugins there’s a PR and various PRs have been merged. While doing this, it was noticed that foreman_setup would need work, but it’s rather unmaintained so I’ve proposed to drop it: Retire foreman_setup plugin

Note mentioned during the meeting, but Fixes #37092 - Use minitest_reporters_github in GHA by ekohl · Pull Request #10010 · theforeman/foreman · GitHub makes reading errors in GitHub Actions easier.

As for Ruby 3, Fixes #37087 - STI preparations for Ruby 3.0 by ofedoren · Pull Request #10006 · theforeman/foreman · GitHub has been merged. Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub now contains the update to matrix.json and one where mocha will become stricter. @ofedoren is verifying this on the various plugin repositories.

Like with webpack we want to set a date where it’s merged. This date will be Tuesday (2024-01-30). That way we don’t (plan to) overlap with the webpack while also providing sufficient time to fix things before branching and before FOSDEM and cfgmgmtcamp.

Something I now notice about last week’s update: we are talking about delivering EL9 support in Foreman 3.10 as experimental. Mostly because this means we can deliver it without any EL8 → EL9 migration support (and postpone that to Foreman 3.11). Of course users can do this on their own, but they’re properly warned.


GitHub Actions

Related: I’ve posted the GitHub Actions in the relevant RFC as a comment Convert Foreman & plugins continuous integration to GitHub Actions - #6 by ekohl

@evgeni raised that various plugins have JS tests and those don’t respect the matrix.json file. Reusable Javascript tests · Issue #42 · theforeman/actions · GitHub has been filed to implement a reusable action for this.


In the past week the focus has been less on Ruby and more on NodeJS. As announced we merged webpack 5 and as expected, there was some fall out. Nothing major, but it occupied us nonetheless. For all the details, see Webpack 5 merge this Friday (2024-01-26).

One uncomfortable thing that showed up NodeJS 16. This is needed for EL9. It looks like we still need legacy peer dependencies handling with NPM. Another is that on EL9 the development packages for NodeJS 16 are missing. Because of that the current investigation is looking at NodeJS 18. That version also ships NPM 8 and we expect few problems. Fedora 38 ships it, which means you can finally just use the system packages in your development environment. Fedora 39 ships NodeJS 20, so that support is also considered. Active work is in Fixes #37134 - upgrade to node 18 and npm 8 by MariaAga · Pull Request #10016 · theforeman/foreman · GitHub.

If we look at the Foreman 3.10 schedule then it’s getting rather close. Next week we have CfgMgmtCamp 2024 where both @evgeni and I will attend. The week after that it’s already stabilization week.

I see 3 options:

  • @ehelms has time to work on the NodeJS > 14 upgrade next week and can get it done
  • We postpone Foreman 3.10 branching for this
  • We postpone Foreman on EL9 to 3.11

@evgeni notes that it’s very useful to have a non-official Foreman on EL9 release (as 3.10 was intended to be) because that makes upgrade testing possible.

@ekohl will discuss with @ehelms and the @releases team what’s the best course of action here.

Ruby 3

We set a target to merge Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub on Tuesday. Due to webpack 5 that didn’t happen, but @ofedoren has continued testing (Fixes #36849 - Run GHA on Ruby 3.0 by ofedoren · Pull Request #9989 · theforeman/foreman · GitHub)

Foreman tasks / Dynflow need work:

Katello needs this and other changes. virt_who_configure depends on Katello and may be fixed once Katello works.

foreman_ansible also has some error that I don’t understand.

We agree to merge it today, even if that breaks some test suites. Maintainers can ignore the Ruby 3.0 result for now.


Good news! Thanks to @MariaAga in Node 18 PR I had the thought we should be able to use the same --legacy-peer-deps in our NodeJS packages that are bundled and achieve the same result. That is, a package that works on 14 and 18, easing the pain of transition by allowing us to rebuild on 14 and then the packages continue to work on 18.

I undertook and tested the following:

  • Building all packages on NodeJS 14 with the --legacy-peer-deps flag
  • Building all packages on NodeJS 18 with the --legacy-peer-deps flag
  • Building Foreman against the NodeJS 18 packages
  • Testing Foreman on a nightly install (Builds for ehelms/foreman-nightly-staging)

Given all of my testing yielded successful results, I have opened a PR to add --legacy-peer-deps to those bundled RPM packages: Use --legacy-peer-deps in bundled NPM packages by ehelms · Pull Request #10389 · theforeman/foreman-packaging · GitHub

I have opened a PR to npm2rpm to add support so that not only will the spec files get generated by our update script properly, but also so that the caches are generated with legacy peer deps.


The meeting yesterday didn’t happen, as several people were either still traveling or recovering from travel/conferences.

Nevertheless, progress happened, so here is a few highlights from me:


Yesterday we met again and went over the updates since last week.

We now have built almost everything. At the time of the meeting foreman_discovery was still blocked on Move to inline settings validations by ekohl · Pull Request #622 · theforeman/foreman_discovery · GitHub but that has since been merged and released (including another fix: Put back webpacked_plugins_js_for, this is required for webpack by evgeni · Pull Request #627 · theforeman/foreman_discovery · GitHub). Ensure glibc-langpack-en is always installed by evgeni · Pull Request #329 · theforeman/puppet-pulpcore · GitHub was needed on more minimal content proxies.

With that our pipelines are passing. That doesn’t mean everything is bug free, but a significant milestone.

The last big issue is Fixes #37134 - upgrade to node 18 and npm 8 by MariaAga · Pull Request #10016 · theforeman/foreman · GitHub.

Some things aren’t covered in our pipelines. There is still Use specific `Host` class #26 by oneiros · Pull Request #27 · betadots/foreman_hdm · GitHub and Fixes #37065 - Call report_row with kwargs in Host Statuses report by jeremylenz · Pull Request #10047 · theforeman/foreman · GitHub (which may also affect users) but overall we should we ready for RC1 next week.


Foreman 3.10 has (mostly) branched and 3.10.0 RC1 is being released. See Release Team Meeting 2024-02-21 - #2 by ekohl as well. There are 2 remaining issues:

They aren’t blockers for RC1, but we should aim to get both resolved for RC2.

To be explicit: the Foreman 3.10 goal is experimental support. We won’t provide a migration path from EL8. We expect it to largely work, but it’s a large upgrade and we may have missed things. Foreman 3.11 aims to complete the work. This means:

Then within the project there are 2 types of issues remaining: GitHub Actions migration and RuboCop changes. For the former Support setting arbitrary environment variables in foreman_plugin · Issue #44 · theforeman/actions · GitHub is an issue that popped up.

For both I’m tempted to split them off to a separate project and rename the existing one to EL9 support, just to keep things clear.

Other follow ups for 3.11 include:

  • Start building for Ubuntu 22.04, which includes Ruby 3.0 so it should just work
  • Start the Zeitwerk migration to get us on a path to Rails 7, opening Ruby 3.1+

Ruby 3.1+ is important, because Debian 12 and Ubuntu 24.04 both ship Ruby 3.1. Right now CentOS Stream 10 is also being branched. Depending on which Fedora version they base it on, I’d expect Ruby 3.2 or 3.3 (with the latter being more likely).

As a last note we’ve started to discuss EL8 server support deprecation. We want to give users a smooth, well tested path so we’re currently looking at 3.13 to drop EL8. This will be communicated in the 3.10+ release notes.


Last week’s meeting was cancelled due to Red Hat’s day of learning, encouraging employees to spend their day learning something new.

The first big update is that the board was split into three boards:

Then there’s a new board for Ubuntu 22.04 support: Ubuntu 22.04 support · GitHub. Because of all the efforts for EL9 this was a rather small effort and everything is on track for Foreman 3.11 to support it.

Since last time the EL9 support on the old manual has been added.

This leaves just a few issues on the board:


There are a few topics we discussed

Ruby 3.0 compatibility

Fixes #37065 - report_row now explicitly accepts args and kwargs by jeremylenz · Pull Request #10047 · theforeman/foreman · GitHub was reviewed and merged during the meeting. That means now there are no known Ruby 3.0 incompatibilities.


The EL9 work is progressing nicely.

Not on EL 9 itself, but on EL 8. The upgrade from PostgreSQL 12 to 13 is implemented. Documentation of this is still in progress. This matches the version to EL 9 and was a blocker for upgrades using leapp.

Leapp changes itself are being reviewed by the leapp team: Foreman/Satellite el8toel9 upgrade by evgeni · Pull Request #1181 · oamg/leapp-repository · GitHub. Another blocking issue [WIP] Rebase libsolv and add epoch by pcreech · Pull Request #863 · theforeman/pulpcore-packaging · GitHub is also making progress.

Ubuntu 22.04

This is done and Foreman 3.11 will release with it. We haven’t decided on a deprecation path, but we’ll encourage users to upgrade. The manual has been updated, but new docs haven’t yet because the changes would conflict with the in progress EL 9 documentation so it’s waiting on that.

Debian 12

Debian 12 uses Ruby 3.1. That should largely work. Rails 6.1 should be compatible with Ruby 3.1, unlike what I first thought. That was because Rails 7.0 had some issues, but that was in changes introduced since 6.1. Safemode is only marked compatible with Ruby 3.0. Once that’s updated, we can start building Foreman on it.


Tiny update from last Thursday’s meeting I forgot to post. There’s not a lot to mention:


Will the documentation/support/scripting to upgrade el8 from PostgreSQL 12 to 13 be available before 3.11, or is it targeted to come out at the same time as 3.11?

I’ve been working on getting my Foreman/Katello install working on el9 with a fresh build, and then restoring from a backup taken on my old el8 instance. However, restore is not currently happy that the DB I’m restoring (v12) is not the current version (v13) :joy:

It will be in Foreman 3.11. Due to modularity on EL8 this is something we have to do with some user interaction so we can’t just add it to 3.10.

1 Like

If you’re curious, see the PostgreSQL 13 tracker for all the changes we had to do.

1 Like

Makes sense, thanks!

I lost last week’s notes so I didn’t post them, but there wasn’t much we talked about and we rehashed it in this week. Here’s a summary where we now are:

  • Foreman 3.10 has delivered initial EL9 support (using Ruby 3.0). We’ll continue to flesh this out in 3.11. There are known bugs, but those will be picked up as regular bug reports.
  • Foreman 3.11 will deliver Ubuntu 22.04 support (using Ruby 3.0)
  • Debian 12 packages are mostly ready (using Ruby 3.1), but blocked on a missing puppetserver package. Once that’s available, our installer will work and we can deliver this. We need Puppet to provide those.
  • Foreman 3.11 on EL8 will update from PostgreSQL 12 to 13 to match EL9, which prepares for a migration using Leapp (which is still in progress).

With all of this, we’ve agreed to consider this RFC resolved.

There will be follow up projects.

Initially we considered updating Rails in scope, but that turned out not to be a blocker. Obviously we still want to continue using Rails versions with security support and it’s only a matter of time before Rails 6.1 goes EOL. That will be its own RFC.

Looking beyond the horizon we can also see EL10 which ships Ruby 3.3 and drops ISC DHCP, meaning we need to migrate to ISC Kea.

And possibly more.