Path to Ruby 3.0, 3.1, EL9 and Ubuntu 22.04

Development RFCs
1

Today Foreman (3.5-develop) is compatible with Ruby 2.7 (older was dropped with Foreman 3.4). It is not yet compatible with Ruby 3, but both EL9 and Ubuntu 22.04 both require that compatibility. This is a rough roadmap with the major milestones to run Foreman and Foreman Proxy on the latest EL and Ubuntu versions. It is likely incomplete and we’ll figure out more things along the way.

How to bootstrap a new platform

This is a small reminder of the general steps to get Foreman running on a new platform.

First of all, there are two major projects that can each independently move. These are Foreman and Foreman Proxy. For both, the steps are largely the same.

  • Get the core project’s test suite runing on the Ruby version
  • Get all plugin’s test suites running on the Ruby version
  • Package them

Once they’re packaged, the installer side can be updated. Additionally, on EL there is an SELinux policy.

In reality it’s not so linear. For example, if it’s a previously unpackaged platform, packaging can start even if plugins aren’t ready. Additionally, some installer modules work with core platform components such as dhcp, dns and tftp.

It should also be noted that a new platform doesn’t mean old platforms are dropped. There is always an overlap which allows users to update.

How to apply this into practice now

Ruby 3 is a big change, but many small steps can be taken. I’ll go over them for each project

Foreman

Today Foreman uses Ruby 2.7 and Rails 6.1. Long term we should aim for for Ruby 3.1, but Rails 7.0 is the first version to support it. Previously I played around with this and created Rails 7 & Ruby 3.1 by ekohl · Pull Request #9328 · theforeman/foreman · GitHub. A lot of smaller tasks have already been split off and merged, but it’s a large task that means we should break this up into smaller pieces. Many of these pieces can move in parallel.

A critical set of plugins should be selected. These plugins should all be made compatible with new versions before changing defaults.

Ruby 3.0

A major step is updating to Ruby 3.0. In Ruby 3.0 some gems changed. The Ruby 3.0.0 release notes mention both rss and rexml were changed from default to bundled gems. This means they must now be specified as a dependency. Additionally some stdlib modules are now gems.

More significant is kwargs. In Ruby 2 the last argument could be a hash and keywords were stored there. In Ruby 2.7 it became possible to have a separate keyword argument parameter. Ruby 3 removes the old behavior. See Separation of positional and keyword arguments in Ruby 3.0 for more information. This is tracked in #35300. Because of this a lot of gems have dropped Ruby older than 2.7, because it’s very hard to support more. Because of this Foreman 3.4 has set the minimum Ruby version to 2.7. Because of this there are gems which even today raise deprecation messages, such as safemode and graphql. Updating gems is a large part of this task.

This is an incomplete list, but some concrete work that must happen:

There are also many more dependencies that could be updated. All of them should be checked.

Once it passes, CI should be modified to also run on Ruby 3.0.

At this point all plugin authors should be notified to perform the same steps and packaging can commence.

Zeitwerk

First of all, there is still some remaining work on Rails 6.1. Updating to Rails 6.1 defaults doesn’t mean we need to have all Rails 6.1 defaults, but at least call load_defaults 6.1. Not all values must match the defaults, but the test must pass. It’s also fine to do this incrementally (first 5.0, then 5.1, etc). Zeitwerk is considered out of scope for this part.

The next major step is Zeitwerk. The new default loader in Rails 6.1 and only loader in Rails 7. This is a large effort. The first goal should be to use the classic loader, but support using Zeitwerk. Once Foreman itself is compatible, we should give plugin developers some time to update before we switch the default.

Rails 7.0

Once Ruby 3.0 and Zeitwerk are in place, Rails 7 can be the next focus. The Rails 7.0 release notes and upgrading guide should be helpful.

The previous mechanism to support multiple Rails versions stopped working, so it was removed. A new way should be found so Foreman can default to Rails 6.1 but run with Rails 7.0 too. This helps plugin authors migrate.

This also involves packaging work.

Ruby 3.1

This depends on both Ruby 3.0 and Rails. It is largely the same as Ruby 3.0. The Ruby 3.1.0 release notes mentions more gem changes. These are likely all rather trivial. It is also not required since both EL9 and Ubuntu 22.04 have Ruby 3.0, though EL9 also has a Ruby 3.1 module. Given this is likely a small task, it’d be good to support.

Once it passes, CI should be modified to also run on Ruby 3.1.

Again, plugins need to be verified too.

Foreman Proxy

Foreman Proxy is a lot smaller than Foreman, both in the size of the code base and it has fewer dependencies. There are no frameworks to update, so it’s limited to the Ruby updates. Today the test suite already passes, but the dependencies do need to be fixed.

Similar to Foreman, there are also plugins to consider. These are usually small, but still need to be verified.

Packaging can start fairly soon.

Packaging

The initial goal is to only create new platforms, not switch over existing platforms. On Debian/Ubuntu there’s only a single Ruby interpreter so that’s certainly out of scope. On EL there are modules, but that requires the whole package set to be compatible. Instead, the work will start with building the new platforms (EL9 and Ubuntu 22.04) with components that are compatible. This means initializing empty repositories and getting foreman-release out. Then once components are ready, they are packaged.

It should be noted that today we have gating by pipelines, but that requires more work so initially they will be pushed to nightly without any verification.

Installer

First all individual modules must be made compatible. Many components describe parts that are delivered as part of the OS, such as Apache, PostgreSQL, ISC DHCP, ISC BIND and TFTP. That work can start and in fact already has. Other modules, such as foreman and foreman_proxy need a packaged version. This means it depends on the previous steps.

Once a complete scenario is available, the foreman-installer package itself can be packaged as well. The foreman-proxy-content scenario is the most likely candidate since the underlying pieces are furthest along (Pulp already runs on EL9).

Timeline

All of this work can happen in parallel, but all projects need to be available before it’s officially released. Having looked at the work needed, it’s very likely work won’t be finished before we branch Foreman 3.5 but are not yet ready. We’ll have to decide if we then have an incomplete repository or only have the changes in nightly.

Update 2024-01-22: We have initial EL9 packages using Ruby 3.0 for most packages, except for Foreman and its plugins itself. The goal is to deliver experimental EL9 support in Foreman 3.10. Version 3.11 will be production quality, including a smooth migration path from EL8.

5 Likes
2

I’m not sure I understand this part. Incomplete repository means some parts Ruby 3.0 compatible but some not? And if that’s the case, parts means Foreman Proxy and Foreman or even e.g. Foreman dependencies?

Otherwise a great write up, it’s good to see what all is ahead of us. That feels like a lot of work and more hands would be… handy :slight_smile:

3

Indeed. This is because today you can’t even get Foreman to run on Ruby 3, so you can’t even build a package. With EL9 we will start with an empty repository (no packages). Then we’ll add foreman-release. Once it’s ready, we can add foreman-proxy. However, until foreman is added I consider the repository incomplete. There are probably more packages that must be present and I didn’t dive into it now, but we will find that out along the way.

1 Like
4

Something I completely forgot, but there’s also NodeJS to consider.

Today we build with NodeJS 12, but support 14. EL9 has version 16 by default and 18 in a module. Ubuntu 22.04 has version 12, but there we’ve used the nodesource repositories to update. I’d suggest to aim for a single version for our packaging environment, but have to rely on people more experienced with it to determine if it should be 16 or 18.

I think our current Webpack 3 (way outdated) is incompatible with NodeJS 16, so that also requires a serious effort. There’s already a plan written for that:

2 Likes
5

Got it, that makes sense.

6

Big + for the summary what needs to be done, suddenly it doesn’t seem to be so easy :smiley:

My addition to this is about the plugin extensions, due to Zeitwerk we will have to drop all the extensions that are monkey patching Foreman internals and replace them with our plugin DSL, like in foreman_ansible for example.
It’s bad practice anyway so even if we won’t needed it will be great improvement.

In some cases DSL is already there, but for many cases we don’t have it. I’ll gladly take responsibility for this part, I will probably start with Katello as there will be a lot of work and then go over other smaller plugins.

3 Likes
7

I’ve created a GitHub project as a way to track the progress:

This is the first time I’m trying an organization wide project, but I hope it’ll be useful.

2 Likes
8

Nice summary of the work required.
Geezus christ though, what a lot of ballache.

Why in hell does EL9 not just provide a ruby 2.7 in addition to the Ruby 3, for christ’s sake.
Pathetic, that Red Hat does not care enough about backward compatibility, and its customers, to do this. It should not be difficult to have Ruby 2.7 and 3 on same server.

9

Quite simple, as Ruby 2.7 is EOL. Better pushing resources to move forward than to support something EOL. But this only my opinion!

2 Likes
10

Ruby 2.7 EOL is no excuse
Windows can still run binaries from Windows 95.

And last time i checked, I could still run Orbital Eunuch Sniper and Quake 3, on linux, and those games are now several decades old.

Just because Ruby 2.7 goes EOL doesn’t mean everything written in Ruby 2.7 is thrown on the scrapheap. And if it does mean that, then the question must be asked,why do people write anything, in these poorly support products that are going to disappear in a few years with no effort made by anyone to maintain backward compatibility.

Even if Ruby 2.7 is EOL there should be a ruby 2.7 legacy or compat package so things that require it can continue to function properly.

11

Hello. Looks like there’s “only” the Rails 7 & Ruby 3.1 task left on “Ruby 3 support” project tracker. Any idea what version of Foreman is targeted for initial el9 support, or is that too far off in the future to tell?

12

It’s hard to commit to any version.

@akumari has done a good job on making sure the Smart Proxy is tested with Ruby 3 in our CI. While it’s not acceptance tests or full system integration, it is a solid base layer.

As for Rails, it’s less positive. While my PR (Rails 7 & Ruby 3.1 by ekohl · Pull Request #9328 · theforeman/foreman · GitHub) might give the impression it’s close, that’s only the start. Once we have Zeitwerk working in Foreman itself, we must also make sure all plugins work with it. That hasn’t even started so who knows which dragons lurk there. I’m trying to work with @ofedoren and @akumari to make progress, but given where we are I wouldn’t expect it in Foreman 3.9.

For EL9 we also need to tackle our NodeJS update, which implies a webpack update. @MariaAga has submitted https://github.com/theforeman/foreman/pull/9834 which is a leap forward. Resurrection of the client-side infrastructure upgrade effort is also tracking that. AFAIK that has been tested in developer environments, but is now in review and packaging also needs to be done. Given its current status this has the potential to make it into Foreman 3.9.

2 Likes
13

Thanks, @ekohl . Appreciate the update.

14

Today @akumari and @ofedoren discussed the Rails part. Essentially we have this dependency chain:

EL9 needs Ruby 3
Ruby 3 needs Rails 7 and other gem updates
Rails 7 needs Zeitwerk

So we first want to focus on the Zeitwerk migration. It’s something we can start shipping with our current Rails 6.1 stack.

Once core is compatible with rake zeitwerk:check that will be added to our CI pipeline to prevent regressions.

It may break a lot of plugins if we just switch core, so we want to introduce a setting in settings.yaml so you can switch it. Once the full test suite passes with it, the CI test suite will be expanded to run it with both the legacy and modern loader. At this point we’ll push plugin developers to make their plugins compatible. The setting can also be switched in production builds, so full end to end testing can be done. We may also introduce an additional pipeline in our nightly runs to do so every run.

Compute resources don’t have automated end-to-end coverage so will need manual verification.

This is something we want to reach for Foreman 3.9. The default isn’t switched yet, but be ready. Then give plugin developers some time to do the same. Shortly after we branch and develop becomes 3.10-develop, we will switch the default.

Then we spoke about future steps. With Zeitwerk in place, we can start Rails 7. It’s unknown how many issues we will find. If all goes well, we should target it for Foreman 3.10.

Even further in the future is Ruby 3 itself. At this point I’d say that Foreman 3.11 may be optimistic, but I’d like to target Foreman 3.12.

As always: these are intentions and no hard commitment. Most of us know how messy these things can be and you always find dragons.

We’ve agreed to meet up weekly and for now I’ll keep posting the updates here.

Short term @ofedoren will look at my Rails 7 PR and see which Zeitwerk parts he can take over and possibly already merge. @akumari will look at the fast_gettext update in there, which is needed for Ruby 3.2 support.

3 Likes
15

I’m curious what drives this requirement? Rails 6.1 doesn’t run against Ruby 3?

16

I tried to look for official support, but these guides don’t explicitly say “now it runs with Ruby x.y”. I looked at:

Rails 7.0.1 was at least the first one to support Ruby 3.1 (Ruby on Rails — Rails 7.0.1 has been released), which is included in Debian 12. It’s also the first one to say Ruby 3.0+ preferred. So perhaps it’s based on an assumption on my side.

But you’re right. Ruby & Rails Compatibility Table - FastRuby.io | Rails Upgrade Service states that is recommended to run on Ruby 3.0.Z. I suppose I when I started, I was very focused on getting it to run on my Fedora which included Ruby 3.1.

That does change the dependencies a bit and allows for more parallelization.

2 Likes
17

As promised, I’m summarizing our weekly sync up here.

We briefly discussed the possibility to run Rails 6.1 with Ruby 3. @ekohl will look at introducing expanding the Foreman Jenkins job to start testing on Ruby 3.

@ofedoren noticed that openscap_parser started to require Ruby 3, though that may have been a mistake (fix: RHICOMPL-3900 disable required ruby version rubocop check by marleystipich2 · Pull Request #56 · OpenSCAP/openscap_parser · GitHub).

@ofedoren is wrapping up some other work before he can start investigating the Zeitwerk status

@akumari is investigating the update to fast_gettext.

3 Likes
18

And another weekly sync up.

4 Likes
19

Due to people having time off we didn’t have the regular sync up, so it’s been a bit quiet on the updates. That doesn’t mean no work has been done.

@ofedoren has been working on testing Foreman with Ruby 3 in CI (Fixes #36849 - Set up CI with Ruby 3.0 by ofedoren · Pull Request #9871 · theforeman/foreman · GitHub). We’ve decided to split this up into two parts: one part that adds GitHub Actions based testing with the current status (Ruby 2.7). The follow up will add Ruby 3.0 to the testing matrix. @ekohl will write up a separate RFC on this topic so it’s also clear what the plan for plugins is. This RFC is already big enough and it would get lost here.

While testing @ofedoren ran into Fix possible nil volumes for libvirt by ofedoren · Pull Request #133 · fog/fog-libvirt · GitHub and @ekohl will do a release of fog-libvirt. That’s useful anyway because there are unreleased changes that bring compatibility with modern libvirt as included in EL9 (Set the default machine type to q35 on i686 and x86_64 by ekohl · Pull Request #127 · fog/fog-libvirt · GitHub & Fix disk type for block devices on QEMU >= 6.0 by ekohl · Pull Request #129 · fog/fog-libvirt · GitHub).

@akumari has been moving forward with the apipie-dsl translations. While no longer a blocker to upgrading, https://github.com/theforeman/foreman/pull/9867 is still a nice step forward.

@akumari has also started to look at updating GitHub - theforeman/theforeman-rubocop: Foreman RuboCop basic rules to the latest version and making it the recommended Foreman & Foreman plugin RuboCop configuration. This is useful because there are various cops that help with the Ruby 3 upgrade (Lint/DeprecatedClassMethods being the most obvious, but there are more).

5 Likes
20

Another week, another update.

First of all, @evgeni and @ehelms will try to assist in the effort. We took a look at the overall structure of the project to try and divide tasks.

Taking a high level overview, we can identify a few subsystems (assuming a full Katello installation):

  • Foreman
  • Foreman Proxy
  • Hammer
  • Pulp
  • Candlepin

For simplicity, I consider plugins part of the subsystem.

Broadly speaking we consider some subsystems ready for EL9, but sometimes untested. Foreman Proxy, Hammer, Pulp & Candlepin all should work but most of the readers will know that theory and practice don’t always agree.

@evgeni wants to get started on packaging effort for those already (and Pulp is already done). So EL9 yolo by evgeni · Pull Request #9990 · theforeman/foreman-packaging · GitHub is the first kick off for that.

For Foreman we still have quite a bit of work to do and that’s what we’ll continue to focus on. As mentioned previously, the whole CI setup needs an overhaul. Foreman has long used Jenkins, but it needs an update. I’m preparing a separate RFC for that, but the summary is that we want to use GitHub Actions.

We also made sure Ruby 3 support · GitHub was updated.

2 Likes