The PR to build packaging on EL9 (Build Rails for EL9 by evgeni · Pull Request #10015 · theforeman/foreman-packaging · GitHub) has been discussed. The approach is to create a new config in COPR that has both EL8 & EL9 chroots, so it can build them in parallel. We can then bootstrap the platform by moving groups over, in the correct order. It will be good to keep notes since we’ll apply the same thing once EL10 comes around the corner.
Next week’s meeting is cancelled because most people have other obligations.
@evgeni has made a lot of progress packaging large parts on EL9. Next steps are to build the Smart Proxy plugins and publish the packages on yum.theforeman.org so we can start updating our Puppet modules to also test it.
@akumari has also been traveling and had time off.
As every year, Ruby has released a new version over Christmas. Ruby 3.3 appears to be small in terms of breaking changes. While reading up on it, I noticed that Ruby 3.0 is EOL in March 2024, so in 3 months. I’d still go forward with releasing it on EL9 on 3.0, even if it is EOL because Ruby 2.7 is already EOL since March 2023 anyway and that hasn’t held us back.
The dependency on foreman-debug and foreman & foreman-proxy has been dropped (Fixes #37022 - Drop requirement on foreman-debug · theforeman/foreman-packaging@acccb9a · GitHub) since it’s not really needed to run and simplifies publishing a minimal set. We’ve already largely switched over to sosreport anyway and for end users it also reduces the required dependencies on minimal installations.
How to install test dependencies? Today Jenkins for regular plugins uses gemfile.d/*.rb and our reusable actions do the same, but Katello has a special job in Jenkins that installs it as a gemspec, which installs all development dependencies. This came up in https://github.com/Katello/katello/pull/10780.
Today Katello’s tests run on every Foreman PR using Jenkins. Should this be ported to GitHub Actions?
One remaining issue that shows up in most plugins is running of tests from Foreman core in a plugin (always execute access_permissions_test from Foreman core by evgeni · Pull Request #16 · theforeman/actions · GitHub). This isn’t an issue in Jenkins, because that runs the entire Foreman test suite (to prevent regressions). GitHub actions only runs the plugin’s but some do run some tests from core. This needs a resolution for plugins using GitHub Actions today.
Packaging wise all EL9 RPM workflows are present: Foreman, Plugins & Katello. The list of packages is incomplete, but a content proxy now passes tests. Remaining is Foreman & its plugins. That’s blocked on Ruby 3 & Webpack 5.
For Webpack 5 the current proposal is to pull the trigger this Friday (2024-01-26). A separate announcement for that will be posted.
Verify it works with all plugins · Issue #1 · theforeman/actions · GitHub has been extended with all the plugins from Foreman landscape that matter. To make this easier, the landscape has also been updated with the testing overview (Jenkins, GitHub Actions, Puppet). This will be even more important because after we merge Webpack 5 we will start to depend on NodeJS 14 and many current GitHub Action implementations still use NodeJS 12. So implicitly there’s also a deadline for this.
In some repositories (at least foreman_openscap) we’ve seen that RuboCop didn’t ran before. For those cases it’s OK to just convert the existing tests, but long term that should be figured out. It’s confusing when there is a RuboCop configuration, but it doesn’t run.
@evgeni has also done some initial work on Debian 12 & Ubuntu 22.04. Debian 12 is Ruby 3.1, but that requires Rails 7 (including Zeitwerk). Ubuntu 22.04 has switched compression, which doesn’t work with our own mirrors right now. This for sure won’t happen (not even experimental) for Foreman 3.10.
The big thing is that we are indeed going to merge webpack 5 tomorrow (Webpack 5 merge this Friday (2024-01-26)). To facilitate this the continued focus has been on the GitHub Actions conversions for plugins. By now for almost all the plugins there’s a PR and various PRs have been merged. While doing this, it was noticed that foreman_setup would need work, but it’s rather unmaintained so I’ve proposed to drop it: Retire foreman_setup plugin
Like with webpack we want to set a date where it’s merged. This date will be Tuesday (2024-01-30). That way we don’t (plan to) overlap with the webpack while also providing sufficient time to fix things before branching and before FOSDEM and cfgmgmtcamp.
Something I now notice about last week’s update: we are talking about delivering EL9 support in Foreman 3.10 as experimental. Mostly because this means we can deliver it without any EL8 → EL9 migration support (and postpone that to Foreman 3.11). Of course users can do this on their own, but they’re properly warned.
In the past week the focus has been less on Ruby and more on NodeJS. As announced we merged webpack 5 and as expected, there was some fall out. Nothing major, but it occupied us nonetheless. For all the details, see Webpack 5 merge this Friday (2024-01-26).
One uncomfortable thing that showed up NodeJS 16. This is needed for EL9. It looks like we still need legacy peer dependencies handling with NPM. Another is that on EL9 the development packages for NodeJS 16 are missing. Because of that the current investigation is looking at NodeJS 18. That version also ships NPM 8 and we expect few problems. Fedora 38 ships it, which means you can finally just use the system packages in your development environment. Fedora 39 ships NodeJS 20, so that support is also considered. Active work is in Fixes #37134 - upgrade to node 18 and npm 8 by MariaAga · Pull Request #10016 · theforeman/foreman · GitHub.
If we look at the Foreman 3.10 schedule then it’s getting rather close. Next week we have CfgMgmtCamp 2024 where both @evgeni and I will attend. The week after that it’s already stabilization week.
I see 3 options:
@ehelms has time to work on the NodeJS > 14 upgrade next week and can get it done
We postpone Foreman 3.10 branching for this
We postpone Foreman on EL9 to 3.11
@evgeni notes that it’s very useful to have a non-official Foreman on EL9 release (as 3.10 was intended to be) because that makes upgrade testing possible.
@ekohl will discuss with @ehelms and the @releases team what’s the best course of action here.
Good news! Thanks to @MariaAga in Node 18 PR I had the thought we should be able to use the same --legacy-peer-deps in our NodeJS packages that are bundled and achieve the same result. That is, a package that works on 14 and 18, easing the pain of transition by allowing us to rebuild on 14 and then the packages continue to work on 18.
I undertook and tested the following:
Building all packages on NodeJS 14 with the --legacy-peer-deps flag
Building all packages on NodeJS 18 with the --legacy-peer-deps flag
I have opened a PR to npm2rpm to add support so that not only will the spec files get generated by our update script properly, but also so that the caches are generated with legacy peer deps.
The meeting yesterday didn’t happen, as several people were either still traveling or recovering from travel/conferences.
Nevertheless, progress happened, so here is a few highlights from me:
thanks to the work of @MariaAga and @ehelms on NodeJS 18 / NPM 8+, we now have rebuilt our RPM packages with --legacy-peer-deps, which allowed us to also build them on NodeJS 18 on EL9 (EL8 remains on NodeJS 14 for now)
this allowed us to build Foreman and almost all Plugins in the “plugins” repository for EL9 (using Ruby 3 and NodeJS 18). notable exceptions are:
when deface and rex packaging changes were applied, and using the patched selinux policy, the “foreman” and “plugins” pipelines pass fine (well, plugins do if we disable discovery), which is a HUGE milestone! (those tests include REX/Ansible, which also validates that dynflow and friends are working)
the next steps will be tackling Katello and the plugins that depend on it
They aren’t blockers for RC1, but we should aim to get both resolved for RC2.
To be explicit: the Foreman 3.10 goal is experimental support. We won’t provide a migration path from EL8. We expect it to largely work, but it’s a large upgrade and we may have missed things. Foreman 3.11 aims to complete the work. This means:
Upgrade all platforms to NodeJS 18 (instead of just EL9)
For both I’m tempted to split them off to a separate project and rename the existing one to EL9 support, just to keep things clear.
Other follow ups for 3.11 include:
Start building for Ubuntu 22.04, which includes Ruby 3.0 so it should just work
Start the Zeitwerk migration to get us on a path to Rails 7, opening Ruby 3.1+
Ruby 3.1+ is important, because Debian 12 and Ubuntu 24.04 both ship Ruby 3.1. Right now CentOS Stream 10 is also being branched. Depending on which Fedora version they base it on, I’d expect Ruby 3.2 or 3.3 (with the latter being more likely).
As a last note we’ve started to discuss EL8 server support deprecation. We want to give users a smooth, well tested path so we’re currently looking at 3.13 to drop EL8. This will be communicated in the 3.10+ release notes.
RuboCop upgrade · GitHub This isn’t blocking EL9 anymore, so it deserves its own effort. A proper RFC is being prepared and will be tracked separate in the future.
Then there’s a new board for Ubuntu 22.04 support: Ubuntu 22.04 support · GitHub. Because of all the efforts for EL9 this was a rather small effort and everything is on track for Foreman 3.11 to support it.
Since last time the EL9 support on the old manual has been added.
Not on EL 9 itself, but on EL 8. The upgrade from PostgreSQL 12 to 13 is implemented. Documentation of this is still in progress. This matches the version to EL 9 and was a blocker for upgrades using leapp.
This is done and Foreman 3.11 will release with it. We haven’t decided on a deprecation path, but we’ll encourage users to upgrade. The manual has been updated, but new docs haven’t yet because the changes would conflict with the in progress EL 9 documentation so it’s waiting on that.
Debian 12
Debian 12 uses Ruby 3.1. That should largely work. Rails 6.1 should be compatible with Ruby 3.1, unlike what I first thought. That was because Rails 7.0 had some issues, but that was in changes introduced since 6.1. Safemode is only marked compatible with Ruby 3.0. Once that’s updated, we can start building Foreman on it.
Will the documentation/support/scripting to upgrade el8 from PostgreSQL 12 to 13 be available before 3.11, or is it targeted to come out at the same time as 3.11?
I’ve been working on getting my Foreman/Katello install working on el9 with a fresh build, and then restoring from a backup taken on my old el8 instance. However, restore is not currently happy that the DB I’m restoring (v12) is not the current version (v13)
