RFC - Enterprise Linux 8 and 9 Support

With the end of life of CentOS 8, and a focus on CentOS 8 and 9 stream, the Infrastructure SIG has been discussing what to do about building of RPMs and testing of RPMs. The choice of distribution has impacts on what we test and support as installation OS targets for Foreman and its plugins. I will use the following terms thusly:

  • tested: operating systems that our release pipelines test on ensure work prior to release
  • supported: operating systems that should work for installation but are not tested

The two options we have for EL8 (and eventually 9) are:

Option 1: Build against RHEL

If we build against RHEL we could in theory have the following:

Tested: CentOS 8 Stream
Supported: RHEL, Alma, Rocky

There is more complexity in build environment to build against RHEL as per the RHEL open source initiative we could have the repositories local to our Koji as long as we ensure they are not publicly accessible. The trade off to building against RHEL is we have to wait for RHEL to be released to update our builds against the newer version and could encounter a period of breakage either stemming from CentOS Stream or within latest RHEL itself. For EL9, we would not be able to begin building until RHEL 9 was released (we might could consider RHEL 9 beta). We would be sitting in the middle of the ecosystem and reacting rather than at the beginning of the ecosystem and being more proactive.

Option 2: Build against CentOS Stream

If we build against CentOS Stream we could in theory have the following:

Tested: CentOS 8 Stream
Supported: N/A

The obvious downside for some users would be that we are only targeting CentOS stream as our EL OS for installation. This does have the benefit of ensuring we are crafting a well working and targeted experience for using our software. Building against CentOS stream means we can catch build and install issues much sooner for the EL ecosystem.

CentOS 8 ends of life at the end of 2021 and the repositories will move to CentOS vault with no further updates. Thus we have some leeway but need to take action on our direction with EL ecosystem.

To that end, we are looking for community input on which direction we go based on what the community needs are balanced with the maintenance cost. Please let us know your thoughts in this RFC so we can arrive at the best decision possible.

If you are still reading this, I likely got something wrong and will edit the post with any corrections pointed out to me.

4 Likes

Hi!
Thanks for bringing this to the community.

EL8:
Option1: Building on EL:
I’d like to add Rocky as build host to the considerations.
Building on Rocky8 for EL8 would be quite the same as building on CentOS7 is for EL7.

Tested: Rocky
Supported: RHEL, Alma, Rocky, OracleLinux, CentOS Stream

Option2: Building on CentOS Stream:
Building on CentOS Stream would mean “running on CentOS Stream only” (at least for a while) until EL has all dependent packages and versions available. I don’t see this as a preferable option.

EL9:
In my opinion we could wait until RHEL9.0 / Rocky9 is released. According to my experience from former EL releases, a .0 release is normally not used for production systems :wink:

  • Mark
4 Likes

Hi,

full disclosure before I throw my 2ct in here: I am somewhat out of the loop about the state of community EL distibutions since EOL for CentOS 8 was announced.

With that in mind, I would highly discourage dropping support for everything but CentOS Stream. From what I am and have been seeing in support toppics, most people want to use Foreman on a more stable, “enterprisy” distibution. After the announced end for CentOS 8, we had a big wave of people trying to deploy to Rocky or Alma since they wanted the stability of “classical” CentOS but could not go paid-for RHEL. Also, a somewhat large portion of the community has historically been on RHEL afaik and dropping support for them would also be a huge bummer.
Many companies require their IT departments to use a certain OS for everything (or at best one out of a selection of a few) and all those people/organizations would then de-facto be locked out of using Foreman. They would either need to break their companies compliance rules or expect their Foreman environments to break when updating.
In my opinion (not beeing a developer myself) the drawbacks of dropping nearly all distro support outweigh the advantages of “beeing at the beginning of the development” stream, especially when that means losing a big part of the community.

If I got something wrong on this topic, feel free to correct or ignore me :wink:

Concerning EL9: It would definitely be nice to have EL9 support up early so people can plan their migration better and can get a longer migration period. But as Mark already mentioned, nearly nobody uses .0 releases in production anyways :wink:

Regards

3 Likes

Hi, thanks for bringing up the discussion!

I don’t think that only testing and supporting CentOS Stream is a valid choice - at least I don’t see my customers willing to adopt this. Most of my customers have been using CentOS and RHEL - most of them switched to Rocky and/or Alma, a very few had a look at CentOS Stream. Don’t get me wrong - I strongly disagree with the typical “CentOS Stream is not stable enough” statement. I think CentOS Stream is perfect if you want to adopt very early and test new changes before arriving in RHEL. But a lot of people just want a RHEL downstream at no additional cost.
Therefore I’d love to see that at least RHEL is supported as well.

Totally agree with this. :slight_smile:

3 Likes

Thank you for soliticing input on this topic in Discourse.

Please consider the obvious choice of simply continuing to build / test on a RHEL downstream distro (Rocky or Alma). I know there are reasons that this was excluded but it really is the right choice for the project. It worked pretty well for the last several years and the turbulence in the CentOS project/leadership doesn’t mean that this approach is any less useful for Foreman.

Building on RHEL is a poor second choice for the reasons you’ve already identified (among others) but it would still provide a path forward in 2022.

Building / testing on RHEL upstream is the least desirable choice. It’s likely to result in less adoption of the Foreman builds since fewer people are using the RHEL upstream today and they won’t want to bring in another distro just to run Foreman. It could also encourage an outside-the-project rebuild to gain adoption (perhaps in a Rocky/Alma SIG or in a refreshed/repainted Oracle Linux Manager build). If any of this happens, it makes the feedback loop less useful for the project.

3 Likes

Perhaps an unpopular opinion, but I would say Foreman / Katello is an upstream project, so building on an upstream distribution would be the way to go. If something on a downstream distribution is not already available, you have to wait. If something got changed there it will fail, but it is not upstream’s fault. Assuming it will always work on a different distribution then it was build on, will likely not be a valid assumption (and was not in the past). And with distributions that rebuild RHEL, there will always be some delay, so there could already be the problem that you have to wait.