RFC - Enterprise Linux 8 and 9 Support

[ehelms]

With the end of life of CentOS 8, and a focus on CentOS 8 and 9 stream, the Infrastructure SIG has been discussing what to do about building of RPMs and testing of RPMs. The choice of distribution has impacts on what we test and support as installation OS targets for Foreman and its plugins. I will use the following terms thusly:

• tested: operating systems that our release pipelines test on ensure work prior to release
• supported: operating systems that should work for installation but are not tested

The two options we have for EL8 (and eventually 9) are:

Option 1: Build against RHEL

If we build against RHEL we could in theory have the following:

Tested: CentOS 8 Stream
Supported: RHEL, Alma, Rocky

There is more complexity in build environment to build against RHEL as per the RHEL open source initiative we could have the repositories local to our Koji as long as we ensure they are not publicly accessible. The trade off to building against RHEL is we have to wait for RHEL to be released to update our builds against the newer version and could encounter a period of breakage either stemming from CentOS Stream or within latest RHEL itself. For EL9, we would not be able to begin building until RHEL 9 was released (we might could consider RHEL 9 beta). We would be sitting in the middle of the ecosystem and reacting rather than at the beginning of the ecosystem and being more proactive.

Option 2: Build against CentOS Stream

If we build against CentOS Stream we could in theory have the following:

Tested: CentOS 8 Stream
Supported: N/A

The obvious downside for some users would be that we are only targeting CentOS stream as our EL OS for installation. This does have the benefit of ensuring we are crafting a well working and targeted experience for using our software. Building against CentOS stream means we can catch build and install issues much sooner for the EL ecosystem.

CentOS 8 ends of life at the end of 2021 and the repositories will move to CentOS vault with no further updates. Thus we have some leeway but need to take action on our direction with EL ecosystem.

To that end, we are looking for community input on which direction we go based on what the community needs are balanced with the maintenance cost. Please let us know your thoughts in this RFC so we can arrive at the best decision possible.

If you are still reading this, I likely got something wrong and will edit the post with any corrections pointed out to me.

[hlawatschek]

Hi!
Thanks for bringing this to the community.

EL8:
Option1: Building on EL:
I’d like to add Rocky as build host to the considerations.
Building on Rocky8 for EL8 would be quite the same as building on CentOS7 is for EL7.

Tested: Rocky
Supported: RHEL, Alma, Rocky, OracleLinux, CentOS Stream

Option2: Building on CentOS Stream:
Building on CentOS Stream would mean “running on CentOS Stream only” (at least for a while) until EL has all dependent packages and versions available. I don’t see this as a preferable option.

EL9:
In my opinion we could wait until RHEL9.0 / Rocky9 is released. According to my experience from former EL releases, a .0 release is normally not used for production systems

• Mark

[areyus]

Hi,

full disclosure before I throw my 2ct in here: I am somewhat out of the loop about the state of community EL distibutions since EOL for CentOS 8 was announced.

With that in mind, I would highly discourage dropping support for everything but CentOS Stream. From what I am and have been seeing in support toppics, most people want to use Foreman on a more stable, “enterprisy” distibution. After the announced end for CentOS 8, we had a big wave of people trying to deploy to Rocky or Alma since they wanted the stability of “classical” CentOS but could not go paid-for RHEL. Also, a somewhat large portion of the community has historically been on RHEL afaik and dropping support for them would also be a huge bummer.
Many companies require their IT departments to use a certain OS for everything (or at best one out of a selection of a few) and all those people/organizations would then de-facto be locked out of using Foreman. They would either need to break their companies compliance rules or expect their Foreman environments to break when updating.
In my opinion (not beeing a developer myself) the drawbacks of dropping nearly all distro support outweigh the advantages of “beeing at the beginning of the development” stream, especially when that means losing a big part of the community.

If I got something wrong on this topic, feel free to correct or ignore me

Concerning EL9: It would definitely be nice to have EL9 support up early so people can plan their migration better and can get a longer migration period. But as Mark already mentioned, nearly nobody uses .0 releases in production anyways

Regards

[stdevel]

Hi, thanks for bringing up the discussion!

I don’t think that only testing and supporting CentOS Stream is a valid choice - at least I don’t see my customers willing to adopt this. Most of my customers have been using CentOS and RHEL - most of them switched to Rocky and/or Alma, a very few had a look at CentOS Stream. Don’t get me wrong - I strongly disagree with the typical “CentOS Stream is not stable enough” statement. I think CentOS Stream is perfect if you want to adopt very early and test new changes before arriving in RHEL. But a lot of people just want a RHEL downstream at no additional cost.
Therefore I’d love to see that at least RHEL is supported as well.

@hlawatschek

In my opinion we could wait until RHEL9.0 / Rocky9 is released. According to my experience from former EL releases, a .0 release is normally not used for production systems

Totally agree with this.

[Tony_Coffman]

Thank you for soliticing input on this topic in Discourse.

Please consider the obvious choice of simply continuing to build / test on a RHEL downstream distro (Rocky or Alma). I know there are reasons that this was excluded but it really is the right choice for the project. It worked pretty well for the last several years and the turbulence in the CentOS project/leadership doesn’t mean that this approach is any less useful for Foreman.

Building on RHEL is a poor second choice for the reasons you’ve already identified (among others) but it would still provide a path forward in 2022.

Building / testing on RHEL upstream is the least desirable choice. It’s likely to result in less adoption of the Foreman builds since fewer people are using the RHEL upstream today and they won’t want to bring in another distro just to run Foreman. It could also encourage an outside-the-project rebuild to gain adoption (perhaps in a Rocky/Alma SIG or in a refreshed/repainted Oracle Linux Manager build). If any of this happens, it makes the feedback loop less useful for the project.

[Dirk]

Perhaps an unpopular opinion, but I would say Foreman / Katello is an upstream project, so building on an upstream distribution would be the way to go. If something on a downstream distribution is not already available, you have to wait. If something got changed there it will fail, but it is not upstream’s fault. Assuming it will always work on a different distribution then it was build on, will likely not be a valid assumption (and was not in the past). And with distributions that rebuild RHEL, there will always be some delay, so there could already be the problem that you have to wait.

[rhwood]

Please build on RHEL or CentOS Stream and not a downstream variant; I have customers who prefer to use a less expensive downstream variant, but who have also found bugs in CentOS (7.something if I recall correctly in that specific instance) not present in the equivalent RHEL. Having experienced that, it would not be good to build on a downstream variant and discover some issue then running Foreman in RHEL or other downstream variants due to how the downstream variant used to build Foreman was itself built.

If building in RHEL, any failures on a specific downstream variant can be declared a problem for the maintainers of the downstream variant to deal with.

[Dirk]

Just as a reference to what can “break” with Stream being not the target: Foreman 3.1/Katello 4.3: pulpcore has newer version of createrepo_c-libs on CentOS 8 stream - #3 by mhjacks

But of course it would also be the other way round if this would be the target and the newer version would be required but has not landed in RHEL and Rebuilds.

And I would still argue that it is better to not break upstream and downstream has to wait than probably not recognize that something would be broken on upstream and than still getting hit on downstream.

[mhjacks]

In my mind building on a downstream only postpones a potential problem for later (unless a package is pushed to upstream, and then updated or retracted before hitting the downstreams). Someone has to deal with the depsolving problems at some point - either it will be stream users earlier (and one can argue they should understand they have accepted some level of risk in this regard) or the downstream users later (when the bits in the upstream get to the downstreams). And I think the users of the downstreams will be more upset, in general, when they discover issues like this.

I get that there may not be an ideal solution for this - since we also have the potential of upstream having a package that the downstreams don’t have yet, and some of the Foreman/Katello/etc packaging depending on that package’s existence. So it’s there in upstream, but not in the downstreams, do we “penalize” downstreams for not having that package? Repackage that dependency in foreman/katello? I’m not sure.

That said, I think I’m still +1 for making -stream the preferred target(s) of testing/packaging.

[ehelms]

Thank you everyone for the input, this has been very valuable in helping us make a decision on direction. Feel free to keep sharing!

Based on the current feedback, and our discussions amongst developers we’ve opted to go with Option 1: Build on RHEL to focus on compatibility. The goal will be to build on RHEL as its released and continue our testing on CentOS stream. We are tracking the work to implement our change in build environments within this foreman-infra issue.

[lzap]

Catching up with some older threads…

@ehelms

The goal will be to build on RHEL as its released and continue our testing on CentOS stream

I like this for a simple reason. As a developer, I suffered through many situations when I had a problem with either one of my personal servers or during Foreman development and having reproducer on fully supported system by Red Hat is key advantage. That is either RHEL or CentOS Stream. Staying as close as possible with these OSes gives us one huge advantage - I can be more closely in touch with Red Hat engineers when solving problems. Try to open an issue in the original CentOS bugtracker (or any other fork) and the reply will be: “not our problem, create a bugzilla for this”. That is right, these projects aim to rebuild an OS, that is vastly different from actually giving support for the bits they ship.

For example yesterday, I was facing an issue that had something to do with RHSM in livemedia-creator. In a bugzilla I filed, I was asked to try on both CentOS and RHEL, apparently there is an issue in OS detection code in Anaconda which enables/disables RHSM component.

https://bugzilla.redhat.com/show_bug.cgi?id=2034601

While this particular bugreport does not require Foreman installation per-se, it is a good example of something that would require me to reinstall on RHEL or CentOS Stream. Therefore I fully support this decision and while few might see it as some bad decision or pushing a product or something, it is really about not allowing arbitrary disconnection between our community and RH/CentOS Stream.

The lesson learned from this is: RHEL, CentOS, CentOS Stream, Alma, Oracle are different. There is an extra work in finding and reproducing some bugs and first and foremost open source projects IMHO should be developer (power-user) friendly first. People who contribute are more likely running into similar issues.

Don’t take me wrong, I really appreciate these forks exist, it is a key thing in FOSS to achieve balance and health. In all of the messy communication and execution, I think the advantages of RHEL and CentOS Stream faded out a bit. This is one of them.

[evgeni]

Small update after a few months:

• With CentOS Stream 8 being used more actively as a base for RHEL 8.6, we had a few places we had to adjust things (most prominently changes to Ansible packaging) for staying compatible with CS8. This is great, as it means Stream works as designed and we keep uptodate with changes that will land in a future version of RHEL (and its rebuilds).
• However, this also meant that we did changes where we have no good way to ensure backwards compatibility with the existing stable RHEL (and its rebuilds).
• Based on discussions in the release and infra teams, we have decided to add Almalinux 8 as a tested platform to our piplines (currently only in nightly, but we’re planning to backport these changes to existing stable releases too).

That means, the listing in the very first post, now should read:
Tested: CentOS 8 Stream, Almalinux 8
Supported: RHEL, Rocky

You may ask why exactly Alma, and not Rocky. This was decided by yours truly using a fair dice and the fact that the Vagrant boxes of Alma looked more regularly updated than the ones from Rocky at the time I have started implementing this. (It really shouldn’t matter which rebuild we’re testing on, they are all compatible, huh?)