CentOS 8 Stream explained

Development
1

Hello,

since we have multiple threads about the CentOS 8 Stream announcement, I would like to make a single place where CentOS 8 Stream can be explained. This page is a wiki, feel free to update information, keep the info on point and relevant for Foreman and Katello users.

Full disclosure: I work at Red Hat, the goal of this post is to iron out some misinterpretation of the original announcement which was a bit confusing.

CentOS 8 Stream does have the standard five years lifecycle. CentOS 8 was announced EOL by end of 2021 which is indeed a significant change from expectations. Scott McCarthy later explained (see links below) that the management did not want to wait until CentOS 9 with the change or until CentOS 8 adoption was too high. This change is indeed controversial, there’s no doubt that this mid-flight change is painful. However, what’s the most important is that CentOS 8 Stream has EOL planned for 2024, all you need is to switch repositories.

CentOS 8 Stream is not a rolling release in Gentoo/Arch Linux sense. The project is not switching from stable server-oriented distribution to stream of upstream packages. That’s what’s Fedora Rawhide for. All updates coming into CentOS 8 Stream are be carefully evaluated and cherry-picked by Red Hat engineers and managers. Proper term should be Continuous Delivery, what is changing is the process not content.

CentOS 8 Stream is not lowering quality. Some people may think that since CentOS is a RHEL rebuild, it has superior quality because all bugs had to be “ironed out” by RHEL customers. Fact is, when a bug slips through Red Hat QA into a RHEL release, CentOS gets it as well. The only difference is it gets it few days later. CentOS has always been mainly rebuild project, bugs in the software itself are tracked in Red Hat bugzilla or upstream trackers. CentOS Stream is not supposed to by a nightly build, updates are pushed only after an explicit QA pass. Every bugzilla has also an explicit (usually fully automated) test. Quality is not changing.

CentOS 8 Stream however will change cadence of updates. Some updates land in RHEL early (during lifetime of a y-release) and others wait until the next y-release. In CentOS Stream however, all updates are pushed as soon as they are signed off by Red Hat Quality Assurance department. While this happens mostly every day, this is far from Gentoo Linux, far from Fedora Rawhide. All RHEL updates are still subject to Red Hat API/ABI promise for RHEL, therefore CentOS 8 Stream is getting the same benefit even there was never any contract or promise for CentOS itself. What is more important than ever is managing updates in some controlled manner, via either rsync/reposync, Pulp or Katello. Content management plays an important role not only for CentOS 8 Stream, but for all operating systems.

CentOS 8 Stream does’t have 8.y branches. This is a major drawback for us, we cannot say “tested on CentOS 8.3 or higher” because there is currently no point in time which can be installed in the future. CentOS Stream installation DVDs are getting published regularly so we should be able to say “works with CentOS-Stream-8-x86_64-20201211-dvd1.iso or newer”. The problem is, the link will not work in few days and since there is no base/kickstart repository available network-based installations do not work without manual DVD download and extraction. We need to look into this to find out if Katello/Pulp can help with this.

The change was not driven by IBM. Red Hat operates as a separate entity, there’s not much to say. CentOS brand was acquired by Red Hat in 2014 and apparently this is where Red Hat would like the project to go in the future. As Scott McCarthy mentioned: “IBM literally play no tactical role in our product and project decisions.”

Foreman and Katello keep fully committed supporting as many operating systems as possible. The Foreman community will do its best to support everything possible, including CentOS Stream, RHEL, Fedora, Debian, Ubuntu, SUSE or Oracle Linux (or any RHEL clone including the newly announced ones). Majority of Red Hat engineers working on the project will likely continue using RHEL/Fedora/CentOS Stream while ATIX engineers filling the list with Debian/Ubuntu/Suse. All patches will be accepted as long as they meet our quality standards.

Running Foreman/Katello on CentOS 8 Stream is something will be looking into. Foreman 2.3 currently support EL7 and EL8 (and clones) and Debian/Ubuntu at the moment. See our installation instructions for more information.

Links:

8 Likes
2

One issue we have is our backup vendors will nor certify centos if it is on a rolling upgrades. They only ever did it because centos was identical to rhel.
I also wonder how the community will provide guides when there will be a lot more fluctuation in builds.

3

Except it never was. CentOS is a complete rebuild of RHEL, there are differences in build environment and some software is missing (rhsm). Truth is, it is very close, you can even switch between RHEL and CentOS back and forth on a single server but you need to be really lucky (yum dependencies). However, the following might surprise you - your CentOS certification might not work as expected. Let me explain:

Suppose you got certification for CentOS 8.2 for example, so you installed it and never performed yum update. This is the only workflow for CentOS8 (for CentOS7 see below) that would guarantee the certification actually. Because if you perform CentOS updates, one day before CentOS 8.3 release your fully updated system is 99% close to CentOS 8.3. Major RHEL version is just a “freeze point”, you already have most of the updates installed when it comes up effectively invalidating the certification.

Certifications make much more sense if you have a RHEL subscription with EUS lifecycle extension where you actually “freeze” your system on a particular version (e.g. 8.2) and you only upgrade it once you are ready (e.g. to 8.6 after your vendor issues certification). Customers essentially skip few releases which lowers their upgrade costs and enables vendors to certify. There are multiple programs/subscriptions available for RHEL to achieve this as well as ELS (extended life-cycle) when you are not ready for major RHEL upgrade (7 to 8).

Fact is, you could not achieve this with CentOS, EUS channels have never been available. CentOS has always been only “main line” RHEL update stream due to lack of resources. I think package sources of some channels are not even publicly available unless you are a Red Hat customer, I might be wrong about that.

You might argue that you can only apply security updates, that’s a difference. Yes, for CentOS 7 or below this is an option. However keep in mind that CentOS project does not publish errata metadata, you can achieve staying on particular version by parsing CentOS announce mailing list and importing data into Pulp/Katello via scripts or by paying a 3rd party which provides these repositories. For CentOS 8 however, due to lack of resources, errata information wasn’t published yet. AFAIK this is, again, due to lack of resources in the CentOS project.

1 Like
4

I appreciate your replies.
I know that in future when I contact third party support they will immediately come back with it’s not on a supported platform". They are not going to want to touch centos the rolling updates will cause far too many issues for them (or they will perceive them to). Centos was indeed not identical RHEL (as you correctly corrected me on), but it was close enough for vendors to recognise that it was effectively the same “product”, and the number of issues would mirror each other, so they could extend the scope of their support.
It’s become far messier than it was and I object to that because we were told one thing and with little notice told another. Fortunately we can switch OS with minimal impact … largely thanks to foreman/katello. Almost certainly oracle Linux (yeah I know), but maybe rocky Linux if it all checks out.

I am aware that your knowledge, experience and development vastly outweighs mine (by orders of magnitude), so you are the person that should be listened to in this debate.

5

My main point is, it is irresponsible from a vendor performing quality assurance on RHEL and then just saying “it works for CentOS too”. This leaves CentOS users in a vacuum, CentOS is only close enough to RHEL in the time of a minor release, as the time goes on between mainline and EUS (or any other frozen subscription certification program) the difference gets larger.

I get that and I respect that most people would rather keep the status-quo. But if you go really into details you will find out that CentOS 8 already was a continious delivery. There are no tools at the moment that would get you RHEL level of update granulality. You can do some things with Pulp and Katello, but without good errata metadata it’s not fair fight. I think the change is for better for both Red Hat and CentOS users because technically there’s not much changing.

I do believe this was communicated poorly and misinformation was spread around. Also it is fair to add that some RHEL clones do provide errata information today so it’s easier to work with it. I haven’t tried Pulp or Katello myself with these tho. :wink:

8

I will be looking into CentOS 8 Stream provisioning soon. I am pretty sure there will be problems, for example the kickstart repository does refresh PXE images regularly, this will cause the TFTP/HTTPBoot smart proxy to corrupt images because we currently do a “dumb” wget -c command.

1 Like
9

Happy with the errata being provided by Oracle Linux 8 here…

10

I do not know if this is the correct place to post. In foreman 1.23.1 I create a new Operating System: CentOS8Stream and associate the media (mirror) and partition tables and the 3 kickstart files. All works fine for the hosts provisioning.
When puppet run for the fisrt time the OS change in generic CentOS the facter in the povisioned host is:

os => {
  architecture => "x86_64",
  family => "RedHat",
  hardware => "x86_64",
  name => "CentOS",
  release => {
    full => "8",
    major => "8"
  },
  selinux => {
    enabled => true
  }
}

There is a way to set the OS name to CentOS8Stream or CentOSStream or CentOS Stream so foreman keep the correct OS for that host in the we interface?

11

Hey @straldi

Welcome to the community !
Would you mind creating a new topic in the support channel? This would mean that there’d be a dedicated topic to your problem!

Foreman 1.23.1 is very old and that is probably a lot of your problem. Would you consider upgrading?

1 Like
12

Hi,
ok I will create a new topic in support channel.
we have a production cluster with CentOS7 and CentOS8 hosts and we use foreman for provisioning and puppet to configure Openstack infrastructure, and all other services. We could update to 1.24 but not to 2.0 release because the Mariadb/Mysql support is not more available and we would not change the DB to Postgress.

13

Please keep in mind that these releases are no longer supported and contain known security vulnerabilities and other bugs. I would highly recommend planning your upgrade to a supported version, despite the requirement to switch the database for it. If any fixes in the code are needed to allow properly supporting CentOS Stream, they will not be backported to the 1.x line.

1 Like