Deprecation plans for Foreman on EL7, Debian 10 and Ubuntu 18.04

Development RFCs
1

It’s that time again when we should talk about deprecation plans for distro support.

The benefit that’s true for all plans below is less work to support a distribution. Over time it becomes harder to support since you’re getting used to features present in newer distributions. Resources are finite so it’s best to spend them wisely.

EL7

This is probably the furthest away, but given the large deployment I think it makes sense to announce it very early.

Let’s start with some facts. RHEL 7’s lifecycle is documented on Red Hat’s product life cycles. It is currently in Maintenance support 2 (see Red Hat Enterprise Linux Life Cycle - Red Hat Customer Portal for details). This means no new features are introduced and it’s only security updates. This phase ends in 2024. CentOS 7 will also go EOL at the same time.

For Foreman we have built for EL 8 since Foreman 2.1, but there have been some feature gaps. Katello being the biggest, but that’s been resolved with Katello 4.0. That means we can start to recommend users moving over.

Then we should also consider that EL9 is on the horizon. It’s expected that CentOS Stream 9 should be on the mirror network in a matter of weeks (source). This brings us to some CI limitations. We don’t have enough resources to do our pipeline testing on all OSes. Dropping EL7 will free up those resources.

For that I’m proposing that we start announcing with Foreman 3.0 that we’re deprecating EL7 support. Actual removal can happen later. This isn’t based on a lot, but I’m thinking about 3.2 or 3.3.

Debian 10 (Buster)

The Debian release table is here:
https://wiki.debian.org/DebianReleases

Debian 11 (Bullseye) was released August 14th. We have not started support to run Foreman on it. This needs to be planned and executed. Once we have Debian 11 support, we can give users 1 or 2 releases to migrate over. That’s how we’ve always dealt with it. Debian 10 will go EOL in ~2022-08.

This means that if we’re optimistic we can deprecate Debian 10 support in Foreman 3.1 and drop it in 3.2 or 3.3, but this means we need to start working on it soon.

Ubuntu 18.04 (Bionic)

The release overview:
https://wiki.ubuntu.com/Releases

Ubuntu 20.04 support was added in Foreman 2.5. This means we can start to deprecate Ubuntu 18.04 support in Foreman 3.0. Actual removal can happen in 1 or 2 releases.

It should be noted that the actual EOL is in 2023, but this saves us resources. We are currently at capacity for Debian building/testing in our CI. Removal of Ubuntu 18.04 in Foreman 3.1 would free up capacity to build Debian 11.

Summary

So I’m proposing this:

Edit: 2021-12-08 Table updated to reflect that Foreman 3.2 will support Debian 11

Foreman version Notes
3.0 Deprecates EL7 and Ubuntu 18.04
3.1 Drops Ubuntu 18.04
3.2 adds Debian 11, deprecates Debian 10
3.3 -
3.4 Drops EL7 & Debian 10

If Debian 11 is not supported in 3.1, Debian 10 deprecation also moves to a next release and removal in 3.3 also moves.

7 Likes
3

:+1: on dropping old OSes.

I think the only concern I’d have (as a user) is that 3.0 deprecates Ubuntu 18.04 and 3.1 already drops it. Yes, 20.04 has been supported since 2.5, but we didn’t announce a deprecation and it feels wrong to do it in one release cycle then.

4

May I chip in here something from the user perspective on deprecating EL7: please leave more time. Much more time.

Two reasons:

  1. As Katello user, the upgrade from katello 3 to 4, i.e. migration to pulp3 was very painful. It took me a couple of months and several workaround, removing products, etc. to get my foreman/katello servers to katello 4 and there are still lots of issues open until it will be as good as it was with 3.18/pulp2. My Foreman servers and proxies all run on CentOS 7 at the moment. Several times, I have considered reinstalling from scratch but didn’t because of the amount of configuration involved to get everything as it was.

So what am saying: after months of painful migration attempts I would find it quite discerning if you tell me know, that in 9-12 months from now you have to move to EL8 (whatever way this may be and however well this migration may work).

Give users some time to get their systems to katello 4, mostly bug-free, and let them run the system smoothly without major administration for at least a year (so well into 2023). It’s just exhausting when it feels like you spend more time on getting something to run, keep it running, keep it updated, than to actually really use it and explore all the usage options you have.

  1. We actually haven’t really decided what comes after CentOS Linux 8. We are still evaluating and monitoring what’s out there and trying to figure out what’s the future for us. Alma, Rocky, CentOS Stream, Oracle Linux, even considering getting RHEL 8 licenses for some hosts until we are sure we have a new stable free platform we are confident about. It’s just my impression but I think this consolidation will go well into 2022 and I am worried I might have to pick an EL8 distro for my new foreman/katello because of the deprecation and may just regret it in the long run, because eventually we have focused on another distro.

So again, don’t tell people now they have to move to EL8 in these times of uncertainty. Give them at least 2022 to figure out which EL8 distro it’s going to be for them before deprecating EL7 support.

On a sidenote (or a third reason if you want): I generally really prefer products/software which in regard to distro support follow the EOL of the distro. I always now the distros and version and their EOL on my hosts. And for many of our servers we use the RHEL derivates to be able to keep them running for many years until EOL. We do this in particular so that we don’t necessarily have to go through every major version. We had quite a lot of hosts running CentOS 6 which we then replaced with CentOS 8 (before they have announced it’s EOL, too).

So my general expectation is that a product officially supports RHEL7/CentOS 7/EL7 would do so until EOL of EL7. That’s why we chose EL to begin with, because it’s supported for many years. If a product before that announces it’s deprecation of a still supported operating system, it’s quite disappointing and I would expect that quite a few people out there would rather keep their Foreman/Katello system running on EL7 with the latest support version then forcing a migration before time (if there are not too many bugs which make it hard to use Foreman/Katello properly).

And a final sidenote: there must be a well working migration process to move the existing server to a new installation on EL8. After the painful pulp migration I really don’t want to spend months again on a EL8 migration. Because otherwise, I’d start thinking if I had known that in the beginning before katello 4, it would have been much better to installation the whole thing from scratch instead of trying to migration first pulp and then to el8…

5 Likes
5

That’s why I wanted to start the discussion sooner rather than later.

I still think we should start deprecating EL7 in the release notes for every release starting 3.0, even if there is no exact end date yet. The same should be done in the installation instructions. This should achieve that new installations happen on EL8. It also makes admins aware that EL8 should become the preferred platform. Even if their Foreman isn’t migrated, they sometimes do install new Smart Proxies and they should be deployed on EL8.

As for the exact date, I can be persuaded that we should see where the dust settles after CentOS Linux 8 is EOL.

What I should have mentioned in the post is that we do have EL9 on the horizon and I would much rather spend my time supporting that than EL7. There are also CI constraints.

6

I now realize I may have been unclear, but this is support to run Foreman + Foreman Proxy (and plugins) on those platforms. With my installer hat on I focused on that. Support for managing EL7, Debian 10 and Ubuntu 18.04 using Foreman will remain supported.

7

I opened changes to the release notes for 3.0 and nightly that reflect these deprecations.

It leaves out any Debian changes since we need to put in effort before making that happen.

I’m not sure my phrasing there is the best it can be so please have a look.

8

Given we didn’t manage to get Debian 11 for 3.1, I am proposing to update the table to read:

  • 3.2 adds Debian 11
  • 3.4 drops Debian 10
1 Like
9

It looks like Foreman 3.2 can ship Debian 11.

Branching is very close. That makes me ask: do we still want to continue with the plan and drop support for running Foreman on EL7 by version 3.3?

10

In the release notes, we tell folks that they should start thinking about planning a migration but I think the current state is that they should be thinking about a reinstall and repointing clients, no?

Assuming no easy migration/upgrade path then: if we drop support for 7.x in 3.3, I would guess a lot of people are going to sit on the upgrade until Foreman is running well on 9.x and then make the effort to reinstall. That could potentially only be 12-15 months away.

It might mean lower uptake for 3.4 and 3.5 and therefore less testing for those versions.

11

That is a good point. Clear upgrade instructions should be provided. @evgeni has been working on an upgrade using leapp:

AFAIK he’s at least tested with CentOS 7 → CentOS 8 Stream once, but I’ll let him correct me if I’m wrong.

1 Like
12

That’s correct, the idea is that the upgrade should be as easy as

# yum install leapp
# leapp preupgrade
# leapp upgrade
# reboot
13

As I have pointed out before and also in EL7 deprecation it all may look much easier if you only have a foreman installation without katello. (although even there I have a couple of puppet parameters which I pass per host group or host which need to be recreated…)

But with katello, I see a lot of additional configuration until everything is set up. The initial setup includes lots of products and repositories to be set up. For each subscribed host I basically have to assume a unique set of repositories enabled. And as far as I can see I cannot simply point a client to a new katello server and it automatically subscribes to all the products it had and enables/disables all the repositories it has.

So to me, a working migration path is essential. Because otherwise, I really do have to start all other again, going through all the settings for all the hosts and recreate everything.

To me, there seem to be two possible ways:

  1. backup and restore. Backup all configuration and restore them into a new foreman/katello installation. The synced pulp content doesn’t have to be migrated. I can live with that being downloaded again (even though it’s close to 1 TB by now). Of course, some intelligent migration for the pulp content would be nice (i.e. the new server doing the initial download from the old server or mounting the old pulp partition on the new server to copy the content).

  2. Extraction of the complete configuration into an ansible playbook to install the new server. That would be the ideal solution to me, because that way I could get into the ansible installation of foreman/katello without starting at scratch.

I think, EL 7 should only be deprecated after the backup/restore process is fully and well tested, so that I can use that to do the migration. No. 2 is far away, if it ever gets there.

Pulp2-3 migration was a major operation for me which took me months until I had s state (with a couple of workaround) to get everything migrated.

Now with katello 4.3 officially out I have to do the puppet module extraction and hope everything will still work after the upgrade. I really do hope that it’s not as difficult as the pulp migration, but because of the problems with the pulp migration and waited with the upgrade from 4.1 as long as possible. I hope it goes smooth but I am worried it may take me again a while to get it properly working again.

If migration is again another pain like the pulp migration, I really would defer any further upgrades until I set up a new server on EL9, because I don’t really don’t want to migrate now from EL7 to EL8 and in two years again from EL8 to EL9. When EL9 is out I would set up a new server on EL9 and start learning how to set up foreman/katello with ansible (which I have never used before).

And I assume many others will do so, too. After all the work and pain the past upgrades have caused you’d rather postpone the updates.

So please, please, give a well working migration plan on how to backup/restore everything before deprecating EL7. And generally, I always think it would be much better, if life time of a product follows the life time of the supported operating system. I wonder what RedHat does with satellite? They have to support satellite until EL7 EOL, don’t they?

1 Like
14
  1. backup and restore is tested and supported. if it doesn’t work for you it’s a bug we have to fix.
  2. the migration using leapp (which you didn’t list) upgrades your system in-place. no data movement whatsoever.
  3. having everything in ansible is great, but extracting that data after the fact is way to complicated.
1 Like
15

I’m also in the distressed to see EL7 support being removed camp.

CentOS 8 not being a thing has left loads of people stuck on CentOS 7 until either the Rocky/AlmaLinux situation settles or their companies stump up the money for RHEL licensing.

16

Red Hat Satellite 7.0 will be the last version to support RHEL7. EL7 is currently in maintenance support 2, which means no new features.

From a developer perspective it’s more and more common to run into limitations. I run into systemd limitations most often and those won’t get fixed, but also more fundamental issues where rpm is too old or various packaging macros. The Apache is too old to support the event MPM, which is supposed to be more efficient. Packaging go on EL7 is also painful and software is starting to show up which uses this.

So the cost of supporting EL7 is going up and at least from the Red Hat perspective there’s no need for it anymore. That means a serious community of people needs to stand up to make sure it continues to work. Time that IMHO is better spent on making sure future versions work well.

17

I don’t know if this has been mentioned, but apparently more recent versions of ansible do not work on RHEL7. No plan to fix that either, AFAIK.

Various dependencies just don’t build.

So, I think RHEL7 is pretty much as dead as it can be, even from the RedHat perspective.
And the only reason people are sour about this is the CentOS 8 “situation”. Else everybody would have moved on.

1 Like
18

O.K. That’s probably what I have to do then. I really do hope it works. It just took me 2 months, various upgrade attempts, countless debugging hours, to upgrade from 4.2 to 4.3, just discovering at least two more issues/bugs.

So considering that, I am worried that the backup/restore project takes me a year until it actually works. Even the docs are extremely limited on how to restore on a different system:

Restoring Foreman server or Smart Proxy server from a Backup

If the original system is unavailable, provision a system with the same configuration settings and host name.

So what are those “same configuration settings”? And using the same hostname is kind of difficult as I cannot really shutdown everything for a longer time…

Just thinking about how I can set up the new system in parallel to the old one in production causes me headache, e.g. hostnames and names in certificates etc. I also have 1 TB for /var/lib/pulp. As far as I understand the backup process, it will backup the content of that and then restore it. So I have 1 TB on the old server, 1 TB in the backup, and 1 TB on the destination. And everything is going to be copied twice. For content which is 99,9% available online anyway. --skip-pulp-content is only for debugging purposes, so it seems copying the pulp content manually before or even simply using cloning the vdi and attaching it to the new server isn’t an option.

So I am really worried that this is going to be a major operation and in the end I will only wonder if it hadn’t be better simply to start all over again. I have 31 products, ~120 repositories, 4 content views, 275 hosts with parameters set, individual subscriptions and enabled repositories sets. Manually setting this up on a new server takes a lot of time, but if backup/restore would take longer in the end, causing much more problems it might be worth it… Of course, it’s futile to assume how long something will really take…

I don’t really like that. Over the time foreman collected so much garbage on the system that I really don’t want to convert the system in-place to CentOS 8. If I run yum autoremote it wants to remove more than 200 packages. I don’t want to keep all that garbage and I don’t feel comfortable what is going to happen with this after the conversion…

Well, extracting all the data manually is more complicated… Using hammer to extract the information about my 120 repositories, converting it into a form that ansible get take. I mean it’s only a limited amount of information that there is.

And if I simply set up a new system I have to get this information out of the old system anyway, even if it would be a simple click/copy/paste…

So either way, this is already causing me headache and sleepless nights, and I really don’t know what’s the best way to go without wasting too much time…

19

Another note on migration options: using ansible isn’t really fully working at the moment either, at least if you want to setup a katello server: it seems even the latest version of the ansible collection is still wants to configure the mirror_on_sync option and also has problems with empty http_proxy_id and ssl_*_id options. During each run it wants to remove the latter even though they are set to null anyway.

So my original plan with running the same playbook against my old and new foreman server, verifying it doesn’t make any changes on the old server and then deploying everything on the new server, doesn’t run so smoothly. Each run I have to manually check what it says needs to be changed and check that it’s only the repositories which are incorrectly flagged…

So now I need to patch the theforeman.foreman collection to make it properly work to be used in a migration…

20

For developers, users of our nightly packages and our Puppet modules, on June 6th we will begin the process of removing EL7 support across the ecosystem for nightly and thus developer setups.

1 Like
21

Is there a documented in-place upgrade path for RHEL7? The leapp method here mentions it won’t work on RHEL7.

If not, is there a tested and working way to export the data (without pulp/packages) from Foreman/Katello installed on RHEL7 to a fresh install on RHEL8 or even RHEL9? I have the same concerns as post #18 above.