Running Foreman on CentOS 8 Stream support

Community
1

Today it’s not possible to run Foreman on CentOS 8 Stream without some additional work. At least Bug #31527: foreman-installer fails to configure Postgres on CentOS Stream 8 - Installer - Foreman is a known blocker. The short summary of that is that the Apache and PostgreSQL Puppet modules assume EL8 has a minor version but stream doesn’t.

Untested, but I think the workaround is adding the following to custom-hiera.yaml:

postgresql::globals::version: 10

I don’t expect to fix this in time for Foreman 2.4. It’s also complicated by the fact that there are no vagrant boxes nor containers which makes automated testing harder.

1 Like
2

I discovered something else giving this a spin earlier: the installer is looking for mod_auth_kerb which apparently doesn’t exist any more.

3

Came by to drop the log, maybe this is helpful…

2021-01-17 14:03:30 [ERROR ] [configure] Execution of '/bin/dnf -d 0 -e 1 -y install mod_auth_kerb' returned 1: Error: Unable to find a match: mod_auth_kerb
2021-01-17 14:03:30 [ERROR ] [configure] /Stage[main]/Apache::Mod::Auth_kerb/Apache::Mod[auth_kerb]/Package[mod_auth_kerb]/ensure: change from 'purged' to 'present' failed: Execution of '/bin/dnf -d 0 -e 1 -y install mod_auth_kerb' returned 1: Error: Unable to find a match: mod_auth_kerb
1 Like
4

It looks like the package is also unavailable in CentOS 8 so it isn’t a Stream specific issue. Would you mind opening a separate issue for that at Overview - Installer - Foreman so we can get it fixed? Personally I don’t know Kerberos so I’d need some guidance on the exact replacement.

1 Like
5

A small update. Recently the project started to publish Vagrant boxes for stream and we just merged a PR to forklift which makes spinning up a box easier:

Now you can run vagrant up centos8-stream-foreman-nightly in forklift. It did reveal some SELinux issue which we don’t see in regular CentOS 8. I don’t quite understand it:

type=AVC msg=audit(1614873402.173:1566): avc:  denied  { getattr } for  pid=30429 comm="httpd" path="/etc/puppetlabs/puppet/ssl/certs/centos8-stream-foreman-nightly.wisse.example.com.pem" dev="vda1" ino=33568393 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:puppet_etc_t:s0 tclass=file permissive=0

However, we have this line in our policy which should allow it:

It may be some breakage due to newer packages and in that case Stream is doing exactly what it should do: show breakages before they hit releases.

6

I did a POC and have now given up on CentOS 8 Stream as a replacement for Centos.

There is a HUGE deficiency in CentOS 8 Stream that I am not comfortable with at all. The repos that Red Hat is providing for CentOS 8 Stream only have the CURRENT version of every RPM. There is no previous version or additional version of rpms in the repos and when an RPM is updated, the old ones disappear. This is totally different behavior from most useful Linux distros, including RHEL and Centos.

a) This makes the Foreman/Katello “On Demand” feature worthless, because you find that a system may go to request an RPM version that no longer exists in the upstream repos, because it might have been updated yesterday and now the Katello wants to download on demand is gone.

b) This makes it extremely hard to do patch management and even plan to do monthly updates.

c) The repos are really only useful for doing installs, and not for updates. My guess is that Red Hat does not want people actually using Stream as an OS that can be updated.

d) This is just asking for RPM dependency hell, and systems which will be impossible to manage and update.

When asked about it, the answer is that is the plan and the repos will be like that. This tells me that Red Hat is not serious about CentOS 8 Stream actually being useful to anyone other than for installing, testing and destroying as a beta test of RHEL. I am moving on from them and looking at other solutions. I am guessing there will be almost zero actual users of Centos Stream in community once people fight with these issues and give up.

7

Turns out this is indeed the case. In EL 8.4 there will be a change and @lzap provided a patch that should also work on < 8.4:

8

I can’t answer these since I’m not responsible for Stream itself. They sound like concerns that should be taken to CentOS. Our goal right now is to make sure we can at least install and run Foreman on it.

However, it also sounds like Katello with immediate sync can help make it more usable for day-to-day users.

Btw, isn’t EPEL also a latest-version-only repository?

1 Like
9

This needs to be resolved, that’s pretty non-standard even Fedora keeps several versions. Probably some overlook, file a CentOS bug please.

Can you link that conversation?

It’s fair to say that SELinux core policy breaking changes happened probably three times for EL7 version. We would hit the bug not because we were on Stream, but because SELinux team decided to make this breaking change.

10

Yeah I can confirm:

http://ftp.cvut.cz/centos/8/AppStream/x86_64/os/Packages/

http://ftp.cvut.cz/centos/8/BaseOS/x86_64/os/Packages/

Compare it to

http://ftp.cvut.cz/centos/8-stream/BaseOS/x86_64/os/Packages/

http://ftp.cvut.cz/centos/8-stream/AppStream/x86_64/os/Packages/

where those older versions are missing. The difference is in the path, 8 vs 8-stream.

1 Like
11

For the record, I emailed with Stephen Smoogen from Fedora IT and he told me that Fedora mirrors and repodata does not keep multiple packages in updates and hasn’t for a while (like 3 or 4 years?). CentOS-8 also does not keep them. There are some technical and hardware resource reasons. This has nothing to do with the CentOS 8 Stream transition.

Anyway, it looks like it depends on the mirror you use, some administrators can decide to keep those versions if they want. In my case, CVUT mirror does keep some versions for CentOS 8 but not for CentOS Stream. You should probably talk to mirror administrators rather than Fedora or CentOS guys. This might have something to do with them thinking that Stream is kind of a “testing ground” and it does not make much sense to keep those versions while in fact it actually does much sense to keep those as the metadata will be getting invalidated more often.

For a small-scale user of Foreman/Katello/Pulp it looks like the only reliable option is to synchronize all packages which might create some unnecessary network load but I guess it is what it is. We should probably consider making the immediate download policy the default choice for CentOS 8.

2 Likes
12

Correction from Stephen: “Fedora does not keep multiple packages but CentOS-8 does.” This also aligns with our findings.

So Stream is deviation from CentOS, but I was told that CentOS devs are looking into this issue and the plan is to keep N and N-1 packages in repos. If you want to hear more details, please ask on the devel list.

2 Likes
13

I can confirm can be reproduced on CentOS 8 and CentOS Stream:
https://projects.theforeman.org/issues/32352