Katello 4.1 and Rocky Linux 8 - not ready for prime time yet it seems

I don’t think this is related to Rocky Linux. I see the same on CentOS 8:

# dnf check-update
Updating Subscription Management repositories.
CentOS-8 - PowerTools                            95 kB/s | 2.6 kB     00:00
Extra Packages for Enterprise Linux 8 - x86_64   84 kB/s | 2.3 kB     00:00
CentOS-8 - AppStream                            113 kB/s | 2.6 kB     00:00
CentOS-8 - Base                                  91 kB/s | 2.3 kB     00:00
Puppet 6 Repository el 8 - x86_64                87 kB/s | 2.0 kB     00:00
ORG el8                                         84 kB/s | 2.0 kB     00:00    
Foreman client 2.5 el8                           91 kB/s | 2.3 kB     00:00

kernel.x86_64               4.18.0-305.10.2.el8_4     ORG_centos8_BaseOS_x86_64
kernel-core.x86_64          4.18.0-305.10.2.el8_4     ORG_centos8_BaseOS_x86_64
kernel-modules.x86_64       4.18.0-305.10.2.el8_4     ORG_centos8_BaseOS_x86_64
kernel-tools.x86_64         4.18.0-305.10.2.el8_4     ORG_centos8_BaseOS_x86_64
kernel-tools-libs.x86_64    4.18.0-305.10.2.el8_4     ORG_centos8_BaseOS_x86_64
python3-perf.x86_64         4.18.0-305.10.2.el8_4     ORG_centos8_BaseOS_x86_64
systemd.x86_64              239-45.el8_4.2            ORG_centos8_BaseOS_x86_64
systemd-libs.x86_64         239-45.el8_4.2            ORG_centos8_BaseOS_x86_64
systemd-pam.x86_64          239-45.el8_4.2            ORG_centos8_BaseOS_x86_64
systemd-udev.x86_64         239-45.el8_4.2            ORG_centos8_BaseOS_x86_64

Update:

# dnf update
...
Error: 
 Problem: The operation would result in removing the following protected packages: systemd, systemd-udev
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

with --nobest:

# dnf update  --nobest
...
 Problem: The operation would result in removing the following protected packages: systemd, systemd-udev
================================================================================
 Package         Arch   Version               Repository                   Size
================================================================================
Installing:
 kernel          x86_64 4.18.0-305.10.2.el8_4 ORG_centos8_BaseOS_x86_64  5.9 M
 kernel-core     x86_64 4.18.0-305.10.2.el8_4 ORG_centos8_BaseOS_x86_64   36 M
 kernel-modules  x86_64 4.18.0-305.10.2.el8_4 ORG_centos8_BaseOS_x86_64   28 M
Upgrading:
 kernel-tools    x86_64 4.18.0-305.10.2.el8_4 ORG_centos8_BaseOS_x86_64  6.1 M
 kernel-tools-libs
                 x86_64 4.18.0-305.10.2.el8_4 ORG_centos8_BaseOS_x86_64  5.9 M
 python3-perf    x86_64 4.18.0-305.10.2.el8_4 ORG_centos8_BaseOS_x86_64  6.0 M
Removing:
 kernel          x86_64 4.18.0-240.22.1.el8_3 @ORG_centos8_BaseOS_x86_64   0
 kernel-core     x86_64 4.18.0-240.22.1.el8_3 @ORG_centos8_BaseOS_x86_64  62 M
 kernel-modules  x86_64 4.18.0-240.22.1.el8_3 @ORG_centos8_BaseOS_x86_64  21 M
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 systemd         x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  3.6 M
 systemd         x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  3.6 M
 systemd-libs    x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  1.1 M
 systemd-libs    x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  1.1 M
 systemd-pam     x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  468 k
 systemd-pam     x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  469 k
 systemd-udev    x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  1.4 M
 systemd-udev    x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  1.4 M

Transaction Summary
================================================================================
Install  3 Packages
Upgrade  3 Packages
Remove   3 Packages
Skip     8 Packages

Total download size: 88 M
Is this ok [y/N]: n

Enabling the CentOS Repo works, too:

# dnf update --enablerepo=baseos
...
Dependencies resolved.
================================================================================
 Package         Arch   Version               Repository                   Size
================================================================================
Installing:
 kernel          x86_64 4.18.0-305.10.2.el8_4 baseos                      5.9 M
 kernel-core     x86_64 4.18.0-305.10.2.el8_4 baseos                       36 M
 kernel-modules  x86_64 4.18.0-305.10.2.el8_4 baseos                       28 M
Upgrading:
 kernel-tools    x86_64 4.18.0-305.10.2.el8_4 baseos                      6.1 M
 kernel-tools-libs
                 x86_64 4.18.0-305.10.2.el8_4 baseos                      5.9 M
 python3-perf    x86_64 4.18.0-305.10.2.el8_4 baseos                      6.0 M
 systemd         x86_64 239-45.el8_4.2        baseos                      3.6 M
 systemd-libs    x86_64 239-45.el8_4.2        baseos                      1.1 M
 systemd-pam     x86_64 239-45.el8_4.2        baseos                      469 k
 systemd-udev    x86_64 239-45.el8_4.2        baseos                      1.4 M
Removing:
 kernel          x86_64 4.18.0-240.22.1.el8_3 @ORG_centos8_BaseOS_x86_64   0 
 kernel-core     x86_64 4.18.0-240.22.1.el8_3 @ORG_centos8_BaseOS_x86_64  62 M
 kernel-modules  x86_64 4.18.0-240.22.1.el8_3 @ORG_centos8_BaseOS_x86_64  21 M

Transaction Summary
================================================================================
Install  3 Packages
Upgrade  7 Packages
Remove   3 Packages

Total download size: 94 M
Is this ok [y/N]: n

Well, I posted that it was Rocky 8 specifically because of course, I would like to see the OS “Supported” (and not reflected as “Unknown”…). And also, that’s the only OS I have loaded into my Katello 4 as of this morning.

It’s actually good to know that this appears to simply be an “8-based” issue (CentOS 8, and MAYBE RedHat 8 also?)-- there are all sorts of issues out there due to the structual changes in version 8 of the OS, most commonly things to do with module support.

For example, there’s an issue where Katello shows updates, but they’re not actually applicable due to Perl module(s) chosen and so on. I totally get that it’s really complicated.

Seeing that this is a larger issue (and that you have seen it too, with CentOS 8), maybe it’ll possibly get addressed sooner than I thought which is good! Is there another thread on the problem? I’d like to follow along if there is.

Meanwhile I am continuing with my testing of all things Katello 4.1-- today Im adding in all my centos7 stuff etc.

Thank you for confirming “I’m seeing this too”. I am happy I am not alone :slight_smile:

Oh, and in my situation (Rocky 8, which is of course a centos CLONE), enabling the rocky8 repos works as well, exactly in your example above with CentOS.

Damn. I wish I could edit the subject line of my post.

Can you check the Tasks to see if you have got an error syncing the baseos repository?

See my topic CentOS 8.4 BaseOS Sync error

I don’t know if it’s a coincidence or not, but the “broken” packages must have arrived during that sync which resulted in a warning…

I don’t see the same error.

Can you check

# dnf provides /usr/bin/kernel-install

I guess you’ll see systemd-udev-239-45.el8_4.1.x86_64 listed twice and systemd-udev-239-45.el8_4.2.x86_64 is missing…

Compare Cannot update puppet-agent to 6.24.0 on foreman servers

Looks like a bigger problem to me, not even related only to EL8…

I only see one listed, which is interesting. But I still have the cannot update systemd thing going on.

[root@butler yum.repos.d]# dnf provides /usr/bin/kernel-install
Updating Subscription Management repositories.
systemd-udev-239-45.el8.x86_64 : Rule-based device node and kernel event manager
Repo        : @System
Matched from:
Filename    : /usr/bin/kernel-install

O.K. In your case the available update is 239-45.el8_4.1. The provides for the update are missing completely for you…

I have analyzed some more in my puppet-agent topic. Maybe you are able to reproduce my findings which I have posted there…

@gvde Found the bug, link is here:

https://bugzilla.redhat.com/show_bug.cgi?id=1984125

He hit this when trying to do an update that involved puppet, I hit it when updating Rocky Linux 8.4 and it barfed on systemd dependencies. This affects Centos8 as well as a client to Katello 4.

It worse: It affects all new packages any RPM repository after the upgrade to Katello 4.1.1 with pulpcore 3.14. 4.1.0 was still O.K.

It only becomes an issue on the client if you have any package installed which has a RPM requires condition for one of the files in the new package.

That’s why you cannot update systemd-udev because it provides /usr/bin/kernel-install and that information is now missing due to this bug…

Well in my case my Katello 4.x server is simply a newer test box (that I will EVENTUALLY migrate everything to), but My 3.18 Katello server is still my production environment for 99% of my hosts. So I am lucky that I can (and will) just… wait.

But Good Lord, what a nasty bug. I feel bad for any Production 4.1+ system out there.

I also ran into that problem today :slight_smile:
Fortunately it looks like there is already a upstream fix for that for 3 days now, so heads up!

Found the second discussion as well CentOS 8.4 BaseOS Sync error

1 Like

Yes, you’ve found the correct pulp bug! Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp

This will be fixed in pulp-rpm 3.14.0 which we’re working on getting pushed out ASAP. however due to the fact that the data in the db is incorrect, a repair process is needed and is detailed here: Activity - RPM Support - Pulp

If you’re wanting to do the repair you’ll want to wait until you’re able to upgrade to python3-pulp-rpm-3.14.0

2 Likes

Upon returning from a week vacation; I did the update, ran the repair script, republished repo metadata, generated new content views.

My rocky 8 client updated properly this time. Woo!

I still have issues with modules and rpm metadata as to applicability, but that’s a different known issue and is slowely being worked on. THIS issue which was pretty serious is now good to go from my point of view!

1 Like

I am on CentOS 8 and Foreman 2.5.2 / Katello 4.1.2 and same problems here.

I ran the repair script:

Package metadata repaired: 10663/10663 ...FINISHED

This is test environment and the problem occured at initial sync so all the packages were broken. I think script worked ok because subsequent run reported 0/0 packages repaired.
I then synced all repositories and published new content view version. However nothing has changed. My test client still reports same conflict:

[root@foremanclienttest1 ~]# dnf update
Updating Subscription Management repositories.
CentOS 8 AppStream                                                                                                                                                                                            27 kB/s | 2.6 kB     00:00
CentOS 8 CentOSPlus                                                                                                                                                                                           23 kB/s | 2.0 kB     00:00
CentOS 8 Extras                                                                                                                                                                                               21 kB/s | 2.0 kB     00:00
CentOS 8 BaseOS                                                                                                                                                                                               25 kB/s | 2.3 kB     00:00
CentOS 8 PowerTools                                                                                                                                                                                           30 kB/s | 2.6 kB     00:00
Error:
 Problem: The operation would result in removing the following protected packages: systemd-udev
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Could you please share exact steps you take? Did you update pulp-rpm, I assume it is not mandatory if I run repair script after repository sync?
Btw, my repositories are not local, they are created with --download-policy 'on_demand' if it makes any difference?

Are you sure your client is using the correct content view and lifecycle environment and you have published and promoted the latest version to that?

Check the file list on the foreman server: go to the Content - Packages page. Search for “systemd-udev-239-45.el8_4.2.x86_64”. Click on the package (or each if you have multiple EL 8 derivates) and check the Files tab. It should list the files in the package:

Files in package systemd-udev-239-45.el8_4.2.x86_64

    /etc/kernel
    /etc/kernel/install.d
    /etc/modules-load.d
    /etc/udev
    /etc/udev/hwdb.bin 
...

Also remember, clients cache metadata. Run “dnf clean all” and then “dnf update” again.

If it is still missing, run

# dnf provides /usr/bin/kernel-install

What packages are listed?

Thanks for fast response.
Content → Packages listed:

### Files in package systemd-udev-239-45.el8_4.2.x86_64

* /etc/kernel
* /etc/kernel/install.d
* /etc/modules-load.d
* /etc/udev
* /etc/udev/hwdb.bin
* /etc/udev/hwdb.d
...

So I think its correct.
I have already ran “dnf clean all” and “dnf update” several times.

[root@foremanclienttest1 ~]# dnf provides /usr/bin/kernel-install
Updating Subscription Management repositories.
Last metadata expiration check: 1:07:08 ago on Tue 03 Aug 2021 01:32:15 PM EEST.
systemd-udev-239-45.el8.x86_64 : Rule-based device node and kernel event manager
Repo        : @System
Matched from:
Filename    : /usr/bin/kernel-install

I am somewhat new to Foreman/Katello, but I have double checked content views and lifecycle environment and I think all is correct.

You can run “dnf repolist -v”. It’ll show you the full baseurl for each repositories. It contains the environment and the content view labels…

dnf provides doesn’t find the new package version although in katello it shows the correct filelist. Did you republish the repository metadata for each repository after the repair? Or did you do an advanced, complete sync of each repository?

Go to the repository page for the baseos repository (Content - Products - <pick or CentOS 8 product> - on the Repositories tab click on the BaseOS repository) and from the “Select Action” dropdown select “Republish Repository Metadata”.

If you client is in the Library Environment on the “Default Organizational View” it should be able to pick up the new metadata. For a content view, I think you’ll have to publish and promote a new version. After a “dnf clean all” it should download the correct metadata…

“dnf repolist -v” shows correct url and content view.

I have somehow missed the metadata republishing step. After repository metadata republish and publishing new CV version it is working now. Thank you for your help.