Cannot update puppet-agent to 6.24.0 on foreman servers

Katello 4.1.1, Foreman 2.5.2 on CentOS 7.9

puppet-agent 6.24.0 just became available. I was able to update on all my clients which connect to my content proxy.

Only my foreman servers (main server and smart proxies) refuse to update. Unlike the clients, they connect to the main server not to the content proxy.

Resolving Dependencies
--> Running transaction check
---> Package puppet-agent.x86_64 0:6.23.0-1.el7 will be updated
---> Package puppet-agent.x86_64 0:6.24.0-1.el7 will be an update
---> Package puppetserver.noarch 0:6.16.0-1.el7 will be updated
---> Package puppetserver.noarch 0:6.16.1-1.el7 will be an update
--> Processing Dependency: /opt/puppetlabs/puppet/bin/ruby for package: 1:foreman-installer-2.5.2-3.el7.noarch
--> Finished Dependency Resolution
Error: Package: 1:foreman-installer-2.5.2-3.el7.noarch (@DKRZ_foreman_2_5_el7_x86_64)
           Requires: /opt/puppetlabs/puppet/bin/ruby
           Removing: puppet-agent-6.23.0-1.el7.x86_64 (@DKRZ_puppet_puppet6_el7_x86_64)
               Not found
           Updated By: puppet-agent-6.24.0-1.el7.x86_64 (DKRZ_puppet_puppet6_el7_x86_64)
               Not found
           Available: puppet-agent-6.0.0-1.el7.x86_64 (DKRZ_puppet_puppet6_el7_x86_64)
               Not found
...

On the foreman servers it doesnā€™t know about the dependency:

# yum provides /opt/puppetlabs/puppet/bin/ruby
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, tracer_upload
Loading mirror speeds from cached hostfile
...
puppet-agent-6.22.1-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.23.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.23.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : @DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby

Curious enough, I really shows 6.2.30-0.1 twice, there.

On my clients, it is correct:

# yum provides /opt/puppetlabs/puppet/bin/ruby
...
puppet-agent-6.22.1-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.23.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.24.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : @DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby

Each version only once and it includes 6.24.0-1. Looks correct to me.

So it kind of looks as if the main server presents the new rpm as puppet-agent-6.23.0-1.el7.x86_64 for provides causing it to be listed twice and breaking dependencies?

O.K. There must be some serious issue here. I see the same problem now with systemd-udev on CentOS 8:

# dnf provides /usr/bin/kernel-install
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:14 ago on Wed 21 Jul 2021 04:59:33 PM CEST.
systemd-udev-239-45.el8.x86_64 : Rule-based device node and kernel event manager
Repo        : ORG_centos8_BaseOS_x86_64
Matched from:
Filename    : /usr/bin/kernel-install

systemd-udev-239-45.el8_4.1.x86_64 : Rule-based device node and kernel event manager
Repo        : @System
Matched from:
Filename    : /usr/bin/kernel-install

systemd-udev-239-45.el8_4.1.x86_64 : Rule-based device node and kernel event manager
Repo        : ORG_centos8_BaseOS_x86_64
Matched from:
Filename    : /usr/bin/kernel-install

This blocks the update to the latest CentOS 8 kernel and systemd. Rocky Linux sees the same. See Katello 4.1 and Rocky Linux 8 - not ready for prime time yet it seems

Maybe also related with the error here: CentOS 8.4 BaseOS Sync error

The problem with puppet-agent only affects my foreman servers which are connected to the main foreman server.

The systemd-udev issue is on my clients which are connected to my (single) foreman content proxy.

I have downloaded the filelists.xml.gz file for the baseos from the content proxy and looked into it, searching for /usr/bin/kernel-install:

...
<package pkgid="c4b34bb4476f7a7dac5cb7adbd1fa710a8fb960a66035a1947c1c0ea7f3187c4" name="systemd-udev" arch="x86_64">
  <version epoch="0" ver="239" rel="45.el8"/>
...
  <file>/usr/bin/kernel-install</file>
...
<package pkgid="22089981a3038158c83bdc693b7a39956c9992a041e88d91461430d1e4d64383" name="systemd-udev" arch="x86_64">
  <version epoch="0" ver="239" rel="45.el8_4.1"/>
...
  <file>/usr/bin/kernel-install</file>
...
<package pkgid="16124eda58e4b07ef18056a11bd20da165eb780c90a663baf99d14770547482f" name="systemd-udev" arch="x86_64">
  <version epoch="0" ver="239" rel="45.el8_4.2"/>
</package>

The complete file list for the latest version is missing.

primary.xml looks good, though:

<package type="rpm">
  <name>systemd-udev</name>
  <arch>x86_64</arch>
  <version epoch="0" ver="239" rel="45.el8_4.2"/>
  <checksum type="sha256" pkgid="YES">16124eda58e4b07ef18056a11bd20da165eb780c90a663baf99d14770547482f</checksum>
  <summary>Rule-based device node and kernel event manager</summary>
  <description>This package contains systemd-udev and the rules and hardware database
needed to manage device nodes. This package is necessary on physical
machines and in virtual machines, but not in containers.</description>
  <packager>CentOS Buildsys &lt;bugs@centos.org&gt;</packager>
  <url>http://www.freedesktop.org/wiki/Software/systemd</url>
  <time file="1626829839" build="1626829051"/>
  <size package="1436628" installed="8009345" archive="8049636"/>
  <location href="Packages/s/systemd-udev-239-45.el8_4.2.x86_64.rpm"/>
  <format>
    <rpm:license>LGPLv2+</rpm:license>
    <rpm:vendor>CentOS</rpm:vendor>
    <rpm:group>Unspecified</rpm:group>
    <rpm:buildhost>x86-01.mbox.centos.org</rpm:buildhost>
    <rpm:sourcerpm>systemd-239-45.el8_4.2.src.rpm</rpm:sourcerpm>
    <rpm:header-range start="5608" end="152592"/>
    <rpm:provides>
      <rpm:entry name="config(systemd-udev)" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="systemd-udev" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="systemd-udev(x86-64)" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="udev" flags="EQ" epoch="0" ver="239"/>
      <rpm:entry name="udev(x86-64)" flags="EQ" epoch="0" ver="239"/>
    </rpm:provides>
    <rpm:requires>
      <rpm:entry name="/bin/bash"/>
      <rpm:entry name="/bin/sh"/>
      <rpm:entry name="/bin/sh" pre="1"/>
      <rpm:entry name="/bin/sh"/>
      <rpm:entry name="grep" pre="1"/>
      <rpm:entry name="kmod" flags="GE" epoch="0" ver="18" rel="4"/>
      <rpm:entry name="libacl.so.1()(64bit)"/>
      <rpm:entry name="libacl.so.1(ACL_1.0)(64bit)"/>
      <rpm:entry name="libblkid.so.1()(64bit)"/>
      <rpm:entry name="libblkid.so.1(BLKID_2.15)(64bit)"/>
      <rpm:entry name="libblkid.so.1(BLKID_2.17)(64bit)"/>
      <rpm:entry name="libblkid.so.1(BLKID_2.18)(64bit)"/>
      <rpm:entry name="libcryptsetup.so.12()(64bit)"/>
      <rpm:entry name="libcryptsetup.so.12(CRYPTSETUP_2.0)(64bit)"/>
      <rpm:entry name="libgcc_s.so.1()(64bit)"/>
      <rpm:entry name="libgcc_s.so.1(GCC_3.0)(64bit)"/>
      <rpm:entry name="libgcc_s.so.1(GCC_3.3.1)(64bit)"/>
      <rpm:entry name="libkmod.so.2()(64bit)"/>
      <rpm:entry name="libkmod.so.2(LIBKMOD_5)(64bit)"/>
      <rpm:entry name="libpthread.so.0()(64bit)"/>
      <rpm:entry name="libpthread.so.0(GLIBC_2.2.5)(64bit)"/>
      <rpm:entry name="libsystemd-shared-239.so()(64bit)"/>
      <rpm:entry name="libsystemd-shared-239.so(SD_SHARED)(64bit)"/>
      <rpm:entry name="rtld(GNU_HASH)"/>
      <rpm:entry name="systemd" pre="1"/>
      <rpm:entry name="systemd"/>
      <rpm:entry name="systemd(x86-64)" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="libc.so.6(GLIBC_2.28)(64bit)"/>
    </rpm:requires>
    <rpm:obsoletes>
      <rpm:entry name="systemd" flags="LT" epoch="0" ver="229" rel="5"/>
      <rpm:entry name="udev" flags="LT" epoch="0" ver="183"/>
    </rpm:obsoletes>
    <rpm:recommends>
      <rpm:entry name="kbd"/>
    </rpm:recommends>
  </format>
</package>

As far as I can tell that is identical to what is on the CentOS mirror servers. So the problem seems to be the filelists.xml.

Iā€™ll go into the postgres database on the content proxy now if I can find anything thereā€¦

O.K. The file list is missing in the database as well. Itā€™s missing in the database on the content proxy as well as on the main server.

content proxy:

pulpcore=# select content_ptr_id, name, version, release, jsonb_array_length(files) from rpm_package where name = 'systemd-udev';
            content_ptr_id            |     name     | version |  release   | jsonb_array_length 
--------------------------------------+--------------+---------+------------+--------------------
 4346017c-e235-4bcd-89eb-6d3030529d8c | systemd-udev | 239     | 45.el8     |                252
 18c0981d-4094-4666-8dfb-cc73ddfb59a1 | systemd-udev | 239     | 45.el8_4.1 |                248
 1217bb8d-0851-45be-9319-9cd1ad5b2539 | systemd-udev | 239     | 45.el8_4.2 |                  0
(3 rows)

on the main server:

pulpcore=# select content_ptr_id, name, version, release, jsonb_array_length(files) from rpm_package where name = 'systemd-udev';
            content_ptr_id            |     name     | version |  release   | jsonb_array_length 
--------------------------------------+--------------+---------+------------+--------------------
 a5d9cfb8-a909-4daf-aa81-8b32b4c1a82f | systemd-udev | 239     | 45.el8     |                255
 0c1ec5b0-436a-405b-8e51-3da9f2d4b7fe | systemd-udev | 239     | 45.el8     |                252
 e77db7b3-9bbc-4269-9b77-ebbcd2774c51 | systemd-udev | 239     | 45.el8_4.1 |                248
 1153dc0b-f8cd-4a36-801c-704231f9ef49 | systemd-udev | 239     | 45.el8_4.1 |                248
 bc9e9c33-4140-424c-9d37-7e1e34912aa5 | systemd-udev | 239     | 45.el8_4.2 |                  0
 8d8ac124-c461-47fb-9734-afeff651ec28 | systemd-udev | 239     | 48.el8     |                250
(6 rows)

(The main server also has CentOS 8 Stream and AlmaLinux 8 which are not synced to the content proxy, thatā€™s why there are more availableā€¦)

So it seems as if the filelist for the package got lost somewhereā€¦

I check a few more packages: puppet-agent as it caused problems and also kernel-core for which an update is also available:

content proxy:

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where name = 'puppet-agent' and jsonb_array_length(files) = 0;
            content_ptr_id            | epoch |     name     | version | release |                              pkgId                               | jsonb_array_length 
--------------------------------------+-------+--------------+---------+---------+------------------------------------------------------------------+--------------------
 b8720eae-0ad3-4f80-98b7-5ada157a9153 | 0     | puppet-agent | 6.24.0  | 1.el8   | be42b9808718a935077b31390e31d294f87ae8e0607a2b534fe2cfa28018d6d5 |                  0
 d742286f-0114-4c1f-914b-8d7aacb6f2c4 | 0     | puppet-agent | 6.24.0  | 1.el7   | 80497802c6dd82891c8c5682a8ce7aa11c45c89ed080369ac295f058ef723ad2 |                  0
(2 rows)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'kernel-core';
            content_ptr_id            | epoch |    name     | version |    release     |                              pkgId                               | jsonb_array_length 
--------------------------------------+-------+-------------+---------+----------------+------------------------------------------------------------------+--------------------
 eb1b8389-1367-4b90-b2e7-8e0b2bb69040 | 0     | kernel-core | 4.18.0  | 305.10.2.el8_4 | 6ec68b3ea6e89936514d650f30c14a49c2162df59149ead31aa026686cbb467e |                  0
(1 row)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'filebeat';
            content_ptr_id            | epoch |   name   | version | release |                              pkgId                               | jsonb_array_length 
--------------------------------------+-------+----------+---------+---------+------------------------------------------------------------------+--------------------
 8c7f4b9f-f9eb-4f7d-b182-d5a18167fca3 | 0     | filebeat | 7.13.4  | 1       | 0d2f70d84dd361bb976c6dd2fb2aa5d0261595bc4c347cb6549ab9d9dec86a65 |                  0
 9dd9ccf7-18f7-4bb0-9ff3-cce47782dee4 | 0     | filebeat | 7.13.4  | 1       | e50aee18037b11610a10040a0f4bd264042d9c080746a51c211b04db79639417 |                  0
 aa246676-2fef-4a99-9bb4-0d6b29695dba | 0     | filebeat | 7.13.4  | 1       | 869b59700c922e910ff0c11eb6794f98b0fa132d6f316ab80a0c5fd8bc719d70 |                  0
(3 rows)

main server:

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where name = 'puppet-agent' and jsonb_array_length(files) = 0;
            content_ptr_id            | epoch |     name     | version | release |                  pkgId                   | jsonb_array_length 
--------------------------------------+-------+--------------+---------+---------+------------------------------------------+--------------------
 792b45f7-04c4-4de6-8d58-24ec5ced447f | 0     | puppet-agent | 6.24.0  | 1.el7   | 3bcf6e568a3ad9fb82b122d46b51f2e566588b5b |                  0
 e11548a2-1e19-4922-a881-56efff6a2d46 | 0     | puppet-agent | 6.24.0  | 1.el8   | 2f0ea8c6767c43308e84a030c80966fa1b84a3aa |                  0
(2 rows)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'kernel-core';
            content_ptr_id            | epoch |    name     | version |    release     |                              pkgId                               |
 jsonb_array_length 
--------------------------------------+-------+-------------+---------+----------------+------------------------------------------------------------------+
--------------------
 96342a34-e16b-47f0-aa75-cf802e55ef26 | 0     | kernel-core | 4.18.0  | 305.10.2.el8_4 | 6ec68b3ea6e89936514d650f30c14a49c2162df59149ead31aa026686cbb467e |
                  0
(1 row)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'filebeat';
            content_ptr_id            | epoch |   name   | version | release |                              pkgId                               | jsonb_arr
ay_length 
--------------------------------------+-------+----------+---------+---------+------------------------------------------------------------------+----------
----------
 b0e5b41e-b7f7-48f0-b2aa-0ce4ebee3daa | 0     | filebeat | 7.13.4  | 1       | 0d2f70d84dd361bb976c6dd2fb2aa5d0261595bc4c347cb6549ab9d9dec86a65 |          
        0
 a82f55fa-9d43-4da7-a4b1-b7d0a103f13e | 0     | filebeat | 7.13.4  | 1       | e50aee18037b11610a10040a0f4bd264042d9c080746a51c211b04db79639417 |          
        0
 60b2e3f3-a771-40f9-99a2-4e5675be7bb5 | 0     | filebeat | 7.13.4  | 1       | 869b59700c922e910ff0c11eb6794f98b0fa132d6f316ab80a0c5fd8bc719d70 |          
        0
(3 rows)

Those are all new package versions which have been available since today. So basically, the repository sync doesnā€™t import the filelist anymore leaving it empty which causes problems, if there are dependencies to one of those files from other packagesā€¦

Unfortunately, I cannot simply search for empty file lists in the database to find all broken entries as some packages really donā€™t contain any filesā€¦

I also already did a ā€œValidate Content Syncā€ on the CentOS 8 BaseOS repository which was successful but obviously didnā€™t fix those empty file listsā€¦

O.K. by checking core_content.pulp_created timestamp I was able to pinpoint the time when this problem started: all packages synced since my update from 4.1.0 to 4.1.1 are missing the file list. All rpm syncs. All rpm repositories.

@ehelms @iballou So I guess this is a pulpcore 3.14 issue, then?

I guess itā€˜s this pulp issue: https://pulp.plan.io/issues/9107

Hi again @gvde,

Ah good catch on the issue there. Itā€™s already got a Bugzilla too. It should be fixed soon then!

Do you have the bugzilla link?

I hope the fix will also correct the synced packages with the missing filelist in the database. A complete, validated repository sync doesnā€˜t seem to affect that partā€¦

Never mind: https://bugzilla.redhat.com/show_bug.cgi?id=1984125

I would voice that concern on the redmine issue. I imagine that would be fixed as well, but itā€™s good to check.

Go figure that I hit this bug the day I decide to light up Katello 4.1 and insert A centos-8-clone-like repository (in my case, Rocky Linux 8.4).

I was right in an early commment on a related post of mine that this was pulp. Ha!

I will comment in my post the link to the bug listed above by @gvde

I just wrote a message in pulp isssue #9107.

I looked at the code of the PR and it seems to me as if there are functions to validate the synced data in the database with the original metadata in the source repository. So if I am not mistaken that is there, itā€™s unclear to me if that works automatically during each sync or if that requires a special pulp repair call from katello like the validated complete syncā€¦

I quote the answer from the pulp issue here to have all the relevant information in this topic, too:

Most likely we will just write a script that will do an in-place repair - it hasnā€™t been decided yet whether Katello would do anything to run it automatically. My assumption is - probably not.

Weā€™ll try to get 3.14.0 released by the end of the day (Thursday), which should let Katello package it by the end of the week. And then once thatā€™s out weā€™ll evaluate the best way to address any existing issues.

O.K. There is hope hereā€¦ Wait until the pulp-rpm 3.14.0 package to get the correct data in for future syncs. Then run a in-place repair. And itā€™s hopefully all up to speed againā€¦

2 Likes

So when I eventually get the new pulp, whatā€™s the ā€œin-place repairā€ process?

It seems pulp_rpm 3.14.0 is out. So I guess next step would be foreman/katello packagingā€¦

@iballou @jjeffers There also seems to be a script available if all RPMs have been downloaded. I guess that means if you have download policy ā€œimmediateā€ and not ā€œon demandā€.

I hope itā€™ll soon gets into the katello repositories and we can get away from that broken version. By now, I have quite a few updates in the queue and I started doing updates from the centos mirrors because some are overdue for more than a week nowā€¦

Just wanted to let you all know that weā€™re in the process of testing out the Pulp teamā€™s fixing script. Iā€™m not sure how itā€™ll be delivered but weā€™ll keep you updated.

1 Like

Excellent. Well will the new pulp_rpm package be available? The sooner I stop syncing rpms with broken metadata into the system, the better. Repairing would be the second step for meā€¦

I wonder what the repair script does exactly. Can it find all RPMs with have broken metadata or does it simply go through everything and compares & repairs it if it is not matching with what it should be?

There is a packaging PR:

https://github.com/theforeman/pulpcore-packaging/pull/198

Is this going into 4.1.2? I really donā€™t quite understand where to look to see whatā€™s going into an upcoming releaseā€¦ If itā€™s going into 4.1.2 and I can expect 4.1.2 very soon, I would wait. Otherwise, I am tempted to patch my 4.1.1 to get my installation working againā€¦