Cannot update puppet-agent to 6.24.0 on foreman servers

gvde · July 21, 2021, 6:07am

Katello 4.1.1, Foreman 2.5.2 on CentOS 7.9

puppet-agent 6.24.0 just became available. I was able to update on all my clients which connect to my content proxy.

Only my foreman servers (main server and smart proxies) refuse to update. Unlike the clients, they connect to the main server not to the content proxy.

Resolving Dependencies
--> Running transaction check
---> Package puppet-agent.x86_64 0:6.23.0-1.el7 will be updated
---> Package puppet-agent.x86_64 0:6.24.0-1.el7 will be an update
---> Package puppetserver.noarch 0:6.16.0-1.el7 will be updated
---> Package puppetserver.noarch 0:6.16.1-1.el7 will be an update
--> Processing Dependency: /opt/puppetlabs/puppet/bin/ruby for package: 1:foreman-installer-2.5.2-3.el7.noarch
--> Finished Dependency Resolution
Error: Package: 1:foreman-installer-2.5.2-3.el7.noarch (@DKRZ_foreman_2_5_el7_x86_64)
           Requires: /opt/puppetlabs/puppet/bin/ruby
           Removing: puppet-agent-6.23.0-1.el7.x86_64 (@DKRZ_puppet_puppet6_el7_x86_64)
               Not found
           Updated By: puppet-agent-6.24.0-1.el7.x86_64 (DKRZ_puppet_puppet6_el7_x86_64)
               Not found
           Available: puppet-agent-6.0.0-1.el7.x86_64 (DKRZ_puppet_puppet6_el7_x86_64)
               Not found
...

On the foreman servers it doesn’t know about the dependency:

# yum provides /opt/puppetlabs/puppet/bin/ruby
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, tracer_upload
Loading mirror speeds from cached hostfile
...
puppet-agent-6.22.1-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.23.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.23.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : @DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby

Curious enough, I really shows 6.2.30-0.1 twice, there.

On my clients, it is correct:

# yum provides /opt/puppetlabs/puppet/bin/ruby
...
puppet-agent-6.22.1-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.23.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby



puppet-agent-6.24.0-1.el7.x86_64 : The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
Repo        : @DKRZ_puppet_puppet6_el7_x86_64
Matched from:
Filename    : /opt/puppetlabs/puppet/bin/ruby

Each version only once and it includes 6.24.0-1. Looks correct to me.

So it kind of looks as if the main server presents the new rpm as puppet-agent-6.23.0-1.el7.x86_64 for provides causing it to be listed twice and breaking dependencies?

gvde · July 21, 2021, 3:04pm

O.K. There must be some serious issue here. I see the same problem now with systemd-udev on CentOS 8:

# dnf provides /usr/bin/kernel-install
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:14 ago on Wed 21 Jul 2021 04:59:33 PM CEST.
systemd-udev-239-45.el8.x86_64 : Rule-based device node and kernel event manager
Repo        : ORG_centos8_BaseOS_x86_64
Matched from:
Filename    : /usr/bin/kernel-install

systemd-udev-239-45.el8_4.1.x86_64 : Rule-based device node and kernel event manager
Repo        : @System
Matched from:
Filename    : /usr/bin/kernel-install

systemd-udev-239-45.el8_4.1.x86_64 : Rule-based device node and kernel event manager
Repo        : ORG_centos8_BaseOS_x86_64
Matched from:
Filename    : /usr/bin/kernel-install

This blocks the update to the latest CentOS 8 kernel and systemd. Rocky Linux sees the same. See Katello 4.1 and Rocky Linux 8 - not ready for prime time yet it seems

Maybe also related with the error here: CentOS 8.4 BaseOS Sync error

gvde · July 21, 2021, 4:28pm

The problem with puppet-agent only affects my foreman servers which are connected to the main foreman server.

The systemd-udev issue is on my clients which are connected to my (single) foreman content proxy.

I have downloaded the filelists.xml.gz file for the baseos from the content proxy and looked into it, searching for /usr/bin/kernel-install:

...
<package pkgid="c4b34bb4476f7a7dac5cb7adbd1fa710a8fb960a66035a1947c1c0ea7f3187c4" name="systemd-udev" arch="x86_64">
  <version epoch="0" ver="239" rel="45.el8"/>
...
  <file>/usr/bin/kernel-install</file>
...
<package pkgid="22089981a3038158c83bdc693b7a39956c9992a041e88d91461430d1e4d64383" name="systemd-udev" arch="x86_64">
  <version epoch="0" ver="239" rel="45.el8_4.1"/>
...
  <file>/usr/bin/kernel-install</file>
...
<package pkgid="16124eda58e4b07ef18056a11bd20da165eb780c90a663baf99d14770547482f" name="systemd-udev" arch="x86_64">
  <version epoch="0" ver="239" rel="45.el8_4.2"/>
</package>

The complete file list for the latest version is missing.

primary.xml looks good, though:

<package type="rpm">
  <name>systemd-udev</name>
  <arch>x86_64</arch>
  <version epoch="0" ver="239" rel="45.el8_4.2"/>
  <checksum type="sha256" pkgid="YES">16124eda58e4b07ef18056a11bd20da165eb780c90a663baf99d14770547482f</checksum>
  <summary>Rule-based device node and kernel event manager</summary>
  <description>This package contains systemd-udev and the rules and hardware database
needed to manage device nodes. This package is necessary on physical
machines and in virtual machines, but not in containers.</description>
  <packager>CentOS Buildsys &lt;bugs@centos.org&gt;</packager>
  <url>http://www.freedesktop.org/wiki/Software/systemd</url>
  <time file="1626829839" build="1626829051"/>
  <size package="1436628" installed="8009345" archive="8049636"/>
  <location href="Packages/s/systemd-udev-239-45.el8_4.2.x86_64.rpm"/>
  <format>
    <rpm:license>LGPLv2+</rpm:license>
    <rpm:vendor>CentOS</rpm:vendor>
    <rpm:group>Unspecified</rpm:group>
    <rpm:buildhost>x86-01.mbox.centos.org</rpm:buildhost>
    <rpm:sourcerpm>systemd-239-45.el8_4.2.src.rpm</rpm:sourcerpm>
    <rpm:header-range start="5608" end="152592"/>
    <rpm:provides>
      <rpm:entry name="config(systemd-udev)" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="systemd-udev" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="systemd-udev(x86-64)" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="udev" flags="EQ" epoch="0" ver="239"/>
      <rpm:entry name="udev(x86-64)" flags="EQ" epoch="0" ver="239"/>
    </rpm:provides>
    <rpm:requires>
      <rpm:entry name="/bin/bash"/>
      <rpm:entry name="/bin/sh"/>
      <rpm:entry name="/bin/sh" pre="1"/>
      <rpm:entry name="/bin/sh"/>
      <rpm:entry name="grep" pre="1"/>
      <rpm:entry name="kmod" flags="GE" epoch="0" ver="18" rel="4"/>
      <rpm:entry name="libacl.so.1()(64bit)"/>
      <rpm:entry name="libacl.so.1(ACL_1.0)(64bit)"/>
      <rpm:entry name="libblkid.so.1()(64bit)"/>
      <rpm:entry name="libblkid.so.1(BLKID_2.15)(64bit)"/>
      <rpm:entry name="libblkid.so.1(BLKID_2.17)(64bit)"/>
      <rpm:entry name="libblkid.so.1(BLKID_2.18)(64bit)"/>
      <rpm:entry name="libcryptsetup.so.12()(64bit)"/>
      <rpm:entry name="libcryptsetup.so.12(CRYPTSETUP_2.0)(64bit)"/>
      <rpm:entry name="libgcc_s.so.1()(64bit)"/>
      <rpm:entry name="libgcc_s.so.1(GCC_3.0)(64bit)"/>
      <rpm:entry name="libgcc_s.so.1(GCC_3.3.1)(64bit)"/>
      <rpm:entry name="libkmod.so.2()(64bit)"/>
      <rpm:entry name="libkmod.so.2(LIBKMOD_5)(64bit)"/>
      <rpm:entry name="libpthread.so.0()(64bit)"/>
      <rpm:entry name="libpthread.so.0(GLIBC_2.2.5)(64bit)"/>
      <rpm:entry name="libsystemd-shared-239.so()(64bit)"/>
      <rpm:entry name="libsystemd-shared-239.so(SD_SHARED)(64bit)"/>
      <rpm:entry name="rtld(GNU_HASH)"/>
      <rpm:entry name="systemd" pre="1"/>
      <rpm:entry name="systemd"/>
      <rpm:entry name="systemd(x86-64)" flags="EQ" epoch="0" ver="239" rel="45.el8_4.2"/>
      <rpm:entry name="libc.so.6(GLIBC_2.28)(64bit)"/>
    </rpm:requires>
    <rpm:obsoletes>
      <rpm:entry name="systemd" flags="LT" epoch="0" ver="229" rel="5"/>
      <rpm:entry name="udev" flags="LT" epoch="0" ver="183"/>
    </rpm:obsoletes>
    <rpm:recommends>
      <rpm:entry name="kbd"/>
    </rpm:recommends>
  </format>
</package>

As far as I can tell that is identical to what is on the CentOS mirror servers. So the problem seems to be the filelists.xml.

I’ll go into the postgres database on the content proxy now if I can find anything there…

gvde · July 21, 2021, 4:49pm

O.K. The file list is missing in the database as well. It’s missing in the database on the content proxy as well as on the main server.

content proxy:

pulpcore=# select content_ptr_id, name, version, release, jsonb_array_length(files) from rpm_package where name = 'systemd-udev';
            content_ptr_id            |     name     | version |  release   | jsonb_array_length 
--------------------------------------+--------------+---------+------------+--------------------
 4346017c-e235-4bcd-89eb-6d3030529d8c | systemd-udev | 239     | 45.el8     |                252
 18c0981d-4094-4666-8dfb-cc73ddfb59a1 | systemd-udev | 239     | 45.el8_4.1 |                248
 1217bb8d-0851-45be-9319-9cd1ad5b2539 | systemd-udev | 239     | 45.el8_4.2 |                  0
(3 rows)

on the main server:

pulpcore=# select content_ptr_id, name, version, release, jsonb_array_length(files) from rpm_package where name = 'systemd-udev';
            content_ptr_id            |     name     | version |  release   | jsonb_array_length 
--------------------------------------+--------------+---------+------------+--------------------
 a5d9cfb8-a909-4daf-aa81-8b32b4c1a82f | systemd-udev | 239     | 45.el8     |                255
 0c1ec5b0-436a-405b-8e51-3da9f2d4b7fe | systemd-udev | 239     | 45.el8     |                252
 e77db7b3-9bbc-4269-9b77-ebbcd2774c51 | systemd-udev | 239     | 45.el8_4.1 |                248
 1153dc0b-f8cd-4a36-801c-704231f9ef49 | systemd-udev | 239     | 45.el8_4.1 |                248
 bc9e9c33-4140-424c-9d37-7e1e34912aa5 | systemd-udev | 239     | 45.el8_4.2 |                  0
 8d8ac124-c461-47fb-9734-afeff651ec28 | systemd-udev | 239     | 48.el8     |                250
(6 rows)

(The main server also has CentOS 8 Stream and AlmaLinux 8 which are not synced to the content proxy, that’s why there are more available…)

So it seems as if the filelist for the package got lost somewhere…

I check a few more packages: puppet-agent as it caused problems and also kernel-core for which an update is also available:

content proxy:

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where name = 'puppet-agent' and jsonb_array_length(files) = 0;
            content_ptr_id            | epoch |     name     | version | release |                              pkgId                               | jsonb_array_length 
--------------------------------------+-------+--------------+---------+---------+------------------------------------------------------------------+--------------------
 b8720eae-0ad3-4f80-98b7-5ada157a9153 | 0     | puppet-agent | 6.24.0  | 1.el8   | be42b9808718a935077b31390e31d294f87ae8e0607a2b534fe2cfa28018d6d5 |                  0
 d742286f-0114-4c1f-914b-8d7aacb6f2c4 | 0     | puppet-agent | 6.24.0  | 1.el7   | 80497802c6dd82891c8c5682a8ce7aa11c45c89ed080369ac295f058ef723ad2 |                  0
(2 rows)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'kernel-core';
            content_ptr_id            | epoch |    name     | version |    release     |                              pkgId                               | jsonb_array_length 
--------------------------------------+-------+-------------+---------+----------------+------------------------------------------------------------------+--------------------
 eb1b8389-1367-4b90-b2e7-8e0b2bb69040 | 0     | kernel-core | 4.18.0  | 305.10.2.el8_4 | 6ec68b3ea6e89936514d650f30c14a49c2162df59149ead31aa026686cbb467e |                  0
(1 row)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'filebeat';
            content_ptr_id            | epoch |   name   | version | release |                              pkgId                               | jsonb_array_length 
--------------------------------------+-------+----------+---------+---------+------------------------------------------------------------------+--------------------
 8c7f4b9f-f9eb-4f7d-b182-d5a18167fca3 | 0     | filebeat | 7.13.4  | 1       | 0d2f70d84dd361bb976c6dd2fb2aa5d0261595bc4c347cb6549ab9d9dec86a65 |                  0
 9dd9ccf7-18f7-4bb0-9ff3-cce47782dee4 | 0     | filebeat | 7.13.4  | 1       | e50aee18037b11610a10040a0f4bd264042d9c080746a51c211b04db79639417 |                  0
 aa246676-2fef-4a99-9bb4-0d6b29695dba | 0     | filebeat | 7.13.4  | 1       | 869b59700c922e910ff0c11eb6794f98b0fa132d6f316ab80a0c5fd8bc719d70 |                  0
(3 rows)

main server:

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where name = 'puppet-agent' and jsonb_array_length(files) = 0;
            content_ptr_id            | epoch |     name     | version | release |                  pkgId                   | jsonb_array_length 
--------------------------------------+-------+--------------+---------+---------+------------------------------------------+--------------------
 792b45f7-04c4-4de6-8d58-24ec5ced447f | 0     | puppet-agent | 6.24.0  | 1.el7   | 3bcf6e568a3ad9fb82b122d46b51f2e566588b5b |                  0
 e11548a2-1e19-4922-a881-56efff6a2d46 | 0     | puppet-agent | 6.24.0  | 1.el8   | 2f0ea8c6767c43308e84a030c80966fa1b84a3aa |                  0
(2 rows)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'kernel-core';
            content_ptr_id            | epoch |    name     | version |    release     |                              pkgId                               |
 jsonb_array_length 
--------------------------------------+-------+-------------+---------+----------------+------------------------------------------------------------------+
--------------------
 96342a34-e16b-47f0-aa75-cf802e55ef26 | 0     | kernel-core | 4.18.0  | 305.10.2.el8_4 | 6ec68b3ea6e89936514d650f30c14a49c2162df59149ead31aa026686cbb467e |
                  0
(1 row)

pulpcore=# select content_ptr_id, epoch, name, version, release, "pkgId", jsonb_array_length(files) from rpm_package where jsonb_array_length(files) = 0 and name = 'filebeat';
            content_ptr_id            | epoch |   name   | version | release |                              pkgId                               | jsonb_arr
ay_length 
--------------------------------------+-------+----------+---------+---------+------------------------------------------------------------------+----------
----------
 b0e5b41e-b7f7-48f0-b2aa-0ce4ebee3daa | 0     | filebeat | 7.13.4  | 1       | 0d2f70d84dd361bb976c6dd2fb2aa5d0261595bc4c347cb6549ab9d9dec86a65 |          
        0
 a82f55fa-9d43-4da7-a4b1-b7d0a103f13e | 0     | filebeat | 7.13.4  | 1       | e50aee18037b11610a10040a0f4bd264042d9c080746a51c211b04db79639417 |          
        0
 60b2e3f3-a771-40f9-99a2-4e5675be7bb5 | 0     | filebeat | 7.13.4  | 1       | 869b59700c922e910ff0c11eb6794f98b0fa132d6f316ab80a0c5fd8bc719d70 |          
        0
(3 rows)

Those are all new package versions which have been available since today. So basically, the repository sync doesn’t import the filelist anymore leaving it empty which causes problems, if there are dependencies to one of those files from other packages…

Unfortunately, I cannot simply search for empty file lists in the database to find all broken entries as some packages really don’t contain any files…

I also already did a “Validate Content Sync” on the CentOS 8 BaseOS repository which was successful but obviously didn’t fix those empty file lists…

gvde · July 21, 2021, 7:16pm

O.K. by checking core_content.pulp_created timestamp I was able to pinpoint the time when this problem started: all packages synced since my update from 4.1.0 to 4.1.1 are missing the file list. All rpm syncs. All rpm repositories.

@ehelms @iballou So I guess this is a pulpcore 3.14 issue, then?

gvde · July 21, 2021, 7:38pm

I guess it‘s this pulp issue: https://pulp.plan.io/issues/9107

iballou · July 21, 2021, 7:39pm

Hi again @gvde,

Ah good catch on the issue there. It’s already got a Bugzilla too. It should be fixed soon then!

gvde · July 21, 2021, 7:44pm

Do you have the bugzilla link?

I hope the fix will also correct the synced packages with the missing filelist in the database. A complete, validated repository sync doesn‘t seem to affect that part…

gvde · July 21, 2021, 7:47pm

Never mind: https://bugzilla.redhat.com/show_bug.cgi?id=1984125

iballou · July 21, 2021, 7:57pm

I would voice that concern on the redmine issue. I imagine that would be fixed as well, but it’s good to check.

caseybea · July 21, 2021, 8:32pm

Go figure that I hit this bug the day I decide to light up Katello 4.1 and insert A centos-8-clone-like repository (in my case, Rocky Linux 8.4).

I was right in an early commment on a related post of mine that this was pulp. Ha!

I will comment in my post the link to the bug listed above by @gvde

gvde · July 22, 2021, 4:54am

I just wrote a message in pulp isssue #9107.

I looked at the code of the PR and it seems to me as if there are functions to validate the synced data in the database with the original metadata in the source repository. So if I am not mistaken that is there, it’s unclear to me if that works automatically during each sync or if that requires a special pulp repair call from katello like the validated complete sync…

gvde · July 22, 2021, 5:05am

I quote the answer from the pulp issue here to have all the relevant information in this topic, too:

Most likely we will just write a script that will do an in-place repair - it hasn’t been decided yet whether Katello would do anything to run it automatically. My assumption is - probably not.

We’ll try to get 3.14.0 released by the end of the day (Thursday), which should let Katello package it by the end of the week. And then once that’s out we’ll evaluate the best way to address any existing issues.

O.K. There is hope here… Wait until the pulp-rpm 3.14.0 package to get the correct data in for future syncs. Then run a in-place repair. And it’s hopefully all up to speed again…

caseybea · July 22, 2021, 6:16pm

So when I eventually get the new pulp, what’s the “in-place repair” process?

gvde · July 24, 2021, 10:53am

It seems pulp_rpm 3.14.0 is out. So I guess next step would be foreman/katello packaging…

gvde · July 26, 2021, 5:57am

@iballou @jjeffers There also seems to be a script available if all RPMs have been downloaded. I guess that means if you have download policy “immediate” and not “on demand”.

I hope it’ll soon gets into the katello repositories and we can get away from that broken version. By now, I have quite a few updates in the queue and I started doing updates from the centos mirrors because some are overdue for more than a week now…

iballou · July 26, 2021, 8:08pm

Just wanted to let you all know that we’re in the process of testing out the Pulp team’s fixing script. I’m not sure how it’ll be delivered but we’ll keep you updated.

gvde · July 27, 2021, 4:38am

Excellent. Well will the new pulp_rpm package be available? The sooner I stop syncing rpms with broken metadata into the system, the better. Repairing would be the second step for me…

I wonder what the repair script does exactly. Can it find all RPMs with have broken metadata or does it simply go through everything and compares & repairs it if it is not matching with what it should be?

ekohl · July 27, 2021, 12:11pm

There is a packaging PR:

gvde · July 28, 2021, 5:41am

Is this going into 4.1.2? I really don’t quite understand where to look to see what’s going into an upcoming release… If it’s going into 4.1.2 and I can expect 4.1.2 very soon, I would wait. Otherwise, I am tempted to patch my 4.1.1 to get my installation working again…