Deployment of Alma9.2 fails with DNF error

Support
1

Problem:
When attempting to deploy Alma 9.2 from the network using the foreman server, DNF seems to break after registering with the foreman server. This has been observed on two different servers. One was my original one, that up until the latest version was working fine, the other is a fresh install done on Alma8.8 that was deployed just to test to see if it was the foreman instance or if it was something more.

[root@wkstst01 ~]# cat ./install.post.log
Updating system time
200 OK
##############################################################
################# SUBSCRIPTION MANAGER #######################
##############################################################

Starting the subscription-manager registration process
package subscription-manager is not installed
AlmaLinux 9 - AppStream                         2.5 MB/s | 7.6 MB     00:03
AlmaLinux 9 - BaseOS                            1.7 MB/s | 2.0 MB     00:01
AlmaLinux 9 - Extras                             28 kB/s |  17 kB     00:00
Last metadata expiration check: 0:00:01 ago on Mon Jun  5 01:20:34 2023.
Dependencies resolved.
======================================================================================
 Package                                  Arch    Version                Repo     Size
======================================================================================
Installing:
 subscription-manager                     x86_64  1.29.33.1-1.el9_2.alma baseos  775 k
Installing dependencies:
 libdnf-plugin-subscription-manager       x86_64  1.29.33.1-1.el9_2.alma baseos   78 k
 python3-chardet                          noarch  4.0.0-5.el9            baseos  209 k
 python3-cloud-what                       x86_64  1.29.33.1-1.el9_2.alma baseos   88 k
 python3-decorator                        noarch  4.4.2-6.el9            baseos   27 k
 python3-ethtool                          x86_64  0.15-2.el9             baseos   40 k
 python3-idna                             noarch  2.10-7.el9             baseos   92 k
 python3-iniparse                         noarch  0.4-45.el9             baseos   42 k
 python3-inotify                          noarch  0.9.6-25.el9           baseos   52 k
 python3-librepo                          x86_64  1.14.5-1.el9           baseos   49 k
 python3-pysocks                          noarch  1.7.1-12.el9           baseos   34 k
 python3-requests                         noarch  2.25.1-6.el9           baseos  114 k
 python3-setuptools                       noarch  53.0.0-12.el9          baseos  839 k
 python3-subscription-manager-rhsm        x86_64  1.29.33.1-1.el9_2.alma baseos  170 k
 python3-urllib3                          noarch  1.26.5-3.el9           baseos  188 k
 subscription-manager-rhsm-certificates   noarch  20220623-1.el9         baseos   20 k
 usermode                                 x86_64  1.114-4.el9            baseos  175 k
 virt-what                                x86_64  1.25-3.el9             baseos   33 k

Transaction Summary
======================================================================================
Install  18 Packages

Total download size: 3.0 M
Installed size: 13 M
Downloading Packages:
(1/18): libdnf-plugin-subscription-manager-1.29 316 kB/s |  78 kB     00:00
(2/18): python3-cloud-what-1.29.33.1-1.el9_2.al 328 kB/s |  88 kB     00:00
(3/18): python3-decorator-4.4.2-6.el9.noarch.rp 281 kB/s |  27 kB     00:00
(4/18): python3-ethtool-0.15-2.el9.x86_64.rpm   515 kB/s |  40 kB     00:00
(5/18): python3-chardet-4.0.0-5.el9.noarch.rpm  585 kB/s | 209 kB     00:00
(6/18): python3-iniparse-0.4-45.el9.noarch.rpm  562 kB/s |  42 kB     00:00
(7/18): python3-inotify-0.9.6-25.el9.noarch.rpm 741 kB/s |  52 kB     00:00
(8/18): python3-idna-2.10-7.el9.noarch.rpm      819 kB/s |  92 kB     00:00
(9/18): python3-pysocks-1.7.1-12.el9.noarch.rpm 576 kB/s |  34 kB     00:00
(10/18): python3-librepo-1.14.5-1.el9.x86_64.rp 756 kB/s |  49 kB     00:00
(11/18): python3-requests-2.25.1-6.el9.noarch.r 1.7 MB/s | 114 kB     00:00
(12/18): python3-subscription-manager-rhsm-1.29 1.5 MB/s | 170 kB     00:00
(13/18): python3-urllib3-1.26.5-3.el9.noarch.rp 1.5 MB/s | 188 kB     00:00
(14/18): subscription-manager-rhsm-certificates 229 kB/s |  20 kB     00:00
(15/18): python3-setuptools-53.0.0-12.el9.noarc 2.5 MB/s | 839 kB     00:00
(16/18): usermode-1.114-4.el9.x86_64.rpm        1.5 MB/s | 175 kB     00:00
(17/18): virt-what-1.25-3.el9.x86_64.rpm        539 kB/s |  33 kB     00:00
(18/18): subscription-manager-1.29.33.1-1.el9_2 2.5 MB/s | 775 kB     00:00
--------------------------------------------------------------------------------
Total                                           2.0 MB/s | 3.0 MB     00:01
AlmaLinux 9 - BaseOS                            3.0 MB/s | 3.1 kB     00:00
Importing GPG key 0xB86B3716:
 Userid     : "AlmaLinux OS 9 <packager@almalinux.org>"
 Fingerprint: BF18 AC28 7617 8908 D6E7 1267 D36C B86C B86B 3716
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux-9
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Installing       : subscription-manager-rhsm-certificates-20220623-1.    1/18
  Installing       : python3-setuptools-53.0.0-12.el9.noarch               2/18
  Installing       : python3-iniparse-0.4-45.el9.noarch                    3/18
  Installing       : python3-idna-2.10-7.el9.noarch                        4/18
  Installing       : python3-inotify-0.9.6-25.el9.noarch                   5/18
  Installing       : virt-what-1.25-3.el9.x86_64                           6/18
  Installing       : usermode-1.114-4.el9.x86_64                           7/18
  Installing       : python3-pysocks-1.7.1-12.el9.noarch                   8/18
  Installing       : python3-urllib3-1.26.5-3.el9.noarch                   9/18
  Installing       : python3-librepo-1.14.5-1.el9.x86_64                  10/18
  Installing       : python3-ethtool-0.15-2.el9.x86_64                    11/18
  Installing       : python3-decorator-4.4.2-6.el9.noarch                 12/18
  Installing       : python3-chardet-4.0.0-5.el9.noarch                   13/18
  Installing       : python3-requests-2.25.1-6.el9.noarch                 14/18
  Installing       : python3-cloud-what-1.29.33.1-1.el9_2.alma.x86_64     15/18
  Installing       : python3-subscription-manager-rhsm-1.29.33.1-1.el9_   16/18
  Installing       : libdnf-plugin-subscription-manager-1.29.33.1-1.el9   17/18
  Running scriptlet: subscription-manager-1.29.33.1-1.el9_2.alma.x86_64   18/18
  Installing       : subscription-manager-1.29.33.1-1.el9_2.alma.x86_64   18/18
  Running scriptlet: subscription-manager-1.29.33.1-1.el9_2.alma.x86_64   18/18
Created symlink /etc/systemd/system/multi-user.target.wants/rhsmcertd.service -> /usr/lib/systemd/system/rhsmcertd.service.

Running in chroot, ignoring command 'daemon-reload'
Running in chroot, ignoring command 'reload-or-restart'

  Verifying        : libdnf-plugin-subscription-manager-1.29.33.1-1.el9    1/18
  Verifying        : python3-chardet-4.0.0-5.el9.noarch                    2/18
  Verifying        : python3-cloud-what-1.29.33.1-1.el9_2.alma.x86_64      3/18
  Verifying        : python3-decorator-4.4.2-6.el9.noarch                  4/18
  Verifying        : python3-ethtool-0.15-2.el9.x86_64                     5/18
  Verifying        : python3-idna-2.10-7.el9.noarch                        6/18
  Verifying        : python3-iniparse-0.4-45.el9.noarch                    7/18
  Verifying        : python3-inotify-0.9.6-25.el9.noarch                   8/18
  Verifying        : python3-librepo-1.14.5-1.el9.x86_64                   9/18
  Verifying        : python3-pysocks-1.7.1-12.el9.noarch                  10/18
  Verifying        : python3-requests-2.25.1-6.el9.noarch                 11/18
  Verifying        : python3-setuptools-53.0.0-12.el9.noarch              12/18
  Verifying        : python3-subscription-manager-rhsm-1.29.33.1-1.el9_   13/18
  Verifying        : python3-urllib3-1.26.5-3.el9.noarch                  14/18
  Verifying        : subscription-manager-1.29.33.1-1.el9_2.alma.x86_64   15/18
  Verifying        : subscription-manager-rhsm-certificates-20220623-1.   16/18
  Verifying        : usermode-1.114-4.el9.x86_64                          17/18
  Verifying        : virt-what-1.25-3.el9.x86_64                          18/18

Installed:
  libdnf-plugin-subscription-manager-1.29.33.1-1.el9_2.alma.x86_64
  python3-chardet-4.0.0-5.el9.noarch
  python3-cloud-what-1.29.33.1-1.el9_2.alma.x86_64
  python3-decorator-4.4.2-6.el9.noarch
  python3-ethtool-0.15-2.el9.x86_64
  python3-idna-2.10-7.el9.noarch
  python3-iniparse-0.4-45.el9.noarch
  python3-inotify-0.9.6-25.el9.noarch
  python3-librepo-1.14.5-1.el9.x86_64
  python3-pysocks-1.7.1-12.el9.noarch
  python3-requests-2.25.1-6.el9.noarch
  python3-setuptools-53.0.0-12.el9.noarch
  python3-subscription-manager-rhsm-1.29.33.1-1.el9_2.alma.x86_64
  python3-urllib3-1.26.5-3.el9.noarch
  subscription-manager-1.29.33.1-1.el9_2.alma.x86_64
  subscription-manager-rhsm-certificates-20220623-1.el9.noarch
  usermode-1.114-4.el9.x86_64
  virt-what-1.25-3.el9.x86_64

Complete!
Retrieving http://foreman.domain.private/pub/katello-ca-consumer-latest.noarch.rpm
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
katello-ca-consumer-foreman.domain.p########################################

WARNING

The yum/dnf plugins: /etc/dnf/plugins/subscription-manager.conf were automatically enabled for the benefit of Subscription Management. If not desired, use "subscription-manager config --rhsm.auto_enable_yum_plugins=0" to block this behavior.

The system has been registered with ID: 5d250cb2-9322-4951-9221-666700df725d
The registered system name is: wkstst01.domain.private
Updating Subscription Management repositories.
Google Chrome                                    17 kB/s | 3.6 kB     00:00
mrochefort vfx-compatibility                     50 kB/s |  11 kB     00:00
Katello Client for EL9 x86_64 - 3.6 RPMs         39 kB/s | 9.0 kB     00:00
Alma 9 for x86_64 - BaseOS RPMs                 8.0 MB/s | 2.0 MB     00:00
Slack                                            62 kB/s |  12 kB     00:00
EPEL 9 for x86_64 - RPMs                         30 MB/s |  16 MB     00:00
NVIDIA rhel9 x86_64                             3.2 MB/s | 657 kB     00:00
mrochefort gnome                                 20 kB/s | 4.3 kB     00:00
Visual Studio Code                               43 MB/s |  36 MB     00:00
Alma 9 for x86_64 - CRB RPMs                    7.4 MB/s | 2.3 MB     00:00
Alma 9 for x86_64 - AppStream RPMs               26 MB/s | 7.5 MB     00:00
terminate called after throwing an instance of 'libdnf::ModulePackageContainer::ResolveException'
  what():  Failed to update from string: Parser error
/tmp/ks-script-tff8l2nz: line 303: 37552 Aborted                 (core dumped) $PACKAGE_MAN install katello-host-tools
Updating Subscription Management repositories.
Google Chrome                                    27 kB/s | 1.5 kB     00:00
mrochefort vfx-compatibility                     35 kB/s | 1.8 kB     00:00
Katello Client for EL9 x86_64 - 3.6 RPMs         36 kB/s | 1.8 kB     00:00
Alma 9 for x86_64 - BaseOS RPMs                  44 kB/s | 2.3 kB     00:00
Slack                                            38 kB/s | 2.1 kB     00:00
EPEL 9 for x86_64 - RPMs                         48 kB/s | 2.3 kB     00:00
NVIDIA rhel9 x86_64                              38 kB/s | 1.8 kB     00:00
mrochefort gnome                                 32 kB/s | 1.8 kB     00:00
Visual Studio Code                               27 kB/s | 1.5 kB     00:00
Alma 9 for x86_64 - CRB RPMs                     48 kB/s | 2.6 kB     00:00
Alma 9 for x86_64 - AppStream RPMs               47 kB/s | 2.6 kB     00:00
terminate called after throwing an instance of 'libdnf::ModulePackageContainer::ResolveException'
  what():  Failed to update from string: Parser error
/tmp/ks-script-tff8l2nz: line 303: 37600 Aborted                 (core dumped) dnf -y update

The exact error that DNF will now though every time throw every time is

terminate called after throwing an instance of 'libdnf::ModulePackageContainer::ResolveException'
  what():  Failed to update from string: Parser error
/tmp/ks-script-tff8l2nz: line 303: 37600 Aborted                 (core dumped) dnf -y update

Sadly as this happens during the kickstart I have not been able to find any logs on it, other than what I posted above, and i can’t run strace as I can’t seem to get it installed.

This also happens before any major configuration of the machine and should just be the default kickstart that comes with foreman.

Foreman and Proxy versions:
foreman-3.6.1-1
katello-4.8.1-1

Distribution and version:
Foreman is installed on Alma8.8
I am attempting to deploy workstations with Alma9.2

I have both servers up, I have a snapshot of my old server i should be able to roll back to a previous version but would that cause issues with the pulp folder that is located on a NFS share?

Thank you for the help!

2

I am not very good with C++ nor do I know DNF source code very well, but from what I could find, the error message you get only printed here.
This would indicate that something is wrong with your repository metadata. Maybe try re-syncing the repos and rebuilding your content-view, if you have not done that yet.
Otherwise, maybe the mirror you are syncing from is broken? You could try syncing from another mirror and see if that solves the problem.

3

Thanks!

So, you gave me an idea to attempt to figure out what repo is dead by removing and readding one at a time.

Turns out it is the nvidia repo that is giving me issues and sadly that is direct from nvidia themselves

https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/

I’m going to attempt to install the repo directly skipping the foreman caching, and see if that gives me issues also, That should at least tell us if there is something wrong with the foreman parsing or if its the actual nvidia repo.

4

Okay, So after adding the repo manually, and attempting a update, it seems to work fine… So it must be something to do with the way foreman syncs the repo that is breaking it. Not sure what else I can do from here other than disable the repo and manually install the repo direct from nvidia.

5

I’m seeing the same behavior here, 9.2 client with NVidia repos enabled from Katello makes any dnf operation fail.
Looking closer with strace, it dies when reading /var/cache/dnf/…CUDA…/repodata/…-modules.yaml
Checking that file with yamllint gives a syntax error:
1673:1 error syntax error: expected ‘’, but found ‘’ (syntax)

In between each section, there is a:
…

At the error line, the — is missing.
– SNIP –
- nvidia-xconfig
- kmod-nvidia-open-dkms
…
document: modulemd
– SNIP –

6

Sigh, looking back, I realized that the important bits were mangled with markup.

Supposed to be between each section of the file
---
...

The line of dashes is missing in one particular spot in the YAML file, causing a syntax error.
type or paste code here