Libsolv from pulpcore repo breaks dnf on CentOS 8 Stream

Problem:

I don’t know if this is the right place to report this, but the libsolv-0.7.20-4.el8.x86_64 package from the pulpcore repository breaks dnf on CentOS 8 Stream.

Which in turn causes foreman-installer to terminate:

2022-06-02 14:23:10 [ERROR ] [root] dnf module enable pki-core --assumeyes failed! Check the output for error!

Running dnf manually fails because of undefined symbols:

...
ImportError: /lib64/libdnf.so.2: undefined symbol: solv_toolversion, version SOLV_1.0
...

I reverted the libsolv package to the version available from the CentOS Stream repositories (libsolv-0.7.20-3.el8). This allows me to run dnf again to install and update packages, but foreman-installer fails:

2022-06-02 14:51:14 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-06-02 14:51:14 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-06-02 14:51:14 [NOTICE] [configure] 2000 configuration steps out of 2042 steps complete.

The traceback from pulcore-manager migrate looks like this:

2022-06-02 14:51:12 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput'
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Traceback (most recent call last):
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib64/python3.8/site-packages/solv.py", line 14, in swig_import_helper
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return importlib.import_module(mname)
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return _bootstrap._gcd_import(name[level:], package, level)
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 556, in module_from_spec
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap_external>", line 1166, in create_module
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: ImportError: /usr/lib64/python3.8/site-packages/_solv.so: undefined symbol: repo_add_deb, version SOLV_1.0
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: During handling of the above exception, another exception occurred:
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Traceback (most recent call last):
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/bin/pulpcore-manager", line 11, in <module>
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     load_entry_point('pulpcore==3.16.8', 'console_scripts', 'pulpcore-manager')()
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     execute_from_command_line(sys.argv)
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     utility.execute()
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     django.setup()
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     apps.populate(settings.INSTALLED_APPS)
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     app_config.ready()
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/pulpcore/app/apps.py", line 96, in ready
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.import_viewsets()
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/pulpcore/app/apps.py", line 137, in import_viewsets
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.viewsets_module = import_module(viewsets_module_name)
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return _bootstrap._gcd_import(name[level:], package, level)
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap_external>", line 843, in exec_module
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/pulp_rpm/app/viewsets.py", line 35, in <module>
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     from pulp_rpm.app import tasks
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/__init__.py", line 3, in <module>
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     from .copy import copy_content  # noqa
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/copy.py", line 6, in <module>
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     from pulp_rpm.app.depsolving import Solver
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.8/site-packages/pulp_rpm/app/depsolving.py", line 3, in <module>
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     import solv
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib64/python3.8/site-packages/solv.py", line 17, in <module>
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     _solv = swig_import_helper()
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib64/python3.8/site-packages/solv.py", line 16, in swig_import_helper
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return importlib.import_module('_solv')
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return _bootstrap._gcd_import(name[level:], package, level)
2022-06-02 14:51:14 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: ImportError: /usr/lib64/python3.8/site-packages/_solv.so: undefined symbol: repo_add_deb, version SOLV_1.0
2022-06-02 14:51:14 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

Expected outcome:

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Stream 8

Other relevant data:

I’m also seeing this exact same issue.

@fbachmann we caught this yesterday in one pipeline for Katello 4.4.1 upgrade,

Looks like the CentosStream package have some patches from 0.7.22, on Pulpcore 3.17 we already ship 0.7.22.

0.7.22 is already cherrypicked into 3.16, it just need to be signed and released.

I’ll update here later today

2 Likes

@jruk @fbachmann

The package is now published in pulpcore 3.16 repo, please use this package https://yum.theforeman.org/pulpcore/3.16/el8/x86_64/libsolv-0.7.22-1.el8.x86_64.rpm

Thanks very much for the prompt attention. I can confirm that with the updated libsolv (0.7.22-1) this fixes the earlier dnf problem :blush:

@Odilhao The pulpcore repo ought to stop providing libsolv on EL8 systems. With EL7 there was no problem with providing it and we basically had to do so because the only packages available were way too out of date, but with EL8 it can be dangerous to provide our own (for this reason) and there’s no need to do so because it’s kept up to date.

We already have published the 0.7.22 version for EL8 and EL7 for pulp 3.16/17/18, and right now the EL8 version that is packaged by Centos/RHEL is at 07.20, this would break upgrades.

Tagging @pcreech and @ehelms to see what we can do about this

To my knowledge, the reasons we have packaged our verions of this package have been due to the base OS not keeping up with the version we need for Pulp to handle everything it needs to in the timeframe it needs.

@dralley can you confirm that the current version in EL8 suffices for all use cases and will continue to do so? For EL9 content will that continue to be true?

If the Pulp project says we can trust the version of libsolv in base OS to handle everything we need then we should look into not carrying it in the future.

To my knowledge, the reasons we have packaged our verions of this package have been due to the base OS not keeping up with the version we need for Pulp to handle everything it needs to in the timeframe it needs.

That was definitely true in the past, but largely because nothing on EL7 really uses libsolv. Thus, the version they use is ancient and nobody updates it. On EL8 however, DNF uses libsolv, which makes it both more dangerous for us to mess with and also ensures that the package is being cared for.

If you look at CentOS 8 Stream you can see the updates, and it’s only slightly behind CentOS 9 Stream currently: https://pkgs.org/search/?q=libsolv

This tracks with what the DNF told me which is that they plan to rebase the DNF stack in EL more frequently going forwards.

@dralley can you confirm that the current version in EL8 suffices for all use cases and will continue to do so? For EL9 content will that continue to be true?

It should, yes. In the early days we had to make a few contributions ourselves to get things we needed, and that was another reason for carrying the package, but that hasn’t been the case for a while. In terms of features we’re covered, we don’t need really need anything in the newer versions.

If the Pulp project says we can trust the version of libsolv in base OS to handle everything we need then we should look into not carrying it in the future.

:+1:

@dralley @ehelms @Odilhao so, based on the amount of re-basing I saw at that link dalley provided, , do we need to “stop providing libsolv” for el8, or will it suffice to just stop updating our el8 libsolv, and eventually the situation will resolve itself?

i.e. Eventually centos stream will update to a libsolv >=0.7.22-1, superceeding what we publish, and no longer pulling in our update?

I’d like to avoid an “unpublish and have users downgrade manually” situation if we can.