Modular Repos randomly break, offering packages from disabled streams

areyus · August 8, 2022, 12:43pm

Problem:
Every once in a while, one of our content smart-proxies seems to lose it’s information about modular repositories and starts offering it’s clients packages from disabled modules. This seems to be unrelated to any particular Content Actions since non (except daily syncs from upstreams to library) are taken in the meantime.
The timeline always looks something like this:

Create and publish new CV and CCV version
Sync the new versions to content smart-proxies
Test the new CVs with a few test systems, everything works as expected
A few days pass
Hosts registred to one of our smart-proxies start getting offered packages from disabled modules, even the test systems that worked a few days prior

So far, the only way to fix this has been to create and pulish new CV versions, but this conflicts with our companies policy on testing patches and applying them cyclical.

Expected outcome:
Hosts only seeing packages from enabled modules, behaviour of CV versions not changing ramdomly.

Foreman and Proxy versions:
2.5.4

Foreman and Proxy plugin versions:

foreman-tasks 	The goal of this plugin is to unify the way of showing task statuses across the Foreman instance. It defines Task model for keeping the information about the tasks and Lock for assigning the tasks to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks. 	Ivan Nečas 	4.1.5
foreman_expire_hosts 	A Foreman plugin that allows hosts to expire at a configurable date. Hosts will be shut down and automatically deleted after a grace period. 	Nagarjuna Rachaneni and Timo Goebel 	7.0.4
foreman_hooks 	Plugin engine for Foreman that enables running custom hook scripts on Foreman events 	Dominic Cleal 	0.3.17
foreman_remote_execution 	A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality. 	Foreman Remote Execution team 	4.5.6
foreman_scc_manager 	Foreman plugin to sync SUSE Customer Center products and repositories into Katello. 	ATIX AG 	1.8.10
foreman_snapshot_management 	Foreman-plugin to manage snapshots in a virtual-hardware environments. 	ATIX AG 	2.0.1
foreman_templates 	Engine to synchronise provisioning templates from GitHub 	Greg Sutcliffe 	9.1.0
katello 	Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp. 	N/A 	4.1.4
puppetdb_foreman 	Disable hosts on PuppetDB after they are deleted or built in Foreman. Follow https://github.com/theforeman/puppetdb_foreman and raise an issue/submit a pull request if you need extra functionality. You can also find some help in #theforeman IRC channel on Freenode. 	Daniel Lobato Garcia 	5.0.0

Distribution and version:
RHEL7

Other relevant data:
Sample output from an affected client:

# dnf module list
Updating Subscription Management repositories.
EPEL-EL8                                                                                                                                                                                                                                                 49 kB/s | 2.3 kB     00:00
Puppet6 EL8                                                                                                                                                                                                                                              39 kB/s | 1.7 kB     00:00
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)                                                                                                                                                                                                 61 kB/s | 2.9 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                                                                 58 kB/s | 2.9 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                                                                    49 kB/s | 2.6 kB     00:00
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)
Name                 Stream          Profiles                     Summary
javapackages-tools   201801          common                       Tools and macros for Java packaging support
mariadb-devel        10.3                                         MariaDB Module
python38-devel       3.8                                          Python programming language, version 3.8
python39-devel       3.9                                          Python programming language, version 3.9
virt-devel           rhel                                         Virtualization module

Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
Name                 Stream          Profiles                     Summary
389-ds               1.4                                          389 Directory Server (base)
ant                  1.10 [d]        common [d]                   Java build tool
container-tools      rhel8 [d]       common [d]                   Most recent (rolling) versions of podman, buildah, skopeo, runc, conmon, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and updated as frequently as ever
                                                                  y 12 weeks.
container-tools      1.0             common [d]                   Stable versions of podman 1.0, buildah 1.5, skopeo 0.1, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and supported for 24 months.
container-tools      2.0             common [d]                   Stable versions of podman 1.6, buildah 1.11, skopeo 0.1, runc, conmon, etc as well as dependencies such as container-selinux built and tested together, and supported as documented on the Application Stream lifecycl
                                                                  e page.
container-tools      3.0             common [d]                   Stable versions of podman 3.0, buildah 1.19, skopeo 1.2, runc, conmon, etc as well as dependencies such as container-selinux built and tested together, and supported as documented on the Application Stream lifecycl
                                                                  e page.
container-tools      4.0             common [d]                   Stable versions of podman 4.0, buildah 1.24, skopeo 1.6, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and supported for 24 months.
eclipse              rhel8           java                         An open, extensible IDE and application platform
freeradius           3.0 [d]         server [d]                   High-performance and highly configurable free RADIUS server
gimp                 2.8 [d]         common [d], devel            gimp module
go-toolset           rhel8 [d]       common [d]                   Go
httpd                2.4 [d][e]      common [d], devel, minimal   Apache HTTP Server
idm                  DL1             adtrust, client, common [d], The Oracle Linux Identity Management system module
                                      dns, server
idm                  client [d]      common [d]                   OL IdM long term support client module
inkscape             0.92.3 [d]      common [d]                   Vector-based drawing program using SVG
javapackages-runtime 201801 [d]      common [d]                   Basic runtime utilities to support Java applications
jmc                  rhel8 [d]       common [d], core             Java Mission Control is a profiling and diagnostics tool for the Hotspot JVM
libselinux-python    2.8             common                       Python 2 bindings for libselinux
llvm-toolset         rhel8 [d]       common [d]                   LLVM
log4j                2 [d]           common [d]                   Java logging library
mailman              2.1 [d]         common [d]                   Electronic mail discussion and e-newsletter lists managing software
mariadb              10.3 [d]        client, galera, server [d]   MariaDB Module
mariadb              10.5            client, galera, server [d]   MariaDB Module
maven                3.5 [d]         common [d]                   Java project management and project comprehension tool
maven                3.6             common [d]                   Java project management and project comprehension tool
mercurial            4.8 [d]         common [d]                   Mercurial -- a distributed SCM
mod_auth_openidc     2.3                                          Apache module suporting OpenID Connect authentication
mysql                8.0 [d][x]      client, server [d]           MySQL Module
nginx                1.14 [d][e]     common [d]                   nginx webserver
nginx                1.16            common [d]                   nginx webserver
nginx                1.18            common [d]                   nginx webserver
nginx                1.20            common [d]                   nginx webserver
nodejs               10 [d]          common [d], development, min Javascript runtime
                                     imal, s2i
nodejs               12              common [d], development, min Javascript runtime
                                     imal, s2i
nodejs               14              common [d], development, min Javascript runtime
                                     imal, s2i
nodejs               16              common [d], development, min Javascript runtime
                                     imal, s2i
parfait              0.5             common                       Parfait Module
perl                 5.24            common [d], minimal          Practical Extraction and Report Language
perl                 5.26 [d][e]     common [d], minimal          Practical Extraction and Report Language
perl                 5.30            common [d], minimal          Practical Extraction and Report Language
perl                 5.32            common [d], minimal          Practical Extraction and Report Language
perl-App-cpanminus   1.7044 [d]      common [d]                   Get, unpack, build and install CPAN modules
perl-DBD-MySQL       4.046 [d]       common [d]                   A MySQL interface for Perl
perl-DBD-Pg          3.7 [d]         common [d]                   A PostgreSQL interface for Perl
perl-DBD-SQLite      1.58 [d]        common [d]                   SQLite DBI driver
perl-DBI             1.641 [d]       common [d]                   A database access API for Perl
perl-FCGI            0.78 [d]        common [d]                   FastCGI Perl bindings
perl-IO-Socket-SSL   2.066 [d][e]    common [d]                   Perl library for transparent TLS
perl-YAML            1.24 [d]        common [d]                   Perl parser for YAML
perl-libwww-perl     6.34 [d][e]     common [d]                   A Perl interface to the World-Wide Web
php                  7.2 [d]         common [d], devel, minimal   PHP scripting language
php                  7.3             common [d], devel, minimal   PHP scripting language
php                  7.4 [e]         common [d] [i], devel, minim PHP scripting language
                                     al
php                  8.0             common [d], devel, minimal   PHP scripting language
pki-core             10.6                                         PKI Core module for PKI 10.6 or later
pki-deps             10.6                                         PKI Dependencies module for PKI 10.6 or later
pmdk                 1_fileformat_v6                              Persistent Memory Development Kit (former NVML)
postgresql           9.6             client, server [d]           PostgreSQL server and client module
postgresql           10 [d]          client, server [d]           PostgreSQL server and client module
postgresql           12              client, server [d]           PostgreSQL server and client module
postgresql           13              client, server [d]           PostgreSQL server and client module
python27             2.7 [d]         common [d]                   Python programming language, version 2.7
python36             3.6 [d][e]      build, common [d]            Python programming language, version 3.6
python38             3.8 [d]         build, common [d]            Python programming language, version 3.8
python39             3.9 [d]         build, common [d]            Python programming language, version 3.9
redis                5 [d]           common [d]                   Redis persistent key-value database
redis                6               common [d]                   Redis persistent key-value database
rhn-tools            1.0 [d]         common [d]                   ULN tools for OL
ruby                 2.5 [d]         common [d]                   An interpreter of object-oriented scripting language
ruby                 2.6             common [d]                   An interpreter of object-oriented scripting language
ruby                 2.7             common [d]                   An interpreter of object-oriented scripting language
ruby                 3.0             common [d]                   An interpreter of object-oriented scripting language
rust-toolset         rhel8 [d]       common [d]                   Rust
satellite-5-client   1.0 [d]         common [d], gui              Red Hat Satellite 5 client packages
scala                2.10 [d]        common [d]                   A hybrid functional/object-oriented language for the JVM
squid                4 [d]           common [d]                   Squid - Optimising Web Delivery
subversion           1.10 [d]        common [d], server           Apache Subversion
subversion           1.14            common [d], server           Apache Subversion
swig                 3.0 [d]         common [d], complete         Connects C/C++/Objective C to some high-level programming languages
swig                 4.0             common [d], complete         Connects C/C++/Objective C to some high-level programming languages
varnish              6 [d]           common [d]                   Varnish HTTP cache
virt                 rhel [d]        common [d]                   Virtualization module

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

And when we try dnf update:

# dnf update -y
Updating Subscription Management repositories.
EPEL-EL8                                                                                                                                                                                                                                                 51 kB/s | 2.3 kB     00:00
Puppet6 EL8                                                                                                                                                                                                                                              39 kB/s | 1.7 kB     00:00
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)                                                                                                                                                                                                 61 kB/s | 2.9 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                                                                 60 kB/s | 2.9 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                                                                    53 kB/s | 2.6 kB     00:00
Error:
 Problem 1: package perl-TermReadKey-2.37-7.el8.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install the best update candidate for package perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install the best update candidate for package perl-TermReadKey-2.37-7.el8.x86_64
 Problem 2: package perf-4.18.0-372.19.1.el8_6.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Carp-1.50-439.module+el8.6.0+13324+628a2397.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Carp-1.42-396.el8.noarch
  - cannot install the best update candidate for package perf-4.18.0-372.19.1.el8_6.x86_64
 Problem 3: package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Data-Dumper-2.174-440.module+el8.6.0+13324+628a2397.x86_64 requires libperl.so.5.32()(64bit), but none of the providers can be installed
  - package perl-Data-Dumper-2.174-440.module+el8.6.0+13324+628a2397.x86_64 requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Data-Dumper-2.167-399.el8.x86_64
  - problem with installed package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64
 Problem 4: package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Digest-1.20-1.module+el8.6.0+13324+628a2397.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64
  - cannot install the best update candidate for package perl-Digest-1.17-395.el8.noarch
 Problem 5: perl-libs-4:5.26.3-421.el8.i686 has inferior architecture
  - package perl-Mozilla-CA-20160104-7.module+el8.3.0+6498+9eecfe51.noarch requires perl(:MODULE_COMPAT_5.26.3), but none of the providers can be installed
  - perl-libs-4:5.26.3-417.el8_3.i686 has inferior architecture
  - perl-libs-4:5.26.3-419.el8_4.1.i686 has inferior architecture
  - perl-libs-4:5.26.3-419.el8.i686 has inferior architecture
  - perl-libs-4:5.26.3-420.el8.i686 has inferior architecture
  - perl-libs-4:5.26.3-416.el8.i686 has inferior architecture
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Digest-MD5-2.58-1.module+el8.6.0+13324+628a2397.x86_64 requires libperl.so.5.32()(64bit), but none of the providers can be installed
  - package perl-Digest-MD5-2.58-1.module+el8.6.0+13324+628a2397.x86_64 requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Mozilla-CA-20160104-7.module+el8.3.0+6498+9eecfe51.noarch
  - cannot install the best update candidate for package perl-Digest-MD5-2.55-396.el8.x86_64
 Problem 6: problem with installed package perf-4.18.0-372.19.1.el8_6.x86_64
  - package perf-4.18.0-372.19.1.el8_6.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Encode-4:3.08-461.module+el8.6.0+13324+628a2397.x86_64 requires libperl.so.5.32()(64bit), but none of the providers can be installed
  - package perl-Encode-4:3.08-461.module+el8.6.0+13324+628a2397.x86_64 requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Encode-4:2.97-3.el8.x86_64
 Problem 7: package perl-IO-Socket-SSL-2.066-4.module+el8.3.0+6446+594cad75.noarch requires perl(Net::SSLeay) >= 1.46, but none of the providers can be installed
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - package perl-Net-SSLeay-1.88-1.module+el8.3.0+6446+594cad75.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Errno-1.30-471.module+el8.6.0+13324+628a2397.x86_64 requires perl-libs(x86-64) = 4:5.32.1-471.module+el8.6.0+13324+628a2397, but none of the providers can be installed
  - package perl-Errno-1.30-471.module+el8.6.0+13324+628a2397.x86_64 requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-IO-Socket-SSL-2.066-4.module+el8.3.0+6446+594cad75.noarch
  - cannot install the best update candidate for package perl-Errno-1.28-421.el8.x86_64
  - package perl-Net-SSLeay-1.88-1.el8.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+30f09725.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+09a687b6.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-1.module+el8.3.0+6452+449fe210.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-1.module+el8.3.0+6446+37a50855.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+6b8485cb.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.85-6.el8.x86_64 is filtered out by modular filtering
 Problem 8: package git-2.31.1-2.el8.x86_64 requires perl(Term::ReadKey), but none of the providers can be installed
  - package perl-TermReadKey-2.37-7.el8.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Exporter-5.74-458.module+el8.6.0+13324+628a2397.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Exporter-5.72-396.el8.noarch
  - cannot install the best update candidate for package git-2.31.1-2.el8.x86_64
 Problem 9: problem with installed package perl-TermReadKey-2.37-7.el8.x86_64
  - package perl-TermReadKey-2.37-7.el8.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-File-Path-2.16-439.module+el8.6.0+13324+628a2397.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-File-Path-2.15-2.el8.noarch
 Problem 10: package perl-Git-2.31.1-2.el8.noarch requires git = 2.31.1-2.el8, but none of the providers can be installed
  - package git-2.31.1-2.el8.x86_64 requires perl(Term::ReadKey), but none of the providers can be installed
  - package perl-TermReadKey-2.37-7.el8.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-File-Temp-1:0.231.100-1.module+el8.6.0+13324+628a2397.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Git-2.31.1-2.el8.noarch
  - cannot install the best update candidate for package perl-File-Temp-0.230.600-1.el8.noarch
 Problem 11: problem with installed package git-2.31.1-2.el8.x86_64
  - package git-2.31.1-2.el8.x86_64 requires perl(Term::ReadKey), but none of the providers can be installed
  - package perl-TermReadKey-2.37-7.el8.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-Getopt-Long-1:2.52-1.module+el8.6.0+13324+628a2397.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-Getopt-Long-1:2.50-4.el8.noarch
 Problem 12: problem with installed package perl-IO-Socket-SSL-2.066-4.module+el8.3.0+6446+594cad75.noarch
  - package perl-IO-Socket-SSL-2.066-4.module+el8.3.0+6446+594cad75.noarch requires perl(Net::SSLeay) >= 1.46, but none of the providers can be installed
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - package perl-Net-SSLeay-1.88-1.module+el8.3.0+6446+594cad75.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-HTTP-Tiny-0.078-1.module+el8.6.0+13324+628a2397.noarch requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-HTTP-Tiny-0.074-1.el8.noarch
  - package perl-Net-SSLeay-1.88-1.el8.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+30f09725.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+f0897f98.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+09a687b6.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-1.module+el8.3.0+6452+449fe210.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-1.module+el8.3.0+6446+37a50855.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.88-2.module+el8.6.0+13392+6b8485cb.x86_64 is filtered out by modular filtering
  - package perl-Net-SSLeay-1.85-6.el8.x86_64 is filtered out by modular filtering
 Problem 13: problem with installed package perl-Git-2.31.1-2.el8.noarch
  - package perl-Git-2.31.1-2.el8.noarch requires git = 2.31.1-2.el8, but none of the providers can be installed
  - package git-2.31.1-2.el8.x86_64 requires perl(Term::ReadKey), but none of the providers can be installed
  - package perl-TermReadKey-2.37-7.el8.x86_64 requires libperl.so.5.26()(64bit), but none of the providers can be installed
  - cannot install both perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64 and perl-libs-4:5.26.3-421.el8.x86_64
  - cannot install both perl-libs-4:5.26.3-421.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-417.el8_3.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8_4.1.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-419.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-420.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - cannot install both perl-libs-4:5.26.3-416.el8.x86_64 and perl-libs-4:5.32.1-471.module+el8.6.0+13324+628a2397.x86_64
  - package perl-IO-1.43-471.module+el8.6.0+13324+628a2397.x86_64 requires libperl.so.5.32()(64bit), but none of the providers can be installed
  - package perl-IO-1.43-471.module+el8.6.0+13324+628a2397.x86_64 requires perl(:MODULE_COMPAT_5.32.1), but none of the providers can be installed
  - cannot install the best update candidate for package perl-IO-1.38-421.el8.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

This is one of the systems where the update went fine just two days before.
I know 2.5 is unsupported for a while now, but maybe someone knows a solution to this or knows whether this is a known bug in our version thats fixed in more recent releases.

Regards

areyus · August 11, 2022, 3:14pm

We managed to dig a little deeper and found the following behaviour:

When we enable the default stream for a module, we can install that module just fine. E.g. dnf module enable mariadb; dnf module install mariadb:10.3/server works just fine and installs mariadb 10.3 (the default stream). Yet at the same time, directly after installing, running dnf update results in the error mentioned in the first post.
If we try to enable a different stream, the system will just stop seeing any packages from that module. Continuing with the example, dnf module enable mariadb:10.5 works, but dnf module install mariadb:10.5 results in the following error:

# dnf module install mariadb:10.5
Updating Subscription Management repositories.
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)                                                                                                                                                                                                 86 kB/s | 4.5 kB     00:00
EPEL-EL8                                                                                                                                                                                                                                                 48 kB/s | 2.3 kB     00:00
Puppet6 EL8                                                                                                                                                                                                                                              35 kB/s | 1.7 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                                                                    52 kB/s | 2.6 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                                                                 58 kB/s | 2.9 kB     00:00
Unable to resolve argument mariadb:10.5
No match for package mariadb-server
Error: Problems in request:
broken groups or modules: mariadb:10.5

Searching also does not show any packages from the module anymore:

# dnf search mariadb
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:28 ago on Thu 11 Aug 2022 03:04:44 PM CEST.
=========================================================================================================================== Name & Summary Matched: mariadb ============================================================================================================================mariadb-connector-c.i686 : The MariaDB Native Client library (C driver)
mariadb-connector-c.x86_64 : The MariaDB Native Client library (C driver)
mariadb-connector-c-devel.x86_64 : Development files for mariadb-connector-c
mariadb-connector-c-devel.i686 : Development files for mariadb-connector-c
mariadb-connector-odbc.x86_64 : The MariaDB Native Client library (ODBC driver)
mariadb-java-client.noarch : Connects applications developed in Java to MariaDB and MySQL databases
================================================================================================================================ Name Matched: mariadb =================================================================================================================================mariadb-connector-c-config.noarch : Configuration files for packages that use /etc/my.cnf as a configuration file
=============================================================================================================================== Summary Matched: mariadb ===============================================================================================================================anope-mysql.x86_64 : MariaDB/MySQL modules for Anope IRC services
mysql-selinux.noarch : SELinux policy modules for MySQL and MariaDB packages
python3-mysqlclient.x86_64 : MySQL/mariaDB database connector for Python

# dnf repoquery mariadb-server
Updating Subscription Management repositories.
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)                                                                                                                                                                                                 93 kB/s | 4.5 kB     00:00
EPEL-EL8                                                                                                                                                                                                                                                 51 kB/s | 2.3 kB     00:00
Puppet6 EL8                                                                                                                                                                                                                                              38 kB/s | 1.7 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                                                                    56 kB/s | 2.6 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                                                                 58 kB/s | 2.9 kB     00:00

The packages returned by dnf search are all non-modular as far as I am able to tell.

I also found this RedHat KB which describes the same Problem, but we are not using filters on our CVs at all.
We also checked the modules.yaml metadata file and it at least looks to be fine.

Any pointers would be really appreciated.

areyus · August 12, 2022, 8:01am

We managed to at least find a solution to recover to a working state.
Regenerating the metadata on the affected repository (rhel8 appstream), the republishing the metadata on both CV and CCV and doing a smart-proxy sync returned the repos to a working state, so it looks like something got messed up inside pulp.
We’re still looking for the underlying problem and/or a way to fix it. Currently we are just hoping it will stop popping up after updating to a more recent release.