Applicable rpms listed from inactive module streams

Problem:
Since migrating to pulp3 and upgrading to katello 4.0 all our centos8 server show a count of installable updates even though those packages are not available in the enabled module stream.

Currently, we have the perl 5.26 module stream enable on those server:

# dnf module list perl*
...
CentOS-8 - AppStream
Name                          Stream                 Profiles                      Summary                                              
perl                          5.24                   common [d], minimal           Practical Extraction and Report Language             
perl                          5.26 [d][e]            common [d], minimal           Practical Extraction and Report Language             
perl                          5.30                   common [d], minimal           Practical Extraction and Report Language             
perl-App-cpanminus            1.7044 [d]             common [d]                    Get, unpack, build and install CPAN modules          
perl-DBD-MySQL                4.046 [d][e]           common [d]                    A MySQL interface for Perl                           
perl-DBD-Pg                   3.7 [d]                common [d]                    A PostgreSQL interface for Perl                      
perl-DBD-SQLite               1.58 [d]               common [d]                    SQLite DBI driver                                    
perl-DBI                      1.641 [d][e]           common [d]                    A database access API for Perl                       
perl-FCGI                     0.78 [d]               common [d]                    FastCGI Perl bindings                                
perl-IO-Socket-SSL            2.066 [d][e]           common [d]                    Perl library for transparent TLS                     
perl-YAML                     1.24 [d]               common [d]                    Perl parser for YAML                                 
perl-libwww-perl              6.34 [d][e]            common [d]                    A Perl interface to the World-Wide Web               

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

For instance, we have the perl-libwww-perl installed:

# rpm -q perl-libwww-perl
perl-libwww-perl-6.34-1.module_el8.3.0+416+dee7bcef.noarch

With Katello 4 it suggest perl-libwww-perl-6.34-1.module_el8.3.0+416+9a1a0b3f.noarch as installable package. However that package requires ā€˜perl(:MODULE_COMPAT_5.30.1)ā€™, i.e. itā€™s for the perl:5.30 module stream and not for the perl:5.26. Itā€™s not offered as update on the server and itā€™s not possible to install it either because of that dependency.

With Katello 3.18.2 and pulp2 all those servers were correctly listed as up-to-date.

Expected outcome:
List only applicable/installable packages which are really available in the currently enabled module streams.

Foreman and Proxy versions:
Foreman 2.4, Katello 4.0

I see the exact same issue with katello 3.18.2 after upgrading to pulp3.

After a detour I just yesterday migrated and updated from Katello 3.18.2 with pulp2 to pulp3 to katello 4.0.1 again.

The applicability for packages from module streams is still broken. For instance, it shows perl-DBI-1.641-3.module_el8.1.0+199+249f9f29.x86_64 as applicable.

On the server, I have the perl-5.26 module enabled. I have the rpm perl-DBI-1.641-3.module_el8.1.0+199+8f0a6bbd.x86_64 installed which is for 5.26.

The suggested rpm perl-DBI-1.641-3.module_el8.1.0+199+249f9f29.x86_64 is not available on the server as that rpm is for the perl 5.24 module.

With 3.18/pulp2 the server did not show any applicable updates.

Created issue Bug #32739: Incorrect applicablity in katello 4 - Katello - Foreman

Bug #32739: Incorrect applicablity in katello 4 - Katello - Foreman has been set to ā€œneed more informationā€. I have asked a couple of days what is needed by got not response.

On issue Bug #32739: Incorrect applicablity in katello 4 - Katello - Foreman there is mention of @iballou and @Justin_Sherrill

@jjeffers originally set it to ā€œneed more informationā€.

Anyone has some insights?

Weā€™ve had this on our immediate backlog for a while. Iā€™m hoping we can get to it soon.

How to reproduce: with any CentOS 8, e.g. minimal installation. We always have perl-LDAP installed. This enables modules perl 5.26, perl-IO-Socket-SSL 2.066 and perl-libwww-perl 6.34 and installs the packages. But then I see 23 upgradable packages listed for the content host.

Upgradable Package
	perl-Data-Dump-1.23-7.module_el8.3.0+416+9a1a0b3f.noarch
	perl-Digest-HMAC-1.03-17.module_el8.3.0+416+9a1a0b3f.noarch
	perl-Encode-Locale-1.05-10.module_el8.3.0+416+9a1a0b3f.noarch
	perl-File-Listing-6.04-17.module_el8.3.0+416+9a1a0b3f.noarch
	perl-HTML-Parser-3.72-15.module_el8.3.0+416+9a1a0b3f.x86_64
	perl-HTML-Tagset-3.20-34.module_el8.3.0+416+9a1a0b3f.noarch
	perl-HTTP-Cookies-6.04-2.module_el8.3.0+416+9a1a0b3f.noarch
	perl-HTTP-Date-6.02-19.module_el8.3.0+416+9a1a0b3f.noarch
	perl-HTTP-Message-6.18-1.module_el8.3.0+416+9a1a0b3f.noarch
	perl-HTTP-Negotiate-6.01-19.module_el8.3.0+416+9a1a0b3f.noarch
	perl-IO-HTML-1.001-11.module_el8.3.0+416+9a1a0b3f.noarch
	perl-IO-Socket-SSL-2.066-4.module_el8.3.0+410+3b5aa49a.noarch
	perl-IO-Socket-SSL-2.066-4.module_el8.3.0+410+4cc2efa4.noarch
	perl-libwww-perl-6.34-1.module_el8.3.0+416+9a1a0b3f.noarch
	perl-LWP-MediaTypes-6.02-15.module_el8.3.0+416+9a1a0b3f.noarch
	perl-Mozilla-CA-20160104-7.module_el8.3.0+416+9a1a0b3f.noarch
	perl-Net-HTTP-6.17-2.module_el8.3.0+416+9a1a0b3f.noarch
	perl-Net-SSLeay-1.88-1.module_el8.3.0+410+3b5aa49a.x86_64
	perl-Net-SSLeay-1.88-1.module_el8.3.0+410+4cc2efa4.x86_64
	perl-NTLM-1.09-17.module_el8.3.0+416+9a1a0b3f.noarch
	perl-TimeDate-1:2.30-15.module_el8.3.0+416+9a1a0b3f.noarch
	perl-Try-Tiny-0.30-7.module_el8.3.0+416+9a1a0b3f.noarch
	perl-WWW-RobotRules-6.02-18.module_el8.3.0+416+9a1a0b3f.noarch

If you cannot reproduce it this way, then I guess it would be some bogus data from the migration in the databaseā€¦

@gvde I have two PRs out that should fix this together. One of them is a patch to subscription-manager. Turns out certain module streams (like perl-IO-Socket-SSL) have multiple ā€œenabledā€ sibling streams but only one is actually active. I thought this was a dnf bug until I talked to the RHEL 8 folks. Thankfully we can find the active one using the dnf library, so weā€™re having sub-man send over the ā€œactiveā€ status to Katello.

https://github.com/Katello/katello/pull/9473

https://github.com/candlepin/subscription-manager/pull/2708

Great. Am I right: this requires changes to the redhat subscription-manager? I guess this will take a while until it gets published thenā€¦

I do wonder, though, why it was working before with 3.18/pulp2? The active flag wasnā€™t available to Katello before, either, so how did it work then?

If I manually patch the profile.py on my CentOS 8 clients as well as the katello patch for testing, what should I do to verify itā€™s working? Is a dnf uploadprofile enough to get the information to the katello server and get the applicability recalculated?

Yep, you need to patch subscription-manager and Katello. The change is relatively small and harmless to sub-man, so Iā€™m hoping it wonā€™t take too long. If it does, weā€™ll have to solve it fully from Katello.

Pulp 2 performed all of the applicability calculations for Katello pre-Pulp 3. Now, the applicability code is in Katello. Pulp 2 didnā€™t match dnf perfectly (mostly with regards to modules I think), so we started the Katello applicability code with a blank slate back in Katello 3.16 for Pulp 3 users.

dnf uploadprofile should do it as long as you see the profile update and applicability regeneration tasks afterwards in the task browser.

I have tried to test your PR. But it doesnā€™t seem to work for me:

I have patched /usr/lib64/python3.6/site-packages/rhsm/profile.py on one of my test clients.

I have patched /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/lib/actions/katello/host/upload_profiles.rb on my main foreman server. foreman-maintain service stop and start.

I ran uploadprofile on the client once foreman was up and running again.

# dnf uploadprofile  --force-upload
Updating Subscription Management repositories.
Package profile updates
        status: 1
        updates: []
        exceptions: 
        

But I still see those perl packages as Upgradable Package in foreman on that host.

The katello PR is only for the main server? The RPM isnā€™t installed on the content proxy thus I didnā€™t apply any patch on the content proxy. My test client is connected to the content proxy, if that matters.

Iā€™ve since updated the PR, there was indeed an issue with it. Should be all good now.

And yeah, you only need to worry about patching your main Katello server and the client youā€™re testing with.

By the way, we found out this issue will really only be for perl-related module streams and their packages. It turns out that pretty much no one else uses this modularity feature where child module streams are enabled but set as inactive. Of course, someone else could start using it, but within the CentOS repositories at least itā€™s just perl.

I tested the PR and it worked for me. Thank you. ā€œdnf uploadprofileā€ was not enough (or I was too impatient?) as I still saw the applicable packages. But after removing and then reinstalling the offending perl package all was fine.

1 Like

User error: I missed part of the katello patch. After a foreman restart and a uploadprofile all works as expected.

1 Like

I havenā€™t patched my systems. I am waiting for the official update for this problem.

However, just today, with some new updates for the container-tools module I see some more odd applicability issues. I am not sure, if they are related to this and going to be fixed or not:

If I check with ā€œContent - Packagesā€ and select the ā€œApplicableā€ checkbox I see now ā€œconmon-2:2.0.29-1.module_el8.4.0+886+c9a8d9ad.x86_64ā€ listed. Now I click on the package name and it shows me ā€œApplicable To:ā€ ā€œ1 Hosts(s)ā€. So now I click on the ā€œ1 Hosts(s)ā€ link and it shows me the ā€œContent Hostsā€ page with the search for those hosts but it doesnā€™t find anything: ā€œYour search returned zero Content Hosts.ā€.

1 Like

O.K. After closer look itā€™s a frontend/encoding problem.

I have opened issue Bug #33256: Incorrect search link from packages view for applicable or upgradable hosts - Katello - Foreman

2 Likes

A few of my systems have around 60-100 packages shown as ā€œneeds updateā€, most of them are perl modules. Some from AlmaLinux, some from harbottle repo. I am glad this is just my future foreman/katello server with around ~10 hosts and not my production environment.

Hi @sjansen,

Let me know if any applicable modular RPMs are incorrect that are not part of perl. Always curious to find more that donā€™t fit the norm.

@iballou Is there an estimated timeline until those patches get into RHEL 8/CentOS 8 Stream/CentOS 8 resp. Katello?