Peculiarity with Foreman 3.2/Katello 4.4

[mhjacks]

Problem: Subscribed hosts are not showing all the repos they should be

Expected outcome: Hosts show the repos that the UI says should be enabled

Foreman and Proxy versions: 3.2.0

Foreman and Proxy plugin versions: 4.4 (Katello)

Distribution and version: CentOS 8 stream

Other relevant data:

Example subscribed system is Elrond, running F36. DNF repolist shows the following:

Meanwhile, these are the repos that should be enabled according to Foreman/Katello (e.g. updates etc):

I have CentOS 8 stream systems subscribed as well that are not showing expected repos like EPEL modular or C8s appstream. I have the community repos grouped into products, such that the EPEL8 product contains EPEL release and EPEL modular repos (for example). The other community “products” are set up similarly. An exception to this behavior is on the foreman/katello server itself where the Foreman/Katello product is showing all the repos I’d expect (Foreman, -plugins, Katello, candlepin, pulpcore).

I am using the Ansible modules to manage content etc (theforeman.foreman collection v3.2.0), and recently refreshed some repo definitions after the Fedora 36 release.

What seems to be happening at first glance is that subscription-manager (and redhat.repo etc) are picking the first listed repo from the “subscription”. I suspect something in the foreman/katello stack as I am seeing this across F36/CentOS 8 stream systems

[jeremylenz]

Hi @mhjacks

I’m curious if this output changes at all after running (on the host) subscription-manager refresh or dnf upload-profile ?

[mhjacks]

Thanks for looking at this! These kinds of things seem not to change the client’s view of available repos, which is super-perplexing to me:

# dnf repolist
Updating Subscription Management repositories.
repo id                                                          repo name
imladris_f36_x86_64_release                                      Fedora 36 x86_64 Release
imladris_google_chrome_chrome_x86_64                             Google Chrome
imladris_ms_vscode_release                                       Visual Studio Code
imladris_rpmfusion_f36_x86_64_free_release                       RPMFusion Free f36 x86_64 Release
root@elrond:~
# dnf upload-profile
Updating Subscription Management repositories.
No updates performed. See /var/log/rhsm/rhsm.log for more information.
root@elrond:~
# dnf repolist
Updating Subscription Management repositories.
repo id                                                          repo name
imladris_f36_x86_64_release                                      Fedora 36 x86_64 Release
imladris_google_chrome_chrome_x86_64                             Google Chrome
imladris_ms_vscode_release                                       Visual Studio Code
imladris_rpmfusion_f36_x86_64_free_release                       RPMFusion Free f36 x86_64 Release

[iballou]

The first thing I like to check are the host’s “bound repositories”:

# foreman-rake console

::Host.find(<host id>).content_facet.bound_repositories

Do they match the repositories that the host is reporting, or the repositories that Katello is reporting in the UI?

Are you able to check repositories that are enabled on the host from Katello at all? What happens if you added a test repository for example?

I suppose it could end up being a subscription-manager bug if the repo file isn’t being updated correctly.

[mhjacks]

I get this:

#<ActiveRecord::Associations::CollectionProxy [#<Katello::Repository id: 80569, pulp_id: "1-Fedora_36_x86_64_Multimedia-Library-018086e3-403...", library_instance_id: 19, content_view_version_id: 18055, relative_path: "imladris/Library/Fedora_36_x86_64_Multimedia/custo...", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 19, remote_href: nil, publication_href: "/pulp/api/v3/publications/rpm/rpm/7c36910f-5f6a-49...", version_href: "/pulp/api/v3/repositories/rpm/rpm/ff6ce17e-d8c2-4b...", last_contents_changed: "2022-03-21 04:08:55", last_applicability_regen: "1970-01-01 00:00:00", last_indexed: "2022-05-12 19:47:44">, #<Katello::Repository id: 80570, pulp_id: "1-Fedora_36_x86_64_Multimedia-Library-8f135f4f-974...", library_instance_id: 20, content_view_version_id: 18055, relative_path: "imladris/Library/Fedora_36_x86_64_Multimedia/custo...", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 20, remote_href: nil, publication_href: "/pulp/api/v3/publications/rpm/rpm/311a059f-5fb6-43...", version_href: "/pulp/api/v3/repositories/rpm/rpm/affbe91d-0624-43...", last_contents_changed: "2022-03-21 04:08:55", last_applicability_regen: "1970-01-01 00:00:00", last_indexed: "2022-05-12 19:47:44">, #<Katello::Repository id: 80565, pulp_id: "1-Fedora_36_x86_64_Multimedia-Library-69519d12-e83...", library_instance_id: 80459, content_view_version_id: 18055, relative_path: "imladris/Library/Fedora_36_x86_64_Multimedia/custo...", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 93, remote_href: nil, publication_href: "/pulp/api/v3/publications/rpm/rpm/ca3fb4c6-edcb-42...", version_href: "/pulp/api/v3/repositories/rpm/rpm/30ddf12c-f209-4d...", last_contents_changed: "2022-03-21 04:08:54", last_applicability_regen: "1970-01-01 00:00:00", last_indexed: "2022-05-12 19:47:44">, #<Katello::Repository id: 80560, pulp_id: "1-Fedora_36_x86_64_Multimedia-Library-ffc40cab-682...", library_instance_id: 80454, content_view_version_id: 18055, relative_path: "imladris/Library/Fedora_36_x86_64_Multimedia/custo...", environment_id: 1, saved_checksum_type: nil, distribution_version: "36", distribution_arch: "x86_64", distribution_bootable: true, distribution_family: "Fedora", distribution_variant: "Everything", container_repository_name: nil, root_id: 88, remote_href: nil, publication_href: "/pulp/api/v3/publications/rpm/rpm/9ceb8c92-a46e-46...", version_href: "/pulp/api/v3/repositories/rpm/rpm/68dc2960-5e77-49...", last_contents_changed: "2022-03-21 04:08:59", last_applicability_regen: "1970-01-01 00:00:00", last_indexed: "2022-05-12 19:47:55">]>

Which seems not to match the repolist I posted a screenshot of above there. I don’t suspect a subscription-manager bug because I’m seeing the same kind of problem on my centos-8-stream hosts (which is on subman 1.28.28), and such a thing would be quite a coincidence.

What might be interesting is that when looking at the repolists on the hosts screens in katello, even if I override repos to “enabled” (that should be enabled by default due to activation key) the repo does not show up in subman/repolist. Seems like some linkage is broken somewhere?

[gvde]

Is it possible you have assigned a content view and lifecycle environment to your hosts and you didn’t publish and promote the content view to the life cycle environment?

You can also check on the client what subscriptions and repos it sees:

# subscription-manager list --consumed
# subscription-manager list --all --available

The former lists the subscriptions assigned/consumed the latter everything known.

# subscription-manager repos

lists all repositories visible as well as whether they are enabled or not. It also shows the repositories URLs used, however you have to add a ‘/’ at the end to see the content if you check in your browser.

[mhjacks]

The only environment I’m using is Library. The point of me reporting the issue is that what I’m seeing on the clients (from multiple different versions of subscription manager) does not match what I see on the corresponding host page in Foreman/Katello. The subman side is seeing fewer repos. The ::Host.find mechanism @iballou showed above seems to show the same thing as subman does.

I know how to check repos available and visible - my question is why there seems to be a discrepancy between what my host detail shows and what subman reports.

[gvde]

You didn’t answer about the content views… That would be an obvious reason for it to be missing. If you had posted the output of those commands it would have been clearer…

[mhjacks]

OK:

Subscription-manager list consumed (for elrond):

+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Fedora 36 x86_64
Provides:            Fedora 36 x86_64
SKU:                 173305444199
Contract:            
Account:             
Serial:              6374058369114177803
Pool ID:             402885a87fa770d5017faa624fa20013
Provides Management: No
Active:              True
Quantity Used:       1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              03/20/2022
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   RPMFusion F36 x86_64
Provides:            RPMFusion F36 x86_64
SKU:                 510821768550
Contract:            
Account:             
Serial:              4538100158436890142
Pool ID:             402885a87fa770d5017faa62602c0015
Provides Management: No
Active:              True
Quantity Used:       1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              03/20/2022
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Microsoft VSCode
Provides:            Microsoft VSCode
SKU:                 225821757096
Contract:            
Account:             
Serial:              7265364219463722780
Pool ID:             402885a87bd2eb69017bd2fdfd390012
Provides Management: No
Active:              True
Quantity Used:       1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Imladris Local Packages
Provides:            Imladris Local Packages
SKU:                 418260279595
Contract:            
Account:             
Serial:              7499682036253348027
Pool ID:             402885a87fc46d78017fd38f7e5a00c2
Provides Management: No
Active:              True
Quantity Used:       1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              03/28/2022
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Google Chrome
Provides:            Google Chrome
SKU:                 903280834997
Contract:            
Account:             
Serial:              54252013618758036
Pool ID:             402885a87bd2eb69017bd2fdf21f0010
Provides Management: No
Active:              True
Quantity Used:       1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

The repos missing are from the F36 and F36 RPMfusion “products” - that’s where I put updates, h264, modular, modular updates and free and nonfree, and free-updates and nonfree-updates respectively. Similar things are true of the other “bases” - c8s base has baseos, appstream, and powertools; yet in subman I only see one of those repos (baseos). EPEL 8 has epel and epel-modular, and I only see epel in subman.

subscription-manager list --all --available shows a number of additional “products”, which are all in library and are expected to be there. The problem isn’t that the products aren’t showing up, it’s that I’m only getting one repository from each of the products, and for the base f36, c8s, c9s, I’m expecting 3 or more repositories.

    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Fedora 35 x86_64
Provides:            
SKU:                 170100407851
Contract:            
Pool ID:             402885a87c159776017c159a66590001
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/23/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   RPMFusion F35 x86_64
Provides:            
SKU:                 198950339414
Contract:            
Pool ID:             402885a87c159776017c159a82b90003
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/23/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   PGDG PostgreSQL Packages
Provides:            
SKU:                 931043459705
Contract:            
Pool ID:             402885a87dcde956017dd56ea5a30031
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              12/19/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   EPEL 9 x86_64
Provides:            
SKU:                 979389216336
Contract:            
Pool ID:             402885a87d83468a017d884ffcde0039
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              12/04/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Ansible Release
Provides:            
SKU:                 113563007768
Contract:            
Pool ID:             402885a87bd2eb69017bd2fe292d0018
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Puppet7 EL8 x86_64
Provides:            
SKU:                 652319862641
Contract:            
Pool ID:             402885a87bd2eb69017bd2fe1f380016
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Puppet7 EL9 x86_64
Provides:            
SKU:                 339232146837
Contract:            
Pool ID:             402885a87f7a3a7d017f7f2049330488
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              03/12/2022
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Docker-CE
Provides:            
SKU:                 869224975369
Contract:            
Pool ID:             402885a87ce43dde017d02604b8d00db
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              11/08/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   TheForeman EL8 x86_64
Provides:            
SKU:                 406088115778
Contract:            
Pool ID:             402885a87bd2eb69017bd2fe3f7e001a
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   CentOS-Stream 8 x86_64
Provides:            
SKU:                 24036363964
Contract:            
Pool ID:             402885a87bd2eb69017bd2fddc1b000c
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   CentOS-Stream 9 x86_64
Provides:            
SKU:                 583045888174
Contract:            
Pool ID:             402885a87d816033017d82c01afe000d
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              12/03/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   EPEL 8 x86_64
Provides:            
SKU:                 261661334712
Contract:            
Pool ID:             402885a87bd2eb69017bd2fde6f0000e
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

Subscription Name:   Puppet7 F34 x86_64
Provides:            
SKU:                 460768654893
Contract:            
Pool ID:             402885a87bd2eb69017bd2fe07dd0014
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              09/10/2021
Ends:                11/30/2049
Entitlement Type:    Physical

As far as content views go, elrond has f36-multimedia, which looks like this:

And the component CVs have the following repo lists (let’s just focus on f36 base since that illustrates the problem, I think):

I believe that I should be seeing more repos syncing on the subman side - the things I’m seeing in the Foreman/Katello interface match what I expect, but subman and ::Host.find don’t. I’m not sure where to look for the disconnect. (And if I’ve misunderstood anything about how I should be managing content, I’m happy to fix it).

Thanks for any input - I have the capacity to build another parallel instance fresh, and I’m considering doing that.

[JendaVodka]

Please check if this is not related to problems described here:

Repositories not listed by subscription-manager (Satellite)

Hello, we faced similar problem but I can not tell from your description if this is your case. In short: our problem was caused by some known bug with missing records in candlepin table, see: Custom repository is not available on the client system even after attaching the right subscription. - Red Hat Customer Portal The workaround depicted in the mentioned article fixed it for us, with several repositories the task failed but it could be fixed with simple renaming of the product and running…

[jeremylenz]

@gvde

Is it possible you have assigned a content view and lifecycle environment to your hosts and you didn’t publish and promote the content view to the life cycle environment?

It looks like you’re using a composite CV. In order for the host’s bound repositories to reflect those content view screenshots, both the component and composite CV must be published after any changes to them. Can you share what content view and version is assigned to the host?

On the content view screen for the composite CV, if you look at the ‘Versions’ tab > ‘Repositories’ section for the version your host is using, I would expect that to match what you’re seeing in the sub-man output.

[mhjacks]

Yes -this seems to match the symptoms. The candlepin table is only showing the repos that subman does. I’ll continue down the road this bug report suggests. Thanks!

FYI the quotes in the query above got converted to “non-quote quotes” so I had to replace them in psql.

[mhjacks]

To follow up: After rct cat-cert’ing several of my entitlements, this seems to exactly be the problem - the certs are missing the repos I’m expecting to be there, which matches what candlepin is saying and what Katello ::Host.find says.

[mhjacks]

Entitlement certificate not containing all content - #2 by Dirk contains a workaround for this issue. I’ve run the script it links (on el8, need to change the shebang line from tfm-ruby to just ruby), but it seems to detect the problem correctly and looks like it would fix it. I haven’t run the script because I’m working with some other red hatters on more diagnostics.

The repo with the script is here: GitHub - ATIX-AG/orcharhino-scripts: Utility scripts for the orcharhino-server. Might also work on foreman-/katello-servers..

I’ll mark this post as a solution and hopefully we can find something more permanent in candlepin itself.