Katello unable to show active subscriptions on some hosts with web and hammer cli 3.0/4.2rc3

Support
1

Problem:

Katello suddently stopped listing installed subscriptions on new registered clients. Subscriptions can be added by GUI and hammer cli but they do not show up in Web or with “hammer host subscription product-content”. I can remove subscriptions only by hammer cli by using the product id.

Expected outcome:
Katello GUI or hammer cli to show subscribed content of a host

Foreman and Proxy versions:
foreman-3.0.0-1.el8.noarch
katello-4.2.0.1-0.1.rc3.el8.noarch
foreman-proxy-3.0.0-1.el8.noarch

Distribution and version:
CentOS 8.4

Other relevant data:

Adding a subscription to my host

[root@fm /]# hammer host subscription attach --host XXX --subscription-id 8ab19e8b7a2f3e33017a3385b6f600eb
Subscription attached to the host successfully.

Show Subscription Status on this host

[root@fm /]# hammer host subscription product-content --host XXX
—|------|------|-----|---------|-------|------------------|---------

ID NAME TYPE URL GPG KEY LABEL DEFAULT ENABLED? OVERRIDE

Nothing here? no base subscriptions and no trace of the newly added subscription. Same on Web.

Check if client gets this repo:

[root@xxx /]# subscription-manager repos | grep Microsoft
Repo ID: XXX_Microsoft_Prod_el8_msprod
Repo URL: https://foreman/pulp/content/myorg/Library/al8/custom/Microsoft_Prod/el8_msprod

Yep, that repo is here and dnf update also loading the metadata of it.

Remove subscription on my host

[root@fm /]# hammer host subscription remove --host XXX --subscription-id 8ab19e8b7a2f3e33017a3385b6f600eb
Subscription removed from the host successfully.

Check if this repo disappear on the client

[root@xxx /]# subscription-manager repos | grep Microsoft
[root@xxx /]#

as expected, dnf update no longer shows this repo.

Again lets see what hammer give us about the subscription status

[root@fm /]# hammer host subscription product-content --host XXX
—|------|------|-----|---------|-------|------------------|---------

ID NAME TYPE URL GPG KEY LABEL DEFAULT ENABLED? OVERRIDE

Still nothing.

Conclusion: adding and removing subscriptions work but katello suddently started to no longer show up subscription status on newly added hosts.

I added around 60 new hosts on sunday and all worked well and subscriptions status still show up for these hosts. Today every host i add show no subscriptions on the hosts. Yesterday i did only one thing on this installation, pressing import roles in the ansible roles window as a preperation to work with ansible in foreman, nothing got assigned to any hosts.

2

One thing i forgot, if i go to Content / Subscriptions and select a subscription > associations > contents hosts, the affacted hosts are missing here (but are subscribed and can use the subscriptions)

3

Hey @sjansen, can you try a few things out?

First, sharing the output of hammer ping would be helpful to ensure all the backend services are working OK.

Second, using foreman-tail on the server, attempt to remove and re-add the subscription(s) that aren’t showing as expected. Do you see any errors in the log? If so, paste them here.

Lastly, you can use a tool on the server to hopefully set everything into place: foreman-rake katello:import_subscriptions Using foreman-tail while doing this may surface some hidden errors as well. Please share the output of the import_subscriptions command.

Is this happening for only a specific subscription, or all of them? And are they for custom products or products from your Red Hat manifest (if you’re using one)?

Sorry for all the questions - but they’ll help pinpoint the issue especially since I don’t see this problem in my own environment. Thank you :slight_smile:

1 Like
4

Hi Jonathon,

hammer ping looks good.

[root@fm /]# hammer ping
database:
    Status:          ok
    Server Response: Duration: 0ms
candlepin:
    Status:          ok
    Server Response: Duration: 45ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 36ms
candlepin_events:
    Status:          ok
    message:         595 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         287 Processed, 0 Failed
    Server Response: Duration: 1ms
pulp3:
    Status:          ok
    Server Response: Duration: 138ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 15ms

Foreman-tail showed nothing unusual or any errros on add/remove/show subscriptions.

After testing some more i found that this issue was now affecting on all hosts, changes to subscriptions did not show up in gui or hammer cli, but on the client with subscription-manager, clients gets properly subscribed but katello does not show anything changed. I did a restart of my Foreman box with no improvment. I thought maybe a “foreman-install do some magic” and it did, katello instantly show the current subscription state of my hosts, any change on client/hammer/web was instantly visible :slight_smile:

Before i rebootet and foreman-install i ran “foreman-rake katello:import_subscriptions” and that fixed the visiblity issue for the moment. Changing some subscription and katello was stuck again showing the last state from the import.

I ran foreman-tail + foreman-rake katello:import_subscriptions and got some tracebacks, all my hosts are affected by this error…

2021-10-05T18:58:33 [I|app|0de38a22]   Parameters: {"facts"=>"[FILTERED]", "name"=>"xyz", "certname"=>"xyz", "apiv"=>"v2", "host"=>{"certname"=>"xyz", "name"=>"xyz"}}
2021-10-05T18:58:33 [W|app|0de38a22] ERF42-9911 [Foreman::Exception]: Host is pending for Build
2021-10-05T18:58:33 [I|app|0de38a22] Backtrace for 'ERF42-9911 [Foreman::Exception]: Host is pending for Build' error (Foreman::Exception): ERF42-9911 [Foreman::Exception]: Host is pending for Build

I hope this is not the reason for the issues i had. All hosts got manually migrated and added to my new Foreman by using the “Register Host” curl script. That register host script was unable to do the subscription-manager register thing and that failed the onboarding, i had to “subscription-manager register --org=… --activationkey=…” by hand and then run the onboarding script again to let in finish.

For now this issue seems to be fixed. This “host pending for build” makes me a little bit nervous.

5

Thank you for all the details. Everything seems to be running OK, but please keep the thread updated if things change.

As far as the “pending for build” errors I’m not sure what may be affected in every case. In your case I think it would be safe to move your hosts out of build mode. You can automate it with hammer, something like:

hammer host update --id=$ID --build=false
1 Like