Unable to get enabled repositories after subscription-manager register

skippy39us · November 8, 2021, 8:39pm

Problem:
After registering with a valid activation key, i get a redhat.repo file but no enabled repositories

Expected outcome:
I expect to see all enabled repositories

Foreman and Proxy versions:
Foreman 1.24.3
Proxy - 1.24.3

Foreman and Proxy plugin versions:

katello - 3.14.1
foreman-tasks - 0.17.5
foreman-docker - 5.0.0
foreman-remote-execution - 2.0.8
foreman_virt_who_configure - 0.3.0

Distribution and version:

CentOS Linux release 7.9.2009 (Core

Other relevant data:

I have been using this version of foreman for a few years now, without issue.

I run the following command:

> subscription-manager register --org="Default_Organization" --activationkey=oracledevkey
The system has been registered with ID: b9ed7af5-ec89-4668-8fdd-49de9181586c 

No products installed.

The /var/log/rhsm/rhsm.log shows this:

2021-11-08 13:25:48,948 [INFO] subscription-manager:35123 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el7.centos', 'subscription-manager': '1.17.6-1.el7.centos'}
2021-11-08 13:25:48,949 [INFO] subscription-manager:35123 @connection.py:815 - Connection built: host=lnxforeftden01.domain.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2021-11-08 13:25:48,949 [INFO] subscription-manager:35123 @connection.py:815 - Connection built: host=lnxforeftden01.domain.com port=443 handler=/rhsm auth=none
2021-11-08 13:25:48,949 [INFO] subscription-manager:35123 @managercli.py:384 - Client Versions: {'python-rhsm': '1.17.2-1.el7.centos', 'subscription-manager': '1.17.6-1.el7.centos'}
2021-11-08 13:25:48,950 [INFO] subscription-manager:35123 @managercli.py:359 - Consumer Identity name=None uuid=None
2021-11-08 13:25:48,950 [INFO] subscription-manager:35123 @managercli.py:359 - Consumer Identity name=None uuid=None
2021-11-08 13:25:48,950 [INFO] subscription-manager:35123 @connection.py:815 - Connection built: host=lnxforeftden01.domain.com port=443 handler=/rhsm auth=none
2021-11-08 13:25:49,061 [INFO] subscription-manager:35123 @hwprobe.py:908 - collected virt facts: virt.is_guest=True, virt.host_type=kvm, virt.uuid=8956340E-BF4C-4174-B53B-DA4BB638BD3E
2021-11-08 13:25:49,062 [INFO] subscription-manager:35123 @facts.py:139 - Loading custom facts from: /etc/rhsm/facts/katello.facts
2021-11-08 13:25:55,584 [INFO] subscription-manager:35123 @managerlib.py:77 - Consumer created: {'consumer_name': 'aacbcd27-dcd2-404d-b7fb-21b01eb55b03, lnxecmftden31', 'uuid': 'aacbcd27-dcd2-404d-b7fb-21b01eb55b03'}
2021-11-08 13:25:55,585 [INFO] subscription-manager:35123 @connection.py:815 - Connection built: host=lnxforeftden01.domain.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2021-11-08 13:25:55,692 [INFO] subscription-manager:35123 @managercli.py:395 - Server Versions: {'rules-version': '5.41', 'candlepin': '3.14.1-Unknown', 'server-type': 'Red Hat Subscription Management'}
2021-11-08 13:25:56,982 [INFO] subscription-manager:35123 @managercli.py:1182 - System registered, updating entitlements if needed
2021-11-08 13:25:57,203 [INFO] subscription-manager:35123 @entcertlib.py:131 - certs updated:
Total updates: 1
Found (local) serial# []
Expected (UEP) serial# [3055295782426035112]
Added (new)
  [sn:3055295782426035112 ( Content Access,) @ /etc/pki/entitlement/3055295782426035112.pem]
Deleted (rogue):
  <NONE>
2021-11-08 13:25:57,206 [ERROR] subscription-manager:35123 @cache.py:128 - Unable to read cache: /var/lib/rhsm/cache/written_overrides.json
2021-11-08 13:25:57,317 [INFO] subscription-manager:35123 @repolib.py:303 - repos updated: Repo updates

Total repo updates: 7
Updated
    <NONE>
Added (new)
    [id:Default_Organization_Oracle_Linux_Oracle_Linux_7_5_Latest Oracle Linux 7.5 Latest]
    [id:Default_Organization_Oracle_Linux_Oracle_Linux_75_GA_installation_media_copy Oracle Linux 7.5 GA installation media copy]
    [id:Default_Organization_Oracle_Linux_centos_extras centos extras]
    [id:Default_Organization_Oracle_Linux_UEK_Release_5 UEK Release 5]
    [id:Default_Organization_Oracle_Linux_Copr_dgoodwin_6 Copr dgoodwin 6]
    [id:Default_Organization_Oracle_Linux_Copr_dgoodwin_7 Copr dgoodwin 7]
    [id:Default_Organization_Oracle_Linux_foreman_client foreman client]
Deleted
    <NONE>
2021-11-08 13:25:57,401 [INFO] subscription-manager:35123 @cert_sorter.py:205 - Product status: valid_products= partial_products= expired_products= unentitled_producs= future_products= valid_until=None
2021-11-08 13:25:57,610 [INFO] rhsmd:35174 @rhsmd:235 - rhsmd started
2021-11-08 13:25:57,618 [INFO] rhsmd:35174 @connection.py:815 - Connection built: host=lnxforeftden01.nnacol.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
2021-11-08 13:25:57,684 [INFO] rhsmd:35174 @cert_sorter.py:205 - Product status: valid_products= partial_products= expired_products= unentitled_producs= future_products= valid_until=None

I have verified that the activation key has the repos, and obviously it does since they show up. Just not sure how to convince the system to enable them.

Any thoughts?

Dewar · November 9, 2021, 4:55pm

Hi @skippy39us,

Have you checked under the subscriptions tab that the products are associated correctly with the activation key?

Also the rhsm.log you provided appears to show some repos.

Could you post the contents of your /etc/yum.repos.d/redhat.repo?

That’s where the subscription-manager writes its repo configuration for yum/dnf.

skippy39us · November 9, 2021, 5:41pm

Hi there

I’ve gone to Content -> Activation Keys -> oracledevkey and it shows the right product associated with the the activation key.

The /etc/yum.repos.d/redhat.repo looks like this:

[Default_Organization_Oracle_Linux_Oracle_Linux_7_5_Latest]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1998403150675871427.pem
baseurl = https://lnxforeftden01.domain.com/pulp/repos/Default_Organization/Development/Oracle_Linux/custom/Oracle_Linux/Oracle_Linux_7_5_Latest
sslverify = 1
name = Oracle Linux 7.5 Latest
sslclientkey = /etc/pki/entitlement/1998403150675871427-key.pem
gpgkey = https://lnxforeftden01.domain.com/katello/api/v2/repositories/27/gpg_key_content
enabled = 0
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

[Default_Organization_Oracle_Linux_Oracle_Linux_75_GA_installation_media_copy]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1998403150675871427.pem
baseurl = https://lnxforeftden01.domain.com/pulp/repos/Default_Organization/Development/Oracle_Linux/custom/Oracle_Linux/Oracle_Linux_75_GA_installation_media_copy
sslverify = 1
name = Oracle Linux 7.5 GA installation media copy
sslclientkey = /etc/pki/entitlement/1998403150675871427-key.pem
gpgkey = https://lnxforeftden01.domain.com/katello/api/v2/repositories/28/gpg_key_content
enabled = 0
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

[Default_Organization_Oracle_Linux_centos_extras]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1998403150675871427.pem
baseurl = https://lnxforeftden01.domain.com/pulp/repos/Default_Organization/Development/Oracle_Linux/custom/Oracle_Linux/centos_extras
sslverify = 1
name = centos extras
sslclientkey = /etc/pki/entitlement/1998403150675871427-key.pem
gpgkey = https://lnxforeftden01.domain.com/katello/api/v2/repositories/141/gpg_key_content
enabled = 0
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

[Default_Organization_Oracle_Linux_UEK_Release_5]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1998403150675871427.pem
baseurl = https://lnxforeftden01.domain.com/pulp/repos/Default_Organization/Development/Oracle_Linux/custom/Oracle_Linux/UEK_Release_5
sslverify = 1
name = UEK Release 5
sslclientkey = /etc/pki/entitlement/1998403150675871427-key.pem
gpgkey = https://lnxforeftden01.domain.com/katello/api/v2/repositories/193/gpg_key_content
enabled = 0
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

[Default_Organization_Oracle_Linux_Copr_dgoodwin_6]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1998403150675871427.pem
baseurl = https://lnxforeftden01.domain.com/pulp/repos/Default_Organization/Development/Oracle_Linux/custom/Oracle_Linux/Copr_dgoodwin_6
sslverify = 1
name = Copr dgoodwin 6
sslclientkey = /etc/pki/entitlement/1998403150675871427-key.pem
gpgkey = https://lnxforeftden01.domain.com/katello/api/v2/repositories/104/gpg_key_content
enabled = 0
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

[Default_Organization_Oracle_Linux_Copr_dgoodwin_7]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1998403150675871427.pem
baseurl = https://lnxforeftden01.domain.com/pulp/repos/Default_Organization/Development/Oracle_Linux/custom/Oracle_Linux/Copr_dgoodwin_7
sslverify = 1
name = Copr dgoodwin 7
sslclientkey = /etc/pki/entitlement/1998403150675871427-key.pem
gpgkey = https://lnxforeftden01.domain.com/katello/api/v2/repositories/105/gpg_key_content
enabled = 0
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

[Default_Organization_Oracle_Linux_foreman_client]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1998403150675871427.pem
baseurl = https://lnxforeftden01.domain.com/pulp/repos/Default_Organization/Development/Oracle_Linux/custom/Oracle_Linux/foreman_client
sslverify = 1
name = foreman client
sslclientkey = /etc/pki/entitlement/1998403150675871427-key.pem
gpgkey = https://lnxforeftden01.domain.com/katello/api/v2/repositories/118/gpg_key_content
enabled = 0
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

Note that all of the subscriptions are not enabled. Its weird because it knows the subs, but it DOESN’T enable them.

Dewar · November 9, 2021, 6:07pm

Ah, they are there but disabled.

If you go to activation keys and click on a key (you might need to create one)
Click the repository sets tab

Under that status column
The above shows enabled, what does yours show?
You may need to set it to enabled.

skippy39us · November 9, 2021, 6:19pm

This is what I have:

They are definitely set to enabled

skippy39us · November 9, 2021, 6:24pm

Meaning it was already set to enabled.

Dewar · November 9, 2021, 6:29pm

hmm…

On the repo details page there’s an option to Restrict to OS version, is it set to ‘No Restriction’?

If not, do try that.

skippy39us · November 9, 2021, 6:38pm

I’m sorry - can you tell me exactly how to get to that page?

Dewar · November 9, 2021, 6:52pm

Products > Click a product > Click a repo

skippy39us · November 9, 2021, 6:57pm

Ah - my version of foreman does not have that. I have the restrict to architecture but not the restrict to os version

skippy39us · November 9, 2021, 6:58pm

Dewar · November 9, 2021, 7:01pm

Could you try to set the “Restrict to Architecture” option to “no restrictions” and see if that helps at all?

skippy39us · November 9, 2021, 7:29pm

So my options are x86_64, i386 or default - i’m not using anything other than x86_64

So you want me to set it to default? And besides - wont that just affect repository syncing? The repo is syncing just fine - and I have other content hosts using that repo successfully

Dewar · November 9, 2021, 8:13pm

Try ‘Default’, and no this doesn’t affect syncing.
This affects whether a repo is enabled on a client or not.

Though this may not be the cause, it would be good to check.

skippy39us · November 9, 2021, 10:08pm

This does not make a difference.

Changed it to default, and no change in subscription status on the client.

Also - I’m curious as to why this change is in the Sync Settings area if it doesn’t affect the “Sync Settings”. That seems strange.

But anyway - it didn’t work.

skippy39us · November 9, 2021, 10:26pm

By the way this seems to happen with ANY of my activation keys for this host. Is there something I need to make sure exists on this host for this to work?

skippy39us · November 9, 2021, 10:31pm

So I don’t seem to have a /var/log/rhsm/rhsmcertd.log file anywhere on this node. Could I have a cert issue?

mcorr · November 10, 2021, 8:18am

At this point, I think it is safe to say that upgrading and checking does the issue go away would be prudent.

Jonathon_Turel · November 10, 2021, 3:36pm

Hello @skippy39us! I agree with @mcorr that upgrading your Katello instance would be the correct course of action. I believe the issue you’re seeing has been fixed in Katello 4.1 → https://github.com/Katello/katello/pull/9262

If you take a backup of your system it may be worth attempting to patch it if you’re currently unable to upgrade. Beyond that I don’t think there’s anything we can do to help with your issue. To patch you would apply the PR changes (less the tests code) manually, run the new rake task via foreman-rake katello:upgrades:4.1:sync_noarch_content, then restart the services with foreman-maintain service restart and then a subscription-manager refresh --force on your clients.

A note about upgrading: depending on the size of your infrastructure and the fact that you’re many versions behind the latest Katello, it may be more efficient to create a new instance with the latest version and migrate your hosts to use the new instance over time that’s convenient for you.

Also, be sure to update subscription-manager on your clients regardless of the Katello upgrade.

skippy39us · November 10, 2021, 8:58pm

Is there a path to upgrade the existing foreman instance as well, even though its old?

Like is there a path to do several upgrades to get to the current version as well? (Or is it possible to just upgrade to the newest version straight from the version that I’m running?)

Todd