Subscription-manager issues

jkalchik · January 20, 2020, 3:08pm

subscription-manager generally works, except for anything around ‘subscription-manager repos’.

List of enabled/disabled repositories

The Foreman/Katello server is running CentOS7, fully patched.
katello-3.14.1-1.el7.noarch
foreman-1.24.1-1.el7.noarch

I’ve tested subscription-manager on CentOS6, CentOS7, Oracle Linux 6 and Oracle Linux 7. All of them exhibit the following sort of output:

# subscription-manager repos









This system has no repositories available through subscriptions.

There are slightly different errors in /var/log/rhsm/rhsm.log on the client hosts.

CentOS6/OL6:
ValueError: Invalid date Bad time value: Invalid month: Bad
2020-01-20 14:16:27,933 [ERROR] subscription-manager:5242 @entcertlib.py:366 - Invalid date Bad time value: Invalid month: Bad
Traceback (most recent call last):
File “/usr/share/rhsm/subscription_manager/entcertlib.py”, line 339, in install
key, cert = self.build_cert(bundle)
File “/usr/share/rhsm/subscription_manager/entcertlib.py”, line 360, in build_cert
cert = create_from_pem(crtpem)
File “/usr/lib64/python2.6/site-packages/rhsm/certificate.py”, line 64, in create_from_pem
return _CertFactory().create_from_pem(pem)
File “/usr/lib64/python2.6/site-packages/rhsm/certificate2.py”, line 74, in create_from_pem
return self._read_x509(_certificate.load(pem=pem), path, pem)
File “/usr/lib64/python2.6/site-packages/rhsm/certificate2.py”, line 100, in _read_x509
raise CertificateException(str(e))
CertificateException: Invalid date Bad time value: Invalid month: Bad
2020-01-20 14:16:27,933 [ERROR] subscription-manager:5242 @entcertlib.py:367 - Bundle not loaded:

CentOS7:
ValueError: unknown string format
2020-01-20 07:17:45,651 [ERROR] subscription-manager:10312:MainThread @entcertlib.py:404 - unknown string format
Traceback (most recent call last):
File “/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py”, line 377, in install
key, cert = self.build_cert(bundle)
File “/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py”, line 398, in build_cert
cert = create_from_pem(crtpem)
File “/usr/lib64/python2.7/site-packages/rhsm/certificate.py”, line 61, in create_from_pem
return _CertFactory().create_from_pem(pem)
File “/usr/lib64/python2.7/site-packages/rhsm/certificate2.py”, line 82, in create_from_pem
return self._read_x509(_certificate.load(pem=pem), path, pem)
File “/usr/lib64/python2.7/site-packages/rhsm/certificate2.py”, line 113, in _read_x509
raise CertificateException(str(e))
CertificateException: unknown string format

OL7: oddly enough, this error looks like the OL6 error.

CentOS6:
subscription-manager-1.17.6-1.el6.x86_64
CentOS7:
subscription-manager-1.24.13-3.el7.centos.x86_64
OL6:
subscription-manager-1.17.6-1.el6.x86_64
OL7:
subscription-manager-1.17.6-1.el7.centos.x86_64

This is all on a fairly fresh install, paved over & rebuilt less than 2 weeks ago. I’m currently using the dgoodwin SubscriptionManager repositories.

What on earth do I have wrong?

jkalchik · January 20, 2020, 7:34pm

So, learning Python on the fly… OL7 client host.

The traceback output from /var/log/rhsm/rhsm.log seems to indicate problems in certificate.py and certificate2.py… Unless I miss my guess, this is raising the error in certificate.py, in the get_datetime_from_x509 function. “Bad time value” is being passed as the time value, and thus, month (“Bad”) is not a valid value (month abbreviation.)

print "DEBUG:	", month
_ssl_months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug",
               "Sep", "Oct", "Nov", "Dec"]

if month not in _ssl_months:
    raise ValueError("Invalid date %s: Invalid month: %s" % (date_str, month))

Output:

DEBUG:  Jan
DEBUG:  Jan
DEBUG:  Jan
DEBUG:  Bad
DEBUG:  Jan
DEBUG:  Bad
DEBUG:  Jan
DEBUG:  Bad
DEBUG:  Jan
DEBUG:  Bad
DEBUG:  Jan
DEBUG:  Bad
DEBUG:  Jan
DEBUG:  Bad
DEBUG:  Jan
DEBUG:  Bad
DEBUG:  Jan
DEBUG:  Bad









DEBUG:  Jan
DEBUG:  Jan
This system has no repositories available through subscriptions.

areyus · January 21, 2020, 1:08pm

Hi,

just a wild guess, but this reminds me of the error described here.
Post 14 has a solution to that problem. Could you check if you have any (custom) subscriptions for those OSes with a validity date set to 01.01.2050 or later?
If so, the solution I linked might solve your problem. I would recommend taking a backup beforehand though.

Regards

jkalchik · January 21, 2020, 3:33pm

Considering I’d been posting in that thread… and that it wasn’t necessarily clear that this is a related issue… see CentOS host cannot subscribe to repositories (post 20.)

areyus · January 22, 2020, 12:29pm

Sorry, I did not catch that you where the one posting there.
I have done a little dive into the subscription-manager source-code (we also have some OL7 seven systems using the dgoodwin repos here). Since our systems do not seem to have any problems, I assume it is something that’s either oddly specific to 3.14 or your setup.
After a lot of digging in the code, I assume that some of your entitlement certificates have bad/weird dates in them, what would match your observations and log messages.
The only way I found to check this (and I assume this is highly overcomplicated):
On one of your affected hosts, execute the following commands:
subscription-manager identity → Get the system identity from here
curl https://YOURRHSMSERVER:8443/rhsm/consumers/SYSTEMID/certificates/serials → replace YOURRHSMSERVER with the server the client is registered to (Either Foreman Server or Smartproxy) and SYSTEMID with the id from above. From the output, you will need the values of all serials returned.
Now for a very over-engineered command to get and parse the certificates:
curl https://YOURRHSMSERVER:8443/rhsm/consumers/SYSTEMID/certificates?serials=SERIALLIST --cert /etc/pki/consumer/cert.pem --key /etc/pki/consumer/key.pem --cacert /etc/pki/ca-trust/source/anchors/katello-server-ca.pem 2>/dev/null | python -m 'json.tool' |grep '"cert"' | awk -F'"' '{print $4}' | awk -F'-----BEGIN ENTITLEMENT DATA' '{print $1}' | while read -r cert ; do echo -e $cert | openssl x509 -noout -text | egrep 'Not (Before|After)' ; done → again, you need to replace YOURRHSMSERVER, SYSTEMID, and SERIALLIST. SERIALLIST needs to be a comma separated list of the serials above, no whitespaces.

This should give you the start and end dates of all your entitlement certificates. Check those for anything strange or datess that exceed 2050. Maybe the problem lies there. If so, I have no ideas how to fix it though

jkalchik · January 22, 2020, 2:44pm

There’s actually a far easier way doc’ed over in the other thread, it’s a simple select in the candlepin database. RH gave taylor1 a solution to modify in place. Other option is to drop and recreate products created between Jan. 1, 2020 and installation of Katello 3.14.1.