CentOS host cannot subscribe to repositories

Purneau · January 14, 2020, 1:36pm

Problem:
Our organization wants to use Katello for Patch & Configuration Management of its CentOS 7 VMs (Hyper-V). After installing Katello (on a fresh CentOS 7.7.1908), we try to add an host (also a fresh CentOS 7.7.1908) to be managed. Registering goes well:

[root@sobsl03 consumer]# subscription-manager register --org="Default_Organization" --activationkey="Non Production"
The system has been registered with ID: 435d1f2d-b671-4788-a797-04ba61db171a
The registered system name is: sobsl03.europe.**<company name>**.corp
No products installed.

However, if we they try a subscription-manager repos we get two blank lines and:

[root@sobsl03 consumer]# subscription-manager repos


This system has no repositories available through subscriptions.

At the time of the last command, the following can be observed in /var/log/rhsm/rhsm.log:

subscription-manager logging

2020-01-14 13:20:29,295 [INFO] rhsmd:17927:MainThread @rhsm_d.py:280 - rhsmd started
2020-01-14 13:20:29,314 [INFO] subscription-manager:17915:MainThread @connection.py:905 - Connection built: host=sobsl08.europe.stater.corp port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2020-01-14 13:20:29,323 [INFO] subscription-manager:17915:MainThread @managercli.py:462 - X-Correlation-ID: 1d4cd75ce9db499caa6e728d5da23ca4
2020-01-14 13:20:29,323 [INFO] subscription-manager:17915:MainThread @managercli.py:351 - Client Versions: {'subscription-manager': '1.24.13-3.el7.centos'}
2020-01-14 13:20:29,324 [INFO] subscription-manager:17915:MainThread @connection.py:905 - Connection built: host=sobsl08.europe.stater.corp port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2020-01-14 13:20:29,324 [INFO] subscription-manager:17915:MainThread @connection.py:905 - Connection built: host=sobsl08.europe.stater.corp port=443 handler=/rhsm auth=none
2020-01-14 13:20:30,500 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/certificates/serials"
2020-01-14 13:20:30,608 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/certificates?serials=5550632573412824304"
2020-01-14 13:20:30,609 [ERROR] subscription-manager:17915:MainThread @certificate2.py:112 - unknown string format
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/rhsm/certificate2.py", line 107, in _read_x509
    return self._create_v3_cert(version, extensions, x509, path, pem)
  File "/usr/lib64/python2.7/site-packages/rhsm/certificate2.py", line 309, in _create_v3_cert
    end=get_datetime_from_x509(x509.get_not_after()),
  File "/usr/lib64/python2.7/site-packages/rhsm/certificate.py", line 90, in get_datetime_from_x509
    return dateutil.parser.parse(date)
  File "/usr/lib/python2.7/site-packages/dateutil/parser.py", line 697, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python2.7/site-packages/dateutil/parser.py", line 303, in parse
    raise ValueError, "unknown string format"
ValueError: unknown string format
2020-01-14 13:20:30,610 [ERROR] subscription-manager:17915:MainThread @entcertlib.py:404 - unknown string format
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py", line 377, in install
    key, cert = self.build_cert(bundle)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py", line 398, in build_cert
    cert = create_from_pem(crtpem)
  File "/usr/lib64/python2.7/site-packages/rhsm/certificate.py", line 61, in create_from_pem
    return _CertFactory().create_from_pem(pem)
  File "/usr/lib64/python2.7/site-packages/rhsm/certificate2.py", line 82, in create_from_pem
    return self._read_x509(_certificate.load(pem=pem), path, pem)
  File "/usr/lib64/python2.7/site-packages/rhsm/certificate2.py", line 113, in _read_x509
    raise CertificateException(str(e))
CertificateException: unknown string format
2020-01-14 13:20:30,610 [ERROR] subscription-manager:17915:MainThread @entcertlib.py:405 - Bundle not loaded:
{u'updated': u'2020-01-14T13:07:58+0000', u'created': u'2020-01-14T13:07:58+0000', u'cert': u'-----BEGIN CERTIFICATE-----\nMIIFcDCCBFigAwIBAgIITQfO6sD71PAwDQYJKoZIhvcNAQELBQAwgYUxCzAJBgNV\nBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQMA4GA1UEBwwHUmFsZWln\naDEQMA4GA1UECgwHS2F0ZWxsbzEUMBIGA1UECwwLU29tZU9yZ1VuaXQxIzAhBgNV\nBAMMGnNvYnNsMDguZXVyb3BlLnN0YXRlci5jb3JwMCAXDTIwMDExNDEyMDc1OFoY\nDzIwNTAwMTA2MTIzNzA0WjBKMR0wGwYDVQQKDBREZWZhdWx0X09yZ2FuaXphdGlv\nbjEpMCcGA1UEAwwgZjRiMWQ3NDhmMGY1NGEwYmEzNDQ3N2EyMWY1ZjgzYWEwggIi\nMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCo8QbDC7gMjCVMeaJ0iHcnY35c\nfuUirMmJJN8hU45/dP+PQS7DIfp5WcMQcIj9tKryRX9molm8CQV+WOslpH+gh1E8\nvgwwWr6g/dARU6qMjQuwxTxR8xe0Ddd/FR0IS4wnC2ZEta8H0p6KPH0fCVAw4YC/\nyyNuPGpeE8pgkbBWLftvpTTal+Svspb+MPxURas8t+mRTvdMmO9yeqyb7tXaSdho\nDdNXSNClB/zKBSM+3BHPFFsSGtYOLtzAKeu+7fYGj6sm9EB+YXkzkJEKeAvxxP71\nWkO6axsfXfp2c3+zLvWdT6P3vJQkrqWPnGdzF9PZbmWMhDWzToPN7I8WunmDl1Ee\nS6IgqrqQ+WNbvDWuECp/QRMwjvJ2XPOgArucXzLygTAMjJhXwAlbQJ/sMPG9jdQh\n6nCyPdn3wojtKgZGjkjtCrgm2m6NHjSSfbznLClAwsQbuf9Jdwon4AUiSdD2SQpN\ntMSHzxWIpZT0epK5wavA/JG7oZssOlG/8o9rWDxoGFGMETM+ce5cR5N1P0pFue04\nHES0FZtCG4omQpU+48EVDfO2nz7A6iFWpCGrPuWqe4zeadulKzNS+KngRtfPlC5H\n6d+pySrwM4FEPNxqLoSujXdsj9IHnWNSRaH8YOhi3BWAEl+13OZjVuI2H8f/Zddy\nCWf3a22edre+JfgMQQIDAQABo4IBGjCCARYwDgYDVR0PAQH/BAQDAgSwMBMGA1Ud\nJQQMMAoGCCsGAQUFBwMCMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMB0G\nA1UdDgQWBBTtjVHC89ghXKtpjzKF0iPb/dqU4zAfBgNVHSMEGDAWgBSu3EOQ5pet\n5vZwOVkJu/jghre7ojASBgkrBgEEAZIICQYEBQwDMy40MBQGCSsGAQQBkggJCAQH\nDAVCYXNpYzBnBgkrBgEEAZIICQcEWgRYeNpzTs0r8Q+Od87PKwGy4sMyU8sZnCFi\n5gxAosLCLN7MhCG5tLgkP5fBJTUtsTSnJN6/KD0xL7MqsSQzP4/BLz8vPqAoP6U0\nGcwFAKwEHTgH/zkNLTVugDANBgkqhkiG9w0BAQsFAAOCAQEAGrO5XN1VlDVZLNxy\ntjLNnEQ1jx7I6+KlLdkeDiL1Ebwb14WZhYnjI9HZRr+vO2X+WqxU2NCYIn+Pu4cA\n9AuxJ1wg+6CouAai9RFtEmlFRb7hInwjz5aXWuSdt0O7zxj6karDyrUQ01j0WDIS\n9QNILkfI+pdpKfIhmZ7tyt1AJDhYKu5Xsys40yrmomCn1oliNvxIqj126gt9ECaA\n1S7fhxnJxipS8q1qHVpk0Bi9Dd8SzCKgm7Oj7tHVesWTENqf/rb0pamzJfmsbJ60\nl58nNMddLsJ6kWn1ItH3v2yVBdUGs2p3ArDm2Jf9jVDPISEwh5wv+D3ju5SAdwj/\nYIhAAg==\n-----END CERTIFICATE-----\n-----BEGIN ENTITLEMENT DATA-----\neJyFkU9vnDAQxb9KNWeI/yxg4FZtj1G3UqocWkXWAN6NtSymxt5ms+K712YThUqV\nKnFg/Dy/efN8hdYMkz8pCzVkm7xje96lTSFYmomyTFFUIqVZgwXrGiYYQgK/PA5O\nuwvULIHJN1Nr9ei0GaC+wnT0gcRyWnJGC1pteOgY8KTC6VYNbvfwScCcgLFdnHld\n0dKIc2hduMoppyllKcu+M15vRE2zHwGkhm4R80WkxUoMzNGazrdugvrnFXT3fx8J\nnJWdFucQCrTts3aqdd6qCIHP9/fwlMSIXGhZY0VFaZ4JSqssNLrLGLkXf/oYsnuQ\nL2Uhi6j32Kg+nH1Re/S9kzt7wEG/YgxN3txIIdcd57CpiW+y9ZMzETuiew41+ReD\nfA2cb7ftY/mG3N58y0etfpN2Ab1LgqynxcX/2vgwHqS30fLdHQnfEZ3qe0Nw1OTM\niVWjmbQzVquJMBJvH9VFvueUwEk57NChVC+jtiEONj/NgTsa08dHX1LkbVVUOYpi\njxnlZUPZ8qcavmkppQXM8x92zs3W\n-----END ENTITLEMENT DATA-----\n-----BEGIN RSA SIGNATURE-----\nlWzQNyR20MRrsrH5GlxVc+oZyDaFOX8tBbscn78PmZH6ev+S+CL3zs37hqpUj+EK\n6G+CdHkxncQkI6S7QUpv6EebYX2cfiXbhA5mNmyE8g7Dm7pGnwOWlEnvZ159IsiL\nNSJ/hNq2j+IyzXc5QHQqBNWIIfVUm/PQVZLfWfBZQEvSt1YQBgV0C2QMX3U9KzbN\nd3f1UXpuzafF6+1G1LLT0omE5XD+RBxq6ZD7Z5rJbGDvAUaYm8zRJOVtlgT2Gkq5\nL+M1koab0s9oevwwCyHcg6NcDqgcUCf0XuoK3laQEKcv0L4tHioLkzCI6f3Vin+C\nlfrqfYaxBTPFy4sA6hMUqA==\n-----END RSA SIGNATURE-----\n', u'key': u'-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEAqPEGwwu4DIwlTHmidIh3J2N+XH7lIqzJiSTfIVOOf3T/j0Eu\nwyH6eVnDEHCI/bSq8kV/ZqJZvAkFfljrJaR/oIdRPL4MMFq+oP3QEVOqjI0LsMU8\nUfMXtA3XfxUdCEuMJwtmRLWvB9Keijx9HwlQMOGAv8sjbjxqXhPKYJGwVi37b6U0\n2pfkr7KW/jD8VEWrPLfpkU73TJjvcnqsm+7V2knYaA3TV0jQpQf8ygUjPtwRzxRb\nEhrWDi7cwCnrvu32Bo+rJvRAfmF5M5CRCngL8cT+9VpDumsbH136dnN/sy71nU+j\n97yUJK6lj5xncxfT2W5ljIQ1s06DzeyPFrp5g5dRHkuiIKq6kPljW7w1rhAqf0ET\nMI7ydlzzoAK7nF8y8oEwDIyYV8AJW0Cf7DDxvY3UIepwsj3Z98KI7SoGRo5I7Qq4\nJtpujR40kn285ywpQMLEG7n/SXcKJ+AFIknQ9kkKTbTEh88ViKWU9HqSucGrwPyR\nu6GbLDpRv/KPa1g8aBhRjBEzPnHuXEeTdT9KRbntOBxEtBWbQhuKJkKVPuPBFQ3z\ntp8+wOohVqQhqz7lqnuM3mnbpSszUvip4EbXz5QuR+nfqckq8DOBRDzcai6Ero13\nbI/SB51jUkWh/GDoYtwVgBJftdzmY1biNh/H/2XXcgln92ttnna3viX4DEECAwEA\nAQKCAgAd/CmnFL1l/LniCPS8mjwL0azr3dntfZ2HqfOwhpQUKz/wvTD/kVhTkYr9\nr/ugjCCLP6qNHOOwbCl9c2rJE5lTdLEGhvTC4FhVWrdDXAY5TA93IIrEQNFEM/C7\nrZOplT8a7vsH1vdp/AgHoRTUHWMOSfgAEp4HQtHeQMT6vuD9asvl1Py8j5kXVitW\nkns2uebasAnUsJ9UCRD20K8yPgHW7KRsU4ffRh1CLVjAwoiu74B5sLM5U2oKumaT\n9NzS1omEQJuwHkoUJFjOvioAIa59Cun1wj+jJr95ocSCoZ1YwEz3ztrbba0dlaHt\nC2WRF/mR7fY4Kn8dhCIAuqvq7M4EMra1WfQ8XQVjGQslSIsY1LURHwbZZaZRfznH\nsO/gZCjvCVwLJ3nq6kI+OSLDmEjk8m1OMMx7LLRon6++R/WS9d6fRCLCIK7APHmL\ndGbnzDB3y9BeWIMIdRnJXri1WQAx0x6lyd8vapETKzQ+0Q5MeEZh4FdL8j9x046W\n/miWVrBMGfE5CLCPttWK3Y1luoVcz0iP+IBN8uHXB5dcXwSHaH6KPRnFwhVufwwV\n7eO2II6XOBYcG3L22VWI73NXVqJgIlIGy5BVZ7Q1cDykysgXcy0bASpGC16OVS3S\nj7C4D1uk1J6fVT4/LZcE/ClcelygmoPHpQZTwv8mfSorMbi4wQKCAQEA7KIm8QZJ\n9FdRMdi4Dy/P3HEEBWZQyqGGHIOX87zfDu8dfnafviwb1W/9H6M80ag7DcfDVVqn\nojYHBT1bziq1ny4K4VWg4bceA5AK+7yAbyzVCCgmztbeT+wlpsf5K5VRP3KMpkam\nmAwHG5Zr7HzbxWjdwOyxmUEL+xwu6J2vPuKf1gjtADMsFiomZ0+nzFPB6yH9K5G1\nwCmuulVm5sTE2/b0sNRlHVHvqeiDLddskJ8G8/FcZslxgDBs4n9RkVsPD5vaMGIw\ny+CiZIH+xKQ51UUPJfK7IXXmFc4tMCLHY7R22R0GKSmfqA2sJdeOKa7Cr/b8M8qX\n5O/rlJYm7/QnWwKCAQEAtsSe9EeKRaq6tUT0O6DGnQ7r00TPSzBikpadIG9kfMfP\nrZ3cAKqCQDrPQ9OOu599rF4M7fDN1s7IgZnJLaS+z7r9kTghL6E6vdETWXRWHrJi\np8Q6cf5+V09XQJaDOe3v9/nNi73Da9icB+T1mWrJEJ9GYuGwB2WoGpQO0Oh5j9+H\n8g50PYKHGoGevkdAbAdBgTTi4kF3C4AlqLlqAJekWg67ZdZdOBVMwCpHcjRTAq8N\nBbrBXIh3tgHK3NnFEeJehZSxYw+WChuIhEyhNnB7GtImTxrlEgACz9qyV/pIWsnW\nhpeNi5jwRI5b61sarKeb+yZ8ruzC+TjTAiknUs3JkwKCAQEArYIAN766v1S/DmOw\nIkKWkTa1EhiBqPJi2cmLw4Z7sAGYiMAGRXUmO4xP98DDuZ83HdWZ3p4ISKKA51Ck\nXjlmzv79r1gQDTEt6sLVqtv++IXQ9pgqTIO0ormuqen7Q//KgVFG3Cz1vosjMxFK\ntS6gHtHTt8BWQN83H7r8kkf81YlqVZBZ50meS+pAGoSHrwTDLdvQPQAsWmi1257/\nQrqI9hjQNlCmi6cmqlj4xsYsrqZMfNY/QIb4mFvpmvJsJr6z/mfyxoNY8EUYDcZb\nvmgEl+J5RwQCPniqatawv5jgxrTuaCgl+kyw6pKL5Xt2nA2uZ0QcjQVB9vYMsPig\nOFW3/QKCAQBm/r6k4nvlbqTw6f7Ia34BnBsBX9GBia7C9OVvFdcHCoLaUA9ewLfO\nAlYy3tGcofBsZkPbAsy8tKMBhex9ME3nOnUSlRL+iFlWBrBkxM6TVeuKp2ZYPxZr\nlWaSGz+wMYn9sGRIz5OE/vhO6pQM9MSB/X/o0U4lnP/ymmUPDTMUBN/TZJnU3+3K\n3ijbCcQgHKapbWg+IK45aOnP/0zhMJOJvKuWZAr8Q+SQlkLU9O+in4szI188ChuM\nEc6ETqRjw7GIjZa5QFtJLuLry1M+Hn0BTFQSWBGm5yKuh4vhUEXeA8hE6Lr3ZCRi\nPedG5nxNt9aK3dF9sibd00v198MVQqrJAoIBAAZb21U8tewufHcBwH6RXom8YEVu\nRcb58F2oOBWa2LLGUe+m7MofoRfuEJiytTEt3m79PjcrJklZQAiDKZl70kvGCtri\nL714wk+IvwKw0fuS7OCdKUF1sB8lAdjhx21d0O4dyPgONzLUUJ0qtibTIjnQFm4n\nkw3RQAxKVumevxVcHZNY4fy0aCU/A0HE3ack1mL/gAq/9dh7+/q8e2YXBAULMXAQ\nhcso74T3Oq+jkppWG7skktkLRyf0gtFWO69caMlyHEC8bbFL14s7kbO8xwUwNPAo\n5mqdyjXDWQ5gWLGWZIVIiBtvwx0adnRtRRXiV5iP61CqMOXK8waZw9IgYfo=\n-----END RSA PRIVATE KEY-----\n', u'serial': {u'updated': u'2020-01-14T13:07:58+0000', u'revoked': False, u'created': u'2020-01-14T13:07:58+0000', u'expiration': u'2050-01-06T12:37:04+0000', u'collected': False, u'serial': 5550632573412824304, u'id': 5550632573412824304}, u'id': u'2c9695a76fa4028b016fa42afd260016'}
unknown string format
2020-01-14 13:20:30,610 [INFO] subscription-manager:17915:MainThread @entcertlib.py:131 - certs updated:
Total updates: 0
Found (local) serial# []
Expected (UEP) serial# [5550632573412824304]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2020-01-14 13:20:30,676 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/status"
2020-01-14 13:20:30,726 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/"
2020-01-14 13:20:30,794 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/content_overrides"
2020-01-14 13:20:30,796 [INFO] subscription-manager:17915:MainThread @repolib.py:464 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-01-14 13:20:30,878 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a"
2020-01-14 13:20:30,939 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/content_overrides"
2020-01-14 13:20:30,940 [INFO] subscription-manager:17915:MainThread @repolib.py:464 - repos updated: Repo updates

Total repo updates: 0
Updated
    <NONE>
Added (new)
    <NONE>
Deleted
    <NONE>
2020-01-14 13:20:30,960 [INFO] subscription-manager:17915:MainThread @dmiinfo.py:76 - Using dmidecode dump file: /dev/mem
2020-01-14 13:20:31,223 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a"
2020-01-14 13:20:31,372 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="PUT /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a"
2020-01-14 13:20:31,464 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/compliance"
2020-01-14 13:20:31,465 [INFO] subscription-manager:17915:MainThread @cert_sorter.py:206 - Product status: valid_products= partial_products= expired_products= unentitled_producs= future_products= valid_until=None
2020-01-14 13:20:31,467 [INFO] subscription-manager:17915:MainThread @connection.py:905 - Connection built: host=sobsl08.europe.stater.corp port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2020-01-14 13:20:31,552 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/compliance"
2020-01-14 13:20:31,553 [INFO] subscription-manager:17915:MainThread @cert_sorter.py:206 - Product status: valid_products= partial_products= expired_products= unentitled_producs= future_products= valid_until=None
2020-01-14 13:20:31,643 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/compliance"
2020-01-14 13:20:31,644 [INFO] subscription-manager:17915:MainThread @cert_sorter.py:206 - Product status: valid_products= partial_products= expired_products= unentitled_producs= future_products= valid_until=None
2020-01-14 13:20:31,730 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/compliance"
2020-01-14 13:20:31,731 [INFO] subscription-manager:17915:MainThread @cert_sorter.py:206 - Product status: valid_products= partial_products= expired_products= unentitled_producs= future_products= valid_until=None
2020-01-14 13:20:31,734 [INFO] rhsmd:17927:MainThread @connection.py:905 - Connection built: host=sobsl08.europe.stater.corp port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2020-01-14 13:20:31,815 [INFO] rhsmd:17927:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/compliance"
2020-01-14 13:20:31,816 [INFO] rhsmd:17927:MainThread @cert_sorter.py:206 - Product status: valid_products= partial_products= expired_products= unentitled_producs= future_products= valid_until=None
2020-01-14 13:20:31,869 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/"
2020-01-14 13:20:31,930 [INFO] subscription-manager:17915:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/consumers/435d1f2d-b671-4788-a797-04ba61db171a/content_overrides"

I have no idea how to fix the above certificate related issue(s).

At the same time, in the Foreman GUI, there is a warning next to this host:

Could not calculate errata status, ensure host is registered and the katello-host-tools package is installed

The mentioned tools are installed.

I also observed there is no redhat.repo file being created in /etc/yum.repos.d.

All prerequisites that I can think of are in place:

GPG Key created
Product created
Repository created
Repository synced
Environment created
Content view created and promoted to all environments
Applicable Activation Key created and used to subscribe

During troubleshooting I followed this tutorial to make sure I did not forget anything: Configure Katello - Download Repositories, Create Environments and Activation Keys

Expected outcome:
Client host should see the available repositories.
No warnings in Foreman GUI.

Foreman and Proxy versions:
Reproducible with:

Foreman 1.24.1-1.el7
Foreman 2.0.0-0.2.develop.20200113195832git0f3da6e.el7

Foreman and Proxy plugin versions:
Reproducible with:

Katello 3.14.0-4.el7 (used with Foreman 1.24)
Katello 3.15.0-0.3.master.el7 (used with Foreman 2.0.0)

Distribution and version:
CentOS 7.7.1908

Other relevant data:

System does not have internet connectivity. Repositories are accessible through Artifactory. That seems to go perfect.
I’m aware the above logging shows certificate and key information, but the VMs containing these keys will be removed immediately after troubleshooting this issue successfully

Thanks for helping!

koshi · January 14, 2020, 2:13pm

For what it’s worth I’m having the same problem. I installed Foreman v1.24 last week, with Centos 7.7 Clients, and get the exact same problem of clients not seeing any subscribed products, & error in the /var/log/rhsm.log:

File “/usr/lib64/python2.7/site-packages/rhsm/certificate2.py”, line 113, in _read_x509
raise CertificateException(str(e))
CertificateException: unknown string format

samos · January 14, 2020, 2:15pm

Hi Team,
I have the exact same issue, on my RHEL 7.6 machines.
/etc/yum.repos.d/redhat.repo < is also empty.

secvirt · January 14, 2020, 5:41pm

Having the same issue here. This issue is related to /usr/lib64/python2.7/site-packages/rhsm/certificate.py

The get_datetime_from_x509 function is unable to parse the datetime from the certificate. I had changed the function to :

dateutil.parser.parse(date, fuzzy=True)

When I do this and re-register, the repository shows up as expired. Thats where I am stuck now.

ekohl · January 14, 2020, 6:34pm

I suspect this is https://bugzilla.redhat.com/show_bug.cgi?id=1789654#c4.

@Jonathon_Turel might have more information about a plan to fix this in upstream.

Jonathon_Turel · January 14, 2020, 6:43pm

Thank you for raising a thread on this. We’re currently prepping Katello 3.14.1 and 3.13.4 to include the fix (Bug #28714: custom products can't be consumed by hosts - Katello - Foreman). Please keep an eye on the Release Announcements area to know when they’re available. I would estimate early next week.

Jonathon_Turel · January 15, 2020, 9:14pm

Closing the loop - Katello 3.14.1 is available with the fix: Katello 3.14.1 has been released!

I expect 3.13.4 to follow along shortly.

jkalchik · January 16, 2020, 4:57pm

Good morning, all.

I’m hoping that I’m not the bearer of bad news… but I think this problem may not be quite fixed yet. I’ve completely rebuilt my Katello server install from the ground up (both for a completely fresh installation as well as a different product layout,) including a fresh install of CentOS 7, Katello & Foreman, and fully patched as of this morning (noting @Jonathon_Turel post above.) I’ve been attempting to provision new clients (OL6 and CentOS6, I can try OL7 & CentOS7 if necessary) and getting no repositories provisioned during the registration process. I do see a similar error in the client’s /var/log/rhsm/rhsm.log:

unknown string format
2020-01-16 10:47:26,301 [ERROR] subscription-manager:7798:MainThread @certificate2.py:112 - unknown string format
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/rhsm/certificate2.py", line 107, in _read_x509

‘subscription-manager repos’ displays a series of blank lines followed by "This system has no repositories available through subscriptions.’

‘subscription-manager repo-override --list’ returns the list of repositories that have specifically been disabled in this host.

I suspect that I’m hitting the bug referenced above, and thus, it’s really not quite fixed completely.

What are my next steps?

Purneau · January 16, 2020, 5:51pm

What exact version of Katello are you using?

I can confirm that on my end Katello 3.14.1 on CentOS 7.7 is no longer affected by this (CentOS 7.7 clients)

jkalchik · January 16, 2020, 6:11pm

$ rpm -q katello foreman
katello-3.14.1-1.el7.noarch
foreman-1.24.1-1.el7.noarch

The further I dig into this, the more that I’m suspecting it’s something I’m doing. Same issue with OL6, CentOS6 and CentOS7. With the string issue I’m experiencing, I’m wondering if I have the wrong subscription-manager RPMs for the clients. The major distro requirement does match (the following is for a CentOS 6 install.)

# rpm -q subscription-manager katello-host-tools
subscription-manager-1.20.10-8.el6.x86_64
katello-host-tools-3.5.1-2.el6.noarch

Jonathon_Turel · January 17, 2020, 2:07am

I could not reproduce on a CentOS 6 VM:

[root@centos6 vagrant]# rpm -q subscription-manager
subscription-manager-1.17.6-1.el6.x86_64
[root@centos6 vagrant]# cat /etc/yum.repos.d/subscription-manager.repo 
[subscription-manager]
name=Subscription manager repository from Candlepin
baseurl=https://copr-be.cloud.fedoraproject.org/results/dgoodwin/subscription-manager/epel-6-x86_64/
gpgcheck=0
enabled=1
[root@centos6 vagrant]# subscription-manager list --available
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   animals
Provides:            
SKU:                 776111295418
Contract:            
Pool ID:             4028f95f6faa6db5016fb13dc65f0009
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Level:       
Service Type:        
Subscription Type:   Standard
Ends:                12/01/2049
System Type:         Physical

[root@centos6 vagrant]# subscription-manager attach --pool 4028f95f6faa6db5016fb13dc65f0009
Successfully attached a subscription for: animals
[root@centos6 vagrant]# subscription-manager repos
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   Default_Organization_animals_zoo
Repo Name: zoo
Repo URL:  https://centos7-katello-3-14.strangeways.example.com/pulp/repos/Default_Organization/Library/custom/animals/zoo
Enabled:   1

I’ll ping the subscription-manager team to see if they have any thoughts. Thanks for sharing @jkalchik

jkalchik · January 17, 2020, 2:34am

IIRC, the subscription manager package I’ve been trying to use on the EL6 systems is 1.20.x, from the slaanesh repository.

On the system I’m tinkering with at home (rebuilt over the weekend,) I can’t even find a 1.17.x el6 rpm.

Gotta be something I’m doing. Should I consider going back to the dgoodwin repositories?

In any event, I need to stop polluting this thread.

jkalchik · January 18, 2020, 6:49pm

Well, shoot…

Went back to the dgoodwin repositories, still experiencing the same thing on both CentOS6 and CentOS7. Stranger, ‘subscription-manager list --available’ does indeed succeed. Anything that lists what should be available repositories is failing, such as ‘subscription-manager repos --list’.

taylor1 · January 20, 2020, 6:45pm

I just had this same problem (I think), and the issue like was mentioned was that certificate expiration for the subscriptions is set 30 years in the future to a date > 2050…and that causes an internal python datetime module to overflow.

There was a solution posted on the redhat satellite forum about this, basically, updating the candlepin database to change the subscription expiration date to something < the year 2050, then updating foreman. See if this works:

sudo -u postgres -H – psql -d candlepin -c “update cp_pool set enddate = ‘2049-12-01 00:00:00’ where enddate > ‘2049-12-31 00:00:00’;”

echo ’
User.current = User.anonymous_admin
m = Katello::Subscription
m.import_all
m = Katello::Pool
m.import_all
’ | foreman-rake console

Thanks for RedHat for that solution. I sorta figured out how to do the database update, but the foreman-rake thing I was clueless about.

Hope this helps

taylor1 · January 20, 2020, 6:45pm

Also, this needs to be done each time you add a product.

jkalchik · January 20, 2020, 7:12pm

Unfortunately, at least for an OL7 client, the above referenced procedure (both the psql update as well as the foreman-rake console work,) doesn’t appear to make a difference.

I’ve opened up a dedicated thread for this.

Jonathon_Turel · January 20, 2020, 9:37pm

Just to be clear: this is not the case in Katello 3.14.1

The dates will not go beyond 2049

kraba · January 21, 2020, 9:39am

I confirm it works, I had the same issue and now I’m able to attach the product.

langesmalle · January 21, 2020, 10:01am

Hi,

We had the same issue.
Yesterday we upgraded from foreman 1.14.0/katello 1.24.0 to foreman 1.14.1/katello 1.24.1 after reading this post and executed the Redhat-Satelite solution provided by taylor1
I can confirm it now creates the redhat.repo.

jkalchik · January 21, 2020, 3:31pm

After a fair amount of discussion with Jonathon Turel, & testing, there’s a small opportunity for problems with custom (non-RedHat) products. In short, custom products created between Jan. 1, 2020 and the installation of Katello 3.14.1 leaves a problem value in cp_pool:enddate (Jan. 1, 2050 or greater.) This is one of the issues that lead to the 3.14.1 release.

To find out if you’re in this scope:

sudo su - postgres -c “psql -d candlepin -c “select id,enddate from cp_pool where enddate > ‘2049-12-31 00:00:00’;””

If this returns one or more rows, there are at least 2 remediation options.

Drop and recreate your products that were originally created during the scope window.
Follow the procedure from @taylor1 above.

Any clients that have been registered to use products with the enddate issue will need to be reregistered.