Cant reach Foreman Katello Server after publishing a new version

Support
1

**Problem: Im new to Katello and i am experimenting right now. I just synced my CentOS 9 repos and created a new version for Libary. My hosts are in the same enviroment. After i created the version and did a “yum update” on one of the servers, the server cant reach the Katello server anymore. I dont know why this happends because with the version before i did work well. The logs dont say anything about it.

**Expected outcome: Repo sync

**Foreman and Proxy versions: Foreman 3.12 Katello 4.14

**Distribution and version: CentOS 9

2

Hi @miwo

“yum update” from a host isn’t the same as repo sync. Repos are synced on Foreman server itself, from the upstream repository. Are you trying to do that? Or just trying to update software on your registered host?

What’s the exact command you’re trying on the host?
What’s the exact error message it gives you?

3

I know. The repo sync just works fine. But i want to publish the packages to the servers and update the packages. But when im publishing a new version of a content view, the server loses the connection to the Foreman server. The yum update is stuck at “Fetching certificate serial numbers” but it did work before, so i dont know if its a certifcate error.

4

Hi @miwo

What is the output of yum repolist -vvv on the client? Can you look at /var/log/rhsm/rhsm.log and see if you see anything odd and post it here from the client?

5

If content view publishing failed, no sense troubleshooting Yum updates yet. What was the CV publish failure?

6

Hi cintrix84,
here is a small output of yum repolist -vvv:

[root@server ~]# yum repolist -vvv
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, notify-packagekit, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, system-upgrade, uploadprofile
Updating Subscription Management repositories.
YUM version: 4.14.0
cachedir: /var/cache/dnf
CentOS Stream 9 - Extras packages 20 kB/s | 18 kB 00:00
CentOS9StreamAppStream 19 kB/s | 2.1 kB 00:00
CentOS9StreamBaseOS 16 kB/s | 1.8 kB 00:00
CentOS9StreamEPEL 23 kB/s | 2.3 kB 00:00
CentOS9StreamExtrasCommon 12 kB/s | 1.5 kB 00:00
Repo-id : CentOS9Stream_CentOS9StreamAppStream
Repo-name : CentOS9StreamAppStream
Repo-revision : 1732617092
Repo-updated : Tue 26 Nov 2024 11:31:31 AM CET
Repo-pkgs : 17,333
Repo-available-pkgs: 16,909
Repo-size : 34 G
Repo-baseurl : https://foreman-server.domain.domain/pulp/content/Wuerth_Elektronik_ICS/Library/custom/CentOS9Stream/CentOS9StreamAppStream
Repo-expire : 1 second(s) (last: Fri 29 Nov 2024 11:59:55 AM CET)
Repo-filename : /etc/yum.repos.d/redhat.repo

Repo-id : CentOS9Stream_CentOS9StreamBaseOS
Repo-name : CentOS9StreamBaseOS
Repo-revision : 1732617087
Repo-updated : Tue 26 Nov 2024 11:31:26 AM CET
Repo-pkgs : 4,554
Repo-available-pkgs: 4,554
Repo-size : 6.1 G
Repo-baseurl : https://foreman-server.domain.domain/pulp/content/Wuerth_Elektronik_ICS/Library/custom/CentOS9Stream/CentOS9StreamBaseOS
Repo-expire : 1 second(s) (last: Fri 29 Nov 2024 12:00:11 PM CET)
Repo-filename : /etc/yum.repos.d/redhat.repo

Repo-id : CentOS9Stream_CentOS9StreamEPEL
Repo-name : CentOS9StreamEPEL
Repo-revision : 1732787951
Repo-updated : Thu 28 Nov 2024 10:59:10 AM CET
Repo-pkgs : 22,531
Repo-available-pkgs: 22,531
Repo-size : 19 G
Repo-baseurl : https://foreman-server.domain.domain/pulp/content/Wuerth_Elektronik_ICS/Library/custom/CentOS9Stream/CentOS9StreamEPEL
Repo-expire : 1 second(s) (last: Fri 29 Nov 2024 12:00:11 PM CET)
Repo-filename : /etc/yum.repos.d/redhat.repo

7

Hey jeremy,
the publishing of the CV doesnt fail. I create a new version with the new packages, this works fine. But now i want to update my registered servers so they can claim those packages with yum update. But as soon i create a new verison the servers cant connect to the foreman server anymore. If i do yum update the command is stuck at “Fetching certificate serial numbers” for about 2 minutes. After that he wants to update the old repos, but doenst do anything because he is up to date. I can ping the foreman server but i cant reach him over 443. The strange thing is that i was able to claim packages until Version 3.0 of the CV. Now i deleted the newer versions and 3.0 is the newest, but it doesnt work anymore. Hope that gives you some information. Thank you for your help.
BR Mirco

8

That looks good, so we don’t see any issues with yum so that leads me to believe we are hitting an issue with subscription-manager with the serial certificate error. Can you look in /var/log/rhsm and see if you see any python tracebacks or clock skew errors?

9

Here is some output of rhsm.log. I hope you can work with this:
2024-12-06 11:26:52,280 [ERROR] rhsmcertd-worker:752498:MainThread @entcertlib.py:98 - Cannot modify subscriptions while disconnected
2024-12-06 11:26:52,281 [WARNING] rhsmcertd-worker:752498:MainThread @base_action_client.py:67 - Exception caught while running <subscription_manager.entcertlib.EntCertActionInvoker object at 0x7fd1d7b70e50> update
2024-12-06 11:26:52,281 [ERROR] rhsmcertd-worker:752498:MainThread @base_action_client.py:68 -
Traceback (most recent call last):
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 95, in perform
expected: List[int] = self._get_expected_serials()
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 277, in get_expected_serials
exp: List[int] = self.get_certificate_serials_list()
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 258, in get_certificate_serials_list
reply: List[Dict] = self.uep.getCertificateSerials(identity.uuid)
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1886, in getCertificateSerials
return self.conn.request_get(method, description=(“Fetching certificate serial numbers”))
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1365, in request_get
result: Dict[str, Any] = self._request(
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1153, in _request
result, response = self._make_request(
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1089, in _make_request
raise NoValidEntitlement(
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1041, in _make_request
conn = self._create_connection(cert_file=cert_file, key_file=key_file)
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 742, in _create_connection
self.close_connection()
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 675, in close_connection
self.__conn.sock.unwrap()
File “/usr/lib64/python3.9/ssl.py”, line 1319, in unwrap
s = self._sslobj.shutdown()
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib64/python3.9/site-packages/subscription_manager/base_action_client.py”, line 59, in _run_update
update_report = lib.update()
File “/usr/lib64/python3.9/site-packages/subscription_manager/certlib.py”, line 34, in update
self.report = self.locker.run(self._do_update)
File “/usr/lib64/python3.9/site-packages/subscription_manager/certlib.py”, line 19, in run
return action()
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 52, in _do_update
return action.perform()
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 99, in perform
raise Disconnected()
subscription_manager.entcertlib.Disconnected
2024-12-06 11:33:56,542 [ERROR] dnf:753658:MainThread @entcertlib.py:97 - The read operation timed out
Traceback (most recent call last):
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 95, in perform
expected: List[int] = self._get_expected_serials()
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 277, in get_expected_serials
exp: List[int] = self.get_certificate_serials_list()
File “/usr/lib64/python3.9/site-packages/subscription_manager/entcertlib.py”, line 258, in get_certificate_serials_list
reply: List[Dict] = self.uep.getCertificateSerials(identity.uuid)
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1886, in getCertificateSerials
return self.conn.request_get(method, description=(“Fetching certificate serial numbers”))
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1365, in request_get
result: Dict[str, Any] = self._request(
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1153, in _request
result, response = self._make_request(
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1089, in _make_request
raise NoValidEntitlement(
File “/usr/lib64/python3.9/site-packages/rhsm/connection.py”, line 1049, in _make_request
response = conn.getresponse()
File “/usr/lib64/python3.9/http/client.py”, line 1377, in getresponse
response.begin()
File “/usr/lib64/python3.9/http/client.py”, line 320, in begin
version, status, reason = self._read_status()
File “/usr/lib64/python3.9/http/client.py”, line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib64/python3.9/socket.py”, line 716, in readinto
return self._sock.recv_into(b)
File “/usr/lib64/python3.9/ssl.py”, line 1275, in recv_into
return self.read(nbytes, buffer)
File “/usr/lib64/python3.9/ssl.py”, line 1133, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
2024-12-06 11:33:56,548 [ERROR] dnf:753658:MainThread @entcertlib.py:98 - Cannot modify subscriptions while disconnected
2024-12-06 11:33:56,548 [ERROR] dnf:753658:MainThread @subscription-manager.py:113 -

10

Ok let’s try one more thing:

From the client let’s do this with filling out your foreman server fqdn:

openssl s_client -connect example.com:443 \
-cert /etc/pki/consumer/cert.pem \
-key /etc/pki/consumer/key.pem \
-CAfile /etc/rhsm/ca/katello-server-ca.pem
11

Hey @cintrix84
whats the expected output of the command?

12

I just tried to unregister the system from RHSM but that doesnt work too. The error says: Unable to reach the server at foreman-server.com:443/rhsm: Tunnel connection failed: 504 Gateway Timeout

Like i said before, the weird thing is that the connection did work with older versions of the CV. I deleted the newer versions and switched to the version that did work before, but it doesnt work either now

13

The issue is solved. Disk space full. Thought of everything but that. Thank you for your help!