Failed tasks to update packages - proxy task gone missing

SunnyK · July 6, 2023, 3:27pm

Job/task/remote action to update packages is failing with error message “Proxy task gone missing from the smart proxy”

Expected outcome is task success

Foreman 3.5.3 and Proxy version 3.5.3

CentOS Stream 8

First of all apologies since I’m new to Foreman/Katello and have inherited the management of this system since one of our Linux admins just left. My main ask here as well as some insight into this particular problem, is how best to troubleshoot Foreman, location of logs etc. so feel free to point me in the right direction.

We have a single smart proxy with 15 hosts, content subscriptions are working fine, however the tasks > Remote action: Update package(s) on all 15 fails with “Proxy task gone missing from the smart proxy”

This is a new problem, the jobs have run successfully in the past but don’t know what’s triggered this.

Please let me know if you need me to post any logs etc. Cheers.

areyus · July 7, 2023, 7:15am

Hi and welcome to the forum

While I cannot help with the actual problem at hand (never seen this error, nor do I have any idea where it may come from), I can at least point you to the logs:

The logs for the main Foreman application are at /var/log/foreman/production.log
The main log for smart-proxies is /var/log/foreman-proxy/proxy.log

Maybe you can find some more info there.

SunnyK · July 11, 2023, 6:25pm

Thanks @areyus I checked the logs and nothing helpful. I did note however that when attempting a manual update on a client manged by foreman I received the following error;

Total size: 385 M
Total download size: 51 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for HH_CENTOS_7_CENTOS_7_BASE
libatomic-4.8.5-44.el7.x86_64. FAILED
https://foreman.lin.ax.local/pulp/content/HH/Library/CENTOS7_MGMT/custom/CENTOS_7/CENTOS_7_BASE/Packages/l/libatomic-4.8.5-44.el7.x86_64.rpm: [Errno 14] curl#60 - “Peer’s certificate issuer has been marked as not trusted by the user.”
Trying other mirror.

Error downloading packages:
libatomic-4.8.5-44.el7.x86_64: [Errno 256] No more mirrors to try.

Any ideas around the cert error?

cintrix84 · July 11, 2023, 6:54pm

Hi @SunnyK

Did you install the Foreman instance with a custom cert initially or change it after the install? On the client if you do subscription-manager refresh does it show any SSL errors in /var/log/rhsm.log

SunnyK · July 12, 2023, 7:11am

Thanks @cintrix84 unfortunately I don’t know the answer to your first question since I inherited the system. I will try and find out.

With regard to the logs, no errors that I can see.

-sh-4.2$ sudo subscription-manager refresh
[sudo] password for adm-sunnyd@ax.local:
All local data refreshed

-sh-4.2$ cat /var/log/rhsm/rhsm.log
2023-07-09 03:11:02,139 [INFO] rhsmd:6706:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionManager provided by rhsmd is deprecated
2023-07-09 03:11:02,139 [INFO] rhsmd:6706:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat.RHSM1 provided by rhsm.service
2023-07-09 03:11:02,142 [INFO] rhsmd:6706:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 05:32:41,805 [INFO] rhsmcertd-worker:7436:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 05:32:42,017 [INFO] rhsmcertd-worker:7436:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 05:32:42,826 [WARNING] rhsmcertd-worker:7436:MainThread @logutil.py:142 - logging already initialized
2023-07-09 05:32:42,830 [INFO] rhsmcertd-worker:7436:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 05:32:42,944 [INFO] rhsmcertd-worker:7436:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 09:32:41,867 [INFO] rhsmcertd-worker:8762:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 09:32:42,075 [INFO] rhsmcertd-worker:8762:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 09:32:42,911 [WARNING] rhsmcertd-worker:8762:MainThread @logutil.py:142 - logging already initialized
2023-07-09 09:32:42,915 [INFO] rhsmcertd-worker:8762:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 09:32:43,037 [INFO] rhsmcertd-worker:8762:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 09:43:01,270 [INFO] rhsmcertd-worker:8904:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 09:43:01,741 [WARNING] rhsmcertd-worker:8904:MainThread @healinglib.py:116 - Got valid status from server but no valid until date.
2023-07-09 09:43:01,856 [INFO] rhsmcertd-worker:8904:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 13:32:41,851 [INFO] rhsmcertd-worker:10102:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 13:32:42,025 [INFO] rhsmcertd-worker:10102:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 13:32:42,853 [WARNING] rhsmcertd-worker:10102:MainThread @logutil.py:142 - logging already initialized
2023-07-09 13:32:42,857 [INFO] rhsmcertd-worker:10102:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 13:32:42,966 [INFO] rhsmcertd-worker:10102:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 17:32:41,880 [INFO] rhsmcertd-worker:11427:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 17:32:42,037 [INFO] rhsmcertd-worker:11427:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 17:32:42,847 [WARNING] rhsmcertd-worker:11427:MainThread @logutil.py:142 - logging already initialized
2023-07-09 17:32:42,851 [INFO] rhsmcertd-worker:11427:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 17:32:42,955 [INFO] rhsmcertd-worker:11427:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 21:32:41,946 [INFO] rhsmcertd-worker:12751:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 21:32:42,177 [INFO] rhsmcertd-worker:12751:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-09 21:32:43,025 [WARNING] rhsmcertd-worker:12751:MainThread @logutil.py:142 - logging already initialized
2023-07-09 21:32:43,029 [INFO] rhsmcertd-worker:12751:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-09 21:32:43,130 [INFO] rhsmcertd-worker:12751:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 01:32:42,013 [INFO] rhsmcertd-worker:14083:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 01:32:42,185 [INFO] rhsmcertd-worker:14083:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 01:32:42,999 [WARNING] rhsmcertd-worker:14083:MainThread @logutil.py:142 - logging already initialized
2023-07-10 01:32:43,003 [INFO] rhsmcertd-worker:14083:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 01:32:43,107 [INFO] rhsmcertd-worker:14083:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 03:12:01,768 [INFO] rhsmd:14731:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionManager provided by rhsmd is deprecated
2023-07-10 03:12:01,768 [INFO] rhsmd:14731:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat.RHSM1 provided by rhsm.service
2023-07-10 03:12:01,771 [INFO] rhsmd:14731:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 05:32:42,098 [INFO] rhsmcertd-worker:15458:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 05:32:42,256 [INFO] rhsmcertd-worker:15458:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 05:32:43,149 [WARNING] rhsmcertd-worker:15458:MainThread @logutil.py:142 - logging already initialized
2023-07-10 05:32:43,152 [INFO] rhsmcertd-worker:15458:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 05:32:43,274 [INFO] rhsmcertd-worker:15458:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 09:32:42,194 [INFO] rhsmcertd-worker:16786:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 09:32:42,391 [INFO] rhsmcertd-worker:16786:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 09:32:43,204 [WARNING] rhsmcertd-worker:16786:MainThread @logutil.py:142 - logging already initialized
2023-07-10 09:32:43,208 [INFO] rhsmcertd-worker:16786:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 09:32:43,316 [INFO] rhsmcertd-worker:16786:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 09:43:01,272 [INFO] rhsmcertd-worker:16927:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 09:43:01,772 [WARNING] rhsmcertd-worker:16927:MainThread @healinglib.py:116 - Got valid status from server but no valid until date.
2023-07-10 09:43:01,892 [INFO] rhsmcertd-worker:16927:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 13:32:42,219 [INFO] rhsmcertd-worker:18127:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 13:32:42,383 [INFO] rhsmcertd-worker:18127:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 13:32:43,216 [WARNING] rhsmcertd-worker:18127:MainThread @logutil.py:142 - logging already initialized
2023-07-10 13:32:43,220 [INFO] rhsmcertd-worker:18127:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 13:32:43,340 [INFO] rhsmcertd-worker:18127:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 17:32:42,287 [INFO] rhsmcertd-worker:19456:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 17:32:42,490 [INFO] rhsmcertd-worker:19456:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 17:32:43,321 [WARNING] rhsmcertd-worker:19456:MainThread @logutil.py:142 - logging already initialized
2023-07-10 17:32:43,324 [INFO] rhsmcertd-worker:19456:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 17:32:43,447 [INFO] rhsmcertd-worker:19456:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 21:32:42,320 [INFO] rhsmcertd-worker:20782:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 21:32:42,537 [INFO] rhsmcertd-worker:20782:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-10 21:32:43,355 [WARNING] rhsmcertd-worker:20782:MainThread @logutil.py:142 - logging already initialized
2023-07-10 21:32:43,359 [INFO] rhsmcertd-worker:20782:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-10 21:32:43,467 [INFO] rhsmcertd-worker:20782:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 01:32:42,438 [INFO] rhsmcertd-worker:22112:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 01:32:42,581 [INFO] rhsmcertd-worker:22112:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 01:32:43,423 [WARNING] rhsmcertd-worker:22112:MainThread @logutil.py:142 - logging already initialized
2023-07-11 01:32:43,427 [INFO] rhsmcertd-worker:22112:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 01:32:43,525 [INFO] rhsmcertd-worker:22112:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 03:35:02,212 [INFO] rhsmd:22894:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionManager provided by rhsmd is deprecated
2023-07-11 03:35:02,212 [INFO] rhsmd:22894:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat.RHSM1 provided by rhsm.service
2023-07-11 03:35:02,215 [INFO] rhsmd:22894:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 05:32:42,484 [INFO] rhsmcertd-worker:23487:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 05:32:42,703 [INFO] rhsmcertd-worker:23487:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 05:32:43,536 [WARNING] rhsmcertd-worker:23487:MainThread @logutil.py:142 - logging already initialized
2023-07-11 05:32:43,540 [INFO] rhsmcertd-worker:23487:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 05:32:43,650 [INFO] rhsmcertd-worker:23487:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 09:32:42,517 [INFO] rhsmcertd-worker:24814:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 09:32:42,677 [INFO] rhsmcertd-worker:24814:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 09:32:43,460 [WARNING] rhsmcertd-worker:24814:MainThread @logutil.py:142 - logging already initialized
2023-07-11 09:32:43,464 [INFO] rhsmcertd-worker:24814:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 09:32:43,618 [INFO] rhsmcertd-worker:24814:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 09:43:01,378 [INFO] rhsmcertd-worker:24955:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 09:43:01,886 [WARNING] rhsmcertd-worker:24955:MainThread @healinglib.py:116 - Got valid status from server but no valid until date.
2023-07-11 09:43:01,986 [INFO] rhsmcertd-worker:24955:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 13:32:42,531 [INFO] rhsmcertd-worker:26152:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 13:32:42,750 [INFO] rhsmcertd-worker:26152:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 13:32:43,595 [WARNING] rhsmcertd-worker:26152:MainThread @logutil.py:142 - logging already initialized
2023-07-11 13:32:43,599 [INFO] rhsmcertd-worker:26152:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 13:32:43,714 [INFO] rhsmcertd-worker:26152:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 17:32:42,623 [INFO] rhsmcertd-worker:27478:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 17:32:42,775 [INFO] rhsmcertd-worker:27478:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 17:32:43,602 [WARNING] rhsmcertd-worker:27478:MainThread @logutil.py:142 - logging already initialized
2023-07-11 17:32:43,606 [INFO] rhsmcertd-worker:27478:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 17:32:43,709 [INFO] rhsmcertd-worker:27478:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 17:38:57,885 [INFO] yum:27688:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 17:38:58,039 [INFO] yum:27688:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 17:39:11,230 [WARNING] yum:27688:MainThread @logutil.py:142 - logging already initialized
2023-07-11 17:39:11,235 [INFO] yum:27688:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 17:39:11,369 [INFO] yum:27688:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 17:39:12,057 [WARNING] yum:27688:MainThread @logutil.py:142 - logging already initialized
2023-07-11 18:55:55,269 [INFO] yum:28189:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 18:55:55,401 [INFO] yum:28189:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 21:32:42,731 [INFO] rhsmcertd-worker:29029:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 21:32:42,897 [INFO] rhsmcertd-worker:29029:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-11 21:32:43,753 [WARNING] rhsmcertd-worker:29029:MainThread @logutil.py:142 - logging already initialized
2023-07-11 21:32:43,757 [INFO] rhsmcertd-worker:29029:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-11 21:32:43,881 [INFO] rhsmcertd-worker:29029:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-12 01:32:42,789 [INFO] rhsmcertd-worker:30365:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 01:32:42,961 [INFO] rhsmcertd-worker:30365:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-12 01:32:43,798 [WARNING] rhsmcertd-worker:30365:MainThread @logutil.py:142 - logging already initialized
2023-07-12 01:32:43,802 [INFO] rhsmcertd-worker:30365:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 01:32:43,924 [INFO] rhsmcertd-worker:30365:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-12 03:22:01,855 [INFO] rhsmd:31056:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionManager provided by rhsmd is deprecated
2023-07-12 03:22:01,855 [INFO] rhsmd:31056:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat.RHSM1 provided by rhsm.service
2023-07-12 03:22:01,858 [INFO] rhsmd:31056:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 05:32:42,840 [INFO] rhsmcertd-worker:31746:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 05:32:43,002 [INFO] rhsmcertd-worker:31746:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-12 05:32:43,841 [WARNING] rhsmcertd-worker:31746:MainThread @logutil.py:142 - logging already initialized
2023-07-12 05:32:43,845 [INFO] rhsmcertd-worker:31746:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 05:32:43,954 [INFO] rhsmcertd-worker:31746:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-12 08:00:26,977 [INFO] rhsmd:32600:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionManager provided by rhsmd is deprecated
2023-07-12 08:00:26,977 [INFO] rhsmd:32600:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat.RHSM1 provided by rhsm.service
2023-07-12 08:00:27,023 [INFO] subscription-manager:32588:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 08:00:27,034 [INFO] subscription-manager:32588:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 08:00:27,034 [INFO] subscription-manager:32588:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=none
2023-07-12 08:00:27,161 [INFO] subscription-manager:32588:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6312051260221814305L]
Expected (UEP) serial# [6312051260221814305]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-07-12 08:00:27,797 [INFO] subscription-manager:32588:MainThread @connection.py:915 - Connection built: http_proxy=proxy.ax.local:3128 host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-07-12 08:00:27,801 [INFO] rhsmd:32600:MainThread @connection.py:915 - Connection built: host=foreman.lin.ax.local port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
-sh-4.2$

cintrix84 · July 12, 2023, 3:33pm

Totally fine on being new to Foreman, we will get it figured out.

So that is good that we don’t see any errors in the rhsm.log

On the Foreman instance, let’s see if we are using a custom SSL cert

Can you run this command for me:

cat /etc/foreman-installer/scenarios.d/satellite-answers.yaml | grep 'server_cert'

SunnyK · July 13, 2023, 1:52pm

Thanks @cintrix84. We don’t have a satellite-answers folder, which I’m going to assume would be available if we had a Red Hat subscription? We don’t have that.

I do have foreman-answers and katello-answers

foreman-answers.yaml
foreman-answers.yaml.rpmnew
foreman.migrations → /usr/share/foreman-installer/config/foreman.migrations
foreman-migrations-applied
foreman-proxy-content-answers.yaml
foreman-proxy-content.migrations → /usr/share/foreman-installer/config/foreman-proxy-content.migrations
foreman-proxy-content-migrations-applied
foreman-proxy-content.yaml
foreman.yaml
katello-answers.yaml
katello-answers.yaml.rpmnew
katello.migrations → /usr/share/foreman-installer/config/katello.migrations
katello-migrations-applied
katello.yaml
last_scenario.yaml → katello.yaml

and we have an entry in katello-answers

[root@foreman scenarios.d]# cat ./*-*.yaml | grep 'server_cert' server_cert: "/etc/pki/katello-certs-tools/certs/katello.crt" server_cert_req: server_certname: foreman.lin.ax.local [root@foreman scenarios.d]#

Does that help?

cintrix84 · July 13, 2023, 5:46pm

That does help, sorry for the satellite-answers.yaml file. When you have multiple boxes sometimes you just ssh to one out of habit.

Are all of the clients giving the SSL error or just this one?

Can you do the following for me:

Client:

md5sum /etc/rhsm/ca/katello-server-ca.pem
md5sum /etc/rhsm/ca/katello-default-ca.pem

Server:

md5sum /etc/pki/katello/certs/katello-default-ca.crt
md5sum /etc/pki/katello/certs/katello-server-ca.crt

Let me know if those match up?

SunnyK · July 13, 2023, 6:36pm

All of the clients have the error
(only confirmed a handful however all the jobs are stopped with error in the console so my assumption is they are all the same. Happy to check each one if required).

Do they certs mach-up? No sir - they do not!

Client
katello-server ends in 51c
katello-default ends in 51c

Server
katello-server ends in 3a8
katello-default ends in 454

I think maybe you’re going to suggest I regenerate the certs for both?

cintrix84 · July 14, 2023, 6:50pm

How many clients do you have connected to the Foreman instance? Also do you have Ansible or Remote Execution setup where you could distribute the updated cert rpm to the clients without going to each of them?

I am trying to think of the best path forward without causing too much disruption.

SunnyK · July 17, 2023, 10:59am

Thanks @cintrix84 … I have 15 clients and Foreman is not currently setup to use Ansible/Remote Execution, however I am interested in getting this setup and using it in the future.

For now and for the purposes of this fix, I’d like to manually update the certs on one client to prove that’s the only problem I have.

I’m just now searching for documentation around regenerating the certs and have found this https://theforeman.org/plugins/katello/nightly/advanced/certificates.html however if you can advise of the best way to do this please do let me know.

cintrix84 · July 17, 2023, 3:34pm

@SunnyK That procedure looks correct to update the certs. That will generate a new katello-ca-consumer with the updated custom certs you are using.

If you just want to use the self-signed ones the installer makes at install time you can pass this to the installer:
--certs-reset

This option will reset any custom certificates and use the self-signed CA instead. Note that any clients will need to be updated with the latest katello-ca-consumer RPM, and any external proxies will need to have the certs updated by generating a new certs tarball.

duanebv · February 1, 2024, 8:05pm

I am encountering this same issue having upgraded to Foreman 3.9.1 with Katello 4.11.0. I am running on Rocky 8.9 for my operating system. I can select a batch of hosts (doesn’t seem to matter how many) and it randomly will have a small percentage (say 2 out of 10, but inconsistent) that get the error “Proxy task gone missing from the smart proxy”. My smart proxy and my Foreman are on the same server and I have not made any changes to the default certificates. In every case, the host(s) on which this happens completes the task even though the job is no longer tracking the actions against the errant host. I see nothing out of the ordinary in the production log or proxy logs when this occurs.

Foreman Version 3.9.1
Foreman Tasks 9.0.1
Foreman Ansible 13.0.3
Foreman Puppet 6.1.1
Foreman Remote Execution 12.0.4
Katello 4.11.0
Candlepin 4.3.1

duanebv · February 6, 2024, 8:22pm

This almost seems like a timeout for long operations, e.g. patching. Is anyone else seeing this behavior? It is still present after the update to Remote Execution 12.0.5.

duanebv · February 16, 2024, 10:50pm

Not sure if it is fully resolved, but since the update from candlepin-4.3.1-1 to candlepin-4.3.12-1 and the same version updates for the candlepin-selinux package, this has no longer occurred. I will continue to monitor.

Bernhard_Suttner · March 12, 2024, 4:58pm

@duanebv Did you have the issue again or is it gone?

duanebv · March 12, 2024, 7:42pm

It is still occurring, yes. I haven’t patched Foreman since my last post, so not sure if any new packages have been released. This is running in an air-gapped network, so updates are quite manual.

nadjaheitmann · March 14, 2024, 12:41pm

I had a look at this as well and what I get in the backtrace is:

2024-03-14T12:33:09 [I|app|04285914] Processing by JobInvocationsController#show as JSON
2024-03-14T12:33:09 [I|app|04285914]   Parameters: {"page"=>"1", "per_page"=>"5000", "id"=>"6772"}
2024-03-14T12:33:09 [E|dyn|433f33cb] Failed to run hook 'emit_event_failure' for action 'Actions::RemoteExecution::RunHostJob'
2024-03-14T12:33:09 [E|dyn|433f33cb] wrong number of arguments (given 0, expected 1) (ArgumentError)
 433f33cb | /usr/share/gems/gems/foreman_webhooks-3.2.2/app/lib/foreman_webhooks/renderer/scope/webhook_template.rb:31:in `payload'
 433f33cb | Ansible Tower - Trigger Job:4:in `get_binding'
 433f33cb | /usr/share/ruby/erb.rb:905:in `eval'
 433f33cb | /usr/share/ruby/erb.rb:905:in `result'
 433f33cb | /usr/share/foreman/app/services/foreman/renderer/unsafe_mode_renderer.rb:7:in `render'
 433f33cb | /usr/share/foreman/app/services/foreman/renderer/base_renderer.rb:18:in `render'
 433f33cb | /usr/share/foreman/app/services/foreman/renderer.rb:46:in `render'
 433f33cb | /usr/share/foreman/app/models/template.rb:172:in `render'
 433f33cb | /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:133:in `rendered_payload'
 433f33cb | /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:70:in `deliver'
 433f33cb | /usr/share/gems/gems/foreman_webhooks-3.2.2/app/models/webhook.rb:48:in `block in deliver'
 433f33cb | /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/delegation.rb:88:in `each'

RunHostJob is the parent task of the one that goes missing as far as I could see. Anyone an idea what went wrong?

m-bucher · March 18, 2024, 1:48pm

@duanebv, I could reproduce this and have created an issue for it: Bug #37272: Long running task is killed with 'Proxy task gone missing from the smart proxy' - foreman-tasks - Foreman

@aruzicka was nice enough to find a fix that solves the issue (at least for me): Fixes #37272 - fix task gone missing by adamruzicka · Pull Request #746 · theforeman/foreman-tasks · GitHub

duanebv · April 23, 2024, 8:21pm

Just to close the loop, I have not had this issue since applying the latest patches. Thank you for your efforts.

Foreman 3.9.1.
Katello 4.11.1
Remote Execution Plugin 12.0.5
Foreman-Tasks Plugin 9.0.4