Unable to register host to new Foreman

rfm33 · February 21, 2023, 5:20pm

Problem:
Hello! I am migrating hosts from an old Foreman instance to a new one. Similar hosts have been able to register successfully with no issues but others are unable to and I am currently stumped. Connectivity is all in place.

When running the generated curl command:

Remote server error. Please check the connection details, or see /var/log/rhsm/rhsm.log for more information.
This system is not yet registered. Try ‘subscription-manager register --help’ for more information.
ERROR: not_found
Host was not found by the subscription UUID: ’ ', this can happen if the host is registered already, but not to this instance

output of /var/log/rhsm/rhsm.log:

2023-02-21 11:24:27,415 [INFO] subscription-manager:23251:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:27,431 [INFO] subscription-manager:23251:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:27,432 [INFO] subscription-manager:23251:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=none
2023-02-21 11:24:27,776 [INFO] subscription-manager:23279:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:28,137 [INFO] yum:23292:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:28,137 [INFO] yum:23292:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# []
Expected (UEP) serial# []
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-02-21 11:24:29,010 [INFO] yum:23304:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:29,010 [INFO] yum:23304:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# []
Expected (UEP) serial# []
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-02-21 11:24:30,464 [INFO] rhsmd:23332:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionManager provided by rhsmd is deprecated
2023-02-21 11:24:30,464 [INFO] rhsmd:23332:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat.RHSM1 provided by rhsm.service
2023-02-21 11:24:30,489 [INFO] subscription-manager:23318:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:30,852 [INFO] subscription-manager:23346:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:31,212 [INFO] subscription-manager:23359:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:31,579 [INFO] subscription-manager:23374:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:31,593 [INFO] subscription-manager:23374:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:31,594 [INFO] subscription-manager:23374:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=none
2023-02-21 11:24:31,598 [INFO] subscription-manager:23374:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=none
2023-02-21 11:24:52,713 [ERROR] subscription-manager:23374:MainThread @managercli.py:217 - Error during registration: ''
2023-02-21 11:24:52,713 [ERROR] subscription-manager:23374:MainThread @managercli.py:218 - ''
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 1389, in _do_command
    type=self.options.consumertype
  File "/usr/lib64/python2.7/site-packages/rhsmlib/services/register.py", line 93, in register
    jwt_token=jwt_token
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1063, in registerConsumer
    return self.conn.request_post(url, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 733, in request_post
    return self._request("POST", method, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 756, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 607, in _request
    response = conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1144, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 457, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 421, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''
2023-02-21 11:24:53,086 [INFO] subscription-manager:23647:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:53,100 [INFO] subscription-manager:23647:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-02-21 11:24:53,101 [INFO] subscription-manager:23647:MainThread @connection.py:915 - Connection built: host=<proxy-address-here> port=443 handler=/rhsm auth=none
2023-02-21 11:46:27,622 [ERROR] rhsmcertd-worker:28288:MainThread @rhsmcertd_worker.py:211 - Either the consumer is not registered or the certificates are corrupted. Certificate update using daemon failed.

Expected outcome:
The system to be successfully registered

Foreman and Proxy versions:
3.1.3

Foreman and Proxy plugin versions:
3.1.3

Distribution and version:
RHEL 7.9

Other relevant data:

Any insight would be appreciated.

lstejska · March 8, 2023, 6:57am

Hi,
are you using registration method from Foreman (Hosts > Register host), or are you downloading rpm with subscription manager and then registering host to second Foreman?

rfm33 · March 8, 2023, 8:14pm

Hi, i’m using the registration method from hosts > register host where it generates the curl command.

lstejska · March 9, 2023, 7:59am

Cool, thanks, can you please share:

Generated curl command for the registration
Output of foreman app log (in var/log/foreman/production.log), ideally with debug level
If you register host through smar-proxy, then I would need also log from the smart proxy, in var/log/foreman-proxy/, ideally with debug level
Do you use default templates or do you have any customization?

You can enable debug level with:
foreman-installer --foreman-logging-level "debug" --foreman-proxy-log-level "DEBUG"

rfm33 · March 9, 2023, 9:01pm

I have enabled debug on both the main foreman as well as the proxy.

Curl Command

curl -sS --insecure 'https://wel4-prfrm01.prd.securekey.com:9090/register?activation_keys=rhel7_akey&force=true&ignore_subman_errors=true&location_id=3&organization_id=1&update_packages=false' -H 'Authorization: Bearer << token omitted >>' | bash

Output of proxy log

2023-03-09T15:46:14  [D] Executor heartbeat
2023-03-09T15:46:29  [D] Executor heartbeat
2023-03-09T15:46:36  [D] accept: 10.88.10.10:41520
2023-03-09T15:46:36  [D] Rack::Handler::WEBrick is invoked.
2023-03-09T15:46:36 1e3626ca [I] Started GET /register activation_keys=rhel7_akey&force=true&ignore_subman_errors=true&location_id=3&organization_id=1&update_packages=false
2023-03-09T15:46:37 1e3626ca [I] Finished GET /register with 200 (544.75 ms)
2023-03-09T15:46:37  [D] close: 10.88.10.10:41520
2023-03-09T15:46:44  [D] Executor heartbeat
2023-03-09T15:46:59  [D] Executor heartbeat
2023-03-09T15:47:14  [D] Executor heartbeat
2023-03-09T15:47:29  [D] Executor heartbeat
2023-03-09T15:47:44  [D] Executor heartbeat
2023-03-09T15:47:59  [D] Executor heartbeat
2023-03-09T15:48:14  [D] Executor heartbeat
2023-03-09T15:48:29  [D] Executor heartbeat
2023-03-09T15:48:44  [D] accept: 10.88.10.10:41534
2023-03-09T15:48:44  [D] Rack::Handler::WEBrick is invoked.
2023-03-09T15:48:44 3110d827 [I] Started POST /register
2023-03-09T15:48:44 3110d827 [I] Finished POST /register with 404 (81.9 ms)
2023-03-09T15:48:44  [D] close: 10.88.10.10:41534

Output of foreman production log has been attached.
production.log (126.4 KB)
Do you use default templates or do you have any customization?

No

lstejska · March 13, 2023, 8:02am

I see that you are ignoring errors from the subscription manager.
I guess that’s going to be the root of the problem.
Ignoring subscription-manager errors is there for backwards compatibility with older versions of subman, I would suggest to run command without ignoring the errors and see why is subscription-manager register command failing

rfm33 · March 13, 2023, 4:28pm

even without the subman_errors flag rhsm.log is still throwing the same errors.

2023-03-13 12:24:12,517 [INFO] rhsmd:74973:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionMa                 nager provided by rhsmd is deprecated
2023-03-13 12:24:12,517 [INFO] rhsmd:74973:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat                 .RHSM1 provided by rhsm.service
2023-03-13 12:24:12,544 [INFO] subscription-manager:74961:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:12,560 [INFO] subscription-manager:74961:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:12,560 [INFO] subscription-manager:74961:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=none
2023-03-13 12:24:12,926 [INFO] subscription-manager:74985:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:13,361 [INFO] yum:74996:MainThread @connection.py:915 - Connection built: host=wel4-prfrm0                 1.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-03-13 12:24:13,362 [INFO] yum:74996:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# []
Expected (UEP) serial# []
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-03-13 12:24:14,255 [INFO] yum:75008:MainThread @connection.py:915 - Connection built: host=wel4-prfrm0                 1.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2023-03-13 12:24:14,255 [INFO] yum:75008:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# []
Expected (UEP) serial# []
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2023-03-13 12:24:15,401 [INFO] rhsmd:75032:MainThread @rhsm_d.py:382 - D-Bus API: com.redhat.SubscriptionMa                 nager provided by rhsmd is deprecated
2023-03-13 12:24:15,401 [INFO] rhsmd:75032:MainThread @rhsm_d.py:383 - Consider using D-Bus API: com.redhat                 .RHSM1 provided by rhsm.service
2023-03-13 12:24:15,434 [INFO] subscription-manager:75020:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:15,834 [INFO] subscription-manager:75044:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:16,224 [INFO] subscription-manager:75055:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:16,629 [INFO] subscription-manager:75068:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:16,645 [INFO] subscription-manager:75068:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecur                 e=False
2023-03-13 12:24:16,646 [INFO] subscription-manager:75068:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=none
2023-03-13 12:24:16,649 [INFO] subscription-manager:75068:MainThread @connection.py:915 - Connection built:                  host=wel4-prfrm01.prd.securekey.com port=443 handler=/rhsm auth=none
2023-03-13 12:25:17,198 [ERROR] subscription-manager:75068:MainThread @managercli.py:217 - Error during reg                 istration: ''
2023-03-13 12:25:17,198 [ERROR] subscription-manager:75068:MainThread @managercli.py:218 - ''
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 1389, in _do_command
    type=self.options.consumertype
  File "/usr/lib64/python2.7/site-packages/rhsmlib/services/register.py", line 93, in register
    jwt_token=jwt_token
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1063, in registerConsumer
    return self.conn.request_post(url, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 733, in request_post
    return self._request("POST", method, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 756, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 607, in _request
    response = conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1144, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 457, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 421, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

rfm33 · March 13, 2023, 4:43pm

i’m led to believe it can’t fetch/create entitlement certificates, any idea how to create those manually?

lstejska · March 14, 2023, 8:06am

Looks like the problem is there, if you try preview global registration template in the UI, can you see the content of the CA file in the template?
I can se from the logs that it generates empty content:

SSL_CA_CERT=$(mktemp)
cat << EOF > $SSL_CA_CERT
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

I would check path define in ssl_ca_file setting and verify that it is set to a valid CA file.

rfm33 · March 14, 2023, 6:51pm

This was actually me removing the certificates from the output. Though I believe I have actually solved the issue.

How to register multiple Content Hosts with same UUID to Red Hat Satellite 6? - Red Hat Customer Portal. In one of the comments someone states:

VMware cloning is an example of “hardware” that can produce duplicate UUID. I solved editing .vmx file with: uuid.bios = “”

In my case, we are indeed using VMs that have been cloned. So these are the steps I have taken to resolve the issue, for anyone else that stumbles upon this thread.

Power down the VM on the ESXi host/vCenter.
Log into the ESXi host.
Navigate to the datastore browser and search for the problem host.
Download the .vmx file
Open the .vmx file in your text editor of choice and change the uuid.bios variable to just empty quotations. i.e:

uuid.bios = “”
Save the changes and re-upload the .vmx file to the same location in the datastore browser to overwrite.
Power on the VM and re-run the registration curl command.

lstejska · March 15, 2023, 6:52am

Nice,
glad you find the solution. Maybe setting excluded_facts will be useful to you,
you can try to exclude this fact from reporting.

lukexjin · March 20, 2024, 8:07am

my ssl connection always error,where can I set excluded_facts?
ssl error with this issue?