Orphaned host in canldepin DB

Support
1

Problem:
We have a host that cannot be registered to Katello via subscription-manager.
When trying to register, the following error occures:

[root@server~]# subscription-manager register --org="ORG" --name="server.example.com" --activationkey="list,of,activationkeys"
Consumer with id 5db95330-0055-46ad-8ca2-3adf105b30f9 could not be found. (HTTP error code 404: Not Found)

The system is a baremetal host we installed via kickstart (entered the required info into Foreman and then did regular netboot).
During the installation, the subscription-manager command threw above error message and trying to run it manually always results in the same error with a different consumer id. Yet the host and content host exist within Foreman.

Expected outcome:
Registration succeeds

Foreman and Proxy versions:
3.2.1

Foreman and Proxy plugin versions:

  • foreman-tasks 6.0.1
  • foreman_expire_hosts 7.0.4
  • foreman_hooks 0.3.17
  • foreman_puppet 3.0.7
  • foreman_remote_execution 6.0.0
  • foreman_scc_manager 1.8.20
  • foreman_snapshot_management 2.0.1
  • foreman_templates 9.1.0
  • foreman_webhooks 3.0.3
  • katello 4.4.2.2
  • puppetdb_foreman 5.0.0

Distribution and version:
RHEL8
Other relevant data:
The host was formerly created as a VM (by error), deleted and then re-created correctly as bare-metal.
I found Error registing host to katello which describes a similar/maybe the same problem and have also tried the linked RH KB article from there. After deleting the host in the described manner and then re-running subscription-manager register, I get the same error, though the subscription-manager registration does re-create the host and the content host.
After digging a little, I found that the candlepin DB still holds a record for a host by the same name, despite the host being deleted from Foreman:

candlepin=# select * from cp_consumer where name LIKE 'server%';
                id                |          created           |          updated           | autoheal | entitlementstatus |            name             | servicelevel |   username    |                 uuid
              |          environment_id          |        consumer_idcert_id        |            keypair_id            |             owner_id             | type_id | releasever |                       complian
cestatushash                       | lastcheckin | annotations | cont_acc_cert_id | content_access_mode | entitlement_count | sp_role | sp_usage |   sp_status   | sp_status_hash | rh_cloud_profile_modified  |
sp_service_type
----------------------------------+----------------------------+----------------------------+----------+-------------------+-----------------------------+--------------+---------------+------------------------
--------------+----------------------------------+----------------------------------+----------------------------------+----------------------------------+---------+------------+-------------------------------
-----------------------------------+-------------+-------------+------------------+---------------------+-------------------+---------+----------+---------------+----------------+----------------------------+-
----------------
 8a206c95887c3ebf0189976f2dee13b4 | 2023-07-27 14:59:53.198+02 | 2023-11-22 15:49:22.655+01 | t        | valid             | server.example.com          |              | foreman_admin | 8ed89969-00c1-4090-acd3
-d6e9488a1cdd | 2ce75e21715e1492487b8eb6ded84c4d | 8a206c95899cdbfb018bf74118a50260 | 8a206c95887c3ebf0189976f376413df | 8a206c95563100b70156310125590001 | 1000    |            | c2848ac2d6d4dbdbdb0ab742d61ce5
2293f2716ffb290368a3eac99dbdfa23ab |             |             |                  |                     |                 6 |         |          | not specified |                | 2023-11-22 15:33:54.144+01 |

I suspect this to be the root cause of the problem and would like to know how to get rid of that if so. But I’d be glad for other help, too :slight_smile:

2

@nofaralfasi I know everyone is currently busy with release stuff, but could you maybe get someone with knowledge of Candlepin to take a look at this?
Thanks in advance :slight_smile:

3

So, I did a little more digging and it turns out that this system somehow got “mixed up” with another system. The uuid associated with the host was somehow associated with a completely different system in the foreman/katello db:

foreman=# select * from katello_subscription_facets where uuid LIKE '8ed89969-00c1-4090-ac%';
  id   | host_id |                 uuid                 |        last_checkin        | service_level | release_version | autoheal |    registered_at    |    registered_through     | user_id | hypervisor | hype
rvisor_host_id | purpose_usage | purpose_role |               dmi_uuid
-------+---------+--------------------------------------+----------------------------+---------------+-----------------+----------+---------------------+---------------------------+---------+------------+-----
---------------+---------------+--------------+--------------------------------------
 16345 |    6063 | 8ed89969-00c1-4090-acd3-d6e9488a1cdd | 2023-12-01 12:22:22.073223 |               |                 | t        | 2023-07-27 12:59:53 | smartproxy.example.com|         | f          |
               |               |              | BAA23EF3-8507-5C48-0000-000000000044
(1 row)
foreman=# select name from hosts where id='6063';
          name
-------------------------
 differentserver.example.com
(1 row)

So I tried subscription-manager unregister followed by a fresh subscription-manager register on differentserver.example.com and that seems to have fixed the issue.
We already tried installing the host with a different name in the meantime, but all that did was change the name in the DB entry from the first post. Now we can cleanly do subscription-manager register on the new host and everything seems to work as expected again.
No idea how these to systems got mangled up in this way, but at least it’s gone now.

1 Like