HTTP error (422 - Unknown): Validation failed: Name has already been taken

Problem:
Trying to register a host (which was possibly registered before) fails with error.
I’ve tried cleaning up everything first:

sudo subscription-manager unregister
sudo subscription-manager remove --all
sudo subscription-manager clean

Then I try to register it:

# subscription-manager register --activationkey=some-key --org=some-org --force
HTTP error (422 - Unknown): Validation failed: Name has already been taken

The server by the name cannot be found in UI or with hammer:

# hammer host list --search server-1.domain.local
---|------|------------------|------------|----|-----|---------------|--------------|-----------------------|-------------
ID | NAME | OPERATING SYSTEM | HOST GROUP | IP | MAC | GLOBAL STATUS | CONTENT VIEW | LIFECYCLE ENVIRONMENT | TRACE STATUS
---|------|------------------|------------|----|-----|---------------|--------------|-----------------------|-------------

It is nowhere to be found in Foreman on any search.
I’ve tried overriding UUID just in case via rhsm facts, no luck.

Expected outcome:
To register normally.

Foreman and Proxy versions:
Foreman 2.3.2

Foreman and Proxy plugin versions:
Katello 3.18.1

Distribution and version:
CentOS 7

Other relevant data:

production.log:

2021-02-18T14:48:27 [I|app|2da915d1] Started POST "/rhsm/consumers?owner=some-org&activation_keys=some-key" for 127.0.0.1 at 2021-02-18 14:48:27 +0000
2021-02-18T14:48:27 [I|app|2da915d1] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
2021-02-18T14:48:27 [I|app|2da915d1]   Parameters: {"name"=>"server-1", "contentTags"=>[], "serviceLevel"=>"", "usage"=>"", "role"=>"", "facts"=>"[FILTERED]", "addOns"=>[], "type"=>"system", "owner"=>"some-org", "activation_keys"=>"some-key"}
2021-02-18T14:48:28 [W|app|2da915d1] Not queueing Host::Managed: ["Name has already been taken"]
2021-02-18T14:48:28 [W|app|2da915d1] Not queueing Host::Managed: ["Name has already been taken"]
2021-02-18T14:48:28 [W|app|2da915d1] Not queueing Host::Managed: ["Name has already been taken"]
2021-02-18T14:48:28 [E|kat|2da915d1] <Class> ActiveRecord::RecordInvalid
2021-02-18T14:48:28 [E|kat|2da915d1] name: ["has already been taken"]
2021-02-18T14:48:28 [I|app|2da915d1] Completed 422 Unprocessable Entity in 220ms (Views: 0.1ms | ActiveRecord: 31.3ms | Allocations: 46822)

The error is very opaque in terms of what is actually raising the validation error. Is it the host name, is it the associated operating system…

I’m curious to see what’s in the database itself. If you run foreman-rake console in the terminal of the Foreman server and enter ::Host.find_by_name('server-1.domain.local') are any results returned?

Are you using puppet on any of your clients?

Thank you for looking into this. My entire infrastructure is managed by Chef, and all my servers are subscribed with subscription-manager via Chef cookbook I wrote using rhsm_register Chef resource. Puppet is only installed on smart proxies deployed around different datacenters during their deployment with foremam-installer.All servers are deployed from the same image.
These facts I’m excluding from DB if this is relevant.

lo,en*v*,usb*,vnet*,macvtap*,_vdsmdummy_,veth*,docker*,tap*,qbr*,qvb*,qvo*,qr-*,qg-*,vlinuxbr*,vovsbr*,proc_cpuinfo*,lscpu*,dmi.baseboard*,dmi.bios*,dmi.chasis*,dmi.memory*,dmi.processor*,cpu*

I have around 10 300 hosts, and this issue has appeared on 5 hosts, 4 in one DC, 1 in other DC.
Checking DB for those hostnames finds nothing, same as hammer and UI search.

# foreman-rake console
Loading production environment (Rails 6.0.3.4)
irb(main):001:0> ::Host.find_by_name('server-1.domain.local')
=> nil
irb(main):002:0> 

Ok! My suspicion is that this doesn’t have anything to do with the name of the hosts (the error would almost certainly be different) but rather the operating systems involved (as I’ve seen this happen a time or two before).

The next step is to take a look at Hosts -> Operating Systems in the UI and see if there are any obvious discrepancies such as anything seemingly duplicated. You may also want to clean up any which are unused (0 hosts associated to them).

It’s be helpful if you could share the output of hammer os list here, as well as subscription-manager facts output from the client host. Based on the facts of the client please also share specific details from the UI for the matching operating system such as the name, major and minor version.