Problem:
Hosts with the exact same fqdn get duplicated.
Expected outcome:
Only 1 host with the same fqdn should ever be created.
How is this possible?
What in the world does Foreman use to determine uniqueness of a host? Should it not be the fqdn?
So, we today discovered another duplicate host on our 3.9 instance, too. This time the creation of the duplicate host was recent enough that I still hat the corresponding logs and I could narrow down where things are going wrong, though I have now clue as to why it is going wrong.
In our case, the issue occured on a config (in our case Puppet) report upload, though as far as I can tell, all config management systems use the same logic in the relevant spots, so it should be identical for salt. There is also nearly identical behavior for fact imports.
Here is what I found in our logs:
2025-03-26T13:14:52 [I|app|c2122db1] Started POST "/api/config_reports" for [smart-proxy ip] at 2025-03-26 13:14:52 +0100
2025-03-26T13:14:52 [I|app|c2122db1] Processing by Api::V2::ConfigReportsController#create as JSON
2025-03-26T13:14:52 [I|app|c2122db1] Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2025-03-26T13:14:52 [W|tax|c2122db1] Current user is set, but not expected. Clearing
2025-03-26T13:14:52 [I|aud|c2122db1] Nic::Managed (27602) create event on mac
[... continues with the complete host creation process]
[Finally runs the report processor]
What it boils down to is that this method is called and tries to find the corresponding host object in Foreman and creates a new object in case it cannot find a matching existing record.
The behavior of fact imports is basically the same, but directly in the Host::Base class.
So for some reason, Foreman seems to sometimes be unable to find the existing host records, though I was unable to find out why this happens. In theory, there should be another safety net since the host model also has a validation for uniqueness on the name (though, I cannot tell if this would help if the existing record is not found in the first place), but both the report processor as well as the Fact importer force-skip validations when saving new host objects.
So the underlying question that needs to be answered is “why can Foreman sometimes not find the existing host?”
I am additionally baffled by the fact that the dupe from my current iteration of this problem has a location (identical to the original host) and an organization (differing from the original) set, and I have absolutely no idea where this data came from. Earlier instances of this happening to us always had org and loc set to “none”.
The “Current user is set, but not expected. Clearing” from here in the code smells somewhat like a multithreading problem, but that might very well be a red hering.
Thanks for the response. Glad to see Im not the only one having this issue.
One thing we did was disable fact import host creation, and left report import host creation enabled. With this running for 2 days, we did not see a single duplicated host. We then disabled report host creation and enabled fact import host creation. Immediately we saw dozens of duplicate hosts coming in. So it must be somewhere in the fact importer. In our case, we dont really want to go down the rabbit hole of modifying foreman source code, thus we are going to likely put in guardrails and additional checkers in our fact upload script. Its not ideal, because now for every single host that runs salt (around 90,000 a day), we’ll have to have one additional API call that checks to see if the host exists in Foreman or not.
I can confirm from experience, that fact imports are much more prone to this behavior that report imports. We had quite a few occurrences of this problem in the past from fact imports (by far not the dozens of hosts you are seeing, but our environement is way smaller with about 3.5k hosts, though these also generat around 85k Puppet reports per day). We then went on to disable host creation on fact uploads and the logs I posted above were the first time we saw this happen in about two years.
I took a look around the issue tracker and found a few issues that look related to this issue:
From the looks of it, your use-case of provisioning hosts outside of Foreman and then have them register via fact uploads might even increase the chance of duplicates happening, if I understand the linked issues correctly.
Maybe adjusting your provisioning workflow to create the Foreman host beforehand or using global registration before the first configuration tool run could be a solution might want to look into, so you do not have to rely on the “import from facts/catalogue” feature or script additional overhead into your fact upload script?
Yes, we are actively working on a better way to manage that. The problem we have is with an infrastructure of some 90,000 hosts and counting, there was no sort of ENC or centralized system, where I work, except that salt is installed on everything, through every OS image. We’ve since changed our cloud system to create the hosts in Foreman first. That reduced about 80-90% of the duplicates. Where we are getting issues is with groups still using “gold” images.
We may end up turning off fact import host creation, or more likely we’ll end up rewriting it to add more guard rails and checks for hosts, before it actually creates one.