Double entries for new hosts in Foreman?

mweilgart · October 12, 2023, 10:46pm

Using Foreman 3.7, smart proxies 3.7, deploying Debian 11.8, deploying on libvirt, I had everything working and now of a sudden I see duplicate host entries getting created in Foreman and provisioning will not complete.

The error in the console shows (with token and hostname changed):

[!!] Finish the installation

Failed to run preseeded command

Execution of preseeded command "wget -Y off http://puppetserver.example.com/unattended/finish?token=[long token here, no brackets] -O /target/tmp/finish.sh && in-target chmod +x /tmp/finish.sh && in-target /tmp/finish.sh" failed with exit code 8"

The foreman UI shows two entries for the host. Looking it up in the database by hostname, I see (with hostname lightly sanitized but nothing else changed):

foreman=# select * from hosts where name = 'my-testhost76';
-[ RECORD 1 ]--------+-------------------------------------
id                   | 77
name                 | my-testhost76
last_compile         | 
last_report          | 
updated_at           | 2023-10-12 22:01:44.200183
created_at           | 2023-10-12 22:01:44.200183
root_pass            | 
architecture_id      | 1
operatingsystem_id   | 3
ptable_id            | 140
medium_id            | 5
build                | t
comment              | 
disk                 | 
installed_at         | 
model_id             | 
hostgroup_id         | 1
owner_id             | 4
owner_type           | User
enabled              | t
puppet_ca_proxy_id   | 1
managed              | t
use_image            | 
image_file           | 
uuid                 | f0c48d44-0f41-4040-bb78-4eccc8972bde
compute_resource_id  | 28
puppet_proxy_id      | 1
certname             | 
image_id             | 
organization_id      | 1
location_id          | 14
type                 | Host::Managed
otp                  | 
realm_id             | 
compute_profile_id   | 5
provision_method     | build
grub_pass            | 
global_status        | 0
lookup_value_matcher | fqdn=my-testhost76.example.com
pxe_loader           | PXELinux BIOS
initiated_at         | 
build_errors         | 
creator_id           | 4
-[ RECORD 2 ]--------+-------------------------------------
id                   | 78
name                 | my-testhost76
last_compile         | 
last_report          | 2023-10-12 22:12:51
updated_at           | 2023-10-12 22:13:32.999312
created_at           | 2023-10-12 22:13:28.588598
root_pass            | 
architecture_id      | 1
operatingsystem_id   | 3
ptable_id            | 
medium_id            | 
build                | f
comment              | 
disk                 | 
installed_at         | 
model_id             | 1
hostgroup_id         | 
owner_id             | 
owner_type           | 
enabled              | t
puppet_ca_proxy_id   | 1
managed              | f
use_image            | 
image_file           | 
uuid                 | 
compute_resource_id  | 
puppet_proxy_id      | 1
certname             | my-testhost76
image_id             | 
organization_id      | 1
location_id          | 
type                 | Host::Managed
otp                  | 
realm_id             | 
compute_profile_id   | 
provision_method     | 
grub_pass            | 
global_status        | 2
lookup_value_matcher | fqdn=my-testhost76.example.com
pxe_loader           | 
initiated_at         | 
build_errors         | 
creator_id           | 3

Looking in the “users” table to compare the creator_id values 3 and 4, I see that I am user 4 and user id 3 belongs to “foreman_api_admin.”

And of course I can’t see any of these details in the UI because both entries show on the “all hosts” tab, but clicking either one goes to a page that does a lookup by hostname, so either link shows info about the same entry.

What causes this problem and how can I fix it?

mweilgart · October 12, 2023, 10:58pm

The only changes I made today that seem possibly relevant (but I think the last change was after this problem appeared) were:

I created a snippet called “Preseed default finish custom snippet” with the following contents:

# Just in case there is weird PTR record, ensure hostname is set correctly.
/bin/echo <%= @host.shortname %> > /etc/hostname
/bin/hostname  <%= @host.shortname %>

In Foreman UI, under Administer → Settings → Provisioning, I set the previously unset values for “Default PXE global template entry” and “Default PXE local template entry” both to “local.” (This didn’t accomplish my intention, which was accomplished by the next change listed below.)
In the UI under Hosts → Provisioning Templates, I clicked the “Build PXE Default” button at the top right to place a correct default pxelinux config on the tftp servers, so that hosts in my infrastructure that reboot from NIC are able to boot from local disk as they should. However as I mentioned, I’m pretty sure that this change was made after the problem with the duplicate entries and the failed builds first appeared.

I will try reverting change 2 listed above and see if that fixes things.

mweilgart · October 12, 2023, 11:24pm

Update: Reverting change 2 mentioned in last comment did NOT fix anything.

Also, I’m not actually sure what the “-Y” option is in the wget command; it’s not documented in the man page or --help output. But that’s a separate question; I’ve written it at What is the undocumented -Y option to wget? - Unix & Linux Stack Exchange

mweilgart · October 13, 2023, 12:14am

I confirmed that the token shown in the console matches what Foreman lists in the “provisioning” section of the host details, and I confirmed from another host that I can wget the correct finish script by passing that token.

wget documentation says exit status 8 is for if the server sent an error in response.

I am pretty sure by now that the exit status 8 isn’t from the first wget listed, but from a wget from INSIDE the finish script.

The last few lines of the finish script are:

if [ -x /usr/bin/curl ]; then
  /usr/bin/curl -o /dev/null --noproxy \* -H 'Content-Type: text/plain' --silent 'http://puppetserver.example.com/unattended/built?token=a17b1a8a-cdb5-4a03-b384-45d4fdc8c0a7'
elif [ -x /usr/bin/wget ]; then
  /usr/bin/wget -q -O /dev/null --no-proxy --method POST --header 'Content-Type: text/plain' 'http://puppetserver.example.com/unattended/built?token=a17b1a8a-cdb5-4a03-b384-45d4fdc8c0a7'
else
  wget -q -O /dev/null --header 'Content-Type: text/plain' 'http://puppetserver.example.com/unattended/built?token=a17b1a8a-cdb5-4a03-b384-45d4fdc8c0a7'
fi

Trying to wget the URL shown (from elsewhere) indeed gives an error:

$ wget 'http://puppetserver.example.com/unattended/built?token=a17b1a8a-cdb5-4a03-b384-45d4fdc8c0a7' --verbose
--2023-10-13 00:12:37--  http://puppetserver.example.com/unattended/built?token=a17b1a8a-cdb5-4a03-b384-45d4fdc8c0a7
Resolving puppetserver.example.com (puppetserver.example.com)... [redacted internal IP]
Connecting to puppetserver.example.com (puppetserver.example.com)|[redacted internal IP]|:80... connected.
HTTP request sent, awaiting response... 409 Conflict
2023-10-13 00:12:37 ERROR 409: Conflict.

Dirk · October 13, 2023, 7:16am

The second one is created by a report (last_report is set and some fields required by the UI are not), so perhaps some mismatch here. I had this in an environment where the fqdn was not set properly, so report came back with only the short name, so I would look here for a similar issue!

mweilgart · October 16, 2023, 3:36pm

I found I had made another change: under Administer → Settings → General, I had set “Append domain names to the host” to “No.” Setting it back to “Yes” solved the behavior described above.

It does seem like it should be possible to have new hosts not have FQDN names shown in the foreman UI, but still not double the host entries. But this (showing FQDN in foreman UI rather than short name) is acceptable for my installation so I’m moving on.