Ubuntu 22.04 host not applying puppet classes and 'kills' hostname in foreman

Support
1

Problem:
Here’s a weird one for you. Doing cloud-init install for Ubuntu 22.04 and after proper build the puppet classes do not apply. Puppet runs fine but shows “Applied catalog in 0.02 seconds” and no classes are applied. Delete the host in Foreman (without removing the host) and the host now will not re-appear after puppet run. Whats even weirder is that after building this host then fully deleting everything and building a host from scratch (not using Foreman) with the same hostname now will not show up at all in foreman after puppet run. This step doesn’t matter which OS, tried Ubuntu 20.04 22.04 and CentOS 7. It’s like the hostname is now tainted or is locked up someplace in the database. It simply won’t appear any more.
Let me know what troubleshooting steps I should take of if clarification on anything is needed. I’ve already looked in the database in the hosts table, and the test host is not present after delete.

Expected outcome:
Classes to apply, host to appear after puppet run.

Foreman and Proxy versions:
3.7.0
Foreman and Proxy plugin versions:
[foreman-tasks] 8.1.1
[foreman_google] 1.0.4
[foreman_puppet] 6.0.0
[foreman_remote_execution] 10.0.7
[foreman_templates] 9.3.3

Distribution and version:
Foreman lives on AlmaLinux 8.8

Other relevant data:
Deploying these hosts to VMWare.
Pretty much standard Foreman install using default templates only modified to point at our expanded ISO for Ubuntu 22.04. Standard puppet run command “agent -t” with zero reports uploaded even though logs show that they are being uploaded:

2023-10-06T11:28:48 [I|app|24d8fde7] Imported report for sbrocktest25 in 37.5 ms, status refreshed in 26.0 ms
2

Just following up here. This behavior isn’t happening in our dev environment and on prod it still happens with 3.8 rc2.

3

I noted this morning that a build for 22.04 which never even went past boot-up also shows this behavior in Foreman.

4

Rooting around in the database brought me to an interesting find. the finish templates run puppet apparently and create a host named “ubuntu-server” which is the hostname in the installer. When this host exists, puppet won’t run on newly created hosts, no matter the OS. Deleting this host allows the newly created hosts to run puppet properly.
Since I caught it in the act, it looks like this is coming from the finish template calling out to the ‘puppet_setup’ snippet which does a puppet run. Going to test disabling the host parameter for doing a puppet run. Puppet running here and creating a host named “ubuntu-server” is a bug imho.