Problem:
I just upgraded a Foreman system from 3.0 → 3.1 → 3.2. After concluding the upgrades, I noticed that iPXE boot was no longer working. I also found that the Foreman server seemed to be registered with itself.
Is this intended behaviour (the self-registering)? AFAIK it should not be and it’s even not supported (also not in Satellite). However, it looks like foreman-installer is doing that for us.
The effect seems to be that, because the system (in my case at least) isn’t properly configured it breaks Foreman proxy from rendering the PXE templates correctly, even for other hosts!
The workarounds I have tried so far:
‘Adopting’ the system (open host → Manage Host), configuring it ‘like the rest’. Fixed some errors in the logs (can’t copy them here, they’re 3 layers of remote desktop deep, but it was complaining about the kickstart_repository not set)
Deleting the host, fixed the issue instantly
Expected outcome:
Not breaking Foreman and Proxy versions:
Foreman 3.2.1 / Katello 4.4.1 Foreman and Proxy plugin versions:
There is an entity (a host) inside Foreman with its own name, yes. It’s used for executing tasks against itself (via Remote Execution). IIRC this was added in 3.1.
Satellite doesn’t support consuming content from itself, which is a completed different thing.
Ah, right, that makes sense, I thought the only means for a system to get in the ‘Hosts’ list was by registering. But with REX it’s logical that it’s there…
But anyway, it seems to break PXE booting, I’ll try to reproduce the issue on a different lab machine and see if I can extract some logs.
Yes, indeed. I have been able to reproduce this in my lab with Foreman 3.2/Katello 4.3.
So when the Foreman server itself (it’s named deploy.lbhr.htm.lan in this log) is present in the hosts list, no systems can boot either FDI or the installer. When I look in /var/log/foreman/production.log I see the following entries:
2022-06-27T10:32:11 [I|app|7942b995] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fdeploy.lbhr.htm.lan%3A8000" for 192.168.255.15 at 2022-06-27 10:32:11 +0200
2022-06-27T10:32:11 [I|app|7942b995] Processing by UnattendedController#host_template as TEXT
2022-06-27T10:32:11 [I|app|7942b995] Parameters: {"bootstrap"=>"1", "url"=>"http://deploy.lbhr.htm.lan:8000", "kind"=>"iPXE", "unattended"=>{}}
2022-06-27T10:32:11 [W|app|7942b995] Could not find a provider for deploy.lbhr.htm.lan. Providers returned {"Katello::ManagedContentMediumProvider"=>["Kickstart repository was not set for host 'deploy.lbhr.htm.lan'", "Content source was not set for host 'deploy.lbhr.htm.lan'"], "MediumProviders::Default"=>["Rocky 8.5 medium was not set for host 'deploy.lbhr.htm.lan'", "Invalid medium '' for 'Rocky 8.5'"]}
2022-06-27T10:32:11 [I|app|7942b995] Rendering text template
2022-06-27T10:32:11 [I|app|7942b995] Rendered text template (Duration: 0.0ms | Allocations: 2)
2022-06-27T10:32:11 [I|app|7942b995] Completed 200 OK in 63ms (Views: 1.0ms | ActiveRecord: 10.0ms | Allocations: 1617
Which will then result in rendering the template below:
[root@deploy ~]# curl http://deploy.lbhr.htm.lan:8000/unattended/iPXE?bootstrap=1
#!ipxe
# Skips booting from network and continues booting from next device
exit 1
Some background about the components it names in the logs:
The operating system Rocky 8.5 is not one I create, it is autocreated by foreman-installer (I think when it ‘registers’ the server itself)
It is therefore completely empty, I only configure a Rocky 8 OS, which I do with Ansible and that holds all the settings it needs.
When I ‘adopt’ the Foreman server and change the OS, the errors in the log go away, but the issue is not resolved. Unknown hosts still receive the ‘boot your local drive’ template.
2022-06-27T10:40:38 [I|app|23f87faa] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fdeploy.lbhr.htm.lan%3A8000" for 192.168.255.15 at 2022-06-27 10:40:38 +0200
2022-06-27T10:40:38 [I|app|23f87faa] Processing by UnattendedController#host_template as TEXT
2022-06-27T10:40:38 [I|app|23f87faa] Parameters: {"bootstrap"=>"1", "url"=>"http://deploy.lbhr.htm.lan:8000", "kind"=>"iPXE", "unattended"=>{}}
2022-06-27T10:40:38 [I|app|23f87faa] Rendering text template
2022-06-27T10:40:38 [I|app|23f87faa] Rendered text template (Duration: 0.0ms | Allocations: 1)
2022-06-27T10:40:38 [I|app|23f87faa] Completed 200 OK in 41ms (Views: 0.5ms | ActiveRecord: 8.5ms | Allocations: 20667)
Only when I delete the host, PXE discovery and installations resume.
What is odd, is that it seems that unknown hosts seem to use the IP address of the foreman server. In both these logs I booted a secondary VM, which has 192.168.255.173 as it’s IP, while the Foreman server has 192.168.255.15.
The controller tries to look up a host based on a number of things, one of those things is a source IP address from the request
If the foreman host exists, it gets matched by that IP
The controller tries to render a template for that host and fails (either due to “blank” os or just assumes the host is already provisioned and renders template for booting from a local disk)
Now why foreman thinks it is processing a request from itself, I cannot tell from the top of my head
@aruzicka Anything we can do to fix this in a more permanent way? The hack I placed in /etc/foreman/setings.yaml works, but each invocation of foreman-installer or other sanity checks will break it
You should be able to pass --foreman-trusted-proxies $TRUSTED_PROXY_LIST to the installer so that installer knows about it and does not stomp all over your changes.
