Foreman server seems to self-register (possibly after upgrade?)

Support
#1

Problem:
I just upgraded a Foreman system from 3.0 → 3.1 → 3.2. After concluding the upgrades, I noticed that iPXE boot was no longer working. I also found that the Foreman server seemed to be registered with itself.

Is this intended behaviour (the self-registering)? AFAIK it should not be and it’s even not supported (also not in Satellite). However, it looks like foreman-installer is doing that for us.

The effect seems to be that, because the system (in my case at least) isn’t properly configured it breaks Foreman proxy from rendering the PXE templates correctly, even for other hosts!

The workarounds I have tried so far:

  • ‘Adopting’ the system (open host → Manage Host), configuring it ‘like the rest’. Fixed some errors in the logs (can’t copy them here, they’re 3 layers of remote desktop deep, but it was complaining about the kickstart_repository not set)
  • Deleting the host, fixed the issue instantly

Expected outcome:
Not breaking :wink:
Foreman and Proxy versions:
Foreman 3.2.1 / Katello 4.4.1
Foreman and Proxy plugin versions:

Distribution and version:
Rocky 8.6

#2

@mcorr Do you know anyone who knows more about this? :slight_smile:

#3

Define “registering”?

There is an entity (a host) inside Foreman with its own name, yes. It’s used for executing tasks against itself (via Remote Execution). IIRC this was added in 3.1.

Satellite doesn’t support consuming content from itself, which is a completed different thing.

#4

Ah, right, that makes sense, I thought the only means for a system to get in the ‘Hosts’ list was by registering. But with REX it’s logical that it’s there… :slight_smile:

But anyway, it seems to break PXE booting, I’ll try to reproduce the issue on a different lab machine and see if I can extract some logs.

#5

It really should not have an influence on PXE.

CC @aruzicka who originally designed this part of “registration” :slight_smile:

#6

Let me recap. After an upgrade:

  1. You have a host representing foreman show up in host list
  2. The presence of this host breaks provisioning of other hosts
  3. Removing the foreman host fixes provisioning of other hosts

That sounds odd. Can’t say I’ve seen that in the wild

#7

Yes, indeed. I have been able to reproduce this in my lab with Foreman 3.2/Katello 4.3.

So when the Foreman server itself (it’s named deploy.lbhr.htm.lan in this log) is present in the hosts list, no systems can boot either FDI or the installer. When I look in /var/log/foreman/production.log I see the following entries:

2022-06-27T10:32:11 [I|app|7942b995] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fdeploy.lbhr.htm.lan%3A8000" for 192.168.255.15 at 2022-06-27 10:32:11 +0200
2022-06-27T10:32:11 [I|app|7942b995] Processing by UnattendedController#host_template as TEXT
2022-06-27T10:32:11 [I|app|7942b995]   Parameters: {"bootstrap"=>"1", "url"=>"http://deploy.lbhr.htm.lan:8000", "kind"=>"iPXE", "unattended"=>{}}
2022-06-27T10:32:11 [W|app|7942b995] Could not find a provider for deploy.lbhr.htm.lan. Providers returned {"Katello::ManagedContentMediumProvider"=>["Kickstart repository was not set for host 'deploy.lbhr.htm.lan'", "Content source was not set for host 'deploy.lbhr.htm.lan'"], "MediumProviders::Default"=>["Rocky 8.5 medium was not set for host 'deploy.lbhr.htm.lan'", "Invalid medium '' for 'Rocky 8.5'"]}
2022-06-27T10:32:11 [I|app|7942b995]   Rendering text template
2022-06-27T10:32:11 [I|app|7942b995]   Rendered text template (Duration: 0.0ms | Allocations: 2)
2022-06-27T10:32:11 [I|app|7942b995] Completed 200 OK in 63ms (Views: 1.0ms | ActiveRecord: 10.0ms | Allocations: 1617

Which will then result in rendering the template below:

[root@deploy ~]# curl http://deploy.lbhr.htm.lan:8000/unattended/iPXE?bootstrap=1
#!ipxe

# Skips booting from network and continues booting from next device
exit 1

Some background about the components it names in the logs:

  • The operating system Rocky 8.5 is not one I create, it is autocreated by foreman-installer (I think when it ‘registers’ the server itself)
  • It is therefore completely empty, I only configure a Rocky 8 OS, which I do with Ansible and that holds all the settings it needs.
  • When I ‘adopt’ the Foreman server and change the OS, the errors in the log go away, but the issue is not resolved. Unknown hosts still receive the ‘boot your local drive’ template.
2022-06-27T10:40:38 [I|app|23f87faa] Started GET "/unattended/iPXE?bootstrap=1&url=http%3A%2F%2Fdeploy.lbhr.htm.lan%3A8000" for 192.168.255.15 at 2022-06-27 10:40:38 +0200
2022-06-27T10:40:38 [I|app|23f87faa] Processing by UnattendedController#host_template as TEXT
2022-06-27T10:40:38 [I|app|23f87faa]   Parameters: {"bootstrap"=>"1", "url"=>"http://deploy.lbhr.htm.lan:8000", "kind"=>"iPXE", "unattended"=>{}}
2022-06-27T10:40:38 [I|app|23f87faa]   Rendering text template
2022-06-27T10:40:38 [I|app|23f87faa]   Rendered text template (Duration: 0.0ms | Allocations: 1)
2022-06-27T10:40:38 [I|app|23f87faa] Completed 200 OK in 41ms (Views: 0.5ms | ActiveRecord: 8.5ms | Allocations: 20667)

Only when I delete the host, PXE discovery and installations resume.

What is odd, is that it seems that unknown hosts seem to use the IP address of the foreman server. In both these logs I booted a secondary VM, which has 192.168.255.173 as it’s IP, while the Foreman server has 192.168.255.15.

#8

Wild theory time:

  1. A request comes in and hits /unattended/iPXE
  2. The controller tries to look up a host based on a number of things, one of those things is a source IP address from the request
  3. If the foreman host exists, it gets matched by that IP
  4. The controller tries to render a template for that host and fails (either due to “blank” os or just assumes the host is already provisioned and renders template for booting from a local disk)

Now why foreman thinks it is processing a request from itself, I cannot tell from the top of my head

#9

Hmm, that reminds me of an issue I forgot about :sweat_smile:

I’ve tried to set the :trusted_proxies (same lab, so same values) and it works! Even without ‘fixing’ the host object for the Foreman server