Cloud-init userdata templates and foreman-proxy 3.4

Support
,
1

Hello,

I’m trying to install Ubuntu 22.04 on baremetal servers with subiquity/cloud-init and foreman 3.4.0, using foreman-proxy (required by our network architecture, the baremetal hosts are not able to contact the foreman server directly) but I’m having a problem when cloud-init tries to fetch the generated meta-data file.

I’m using the foreman_request_addr variable in the bootloader configuration template as the URL for cloud-init (ds='nocloud-net;s=http://<%= foreman_request_addr %>/userdata/') which points to the foreman-proxy responsible for the host (checked on the actual grub configuration on the proxy server).
The problem is that foreman (server) answers with a 404 because it’s not able to find the host.

I have the following logs (with redacted IP addresses and hostnames) on foreman:

2022-09-27T09:14:15 [I|app|9c5341ca] Started GET "/userdata/meta-data?url=http%3A%2F%2F192.0.2.1%3A8080" for 192.0.2.1 at 2022-09-27 09:14:15 +0000
2022-09-27T09:14:15 [I|app|9c5341ca] Processing by UserdataController#metadata as TEXT
2022-09-27T09:14:15 [I|app|9c5341ca]   Parameters: {"url"=>"http://192.0.2.1:8080", "userdatum"=>{}}
2022-09-27T09:14:15 [E|app|9c5341ca] Could not find host for request 192.0.2.1
2022-09-27T09:14:15 [I|app|9c5341ca]   Rendered text template (Duration: 0.1ms | Allocations: 2)
2022-09-27T09:14:15 [I|app|9c5341ca] Filter chain halted as :find_host rendered or redirected
2022-09-27T09:14:15 [I|app|9c5341ca] Completed 404 Not Found in 81ms (Views: 1.7ms | ActiveRecord: 27.6ms | Allocations: 5633)

On foreman-proxy:

2022-09-27T09:13:43 1912d831 [I] Started GET /userdata/meta-data
2022-09-27T09:14:15 1912d831 [E] Failed to retrieve meta-data userdata template for {"kind"=>"meta-data"}: Error retrieving userdata/meta-data for {"url"=>"http://192.0.2.1:8080"} from foreman.domain.tld: Net::HTTPNotFound
2022-09-27T09:14:15 1912d831 [W] Error details for Failed to retrieve meta-data userdata template for {"kind"=>"meta-data"}: Error retrieving userdata/meta-data for {"url"=>"http://192.0.2.1:8080"} from foreman.domain.tld: Net::HTTPNotFound: <RuntimeError>: Error retrieving userdata/meta-data for {"url"=>"http://192.0.2.1:8080"} from foreman.domain.tld: Net::HTTPNotFound

The host I’m trying to install is correctly defined in foreman, with an associated operating system that uses a userdata template.

After digging a bit it seems that foreman is using the IP address (in find_host here, ip: request.remote_ip) of the HTTP client that made the request as search criteria for the host that the meta-data file should be templated for, but in my case that IP address is the proxy’s address and not one that’s configured on the host that’s being installed.

Just to confirm this I edited this file on a test environment and used the X-Forwarded-For header which contains the host’s address (with ip: request.headers["X-Forwarded-For"]), with that change foreman finds the host, properly templates the files (both meta-data and user-data) and the installation on the host can proceed.

foreman-proxy supports proxying that sort of requests so I’m not sure if that’s a bug or if I’m missing something, has anyone had that problem? For information, foreman and foreman-proxy are running on Ubuntu 20.04.

2

I think what you’re running into is that Foreman doesn’t know that 192.0.2.1 is a Foreman Proxy and thus doesn’t “allow” it to inject the “Fordwarded-For” header.

Is your foreman proxy listed as a “trusted proxy” in /etc/foreman/settings.yml? You can modify the values of that using the --foreman-trusted-proxies parameter to the installer.

3

Thanks for you answer, the trusted_proxies part was indeed missing from settings.yml.
I’ve added the following at the end of the file (with the right IP address) and restarted the foreman service:

:trusted_proxies:
  - "192.0.2.1/32"

But that doesn’t help and I get the same 404 from foreman and 500 from foreman-proxy. I see no mention of trusted_proxies in the logs

4

Yesterday I did merge it mentioned for the modern documentation (only used for Katello today):

And now I also submitted the manual way for the old documentation:

Note that using the installer is preferred over manually changing settings.yml.

I wonder if perhaps you need to include localhost IPs too.

To make this easier to debug, I have started 2 PRs but probably won’t have time to finish it soon:

5

I poked our a test environment a bit more today and I turned out to have an intermittent issue with apache sitting in front of the foreman server that was messing with the X-Forward-For, that was fixed with ProxyAddHeaders Off. Sorry for the noise.
The actual problem was that the trusted proxies were not set properly, being able to easily debug the detected remote IP would definitely be helpful though.
Thank you both very much for your help!

1 Like