Cloud-init userdate templates not found on smart proxy

Support
,
#1

Problem:
Getting a 500 Internal Server Error when trying to get the cloud-init config from the smart proxy which resides within the actual foreman server (I have a NAT’d network and I need some way to pass along the real IP of the build host). As a test, I run the following from a different host:

curl -D - -H 'X-Forwarded-For x.x.x.x' http://foreman.domain.com:8080/userdata/user-data

The results are the following:

Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8080"} from foreman.domain.com: Net::HTTPNotFound

Expected outcome:
Should be able to pull the data by running that curl command and via the build process as well

Foreman and Proxy versions:
3.4.1
Foreman and Proxy plugin versions:

Distribution and version:
Rocky Linux 8

Other relevant data:

The following related config is set:

/etc/foreman/settings.yaml:

:trusted_proxies:
 - '127.0.0.1/8'
 - '::1'

/etc/foreman-proxy/settings.yml:

:http_port: 8000

/etc/foreman-proxy/settings.d/templates.yml:

:enabled: true
:template_url: http://foreman.domain.com:8000

All of these values were set using foreman-installer

#2

Just realize that the curl command had the wrong port. Should be this
curl -D - -H 'X-Forwarded-For x.x.x.x' http://foreman.domain.com:8000/userdata/user-data

Error is this

Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
#3

OK some progress on this. I figured I’d add the actual IPs of the Foreman server itself as trusted proxy like so:

:trusted_proxies:
 - '127.0.0.1/8'
 - '::1'
 - '10.xxx.xxx.xxx/32'
 - '10.yyy.yyy.yyy/32'

Also realized that I had to modify my curl command slightly:

curl -D - -H "X-Forwarded-For: x.x.x.x" http://foreman.domain.com:8000/userdata/user-data

I’m now able to retrieve the userdata config from any host. The problem now is that the build still fails. Judging from the cloud-init logs on the host that failed to build, it’s only sending the User-Agent header.

#4

So I’ve confirmed via a packet capture that the only header being sent during cloud-init provisioning is that of the User-Agent. Here are the relevant logs:

From /var/log/foreman-proxy/proxy.log

2023-01-27T08:40:31 905fe381 [I] Started GET /userdata/user-data
2023-01-27T08:40:31 905fe381 [E] Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
2023-01-27T08:40:31 905fe381 [W] Error details for Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound: <RuntimeError>: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
2023-01-27T08:40:31 905fe381 [W] Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound: <RuntimeError>: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
2023-01-27T08:40:31 905fe381 [I] Finished GET /userdata/user-data with 500 (48.12 ms)

from /var/log/foreman/production.log

2023-01-27T08:40:31 [I|app|f4801ae9] Started GET "/userdata/user-data?url=http%3A%2F%2Fforeman.domain.com%3A8000" for 10.XX.XX.XX at 2023-01-27 08:40:31 -0500
2023-01-27T08:40:31 [I|app|f4801ae9] Processing by UserdataController#userdata as TEXT
2023-01-27T08:40:31 [I|app|f4801ae9]   Parameters: {"url"=>"http://foreman.domain.com:8000", "userdatum"=>{}}
2023-01-27T08:40:31 [E|app|f4801ae9] Unable to find user-data or cloud-init template for host foreman.domain.com running Rocky 8.7
2023-01-27T08:40:31 [I|app|f4801ae9]   Rendered text template (Duration: 0.0ms | Allocations: 2)
2023-01-27T08:40:31 [I|app|f4801ae9] Completed 404 Not Found in 25ms (Views: 0.3ms | ActiveRecord: 8.3ms | Allocations: 8100)

Is there any config I should check for to ensure that the host sends the X-Forwarded-For header during cloud-init provisioning?

#5

So after debugging this for a few days, I believe this isn’t going to be possible the way that I’m trying to get it set up. Can someone confirm that the smart proxy still needs to be able to receive the traffic from the IP of the host that is configured in Foreman and then the smart proxy is the one adding the X-Forwarded-For header back to the main Foreman server? If so, is there any other way around this that doesn’t involve adding additional smart proxies or is that my only solution?