Problem:
Getting a 500 Internal Server Error when trying to get the cloud-init config from the smart proxy which resides within the actual foreman server (I have a NAT’d network and I need some way to pass along the real IP of the build host). As a test, I run the following from a different host:
curl -D - -H 'X-Forwarded-For x.x.x.x' http://foreman.domain.com:8080/userdata/user-data
The results are the following:
Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8080"} from foreman.domain.com: Net::HTTPNotFound
Expected outcome:
Should be able to pull the data by running that curl command and via the build process as well
Foreman and Proxy versions:
3.4.1
Foreman and Proxy plugin versions:
Distribution and version:
Rocky Linux 8
Other relevant data:
The following related config is set:
/etc/foreman/settings.yaml:
:trusted_proxies:
- '127.0.0.1/8'
- '::1'
/etc/foreman-proxy/settings.yml:
:http_port: 8000
/etc/foreman-proxy/settings.d/templates.yml:
:enabled: true
:template_url: http://foreman.domain.com:8000
All of these values were set using foreman-installer
Just realize that the curl command had the wrong port. Should be this
curl -D - -H 'X-Forwarded-For x.x.x.x' http://foreman.domain.com:8000/userdata/user-data
Error is this
Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
OK some progress on this. I figured I’d add the actual IPs of the Foreman server itself as trusted proxy like so:
:trusted_proxies:
- '127.0.0.1/8'
- '::1'
- '10.xxx.xxx.xxx/32'
- '10.yyy.yyy.yyy/32'
Also realized that I had to modify my curl command slightly:
curl -D - -H "X-Forwarded-For: x.x.x.x" http://foreman.domain.com:8000/userdata/user-data
I’m now able to retrieve the userdata config from any host. The problem now is that the build still fails. Judging from the cloud-init logs on the host that failed to build, it’s only sending the User-Agent header.
So I’ve confirmed via a packet capture that the only header being sent during cloud-init provisioning is that of the User-Agent. Here are the relevant logs:
From /var/log/foreman-proxy/proxy.log
2023-01-27T08:40:31 905fe381 [I] Started GET /userdata/user-data
2023-01-27T08:40:31 905fe381 [E] Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
2023-01-27T08:40:31 905fe381 [W] Error details for Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound: <RuntimeError>: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
2023-01-27T08:40:31 905fe381 [W] Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound: <RuntimeError>: Error retrieving userdata/user-data for {"url"=>"http://foreman.domain.com:8000"} from foreman.domain.com: Net::HTTPNotFound
2023-01-27T08:40:31 905fe381 [I] Finished GET /userdata/user-data with 500 (48.12 ms)
from /var/log/foreman/production.log
2023-01-27T08:40:31 [I|app|f4801ae9] Started GET "/userdata/user-data?url=http%3A%2F%2Fforeman.domain.com%3A8000" for 10.XX.XX.XX at 2023-01-27 08:40:31 -0500
2023-01-27T08:40:31 [I|app|f4801ae9] Processing by UserdataController#userdata as TEXT
2023-01-27T08:40:31 [I|app|f4801ae9] Parameters: {"url"=>"http://foreman.domain.com:8000", "userdatum"=>{}}
2023-01-27T08:40:31 [E|app|f4801ae9] Unable to find user-data or cloud-init template for host foreman.domain.com running Rocky 8.7
2023-01-27T08:40:31 [I|app|f4801ae9] Rendered text template (Duration: 0.0ms | Allocations: 2)
2023-01-27T08:40:31 [I|app|f4801ae9] Completed 404 Not Found in 25ms (Views: 0.3ms | ActiveRecord: 8.3ms | Allocations: 8100)
Is there any config I should check for to ensure that the host sends the X-Forwarded-For header during cloud-init provisioning?
So after debugging this for a few days, I believe this isn’t going to be possible the way that I’m trying to get it set up. Can someone confirm that the smart proxy still needs to be able to receive the traffic from the IP of the host that is configured in Foreman and then the smart proxy is the one adding the X-Forwarded-For header back to the main Foreman server? If so, is there any other way around this that doesn’t involve adding additional smart proxies or is that my only solution?
I’m also running into this issue. I cloned the default “Preseed Autoinstall cloud-init user data” template and just added a new user stanza, but the userdata template isn’t being found for whatever reason. In the foreman-proxy logs I’m seeing this:
[E] Failed to retrieve user-data userdata template for {"kind"=>"user-data"}: Error retrieving userdata/user-data for {"url"=>"ht
tp://foreman.domain.com:8000"} from testnode02.domain.com: Net::HTTPNotFound
Unlike @lravelo though, even adding the proxy IP to the the trusted_proxies list doesn’t allow me to curl the data either.
Does anyone have any recommendations on things that can be checked or verified to get this to work?
@mgilbert one thing I was checking yesterday as I was looking into this topic again was to revise the cloud-init template config. I made sure it was associated with the applicable operating systems, locations and orgs. I also checked under Operating Systems and under the OS I was deploying I made sure that said cloud-init template was selected there as well.
1 Like
@lravelo, thanks for the response! I double-checked that the template was associated with the right OS, location, and org and that the OS had the right template selected. All that looked good.
I waited long enough and was given a cloud-init timeout message that allowed me to drop into a shell. While in there, I poked around and saw
dmesg:[ 326.260279] kernel: Out of memory: Killed process 1581 (cloud-init) total-vm:1842236kB, anon-rss:1821980kB, file-rss:2712kB, shmem-rss:0kB, UID:0 pgtables:3644kB oom_score_adj:0
Binary file journal/1e34e6646878455f86b2ef88624a8eff/system.journal matches
I googled around and found this post that seemed to run into the same problem. I tried adding cloud-config-url=/dev/null to the kernel line and for good measure I upped the VM I’m testing on to 6GB. I haven’t been able to confirm that either of these things fixed anything though because I’m running into an issue with my mirror server.
