Problem:
Unable to use image-based provisioning in a somewhat complex network
Expected outcome:
To be able to use either UserData or SSH-based finish job with image-based provisioning in a complex network
Foreman and Proxy versions:
3.13
Foreman and Proxy plugin versions:
Katello 4.15
Distribution and version:
Oracle Linux 9
Other relevant data:
I’m trying to get VMware imaged-based provisioning to work in a scenario where there’s a load balancer and firewall between the smart proxies and the main server.
I started with UserData/cloud-init as the finish script mechanism, but because of the load balancer, every request looks to the main server like it’s coming from the load balancer’s IP, so I believe I’d have to add that IP to the trusted proxies list, which would mean that I’d effectively be trusting every IP going through that load balancer, and that seems like a non-starter.
I asked if the load balancer could strip the X-Forwarded-For header from requests not coming from our smart proxies, but apparently that’s not something it’s capable of.
So I (temporarily) added the load balancer IP to the list of trusted proxies, and it almost works? I’ve got seedfrom set to https://smart-proxy.example.com:9090/userdata/, and on boot new hosts register with katello via subscription-manager, but phoning home fails with:
2025-04-23T15:14:37 51858740 [E] Failed to proxy /built for {"token"=>"aaaaaaa-2222-4444-8888-fffffffffffffff", "kind"=>"built"}: Error retrieving unattended/built for {"token"=>"aaaaaaa-2222-4444-8888-fffffffffffffff", "url"=>"http://smart-proxy.example.com:8000"} from foreman.example.com: Net::HTTPNotFound: 404
2025-04-23T15:14:37 51858740 [W] Error details for Failed to proxy /built for {"token"=>"aaaaaaa-2222-4444-8888-fffffffffffffff", "kind"=>"built"}: Error retrieving unattended/built for {"token"=>"aaaaaaa-2222-4444-8888-fffffffffffffff", "url"=>"http://smart-proxy.example.com:8000"} from foreman.example.com: Net::HTTPNotFound: 404 : <Proxy::Error::HttpError>: Error retrieving unattended/built for {"token"=>"aaaaaaa-2222-4444-8888-fffffffffffffff", "url"=>"http://smart-proxy.example.com:8000"} from foreman.example.com: Net::HTTPNotFound: 404
ten times in the logs. Oddly enough, if I go and look at the host in Foreman, the provisioning box is empty now instead of having the token in it, and the webhook I have configured to fire on Build Exited fired. So maybe that 404 is a red herring? I don’t know.
But even if this is expected behavior, aren’t I risking leaking sensitive data and/or adverse actions if I leave the LB in the list of trusted proxies?
Anyway, I also tried pivoting to using SSH in lieu of UserData, but that fails because the main server initiates that finish script despite having remote execution installed and working on the smart proxy that can directly SSH to that host, and from what I understand that’s just a limitation of the software at this point?
There’s really no practical way to get the main server to be able to directly SSH to those hosts that I can think of. They’re in very separate networks with only the required ports open, which works fine for everything else I’ve needed to do.
So at this point I’m trying to think of some out of the box options to complete registration and mark the system as built. Maybe a shellhook on the smart proxy that’s subscribed to the Build Entered event that waits for the host to come up, then schedules a remote execution task on it that’s effectively a derivative of the kickstart finish script?
Any other ideas or things I’m overlooking?