We are running Foreman 3.17.2 / Katello 4.19.2 with three Smart Proxies sitting behind a load balancer.
Our hosts register via the load balancer URL, and both /etc/rhsm/rhsm.conf and /etc/yum.repos.d/redhat.repo correctly reference the load balancer on the client side.
In the Foreman UI, everything appears correct at first glance:
The Host Details page shows the host is registered to the load balancer.
It also states that “Content may come from <smart_proxy_hostname> or any other Smart Proxy behind the load balancer.”
However, when looking at the Smart Proxy Details page (Infrastructure → Smart Proxies → <smart_proxy_hostname>), it shows that all hosts are managed by the exact same Smart Proxy instead of being distributed. For example, one Smart Proxy manages 243 hosts, while the other Smart Proxies only show 1 host managed.
Is this expected behaviour, or am I missing a configuration step to reflect the balancing in the UI?
Just to check, you followed the load balancing configuration documentation from docs.theforeman.org?
Generally with load balancing the registration records aren’t indicative of the actual load being distributed. I can’t remember though exactly which smart proxy record retains the host registration information, @jeremylenz may know. One way to check if your load balancing setup is really working would be to disconnect each capsule and see if Pulp content is still generally available to your hosts.
Yes, I followed the “Configuring Smart Proxies with a load balancer” documentation. The only difference is that we are using a DNS-based load balancer (GSLB) instead of a TCP load balancer.
As a test, I shut down the specific Smart Proxy that was actively delivering Pulp content to a test client (which also happens to be the proxy the GUI lists as the “owner” of all clients). The client didn’t notice any disruption! dnf command continued working perfectly because the GSLB immediately rerouted the DNS lookup to the remaining active Smart Proxy. Remote Execution also worked fine, as Foreman balanced the jobs across the available proxies.
So it appears everything is working fine and that registration ownership by a specific Smart Proxy is mostly a cosmetic artifact in the web UI.
And another observation that seems to confirm this: enabling the “Prefer registered through Smart Proxy for remote execution” setting has no effect in this setup. Foreman continues to distribute jobs evenly across all healthy Smart Proxies rather than attempting to prefer Smart Proxy the client is registered to.
It might be worth adding a brief note to the official documentation to clarify this.
This is expected behavior. The way load-balanced smart proxies work is that you set the registration_url and templates_url on each underlying smart proxy to the url of the load balancer. But the problem is that when a host is registered through the LB, we literally don’t know which smart proxy is going to serve the content - it could be any of them, and it could change for each request.
However, we have to have something in the host’s “content source” record. (This is our field that answers the question, “Where does this host get its content from?”) Rather than leaving that field blank, we just do a search for any smart proxies with that registration_url and return the first matching one.
It’s not ideal, but it works. There has been talk of improving the load balancing code in Foreman so that it has a more proper record of the load balancer itself (right now it knows nothing about it.) If that work were to move forward, this UX issue could get better.
