Foreman 2.4 / Katello 4.0 + Smart Proxy issue

Problem:
I have a working Foreman 2.4 + Katello 4.0 main server working. I’m trying to install another Foreman Smart Proxy. Used katello-certs-check + foreman-proxy-certs-generate for the Ssl certs.
But this option fails…
–foreman-proxy-register-in-foreman true

2021-05-19 18:10:45 [ERROR ] [configure] Error making POST request to Foreman at https://foremankatello.xxx/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([Net::HTTPServerException]: 403 “Forbidden”) for proxy https://foremansmartpxy01.yyy:9090/v2/features Please check the proxy is configured and running on the host.
2021-05-19 18:10:45 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foremansmartpxy01.yyy]/ensure: change from ‘absent’ to ‘present’ failed: Error making POST request to Foreman at https://foremankatello01.xxx/api/v2/smart_proxies: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([Net::HTTPServerException]: 403 “Forbidden”) for proxy https://foremansmartpxy01.yyy:9090/v2/features Please check the proxy is configured and running on the host.
2021-05-19 18:10:47 [NOTICE] [configure] System configuration has finished.

From the Smart Proxy
https://foremankatello01.xxx:9090/features
[“ansible”,“dhcp”,“dynflow”,“httpboot”,“logs”,“openscap”,“pulpcore”,“puppet”,“puppetca”,“registration”,“ssh”,“templates”,“tftp”]

https://foremankatello01.xxx:9090/v2/features
could not read client cert from environment

From the main Foreman/Katello server
https://foremansmartpxy01.yyy:9090/features
[“ansible”,“container_gateway”,“dhcp”,“discovery”,“dynflow”,“httpboot”,“logs”,“pulpcore”,“puppet”,“puppetca”,“registration”,“ssh”,“templates”,“tftp”]

https://foremansmartpxy01.yyy:9090/v2/features
could not read client cert from environment

I don’t understand why the /v2/ is causing problems…

Expected outcome:

Foreman and Proxy versions:

Foreman and Proxy plugin versions:
Foreman 2.4 + Katello 4.0

Distribution and version:
CentOS 7.9.2009 (fresh install)

Other relevant data:

On the Smart Proxy after re-running the installation with:
–foreman-proxy-register-in-foreman false
it completed ok and it’s running but I still can’t register the Smart Proxy in the main Foreman/Katello server because of the /v2/

systemctl status foreman-proxy
● foreman-proxy.service - Foreman Proxy
Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-05-19 18:04:11 UTC; 59min ago
Main PID: 9183 (ruby)
CGroup: /system.slice/foreman-proxy.service
└─9183 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize
May 19 18:35:36 foremansmartpxy01.yyy smart-proxy[9183]: 10.61.21.18 - - [19/May/2021:18:35:36 UTC] “GET /features HTTP/1.1” 200 149
May 19 18:35:36 foremansmartpxy01.yyy smart-proxy[9183]: - -> /features
May 19 18:45:07 foremansmartpxy01.yyy smart-proxy[9183]: 10.61.21.18 - - [19/May/2021:18:45:07 UTC] “GET /features HTTP/1.1” 200 149
May 19 18:45:07 foremansmartpxy01.yyy smart-proxy[9183]: - -> /features
May 19 18:45:26 foremansmartpxy01.yyy smart-proxy[9183]: 10.61.21.18 - - [19/May/2021:18:45:26 UTC] “GET /features HTTP/1.1” 200 149
May 19 18:45:26 foremansmartpxy01.yyy smart-proxy[9183]: - -> /features
May 19 19:00:02 foremansmartpxy01.yyy smart-proxy[9183]: 10.61.21.18 - - [19/May/2021:19:00:02 UTC] “GET /features HTTP/1.1” 200 149
May 19 19:00:02 foremansmartpxy01.yyy smart-proxy[9183]: - -> /features
May 19 19:00:55 foremansmartpxy01.yyy smart-proxy[9183]: 10.61.21.18 - - [19/May/2021:19:00:55 UTC] “GET /v2/features HTTP/1.1” 403 43
May 19 19:00:55 foremansmartpxy01.yyy smart-proxy[9183]: - -> /v2/features

Double check that on the smart-proxy the set of trusted hosts is accurate.

The smart proxy was set up using:

–foreman-proxy-trusted-hosts “foremankatello01.xxx”
–foreman-proxy-trusted-hosts “foremansmartpxy01.yyy”

Where can I check if this has been picked up correctly in the smart proxy?
Would I need to re-run the installed on the main foreman/katello server with similar params?

You’ll find it in /etc/foreman-proxy/settings.yml

Definitely in there

:trusted_hosts:

  • foremankatello01.xxx
  • foremansmartpxy01.yyy

Well well it’s fixed. Foreman/Katello server <-> Foreman Smart Proxy comms were going via a proxy… which messed up things. I’ve excluded the Smart Proxy host in the Administer -> General tab (HTTP(S) proxy except hosts) and my life went back to normal! :slight_smile:

To clarify, the v2 endpoint requires authentication while the other one allows anonymous access. You need to present the client certificates that Foreman uses.