No smart proxy server found on ["foreman-puppet.example.com"] and is not in trusted_hosts

Problem:

Today, I have enabled the ansible plugin and proxy plugin, i.e. I ran

  # foreman-installer --enable-foreman-plugin-ansible \
  --enable-foreman-proxy-plugin-ansible

on my main server foreman.example.com which was successful. I can see the ansible menu items, tabs, etc.

However, I have noticed that no puppet reports arrived on the main server through my puppet smart proxy foreman-puppet.example.com anymore.

The access logs now show a 403 for the config report posts on the main server:

... "POST /api/config_reports HTTP/1.1" 403 166 "-" "Ruby"

production.log shows this:

2021-11-18T14:04:18 [I|app|f66a0bac] Started POST "/api/config_reports" for 10.10.25.2 at 2021-11-18 14:04:18 +0100
2021-11-18T14:04:18 [I|app|f66a0bac] Processing by Api::V2::ConfigReportsController#create as JSON
2021-11-18T14:04:18 [I|app|f66a0bac]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2021-11-18T14:04:18 [W|app|f66a0bac] No smart proxy server found on ["foreman-puppet.example.com"] and is not in trusted_hosts
2021-11-18T14:04:18 [I|app|f66a0bac]   Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout
2021-11-18T14:04:18 [I|app|f66a0bac]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 6.5ms | Allocations: 5866)
2021-11-18T14:04:18 [I|app|f66a0bac] Filter chain halted as #<Proc:0x000000000fc730f8 /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected
2021-11-18T14:04:18 [I|app|f66a0bac] Completed 403 Forbidden in 23ms (Views: 11.3ms | ActiveRecord: 2.7ms | Allocations: 15887)

This worked just fine before the installer run. All smart proxies show green “active” status. All features are shown as they should. As I did the installer run while everything was running, I have tried another run with services stopped:

# foreman-maintain service stop
# foreman-installer

And I did the same on the smart proxy foreman-puppet.example.com, too, in case it needed it re-register itself.

Eventually, I have found a hint in a topic to add the host name to TrustedHost, as the error suggests, i.e. Settings - Authentication - TrustedHost, setting it to the names of all my smart proxies in my network.

Now, reports are accepted again on the main server.

Of course, as I understand the description of the TrustedHost setting and as it was working fine before, I think by default foreman should automatically trust the names of all smart proxies registered to the main server.

If I clear our TrustedHosts, I see the 403s, so it’s reproducible. It seems I am not the first one with these kinds of problem, but beyond the manual workaround with TrustedHosts I haven’t found a real solution/fix.

Expected outcome:

All smart proxies should be trusted, even with the ansible plugins installed…

Foreman and Proxy versions:
katello-4.1.4-1.el7.noarch
foreman-2.5.4-1.el7.noarch
foreman-proxy-2.5.4-1.el7.noarch

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

O.K. I have went into the source code and found the reason, why it doesn’t accepts my foreman-puppet smart proxy host as trusted host:

The error is here: https://github.com/theforeman/foreman/blob/ccd67513052494e1d5ae79f4450cd04b8020633e/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb#L100

which checks allowed_hosts which consist of the setting trusted_hosts and the proxies parameter passed to auth_smart_proxy.

This is passed here: https://github.com/theforeman/foreman/blob/ccd67513052494e1d5ae79f4450cd04b8020633e/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb#L46 in allow_smart_proxies which is set right before that line.

And that’s where the code deviates from before: as far as I understand the features parameter of the function require_smart_proxy_or_login is set to ConfigReportImporter.authorized_smart_proxy_features (by this and this)

and that now contains:

irb(main):024:0> ConfigReportImporter.authorized_smart_proxy_features
=> ["Ansible"]

I am still running 2.5.4 that puppet is still part of the core. So back to require_smart_proxy_or_login it takes the features parameter, it’s not blank anymore and thus it selects all smart proxies with the “Ansible” feature enabled. Of course, as I have just started, only may main server foreman.example.com has the feature and not my two proxies foreman-content nor foreman-puppet.

Due to this, the puppet config report is not accepted because my foreman-puppet server doesn’t have the ansible feature and ConfigReportImporter.authorized_smart_proxy_features doesn’t list “Puppet” as authorized smart proxy feature.

But that’s where it looks strange to me, because according to https://github.com/theforeman/foreman/blob/ccd67513052494e1d5ae79f4450cd04b8020633e/app/services/config_report_importer.rb#L3

    @authorized_smart_proxy_features ||= super + ['Puppet']

I think it should always contain “Puppet”. But it doesn’t…