No smart proxy server found on ["foreman-puppet.example.com"] and is not in trusted_hosts

Support
1

Problem:

Today, I have enabled the ansible plugin and proxy plugin, i.e. I ran

  # foreman-installer --enable-foreman-plugin-ansible \
  --enable-foreman-proxy-plugin-ansible

on my main server foreman.example.com which was successful. I can see the ansible menu items, tabs, etc.

However, I have noticed that no puppet reports arrived on the main server through my puppet smart proxy foreman-puppet.example.com anymore.

The access logs now show a 403 for the config report posts on the main server:

... "POST /api/config_reports HTTP/1.1" 403 166 "-" "Ruby"

production.log shows this:

2021-11-18T14:04:18 [I|app|f66a0bac] Started POST "/api/config_reports" for 10.10.25.2 at 2021-11-18 14:04:18 +0100
2021-11-18T14:04:18 [I|app|f66a0bac] Processing by Api::V2::ConfigReportsController#create as JSON
2021-11-18T14:04:18 [I|app|f66a0bac]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2021-11-18T14:04:18 [W|app|f66a0bac] No smart proxy server found on ["foreman-puppet.example.com"] and is not in trusted_hosts
2021-11-18T14:04:18 [I|app|f66a0bac]   Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout
2021-11-18T14:04:18 [I|app|f66a0bac]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 6.5ms | Allocations: 5866)
2021-11-18T14:04:18 [I|app|f66a0bac] Filter chain halted as #<Proc:0x000000000fc730f8 /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected
2021-11-18T14:04:18 [I|app|f66a0bac] Completed 403 Forbidden in 23ms (Views: 11.3ms | ActiveRecord: 2.7ms | Allocations: 15887)

This worked just fine before the installer run. All smart proxies show green “active” status. All features are shown as they should. As I did the installer run while everything was running, I have tried another run with services stopped:

# foreman-maintain service stop
# foreman-installer

And I did the same on the smart proxy foreman-puppet.example.com, too, in case it needed it re-register itself.

Eventually, I have found a hint in a topic to add the host name to TrustedHost, as the error suggests, i.e. Settings - Authentication - TrustedHost, setting it to the names of all my smart proxies in my network.

Now, reports are accepted again on the main server.

Of course, as I understand the description of the TrustedHost setting and as it was working fine before, I think by default foreman should automatically trust the names of all smart proxies registered to the main server.

If I clear our TrustedHosts, I see the 403s, so it’s reproducible. It seems I am not the first one with these kinds of problem, but beyond the manual workaround with TrustedHosts I haven’t found a real solution/fix.

Expected outcome:

All smart proxies should be trusted, even with the ansible plugins installed…

Foreman and Proxy versions:
katello-4.1.4-1.el7.noarch
foreman-2.5.4-1.el7.noarch
foreman-proxy-2.5.4-1.el7.noarch

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

2

O.K. I have went into the source code and found the reason, why it doesn’t accepts my foreman-puppet smart proxy host as trusted host:

The error is here: https://github.com/theforeman/foreman/blob/ccd67513052494e1d5ae79f4450cd04b8020633e/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb#L100

which checks allowed_hosts which consist of the setting trusted_hosts and the proxies parameter passed to auth_smart_proxy.

This is passed here: https://github.com/theforeman/foreman/blob/ccd67513052494e1d5ae79f4450cd04b8020633e/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb#L46 in allow_smart_proxies which is set right before that line.

And that’s where the code deviates from before: as far as I understand the features parameter of the function require_smart_proxy_or_login is set to ConfigReportImporter.authorized_smart_proxy_features (by this and this)

and that now contains:

irb(main):024:0> ConfigReportImporter.authorized_smart_proxy_features
=> ["Ansible"]

I am still running 2.5.4 that puppet is still part of the core. So back to require_smart_proxy_or_login it takes the features parameter, it’s not blank anymore and thus it selects all smart proxies with the “Ansible” feature enabled. Of course, as I have just started, only may main server foreman.example.com has the feature and not my two proxies foreman-content nor foreman-puppet.

Due to this, the puppet config report is not accepted because my foreman-puppet server doesn’t have the ansible feature and ConfigReportImporter.authorized_smart_proxy_features doesn’t list “Puppet” as authorized smart proxy feature.

But that’s where it looks strange to me, because according to https://github.com/theforeman/foreman/blob/ccd67513052494e1d5ae79f4450cd04b8020633e/app/services/config_report_importer.rb#L3

    @authorized_smart_proxy_features ||= super + ['Puppet']

I think it should always contain “Puppet”. But it doesn’t…

3

I think this is a leftover from a patch that shipped in Foreman 2.1, while the hosts endpoint is authorized correctly, config reports API endpoint is not. This should fix it, please confirm:

https://github.com/theforeman/foreman/pull/8958

What I don’t understand is why we get this reported now, after all those months/years? I would expect this to blow up much quicker. I do not understand tho why puppet would use this endpoint instead of the /hosts/xyz/facts?

4

That patch doesn’t work for me. I have cleared out the trusted_hosts settings and after foreman-maintain service stop and start, I get this:

2021-11-29T14:05:11 [I|app|524dc2a9] Started POST "/api/config_reports" for 10.1.2.3 at 2021-11-29 14:05:11 +0100
2021-11-29T14:05:11 [I|app|524dc2a9] Processing by Api::V2::ConfigReportsController#create as JSON
2021-11-29T14:05:11 [I|app|524dc2a9]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2021-11-29T14:05:11 [I|app|524dc2a9]   Rendering api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout
2021-11-29T14:05:11 [I|app|524dc2a9]   Rendered api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout (Duration: 7.7ms | Allocations: 5804)
2021-11-29T14:05:11 [I|app|524dc2a9] Filter chain halted as :authorize rendered or redirected
2021-11-29T14:05:11 [I|app|524dc2a9] Completed 401 Unauthorized in 17ms (Views: 12.9ms | ActiveRecord: 0.5ms | Allocations: 12470)

If I add the trusted_hosts again, it still doesn’t work:

2021-11-29T14:10:25 [I|app|826d8296] Started POST "/api/config_reports" for 10.1.2.3 at 2021-11-29 14:10:25 +0100
2021-11-29T14:10:25 [I|app|826d8296] Processing by Api::V2::ConfigReportsController#create as JSON
2021-11-29T14:10:25 [I|app|826d8296]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2021-11-29T14:10:25 [W|app|826d8296] Brute-force attempt blocked from IP: 10.1.2.3
2021-11-29T14:10:25 [I|app|826d8296]   Rendering api/v2/errors/bruteforce_attempt.json.rabl within api/v2/layouts/error_layout
2021-11-29T14:10:25 [I|app|826d8296]   Rendered api/v2/errors/bruteforce_attempt.json.rabl within api/v2/layouts/error_layout (Duration: 7.4ms | Allocations: 5827)
2021-11-29T14:10:25 [I|app|826d8296] Filter chain halted as :authorize rendered or redirected
2021-11-29T14:10:25 [I|app|826d8296] Completed 401 Unauthorized in 15ms (Views: 12.2ms | ActiveRecord: 0.5ms | Allocations: 12541)

I reverted the patch and restarted foreman services and it’s working again (with trusted_hosts configured).

5

Foreman should match the CN against smart proxy URL hostname. Can you double check that?

I really don’t know what is wrong at this moment. Can you debug this providing some log statements on key places perhaps?

6

Oh this looks suspicious, it is the same IP address as the one that is trying to upload config report. Could this be the culprit? FYI this is the brute-force password protection system. It is enabled by default for both UI and API since API also supports passwords.

It works the following way: everytime a password is invalid (or even empty) an entry in Rails cache is created named failed_login_REMOTEIP. When it exceeds the treshold, request is ended immediately with 401. Cache expires in 5 minutes tho.

7

It looks like your client doesn’t have a valid SSL certificate, therefore Foreman fallsback to username/password. Not sure.

8

That matches. It’s foreman-puppet.example.com in both cases.

I don’t really know where to look…

That’s the IP address of the smart proxy running the puppet master, i.e. forwarding the config reports.

I don’t understand how that should happen. It’s the smart proxy, not the clients. Clients report nicely to the smart proxy foreman-puppet. Shouldn’t it also fail after I add the hostname to trusted_hosts, because it still needs to verify the hostname?

And remember, it all worked fine before. The only change was to enable the ansible plugin.

Maybe I should disable it again and see if it’s back to normal then?

9

It does not matter who this is. Foreman API require authentication OR valid X509 certificate for some endpoints (this one does). If it, for some reason, is not considered valid, Foreman can likely fallback to username/password authentication and since there is nothing in the request this likely is seen as empty strings or nil values which are considered as incorrect passwords, counter of failed logins increases for this IP until it is banned for 5 minutes.

Please test this and if you confirm, we would ask @aruzicka for an insight.

10

O.K. Dumb question: how do I disable the plugins, i.e. undo the enable before? I have tried

# foreman-installer --no-enable-foreman-plugin-ansible \
  --no-enable-foreman-proxy-plugin-ansible

but that doesn’t undo what was set up before. It remains enabled.

11

Hmmm not sure how to do it via installer, however, edit /etc/foreman-proxy/settings.d/ansible.yaml and disable it there, then restart foreman-proxy. This should do it as a temporary thing, note that the installer will overwrite this.

12

O.K. I have disabled the ansible feature in the smart proxy, restarted the proxy, refresh features and eventually even restarted all foreman services. It still doesn’t make a difference: it only works as long as the foreman-puppet hostname is mentioned in the trustedhost settings.

irb(main):013:0> ConfigReportImporter.authorized_smart_proxy_features
=> ["Ansible"]

It’s still not the same as before I have enabled the ansible plugin in foreman…

13

As far as I know enabling (or not) the ansible plugin should not really touch which reports are accepted and which are rejected. It feels like we’re getting sidetracked a bit by focuing on ansible. Isn’t it possible that the installer stomped over some other configuration?

14

Please try my patch again, I had an error there (copy and paste typo). That is why it did not make any difference.

Remember to restart the process and stuff like that :slight_smile:

1 Like
15

Applied the patch and restarted all foreman services. Works for me now. Config reports coming in via my puppet only smart proxy.

1 Like
16

Unfortunately, the patch is going to be more complicated. See the PR.

17

I have noticed. But at this time, it doesn’t bother me as long as it works, even if it’s technically still “broken” at the moment.