Foreman and Puppet-CA Failure: ERF50-5345 and ERF12-5356. Not able to view or approve certificates

Problem:
When trying to view certificates from Foreman, the following error message shows up in the webgui:

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy *hostname*:8443/puppet/ca)

and we are not able to view certificates or approve new requests.
Expected outcome:
I should be able to view existing certificates as well as approve certificate requests through Foreman
Foreman and Proxy versions:
Puppetserver: 6.7.2 , Foreman: 1.22.2 , Proxy: Maybe N/A (not sure how to find this) )
Foreman and Proxy plugin versions:
N/A (?)
Distribution and version:
Ubuntu 16.04.6 LTS
Other relevant data:

I have been troubleshooting this a bit. Im new to Linux admin, puppet and foreman, but also eager to learn of course. Below are the logs i found with relevant info, and also the auth.conf.

Puppet and foreman are running on the same server.

/var/log/foreman-proxy/proxy.log

2021-07-08T09:30:47 5d795038 [E] Failed to list certificates: Failed to query Puppet CA search v1 API: 403 Forbidden request: /puppet-ca/v1/certificate_statuses/foreman (method :get). Please see the server logs for details.

/var/log/foreman/production.log

2021-07-08T09:30:47 [I|app|f81877a8] Started GET "/smart_proxies/1-*hostname*/puppetca/1-*hostname*/expiry" for 1.2.3.4 at 2021-07-08 09:30:47 +0200
2021-07-08T09:30:47 [I|app|f81877a8] Processing by PuppetcaController#expiry as HTML
2021-07-08T09:30:47 [I|app|f81877a8]   Parameters: {"smart_proxy_id"=>"1-*hostname*", "id"=>"1-*hostname*"}
2021-07-08T09:30:47 [I|app|f81877a8] Current user set to me@user.com (admin)
2021-07-08T09:30:47 [W|app|f81877a8] Failed to expiry
 | Foreman::WrappedException: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://*hostname*:8443/puppet/ca)

/var/log/puppetlabs/puppetserver/puppetserver-access.log
(2.3.4.5 puppetmaster ip)

2.3.4.5 - - [08/Jul/2021:09:30:47 +0200] "GET /puppet-ca/v1/certificate_statuses/foreman HTTP/1.1" 403 116 "-" "Ruby" 7 - -
2.3.4.5 - - [08/Jul/2021:09:30:47 +0200] "GET /puppet-ca/v1/certificate_statuses/foreman HTTP/1.1" 403 116 "-" "Ruby" 2 - -
2.3.4.5 - - [08/Jul/2021:09:30:47 +0200] "GET /puppet-ca/v1/certificate_statuses/foreman HTTP/1.1" 403 116 "-" "Ruby" 2 - -

/var/log/puppetlabs/puppetserver/puppetserver.log

2021-07-08T09:30:47.120+02:00 ERROR [qtp600582592-143478] [p.t.a.rules] Forbidden request: 2.3.4.5 access to /puppet-ca/v1/certificate_statuses/foreman (method :get) (authenticated: false) denied by rule 'certificate_statuses'.
2021-07-08T09:30:47.282+02:00 ERROR [qtp600582592-139132] [p.t.a.rules] Forbidden request: 2.3.4.5 access to /puppet-ca/v1/certificate_statuses/foreman (method :get) (authenticated: false) denied by rule 'certificate_statuses'.
2021-07-08T09:30:47.558+02:00 ERROR [qtp600582592-143590] [p.t.a.rules] Forbidden request: 2.3.4.5 access to /puppet-ca/v1/certificate_statuses/foreman (method :get) (authenticated: false) denied by rule 'certificate_statuses'.

I also noticed the following in /etc/puppetlabs/puppetserver/conf.d/auth.conf

authorization: {
    version: 1
    rules: [
        {
            "allow": [
                "localhost",
		"2.3.4.5",
                "hostname"
            ],
            "match-request": {
                "method": [
                    "get",
                    "put",
                    "delete"
                ],
                "path": "/puppet-ca/v1/certificate_status/",
                "query-params": {},
                "type": "path"
            },
            "name": "certificate_status",
            "sort-order": 200

Reading the documentation for auth.conf i see that there is a possibility of using allow-unauthenticated , but im wondering why it isnt authenticated.
Can anyone give me a hint as to which direction to go, or what more to check to see?
I have not seen other problems with the installation. I dont know the history as the whole environment is a hand-me-down, but im learning a lot from this… :slight_smile:

Is this the puppet CA deployed by foreman-installer? Did you touch any certificates? See what certificates are used at /etc/foreman-proxy/settings.d/puppetca_http_api.yml, it seems your puppet CA does not trust this client for some reason.

Oh and your Foreman 1.22.2 is very very old. It would be good to upgrade first to newer version as there can be various problems fixed already. Current version is 2.5.

Hello,

I dont know how this instance was upgraded, this was before my time. As mentioned this is a hand-me-down environment so im still working on getting a good overview.
The certificate paths in /etc/foreman-proxy/settings.d/puppetca_http_api.yml are commented out for some reason.

I set the proper config and restarted foreman-proxy and it works. Without puppetca_http_api.yml being set i guess its no wonder it does not work.

Thanks for the nudge in the right direction :slight_smile: