Updating certificates on content proxy

Problem:

I am using externally signed certificates on my foreman server and proxies. I have already updated the certs on my proxies before and using my instructions I have wrote down the last time, I have tried to do it again, but it doesn’t replace the certificate on the proxy port 9090 of my proxy server.

I have received the new certificate (which also requires a new ca-bundle, if that matters) and checked it on the main foreman server:

[root@foreman certs]# katello-certs-check -t foreman-proxy -c foreman-puppet.cer -k foreman-puppet.key -b ca-bundle.crt

then ran the command as suggested:

[root@foreman certs]# foreman-proxy-certs-generate --foreman-proxy-fqdn "foreman-puppet.example.com" \
> --certs-tar  "~/foreman-puppet.example.com-certs.tar" \
>                                    --server-cert "/root/certs/foreman-puppet.cer" \
>                                    --server-key "/root/certs/foreman-puppet.key" \
>                                    --server-ca-cert "/root/certs/ca-bundle.crt" \
>                                    --certs-update-server

Then copied /root/foreman-puppet.example.com-certs.tar to the proxy server “foreman-puppet”. On the proxy server I run

[root@foreman-puppet ~]# foreman-maintain service stop
[root@foreman-puppet ~]# foreman-installer --certs-tar-file /root/foreman-puppet.dkrz.de-certs.tar --certs-update-all

That runs without errors. However, it doesn’t seem to replace the certs anymore as it did before (with some older katello version a year ago). In particular, on port 9090 it still shows the old certificate.

A year ago the same procedure worked…

I have checked the content of the tar and the content of /root/ssl-build on the proxy server. It contains the new certificate and chain. The configured file in the foreman-proxy settings is /etc/foreman-proxy/ssl_cert.pem and it contains the old certificate.

What do I have to do to get the foreman-installer on the proxy server update the certificates?

Expected outcome:
foreman-installer reconfiguring the proxy for the new cert and chain.

Foreman and Proxy versions:
Foreman 3.1.2, Katello 4.3.1

Distribution and version:
CentOS 7.9

Does anyone has an idea why foreman-installer doesn’t update the certificates? It’s expiring and soon I’ll have to replace it manually wherever I can find it in the system…

One week left… @katello Does anyone have any insight why the foreman-installer doesn’t update the certificates?

This does look correct to me, procedure wise.

Is this a foreman-only proxy (aka: without content) or a katello proxy (aka with content)?

Well, kind of both: it’s an installation of a content proxy (i.e. scenario foreman-proxy-content) with all content functions disabled and only puppet enabled, i.e. only enabled features are logs and puppet.

Interesting. Would you mind sharing /var/log/foreman-installer/foreman-proxy-content.log after the foreman-installer --certs-tar-file … run?

It might contain sensitive information, so maybe better mail it to evgeni at redhat dot com :wink:

Just sent an email.

So reading your installer log, it seems that nothing has loaded and executed the foreman_proxy_content Puppet module, but that’s the one that’s responsible for deploying the certs in a proxy content scenario.

Well, yes. I don’t want to provide any content from that proxy. In the answer file foreman-proxy-content-answers.yaml there is foreman_proxy_content: false set.

During the original installation with Foreman 2.0/Katello 3.15 I have used the --no-enable-foreman-proxy-content option to get rid of the content functions and only use it as puppetmaster and puppet proxy…

So I need to set --enable-foreman-proxy-content?

You probably don’t want to serve content, but that’s the place that’s responsible for the certs…

Yes, that would probably work.

O.K. Yes it does work. In a way…

I have run the installer with --enable-foreman-proxy-content. But I don’t really like it as it installs and enables the whole pulpcore content management which I don’t need and want on that server. There doesn’t seem to be an option to disable or prevent pulpcore installation with foreman-proxy-content enabled.

It doesn’t really makes sense to me to expect a full content proxy to update the certificate for puppet. And there is no simple foreman-installer scenario to install a proxy without content (see Foreman-installer scenario for foreman-proxy installation

Worse: now the main server cannot connect to the proxy with the new certificates (using a different ca) anymore. Although katello-certs-check was happy, the main server doesn’t accept the new certificate:

2022-05-04 19:02:49 [ERROR ] [configure] Error making PUT request to https://foreman.example.com/api/v2/smart_proxies/6/refresh: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) for proxy https://foreman-puppet.example.com:9090/v2/features Please check the proxy is configured and running on the host.
2022-05-04 19:02:49 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-puppet.example.com]/features: change from ["Logs", "Puppet"] to ["Logs", "Pulpcore", "Puppet"] failed: Error making PUT request to https://foreman.example.com/api/v2/smart_proxies/6/refresh: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) for proxy https://foreman-puppet.example.com:9090/v2/features Please check the proxy is configured and running on the host.
2022-05-04 19:02:49 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-puppet.example.com]: Failed to call refresh: Error making PUT request to https://foreman.example.com/api/v2/smart_proxies/6/refresh: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) for proxy https://foreman-puppet.example.com:9090/v2/features Please check the proxy is configured and running on the host.
2022-05-04 19:02:49 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman-puppet.example.com]: Error making PUT request to https://foreman.example.com/api/v2/smart_proxies/6/refresh: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) for proxy https://foreman-puppet.example.com:9090/v2/features Please check the proxy is configured and running on the host.

So I do I tell the main server which ca chains to accept to connect to the proxies? Currently, my main server and the two proxies all used the ca with the same ca chain. We have to move to a different ca and the puppet proxy is the first to move because the old certificate is expiring. Thus, for the transition period I need to tell the main server to accept both ca chains to be able to connect to the proxies with the new or old certificates.

And possibly, I need to do the same on the proxy to be able to connect to the main server?

Looking through my installation notes I suppose I need tell the installer both ca chains with --certs-server-ca-cert ca-bundle.crt, i.e. putting both chains into the ca-bundle.crt.

Checking the certificates presented by the foreman-proxy ssl server I have noticed that it only presents the server certificate itself and no intermediate ca certificates.

Technically, a ssl server should present its own server certificate and all intermediate ca certificates (and no root ca certificate as that should be known to the client).

The foreman proxy on port 9090 only presents the server certificate, i.e. intermediates are missing.

The foreman server on port 443 presents the server certificate, the intermediate CAs but also the root ca. The latter should not be in there.

O.K. Now a ran a

[root@foreman ~]# foreman-installer --scenario katello --certs-server-ca-cert "/etc/pki/tls/certs/ca-bundle.crt" --certs-update-server-ca

on the main server with ca-bundle.crt containing the intermediate and root ca certs of both chains.

Then I ran

[root@foreman-puppet ~]# foreman-installer --enable-foreman-proxy-content

On the proxy. This installs and enables the pulpcore feature but at least the main server can contact the proxy and puppet reports from the proxy are getting into the main server.

What’s ugly is that the main server wants to check pulpcore on the overview page for the smart proxy but cannot connect:

Oops, we're sorry but something went wrong foreman-puppet.example.com is unreachable. Failed to open TCP connection to foreman-puppet.example.com:443 (Connection refused - connect(2) for "foreman-puppet.example.com" port 443)

I have been looking through foreman-installer options on the proxy but I don’t see anything to have foreman-proxy-content enabled but disable/deinstall/deactive pulpcore. Is that true?

Does anyone have an idea how to best handle this kind of setup?