Foreman 2.2.1 + Katello 3.17 - custom CA certificates issue

Problem:

Unable to replace/update certificates on 1 out of 3 remote foreman-proxy servers using the generated tar certificate bundle.

Expected outcome:

Update/replace relevant certificates (with custom CA) on the remote foreman-proxy server and get it registered to foreman.

Foreman and Proxy versions:

foreman-proxy-2.2.1-1.el7.noarch
foreman-release-2.2.1-1.el7.noarch

Foreman and Proxy plugin versions:

Distribution and version:
RHEL 7
Other relevant data:

We are using katello 3.17 and we’re trying to use a custom CA on our existing environment. The environment consists of 4 RHEL 7 servers.

1 main katello server
2 foreman-proxy compilers (puppetserver)
1 foreman-proxy CA

We managed to get the certificates updated easily on 3 servers.

1 main katello server
2 foreman-proxy compilers (puppetserver)

But not on the foreman-proxy CA.

The process that we’re doing to update the certificate at the remote foreman-proxy is as follows.

At the main katello server:

  1. Check the validity of the certificates
katello-certs-check -c /etc/ssl/fmanca01/cert/fmanca01.pem \
                    -k /etc/ssl/fmanca01/key/fmanca01.key \
                    -b /etc/ssl/net.ca-bundle
  1. Generate a certificate bundle tar file
foreman-proxy-certs-generate --foreman-proxy-fqdn "fmanca01.inet.net"\
                             --certs-tar /root/custom_cert/fmanca01.inet.net-certs.tar\
                             --server-cert /etc/ssl/fmanca01.inet.net/cert/fmanca01.pem\
                             --server-key /etc/ssl/fmanca01.inet.net/key/fmanca01.key\
                             --server-ca-cert /etc/ssl/net.ca-bundle\
                             --certs-update-all	 

At the remote foreman-proxy:

  1. Install the katello-ca-consumer-latest.noarch.rpm

  2. Run the command shown after running the foreman-proxy-certs-generate command.

foreman-installer \
                    --scenario foreman-proxy-content \
                    --certs-tar-file                              "/root/fmanca01.inet.net-certs.tar"\
                    --foreman-proxy-content-parent-fqdn           "fmanp02.inet.net"\
                    --foreman-proxy-register-in-foreman           "true"\
                    --foreman-proxy-foreman-base-url              "https://fmanp02.inet.net"\
                    --foreman-proxy-trusted-hosts                 "fmanp02.inet.net"\
                    --foreman-proxy-trusted-hosts                 "fmanca01.inet.net"\
                    --foreman-proxy-oauth-consumer-key            "ZuQX3XETDtBhboo4WuxqKufKEytkh5FQ"\
                    --foreman-proxy-oauth-consumer-secret         "7UKsqvTcJ5DY4Lrxhqx3PQBFXoje34gR"\
                    --puppet-server-foreman-url                   "https://fmanp02.inet.net"

The output of the command

Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
Preparing installation Done
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/foreman-installer/foreman-proxy-content.log

Error from /var/log/foreman-installer/foreman-proxy-content.log

[ERROR 2021-05-19T07:47:18 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[fmanca01.inet.net]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://fmanp02.inet.net/api/v2/smart_proxies?search=name="fmanca01.inet.net"

Some observations:

I compared some of the certifcates with the other foreman-proxy that worked and found the following differences:

The following certificates were updated on the 2 remote foreman-proxy that worked but not on fmanca01.inet.net

/etc/pki/ca-trust/source/anchors/katello_server-host-cert.crt
/etc/foreman-proxy/ssl_cert.pem
/etc/foreman-proxy/foreman_ssl_ca.pem

The following packages are not being updated at the fmanca01.inet.net which I expected to be updated because when I extract the contents of the tar bundle, there are already fmanca01.inet.net-xxxx-xxxxx-xxxxx-1.0-3.noarch in it.

fmanca01.inet.net-foreman-proxy-client-1.0-1.noarch
fmanca01.inet.net-qpid-router-client-1.0-1.noarch
fmanca01.inet.net-puppet-client-1.0-1.noarch
fmanca01.inet.net-apache-1.0-1.noarch
fmanca01.inet.net-qpid-client-cert-1.0-1.noarch
fmanca01.inet.net-qpid-broker-1.0-1.noarch
fmanca01.inet.net-foreman-proxy-1.0-1.noarch
fmanca01.inet.net-qpid-router-server-1.0-1011.noarch

Any help is greatly appreciated.

I managed to get the foreman-proxy registered by replacing the contents of the following files with the custom server cert, private key, and ca-bundle/chain

cat /etc/ssl/fmanca01.vestasext.net/cert/fmanca01.pem > /etc/foreman-proxy/ssl_cert.pem
cat /etc/ssl/fmanca01.vestasext.net/key/fmanca01.key > /etc/foreman-proxy/ssl_key.pem
cat /etc/ssl/net.ca-bundle > /etc/foreman-proxy/foreman_ssl_ca.pem

Am I right in thinking that this should have been taken cared of by the command below?

foreman-installer
–scenario foreman-proxy-content
–certs-tar-file “/root/fmanca01.inet.net-certs.tar”
–foreman-proxy-content-parent-fqdn “fmanp02.inet.net
–foreman-proxy-register-in-foreman “true”
–foreman-proxy-foreman-base-url “https://fmanp02.inet.net
–foreman-proxy-trusted-hosts “fmanp02.inet.net
–foreman-proxy-trusted-hosts “fmanca01.inet.net
–foreman-proxy-oauth-consumer-key “ZuQX3XETDtBhboo4WuxqKufKEytkh5FQ”
–foreman-proxy-oauth-consumer-secret “7UKsqvTcJ5DY4Lrxhqx3PQBFXoje34gR”
–puppet-server-foreman-url “https://fmanp02.inet.net

An issue that I’m facing now is that all of the fine tuning that I did such as increasing the jruby instances, sending metrics to graphite, additional puma workers, etc were reset back to default so we’re back to having performance issues but this should be an easy fix. I’ll start finetuning the servers and then see if we get back to the original working condition.