Setup Foreman webserver to use custom CA

I’ve dug through dozens upon dozens of posts about setting up external CA certs to use with Katello / Foreman.

I’m trying to setup a simple Katello/Foreman server in one data center and a smart proxy in the other, following the install directions for 3.10 worked, sort of… to give me a working server and smart-proxy. This effort is to retire out aging Spacewalk Server as we roll out a new Foreman server build.

Problem is that my company requires all websites to use HTTPS. Again, no problem, if I only want to use IE to acccess the website. The self-signed certs kick off the HSTS errors in Chrome, Firefox and Brave. We also have to use a wildcard cert, so the self-signed cert is also a non-starter.

On my old Foreman / Puppet setup it was no problem to setup the web interface to use SSL with the wildcard cert. I can force it in using the --foreman-server-ssl*, --foreman-websockets-ssl* and --certs-server* flags. It gives me a good HTTPS session, but the smart proxy doesn;t work. Passing flags --server-ca-cert, --server-cert and --server-key appears to have fixed that issue. Puppet is happy and all appeared well, but then it wasn’t.

Subscription manager lost it’s mind with SSL invalid certs errors. I did a katello-remove on the master and smart-proxy to try with only passing the Java,io override to use /var/tmp since /tmp has the noexec flag.

The install guide and forums don’t really have any solutions and all the docs on changing certs to external CA didn’t do anything.

So for the question. Can Katello be configured to use external CAs with smart-proxy hosts where everything works as it should? If so, is there a reference document on making it work? Are there workarounds is it that everyone just excepts that you can’t access the Foreman GUI with a valid external CA signed certificate (Digicert in this case)?

And before I get the hundreds of RTFM, I’ve gone through the Katello install and users guide, the Foremen troubleshooting FAQ and spent a week reading every single article at Red Hat, Google Groups, and this forum looking for answers.

Assistance in making this work would be greatly appreciated.

It seems that when Katello 3.10 came out, the katello-installer was now integrated with the foreman-installer. Unfortunately it seems the documentation for the install wasn’t updated to include the readme details, or i didn’t see them in the 3.10 install docs, from the katello-installer readme.md on version 3.9 and back located at https://github.com/Katello/katello-installer/tree/KATELLO-3.9.

Specifically the certificates section that was exactly what I needed to resolve the issues I had with custom certs:

Certificates

Katello installer comes with a default CA used both for the server SSL certificates as well as the client certificates used for authentication of the subservices (pulp, foreman-proxy, subscription-manager etc.).

Custom Server Certificates

Katello-installer runs a validation script for passed input certificate files. One can run it manually as follows:

katello-certs-check -c ~/path/to/server.crt\
                    -r ~/path/to/server.crt.req\
                    -k ~/path/to/server.key\
                    -b ~/path/to/cacert.crt

The check is performed also as part of the installer script. In case the script marked the certs as invalid incorrectly, one can skip this check by passing --certs-skip-check to the installer.

When running the installer for the first time:

foreman-installer --scenario katello\
                  --certs-server-cert ~/path/to/server.crt\
                  --certs-server-cert-req ~/path/to/server.crt.req\
                  --certs-server-key ~/path/to/server.key\
                  --certs-server-ca-cert ~/path/to/cacert.crt

Where the --certs-server-ca-cert is the CA used for issuing the server certs (this CA gets distributed to the consumers and Foreman proxies).

For the Foreman proxy, these options are passed as part of the foreman-proxy-certs-generate script:

foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY"\
                             --certs-tar ~/$FOREMAN_PROXY-certs.tar\
                             --server-cert ~/path/to/server.crt\
                             --server-cert-req ~/path/to/server.crt.req\
                             --server-key ~/path/to/server.key\
                             --server-ca-cert ~/cacert.crt

The rest of the procedure is identical to the default CA setup.

Setting custom certificates after ‘foreman-installer --scenario katello’ was already run.

The first run of foreman-installer --scenario katello uses the default CA for both server and client certificates. To enforce the custom certificates to be deployed, on needs to set --certs-update-server for updating the CA certificate and --certs-update-server-ca because the server CA changed as well (and new katello-ca-consumer rpm needs to be regenerated):

foreman-installer --scenario katello\
                  --certs-server-cert ~/path/to/server.crt\
                  --certs-server-cert-req ~/path/to/server.crt.req\
                  --certs-server-key ~/path/to/server.key\
                  --certs-server-ca-cert ~/path/to/cacert.crt\
                  --certs-update-server --certs-update-server-ca

After the server CA changes, the new version of the consumer-ca-consumer rpm needs to be installed on the consumers, as well as:

rpm -Uvh http://katello.example.com/pub/katello-ca-consumer-latest.noarch.rpm

When using the custom server CA, the CA needs to be used for the server certificates on the Foreman proxies as well. The certificates for the Foreman proxy are deployed to the Foreman proxy through the use of the foreman-proxy-certs-generate script (followed by copying the certs tar to the Foreman proxy and running the ‘foreman-installer --scenario foreman-proxy-content’ to refresh the certificates).:

foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY_CONTENT"\
                             --certs-tar ~/$FOREMAN_PROXY_CONTENT-certs.tar\
                             --server-cert ~/path/to/server.crt\
                             --server-cert-req ~/path/to/server.crt.req\
                             --server-key ~/path/to/server.key\
                             --server-ca-cert ~/cacert.crt\
                             --certs-update-server

Updating Certificates

On a server

To regenerate the server certificates when using the default CA or enforce deploying new certificates for the custom server CA run the ‘foreman-installer --scenario katello’ with --certs-update-server option:

foreman-installer --scenario katello --certs-update-server

To regenerate all the certificates used in the Katello server, there is a --certs-update-all . This will generate and deploy the certificates as well as restart corresponding services.

On a Foreman proxy

For updating the certificates on a Foreman proxy pass the same options (either --certs-update-server or --certs-update-all ) to the foreman-proxy-certs-generate script. The new certs tar gets generated that needs to be transferred to the Foreman proxy and then foreman-installer --scenario foreman-content-proxy needs to be re-run to apply the updates and restart corresponding services.

4 Likes

@jjperry69 I’m sorry that this went unacknowledged for as long as it did and for the difficulty involved in finding the documentation you were looking for. Very nice catch with the katello-installer deprecation though!

I’ve taken a shot at moving the custom certs writeup you mentioned and formalizing it for our official documentation. If you wouldn’t mind taking a look at it I would appreciate any feedback since you just ran through the scenario recently and you can tell me if it maps properly to your experience. Maybe you’ll have some helpful details too.

Here’s my pull request: https://github.com/theforeman/theforeman.org/pull/1299

1 Like