How to configure ssl_ca in foreman.yaml on Smart Proxy?

Support
,
1

Problem:
Everytime I run the foreman-installer on my katello/puppet smart proxy it overrides the /etc/puppetlabs/puppet/foreman.yamlwith:

:url: "https://foreman-main.foo.com"
:ssl_ca: "/etc/pki/katello/puppet/puppet_client_ca.crt"

I then need to edit this file to update the :ssl_ca parameter to target my corporate CA used to sign the upstream foreman server (defined by :url)

I can’t seem to find any foreman-installer parameter that would allow me to specify which CA file I want to be used in foreman.yaml.

Here’s the full foreman-installer I use:

foreman-installer --scenario foreman-proxy-content \
--certs-tar-file "/root/smart-proxy.foo.com-certs.tar" \
--enable-puppet \
--puppet-server true \
--puppet-agent-server-hostname smart-proxy.foo.com \
--puppet-server-ca true \
--puppet-ca-server smart-proxy.foo.com \
--puppet-server-foreman-url "https://foreman-main.foo.com" \
--foreman-proxy-foreman-base-url "https://foreman-main.foo.com" \
--foreman-proxy-trusted-hosts "foreman-main.foo.com" \
--foreman-proxy-trusted-hosts "smart-proxy.foo.com" \
--foreman-proxy-oauth-consumer-key "yXiqLcdvfgdwv7mL4oBfX9Vt4MUbxFdE3j" \
--foreman-proxy-oauth-consumer-secret "Sb7DfBHkA4wdvwHBMJo3RjtotPnwryZs9" \
--foreman-proxy-puppet true \
--foreman-proxy-puppetca true \
--foreman-proxy-puppet-url "https://smart-proxy.foo.com:8140" \
--foreman-proxy-puppet-ssl-cert "/etc/puppetlabs/puppet/ssl/certs/smart-proxy.foo.com.pem" \
--foreman-proxy-puppet-ssl-key "/etc/puppetlabs/puppet/ssl/private_keys/smart-proxy.foo.com.pem" \
--foreman-proxy-ssl-ca "/etc/foreman-proxy/ssl_ca_combined.pem" \
--foreman-proxy-content-enable-yum true

Expected outcome:
Have an option to tell the foreman-installer which :ssl_ca file to configure in /etc/puppetlabs/puppet/foreman.yaml.

Foreman and Proxy versions: 3.17 (katello 4.19)

Distribution and version: Alma 9.7

2

That’s one of the many problems you’ll see if you start messing with the certificate configuration of the modules instead of following the docs and let foreman-installer assemble the necessary certificates in the right places using the certs module. foreman-installer then assembles the files in other places then you point your configuration and it doesn’t fit anymore…

So either reset your manual changes and use the certs module or copy the correct content into the expected file. I think you cannot change the path in foreman.yaml because it’s used by node.rb and thus is to be expected to have the appropriate permissions and selinux labels.

The certs module of foreman-installer would copy the required content into /etc/pki/katello/puppet/puppet_client_ca.crt.

Otherwise, you’ll have to find the right option which affects those settings. As they are puppet server settings for the connection to foreman, I suspect you have to mess with the puppet-server-foreman options for foreman-installer.

3

Yes I agree, we need to regenerate proper katello certificates on the primary foreman instance as this is where the certificates then used on the smart proxy are sourced from.
In my case it’s the katello certificate on the main foreman server that needs to already allow connecting to itself on foreman-main:443.
With this I shouldn’t have to manipulate any files by hand.

4

No: with this you should not modify the foreman-installer options for the certificate handling.

For instance

should be pointing to /etc/foreman-proxy/ssl_ca.pem which contains the self-signed foreman ca certificate which is used to secure the internal communication between the main server and the proxy. /etc/foreman-proxy/ssl_ca.pem contains what is usually found in katello-default-ca.crt.

If you point the foreman-proxy specifically to ssl_ca_combined.pem (I don’t know what you have put in there…) you’ll run into problems. foreman-installer writes /etc/foreman-proxy/ssl_ca.pem but does not handle anything in /etc/foreman-proxy/ssl_ca_combined.pem.

If the contents of your file doesn’t contain the foreman ca certificate, the proxy cannot talk to the server anymore.

As I wrote before: don’t change the certificate settings for specific parts of foreman/katello. Follow the docs if you want to install custom certificates. The certs modules of foreman-installer handles deployment of certificates for you and puts all certificates, chains and keys into the places where they are needed…

1 Like
5

(post deleted by author)

6

@gvde Yes, if I can avoid any extra foreman-installer options all the better.

My primary foreman/katello instance uses a corporate CA for the web-server, which was installed using (as per doc):

foreman-installer --certs-server-cert /etc/foreman/ssl/foreman.crt \
--certs-server-key /etc/foreman/ssl/foreman.key \
--certs-server-ca-cert /etc/foreman/ssl/FOO-CERT-CHAIN.crt \
--certs-update-server --certs-update-server-ca

As such foreman-main:443 presents an SSL certificate signed by FOO-CERT-CHAIN.crt

I then followed the documentation to install the smart proxy:

  1. Generate the smart proxy certificates tar file from the main foreman instance:
foreman-proxy-certs-generate \
--foreman-proxy-fqdn smart-proxy.foo.com \
--certs-tar /root/smart-proxy_cert/smart-proxy.foo.com-certs.tar
  1. Copy the resulting smart-proxy.foo.com-certs.tar to the smart proxy server.
  2. Configure the smart proxy using:
foreman-installer\
--scenario foreman-proxy-content\
--certs-tar-file                       "/root/smart-proxy.foo.com-certs.tar"\
--foreman-proxy-foreman-base-url       "https://foreman-main.foo.com"\
--foreman-proxy-trusted-hosts          "foreman-main.foo.com"\
--foreman-proxy-trusted-hosts          "smart-proxy.foo.com"\
--foreman-proxy-oauth-consumer-key     "yXiqLcnPBTg7mL4oBfX9Vt4MUbxFdE3j"\
--foreman-proxy-oauth-consumer-secret  "Sb7DfBHkA4SxbHBMJo3RjtotPnwryZs9"

SSL issue
The issue is that the resulting foreman.yaml file points to a CA file

[root@foreman-main]# grep ssl_ca /etc/puppetlabs/puppet/foreman.yaml
:ssl_ca: "/etc/pki/katello/puppet/puppet_client_ca.crt"

that doesn’t allow connecting to foreman-main:443(which I understand is the endpoint used to retrieve nodes ENC):

[root@smart-proxy]# openssl s_client -connect foreman-main.foo.com:443 2>/dev/null -CAfile /etc/pki/katello/puppet/puppet_client_ca.crt 2>&1 | grep "Verify 
return"
Verify return code: 19 (self-signed certificate in certificate chain)

This error is because the CA file puppet_client_ca.crt used to check the connection between the smart proxy and foreman-main (ENC) is signed by Katello:

[root@smart-proxy]# openssl x509 -noout -subject -issuer -in /etc/pki/katello/puppet/puppet_client_ca.crt
subject=C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman-main.foo.com
issuer=C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman-main.foo.com

whereas the SSL certificate presented by the web-server on foreman-main is signed by the corporate CA:

[root@smart-proxy]# openssl s_client -connect foreman-main.foo.com:443 2>/dev/null | openssl x509 -noout -issuer
issuer=OU=FOO, CN=FOO-CERT
7

O.K. The problem with foreman-installer is that if you ever later used any custom certificate configuration command for any component of foreman this change is saved in the answers file and thus is always applied until you reset or restore the original value. So while the above is good you have to write up all other configuration changes you have done later to give a complete and full picture of what foreman-installer is actually configuring for you.

And that’s why it’s a dangerous path once you start setting those customized configuration in foreman-installer.

Did you use katello-certs-check to check the cert, key and chain. Does FOO-CERT-CHAIN.crt contain the full chain including the root CA certificate?

So you don’t deploy certificates for your proxy? I guess, this means that your smart-proxy will get certificates signed by the foreman server. This kind of setup is not supported according to the docs:

If you use a custom SSL certificate, you must also configure each Smart Proxy server with a distinct custom SSL certificate. For more information, see Configuring Smart Proxy server with a custom SSL certificate.

And you did that for the initial installation and configuration? Otherwise there would be the certs-update-server missing to tell the installer to actually update the certs.

This is also for the initial configuration only.

The relevant command should be

# openssl s_client -connect foreman-main.foo.com:443 -no-CApath -no-CAstore -CAfile /etc/pki/katello/puppet/puppet_client_ca.crt

With a standard deployment, foreman-installer would copy the content of your file FOO-CERT-CHAIN.crt into /etc/pki/katello/certs/katello-server-ca.crt and also into /etc/pki/katello/puppet/puppet_client_ca.crt.

However, I am not sure if this applies in your case, when you don’t supply your own certificate to the smart-proxy. One of the developers should probably be able to tell, where exactly the file puppet_client_ca.crt is used and whether it is deployed with the main server ca file if the smart proxy is using the default ca.

This very much suggests that you either did the full foreman initial installation without custom certificates or at least for the smart proxy.

This is not supported according to the docs. This might be considered a bug in foreman-installer, that it doesn’t handle this case properly, but at least so far it seems to me that you have deploy custom certificates on the main server but used default certificates with the CA you see in etc/pki/katello/puppet/puppet_client_ca.crt on the smart proxy.

It seems foreman-installer does not support that and thus deployed the default CA on the smart proxy using the default CA. (Unless the update option is missing and it would update the ca if you gave the option to do so.) The docs say this case isn’t supported.

1 Like
8

@gvde Thank you for your help and your thorough explanations!

Indeed, I missed a key instruction: use SSL certificates signed by the same CA on both the foreman server and the smart proxy server.

I went through the procedure again and was able to get a working smart proxy without having to edit anything manually after running the foreman-installer:

foreman-installer \
--scenario foreman-proxy-content \
--certs-tar-file "/root/smart-proxy.foo.com-certs.tar" \
--foreman-proxy-foreman-base-url "https://foreman-main.foo.com" \
--foreman-proxy-trusted-hosts "foreman-main.foo.com" \
--foreman-proxy-trusted-hosts "smart-proxy.foo.com" \
--foreman-proxy-oauth-consumer-key "yXiqLcnPBTg7mL4oBfX9Vt4MUbxFdE3j" \
--foreman-proxy-oauth-consumer-secret "Sb7DfBHkA4SxbHBMJo3RjtotPnwryZs9" \
--foreman-proxy-foreman-ssl-ca "/etc/pki/ca-trust/source/anchors/FOO-CERT-CHAIN.crt"

However, I still needed to add the --foreman-proxy-foreman-ssl-ca parameter (didn’t find any other way) to the foreman-installer command to update the :foreman_ssl_ca in /etc/foreman-proxy/settings.yml (tells foreman-proxy which CA to use when talking to the main foreman):

[root@smart-proxy ~]# grep -B6 foreman_ssl_ca /etc/foreman-proxy/settings.yml
# Endpoint for reverse communication
:foreman_url: https://foreman-main.foo.com

# SSL settings for client authentication against Foreman. If undefined, the values
# from general SSL options are used instead. Mainly useful when Foreman uses
# different certificates for its web UI and for smart-proxy requests.
:foreman_ssl_ca: /etc/pki/ca-trust/source/anchors/FOO-CERT-CHAIN.crt

otherwise this parameter, when unset, resolves to the same value as :ssl_ca_file:

[root@smart-proxy ~]# grep -B3 :ssl_ca /etc/foreman-proxy/settings.yml
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem

which doesn’t allow communication with the main foreman server:

[root@smart-proxy ~]# openssl s_client -connect foreman-main.foo.com:443 2>/dev/null -no-CApath -no-CAstore -CAfile /etc/puppetlabs/puppet/ssl/certs/ca.pem 2>&1 | grep "Verify return"
Verify return code: 19 (self-signed certificate in certificate chain)

Additionally, as you pointed out, the content of FOO-CERT-CHAIN.crt is copied into katello-server-ca.pem as well as puppet_client_ca.crt:

[root@smart-proxy ~]# diff -bw /etc/pki/ca-trust/source/anchors/FOO-CERT-CHAIN.crt /etc/pki/ca-trust/source/anchors/katello-server-ca.pem 
[root@smart-proxy ~]# diff -bw /etc/pki/ca-trust/source/anchors/FOO-CERT-CHAIN.crt /etc/pki/katello/puppet/puppet_client_ca.crt
[root@smart-proxy ~]#

which solves the issue I initially created this post for:

the :ssl_ca parameter in foreman.yaml

[root@smart-proxy ~]# grep ssl_ca /etc/puppetlabs/puppet/foreman.yaml 
:ssl_ca: "/etc/pki/katello/puppet/puppet_client_ca.crt"

points to a CA that verifies the connection to the main foreman:

[root@smart-proxy ~]# openssl s_client -connect foreman-main.foo.com:443 2>/dev/null -no-CApath -no-CAstore -CAfile /etc/pki/katello/puppet/puppet_client_ca.crt 2>&1 | grep "Verify return"
Verify return code: 0 (ok)
9

As I wrote before: that is the wrong approach and it will just break your system. You will have to make sure that the file /etc/pki/ca-trust/source/anchors/FOO-CERT-CHAIN.crt will always contain the certificates the foreman expects.

By default, it points to /etc/foreman-proxy/foreman_ssl_ca.pem and foreman-installer will write the necessary certificates into this file. It’s actually the same file as the puppet one above. A quick scan by size on my content proxy suggests that

/etc/pki/katello/certs/katello-server-ca.crt
/etc/pki/katello/puppet/puppet_client_ca.crt
/etc/rhsm/ca/katello-server-ca.pem
/etc/foreman-proxy/foreman_ssl_ca.pem
/var/www/html/pub/katello-server-ca.crt

all contain the ca certs for the server connection.

Thus, if a file does not contain what you expect, find out why it hasn’t been updated. Do not reconfigure foreman to point to a different file which you have to maintain. That’s just the wrong approach.

That’s the problem: resetting sets the option to the foreman-installer puppet module default. However, the foreman-installer scenarios have specific preset defaults defined in /etc/foreman-installer/scenarios.d/foreman-proxy-content-answers.yaml in the rpm. (the file is overwritten with the latest applied settings, but if you have a /etc/foreman-installer/scenarios.d/foreman-proxy-content-answers.yaml.rpmnew file lying around you’ll see the original content from the rpm).

By default, foreman-installer sets

foreman_proxy:
  foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem

Please check /etc/foreman-proxy/foreman_ssl_ca.pem. It should contain what you need.

You have to set --foreman-proxy-foreman-ssl-ca=/etc/foreman-proxy/foreman_ssl_ca.pem to get the default for a content proxy. foreman-installer should maintain the content of that file.

If the file does not contain the correct certificates at the moment, you need to run foreman-installer to update the certificates. Check the docs on how to renew certificates. You’ll have to add --certs-update-server and probably --certs-update-server-ca to make sure that all certificates and ca certs are updated.

It should never be necessary to point foreman to a different file nor should it be necessary to update on of those files from foreman manually.

If even after following the docs a file in one location still does not contain the correct content (and you really didn’t point foreman-installer elsewhere before) then you should ask here. That would be a bug.

We have been running our foreman setup for a couple of years now and upgraded through various versions and always used the certs module to update certificates and even used it when we switched the ca to a different provider, providing temporarily two ca chains.