Using custom SSL certficates with foreman-proxy

Support
,
1

Problem:
“Unable to get local issuer certificate” when curling https://foreman-proxy.example.com:9090/features.

The certificates work when using openssl verify on the certs on the foreman instance.

Expected outcome:
Output of proxy features

Foreman and Proxy versions:
3.14/4.16

Foreman and Proxy plugin versions:

Plugins - Packages 
ansible-collection-theforeman-foreman-5.1.0-2.el9.noarch
candlepin-4.4.20-1.el9.noarch
candlepin-selinux-4.4.20-1.el9.noarch
dynflow-utils-1.6.3-1.el9.x86_64
foreman-3.14.0-1.el9.noarch
foreman-cli-3.14.0-1.el9.noarch
foreman-dynflow-sidekiq-3.14.0-1.el9.noarch
foreman-installer-3.14.0-1.el9.noarch
foreman-installer-katello-3.14.0-1.el9.noarch
foreman-libvirt-3.14.0-1.el9.noarch
foreman-postgresql-3.14.0-1.el9.noarch
foreman-proxy-3.14.0-1.el9.noarch
foreman-redis-3.14.0-1.el9.noarch
foreman-release-3.14.0-1.el9.noarch
foreman-selinux-3.14.0-1.el9.noarch
foreman-service-3.14.0-1.el9.noarch
katello-4.16.0-1.el9.noarch
katello-certs-tools-2.10.0-1.el9.noarch
katello-client-bootstrap-1.7.9-2.el9.noarch
katello-common-4.16.0-1.el9.noarch
katello-repos-4.16.0-1.el9.noarch
katello-selinux-5.2.0-1.el9.noarch
pulpcore-selinux-2.0.1-1.el9.x86_64
python3.11-pulp-ansible-0.22.4-1.el9.noarch
python3.11-pulp-cli-0.31.0-1.el9.noarch
python3.11-pulp-container-2.22.1-1.el9.noarch
python3.11-pulp-deb-3.5.1-1.el9.noarch
python3.11-pulp-glue-0.31.0-1.el9.noarch
python3.11-pulp-python-3.12.6-1.el9.noarch
python3.11-pulp-rpm-3.27.2-1.el9.noarch
python3.11-pulpcore-3.63.11-1.el9.noarch
rubygem-dynflow-1.9.0-1.el9.noarch
rubygem-foreman-tasks-10.0.2-1.fm3_14.el9.noarch
rubygem-foreman_ansible-15.0.7-1.fm3_14.el9.noarch
rubygem-foreman_bootdisk-22.0.3-1.fm3_14.el9.noarch
rubygem-foreman_maintain-1.10.3-1.el9.noarch
rubygem-foreman_puppet-8.1.1-1.fm3_14.el9.noarch
rubygem-foreman_remote_execution-15.0.2-1.fm3_14.el9.noarch
rubygem-foreman_remote_execution-cockpit-15.0.2-1.fm3_14.el9.noarch
rubygem-foreman_snapshot_management-4.0.0-1.fm3_13.el9.noarch
rubygem-foreman_templates-10.0.7-1.fm3_14.el9.noarch
rubygem-foreman_webhooks-4.0.1-1.fm3_14.el9.noarch
rubygem-hammer_cli-3.14.0-1.el9.noarch
rubygem-hammer_cli_foreman-3.14.0-1.el9.noarch
rubygem-hammer_cli_foreman_ansible-0.7.0-1.fm3_11.el9.noarch
rubygem-hammer_cli_foreman_puppet-0.1.0-1.fm3_11.el9.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.2-1.fm3_14.el9.noarch
rubygem-hammer_cli_foreman_ssh-0.0.3-1.el9.noarch
rubygem-hammer_cli_foreman_tasks-0.0.22-1.fm3_14.el9.noarch
rubygem-hammer_cli_foreman_templates-0.3.0-1.el9.noarch
rubygem-hammer_cli_foreman_webhooks-0.1.0-1.el9.noarch
rubygem-hammer_cli_katello-1.16.0-1.el9.noarch
rubygem-katello-4.16.0-1.el9.noarch
rubygem-pulp_ansible_client-0.22.3-1.el9.noarch
rubygem-pulp_certguard_client-3.63.9-1.el9.noarch
rubygem-pulp_container_client-2.22.1-1.el9.noarch
rubygem-pulp_deb_client-3.3.1-1.el9.noarch
rubygem-pulp_file_client-3.63.9-1.el9.noarch
rubygem-pulp_ostree_client-2.4.5-1.el9.noarch
rubygem-pulp_python_client-3.12.5-1.el9.noarch
rubygem-pulp_rpm_client-3.27.2-1.el9.noarch
rubygem-pulpcore_client-3.63.9-1.el9.noarch
rubygem-smart_proxy_dynflow-0.9.4-1.fm3_14.el9.noarch
rubygem-smart_proxy_pulp-3.4.0-1.fm3_13.el9.noarch

Distribution and version:
RHEL9

Other relevant data:

Foreman Install Command 
foreman-installer -v -l INFO
  --scenario katello
  --enable-foreman
  --enable-foreman-cli
  --enable-foreman-cli-ansible
  --enable-foreman-cli-puppet
  --enable-foreman-cli-remote-execution
  --enable-foreman-cli-ssh
  --enable-foreman-cli-webhooks
  --enable-foreman-compute-libvirt
  --enable-foreman-plugin-ansible
  --enable-foreman-plugin-bootdisk
  --enable-foreman-plugin-puppet
  --enable-foreman-plugin-remote-execution
  --enable-foreman-plugin-remote-execution-cockpit
  --enable-foreman-plugin-snapshot-management
  --enable-foreman-plugin-webhooks
  --no-enable-foreman-plugin-acd
  --no-enable-foreman-plugin-dhcp-browser
  --no-enable-foreman-plugin-discovery
  --no-enable-foreman-cli-discovery
  --enable-foreman-proxy
  --enable-puppet
  --enable-foreman-plugin-templates
  --enable-foreman-cli-templates
  --enable-foreman-proxy-plugin-ansible
  --foreman-initial-admin-password "<REDACTED>"
  --foreman-initial-location 'Main'
  --foreman-initial-organization 'MyOrg'
  --foreman-proxy-dhcp=true
  --foreman-proxy-dhcp-interface 'eno1'
  --foreman-proxy-dhcp-gateway '192.168.0.70'
  --foreman-proxy-dhcp-nameservers '192.168.0.30'
  --foreman-proxy-dhcp-netmask '255.255.255.128'
  --foreman-proxy-dhcp-network '192.168.0.1'
  --foreman-proxy-dhcp-key-name 'omapi_key'
  --foreman-proxy-dhcp-key-secret '<REDACTED>'
  --foreman-proxy-dns=true
  --foreman-proxy-dns-managed=false
  --foreman-proxy-dns-server '192.168.0.20'
  --foreman-proxy-dns-ttl '3600'
  --foreman-proxy-dns-provider 'nsupdate'
  --foreman-proxy-keyfile '/etc/foreman-proxy/dnsupdate.txt'
  --foreman-proxy-tftp=true
  --foreman-proxy-tftp-servername '192.168.0.70'
  --foreman-proxy-http=true
  --foreman-proxy-bmc=true
  --foreman-proxy-bmc-default-provider=ipmitool
  --enable-foreman-proxy-plugin-remote-execution-script
  --foreman-server-ssl-key "/etc/ssl/foreman-certs/key.pem"
  --foreman-server-ssl-cert "/etc/ssl/foreman-certs/cert.pem"
  --foreman-server-ssl-chain "/etc/ssl/foreman-certs/ca.pem"
  --puppet-server-foreman-ssl-ca "/etc/ssl/foreman-certs/ca.pem"
  --foreman-proxy-foreman-ssl-ca "/etc/ssl/foreman-certs/ca.pem"
  --certs-server-cert "/etc/ssl/foreman-certs/cert.pem"
  --certs-server-key "/etc/ssl/foreman-certs/key.pem"
  --certs-server-ca-cert "/etc/ssl/foreman-certs/ca.pem"
  --certs-update-server
  --certs-update-server-ca
foreman-proxy-certs command 
  foreman-proxy-certs-generate
    --foreman-proxy-fqdn foreman-proxy.example.com
    --certs-tar /etc/ssl/smart-proxy/certs.tar
    --server-cert /etc/ssl/smart-proxy/cert.pem
    --server-key /etc/ssl/smart-proxy/key.pem
    --server-ca-cert /etc/ssl/smart-proxy/ca.pem
Foreman Proxy Install Command 
  foreman-installer -v -l INFO
  --scenario foreman-proxy-content
  --foreman-proxy-bmc false
  --foreman-proxy-dhcp false
  --foreman-proxy-dns false
  --foreman-proxy-httpboot false
  --foreman-proxy-puppet false
  --foreman-proxy-puppetca false
  --foreman-proxy-realm false
  --foreman-proxy-tftp false
  --certs-tar-file "/root/certs.tar"
  --foreman-proxy-register-in-foreman "false"
  --foreman-proxy-foreman-base-url "https://foreman.example.com"
  --foreman-proxy-trusted-hosts "foreman.example.com"
  --foreman-proxy-trusted-hosts "foreman-proxy.example.com"
  --foreman-proxy-oauth-consumer-key "<REDACTED>"
  --foreman-proxy-oauth-consumer-secret "<REDACTED>"

Temporary fix:
Foreman seems to add the certificates to these files
ca.pem → /etc/foreman-proxy/foreman_ssl_ca.pem
cert.pem → /etc/foreman-proxy/ssl_cert.pem
key.pem → /etc/foreman-proxy/ssl_key.pem

instead of:
ca.pem → /etc/foreman-proxy/ssl_ca.pem
cert.pem → /etc/foreman-proxy/ssl_cert.pem
key.pem → /etc/foreman-proxy/ssl_key.pem

Adding --foreman-proxy-ssl-ca "/etc/foreman-proxy/foreman_ssl_ca.pem" seems to fix the issue.
Is the any better way to fix this?

2

I think that’s where the certificate issues start, once you set some of them manually instead of using the certs module only. foreman uses certificates in various places, some are self-signed. The certs module puts everything into the right places. If you overwrite paths, it doesn’t fit anymore.

ssl_ca.pem contains the self-signed certificate for the internal communication in foreman from server to the proxy. foreman_ssl_ca.pem contains the (possibly) public ca chain for the communication between the proxy and the server.

But you have changed some of those paths and thus the certs-update-server won’t really fit everywhere…

1 Like
3

I have now tested an reinstall with the removal of these options:

  --foreman-server-ssl-key "/etc/ssl/foreman-certs/key.pem"
  --foreman-server-ssl-cert "/etc/ssl/foreman-certs/cert.pem"
  --foreman-server-ssl-chain "/etc/ssl/foreman-certs/ca.pem"
  --puppet-server-foreman-ssl-ca "/etc/ssl/foreman-certs/ca.pem"
  --foreman-proxy-foreman-ssl-ca "/etc/ssl/foreman-certs/ca.pem"

From what I see the Foreman install is working as it should, and the webgui is still being delivered on the correct custom SSL certificate.

However when testing a reinstall of the smart proxy, the issue regarding ssl still seems to persist.

Using with custom ssl, we’re getting the same Unable to get local issuer certificate issue.

foreman-proxy-certs-generate
    --foreman-proxy-fqdn foreman-proxy.example.com
    --certs-tar /etc/ssl/smart-proxy/certs.tar
    --server-cert /etc/ssl/smart-proxy/cert.pem
    --server-key /etc/ssl/smart-proxy/key.pem
    --server-ca-cert /etc/ssl/smart-proxy/ca.pem

Setting up the smart proxy with selfsigned certificates instead, gives us the error: certificate verify failed (self-signed certificate in certificate chain)

foreman-proxy-certs-generate
    --foreman-proxy-fqdn foreman-proxy.example.com
    --certs-tar /etc/ssl/smart-proxy/certs.tar

Do you know of any other things that should be corrected @gvde?

4

It’s really extremely hard to follow if you don’t post the exact, full commands put instead the options you have left out.

If you do a reinstallation (i.e. an installation on a server with a fresh operating system) I would highly recommend to follow the docs Installing Foreman Server with Katello 4.16 plugin on Enterprise Linux and start with minimal options to test the certificates. Most options which you have used in the original post are probably not even necessary.

Install the main server following the docs. Install the proxy with custom certs following the docs. If you still see the problems it maybe a bug. Don’t make it overly complicated.

For the initial foreman-installer run it should be enough to run it like this:

# foreman-installer --scenario katello \
--foreman-initial-organization "My_Organization" \
--foreman-initial-location "My_Location" \
--foreman-initial-admin-username admin_user_name \
--foreman-initial-admin-password admin_password
--certs-server-ca-cert /etc/pki/tls/certs/foreman-chain.crt \
--certs-server-cert /etc/pki/tls/certs/foreman.crt \
--certs-server-key /etc/pki/tls/private/foreman.key

That’s how we did it.

5

The whole install command was:

foreman-installer -v -l INFO
  --scenario katello
  --enable-foreman
  --enable-foreman-cli
  --enable-foreman-cli-ansible
  --enable-foreman-cli-puppet
  --enable-foreman-cli-remote-execution
  --enable-foreman-cli-ssh
  --enable-foreman-cli-webhooks
  --enable-foreman-compute-libvirt
  --enable-foreman-plugin-ansible
  --enable-foreman-plugin-bootdisk
  --enable-foreman-plugin-puppet
  --enable-foreman-plugin-remote-execution
  --enable-foreman-plugin-remote-execution-cockpit
  --enable-foreman-plugin-snapshot-management
  --enable-foreman-plugin-webhooks
  --no-enable-foreman-plugin-acd
  --no-enable-foreman-plugin-dhcp-browser
  --no-enable-foreman-plugin-discovery
  --no-enable-foreman-cli-discovery
  --enable-foreman-proxy
  --enable-puppet
  --enable-foreman-plugin-templates
  --enable-foreman-cli-templates
  --enable-foreman-proxy-plugin-ansible
  --foreman-initial-admin-password "<REDACTED>"
  --foreman-initial-location 'Main'
  --foreman-initial-organization 'MyOrg'
  --foreman-proxy-dhcp=true
  --foreman-proxy-dhcp-interface 'eno1'
  --foreman-proxy-dhcp-gateway '192.168.0.70'
  --foreman-proxy-dhcp-nameservers '192.168.0.30'
  --foreman-proxy-dhcp-netmask '255.255.255.128'
  --foreman-proxy-dhcp-network '192.168.0.1'
  --foreman-proxy-dhcp-key-name 'omapi_key'
  --foreman-proxy-dhcp-key-secret '<REDACTED>'
  --foreman-proxy-dns=true
  --foreman-proxy-dns-managed=false
  --foreman-proxy-dns-server '192.168.0.20'
  --foreman-proxy-dns-ttl '3600'
  --foreman-proxy-dns-provider 'nsupdate'
  --foreman-proxy-keyfile '/etc/foreman-proxy/dnsupdate.txt'
  --foreman-proxy-tftp=true
  --foreman-proxy-tftp-servername '192.168.0.70'
  --foreman-proxy-http=true
  --foreman-proxy-bmc=true
  --foreman-proxy-bmc-default-provider=ipmitool
  --enable-foreman-proxy-plugin-remote-execution-script
  --certs-server-cert "/etc/ssl/foreman-certs/cert.pem"
  --certs-server-key "/etc/ssl/foreman-certs/key.pem"
  --certs-server-ca-cert "/etc/ssl/foreman-certs/ca.pem"
  --certs-update-server
  --certs-update-server-ca

I will try it again with a minimal install, and if that does not work it is probably as you said, a bug.

6

Just did a test with this minimal install

foreman-installer -v -l INFO
  --scenario katello
  --foreman-initial-admin-password "<REDACTED>"
  --foreman-initial-location 'Main'
  --foreman-initial-organization 'MyOrg'
  --certs-server-cert "/etc/ssl/foreman-certs/cert.pem"
  --certs-server-key "/etc/ssl/foreman-certs/key.pem"
  --certs-server-ca-cert "/etc/ssl/foreman-certs/ca.pem"

Generated proxy certs with this command

foreman-proxy-certs-generate
    --foreman-proxy-fqdn foreman-proxy.example.com
    --certs-tar /etc/ssl/smart-proxy/certs.tar
    --server-cert /etc/ssl/smart-proxy/cert.pem
    --server-key /etc/ssl/smart-proxy/key.pem
    --server-ca-cert /etc/ssl/smart-proxy/ca.pem

And ran the installation of the proxy using the provided install command:

foreman-installer\
                    --scenario foreman-proxy-content\
                    --certs-tar-file                                                "/root/certs.tar"\
                    --foreman-proxy-register-in-foreman            "true"\
                    --foreman-proxy-foreman-base-url               "https://foreman.example.com"\
                    --foreman-proxy-trusted-hosts                      "foreman.example.com"\
                    --foreman-proxy-trusted-hosts                      "foreman-proxy.example.com"\
                    --foreman-proxy-oauth-consumer-key          "REDACTED"\
                    --foreman-proxy-oauth-consumer-secret      "REDACTED"

After this the issue is still persisting. So I believe this has to be a bug.