Custom tls certifcate break smart-proxy

Support
,
1

Problem:

Hello,

after deploying a custom certificate via the following command:

foreman-installer --scenario katello \
                      --certs-server-cert "/root/foreman/forman_info.pem" \
                      --certs-server-key "/root/foreman/foreman_key.pem" \
                      --certs-server-ca-cert "/root/foreman/foreman_info-bundle.pem" \
                      --certs-update-server --certs-update-server-ca

i have an issue with communication with the smart-proxy.

The katello-certs-ckeck successfully validated the certificate and the webui is working fine.

Expected outcome:

Communication with the smart-proxy should works

Foreman and Proxy versions:

Foreman 3.19.1

Foreman and Proxy plugin versions:

foreman-tasks 12.1.1

foreman_remote_execution 16.6.4

katello 4.21.0

Distribution and version:

Red Hat Enterprise Linux 9.8

Other relevant data:

foreman-rake console:
Loading production environment (Rails 7.0.10)
irb(main):001:0> roxy.all.each do |smart_proxy|
ForemanTasks.sync_task(Actions::Pulp3::ContentGuard::Refresh,
irb(main):001:1* SmartProxy.all.each do |smart_proxy|
irb(main):002:1* ForemanTasks.sync_task(Actions::Pulp3::ContentGuard::Refresh, smart_proxy)
irb(main):003:0> end
/usr/share/gems/gems/foreman-tasks-12.1.1/lib/foreman_tasks.rb:56:in block in sync_task': Task 6eb1be62-1e28-49cf-91f7-b919301f3134: Faraday::SSLError: SSL_read: tlsv1 alert unknown ca (ForemanTasks::TaskError) from /usr/share/gems/gems/foreman-tasks-12.1.1/lib/foreman_tasks.rb:55:in sync_task’

/var/log/foreman/production.log:
2026-08-26T14:57:23 [I|app|] Everything ready for world: fee6e5af-1654-4bac-acdd-c4eed7ea6206
2026-08-26T14:57:23 [I|dyn|] Performing validity checks
2026-08-26T14:57:23 [I|dyn|] Finished performing validity checks
2026-08-26T15:00:32 [I|app|] Rails cache backend: Redis
2026-08-26T15:00:35 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
2026-08-26T15:00:37 [W|app|] You are trying to replace view_smart_proxies from katello. Adding allowed actions from plugin permissions to the existing one.
2026-08-26T15:00:38 [I|app|] Registered app metadata for plugin katello
2026-08-26T15:00:43 [W|app|] Console started with ‘foreman_console_admin’ user, call User.current= to change it
2026-08-26T15:00:46 [I|bac|] Task {label: , execution_plan_id: 25ce4027-e11a-4507-be82-b7dd3223d7e6} state changed: pending
2026-08-26T15:00:46 [I|bac|] Task {label: Actions::Pulp3::ContentGuard::Refresh, id: 6eb1be62-1e28-49cf-91f7-b919301f3134, execution_plan_id: 25ce4027-e11a-4507-be82-b7dd3223d7e6} state changed: planning
2026-08-26T15:00:47 [I|bac|] Task {label: Actions::Pulp3::ContentGuard::Refresh, id: 6eb1be62-1e28-49cf-91f7-b919301f3134, execution_plan_id: 25ce4027-e11a-4507-be82-b7dd3223d7e6} state changed: planned
2026-08-26T15:00:47 [I|bac|] Task {label: Actions::Pulp3::ContentGuard::Refresh, id: 6eb1be62-1e28-49cf-91f7-b919301f3134, execution_plan_id: 25ce4027-e11a-4507-be82-b7dd3223d7e6} state changed: running
2026-08-26T15:00:47 [E|bac|] SSL_read: tlsv1 alert unknown ca (Faraday::SSLError)

2

Hi,

Please note that you do not just configure the certificates directly on the Smart Proxy. You need to generate them on the Foreman server first. That will produce a tar file and the exact foreman-installer command you need to run on the proxy.

Did you follow the documentation?
→ Installing a Smart Proxy Server 3.19 on Enterprise Linux

3

Hello,

oh sorry, forgot to mention that it’s a foreman katello installation with the default smart-proxy.

i followed this procedure Installing Foreman Server with Katello 4.21 plugin on Enterprise Linux

i thought that il will just replace the certificate for the web ui but leave the default internal ca for the proxy.

4

That’s the installation guide. From what you write it sounds like you have a running foreman server and you are trying to replace the certificates. That would be the renewal as described in

Does the CA cert bundle include the root certificate?

5

yes, as posted in the op. i ran

foreman-installer --scenario katello \
--certs-server-cert "/root/foreman/forman_info.pem" \
--certs-server-key "/root/foreman/foreman_key.pem" \
--certs-server-ca-cert "/root/foreman/foreman_info-bundle.pem" \
--certs-update-server --certs-update-server-ca

the cert bundle contain the intermediate ca on top of the root ca

6

O.K. Did you read the section? The command is only one step.

What is the output of

# openssl verify -no-CAfile -no-CApath -no-CAstore -CAfile /root/foreman/foreman_info-bundle.pem /root/foreman/forman_info.pem
7

Hello,

here is the result:
/root/foreman/foreman_info.pem: OK

8

O.K. Then check which certificate is presented by the proxy:

$ openssl s_client -connect foreman.example.com:9090

This shows you subject/issuers of the proxy server certificate and chain.

Also check whether /etc/foreman-proxy/ssl_cert.pem contains the server certificate from foreman_info.pem and /etc/foreman-proxy/settings.yml contains

:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem
9

the certificate presented by the proxy return the complete certificate chain.

here is the content of settings.yml

[root@foreman foreman-proxy]# grep -i ssl_ settings.yml
:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem
#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]
:foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
:foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem
# ssl_certificate, ssl_ca_file, and ssl_private_key correspondingly

but i found something strange with these certificates, the ca and cert are swapped between ssl_ca.pem and foreman_ssl_ca:

[root@foreman foreman-proxy]# openssl x509 -in ssl_cert.pem -noout -subject -issuer
subject=O=MyCorp, OU=IT, CN=foreman.info
issuer=O=MyCorp, CN=Applications CA
[root@foreman foreman-proxy]# openssl x509 -in ssl_ca.pem -noout -subject -issuer
subject=C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman.info
issuer=C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman.info
[root@foreman foreman-proxy]# openssl x509 -in foreman_ssl_cert.pem -noout -subject -issuer
subject=C=US, ST=North Carolina, O=FOREMAN, OU=FOREMAN_PROXY, CN=foreman.info
issuer=C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman.info
[root@foreman foreman-proxy]# openssl x509 -in foreman_ssl_ca.pem -noout -subject -issuer
subject=O=MyCorp, CN=Applications CA
issuer=O=MyCorp, CN=Root CA
10

I have only asked whether it returned the correct server certificate. Is it?

It should not return the complete chain. The server certificate should not contain the chain. The proxy on port 9090 only uses the server certificate (unlike public tls servers). The client side checks against the full ca chain (which is why the root certificate must be included).

openssl can show you all the certificates which the server delivers:

$ openssl s_client -connect foreman.example.com:9090 -showcerts

It should only show you the server certificate as PEM, no chain certs.

So the hostname of the foreman server is foreman.info and the Applications CA is your issuing ca.

That’s the internal foreman ca for client authentication between foreman services, proxies, etc.

That is the client certificate issued for this smart proxy by the internal foreman ca.

This file should contain multiple certificates. It should include the Root CA certificate. That’s the ca chain checked when the smart proxy contacts the main server. Please make sure that this file contains the full chain including the root ca certificate.

So far, everything looks normal. Nothing is swapped. the ssl_*.pem files are for the smart proxy server configuration. The foreman_ssl_*.pem files are the smart proxy client configuration. That’s why it may seem swapped. But the server side has to check the client certificates from the connecting main server and the client side has to check the server certificate of the main server…

I think from the logs it looks as if the issue is the main server connecting to the smart proxy. Can you check /etc/foreman/proxy_ca.pem? It should contain the internal ca certificate from ssl_ca.pem above followed by the full chain including your root ca certificate.

11

Hello,

regarding

openssl s_client -connect foreman.example.com:9090 -showcerts

i misinterpreted the result, indeed, it show the custom server certificate.

this the content of /etc/foreman-proxy/ssl_ca.pem:

openssl crl2pkcs7 -nocrl -certfile /etc/foreman-proxy/ssl_ca.pem | openssl pkcs7 -print_certs -text | grep -E ‘Subject:|Issuer:’
Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman.info
Subject: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman.info

and this is the content of /etc/foreman/proxy_ca.pem:
openssl crl2pkcs7 -nocrl -certfile /etc/foreman/proxy_ca.pem | openssl pkcs7 -print_certs -text | grep -E ‘Subject:|Issuer:’
Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman.info
Subject: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=foreman.info
Issuer: O=MyCorp, CN=EP Root CA
Subject: O=MyCorp, CN=EP Applications CA
Issuer: O=MyCorp, CN=EP Root CA
Subject: O=MyCorp, CN=EP Root CA