Error during proxy install - "certificate verify failed (self signed certificate in certificate chain)"

Support
, ,
1

Problem:
During install of Smart Proxy w/ Content, I get the following error:

Error 1: Puppet Foreman_host resource 'foreman-proxy-gpnixfor02.ipa.medforest.org' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1134 of 1149)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org](provider=rest_v3)
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Error 2: Puppet Foreman_smartproxy resource 'gpnixfor02.ipa.medforest.org' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1136 of 1149)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
    Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_smartproxy[gpnixfor02.ipa.medforest.org](provider=rest_v3)
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2 errors were detected.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/foreman-proxy-content.log

Expected outcome:
Successful Smart Proxy installation

Foreman and Proxy versions:
3.11

Foreman and Proxy plugin versions:

Distribution and version:
RHEL 9.4

Other relevant data:

  1. Did a fresh install of Foreman 3.11 on RHEL 9:
  # foreman-installer --scenario katello \
  --certs-server-cert "/root/foreman_cert/gpnixfor01.pem" \
  --certs-server-key "/root/foreman_cert/gpnixfor01_key.pem" \
  --certs-server-ca-cert "/root/foreman_cert/ca_cert_bundle.pem" \
  --foreman-initial-organization "<Redacted>" \
  --foreman-initial-location "SDC" \
  --foreman-initial-admin-username foreman-admin \
  --foreman-initial-admin-password "<Redacted>" \
  --foreman-proxy-bmc "true" \
  --foreman-proxy-bmc-default-provider "freeipmi"
  • Command runs successfully. Foreman runs well and certs are working.

  1. Registered Smart Proxy host as a Foreman Client.

  2. Generated cert package tar for smart proxy:

# foreman-proxy-certs-generate \
--foreman-proxy-fqdn gpnixfor02.ipa.medforest.org \
--certs-tar /root/smart-proxy_cert/gpnixfor02.ipa.medforest.org-certs.tar

  1. Copied tar to Smart Proxy host.

  2. Ran installer on Proxy:

# foreman-installer \
--scenario foreman-proxy-content \
--certs-tar-file                              "/root/gpnixfor02.ipa.medforest.org-certs.tar" \
--foreman-proxy-register-in-foreman           "true" \
--foreman-proxy-foreman-base-url              "https://gpnixfor01.ipa.medforest.org" \
--foreman-proxy-trusted-hosts                 "gpnixfor01.ipa.medforest.org" \
--foreman-proxy-trusted-hosts                 "gpnixfor02.ipa.medforest.org" \
--foreman-proxy-oauth-consumer-key            "<redacted>" \
--foreman-proxy-oauth-consumer-secret         "<redacted>"
2024-08-02 22:53:06 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-08-02 22:53:10 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-08-02 22:53:10 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-08-02 22:54:15 [NOTICE] [configure] Starting system configuration.
2024-08-02 22:54:58 [NOTICE] [configure] 250 configuration steps out of 1120 steps complete.

2024-08-02 22:55:33 [NOTICE] [configure] 500 configuration steps out of 1122 steps complete.
2024-08-02 22:55:44 [NOTICE] [configure] 750 configuration steps out of 1147 steps complete.
2024-08-02 22:57:08 [NOTICE] [configure] 1000 configuration steps out of 1148 steps complete.
2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-02 22:57:29 [ERROR ] [configure] Wrapped exception:
2024-08-02 22:57:29 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-02 22:57:29 [ERROR ] [configure] Wrapped exception:
2024-08-02 22:57:29 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-02 22:57:29 [ERROR ] [configure] Wrapped exception:
2024-08-02 22:57:29 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-02 22:57:32 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Foreman_host resource 'foreman-proxy-gpnixfor02.ipa.medforest.org' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1134 of 1149)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org](provider=rest_v3)
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Error 2: Puppet Foreman_smartproxy resource 'gpnixfor02.ipa.medforest.org' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1136 of 1149)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
    Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_smartproxy[gpnixfor02.ipa.medforest.org](provider=rest_v3)
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2 errors were detected.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/foreman-proxy-content.log

Relevant info from /var/log/foreman-installer/foreman-proxy-content.log:

2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2024-08-02 22:57:29 [ERROR ] [configure] Wrapped exception:

2024-08-02 22:57:29 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

2024-08-02 22:57:29 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]: Evaluated in 0.01 seconds

2024-08-02 22:57:29 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Starting to evaluate the resource (1135 of 1149)

2024-08-02 22:57:29 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Collected {"features"=>["Templates", "Logs", "Registration", "Pulpcore", "Container_Gateway"]}

2024-08-02 22:57:29 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Selecting source_key features

2024-08-02 22:57:29 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Now setting field :features

2024-08-02 22:57:29 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Evaluated in 0.00 seconds

2024-08-02 22:57:29 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Starting to evaluate the resource (1136 of 1149)

2024-08-02 22:57:29 [DEBUG ] [configure] Foreman_smartproxy[gpnixfor02.ipa.medforest.org](provider=rest_v3): Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2024-08-02 22:57:29 [ERROR ] [configure] Wrapped exception:

2024-08-02 22:57:29 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

2024-08-02 22:57:29 [DEBUG ] [configure] Foreman_smartproxy[gpnixfor02.ipa.medforest.org](provider=rest_v3): Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2024-08-02 22:57:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2024-08-02 22:57:29 [ERROR ] [configure] Wrapped exception:

2024-08-02 22:57:29 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

I’ve used openssl to check the cert, key, and ca cert package that I provided.

Thank you for your time. Please let me know if I can provide any more info that might be helpful.

2

Any news? We have the same problems with all Smart Proxies during Katello upgrade from 4.12 to 4.13 (Foreman from 3.10 to 3.11).

3

You seem to use a custom certificate on the main server and default certificates on the proxy. That wouldn’t work because the proxy doesn’t know about the server chain.

See Installing a Smart Proxy Server 3.11 on Enterprise Linux

4

Thank you, I’ve changed that. Here’s the latest thing we’ve tried:

1. Confirm CA bundle on Foreman Server. Both of these commands are successful:

# openssl verify -CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem /root/smart-proxy_cert/gpnixfor02_cert.pem

openssl s_client --connect gpnixfor01.ipa.medforest.org:443 -CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem

2. Generate certs for Proxy:

[root@gpnixfor01 ~]# foreman-proxy-certs-generate \
--foreman-proxy-fqdn gpnixfor02.ipa.medforest.org \
--certs-tar ~/gpnixfor02.ipa.medforest.org-certs.tar \
--server-cert /root/smart-proxy_cert/gpnixfor02_cert.pem \
--server-key /root/smart-proxy_cert/gpnixfor02_key.pem \
--server-ca-cert /root/foreman_cert/medforest_ca_cert_bundle.pem
Preparing installation Done                                              
  Success!

3. SCP tar package to new Smart Proxy.

4. On Smart Proxy, run install commands:

yum -y --nogpgcheck localinstall http://gpnixfor01.ipa.medforest.org/pub/katello-ca-consumer-latest.noarch.rpm

Then:

foreman-installer \
--scenario foreman-proxy-content \
--certs-tar-file                              "/root/gpnixfor02.ipa.medforest.org-certs.tar" \
--foreman-proxy-register-in-foreman           "true" \
--foreman-proxy-foreman-base-url              "https://gpnixfor01.ipa.medforest.org" \
--foreman-proxy-trusted-hosts                 "gpnixfor01.ipa.medforest.org" \
--foreman-proxy-trusted-hosts                 "gpnixfor02.ipa.medforest.org" \
--foreman-proxy-oauth-consumer-key            "<Redacted>" \
--foreman-proxy-oauth-consumer-secret         "<Redacted>"

4. I still get this error:

2024-08-16 16:14:15 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-08-16 16:14:18 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-08-16 16:14:18 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-08-16 16:15:22 [NOTICE] [configure] Starting system configuration.
2024-08-16 16:16:01 [NOTICE] [configure] 250 configuration steps out of 1120 steps complete.
2024-08-16 16:16:34 [NOTICE] [configure] 500 configuration steps out of 1122 steps complete.
2024-08-16 16:16:43 [NOTICE] [configure] 750 configuration steps out of 1147 steps complete.
2024-08-16 16:18:09 [NOTICE] [configure] 1000 configuration steps out of 1148 steps complete.
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:34 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Foreman_host resource 'foreman-proxy-gpnixfor02.ipa.medforest.org' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1134 of 1149)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.01 seconds
  Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org](provider=rest_v3)
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Error 2: Puppet Foreman_smartproxy resource 'gpnixfor02.ipa.medforest.org' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1136 of 1149)
    Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
    Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
    Evaluated in 0.02 seconds
  Foreman_smartproxy[gpnixfor02.ipa.medforest.org](provider=rest_v3)
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
    Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22

2 errors were detected.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/foreman-proxy-content.log

- From /var/log/foreman-installer/foreman-proxy-content.log:

2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in 
certificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/hosts?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:31 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-gpnixfor02.ipa.medforest.org]: Evaluated in 0.01 seconds
2024-08-16 16:18:31 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Starting to evaluate the resource (1135 of 1149)
2024-08-16 16:18:31 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Collected {"features"=>["Templates", "Logs", "Registration", "Pulpcore", "Container_Gateway"]}
2024-08-16 16:18:31 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Selecting source_key features
2024-08-16 16:18:31 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Now setting field :features
2024-08-16 16:18:31 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Evaluated in 0.00 seconds
2024-08-16 16:18:31 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Starting to evaluate the resource (1136 of 1149)
2024-08-16 16:18:31 [DEBUG ] [configure] Foreman_smartproxy[gpnixfor02.ipa.medforest.org](provider=rest_v3): Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certific
ate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2024-08-16 16:18:31 [DEBUG ] [configure] Foreman_smartproxy[gpnixfor02.ipa.medforest.org](provider=rest_v3): Making get request to https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in cert
ificate chain) in get request to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[gpnixfor02.ipa.medforest.org]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) in get re
quest to: https://gpnixfor01.ipa.medforest.org/api/v2/smart_proxies?search=name%3D%22gpnixfor02.ipa.medforest.org%22
2024-08-16 16:18:31 [ERROR ] [configure] Wrapped exception:
2024-08-16 16:18:31 [ERROR ] [configure] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

5. Try to replicate the problem:

openssl s_client -connect gpnixfor01.ipa.medforest.org:443 -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

This command is succesful.

5

The katello-certs-check was successful when installing custom certs on the primary Foreman server:

[root@gpnixfor01 ~]# katello-certs-check \
-c /root/foreman_cert/gpnixfor01_cert.pem \
-k /root/foreman_cert/gpnixfor01_key.pem \
-b /root/foreman_cert/medforest_ca_cert_bundle.pem

This produced the install command we used for the Foreman server.

6

Not sure if this will help, but-- I’m trying to mimic the SSL verification process that might be causing the Foreman Proxy installer to fail.

This is a small Ruby script using the Net::HTTP library (which maybe the Foreman installer uses under the hood?).


require 'net/http'

require 'openssl'

# Foreman server URL

uri = URI.parse("https://gpnixfor01.ipa.medforest.org")

# New HTTP object

http = Net::HTTP.new(uri.host, uri.port)

http.use_ssl = true

# Path to your CA bundle

http.ca_file = "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"

# SSL verification mode

http.verify_mode = OpenSSL::SSL::VERIFY_PEER

# Enable detailed SSL debugging output

http.set_debug_output($stdout)

# Create HTTP request

request = Net::HTTP::Get.new(uri.request_uri)

# Execute request

begin

response = http.request(request)

puts "Response Code: #{response.code}"

puts "Response Body: #{response.body}"

rescue OpenSSL::SSL::SSLError => e

puts "SSL Error: #{e.message}"

end

This runs successfully, verifying the certs. And all the openssl commands we’ve tried in order to replicate the issue have all been successful.

7

Is this a completely installation or is this the installation where you have tried before?

Because if it is the latter I think you have to follow the instructions to renew/update the certificate on the smart proxy: Administering Foreman

This adds the --certs-update-server option to the generate and installer commands…

8

We’ve done a complete, from-scratch install (server, then proxy) 4 times now, with the same result.

We also did a successful install with the default certs (not custom) and it worked. Then, we added custom certs, following the docs you linked to, using --certs-update-server, and got the same ssl errors.

9

To be more clear:

We’ve done a complete, from-scratch install (server, then proxy) 4 times now. Every time we try to install custom certs on a smart proxy, we get the same result.

We also did a successful install with the default certs (not custom) and it was successful on the primary Foreman server. Then, we added custom certs, following the docs you linked to (using --certs-update-server). The custom certs worked fine when adding them to the primary Foreman server. But we got the same ssl error when trying to add them to the Smart Proxy.

10

I have double-checked with my original installation notes and I basically did the same when installing the proxy. I did it right from the start (not using default certs).

For the certs I have used the parameters generated by katello-certs-check, i.e. for the main server

# katello-certs-check -t foreman -c foreman.crt -k foreman.key -b foreman_interm.cer

for the proxy

# katello-certs-check -t foreman-proxy -c foreman-proxy.crt -k foreman-proxy.key -b foreman_interm.cer

sharing the same bundle foreman_interm.cer.

When using openssl to check verification, you must make sure that no default CAfile or CApath are loaded! Thus your command

openssl verify -CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem /root/smart-proxy_cert/gpnixfor02_cert.pem

may succeed because the verification uses the the certs from the default CA path. Thus you have to run

openssl verify -no-CApath -CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem /root/smart-proxy_cert/gpnixfor02_cert.pem

instead to really verify the cert to the given bundle only, or

openssl s_client -no-CApath -connect gpnixfor01.ipa.medforest.org:443 -CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem
...
    Verify return code: 19 (self signed certificate in certificate chain)

to see the verification error during connect. I think foreman does not use the default path nor default file. It uses ca pinning only accepting the CA certs in the bundle file.

Does /root/foreman_cert/medforest_ca_cert_bundle.pem contain the full chain for your certs including the root ca certificate?

11

Thank you for responding.

Both commands are successful.

SSL handshake has read 9041 bytes and written 442 bytes

Verification: OK

openssl s_client \
-no-CApath \
-connect gpnixfor01.ipa.medforest.org:443 \
-CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem

---
SSL handshake has read 9041 bytes and written 442 bytes
Verification: OK
---
12 
[root@gpnixfor01 ~]# openssl verify -CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem /root/smart-proxy_cert/gpnixfor02_cert.pem
/root/smart-proxy_cert/gpnixfor02_cert.pem: OK

openssl s_client \
-no-CApath \
-connect gpnixfor01.ipa.medforest.org:443 \
-CAfile /root/foreman_cert/medforest_ca_cert_bundle.pem

---
SSL handshake has read 9041 bytes and written 442 bytes
Verification: OK
---
13

It has an intermediate cert and root cert (in that order).

14

Hmm, the only thing I could think of here would be to try to add the --certs-update-all option instead of the other --certs-update... option to the commands. That should set all certs as required.

I guess one of the devs must help out here and explain, which certs and CAs are exactly used for /Stage[main]/Foreman_proxy::Register/Foreman_host. You have everything the way I have. It works fine for me that way and I have renewed those certs that way a couple of time. So it’s maybe an issue with the current installer.

I know there may be issues if there are still old files in the file systems and you don’t start on a new server with a new, fresh base os. But on a freshly installed base os it should work.

1 Like
15

I suspect the issue is related to how Foreman is creating /etc/foreman-proxy/foreman_ssl_ca.pem. I was able to complete a Foreman Proxy 3.11 install w/ custom certs by doing the following:

  1. Ran this command, which is unsuccessful (I got the error that says I have self-signed certs, which is not the case):
foreman-installer \
--scenario foreman-proxy-content \
--certs-tar-file                              "/root/myforemanproxy-certs.tar" \
--foreman-proxy-register-in-foreman           "true" \
--foreman-proxy-foreman-base-url              "https://myforemanserver.example.org" \
--foreman-proxy-trusted-hosts                 "https://myforemanproxy.example.org" \
--foreman-proxy-trusted-hosts                 "myforemanproxy.example.org" \
--foreman-proxy-oauth-consumer-key            "<Redacted>" \
--foreman-proxy-oauth-consumer-secret         "<Redacted>"

  1. Manually placed my CA cert bundle (for our local CA) at /root/ssl-build/local-ca_cert_bundle.pem.

  2. Edited /etc/foreman-installer/scenarios.d/foreman-proxy-content-answers.yaml and set the following:

server_ca_cert: "/root/ssl-build/local-ca_cert_bundle.pem"
  1. Reran the installer:
foreman-installer --scenario foreman-proxy-content
  1. The proxy successfully completed installation. :smiley: