SmartProxy "SSLCertificateNotVerified" Error Upon Installation

Hi all, hope everyone is doing OK.

Problem: Attempting to install a SmartProxy on an end-point using the below command:

foreman-installer \
                  --scenario foreman-proxy-content \
                  --certs-tar-file                              "/opt/OD/data/myhost.example.net-certs.tar"\
                  --foreman-proxy-content-parent-fqdn           "foreman_server.example.net"\
                  --foreman-proxy-register-in-foreman           "true"\
                  --foreman-proxy-foreman-base-url              "https://foreman_server.example.net"\
                  --foreman-proxy-trusted-hosts                 "foreman_server.example.net"\
                  --foreman-proxy-trusted-hosts                 "myhost.example.net"\
                  --foreman-proxy-oauth-consumer-key            "fu5t4tsDy4nVAwbxaiqEwbnynvQ3VfnB"\
                  --foreman-proxy-oauth-consumer-secret         "mhq8EgVDYmZfUC2i4obK2e5aoNjiN72o"\
                  --puppet-server-foreman-url                   "https://foreman_server.example.net"\
                  --foreman-proxy-tftp							"true"

After executing, I get the below error message:

Proxy myhost.example.net cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://myhost.example.net:9090/v2/features Please check the proxy is configured and running on the host.
[ERROR 2022-03-21T17:49:47 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[ods00712w10.odretail.net]/ensure: change from 'absent' to 'present' failed: Proxy myhost.example.net cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)) for proxy https://myhost.example.net:9090/v2/features Please check the proxy is configured and running on the host.

Note I was able to successfully generate the certs tar file from the Foreman server.

I do have custom SSL certificates installed on my Foreman/Katello server, which I suppose may have something to do with this issue?

Expected outcome: SmartProxy is successfully installed.

Foreman and Proxy versions: Foreman 2.2.3

Foreman and Proxy plugin versions: Katello 3.17.3

Distribution and version: CentOS 7

Other relevant data:

Thanks!

Just experimenting a bit to see if I can get this working.

So on my Katello server, I have my signed SSL certificate files that it is using. Here are the three files, renamed for simplicity:

  • cert.crt
  • key.key
  • cacert.crt

I can use katello-certs-check to validate these are kosher:

katello-certs-check -c ./cert.crt -k ./key.key -b ./cacert.crt

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[OK]

Checking CA bundle size:
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

Validation succeeded

OK, cool! So are these what I need when setting up my Smart Proxy? I copied them to the Smart Proxy system and attempted to reference them in foreman-installer, as shown below:

foreman-installer \
                  --scenario foreman-proxy-content \
                  --certs-tar-file                              "/opt/OD/data/myhost.example.net-certs.tar"\
                  --foreman-proxy-content-parent-fqdn           "foreman_server.example.net"\
                  --foreman-proxy-register-in-foreman           "true"\
                  --foreman-proxy-foreman-base-url              "https://foreman_server.example.net"\
                  --foreman-proxy-trusted-hosts                 "foreman_server.example.net"\
                  --foreman-proxy-trusted-hosts                 "myhost.example.net"\
                  --foreman-proxy-oauth-consumer-key            "fu5t4tsDy4nVAwbxaiqEwbnynvQ3VfnB"\
                  --foreman-proxy-oauth-consumer-secret         "mhq8EgVDYmZfUC2i4obK2e5aoNjiN72o"\
                  --puppet-server-foreman-url                   "https://foreman_server.example.net"\
                  --foreman-proxy-foreman-ssl-ca                "./cacert.crt" \
                  --foreman-proxy-foreman-ssl-cert              "./cert.crt" \
                  --foreman-proxy-foreman-ssl-key               "./key.key" \
                  --foreman-proxy-tftp							"true"

However, I get the same error. I also noticed there are similar options, which I tried swapping out and adding to the foreman-installer command:

--foreman-proxy-ssl-ca "./cacert.crt" \
--foreman-proxy-ssl-cert "./cert.crt" \
--foreman-proxy-ssl-key "./key.key" \

Also, this results in the same error.

I feel like I’m overlooking something silly. Any tips would be greatly appreciated. Thanks!

I think you have to pass the certs to foreman-proxy-certs-generate and then use the generated tar which includes the certificates. This is how I have generated the tar for my content proxy:

foreman-proxy-certs-generate --foreman-proxy-fqdn "foreman-content.example.com" \
       --certs-tar "~/foreman-content.example.com-certs.tar" \
       --server-cert /root/foreman-content.crt \
       --server-key /root/foreman-content.key \
       --server-ca-cert /etc/pki/tls/certs/ca-bundle-noroot.crt

(ca-bundle-noroot.crt contains the ca chain for the certificate excluding the root ca which is a public ca)

Ah, that would make perfect sense!

Sadly, I’m not having much luck after trying that. I’m 99.99% sure it is something stupid I’m doing… I’m not great with configuring SSL and admittedly stumbled through getting the signed SSL certs working with the Katello server to begin with.

When I originally setup the signed SSL certs, I just needed these three files that I mentioned above:

  • cert.crt being the certificate provided to me by DigiCert after sending in my CSR.
  • key.key would be the key generated when I created my CSR using the openssl command.
  • cacert.crt if I recall correctly was built by concatenating the below:
    • The intermediate certificate DigiCertCA.crt provided by DigiCert.
    • The root certificate TrustedRoot.crt provided by DigiCert.

I’d imagine my issue is likely regarding the server-ca-cert, which you mentioned is the CA chain for the certificate excluding the root (public) CA. Wouldn’t this just be the intermediate certificate DigiCertCA.crt I mentioned above?

I successfully built the cert tar doing just that, copied it to my destined Smart Proxy, but end up with the same error after executing the install command.

Sorry, I know this isn’t a support forum for basic SSL stuff, but any guidance you can offer would be awesome.

So you ran foreman-proxy-certs-generate on the main server, then copied the tar to the proxy and used the command printed by foreman-proxy-certs-generate and still got the same error?

You can check if the certificate fits the ca cert(s) with openssl:

# openssl verify -CApath /dev/null -CAfile /etc/pki/tls/certs/ca-bundle.crt foreman.crt 
foreman.crt: OK

ca-bundle.crt would be your cacert.crt file, which must contain the intermediates and the root for verification to succeed.

(The root ca isn’t necessary later if it’s a public ca which is included in the root bundle of the system, but it causes no problems if it’s included anyway)

Yeah, I did exactly that.

Testing with the openssl command:

# openssl verify -CApath /dev/null -CAfile ./cacert.crt cert.crt
cert.crt: OK

Here is the command I ran to generate the tar:

foreman-proxy-certs-generate --foreman-proxy-fqdn "smartproxy.example.net" \
       --certs-tar "/root/smartproxy.example.net-certs.tar" \
       --server-cert "/root/cert.crt" \
       --server-key "/root/key.key" \
       --server-ca-cert "/root/cacert.crt"

And the output:

Preparing installation Done
  Success!

  To finish the installation, follow these steps:

  If you do not have the Smart Proxy registered to the Katello instance, then please do the following:

  1. yum -y localinstall http://katello_server.example.net/pub/katello-ca-consumer-latest.noarch.rpm
  2. subscription-manager register --org "myOrg"

  Once this is completed run the steps below to start the Smart Proxy installation:

  1. Ensure that the foreman-installer-katello package is installed on the system.
  2. Copy the following file /root/smartproxy.example.net-certs.tar to the system ods00712w10.odretail.net at the following location /root/smartproxy.example.net-certs.tar
  scp /tmp/smartproxy.example.net-certs.tar root@smartproxy.example.net:/root/smartproxy.example.net-certs.tar
  3. Run the following commands on the Smart Proxy (possibly with the customized
     parameters, see foreman-installer --scenario foreman-proxy-content --help and
     documentation for more info on setting up additional services):

  foreman-installer \
                    --scenario foreman-proxy-content \
                    --certs-tar-file                              "/root/smartproxy.example.net-certs.tar"\
                    --foreman-proxy-content-parent-fqdn           "katello_server.example.net"\
                    --foreman-proxy-register-in-foreman           "true"\
                    --foreman-proxy-foreman-base-url              "https://katello_server.example.net"\
                    --foreman-proxy-trusted-hosts                 "katello_server.example.nett"\
                    --foreman-proxy-trusted-hosts                 "smartproxy.example.net"\
                    --foreman-proxy-oauth-consumer-key            "..."\
                    --foreman-proxy-oauth-consumer-secret         "..."\
                    --puppet-server-foreman-url                   "https://katello_server.example.net"

I then copy the tar file to smartproxy.example.net and run the command above. Unfortunately, the same error occurs.

If I open up the tar, this is what I find inside:

.:
total 4.0K
   0 drwxr-xr-x   3 root root   23 Mar 22 14:46 ./
4.0K dr-xr-x---. 17 root root 4.0K Mar 22 14:46 ../
   0 drwxr-xr-x   3 root root  207 Mar 22 14:46 ssl-build/

./ssl-build:
total 28K
   0 drwxr-xr-x 3 root root  207 Mar 22 14:46 ./
   0 drwxr-xr-x 3 root root   23 Mar 22 14:46 ../
8.0K -rw-r--r-- 1 root root 4.9K Oct 27  2020 katello-default-ca-1.0-1.noarch.rpm
8.0K -rw-r--r-- 1 root root 4.9K Oct 27  2020 katello-server-ca-1.0-1.noarch.rpm
4.0K -rw-r--r-- 1 root root 4.0K Sep  9  2021 katello-server-ca-1.0-2.noarch.rpm
4.0K -rw-r--r-- 1 root root 4.0K Sep  9  2021 katello-server-ca-1.0-3.noarch.rpm
4.0K drwxr-xr-x 2 root root 4.0K Mar 22 14:46 smartproxy.example.net/

./ssl-build/smartproxy.example.net:
total 112K
4.0K drwxr-xr-x 2 root root 4.0K Mar 22 14:46 ./
   0 drwxr-xr-x 3 root root  207 Mar 22 14:46 ../
 12K -rw------- 1 root root  11K Feb  7  2021 smartproxy.example.net-apache-1.0-1.noarch.rpm
 12K -rw------- 1 root root  12K Feb  7  2021 smartproxy.example.net-foreman-client-1.0-1.noarch.rpm
 12K -rw------- 1 root root  12K Feb  7  2021 smartproxy.example.net-foreman-proxy-1.0-1.noarch.rpm
 12K -rw------- 1 root root  11K Feb  7  2021 smartproxy.example.net-foreman-proxy-client-1.0-1.noarch.rpm
 12K -rw------- 1 root root  12K Feb  7  2021 smartproxy.example.net-puppet-client-1.0-1.noarch.rpm
 12K -rw------- 1 root root  12K Feb  7  2021 smartproxy.example.net-qpid-broker-1.0-1.noarch.rpm
 12K -rw------- 1 root root  12K Feb  7  2021 smartproxy.example.net-qpid-client-cert-1.0-1.noarch.rpm
 12K -rw------- 1 root root  11K Feb  7  2021 smartproxy.example.net-qpid-router-client-1.0-1.noarch.rpm
 12K -rw------- 1 root root  11K Feb  7  2021 smartproxy.example.net-qpid-router-server-1.0-1.noarch.rpm

I haven’t attempted unpacking any of these RPMs yet, but I suppose my certificate information should be within them? And if so, I would expect the date of the RPMs to be today?

I am probably way off, but just something I noticed…

Thank you so much for your help so far…