Issues setting up new foreman-proxy instances in remote sites after switching SSL cert to internal CA on foreman master

Support
1

Hi,

Problem:
I was able to change the web interface certificate to our internal PKI issued certs in the main foreman/puppet6 instance following user boffin’s outline in the comments here (thank you boffin).
https://theforeman.org/2015/11/foreman-ssl.html

I am now having issues when trying to setup a new foreman-proxy/puppet instance in remote sites to connect to the main foreman instance.

The foreman-proxy instance that lives on the main foreman box works fine, its only new instances I try to setup in remote locations. Below is the error that I see ( appears a few times ) in the resulting log file after running the command at the bottom of this message.

/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[hostname.fqdn]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://foreman.domain.com/api/v2/smart_proxies?search=name="hostname.fqdn"

Expected outcome:
foreman-installer should complete and register the new proxy inside the main foreman instance.

Foreman and Proxy versions:
foreman version 1.24.3-2
foreman-proxy version 1.24.3-1
Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:
below is most of the various arguments i have tried to get this to complete successfully, but no matter which i use/remove, i always seem to get the same error above.

I have also tried various ca-bundle.crt combinations for all the different ca values below. Mashing both the internal PKI cert chain, the ca_crt.pem from the foreman master, etc… Nothing seems to work.

foreman-installer
–no-enable-foreman
–no-enable-foreman-plugin-bootdisk
–no-enable-foreman-plugin-setup
–puppet-agent-additional-settings=server:foreman.domain.com
–enable-puppet
–puppet-server-ca=false
–puppet-server-foreman-url=https://foreman.domain.com
–puppet-server-foreman=true
–puppet-server-foreman-ssl-ca=/etc/pki/tls/certs/ca-bundle.crt
–enable-foreman-proxy
–foreman-proxy-puppetca=false
–foreman-proxy-tftp=false
–foreman-proxy-foreman-ssl-ca=/etc/pki/tls/certs/ca-bundle.crt
–foreman-proxy-foreman-ssl-cert=/etc/puppetlabs/puppet/ssl/certs/remproxy.domain.com.pem
–foreman-proxy-foreman-ssl-key=/etc/puppetlabs/puppet/ssl/private_keys/remproxy.domain.com.pem
–foreman-proxy-register-in-foreman=true
–foreman-proxy-trusted-hosts=foreman.domain.com
–foreman-proxy-trusted-hosts=remproxy.domain.com
–foreman-proxy-foreman-base-url=https://foreman.domain.com
–foreman-proxy-oauth-consumer-key=xxxxxxxxxxxxx
–foreman-proxy-oauth-consumer-secret=xxxxxxxxxxx

Any help is welcomed and appreciated.

Thanks

2

Inside the file /etc/foreman-proxy/settings.yml inspect the certificate declared over there under the key: :foreman_ssl_ca:

This certificate is a chain that must include:

  • Your katello ca cert
  • Your custom cert Root CA, for instance if I bought a cert from Digicert, then I can use the Root CA from their side which signed my CSR
  • Your Puppet CA cert

So, in your scenario, I would say that you can modify /etc/pki/tls/certs/ca-bundle.crt to include all 3 CAs. Its just an idea.

1 Like
3

Thanks for the suggestion. I have verified that the cert chain does already have the Puppet CA cert and my internal CA chain mashed together. Im not using katello at all so that doesnt apply. Still not able to get this working right.

Has anyone been able to get this working? Seems like its something most people would want and shouldnt be too difficult to setup the ca. Just not finding any docs on how to make it happen other than what I mentioned earlier from 2015.

4

Can you paste the stack trace from?
/var/log/foreman-proxy/proxy.log

6

It wont let me upload a file since im a ‘new user’. Heres the contents of that proxy.log. I had to scrub my company’s info so the certs show as and the DNS names are generic. Thanks again.

2020-11-05T16:11:52  [I] Successfully initialized 'foreman_proxy'
2020-11-05T16:11:52  [W] Missing SSL setup, https is disabled.
2020-11-05T16:11:52  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2020-11-05T16:13:37  [I] Successfully initialized 'foreman_proxy'
2020-11-05T16:13:37  [I] Started puppet class cache initialization
2020-11-05T16:13:37  [I] Successfully initialized 'puppet_proxy_puppet_api'
2020-11-05T16:13:37  [I] Successfully initialized 'puppet'
2020-11-05T16:13:37  [I] Successfully initialized 'logs'
2020-11-05T16:13:37  [I] WEBrick 1.4.2
2020-11-05T16:13:37  [I] ruby 2.5.1 (2018-03-29) [x86_64-linux-gnu]
2020-11-05T16:13:37  [I] 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 15 (0xf)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: foreman
        Validity
            Not Before: Nov  4 16:57:46 2020 GMT
            Not After : Nov  4 16:57:46 2025 GMT
        Subject: CN=newproxy.domain.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
      <omitted>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Comment: 
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier: 
                keyid:73:6E:FB:5E:D7:7C:15:17:30:40:E1:FD:54:5A:60:4F:64:4C:F1:EA

            X509v3 Subject Key Identifier: 
                CA:56:56:27:1A:CF:E3:2D:A1:38:7B:04:81:D3:2D:AD:4B:57:42:61
            X509v3 Subject Alternative Name: 
                DNS:puppet, DNS:puppet.domain.com, DNS:newproxy.domain.com
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
     <omitted>

2020-11-05T16:13:37  [I] WEBrick::HTTPServer#start: pid=10575 port=8443
2020-11-05T16:13:37  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2020-11-05T16:13:39  [I] Finished puppet class cache initialization
2020-11-05T16:32:07  [I] going to shutdown ...
2020-11-05T16:32:07  [I] WEBrick::HTTPServer#start done.
2020-11-05T16:32:07  [I] Successfully initialized 'foreman_proxy'
2020-11-05T16:32:07  [I] Started puppet class cache initialization
2020-11-05T16:32:07  [I] Successfully initialized 'puppet_proxy_puppet_api'
2020-11-05T16:32:07  [I] Successfully initialized 'puppet'
2020-11-05T16:32:07  [I] Successfully initialized 'logs'
2020-11-05T16:32:07  [I] WEBrick 1.4.2
2020-11-05T16:32:07  [I] ruby 2.5.1 (2018-03-29) [x86_64-linux-gnu]
2020-11-05T16:32:07  [I] 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 15 (0xf)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: foreman
        Validity
            Not Before: Nov  4 16:57:46 2020 GMT
            Not After : Nov  4 16:57:46 2025 GMT
        Subject: CN=newproxy.domain.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
      <omitted>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Comment: 
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier: 
                keyid:73:6E:FB:5E:D7:7C:15:17:30:40:E1:FD:54:5A:60:4F:64:4C:F1:EA

            X509v3 Subject Key Identifier: 
                CA:56:56:27:1A:CF:E3:2D:A1:38:7B:04:81:D3:2D:AD:4B:57:42:61
            X509v3 Subject Alternative Name: 
                DNS:puppet, DNS:puppet.domain.com, DNS:newproxy.domain.com
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
     <omitted>

2020-11-05T16:32:07  [I] WEBrick::HTTPServer#start: pid=14392 port=8443
2020-11-05T16:32:07  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2020-11-05T16:32:09  [I] Finished puppet class cache initialization
2020-11-06T12:01:15  [E] <OpenSSL::SSL::SSLError> SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization
    /usr/lib/ruby/2.5.0/webrick/server.rb:299:in `accept'
    /usr/lib/ruby/2.5.0/webrick/server.rb:299:in `block (2 levels) in start_thread'
    /usr/lib/ruby/2.5.0/webrick/utils.rb:263:in `timeout'
    /usr/lib/ruby/2.5.0/webrick/server.rb:297:in `block in start_thread'
    /usr/lib/ruby/vendor_ruby/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-11-06T12:01:17 c92f98e1 [I] Started GET / 
2020-11-06T12:01:17 c92f98e1 [I] Finished GET / with 404 (8.58 ms)
2020-11-06T12:01:17 47c409b3 [I] Started GET /ui/ 
2020-11-06T12:01:17 47c409b3 [I] Finished GET /ui/ with 404 (0.21 ms)
2020-11-06T12:47:00  [I] going to shutdown ...
2020-11-06T12:47:00  [I] WEBrick::HTTPServer#start done.
2020-11-06T12:47:00  [I] Successfully initialized 'foreman_proxy'
2020-11-06T12:47:00  [I] Started puppet class cache initialization
2020-11-06T12:47:00  [I] Successfully initialized 'puppet_proxy_puppet_api'
2020-11-06T12:47:00  [I] Successfully initialized 'puppet'
2020-11-06T12:47:00  [I] Successfully initialized 'logs'
2020-11-06T12:47:00  [I] WEBrick 1.4.2
2020-11-06T12:47:00  [I] ruby 2.5.1 (2018-03-29) [x86_64-linux-gnu]
2020-11-06T12:47:00  [I] 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 15 (0xf)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: foreman
        Validity
            Not Before: Nov  4 16:57:46 2020 GMT
            Not After : Nov  4 16:57:46 2025 GMT
        Subject: CN=newproxy.domain.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
     <omitted>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Comment: 
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier: 
                keyid:73:6E:FB:5E:D7:7C:15:17:30:40:E1:FD:54:5A:60:4F:64:4C:F1:EA

            X509v3 Subject Key Identifier: 
                CA:56:56:27:1A:CF:E3:2D:A1:38:7B:04:81:D3:2D:AD:4B:57:42:61
            X509v3 Subject Alternative Name: 
                DNS:puppet, DNS:puppet.domain.com, DNS:newproxy.domain.com
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
     <omitted>

2020-11-06T12:47:00  [I] WEBrick::HTTPServer#start: pid=4778 port=8443
2020-11-06T12:47:00  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2020-11-06T12:47:01  [I] Finished puppet class cache initialization
2020-11-06T12:57:06  [I] going to shutdown ...
2020-11-06T12:57:06  [I] WEBrick::HTTPServer#start done.
2020-11-06T12:57:06  [I] Successfully initialized 'foreman_proxy'
2020-11-06T12:57:06  [I] Started puppet class cache initialization
2020-11-06T12:57:06  [I] Successfully initialized 'puppet_proxy_puppet_api'
2020-11-06T12:57:06  [I] Successfully initialized 'puppet'
2020-11-06T12:57:06  [I] Successfully initialized 'logs'
2020-11-06T12:57:06  [I] WEBrick 1.4.2
2020-11-06T12:57:06  [I] ruby 2.5.1 (2018-03-29) [x86_64-linux-gnu]
2020-11-06T12:57:06  [I] 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 15 (0xf)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: foreman
        Validity
            Not Before: Nov  4 16:57:46 2020 GMT
            Not After : Nov  4 16:57:46 2025 GMT
        Subject: CN=newproxy.domain.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
     <omitted>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Comment: 
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier: 
                keyid:73:6E:FB:5E:D7:7C:15:17:30:40:E1:FD:54:5A:60:4F:64:4C:F1:EA

            X509v3 Subject Key Identifier: 
                CA:56:56:27:1A:CF:E3:2D:A1:38:7B:04:81:D3:2D:AD:4B:57:42:61
            X509v3 Subject Alternative Name: 
                DNS:puppet, DNS:puppet.domain.com, DNS:newproxy.domain.com
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
    <omitted>

2020-11-06T12:57:06  [I] WEBrick::HTTPServer#start: pid=6748 port=8443
2020-11-06T12:57:06  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2020-11-06T12:57:08  [I] Finished puppet class cache initialization