Unknown CA thrown by Dynflow Core

Problem:
The dynflow core proxy rejects the certificate from the foreman-proxy with OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca

I used the smart proxy puppet module v7.1.0 to do the proxy setup, and communication from the master Foreman instance to foreman-proxy over https, port 8443, works as expected. It is from foreman-proxy over https, port 8008, to smart_proxy_dynflow_core that is broken.
Expected outcome:
Successful job submission to dynflow core
Foreman and Proxy versions:
1.17
Foreman and Proxy plugin versions:

  • Dynflow proxy plugin 0.1.10
  • Foreman remote execution 1.4.5

Other relevant data:
From what I can tell, foreman-proxy and smart_proxy_dynflow_core are both configured to use the system’s puppet certificate and CA.

Could you please post your /etc/foreman-proxy/settings.yml and /etc/smart_proxy_dynflow_core/settings.yml? Also output of curl -k https://localhost:8443/dynflow/tasks/count and parts of /var/log/foreman-proxy/smart_proxy_dynflow_core.log during the time when the curl was run would be helpful for moving this forward.

Absolutely, they are as follows:

Smart Proxy

# /etc/foreman-proxy/settings.yml
---
### File managed with puppet ###
## Module:           'foreman_proxy'

:settings_directory: /etc/foreman-proxy/settings.d

# SSL Setup

:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/aalvp003.company.com.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/aalvp003.company.com.pem

:trusted_hosts:
  - foreman_proxy_endpoint.dvn.com

:foreman_url: https://foreman.dvn.com

:foreman_ssl_ca: /etc/pki/ca-trust/source/anchors/Company_Root_CA.crt
#:foreman_ssl_cert: ssl/certs/fqdn.pem
#:foreman_ssl_key: ssl/private_keys/fqdn.pem

:daemon: true

:bind_host: '*'
:https_port: 8443
#:http_port: 8000
# Log configuration
:log_file: /var/log/foreman-proxy/proxy.log
:log_level: INFO

# Log buffer size and extra buffer size (for errors). Defaults to 3000 messages in total,
# which is about 500 kB request.
:log_buffer: 2000
:log_buffer_errors: 1000

Dynflow Core

# /etc/smart_proxy_dynflow_core/settings.yml
---
# Path to dynflow database, leave blank for in-memory non-persistent database
:database:
:console_auth: true

# URL of the foreman, used for reporting back
:foreman_url: https://foreman.company.com

# SSL settings for client authentication against foreman.
:foreman_ssl_ca: /etc/pki/ca-trust/source/anchors/Company_Root_CA.crt
#:foreman_ssl_certificate: ssl/certs/fqdn.pem
#:foreman_ssl_private_key: ssl/private_keys/fqdn.pem

# Listen on address
:listen: 0.0.0.0

# Listen on port
:port: 8008

:use_https: true
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/aalvp003.company.com.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/aalvp003.company.com.pem
# :ssl_ca_file: ssl/ca.pem
# :ssl_private_key: ssl/localhost.pem
# :ssl_certificate: ssl/certs/localhost.pem

# Log level, one of UNKNOWN, FATAL, ERROR, WARN, INFO, DEBUG
:log_level: DEBUG

Curl Log Response

E, [2018-05-02T17:04:29.113128 #5621] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca
        /opt/rh/rh-ruby24/root/usr/share/ruby/openssl/ssl.rb:382:in `accept'

That error is the same as when attempting to kick off a job. Setting the log to debug reveals the below as the client cert being presented:

 Data:
        Version: 3 (0x2)
        Serial Number: 4644 (0x1224)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: oilvp044.company.com
        Validity
            Not Before: Apr 19 21:06:29 2018 GMT
            Not After : Apr 19 21:06:29 2023 GMT
        Subject: CN=aalvp003.company.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

I trimmed off the rest, but that is the smart proxy’s Puppet client cert, and what I would think should be in use. I can post the rest of the certificate if it’ll help.

Have You managed to solve the issue?