Help getting forman and foreman-proxy to communicate using custom CA and Intermediate certificate

All:

I am trying to replace foreman's built in CA with our in house CA which
includes an Intermediate certificate and am having no luck.

What works::

   - invoking https://foreman.awt.local as a local user or as a user on
   another system works - I can load the foreman web client.
   - Invoking https://forman.awt.local:8443/features as a local user or as
   a user on another system works - I get a list of features

What doesn't work is trying to create a smart-proxy from within the web
app. Doing so results in an error and the production.log message:

Failed to save: Unable to communicate with the proxy: ERF12-2530
[ProxyAPI::ProxyException]: Unable to detect features
([RestClient::SSLCertificateNo
tVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify
failed) for proxy https://dev-foreman.awt.local:8443/features, Please check
the proxy is configured and running o
n the host.

Here is my foreman settings.yml
### File managed with puppet ###

## Module: 'foreman'

:unattended: true

:login: true

:require_ssl: true

:locations_enabled: false

:organizations_enabled: false

:puppetrun: false

:puppetssldir: /etc/puppetlabs/puppet/ssl

# The following values are used for providing default settings during db
migrate

:oauth_active: true

:oauth_map_users: false

:oauth_consumer_key: kSAzFYd2My8ec9abyK7VadTcscD2hxAe

:oauth_consumer_secret: oBXX7DsRKbtAi8t5CtBrDuVeQBXDJ4EB

# Websockets

:websockets_encrypt: on

:websockets_ssl_key: /usr/local/share/ca-certificates/foreman/foreman_key.
pem

:websockets_ssl_cert: /usr/local/share/ca-certificates/foreman/foreman_cert.
pem

# SSL-settings

:ssl_certificate: /usr/local/share/ca-certificates/foreman/foreman_cert.pem

:ssl_ca_file: /usr/local/share/ca-certificates/foreman/foreman_ssl_ca.pem

:ssl_priv_key: /usr/local/share/ca-certificates/foreman/foreman_key.pem

# Log settings for the current environment can be adjusted by adding them

# here. For example, if you want to increase the log level.

:logging:

  :level: debug

# Individual logging types can be toggled on/off here

:loggers:

:dynflow:

  :pool_size: 5

and my foreman-proxy settings:

···

---

### File managed with puppet ###

## Module: 'foreman_proxy'

:settings_directory: /etc/foreman-proxy/settings.d

# SSL Setup

# if enabled, all communication would be verified via SSL

# NOTE that both certificates need to be signed by the same CA in order for
this to work

# see SSL - Smart Proxy - Foreman for more
information

:ssl_ca_file: /usr/local/share/ca-certificates/foreman/foreman_ssl_ca.pem

:ssl_certificate: /usr/local/share/ca-certificates/foreman/foreman_cert.pem

:ssl_private_key: /usr/local/share/ca-certificates/foreman/foreman_key.pem

# Use this option only if you need to disable certain cipher suites.

# Note: we use the OpenSSL suite name, take a look at:

#
https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER-SUITE-NAMES

# for more information.

#:ssl_disabled_ciphers: [CIPHER-SUITE-1, CIPHER-SUITE-2]

# the hosts which the proxy accepts connections from

# commenting the following lines would mean every verified SSL connection
allowed

:trusted_hosts:

  - dev-foreman.awt.local

# Endpoint for reverse communication

:foreman_url: https://dev-foreman.awt.local

# SSL settings for client authentication against Foreman. If undefined, the
values

# from general SSL options are used instead. Mainly useful when Foreman uses

# different certificates for its web UI and for smart-proxy requests.

:foreman_ssl_ca: /usr/local/share/ca-certificates/foreman/foreman_ssl_ca.pem

:foreman_ssl_cert: /usr/local/share/ca-certificates/foreman/foreman_cert.pm

:foreman_ssl_key: /usr/local/share/ca-certificates/foreman/foreman_key.pem

# by default smart_proxy runs in the foreground. To enable running as a
daemon, uncomment 'daemon' setting

:daemon: true

# Only used when 'daemon' is set to true.

# Uncomment and modify if you want to change the default pid file
'/var/run/foreman-proxy/foreman-proxy.pid'

#:daemon_pid: /var/run/foreman-proxy/foreman-proxy.pid

# host and ports configuration

# Host or IP to bind ports to (e.g. *, localhost, 0.0.0.0, ::, 192.168.1.20)

:bind_host: '*'

# http is disabled by default. To enable, uncomment 'http_port' setting

# https is enabled if certificate, CA certificate, and private key are
present in locations specifed by

# ssl_certificate, ssl_ca_file, and ssl_private_key correspondingly

# default values for https_port is 8443

:https_port: 8443

#:http_port: 8000

# Log configuration

# Uncomment and modify if you want to change the location of the log file
or use STDOUT or SYSLOG values

:log_file: /var/log/foreman-proxy/proxy.log

# Uncomment and modify if you want to change the log level

# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN

:log_level: DEBUG

# Log buffer size and extra buffer size (for errors). Defaults to 3000
messages in total,

# which is about 500 kB request.

:log_buffer: 2000

:log_buffer_errors: 1000

Here's my foreman cert (openssl x509 -noout -text -in foreman_cert.pem
Certificate:

    Data:

        Version: 3 (0x2)

        Serial Number:

            37:8e:aa:65:d0:69:79:53:31:d8:dd:47:2a:66:19:0e:f9:f0:ad:f9

    Signature Algorithm: sha256WithRSAEncryption

        Issuer: CN=AWT Intermediate Certificate Authority

        Validity

            Not Before: Dec 16 02:32:06 2017 GMT

            Not After : Jan 15 02:32:31 2018 GMT

        Subject: CN=dev-foreman.awt.local

        Subject Public Key Info:

            Public Key Algorithm: rsaEncryption

                Public-Key: (2048 bit)

                Modulus:

                    00:b6:e9:e9:87:5b:df:6c:b0:1a:c4:0f:9a:91:75:

                    c8:43:f2:18:69:6b:89:44:56:31:0b:c7:f4:0d:50:

                    52:0a:19:df:c1:77:30:cd:1b:10:64:5b:fa:74:b6:

                    2c:a2:38:a3:6e:b8:9c:6b:3b:7d:d3:c8:40:68:9e:

                    40:0d:64:fa:37:d0:d2:dc:5f:9c:7d:5d:e0:8d:ea:

                    da:7a:31:fd:a9:ed:9f:ee:99:b0:9e:40:c2:6e:d1:

                    05:7b:34:ee:4a:d5:9d:dd:5c:ac:be:13:a5:bb:72:

                    f5:4e:f2:31:e4:08:6b:58:d3:8e:d0:c3:39:64:20:

                    08:c7:27:0c:74:99:d5:32:b1:bc:4a:f4:84:b2:5d:

                    1c:bb:5a:54:a5:91:0a:08:6c:c1:8a:6e:43:21:d0:

                    d6:a6:84:c3:2e:14:0d:a3:a8:68:c8:aa:0c:79:05:

                    e6:51:73:1b:69:96:79:39:82:e9:20:24:cc:88:0e:

                    e0:5a:11:6d:08:ed:0b:7a:13:1c:f7:28:68:b6:8f:

                    e4:50:88:1e:de:d6:5e:5f:f7:53:da:51:03:a3:a5:

                    50:5c:41:a5:69:ab:53:6c:a8:40:4f:aa:82:4d:91:

                    32:95:e2:92:73:a5:38:17:f6:1f:93:37:b3:cd:5f:

                    12:e9:5f:6a:cd:c2:55:2d:45:3c:a1:50:ea:7d:c1:

                    56:07

                Exponent: 65537 (0x10001)

        X509v3 extensions:

            X509v3 Key Usage: critical

                Digital Signature, Key Encipherment, Key Agreement

            X509v3 Extended Key Usage:

                TLS Web Server Authentication, TLS Web Client Authentication

            X509v3 Subject Key Identifier:

                CF:7B:F3:97:54:AC:20:38:C7:CB:92:FB:A8:C5:A8:45:D1:59:0B:C9

            X509v3 Authority Key Identifier:

                keyid:C0:41:49:9D:44:AB:32:36:EB:81:27:E0:18:F6:E6:9E:5B:9E:
74:BF

            Authority Information Access:

                CA Issuers - URI:http://127.0.0.1:8200/v1/pki_int/ca

            X509v3 Subject Alternative Name:

                DNS:dev-foreman.awt.local

            X509v3 CRL Distribution Points:

                Full Name:

                  URI:http://127.0.0.1:8200/v1/pki_int/crl

    Signature Algorithm: sha256WithRSAEncryption

         43:7a:fb:bb:1c:2c:1a:13:eb:48:93:bd:68:c7:8f:e9:62:18:

         42:44:22:c4:cd:56:c6:5c:e6:fc:ac:fd:92:cf:7e:0e:50:03:

         07:1b:bf:17:b2:32:53:74:83:c2:9b:d7:5b:f4:d3:e6:70:89:

         fd:e2:1e:25:4c:c0:bc:81:53:da:ae:e1:68:cf:62:33:af:d2:

         df:66:5c:86:3f:14:7e:e4:11:e5:60:17:ea:d7:fb:28:80:ca:

         8a:80:23:c6:1b:37:f0:f5:91:5f:55:88:1d:fe:1f:0f:b5:1d:

         0b:3f:1f:4d:0e:34:33:68:52:67:c3:29:a8:5a:c2:87:47:a4:

         9b:e1:80:61:33:14:7d:3f:3e:a7:6d:f6:2f:ab:41:d7:8f:14:

         35:e9:e0:d6:12:fc:f8:f9:f6:64:ad:94:72:55:f8:d7:ed:a8:

         11:f0:ee:b2:49:b7:d4:64:56:7e:df:34:ed:1a:fe:04:7a:87:

         07:37:19:ac:bb:0f:9f:c6:a4:4d:72:92:c5:9d:20:16:b2:6b:

         db:27:61:f9:62:d1:95:3d:1f:29:88:74:9a:89:43:57:07:cd:

         18:77:77:8a:72:f3:49:a3:40:ba:33:ce:5b:b2:50:ff:42:c7:

         a8:8a:59:ca:7e:25:93:52:64:4f:67:7d:96:3c:f3:cd:79:1d:

         99:2d:89:e7

and my ca (openssl x509 -noout -text -in foreman_ssl_ca.pem
Certificate:

    Data:

        Version: 3 (0x2)

        Serial Number:

            6b:03:b3:1b:27:0a:dd:96:c6:5a:c4:f4:48:f4:dc:dd:19:d6:5b:91

    Signature Algorithm: sha256WithRSAEncryption

        Issuer: CN=aussieswithtails.com

        Validity

            Not Before: Dec 15 01:56:06 2017 GMT

            Not After : Jan 16 01:56:36 2018 GMT

        Subject: CN=AWT Intermediate Certificate Authority

        Subject Public Key Info:

            Public Key Algorithm: rsaEncryption

                Public-Key: (2048 bit)

                Modulus:

                    00:cc:e7:17:86:74:a5:25:50:a7:5b:1e:ef:28:a2:

                    24:72:6d:1d:06:d6:cb:5e:a4:14:d2:fe:31:4c:ab:

                    ae:cd:06:26:77:c9:11:f6:42:cf:56:f8:35:c7:e0:

                    a8:d8:80:8c:61:4f:62:79:e9:0f:5c:ea:60:68:00:

                    6f:31:63:68:74:9a:8e:6c:a4:f4:e6:3a:ef:01:a7:

                    1f:d2:97:b2:6b:83:a6:f0:fe:63:d5:51:3a:24:c2:

                    38:10:d7:2a:99:fd:40:e8:e7:31:4c:95:52:69:37:

                    1b:2c:65:f4:68:fa:e2:c9:76:1f:fb:3e:d9:d7:51:

                    e2:92:75:d4:d9:f6:fe:2e:0a:e8:8a:d9:1c:aa:ab:

                    2f:b4:f4:28:f5:6c:e0:f6:9f:4e:0c:c5:1b:30:90:

                    3f:16:12:5f:a0:22:f0:4a:7f:ea:8b:c3:03:17:00:

                    de:ab:b3:d8:e3:02:55:a8:64:e1:91:c2:7d:59:4d:

                    54:8f:45:7f:93:8a:40:98:bf:9b:bb:30:83:43:d4:

                    57:35:4a:3d:5c:d2:61:62:c5:39:46:03:ee:a5:25:

                    c1:bb:a9:3f:60:35:03:14:51:17:f2:7b:fb:36:9c:

                    a4:88:85:a6:eb:4f:c3:f5:24:44:41:90:8b:1b:1d:

                    1d:45:7c:e5:a8:74:54:92:b3:71:96:d0:5e:50:87:

                    31:dd

                Exponent: 65537 (0x10001)

        X509v3 extensions:

            X509v3 Key Usage: critical

                Certificate Sign, CRL Sign

            X509v3 Basic Constraints: critical

                CA:TRUE

            X509v3 Subject Key Identifier:

                C0:41:49:9D:44:AB:32:36:EB:81:27:E0:18:F6:E6:9E:5B:9E:74:BF

            X509v3 Authority Key Identifier:

                keyid:0B:A4:90:CE:FA:61:FD:69:4F:11:F7:98:25:2D:20:FF:D8:AF:
58:D4

            Authority Information Access:

                CA Issuers - URI:http://vault.example.com:8200/v1/pki/ca

            X509v3 Subject Alternative Name:

                DNS:AWT Intermediate Certificate Authority

            X509v3 CRL Distribution Points:

                Full Name:

                  URI:http://vault.example.com:8200/v1/pki/crl

    Signature Algorithm: sha256WithRSAEncryption

         73:09:4b:32:e2:39:ad:1c:4a:79:5c:c8:b1:e6:de:4a:32:55:

         42:cc:e7:82:1c:e7:58:40:40:da:39:10:57:c8:14:ec:b4:72:

         39:3c:92:6d:eb:65:b2:36:40:da:f0:24:c3:ac:c9:9d:37:55:

         c6:6b:d9:55:d0:e2:74:8f:b4:48:ad:15:f4:6f:76:fb:84:69:

         ee:b6:b3:31:f3:8d:6d:ac:5a:0c:75:81:98:17:20:9b:f6:33:

         be:61:6d:0f:0f:78:c2:b5:90:45:22:82:55:94:57:96:00:ee:

         68:65:95:b8:d0:72:45:46:07:69:36:47:d3:a4:ae:ce:ad:e5:

         87:79:5a:64:e9:6f:e4:56:64:a7:01:f8:40:47:3e:58:ae:52:

         9b:71:4a:61:d8:fa:b2:f5:5e:1a:e4:56:c9:5c:21:37:06:e2:

         d2:a2:7a:7e:90:9f:8b:e9:93:a5:1b:ab:99:e3:c0:cb:96:ff:

         19:0d:e5:fa:04:84:ed:ab:f7:9e:15:3b:91:44:ab:2c:aa:b9:

         66:a8:06:ab:10:14:db:01:c0:5e:6a:b7:62:bf:ab:1e:35:14:

         49:34:fc:5a:03:8b:b3:22:c9:97:f4:17:fc:87:21:6c:39:18:

         26:60:9a:57:62:2f:48:0c:4a:87:1a:c6:39:1b:6f:1b:39:7a:

         c6:14:b6:70

The root certificate is trusted. All certificates and keys were generated
through Hashicorp's Vault tool

I'm pretty much an ssl newbie so I may have made an obvious error.

Any assistance would be much appreciated.

-steve