Certificate setup failure with custom CA

barn · August 28, 2020, 3:19pm

I might be going about this the wrong way. I’ve been assuming the issue is the Smart Proxy cert/ssl because that’s what the errors say.

However, the Smart proxy is completely offline and requests to myserver.com:9090 just timeout. So its possible the installer is creating a bad config that stops the Smart Proxy from starting-up?

ekohl · August 28, 2020, 3:35pm

This is the registration part. That works by talking to the Foreman API, which is why you see it connects to https://foreman.example.com and not https://foreman.example.com:9090. That connection fails and must be debugged.

For the actual registration, Foreman will connect to the Foreman Proxy and if that doesn’t work, it’ll show you an error.

Right now I have a hard time explaining why it would fail. Do you by any chance have some HTTP proxy set up?

barn · August 28, 2020, 3:47pm

I have a /etc/yum.conf proxy. But not a system-wide proxy.

barn · August 28, 2020, 6:26pm

I tried removing the proxy and retrying, but its inconclusive: this time the installer hung for over an hour before it failed noting that a dependency was missing.

Is there a list of dependencies available so i can preinstall them all prior to using foreman-installer?

Execution of '/bin/yum -d 0 -e 0 -y install foreman-service' returned 1: Error downloading packages:
[ERROR 2020-08-28T10:54:10 verbose]   tfm-rubygem-puma-4.3.3-4.el7.x86_64: [Errno 256] No more mirrors to try.
[ERROR 2020-08-28T10:54:10 verbose]   foreman-service-2.1.2-1.el7.noarch: [Errno 256] No more mirrors to try.

barn · August 29, 2020, 5:46pm

Without the proxy, i cant get any internet access to my box. It’s currently the policy, so i attempted the following workaround that i thought would be sufficient, but it still failed.
NOTE: the steps below were completed on a new VM install.

1) Enable yum.conf proxy, install required repos and packages, and do a yum update.

2) Clean install:

foreman-installer -v --scenario katello
Install completed without errors, as it always does.

3) Remove proxy from yum.conf

4) Check certs

katello-certs-check -c /root/certs/myserver.crt -k /root/certs/myserver-d.key -b /etc/pki/tls/certs/ROOTCA-CA_2019.crt

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[OK]

Checking CA bundle size:
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

Validation succeeded


To install the Katello main server with the custom certificates, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/root/certs/myserver.crt" \
                      --certs-server-key "/root/certs/myserver-d.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt"

To update the certificates on a currently running Katello installation, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/root/certs/myserver.crt" \
                      --certs-server-key "/root/certs/myserver-d.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt" \
                      --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy

5) Update certs…

foreman-installer --scenario katello \
  --certs-server-cert "/root/certs/myserver.crt" \
  --certs-server-key "/root/certs/myserver-d.key" \
  --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt" \
  --certs-update-server --certs-update-server-ca -v

6) Still failed with certificate errors. Log below.

As previously noted, after adding the certificates, the main web-access works successfully and the previous step correctly implemented my certificates. However, the Smart Proxy certificates appear broken and the Smart Proxy is offline in a failed state.

Any help understanding the issue or a possible workaround is appreciated.

grep ERROR /var/log/foreman-installer/katello.log

[ERROR 2020-08-28T21:17:34 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myserver.com]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myserver.com/api/v2/smart_proxies?search=name=%22myserver.com%22
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1115:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1143:in `retrieve_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:307:in `from_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-28T21:17:34 main] Wrapped exception:
[ERROR 2020-08-28T21:17:34 main] SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:985:in `connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:909:in `start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:1458:in `request'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1115:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1143:in `retrieve_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:307:in `from_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-28T21:17:34 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myserver.com]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myserver.com/api/v2/smart_proxies?search=name=%22myserver.com%22
[ERROR 2020-08-28T21:17:34 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myserver.com]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myserver.com/api/v2/smart_proxies?search=name=%22myserver.com%22
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:72:in `refresh'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:149:in `process_callback'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:121:in `block in queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `each'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:288:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-28T21:17:34 main] Wrapped exception:
[ERROR 2020-08-28T21:17:34 main] SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:985:in `connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:909:in `start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:1458:in `request'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:72:in `refresh'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:149:in `process_callback'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:121:in `block in queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `each'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:288:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[DEBUG 2020-08-28T21:17:36 main] Hook /usr/share/foreman-installer/katello/hooks/post/31-cdn_setting.rb returned [#<Logging::Logger:0x0000000003728ed8 @name="main", @parent=#<Logging::RootLogger:0x0000000002e15030 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::RollingFile:0x0000000003723140 @roller=#<Logging::Appenders::RollingFile::Roller:0x0000000003723118 @fn="/var/log/foreman-installer/katello{{.%d}}.log", @roll_by=:number, @filename="/var/log/foreman-installer/katello.log", @roll=false, @keep=nil, @copy_file="/var/log/foreman-installer/katello.log._copy_", @glob="/var/log/foreman-installer/katello.*.log", @number_rgxp=/\/var\/log\/foreman-installer\/katello.(\d+).log/, @format="/var/log/foreman-installer/katello.%d.log">, @size=nil, @age_fn="/var/log/foreman-installer/katello.log.age", @age_fn_mtime=nil, @age=nil, @encoding=#<Encoding:UTF-8>, @mode="a+:UTF-8", @io=#<File:/var/log/foreman-installer/katello.log>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="configure", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x0000000003722b50 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x0000000002e22488 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-28 20:11:18 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=nil>, @level=0, @write_size=500>], @additive=true, @level=0, @caller_tracing=false>, #<Logging::Logger:0x000000000371bcd8 @name="fatal", @parent=#<Logging::RootLogger:0x0000000002e15030 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::Stderr:0x0000000003718498 @io=#<IO:<STDERR>>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="stderr", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x0000000003718150 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x0000000002df3200 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-28 20:11:18 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=#<Logging::ColorScheme:0x0000000002df38e0 @scheme={"date"=>"\e[34m", "logger"=>"\e[36m", "line"=>"\e[33m", "file"=>"\e[33m", "method"=>"\e[33m", "info"=>"\e[32m", "warn"=>"\e[33m", "error"=>"\e[31m", "fatal"=>"\e[37m\e[41m"}, @lines=false, @levels=true>, @name_map_0=["DEBUG", "\e[32m INFO\e[0m", "\e[33m WARN\e[0m", "\e[31mERROR\e[0m", "\e[37m\e[41mFATAL\e[0m"]>, @level=0, @encoding=nil, @write_size=500>], @additive=true, @level=4, @caller_tracing=false>, #<Logging::Logger:0x0000000003821ba0 @name="verbose", @parent=#<Logging::RootLogger:0x0000000002e15030 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::Stdout:0x00000000037e9610 @io=#<IO:<STDOUT>>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="stdout", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x00000000037e7f40 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x0000000002df3200 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-28 20:11:18 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=#<Logging::ColorScheme:0x0000000002df38e0 @scheme={"date"=>"\e[34m", "logger"=>"\e[36m", "line"=>"\e[33m", "file"=>"\e[33m", "method"=>"\e[33m", "info"=>"\e[32m", "warn"=>"\e[33m", "error"=>"\e[31m", "fatal"=>"\e[37m\e[41m"}, @lines=false, @levels=true>, @name_map_0=["DEBUG", "\e[32m INFO\e[0m", "\e[33m WARN\e[0m", "\e[31mERROR\e[0m", "\e[37m\e[41mFATAL\e[0m"]>, @level=0, @encoding=nil, @write_size=500>], @additive=true, @level=1, @caller_tracing=false>]

barn · August 29, 2020, 6:16pm

Additional info:

From the attached full log, you will see that around line 29311 Katello is generating a certificate that is NOT my custom CA certificate. As @ekohl has alluded to earlier, this may be what is causing error (?)

Full log (though beginning appears to be lost):
katello.log (2.8 MB)

barn · September 1, 2020, 5:46pm

Update. I was able to get my VM to bypass any need for a proxy. So I rolled back my VM to a pre-foreman state and attempted a fresh foreman/katello install (see below), however, I received the exact same errors as previously shared.

1) Prereqs

No proxy whatsoever. Install required repos and packages, and do a yum update.

2) Check certs and install

katello-certs-check -c /root/certs/myserver.crt -k /root/certs/myserver-d.key -b /etc/pki/tls/certs/ROOTCA-CA_2019.crt

foreman-installer --scenario katello \
  --certs-server-cert "/root/certs/myserver.crt" \
  --certs-server-key "/root/certs/myserver-d.key" \
  --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt" \
  --certs-update-server --certs-update-server-ca -v

3) Errors

Got the same errors as previously shared. As previously documented, the main web console appears to all work correctly and successfully utilizes new certificates, however, the Katello Smart Proxy breaks and is unusable.

Still dont have a good understanding of what the underlying issue is. Help appreciated.

gvde · September 1, 2020, 6:42pm

I don‘t know if it makes a difference, but you are using the instructions to update the certificate in an existing installation and not the one for the initial, first installation…

barn · September 1, 2020, 8:14pm

sorry, that looks like a typo on my side. you can ignore that part.

I have tried doing clean install with certs from beginning, as well as, a clean install without certs, and then adding certs as a second step. but no luck.

barn · September 1, 2020, 9:27pm

Rolled back VM again, and tried clean install with certificates. This time I made sure to use SubCA/intermediateCA instead of RootCA. However, it still failed with what appears are the same errors.

katello.log.short.log (2.5 MB)

barn · September 2, 2020, 3:36am

I have done some backtracking, and noticed something I hadn’t realized before because of preliminary steps I had done. This may or may-not be related to my issue.

On a clean box, I run:

# katello-certs-check -c /root/certs/myserver.crt -k /root/certs/myserver.key -b /root/certs/SUBCA-CA_2019.cer

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[FAIL]

The /root/certs/SUBCA-CA_2019.cer does not verify the /root/certs/myserver.crt
/root/certs/myserver.crt: DC = com, DC = censored, DC = censored, CN = SUBCA-CA error 2 at 1 depth lookup:unable to get issuer certificate
Checking CA bundle size:
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

As you can see, there is a trust error, “unable to get issuer certificate”. This is the same error that appears during the foreman-installer execution.

The way i had circumvented this was by doing the following for both RootCA and intermediate/SubCA:

cp example.crt /etc/pki/tls/certs/
ln -s /etc/pki/tls/certs/example.crt /etc/pki/tls/certs/$(openssl x509 -noout -hash -in /etc/pki/tls/certs/example.crt).0

and

cp /root/certs/rootCA /etc/pki/ca-trust/source/anchors/
cp /root/certs/subCA /etc/pki/ca-trust/source/anchors/
update-ca-trust

Then, when i re-run “katello-certs-check”, all checks pass [OK].

Is it possible that the main foreman certificates inherit this^ trust, but the Smart Proxy does not inherit it? Is there another way recommended to trust my custom CA?

gvde · September 2, 2020, 4:06am

I have compared with my notes from my initial 3.15 installation.

I did not use --certs-update-server --certs-update-server-ca for the initial installation.
I have used the certificate chain bundle for --certs-server-ca-cert. The file contains the issuing CA, followed by the sub CA followed by the root CA.

ekohl · September 2, 2020, 9:28am

That is correct. The Smart Proxy and Smart Proxy registration in the Foreman do not use the system trust store. You must configure things explicitly.

Are you by any chance using any intermediate CA?

barn · September 2, 2020, 2:03pm

Yes. And I had tried using both the RootCA and Intermediate/subca as --certs-server-ca-cert. What is the recommended way to identify them explicitly? Can i include --certs-server-ca-cert switch twice, once for Root and once for intermediate?

ekohl · September 2, 2020, 2:05pm

I don’t think so. They both need to be in the same file. The order is also important. IIRC it must be first the intermediate and then the CA.

barn · September 2, 2020, 2:07pm

I don’t immediately know how to create chain bundle (suggestions/tips welcome), but ill give this a try and circle back.

ekohl · September 2, 2020, 2:07pm

You can just cat them together.

barn · September 2, 2020, 2:32pm

Y’alls suggestions worked. Simply creating a chain bundle resolved issue. Looks like all certs are good, AND smart proxy is up.

Obviously, im not very familiar with custom cert implementations. Really appreciate y’alls patience.

ekohl · September 2, 2020, 4:41pm

Glad to hear that. SSL implementations often require some knowledge that isn’t really documented. Those who use it typically know these pitfalls and don’t write docs, those who don’t are doomed to learn it the hard way.

gvde · September 2, 2020, 5:56pm

Well, generally, if you configure an SSL server using a certificate from a sub CA of a root or intermediate CA the server must be configured with the CA chain. Thus, you’ll have to specify a bundle which contains those sub/intermediate CAs (and usually you can omit the root CA because the client should have preinstalled trusted CAs). That is how it always works. It has nothing to do with custom certs…

When a client connects, the server presents its own certificate together with the CA chain. The client uses the chain and it’s builtin list of trusted root CAs to verify the chain and the server certificate.

If the server does not deliver all intermediate CAs together the client won’t be able to verify the certificate and will fail. Unfortunately, browsers like firefox cache intermediate CAs whenever they come across one. When you later browse to a site which does not present all intermediate CAs, Firefox is still able to verify the chain because it has already learned the CAs before.