Certificate setup failure with custom CA

barn · August 21, 2020, 10:28pm

Problem:
I cant get my custom CA certs to work as expected. I dont grasp what is causing the issue. This is a brand new deployment of foreman that i did today.
Note: in the logs below, I have replaced my hostnames with alternative names.

Test my certs:

katello-certs-check -c /etc/pki/tls/certs/myforemanserver.crt \
> -k /etc/pki/tls/certs/myforemanserverD.key \
> -b /etc/pki/tls/certs/rootca-CA_2019.crt

Output:

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[OK]

Checking CA bundle size:
[OK]

Checking Subject Alt Name on certificate
[WARNING]

The /etc/pki/tls/certs/myforemanserver.crt does not contain a Subject Alt Name.
Checking Key Usage extension on certificate for Key Encipherment
[OK]

Validation succeeded


To install the Katello main server with the custom certificates, run:

    foreman-installer --scenario katello \
                    --certs-server-cert "/etc/pki/tls/certs/myforemanserver.crt" \
                    --certs-server-key "/etc/pki/tls/certs/myforemanserverD.key" \
                    --certs-server-ca-cert "/etc/pki/tls/certs/rootca-CA_2019.crt"

To update the certificates on a currently running Katello installation, run:

    foreman-installer --scenario katello \
                    --certs-server-cert "/etc/pki/tls/certs/myforemanserver.crt" \
                    --certs-server-key "/etc/pki/tls/certs/myforemanserverD.key" \
                    --certs-server-ca-cert "/etc/pki/tls/certs/rootca-CA_2019.crt" \
                    --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy

So next I run the command on my existing server:

foreman-installer --scenario katello \
 --certs-cname "myforemanserver" \
 --certs-server-cert /etc/pki/tls/certs/myforemanserver.crt \
 --certs-server-key /etc/pki/tls/certs/myforemanserverD.key \
 --certs-server-ca-cert /etc/pki/tls/certs/rootca-CA_2019.crt \
 --certs-update-server --certs-update-server-ca

Initial output:

... ... ...
Preparing installation Done
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/katello.log

I have pasted more logs in the “Other relevant data” section below.

Expected outcome:
Im trying to configure a foreman deployment with katello. Since i have custom CA, im trying to configure certs for my CA.

Foreman and Proxy versions:
This instance was just deployed today:
foreman 2.1
katello 3.16

Distribution and version:
CentOS Linux release 7.8.2003 (Core)

Other relevant data:

This a clean/new foreman install. I have only deployed server and configured LDAP/Active Directory which is authenticating successfully with my root and sub CA to port 636.

I installed server with following which deployed successfully:

foreman-installer -v --enable-foreman --enable-foreman-cli --enable-foreman-cli-ansible --enable-foreman-plugin-ansible --enable-foreman-cli-discovery --enable-foreman-plugin-discovery --enable-foreman-cli-remote-execution --enable-foreman-plugin-remote-execution --enable-foreman-cli-tasks --enable-foreman-plugin-setup --enable-foreman-plugin-tasks --enable-foreman-compute-vmware --scenario katello

I have a root CA and a sub CA, however, i did not understand how to include that in the configuration.

Here is a snapshot of the log file right before the errors begin:

[DEBUG 2020-08-21T14:55:16 main]  /File[/etc/puppetlabs/puppet/ssl/public_keys/myforemanserver.com.pem]: Adding autorequire
relationship with File[/etc/puppetlabs/puppet/ssl/public_keys]
[DEBUG 2020-08-21T14:55:16 main]  /File[/etc/puppetlabs/puppet/ssl/certs/ca.pem]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl/c
erts]
[DEBUG 2020-08-21T14:55:16 main]  /File[/etc/puppetlabs/puppet/ssl/crl.pem]: Adding autorequire relationship with File[/etc/puppetlabs/puppet/ssl]
[DEBUG 2020-08-21T14:55:16 main]  /File[/opt/puppetlabs/puppet/cache/facts.d]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache]
[DEBUG 2020-08-21T14:55:16 main]  /File[/opt/puppetlabs/puppet/cache/locales]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache]
[DEBUG 2020-08-21T14:55:16 main]  Finishing transaction 60092600
[DEBUG 2020-08-21T14:55:16 main]  Received report to process from myforemanserver.com
[ INFO 2020-08-21T14:55:17 main] Puppet has finished, bye!
[ INFO 2020-08-21T14:55:17 main] Executing hooks in group post
[DEBUG 2020-08-21T14:55:17 main] Hook /usr/share/foreman-installer/hooks/post/30-upgrade.rb returned nil
[DEBUG 2020-08-21T14:55:17 main] Hook /usr/share/foreman-installer/hooks/post/99-post_install_message.rb returned nil
[DEBUG 2020-08-21T14:55:17 main] cdn_ssl_version already migrated, skipping
[DEBUG 2020-08-21T14:55:17 main] Hook /usr/share/foreman-installer/katello/hooks/post/31-cdn_setting.rb returned [#<Logging::Logger:0x0000000002da1810 @
name="main", @parent=#<Logging::RootLogger:0x000000000244ebc8 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders
=[#<Logging::Appenders::RollingFile:0x0000000002d978d8 @roller=#<Logging::Appenders::RollingFile::Roller:0x0000000002d978b0 @fn="/var/log/foreman-instal
ler/katello{{.%d}}.log", @roll_by=:number, @filename="/var/log/foreman-installer/katello.log", @roll=false, @keep=nil, @copy_file="/var/log/foreman-inst
aller/katello.log._copy_", @glob="/var/log/foreman-installer/katello.*.log", @number_rgxp=/\/var\/log\/foreman-installer\/katello.(\d+).log/, @format="/
var/log/foreman-installer/katello.%d.log">, @size=nil, @age_fn="/var/log/foreman-installer/katello.log.age", @age_fn_mtime=nil, @age=nil, @encoding=#<En
coding:UTF-8>, @mode="a+:UTF-8", @io=#<File:/var/log/foreman-installer/katello.log>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1,
@async=false, @async_flusher=nil, @flush_period=nil, @name="configure", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x0000000002d97108 @locker=n
il>, @layout=#<Logging::Layouts::Pattern:0x00000000024639b0 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-2
1 14:52:58 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=nil>, @level=0, @write_size=500>], @ad
ditive=true, @level=0, @caller_tracing=false>, #<Logging::Logger:0x0000000002d83b08 @name="fatal", @parent=#<Logging::RootLogger:0x000000000244ebc8 @nam
e="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::Stderr:0x0000000002d80070 @io=#<IO:<STDERR
>>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="stderr", @closed=fals
e, @filters=[], @mutex=#<ReentrantMutex:0x0000000002d77d30 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x000000000213db78 @obj_format=:string, @ba
cktrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-21 14:52:58 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5
l %d %c] %m\n", @color_scheme=#<Logging::ColorScheme:0x000000000213f6d0 @scheme={"date"=>"\e[34m", "logger"=>"\e[36m", "line"=>"\e[33m", "file"=>"\e[33m
", "method"=>"\e[33m", "info"=>"\e[32m", "warn"=>"\e[33m", "error"=>"\e[31m", "fatal"=>"\e[37m\e[41m"}, @lines=false, @levels=true>, @name_map_0=["DEBUG
", "\e[32m INFO\e[0m", "\e[33m WARN\e[0m", "\e[31mERROR\e[0m", "\e[37m\e[41mFATAL\e[0m"]>, @level=0, @encoding=nil, @write_size=500>], @additive=true, @
level=4, @caller_tracing=false>]
[DEBUG 2020-08-21T14:55:17 main] Hook /usr/share/foreman-installer/katello/hooks/post/99-version_locking.rb returned nil
[ INFO 2020-08-21T14:55:17 main] All hooks in group post finished
[DEBUG 2020-08-21T14:55:17 main] Exit with status code: 6 (signal was 6)
[ERROR 2020-08-21T14:55:17 main] Errors encountered during run:
[ERROR 2020-08-21T14:55:17 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
[ERROR 2020-08-21T14:55:17 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myforemanserver.com]: Could not e
valuate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: htt
ps://myforemanserver.com/api/v2/smart_proxies?search=name=%22myforemanserver.com%22
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1115:in `retrieve'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1143:in `retrieve_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:307:in `from_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:734:in `exit_on_fail'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-21T14:55:17 main] Wrapped exception:
[ERROR 2020-08-21T14:55:17 main] SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:985:in `connect'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:909:in `start'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:1458:in `request'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1115:in `retrieve'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1143:in `retrieve_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:307:in `from_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:734:in `exit_on_fail'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-21T14:55:17 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myforemanserver.com]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myforemanserver.com/api/v2/smart_proxies?search=name=%22myforemanserver.com%22
[ERROR 2020-08-21T14:55:17 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myforemanserver.com]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myforemanserver.com/api/v2/smart_proxies?search=name=%22myforemanserver.com%22
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:72:in `refresh'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:149:in `process_callback'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:121:in `block in queued_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `each'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `queued_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:288:in `eval_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:734:in `exit_on_fail'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-21T14:55:17 main] Wrapped exception:
[ERROR 2020-08-21T14:55:17 main] SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:985:in `connect'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:909:in `start'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:1458:in `request'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-21T14:55:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:72:in `refresh'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:149:in `process_callback'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:121:in `block in queued_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `each'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `queued_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:288:in `eval_resource'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:734:in `exit_on_fail'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-21T14:55:17 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[DEBUG 2020-08-21T14:55:17 main] Cleaning /tmp/kafo_installation20200821-41239-4guz7i
[DEBUG 2020-08-21T14:55:17 main] Cleaning /tmp/kafo_installation20200821-41239-8h0bjz
[DEBUG 2020-08-21T14:55:17 main] Cleaning /tmp/default_values.yaml
[ INFO 2020-08-21T14:55:17 main] Installer finished in 133.162121914 seconds

ekohl · August 27, 2020, 4:56pm

You generated a custom certificate without a Subject Alt Name extension. Browsers have started to reject these. That’s why the verification is telling you that you should really do so.

I’m not sure why the SSL connect is failing though. Might be related, but I don’t know the stack well enough to say for sure.

barn · August 28, 2020, 1:28pm

Yea, i did notice that too. So i recreated the cert with a Subject Alt Name. Now, when i run “katello-certs-check”, it passes all checks as OK. However, i still get the same issue.

Looks like problem revolves around “certificate verify failed (unable to get local issuer certificate)”, however, I am not sure what the solution is. If I do a clean install without any certificates, LDAPS does pass SSL checks; so, I am not sure why a clean install with the CA creates these errors, or what these errors mean:

Below i have highlighted the 3 main errors i see from log:

[DEBUG 2020-08-27T19:37:18 main] Hook /usr/share/foreman-installer/katello/hooks/post/99-version_locking.rb returned nil
[ INFO 2020-08-27T19:37:18 main] All hooks in group post finished
[DEBUG 2020-08-27T19:37:18 main] Exit with status code: 6 (signal was 6)
[ERROR 2020-08-27T19:37:18 main] Errors encountered during run:
[ERROR 2020-08-27T19:37:18 main] foreman-maintain packages is-locked --assumeyes failed! Check the output for error!
[ERROR 2020-08-27T19:37:18 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[MyServer.com]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://MyServer.com/api/v2/smart_proxies?search=name=%22MyServer.com%22
[ERROR 2020-08-27T19:37:18 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'


/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[MyServer.com]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://MyServer.com/api/v2/smart_proxies?search=name=%22MyServer.com%22
[ERROR 2020-08-27T19:37:18 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[MyServer.com]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://MyServer.com/api/v2/smart_proxies?search=name=%22MyServer.com%22


Wrapped exception:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

barn · August 28, 2020, 2:28pm

I should note, when I get these error, the main foreman URL websocket works as expected and receives correct certificate!

However, the built-in Smart Proxy has SSL errors via Infrastructure > Smart Proxies:

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) for proxy https://myServer.com:9090/logs)

Also, the smart proxy reports a “down” or “broken” status (red X).

Maybe the error is something specific to how the certificate is generated?
- When creating the certificate, do I need to define something specific for the Smart Proxy?
Or do I need to generate a separate certificate for the built-in default katello smart proxy? If so, where is that done?

ekohl · August 28, 2020, 2:30pm

The code for registration is here:

github.com

theforeman/puppet-foreman_proxy/blob/master/manifests/register.pp

# @summary Register the foreman proxy
# @api private
class foreman_proxy::register {
  if $foreman_proxy::register_in_foreman {
    foreman_smartproxy { $foreman_proxy::registered_name:
      ensure          => present,
      base_url        => $foreman_proxy::foreman_base_url,
      consumer_key    => $foreman_proxy::oauth_consumer_key,
      consumer_secret => $foreman_proxy::oauth_consumer_secret,
      effective_user  => $foreman_proxy::oauth_effective_user,
      ssl_ca          => pick($foreman_proxy::foreman_ssl_ca, $foreman_proxy::ssl_ca),
      url             => $foreman_proxy::real_registered_proxy_url,
    }

    # Ensure puppet agent is started after registering the Foreman proxy
    # By using collectors, we don't have to test if the collected resource actually exists
    Foreman_smartproxy[$foreman_proxy::registered_name] -> Cron <| title == 'puppet' |>
    Foreman_smartproxy[$foreman_proxy::registered_name] -> Service <| title == 'puppet' |>
    Foreman_smartproxy[$foreman_proxy::registered_name] -> Service <| title == 'puppet-run.timer' |>

This file has been truncated. show original

Somehow the CA it uses there is not the right one. You can verify the values for foreman_ssl_ca (or ssl_ca_file if that’s unset). That file should contain the correct CA.

Then you can use openssl s_client -connect myserver.com:443 -CAfile /path/to/ca to see if it can connect.

barn · August 28, 2020, 3:04pm

Without making any changes, when i run…
openssl s_client -connect myserver.com:443 -CAfile /path/to/ca

It returns correct cert details, and concludes:
Verify return code: 0 (ok)

foreman_ssl_ca is unset, so i found the path for ssl_ca_file in /etc/foreman/settings.yml which is defined as…
:ssl_ca_file: /etc/foreman/proxy_ca.pem

So… i did…
openssl s_client -connect myserver.com:443 -CAfile /etc/foreman/proxy_ca.pem
… but that came back OK.

And…
openssl s_client -connect myserver.com:9090 -CAfile /etc/foreman/proxy_ca.pem
… also came back OK.

So i think that cert is ok, right?

I’ll keep digging, because there must be another location where a CA is defined for the Smart Proxy, and I havn’t found it yet. (suggestions welcome)

barn · August 28, 2020, 3:19pm

I might be going about this the wrong way. I’ve been assuming the issue is the Smart Proxy cert/ssl because that’s what the errors say.

However, the Smart proxy is completely offline and requests to myserver.com:9090 just timeout. So its possible the installer is creating a bad config that stops the Smart Proxy from starting-up?

ekohl · August 28, 2020, 3:35pm

This is the registration part. That works by talking to the Foreman API, which is why you see it connects to https://foreman.example.com and not https://foreman.example.com:9090. That connection fails and must be debugged.

For the actual registration, Foreman will connect to the Foreman Proxy and if that doesn’t work, it’ll show you an error.

Right now I have a hard time explaining why it would fail. Do you by any chance have some HTTP proxy set up?

barn · August 28, 2020, 3:47pm

I have a /etc/yum.conf proxy. But not a system-wide proxy.

barn · August 28, 2020, 6:26pm

I tried removing the proxy and retrying, but its inconclusive: this time the installer hung for over an hour before it failed noting that a dependency was missing.

Is there a list of dependencies available so i can preinstall them all prior to using foreman-installer?

Execution of '/bin/yum -d 0 -e 0 -y install foreman-service' returned 1: Error downloading packages:
[ERROR 2020-08-28T10:54:10 verbose]   tfm-rubygem-puma-4.3.3-4.el7.x86_64: [Errno 256] No more mirrors to try.
[ERROR 2020-08-28T10:54:10 verbose]   foreman-service-2.1.2-1.el7.noarch: [Errno 256] No more mirrors to try.

barn · August 29, 2020, 5:46pm

Without the proxy, i cant get any internet access to my box. It’s currently the policy, so i attempted the following workaround that i thought would be sufficient, but it still failed.
NOTE: the steps below were completed on a new VM install.

1) Enable yum.conf proxy, install required repos and packages, and do a yum update.

2) Clean install:

foreman-installer -v --scenario katello
Install completed without errors, as it always does.

3) Remove proxy from yum.conf

4) Check certs

katello-certs-check -c /root/certs/myserver.crt -k /root/certs/myserver-d.key -b /etc/pki/tls/certs/ROOTCA-CA_2019.crt

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[OK]

Checking CA bundle size:
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

Validation succeeded


To install the Katello main server with the custom certificates, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/root/certs/myserver.crt" \
                      --certs-server-key "/root/certs/myserver-d.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt"

To update the certificates on a currently running Katello installation, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/root/certs/myserver.crt" \
                      --certs-server-key "/root/certs/myserver-d.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt" \
                      --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy

5) Update certs…

foreman-installer --scenario katello \
  --certs-server-cert "/root/certs/myserver.crt" \
  --certs-server-key "/root/certs/myserver-d.key" \
  --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt" \
  --certs-update-server --certs-update-server-ca -v

6) Still failed with certificate errors. Log below.

As previously noted, after adding the certificates, the main web-access works successfully and the previous step correctly implemented my certificates. However, the Smart Proxy certificates appear broken and the Smart Proxy is offline in a failed state.

Any help understanding the issue or a possible workaround is appreciated.

grep ERROR /var/log/foreman-installer/katello.log

[ERROR 2020-08-28T21:17:34 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myserver.com]: Could not evaluate: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myserver.com/api/v2/smart_proxies?search=name=%22myserver.com%22
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1115:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1143:in `retrieve_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:307:in `from_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-28T21:17:34 main] Wrapped exception:
[ERROR 2020-08-28T21:17:34 main] SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:985:in `connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:909:in `start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:1458:in `request'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1115:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1143:in `retrieve_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:307:in `from_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-28T21:17:34 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myserver.com]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myserver.com/api/v2/smart_proxies?search=name=%22myserver.com%22
[ERROR 2020-08-28T21:17:34 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[myserver.com]: Exception SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) in get request to: https://myserver.com/api/v2/smart_proxies?search=name=%22myserver.com%22
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:72:in `refresh'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:149:in `process_callback'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:121:in `block in queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `each'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:288:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-28T21:17:34 main] Wrapped exception:
[ERROR 2020-08-28T21:17:34 main] SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:985:in `connect'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:909:in `start'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:1458:in `request'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
[ERROR 2020-08-28T21:17:34 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:72:in `refresh'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:149:in `process_callback'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:121:in `block in queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `each'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:120:in `queued_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:288:in `eval_resource'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:341:in `apply_catalog'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:253:in `block in main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-28T21:17:34 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[DEBUG 2020-08-28T21:17:36 main] Hook /usr/share/foreman-installer/katello/hooks/post/31-cdn_setting.rb returned [#<Logging::Logger:0x0000000003728ed8 @name="main", @parent=#<Logging::RootLogger:0x0000000002e15030 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::RollingFile:0x0000000003723140 @roller=#<Logging::Appenders::RollingFile::Roller:0x0000000003723118 @fn="/var/log/foreman-installer/katello{{.%d}}.log", @roll_by=:number, @filename="/var/log/foreman-installer/katello.log", @roll=false, @keep=nil, @copy_file="/var/log/foreman-installer/katello.log._copy_", @glob="/var/log/foreman-installer/katello.*.log", @number_rgxp=/\/var\/log\/foreman-installer\/katello.(\d+).log/, @format="/var/log/foreman-installer/katello.%d.log">, @size=nil, @age_fn="/var/log/foreman-installer/katello.log.age", @age_fn_mtime=nil, @age=nil, @encoding=#<Encoding:UTF-8>, @mode="a+:UTF-8", @io=#<File:/var/log/foreman-installer/katello.log>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="configure", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x0000000003722b50 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x0000000002e22488 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-28 20:11:18 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=nil>, @level=0, @write_size=500>], @additive=true, @level=0, @caller_tracing=false>, #<Logging::Logger:0x000000000371bcd8 @name="fatal", @parent=#<Logging::RootLogger:0x0000000002e15030 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::Stderr:0x0000000003718498 @io=#<IO:<STDERR>>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="stderr", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x0000000003718150 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x0000000002df3200 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-28 20:11:18 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=#<Logging::ColorScheme:0x0000000002df38e0 @scheme={"date"=>"\e[34m", "logger"=>"\e[36m", "line"=>"\e[33m", "file"=>"\e[33m", "method"=>"\e[33m", "info"=>"\e[32m", "warn"=>"\e[33m", "error"=>"\e[31m", "fatal"=>"\e[37m\e[41m"}, @lines=false, @levels=true>, @name_map_0=["DEBUG", "\e[32m INFO\e[0m", "\e[33m WARN\e[0m", "\e[31mERROR\e[0m", "\e[37m\e[41mFATAL\e[0m"]>, @level=0, @encoding=nil, @write_size=500>], @additive=true, @level=4, @caller_tracing=false>, #<Logging::Logger:0x0000000003821ba0 @name="verbose", @parent=#<Logging::RootLogger:0x0000000002e15030 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::Stdout:0x00000000037e9610 @io=#<IO:<STDOUT>>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="stdout", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x00000000037e7f40 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x0000000002df3200 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2020-08-28 20:11:18 -0700, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=#<Logging::ColorScheme:0x0000000002df38e0 @scheme={"date"=>"\e[34m", "logger"=>"\e[36m", "line"=>"\e[33m", "file"=>"\e[33m", "method"=>"\e[33m", "info"=>"\e[32m", "warn"=>"\e[33m", "error"=>"\e[31m", "fatal"=>"\e[37m\e[41m"}, @lines=false, @levels=true>, @name_map_0=["DEBUG", "\e[32m INFO\e[0m", "\e[33m WARN\e[0m", "\e[31mERROR\e[0m", "\e[37m\e[41mFATAL\e[0m"]>, @level=0, @encoding=nil, @write_size=500>], @additive=true, @level=1, @caller_tracing=false>]

barn · August 29, 2020, 6:16pm

Additional info:

From the attached full log, you will see that around line 29311 Katello is generating a certificate that is NOT my custom CA certificate. As @ekohl has alluded to earlier, this may be what is causing error (?)

Full log (though beginning appears to be lost):
katello.log (2.8 MB)

barn · September 1, 2020, 5:46pm

Update. I was able to get my VM to bypass any need for a proxy. So I rolled back my VM to a pre-foreman state and attempted a fresh foreman/katello install (see below), however, I received the exact same errors as previously shared.

1) Prereqs

No proxy whatsoever. Install required repos and packages, and do a yum update.

2) Check certs and install

katello-certs-check -c /root/certs/myserver.crt -k /root/certs/myserver-d.key -b /etc/pki/tls/certs/ROOTCA-CA_2019.crt

foreman-installer --scenario katello \
  --certs-server-cert "/root/certs/myserver.crt" \
  --certs-server-key "/root/certs/myserver-d.key" \
  --certs-server-ca-cert "/etc/pki/tls/certs/ROOTCA-CA_2019.crt" \
  --certs-update-server --certs-update-server-ca -v

3) Errors

Got the same errors as previously shared. As previously documented, the main web console appears to all work correctly and successfully utilizes new certificates, however, the Katello Smart Proxy breaks and is unusable.

Still dont have a good understanding of what the underlying issue is. Help appreciated.

gvde · September 1, 2020, 6:42pm

I don‘t know if it makes a difference, but you are using the instructions to update the certificate in an existing installation and not the one for the initial, first installation…

barn · September 1, 2020, 8:14pm

sorry, that looks like a typo on my side. you can ignore that part.

I have tried doing clean install with certs from beginning, as well as, a clean install without certs, and then adding certs as a second step. but no luck.

barn · September 1, 2020, 9:27pm

Rolled back VM again, and tried clean install with certificates. This time I made sure to use SubCA/intermediateCA instead of RootCA. However, it still failed with what appears are the same errors.

katello.log.short.log (2.5 MB)

barn · September 2, 2020, 3:36am

I have done some backtracking, and noticed something I hadn’t realized before because of preliminary steps I had done. This may or may-not be related to my issue.

On a clean box, I run:

# katello-certs-check -c /root/certs/myserver.crt -k /root/certs/myserver.key -b /root/certs/SUBCA-CA_2019.cer

Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[FAIL]

The /root/certs/SUBCA-CA_2019.cer does not verify the /root/certs/myserver.crt
/root/certs/myserver.crt: DC = com, DC = censored, DC = censored, CN = SUBCA-CA error 2 at 1 depth lookup:unable to get issuer certificate
Checking CA bundle size:
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

As you can see, there is a trust error, “unable to get issuer certificate”. This is the same error that appears during the foreman-installer execution.

The way i had circumvented this was by doing the following for both RootCA and intermediate/SubCA:

cp example.crt /etc/pki/tls/certs/
ln -s /etc/pki/tls/certs/example.crt /etc/pki/tls/certs/$(openssl x509 -noout -hash -in /etc/pki/tls/certs/example.crt).0

and

cp /root/certs/rootCA /etc/pki/ca-trust/source/anchors/
cp /root/certs/subCA /etc/pki/ca-trust/source/anchors/
update-ca-trust

Then, when i re-run “katello-certs-check”, all checks pass [OK].

Is it possible that the main foreman certificates inherit this^ trust, but the Smart Proxy does not inherit it? Is there another way recommended to trust my custom CA?

gvde · September 2, 2020, 4:06am

I have compared with my notes from my initial 3.15 installation.

I did not use --certs-update-server --certs-update-server-ca for the initial installation.
I have used the certificate chain bundle for --certs-server-ca-cert. The file contains the issuing CA, followed by the sub CA followed by the root CA.

ekohl · September 2, 2020, 9:28am

That is correct. The Smart Proxy and Smart Proxy registration in the Foreman do not use the system trust store. You must configure things explicitly.

Are you by any chance using any intermediate CA?

barn · September 2, 2020, 2:03pm

Yes. And I had tried using both the RootCA and Intermediate/subca as --certs-server-ca-cert. What is the recommended way to identify them explicitly? Can i include --certs-server-ca-cert switch twice, once for Root and once for intermediate?