Connection refused - connect(2) port 443

Problem:
At the very end of the foreman installation it fails with the following error message:

/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[vlt01429.ourcompany.net]: Could not 
evaluate: Exception Failed to open TCP connection to vlt01429.ourcompany.net:443 (Connection 
refused - connect(2) for "vlt01429.ourcompany.net" port 443) in get request to:  
https://vlt01429.ourcompany.net/api/v2/smart_proxies? 
search=name=%22vlt01429.ourcompany.net%22

Expected outcome:
Installer finishes without errors

Foreman and Proxy versions:
Foreman 1.20
Katello 3.10

Details below:

tfm-rubygem-hammer_cli_foreman-0.15.1-1.el7.noarch
tfm-rubygem-foreman_docker-4.1.0-2.fm1_20.el7.noarch
rubygem-foreman_maintain-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-7.el7.noarch
foreman-postgresql-1.20.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.13-1.fm1_20.el7.noarch
foreman-1.20.2-1.el7.noarch
tfm-rubygem-foreman-tasks-0.14.3-1.fm1_20.el7.noarch
vlt01429.ourcompany.net-foreman-proxy-1.0-1.noarch
tfm-rubygem-foreman-tasks-core-0.2.5-2.fm1_20.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.4-4.el7.noarch
foreman-release-scl-7-2.el7.noarch
foreman-debug-1.20.2-1.el7.noarch
foreman-installer-1.20.2-1.el7.noarch
foreman-installer-katello-3.10.0-1.el7.noarch
vlt01429.ourcompany.net-foreman-client-1.0-1.noarch
vlt01429.ourcompany.net-foreman-proxy-client-1.0-1.noarch
foreman-cli-1.20.2-1.el7.noarch
foreman-selinux-1.20.2-1.el7.noarch
foreman-proxy-1.20.2-1.el7.noarch
foreman-compute-1.20.2-1.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-selinux-3.0.3-1.el7.noarch

Other relevant data:

After it fails I was able to connect to the port 443 (the authorization is missing though):

[root@vlt01429 ~]# wget https://vlt01429.ourcompany.net/api/v2/smart_proxies?
search=name="vlt01429.ourcompany.net"
--2019-03-21 12:52:33--  https://vlt01429.ourcompany.net/api/v2/smart_proxies? 
search=name=vlt01429.ourcompany.net
Resolving vlt01429.ourcompany.net (vlt01429.ourcompany.net)... 10.200.1.66
Connecting to vlt01429.ourcompany.net (vlt01429.ourcompany.net)|10.200.1.66|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authorization failed.


[root@vlt01429 ~]# foreman-installer --scenario katello --foreman-initial-organization "ourcompany_Playground" --foreman-initial-location "Frankfurt" --foreman-admin-first-name Katello --foreman-admin-last-name Administrator --foreman-admin-username kat-admin --foreman-admin-password 'xxxxxxxxx' --foreman-proxy-tftp false --katello-proxy-url http://proxy.ourcompany.de --katello-proxy-port 3128 --foreman-proxy-bind-host '*' --puppet-server false --puppet-server-ca false --foreman-proxy-puppet false --foreman-proxy-manage-puppet-group false --foreman-proxy-puppetca false --foreman-proxy-content-puppet false --no-enable-puppet --enable-certs --certs-user foreman
Resetting puppet server version param...
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[vlt01429.ourcompany.net]: Could not evaluate: Exception Failed to open TCP connection to vlt01429.ourcompany.net:443 (Connection refused - connect(2) for "vlt01429.ourcompany.net" port 443) in get request to: https://vlt01429.ourcompany.net/api/v2/smart_proxies?search=name=%22vlt01429.ourcompany.net%22
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:81:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1070:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1098:in `retrieve_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:302:in `from_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:515:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:514:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:515:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:514:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:515:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:514:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:663:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:136:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
Wrapped exception:
Failed to open TCP connection to vlt01429.ourcompany.net:443 (Connection refused - connect(2) for "vlt01429.ourcompany.net" port 443)
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:906:in `rescue in block in connect'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:903:in `block in connect'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:902:in `connect'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:887:in `do_start'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:876:in `start'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1407:in `request'
/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:81:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1070:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1098:in `retrieve_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:302:in `from_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:515:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:514:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:515:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:514:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:515:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:514:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:663:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:136:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 
Installing             Done                                               [100%] [..........................................................................................................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/katello.log
yum install -y -q rh-mongodb34-syspaths finished successfully!

Not sure what is the root cause of this (hostname pointing to a different host probably) but you can use “skip-registration-in-foreman” flag (don’t remember the proper name check docs or help) to skip this step and then you can register the proxy manually in the web interface.

Often this points to a proxy somewhere that is in between and breaks it. I thought we had checks on environment variables, but that’s where I’d look.

I wouldn’t advise on disabling the registration because it provides some nice guarantees the proxy is actually in the correct state that you otherwise need to verify manually. It will also make upgrading harder, especially in the future where we’re going to rely on it more. There are also callbacks from the proxy that might break in the same way.