Proxy centos7-devel.localhost.example.com cannot be retrieved: unknown error (response 503)

rabajaj · May 24, 2018, 3:30pm

Problem:

Not able to install the dhcp dns and tftp services on my smart proxy.
Running this command:

sudo foreman-installer --scenario katello-devel --foreman-proxy-dhcp=true --foreman-proxy-dns=true --foreman-proxy-tftp=true --disable-system-checks

error:

/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[centos7-devel.localhost.example.com]: Could not evaluate: Proxy centos7-devel.localhost.example.com cannot be retrieved: unknown error (response 503)
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:7:in `proxy'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:81:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1070:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1098:in `retrieve_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:302:in `from_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:661:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

logs for the error :



I, [2018-05-23T18:16:11.336375 ]  INFO -- : WEBrick 1.3.1
I, [2018-05-23T18:16:11.336425 ]  INFO -- : ruby 2.0.0 (2015-12-16) [x86_64-linux]
I, [2018-05-23T18:16:11.352351 ]  INFO -- : WEBrick::HTTPServer#start: pid=852 port=9090
I, [2018-05-23T18:16:11.352485 ]  INFO -- : WEBrick::HTTPServer#start: pid=852 port=8000
I, [2018-05-23T18:16:11.352528 ]  INFO -- : Smart proxy has launched on 2 socket(s), waiting for requests
W, [2018-05-23T18:16:11.398227 ]  WARN -- : Failed to initialize puppet class cache, deferring initialization. Is puppetserver running?
W, [2018-05-23T18:20:06.133374 739e951b]  WARN -- : File at /var/lib/pulp/content defined in pulp_content_dir parameter doesn't exist or is unreadable
I, [2018-05-23T18:20:06.150185 739e951b]  INFO -- : ::ffff:192.168.121.102 - - [23/May/2018:18:20:06 +0000] "GET /pulp/status/disk_usage HTTP/1.1" 200 383 0.0174

I, [2018-05-23T19:00:57.387864 739e951b]  INFO -- : ::ffff:192.168.121.102 - - [23/May/2018:19:00:57 +0000] "GET /version HTTP/1.1" 200 109 0.0014

I, [2018-05-23T19:10:09.204223 ]  INFO -- : going to shutdown ...
I, [2018-05-23T19:10:09.206965 ]  INFO -- : WEBrick::HTTPServer#start done.
I, [2018-05-23T19:10:09.207728 ]  INFO -- : going to shutdown ...
I, [2018-05-23T19:10:09.207787 ]  INFO -- : WEBrick::HTTPServer#start done.
E, [2018-05-23T19:10:09.648077 ] ERROR -- : Disabling all modules in the group ['dns_libvirt', 'dns'] due to a failure in one of them: cannot load such file -- libvirt
E, [2018-05-23T19:10:09.650888 ] ERROR -- : Disabling all modules in the group ['dhcp_libvirt', 'dhcp'] due to a failure in one of them: cannot load such file -- libvirt
I, [2018-05-23T19:10:09.657322 ]  INFO -- : Successfully initialized 'pulp'
I, [2018-05-23T19:10:09.657393 ]  INFO -- : Successfully initialized 'foreman_proxy'
I, [2018-05-23T19:10:09.657430 ]  INFO -- : Successfully initialized 'tftp'
I, [2018-05-23T19:10:09.657461 ]  INFO -- : Successfully initialized 'puppetca'
I, [2018-05-23T19:10:09.657618 ]  INFO -- : Started puppet class cache initialization
I, [2018-05-23T19:10:09.658055 ]  INFO -- : Successfully initialized 'puppet_proxy_puppet_api'
I, [2018-05-23T19:10:09.658259 ]  INFO -- : Successfully initialized 'puppet'
I, [2018-05-23T19:10:09.658316 ]  INFO -- : Successfully initialized 'logs'
I, [2018-05-23T19:10:09.673598 ]  INFO -- : WEBrick 1.3.1
I, [2018-05-23T19:10:09.673752 ]  INFO -- : ruby 2.0.0 (2015-12-16) [x86_64-linux]
I, [2018-05-23T19:10:09.674961 ]  INFO -- : 
Certificate:
    Data:

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]

found a similar issue here: Bug #19069: DNS and DHCP providers for libvirt fail to start - Packaging - Foreman

lzap · May 24, 2018, 3:39pm

Rahul, this error is from a puppet “step” which creates smart proxy record in Foreman. What is actualy does is CREATE call with proxy name and url. That’s pretty much all of it. You can skip this step via --foreman-proxy-register-in-foreman false and it will work.

The root cause is perhaps “cannot load such file – libvirt” which indicates you don’t have libvirt rubygem installed or something? During proxy creation, Foreman tries to access it and list all its feature, that’s why it fails.

rabajaj · May 24, 2018, 3:48pm

@lzap , hey
So, after running gem install ruby-libvirt I get an error:

[vagrant@centos7-devel ~]$ sudo foreman-installer --scenario katello-devel --foreman-proxy-dhcp=true --foreman-proxy-dns=true --foreman-proxy-tftp=true --disable-system-checks
WARNING: '--emit-json-stdout' is deprecated. Use '--format json' instead.
WARNING: '--emit-json-stdout' is deprecated. Use '--format json' instead.
WARNING: '--emit-json-stdout' is deprecated. Use '--format json' instead.
WARNING: '--emit-json-stdout' is deprecated. Use '--format json' instead.
WARNING: '--emit-json-stdout' is deprecated. Use '--format json' instead.
WARNING: '--emit-json-stdout' is deprecated. Use '--format json' instead.
Resetting puppet server version param...
 Proxy centos7-devel.localhost.example.com cannot be refreshed: unknown error (response 500)
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:53:in `refresh_features!'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:48:in `features='
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:195:in `call_provider'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:506:in `set'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:568:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:661:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[centos7-devel.localhost.example.com]/features: change from ["Logs", "Pulp", "Puppet", "Puppet CA", "TFTP"] to ["DHCP", "DNS", "Logs", "Puppet", "Puppet CA", "TFTP"] failed: Proxy centos7-devel.localhost.example.com cannot be refreshed: unknown error (response 500)
Installing             Done                                               [100%] [..............................

Also, if I run the command with --foreman-proxy-register-in-foreman option, will it have an adverse effect while discovering the host or provisioning?

logs for the error are here : https://paste.fedoraproject.org/paste/I2QShYCMdnEq~wve9Ls4cQ

lzap · May 24, 2018, 3:59pm

You want to see proxy.log in this case - it does not return features for some reason.

rabajaj · May 24, 2018, 4:12pm

I am not sure how but I am looking at /var/log/foreman-proxy/proxy.log but that seems to be empty in my case. It has its user and group both set to foreman-proxy, is that correct or it should be root?

rabajaj · May 24, 2018, 4:28pm

Hey solved the issue, checked for the libvirt services and found that the libvirt-guests service was not active.
Just ran systemctl status libvirt-guests.service and things worked for me thanks @lzap for your help!

lzap · May 25, 2018, 6:27am

I am pretty sure that this service has nothing to do with proxy. This starts/stops libvirt guests during hypervisor boot or shutdown. But glad you sorted it out