Foreman/Katello breaks on enable-foreman-plugin-remote-execution

Support
1

Problem:
–enable-foreman-plugin-remote-execution
–enable-foreman-proxy-plugin-remote-execution-ssh

Breaks fresh foreman/katello installation on CentOS 7.

Expected outcome:
Documented installation guidelines should produce a working instance.

Foreman and Proxy versions:

Installed Packages
foreman.noarch                      1.21.0-1.el7        @foreman
foreman-cli.noarch                  1.21.0-1.el7        @foreman
foreman-compute.noarch              1.21.0-1.el7        @foreman
foreman-debug.noarch                1.21.0-1.el7        @foreman
foreman-installer.noarch            1:1.21.0-1.el7      @foreman
foreman-installer-katello.noarch    1:1.21.0-1.el7      @katello
foreman-libvirt.noarch              1.21.0-1.el7        @foreman
foreman-openstack.noarch            1.21.0-1.el7        @foreman
foreman-postgresql.noarch           1.21.0-1.el7        @foreman
foreman-proxy.noarch                1.21.0-1.el7        @foreman
foreman-release.noarch              1.21.0-1.el7        installed
foreman-release-scl.noarch          7-2.el7             @foreman
foreman-selinux.noarch              1.21.0-1.el7        @foreman
foreman-vmware.noarch               1.21.0-1.el7        @foreman

Foreman and Proxy plugin versions:

*** LOCAL GEMS ***

ansi (1.4.3)
bigdecimal (1.2.0)
bundler (1.7.8)
bundler_ext (0.4.1)
clamp (1.1.2)
concurrent-ruby (1.1.4)
daemon_controller (1.1.2)
ffi (1.9.10)
foreman_maintain (0.3.0)
gssapi (1.1.2)
hashie (3.6.0)
highline (1.7.8)
io-console (0.4.2)
json (1.7.7)
jwt (1.2.1)
kafo (2.1.0)
kafo_parsers (1.0.0)
kafo_wizards (0.0.1)
little-plugger (1.1.3)
logging (2.2.2)
mime-types (1.19)
multi_json (1.12.2)
net-http-persistent (2.8)
netrc (0.7.7)
powerbar (2.0.1)
psych (2.0.0)
rack (1.6.4)
rack-protection (1.5.3)
rake (0.9.6)
rb-inotify (0.9.7)
rdoc (4.0.0)
rest-client (1.6.7)
rkerberos (0.1.3)
rsec (0.4.2)
rubyipmi (0.10.0)
sinatra (1.4.8)
smart_proxy_ansible (2.1.2)
smart_proxy_discovery (1.0.4)
smart_proxy_dynflow (0.2.2)
smart_proxy_pulp (1.3.0)
smart_proxy_remote_execution_ssh (0.2.0)
thor (0.19.1)
tilt (2.0.7)

Other relevant data:

error from /var/log/foreman-installer/katello.log

[ERROR 2019-03-27T22:40:17 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[CENSORED_HOSTNAME]: Failed to call refresh: Proxy CENSORED_HOSTNAME cannot be retrieved: unknown error (response 500)
[ERROR 2019-03-27T22:40:17 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[CENSORED_HOSTNAME]: Proxy CENSORED_HOSTNAME cannot be retrieved: unknown error (response 500)
[ERROR 2019-03-27T22:40:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:7:in `proxy'
[ERROR 2019-03-27T22:40:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2019-03-27T22:40:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:81:in `retrieve'
[ERROR 2019-03-27T22:40:17 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/type/foreman_smartproxy.rb:72:in `refresh'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:147:in `process_callback'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:34:in `block in process_events'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:119:in `block in queued_events'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:118:in `each'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:118:in `queued_events'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:33:in `process_events'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:250:in `eval_resource'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `call'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block (2 levels) in evaluate'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:507:in `block in thinmark'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:506:in `thinmark'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block in evaluate'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:154:in `evaluate'
[ERROR 2019-03-27T22:40:17 main] /usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:222:in `block in apply'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:155:in `with_destination'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:221:in `apply'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:171:in `block in apply_catalog'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:224:in `block in benchmark'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:170:in `apply_catalog'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:343:in `run_internal'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:221:in `block in run'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:306:in `override'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:195:in `run'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:348:in `apply_catalog'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:274:in `block in main'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:306:in `override'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:225:in `main'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:662:in `exit_on_fail'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
[ERROR 2019-03-27T22:40:17 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

Installation commands. Fresh and updated CentOS 7.

firewall-cmd --zone=public --permanent --add-port=53/udp
firewall-cmd --zone=public --permanent --add-port=67/udp
firewall-cmd --zone=public --permanent --add-port=68/udp
firewall-cmd --zone=public --permanent --add-port=69/udp
firewall-cmd --zone=public --permanent --add-port=80/tcp
firewall-cmd --zone=public --permanent --add-port=443/tcp
firewall-cmd --zone=public --permanent --add-port=5647/tcp
firewall-cmd --zone=public --permanent --add-port=8000/tcp
firewall-cmd --zone=public --permanent --add-port=8140/tcp
firewall-cmd --zone=public --permanent --add-port=8443/tcp
firewall-cmd --zone=public --permanent --add-port=9090/tcp
firewall-cmd --reload

yum install -y http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install -y https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
yum install -y https://yum.theforeman.org/releases/1.21/el7/x86_64/foreman-release.rpm
yum install -y http://fedorapeople.org/groups/katello/releases/yum/3.11/katello/el7/x86_64/katello-repos-latest.rpm

yum install -y foreman-release-scl
yum install -y python2-django
yum install -y katello 

foreman-installer \
--scenario katello \
--foreman-initial-organization=CENSORED \
--foreman-initial-location=CENSORED \
--foreman-proxy-dhcp=true \
--foreman-proxy-dhcp-managed=true \
--foreman-proxy-dhcp-interface=eth0 \
--foreman-proxy-dhcp-nameservers=10.0.13.1 \
--foreman-proxy-dhcp-range="10.0.13.100 10.0.13.200" \
--foreman-proxy-dhcp-server=10.0.13.1 \
--foreman-proxy-tftp=true \
--foreman-proxy-tftp-managed=true \
--foreman-proxy-dns=true \
--foreman-proxy-dns-managed=true \
--foreman-proxy-dns-zone=CENSORED \
--foreman-proxy-dns-interface=eth0 \
--foreman-proxy-dns-forwarders=10.0.11.51 \
--foreman-proxy-dns-reverse=13.0.10.in-addr.arpa \
--foreman-proxy-bmc=true \
--enable-foreman-compute-libvirt \
--enable-foreman-compute-vmware \
--enable-foreman-plugin-remote-execution \
--enable-foreman-proxy-plugin-remote-execution-ssh
2

Seems like the solution is to bump up the “PassengerStartTimeout” to more than 90. I’ve tried 300 it works. Otherwise I get a non working passenger and I guess that is because of the slow cpu (1,5ghz). It needs more time for startup… I’m only guessing here. But it works at least. However, it feels like hacky solution.

If I do the “main” installation first I can specify all the variables and dhcp config for proxy. That install is successful. After that I do an other installation with enable remote execution and ansible plugins. That fails.

Edit /etc/httpd/conf.d/05-foreman-ssl.conf and /etc/httpd/conf.d/05-foreman.conf and change that value at the bottom. systemctl restart httpd and it works again. Even thou the plugin installation failed.

1 Like