Katello 3.13 to 3.14 upgrade issues on puppetserver smartproxy

Problem:
Upgrading Katello was a success, but I don’t use the foreman host for puppetserver, instead use smartproxy for this role.

When updating a smartproxy with puppetserver/pulp roles foreman-installer returned this error:

/opt/puppetlabs/puppet/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in 'require': cannot load such file -- kafo (LoadError) from /opt/puppetlabs/puppet/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in 'require' from /usr/sbin/foreman-installer:3:in <main>'

Expected outcome:
Upgrade would proceed without error.
We are still running an older puppet version but the docs for 3.14 mention puppet 5.5.10 is min req and we do meet that.

Foreman and Proxy versions:
Foreman and Proxy plugin versions:
[root@foreman-puppet001 ~ #] rpm -qa | egrep ‘foreman|katello|puppet|kafo|ruby’ | sort
foreman-debug-1.23.2-1.el7.noarch
foreman-installer-1.23.2-1.el7.noarch
foreman-installer-katello-1.23.2-1.el7.noarch
foreman-proxy-1.23.2-1.el7.noarch
foreman-proxy-content-3.13.4-1.el7.noarch
foreman-puppet001.test.domain-apache-1.0-2.noarch
foreman-puppet001.test.domain-foreman-proxy-1.0-2.noarch
foreman-puppet001.test.domain-foreman-proxy-client-1.0-2.noarch
foreman-puppet001.test.domain-puppet-client-1.0-2.noarch
foreman-puppet001.test.domain-qpid-broker-1.0-2.noarch
foreman-puppet001.test.domain-qpid-client-cert-1.0-2.noarch
foreman-puppet001.test.domain-qpid-router-client-1.0-2.noarch
foreman-puppet001.test.domain-qpid-router-server-1.0-2.noarch
foreman-release-1.24.3-1.el7.noarch
foreman-selinux-1.23.2-1.el7.noarch
katello-agent-3.1.0-1.el7.noarch
katello-ca-consumer-foreman001.prod.domain-1.0-20.noarch
katello-certs-tools-2.5.3-1.el7.noarch
katello-client-bootstrap-1.7.4-1.el7.noarch
katello-common-3.13.4-1.el7.noarch
katello-debug-3.13.4-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.1.0-1.el7.noarch
katello-host-tools-fact-plugin-3.1.0-1.el7.noarch
katello-repos-3.14.1-1.el7.noarch
katello-selinux-3.1.1-1.el7.noarch
katello-server-ca-1.0-12.noarch
katello-service-3.13.4-1.el7.noarch
pulp-puppet-plugins-2.20.1-2.el7.noarch
puppet-agent-5.5.22-1.el7.x86_64
puppet-agent-oauth-0.5.1-3.el7.noarch
puppetdb-termini-5.2.19-1.el7.noarch
puppetserver-5.3.16-1.el7.noarch
python-pulp-puppet-common-2.20.1-2.el7.noarch
ruby-2.0.0.648-36.el7.x86_64
rubygem-ansi-1.4.3-2.el7.noarch
rubygem-bigdecimal-1.2.0-36.el7.x86_64
rubygem-bundler-1.7.8-3.el7.noarch
rubygem-bundler_ext-0.4.1-3.el7.noarch
rubygem-clamp-1.1.2-4.el7.noarch
rubygem-concurrent-ruby-1.1.4-2.el7.noarch
rubygem-ffi-1.9.10-2.el7.x86_64
rubygem-foreman_maintain-0.4.5-1.el7.noarch
rubygem-gssapi-1.1.2-3.el7.noarch
rubygem-hashie-3.6.0-1.el7.noarch
rubygem-highline-1.7.8-4.el7.noarch
rubygem-io-console-0.4.2-36.el7.x86_64
rubygem-json-1.7.7-36.el7.x86_64
rubygem-jwt-1.2.1-1.el7.noarch
rubygem-kafo-3.0.0-1.el7.noarch
rubygem-kafo_parsers-1.0.0-1.el7.noarch
rubygem-kafo_wizards-0.0.1-2.el7.noarch
rubygem-little-plugger-1.1.4-1.el7.noarch
rubygem-logging-2.2.2-3.el7.noarch
rubygem-multi_json-1.12.2-3.el7.noarch
rubygem-net-http-persistent-2.8-5.el7.noarch
rubygem-powerbar-2.0.1-1.el7.noarch
rubygem-psych-2.0.0-36.el7.x86_64
rubygem-rack-1.6.12-1.el7.noarch
rubygem-rack-protection-1.5.3-3.el7.noarch
rubygem-rake-0.9.6-36.el7.noarch
rubygem-rb-inotify-0.9.7-2.el7.noarch
rubygem-rdoc-4.0.0-36.el7.noarch
rubygem-rkerberos-0.1.3-6.el7.x86_64
rubygem-rsec-0.4.3-1.el7.noarch
rubygem-rubyipmi-0.10.0-3.el7.noarch
rubygems-2.0.14.1-36.el7.noarch
rubygem-sinatra-1.4.8-2.el7.noarch
rubygem-smart_proxy_pulp-1.4.1-1.el7.noarch
rubygem-thor-0.19.1-1.el7.noarch
rubygem-tilt-2.0.7-1.el7.noarch
ruby-irb-2.0.0.648-36.el7.noarch
ruby-libs-2.0.0.648-36.el7.x86_64

Distribution and version:
[root@foreman-puppet001 ~ #] cat /etc/redhat-release CentOS Linux release 7.9.2009 (Core)

That’s weird, the error basically means that foreman-installer can’t find kafo library (rubygem). You seem to have it installed rubygem-kafo-3.0.0-1.el7.noarch. Are you by any chance in some SCL context? What’s slightly confusing is you foreman-installer version, 1.23.2 look quite old as well as katello 3.14.

Do you by any chance have the Ruby SCL enabled while running the installer? Current Kafo should have fixes for that, but I think those only landed after started to run the installer in a modern ruby (using Ruby SCL). That wasn’t in 1.23.

Using RVM can also trigger that.

No RVM or SCL loaded that I can see:

[root@foreman-puppet001 ~ #] rvm
-bash: rvm: command not found

[root@foreman-puppet001 rh #] echo $X_SCLS

[root@foreman-puppet001 rh #]

The package versions I listed were from after I reverted the install, here are the correct versions after enabling the repos and updating:

foreman-debug-1.24.3-1.el7.noarch
foreman-installer-1.24.3-1.el7.noarch
foreman-installer-katello-1.24.3-1.el7.noarch
foreman-proxy-1.24.3-1.el7.noarch
foreman-proxy-content-3.14.1-1.el7.noarch
foreman-puppet001.test.domain-apache-1.0-2.noarch
foreman-puppet001.test.domain-foreman-proxy-1.0-2.noarch
foreman-puppet001.test.domain-foreman-proxy-client-1.0-2.noarch
foreman-puppet001.test.domain-puppet-client-1.0-2.noarch
foreman-puppet001.test.domain-qpid-broker-1.0-2.noarch
foreman-puppet001.test.domain-qpid-client-cert-1.0-2.noarch
foreman-puppet001.test.domain-qpid-router-client-1.0-2.noarch
foreman-puppet001.test.domain-qpid-router-server-1.0-2.noarch
foreman-release-1.24.3-1.el7.noarch
foreman-selinux-1.24.3-1.el7.noarch
katello-agent-3.1.0-1.el7.noarch
katello-ca-consumer-foreman001.prod.domain-1.0-20.noarch
katello-certs-tools-2.6.0-1.el7.noarch
katello-client-bootstrap-1.7.4-1.el7.noarch
katello-common-3.14.1-1.el7.noarch
katello-debug-3.14.1-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.1.0-1.el7.noarch
katello-host-tools-fact-plugin-3.1.0-1.el7.noarch
katello-repos-3.14.1-1.el7.noarch
katello-selinux-3.1.1-1.el7.noarch
katello-server-ca-1.0-12.noarch
katello-service-3.14.1-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch
puppet-agent-5.5.22-1.el7.x86_64
puppet-agent-oauth-0.5.1-3.el7.noarch
puppetdb-termini-5.2.19-1.el7.noarch
puppetserver-5.3.16-1.el7.noarch
python-pulp-puppet-common-2.21.5-1.el7.noarch
ruby-2.0.0.648-36.el7.x86_64
rubygem-ansi-1.4.3-2.el7.noarch
rubygem-bigdecimal-1.2.0-36.el7.x86_64
rubygem-bundler-1.7.8-3.el7.noarch
rubygem-bundler_ext-0.4.1-3.el7.noarch
rubygem-clamp-1.1.2-4.el7.noarch
rubygem-concurrent-ruby-1.1.4-2.el7.noarch
rubygem-ffi-1.9.10-2.el7.x86_64
rubygem-foreman_maintain-0.5.4-1.el7.noarch
rubygem-gssapi-1.1.2-3.el7.noarch
rubygem-hashie-3.6.0-1.el7.noarch
rubygem-highline-1.7.8-4.el7.noarch
rubygem-io-console-0.4.2-36.el7.x86_64
rubygem-json-1.7.7-36.el7.x86_64
rubygem-jwt-1.2.1-1.el7.noarch
rubygem-kafo-4.0.0-1.el7.noarch
rubygem-kafo_parsers-1.0.0-1.el7.noarch
rubygem-kafo_wizards-0.0.1-2.el7.noarch
rubygem-little-plugger-1.1.4-1.el7.noarch
rubygem-logging-2.2.2-3.el7.noarch
rubygem-multi_json-1.12.2-3.el7.noarch
rubygem-net-http-persistent-2.8-5.el7.noarch
rubygem-powerbar-2.0.1-1.el7.noarch
rubygem-psych-2.0.0-36.el7.x86_64
rubygem-rack-1.6.12-1.el7.noarch
rubygem-rack-protection-1.5.3-3.el7.noarch
rubygem-rake-0.9.6-36.el7.noarch
rubygem-rb-inotify-0.9.7-3.el7.noarch
rubygem-rdoc-4.0.0-36.el7.noarch
rubygem-rkerberos-0.1.3-6.el7.x86_64
rubygem-rsec-0.4.3-2.el7.noarch
rubygem-rubyipmi-0.10.0-4.el7.noarch
rubygems-2.0.14.1-36.el7.noarch
rubygem-sinatra-1.4.8-2.el7.noarch
rubygem-smart_proxy_pulp-1.5.0-1.el7.noarch
rubygem-thor-0.19.1-1.el7.noarch
rubygem-tilt-2.0.7-1.el7.noarch
ruby-irb-2.0.0.648-36.el7.noarch
ruby-libs-2.0.0.648-36.el7.x86_64

And then running the installer:

[root@foreman-puppet001 ~ #] foreman-installer --scenario foreman-proxy-content --upgrade --certs-tar-file ~/-certs.tar --certs-update-all --certs-regenerate true --certs-deploy true/opt/puppetlabs/puppet/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require': cannot load such file -- kafo (LoadError) from /opt/puppetlabs/puppet/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require’
from /usr/sbin/foreman-installer:3:in `’
[root@foreman-puppet001 ~ #]

I am interested in the output of ENV. It looks like it’s trying to use the Puppet Ruby environment rather than the system Ruby. Can you also check what /usr/bin/foreman-installer uses as shebang? If it’s /usr/bin/env ruby then it may be sufficient to change it to /usr/bin/ruby to be sure the system Ruby is used.

Satellite upgrade 6.3 -> 6.4 fails with '/opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- kafo (LoadError)' - Red Hat Customer Portal says to move the puppet paths to the end of $PATH.

I did also find that forum thread mentioning foreman-installer using the wrong shebang, and changed it to /usr/bin/ruby.

I then got the following error:

[root@foreman-puppet001 ~ #] foreman-installer --scenario foreman-proxy-content --upgrade --certs-tar-file ~/`hostname -f`-certs.tar --certs-update-all --certs-regenerate true --certs-deploy truePlease unset the following environment variables before running the installer: http_proxy, https_proxy, ssl_cert_file, HTTP_PROXY, HTTPS_PROXY, SSL_CERT_FILE
Your system does not meet configuration criteria

Only SSL_CERT_FILE was set:

[root@foreman-puppet001 ~ #] unset SSL_CERT_FILE

And it looks like the installer has completed now.

Thanks for pointing me in the right direction.