Error after installation of Foreman/Katello 3.11.2

Problem:
Getting some strange errors related to ruby after install

------------------------------------------------------------------------------------------------------------
========================================================================================================================================================================================================================================
 Package                                                             Architecture                                     Version                                                   Repository                                         Size
========================================================================================================================================================================================================================================
Installing:
 foreman-installer-katello                                           noarch                                           1:3.11.2-1.el9                                            foreman                                            83 k
Installing dependencies:
 katello-certs-tools                                                 noarch                                           2.10.0-1.el9                                              foreman                                            65 k

Transaction Summary
========================================================================================================================================================================================================================================
Install  2 Packages

Total download size: 148 k
Installed size: 636 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): foreman-installer-katello-3.11.2-1.el9.noarch.rpm                                                                                                                                                830 kB/s |  83 kB     00:00    
(2/2): katello-certs-tools-2.10.0-1.el9.noarch.rpm                                                                                                                                                      625 kB/s |  65 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                   1.1 MB/s | 148 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                1/1 
  Installing       : katello-certs-tools-2.10.0-1.el9.noarch                                                                                                                                                                        1/2 
  Installing       : foreman-installer-katello-1:3.11.2-1.el9.noarch                                                                                                                                                                2/2 
  Running scriptlet: foreman-installer-katello-1:3.11.2-1.el9.noarch                                                                                                                                                                2/2 
<internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require': libruby.so.2.7: cannot open shared object file: No such file or directory - /usr/local/lib64/gems/ruby/stringio-3.0.2/stringio.so (LoadError)
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/clamp-1.3.2/lib/clamp/help.rb:3:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:5:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/clamp-1.3.2/lib/clamp.rb:5:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/kafo-7.4.0/lib/kafo/kafo_configure.rb:11:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/kafo-7.4.0/lib/kafo.rb:3:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:149:in `require'
        from /sbin/foreman-installer:3:in `<main>'
<internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- kafo (LoadError)
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /sbin/foreman-installer:3:in `<main>'
<internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require': libruby.so.2.7: cannot open shared object file: No such file or directory - /usr/local/lib64/gems/ruby/stringio-3.0.2/stringio.so (LoadError)
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/clamp-1.3.2/lib/clamp/help.rb:3:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:5:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/clamp-1.3.2/lib/clamp.rb:5:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/kafo-7.4.0/lib/kafo/kafo_configure.rb:11:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:96:in `require'
        from /usr/share/gems/gems/kafo-7.4.0/lib/kafo.rb:3:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:149:in `require'
        from /sbin/foreman-installer:3:in `<main>'
<internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- kafo (LoadError)
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /sbin/foreman-installer:3:in `<main>'
warning: %post(foreman-installer-katello-1:3.11.2-1.el9.noarch) scriptlet failed, exit status 1

Error in POSTIN scriptlet in rpm package foreman-installer-katello
  Verifying        : foreman-installer-katello-1:3.11.2-1.el9.noarch                                                                                                                                                                1/2 
  Verifying        : katello-certs-tools-2.10.0-1.el9.noarch                                                                                                                                                                        2/2 
Installed products updated.

Installed:
  foreman-installer-katello-1:3.11.2-1.el9.noarch                                                                        katello-certs-tools-2.10.0-1.el9.noarch                                                                       

Complete!


--------------------------------------------------------------------------------------------------------------------

Expected outcome:
I have expected the install to go fine without errors.

Foreman and Proxy versions:
Foreman 3.11.2

Foreman and Proxy plugin versions:

Distribution and version:
RHEL 9.4

Other relevant data:

On EL9 we use Ruby 3.0, but your stacktrace indicates 2.7 (which isn’t shipped natively on EL9 at all), where is that coming from?
Why is it looking in /usr/local/lib64/gems/?

I am a little clueless on this one, need some help from the GODs :slight_smile:

This is the output form my ruby packages installed on the box:

rpm -qa | grep -i ruby
rubygem-concurrent-ruby-edge-0.6.0-3.fm2_5.el8.noarch
ruby-libs-3.0.7-162.el9_4.x86_64
ruby-3.0.7-162.el9_4.x86_64
rubygem-bigdecimal-3.0.0-162.el9_4.x86_64
ruby-default-gems-3.0.7-162.el9_4.noarch
rubygem-bundler-2.2.33-162.el9_4.noarch
rubygem-io-console-0.5.7-162.el9_4.x86_64
rubygem-json-2.5.1-162.el9_4.x86_64
rubygem-rdoc-6.3.4.1-162.el9_4.noarch
rubygems-3.2.33-162.el9_4.noarch
ruby-devel-3.0.7-162.el9_4.x86_64
rubygem-irb-1.3.5-162.el9_4.noarch
rubygem-minitest-5.14.2-162.el9_4.noarch
rubygem-rake-13.0.3-162.el9_4.noarch
rubygem-highline-2.1.0-1.el9.noarch
rubygem-concurrent-ruby-1.1.10-1.el9.noarch
rubygem-rack-protection-2.2.4-1.el9.noarch
rubygem-i18n-1.14.5-1.el9.noarch
rubygem-tzinfo-2.0.6-1.el9.noarch
rubygem-kafo_wizards-0.0.2-2.el9.noarch
rubygem-zeitwerk-2.6.14-1.el9.noarch
rubygem-webrick-1.8.1-1.el9.noarch
rubygem-tilt-2.3.0-1.el9.noarch
rubygem-server_sent_events-0.1.3-1.el9.noarch
rubygem-ruby2_keywords-0.0.5-1.el9.noarch
rubygem-mustermann-2.0.2-1.el9.noarch
rubygem-netrc-0.11.0-6.el9.noarch
rubygem-mime-types-data-3.2024.0206-1.el9.noarch
rubygem-mime-types-3.5.2-1.el9.noarch
rubygem-little-plugger-1.1.4-3.el9.noarch
rubygem-logging-2.3.1-1.el9.noarch
rubygem-kafo_parsers-1.2.1-1.el9.noarch
rubygem-http-accept-1.7.0-1.el9.noarch
rubygem-hashie-5.0.0-1.el9.noarch
rubygem-powerbar-2.0.1-3.el9.noarch
rubygem-excon-0.110.0-1.el9.noarch
rubygem-domain_name-0.6.20240107-1.el9.noarch
rubygem-http-cookie-1.0.5-1.el9.noarch
rubygem-clamp-1.3.2-1.el9.noarch
rubygem-ansi-1.5.0-3.el9.noarch
rubygem-kafo-7.4.0-1.el9.noarch
rubygem-rest-client-2.1.0-1.el9.noarch
rubygem-redfish_client-0.5.4-1.el9.noarch
rubygem-sinatra-2.2.4-1.el9.noarch
rubygem-xmlrpc-0.3.3-1.el9.noarch
rubygem-activesupport-6.1.7.8-1.el9.noarch
rubygem-rubyipmi-0.11.1-2.el9.noarch
rubygem-mqtt-0.5.0-1.el9.noarch
rubygem-statsd-instrument-2.9.2-1.el9.noarch
rubygem-sequel-5.80.0-1.el9.noarch
rubygem-sd_notify-0.1.1-1.el9.noarch
rubygem-rsec-0.4.3-5.el9.noarch
rubygem-pg-1.5.6-1.el9.x86_64
rubygem-net-ssh-7.2.3-1.el9.noarch
rubygem-jwt-2.7.1-1.el9.noarch
rubygem-bundler_ext-0.4.1-6.el9.noarch
rubygem-apipie-params-0.0.5-5.el9.noarch
rubygem-algebrick-0.7.5-1.el9.noarch
rubygems-devel-3.2.33-162.el9_4.noarch
rubygem-rack-2.2.4-1.el9.noarch
rubygem-multi_json-1.15.0-3.el9.noarch
rubygem-nokogiri-1.15.6-1.el9.x86_64
rubygem-rails-dom-testing-2.2.0-1.el9.noarch
rubygem-mini_mime-1.1.5-1.el9.noarch
rubygem-mail-2.8.1-2.el9.noarch
rubygem-activemodel-6.1.7.8-1.el9.noarch
rubygem-activerecord-6.1.7.8-1.el9.noarch
rubygem-websocket-extensions-0.1.5-2.el9.noarch
rubygem-websocket-driver-0.7.6-1.el9.x86_64
rubygem-thor-1.3.0-1.el9.noarch
rubygem-sprockets-4.2.1-1.el9.noarch
rubygem-rack-test-2.1.0-1.el9.noarch
rubygem-nio4r-2.7.3-1.el9.x86_64
rubygem-method_source-1.1.0-1.el9.noarch
rubygem-marcel-1.0.4-1.el9.noarch
rubygem-globalid-1.2.1-1.el9.noarch
rubygem-activejob-6.1.7.8-1.el9.noarch
rubygem-erubi-1.12.0-1.el9.noarch
rubygem-crass-1.0.6-2.el9.noarch
rubygem-loofah-2.22.0-1.el9.noarch
rubygem-rails-html-sanitizer-1.6.0-1.el9.noarch
rubygem-builder-3.2.4-5.el9.noarch
rubygem-actionview-6.1.7.8-1.el9.noarch
rubygem-actionpack-6.1.7.8-1.el9.noarch
rubygem-activestorage-6.1.7.8-1.el9.noarch
rubygem-actionmailbox-6.1.7.8-1.el9.noarch
rubygem-actiontext-6.1.7.8-1.el9.noarch
rubygem-actioncable-6.1.7.8-1.el9.noarch
rubygem-actionmailer-6.1.7.8-1.el9.noarch
rubygem-railties-6.1.7.8-1.el9.noarch
rubygem-sprockets-rails-3.4.2-1.el9.noarch
rubygem-rails-6.1.7.8-1.el9.noarch
rubygem-psych-3.3.2-162.el9_4.x86_64

How was this machine set up?
The presence of rubygem-concurrent-ruby-edge-0.6.0-3.fm2_5.el8.noarch is confusing me.

Hi @evgeni , this machine was recently upgraded from RHEL 8 to RHEL 9, perhaps that package was leftover

Possible.
Was there a Foreman installation before on that machine?
How was it upgraded?

Hi @evgeni , yes there was a foreman/katello install on it before on this box…this is basiclaly a proxy box. The upgrade was perform using leapp following the standard RHEL documents.

Were you following Upgrading Foreman to 3.11 and using our leapp packages, or the ones from RHEL?

I actually upgraded the system using the RHEL 8 to RHEL 9 procedures provided by RH, then I proceeded to upgrade the packages as per the Foreman/Katello documents. I already did this with a few other proxies without issues and it all worked great, but this one is giving me an actitutude with ruby:-)

Yeah, no idea what’s happening here. It seems there is an (unpackaged?) Ruby 2.7 lying around on your system, breaking things.

My ruby when I checked is showing correctly:

ruby -v
ruby 3.0.7p220 (2024-04-23 revision 724a071175) [x86_64-linux]

I am not sure abou tthat old reference to 2.7 or where is coming from, need some help on this one.

@jeremylenz , any clues as what may be happening on thi sone with ruby?

Team,
disregard this…since this was only a proxy I end up creating a new box and rebuilding it. Spend to much time tryuing to figure out the ruby issue, thank you all for the input.