Facing difficulties when trying to upgrade 2.4 to 2.5

Problem:
# dnf update -y
# dnf update -y https://yum.theforeman.org/releases/2.5/el8/x86_64/foreman-release.rpm
# dnf update -y https://yum.theforeman.org/katello/4.1/katello/el8/x86_64/katello-repos-latest.rpm
# dnf install -y centos-release-scl-rh
Katello 4.1 354 kB/s | 447 kB 00:01
Candlepin: an open source entitlement management system. 24 kB/s | 26 kB 00:01
pulpcore: Fetch, Upload, Organize, and Distribute Software Packages. 213 kB/s | 253 kB 00:01
No match for argument: centos-release-scl-rh

# dnf -y update
CentOS Linux 8 - AppStream                                                                   3.1 MB/s | 7.5 MB     00:02
CentOS Linux 8 - BaseOS                                                                      2.9 MB/s | 2.6 MB     00:00
CentOS Linux 8 - Extras                                                                       16 kB/s | 9.6 kB     00:00
CentOS Linux 8 - PowerTools                                                                  1.6 MB/s | 2.3 MB     00:01
Extra Packages for Enterprise Linux Modular 8 - x86_64                                       1.2 MB/s | 611 kB     00:00
Extra Packages for Enterprise Linux 8 - x86_64                                               804 kB/s | 9.5 MB     00:12
Foreman plugins 2.5                                                                          959 kB/s | 2.0 MB     00:02
Foreman 2.5                                                                                  1.4 MB/s | 1.6 MB     00:01
Katello 4.1                                                                                  559 kB/s | 447 kB     00:00
Candlepin: an open source entitlement management system.                                      43 kB/s |  26 kB     00:00
pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.                         375 kB/s | 253 kB     00:00
Puppet 6 Repository el 8 - x86_64                                                            2.6 MB/s |  28 MB     00:10
Error:
 Problem 1: package rubygem-websocket-driver-0.7.1-2.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-websocket-driver-0.7.1-1.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 2: package rubygem-unicode-0.4.4.4-4.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-unicode-0.4.4.4-3.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 3: package rubygem-unf_ext-0.0.7.2-4.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-unf_ext-0.0.7.2-3.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 4: package rubygem-ruby-libvirt-0.7.1-2.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-ruby-libvirt-0.7.1-1.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 5: package rubygem-rkerberos-0.1.5-20.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-rkerberos-0.1.5-19.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 6: package rubygem-puma-5.3.2-1.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-puma-5.1.1-1.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 7: package rubygem-pg-1.1.4-4.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-pg-1.1.4-3.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 8: package rubygem-nokogiri-1.10.9-2.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-nokogiri-1.10.9-1.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 9: package rubygem-nio4r-2.5.4-2.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-nio4r-2.5.4-1.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 10: package rubygem-ffi-1.12.2-2.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-ffi-1.12.2-1.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 11: package rubygem-bcrypt-3.1.12-4.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-bcrypt-3.1.12-3.el8.x86_64
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 12: package rubygem-foreman_remote_execution_core-1.4.3-1.el8.noarch requires rubygem(bcrypt_pbkdf), but none of the providers can be installed
  - package rubygem-bcrypt_pbkdf-1.1.0-1.el8.x86_64 requires libruby.so.2.7()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-foreman_remote_execution_core-1.4.0-1.el8.noarch
  - package ruby-libs-2.7.1-133.module_el8.3.0+429+625e5e4e.x86_64 is filtered out by modular filtering
 Problem 13: package foreman-2.5.0-1.el8.noarch requires rubygem(net-ssh) = 4.2.0, but none of the providers can be installed
  - cannot install both rubygem-net-ssh-5.1.0-2.el8.noarch and rubygem-net-ssh-4.2.0-2.el8.noarch
  - cannot install both rubygem-net-ssh-4.2.0-3.el8.noarch and rubygem-net-ssh-5.1.0-2.el8.noarch
  - cannot install the best update candidate for package rubygem-net-ssh-4.2.0-2.el8.noarch
  - cannot install the best update candidate for package foreman-2.4.1-1.el8.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Expected outcome:

Foreman and Proxy versions:
Foreman 2.4
Katello 4.0

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 8.3.2011
Other relevant data:
Docs iā€™ve follwed: Upgrading and Updating Foreman server

Check the installation docs to see which repositories are required.

https://docs.theforeman.org/2.5/Installing_Server_on_Red_Hat/index-katello.html#repositories-centos-8

1 Like

Iā€™ll just add that if you are using Foreman+Katello you have to follow the Katello upgrade docs as there can be differences in the procedures.

In this case, the specific issue you are seeing is due to the need to switch to the Ruby 2.7 stream of the Ruby module as we upgraded from Ruby 2.5 to Ruby 2.7 from Foreman 2.4 to Foreman 2.5.

1 Like

But currently, the upgrade docs are basically for centos7 only. So if you are upgrading on centos8 youā€™ll have to consult the installation docs as wellā€¦

1 Like

It isnā€™t very helpful to say ā€œyou have to follow the Katello upgrade docsā€ ā€¦ and then say you ā€œneed to switch to the Ruby 2.7 streamā€.

Do the Katello upgrade docs mention the need to switch to the Ruby 2.7 stream?
No, they do not.

Does your answer assist the user in the slightest way, to actually switch to the Ruby 2.7 stream?
No, it does not.

So i have some more questions:
Is it even feasible to switch to the Ruby 2.7 stream?
When i attempt to enable Ruby 2.7 it tells me I need to remove all Ruby 2.5 packages, before i can switch to 2.7.
How do I do this? Is it even possible to do this, without breaking foreman/katello completely? I am not sure it is going to be possible.
I have 200 ruby packages installed on my foreman server. If I remove those, it will take all of foreman with it.

Perhaps I could remove them all, without removing dependencies. Then i could update all my packages, including foreman, and perhaps it would then be able to pull in all of the Ruby 2.7 dependencies it needs. Maybe.

Is that the correct procedure?
Has anyone tested this?
Is this procedure documented in the Katello upgrade docs?
Will it be documented soon, if it isnā€™t yet?

Or are we going to keep pretending users can just happily upgrade from 2.4 to 2.5, without hitting this issue, and then make them feel like idiots, when they have to come in here and ask what they are doing wrong?

2 Likes

Aaaah i see, the installation docs DO mention the need to:

Enable Ruby 2.7 module:
dnf module reset ruby
dnf module enable ruby:2.7

But this is not mentioned at all, in the upgrade docs.

This is a big omission, is it not?
If you have 200 ruby 2.5 packages installed, in order to run foreman 2.4, itā€™s not a good idea to reset the ruby module and enable 2.7, is it?

When I try to enable ruby 2.7, it tells me I should remove all ruby 2.5 content firstā€¦ and obviously this is a problem, as it will completely break foreman 2.4.

What is the way forward here?
Is it even possible to successfully upgrade an el8 Foreman 2.4/Katello 4.0/Ruby 2.5 installation to Foreman 2.5/Katello 4.1/Ruby 2.7 ?

As I have mentioned before:

Itā€™s work in progress. They started the new docs for 2.4/4.0 and itā€™s incomplete or in some parts redhat satellite specific.

FWIW, I did:

dnf module reset ruby
dnf module enable ruby:2.7
rpm -e --nodeps rubygem-qpid_proton
dnf up

This seemed to be enough to make dnf happy.

TY John,
yep that seems to have worked for me.
Someone should add that to the 2.4 to 2.5 upgrade docsā€¦ because it IS essentialā€¦

Unreal.
I reset ruby, enabled 2.7, removed rubygem-qpid_proton, as John suggested.
I then managed to to a completely successful upgrade.
No errors showed on dnf update, and it upgraded foreman and katello, without any hint of any problems.

Now, after the upgrade, if I run ā€œdnf updateā€ to check things are looking nice and clean, i get this:

dnf update
Updating Subscription Management repositories.
Last metadata expiration check: 0:21:06 ago on Tue 17 Aug 2021 02:54:12 PM AEST.
Error:
Problem 1: package rubygem-sqlite3-1.4.2-2.el8.x86_64 requires libruby.so.2.5()(64bit), but none of the providers can be installed

  • cannot install the best update candidate for package rubygem-sqlite3-1.3.13-7.el8.x86_64
  • package ruby-libs-2.5.3-103.module+el8+2671+ebcc7ee0.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.3-104.module+el8.0.0+3250+4b7d6d43.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.5-105.module+el8.1.0+3656+f80bfa1d.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.5-106.module+el8.3.0+7153+c6f6daa5.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.9-107.module+el8.4.0+10822+fe4fffb1.x86_64 is filtered out by modular filtering
    Problem 2: package rubygem-qpid_proton-0.35.0-1.el8.x86_64 requires libruby.so.2.5()(64bit), but none of the providers can be installed
  • cannot install the best update candidate for package rubygem-qpid_proton-0.32.0-3.el8.x86_64
  • package ruby-libs-2.5.3-103.module+el8+2671+ebcc7ee0.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.3-104.module+el8.0.0+3250+4b7d6d43.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.5-105.module+el8.1.0+3656+f80bfa1d.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.5-106.module+el8.3.0+7153+c6f6daa5.x86_64 is filtered out by modular filtering
  • package ruby-libs-2.5.9-107.module+el8.4.0+10822+fe4fffb1.x86_64 is filtered out by modular filtering
    Problem 3: package foreman-2.5.2-1.el8.noarch requires rubygem(net-ssh) = 4.2.0, but none of the providers can be installed
  • cannot install both rubygem-net-ssh-5.1.0-2.el8.noarch and rubygem-net-ssh-4.2.0-3.el8.noarch
  • cannot install both rubygem-net-ssh-4.2.0-3.el8.noarch and rubygem-net-ssh-5.1.0-2.el8.noarch
  • cannot install the best update candidate for package rubygem-net-ssh-4.2.0-3.el8.noarch
  • cannot install the best update candidate for package foreman-2.5.2-1.el8.noarch
    (try to add ā€˜ā€“allowerasingā€™ to command line to replace conflicting packages or ā€˜ā€“skip-brokenā€™ to skip uninstallable packages or ā€˜ā€“nobestā€™ to use not only best candidate packages)

So i removed rubygem-qpid_proton before the upgrade, but now it is back, and it appears to still have a broken dependency on ruby 2.5

rpm -q foreman
foreman-2.5.2-1.el8.noarch

rpm -q rubygem-qpid_proton
rubygem-qpid_proton-0.32.0-3.el8.x86_64

rpm -q rubygem-net-ssh
rubygem-net-ssh-4.2.0-3.el8.noarch

Just ridiculous.

rubygem-sqlite3-1.4.2-2.el8.x86_64 comes out of EPEL 8. You must not enable EPEL 8 on your foreman server. EPEL 8 is not supported. See Installing Foreman 2.5 server with Katello 4.1 plugin on Enterprise Linux

Make sure to remove/downgrade all packages you have already installed from EPEL 8 before continuingā€¦

AH!

Thank you gvde.

I had epel 8 disabled, I only just enabled epel 8 AFTER the upgrade, because when i ran foreman-installer (to reconfigure after the updates), it completed without error, EXCEPT for this:

2021-08-17 14:41:41 [NOTICE] [configure] 750 configuration steps out of 2051 steps complete.
2021-08-17 14:41:53 [ERROR ] [configure] Execution of ā€˜/bin/dnf -d 0 -e 1 -y install python3-pulp-ansibleā€™ returned 1: Error:
2021-08-17 14:41:53 [ERROR ] [configure] Problem: package python3-pulp-ansible-1:0.9.0-1.el8.noarch requires python3-galaxy-importer >= 0.3.1, but none of the providers can be installed
2021-08-17 14:41:53 [ERROR ] [configure] - cannot install the best candidate for the job
2021-08-17 14:41:53 [ERROR ] [configure] - nothing provides ansible needed by python3-galaxy-importer-0.3.2-1.el8.noarch
2021-08-17 14:41:53 [ERROR ] [configure] - nothing provides /usr/bin/ansible-test needed by python3-galaxy-importer-0.3.2-1.el8.noarch
2021-08-17 14:41:53 [ERROR ] [configure] /Stage[main]/Pulpcore::Plugin::Ansible/Pulpcore::Plugin[ansible]/Package[python3-pulp-ansible]/ensure: change from ā€˜purgedā€™ to ā€˜presentā€™ failed: Execution of ā€˜/bin/dnf -d 0 -e 1 -y install python3-pulp-ansibleā€™ returned 1: Error:
2021-08-17 14:41:53 [ERROR ] [configure] Problem: package python3-pulp-ansible-1:0.9.0-1.el8.noarch requires python3-galaxy-importer >= 0.3.1, but none of the providers can be installed
2021-08-17 14:41:53 [ERROR ] [configure] - cannot install the best candidate for the job
2021-08-17 14:41:53 [ERROR ] [configure] - nothing provides ansible needed by python3-galaxy-importer-0.3.2-1.el8.noarch
2021-08-17 14:41:53 [ERROR ] [configure] - nothing provides /usr/bin/ansible-test needed by python3-galaxy-importer-0.3.2-1.el8.noarch

It was trying to install ansible, for remote execution, so i enabled epel, re-ran foreman-installer, and it was successful.

I now have these rpms from epel:

dnf list | grep ā€œ@epelā€
ansible.noarch 2.9.24-1.el8 @epel
ansible-test.noarch 2.9.24-1.el8 @epel
libsodium.x86_64 1.0.18-2.el8 @epel
python3-bcrypt.x86_64 3.1.6-2.el8.1 @epel
python3-bracex.noarch 2.1.1-2.el8 @epel
python3-dataclasses.noarch 0.8-3.el8 @epel
python3-mccabe.noarch 0.6.1-11.el8 @epel
python3-paramiko.noarch 2.4.3-1.el8 @epel
python3-pynacl.x86_64 1.3.0-5.el8 @epel
sshpass.x86_64 1.06-9.el8 @epel

So the question is, where is foreman supposed to get ansible from on el8, if not from epel?
Rgh
Thats a stupid question
I suppose I should be getting Ansible and so on from some official Red Hat repo.
Damn
ok i will see if i can enable the correct Red Hat repo, and replace those epel rpms with Red Hat ones.
And will delete Epel repo off this machine so i donā€™t forget, and enable it again at some point in future.

Look into the installation docs. It says which ansible to use: centos-release-ansible-29

Iā€™m running foreman/katello on Red Hat 8, not CentOS.
So i think Iā€™m probably better off using the Red Hat repos to get ansible 2.9.

Itā€™s probably time for docs to be updated, to show foreman 2.5 works on Red Hat 8 too, since this minor issue with ansible is the only issue Iā€™ve hit, which I would not also have hit on CentOS. And the Red Hat ansible 2.9 should work as well as the CentOS one.

There is a similar issue running on Oracle Linux 8, where no centos-release-ansible-29 package exists. Presumably I can just add this CentOS repo to my OL8 server and itā€™ll just workā€¦?

Sadly not so clean here:

$ sudo dnf list |grep epel
epel-release.noarch                                               8-10.el8                                              @@commandline
joe.x86_64                                                        4.6-6.el8                                             @epel8
libbsd.x86_64                                                     0.9.1-4.el8                                           @epel8
libretls.x86_64                                                   3.3.2-1.el8                                           @epel8
libwebsockets.x86_64                                              4.0.1-2.el8                                           @epel
netcat.x86_64                                                     1.217-3.el8                                           @epel8
python-idna-ssl.noarch                                            1.1.0-9.el8                                           @epel
python3-aiohttp.x86_64                                            3.7.4-1.el8                                           @epel
python3-async-timeout.noarch                                      3.0.1-8.el8                                           @epel
python3-django-filter.noarch                                      2.4.0-1.el8                                           @epel
python3-django-prometheus.noarch                                  2.1.0-1.el8                                           @epel
python3-inflection.noarch                                         0.5.1-1.el8                                           @epel
python3-prometheus_client.noarch                                  0.9.0-1.el8                                           @epel
python3-pyrsistent.x86_64                                         0.17.3-5.el8                                          @epel
python3-qpid-proton.x86_64                                        0.34.0-1.el8                                          @epel
python3-redis.noarch                                              3.5.3-1.el8                                           @epel
qpid-dispatch-router.x86_64                                       1.16.0-1.el8                                          @epel
qpid-proton-c.x86_64                                              0.34.0-1.el8                                          @epel
rubygem-gssapi.noarch                                             1.3.0-2.el8                                           @epel
rubygem-locale.noarch                                             2.1.2-3.el8.2                                         @epel
rubygem-mail.noarch                                               2.7.1-3.el8                                           @epel
rubygem-mime-types-data.noarch                                    3.2019.0331-1.el8                                     @epel
rubygem-mini_mime.noarch                                          1.1.0-1.el8                                           @epel
rubygem-rb-inotify.noarch                                         0.10.0-1.el8                                          @epel
screen.x86_64                                                     4.6.2-12.el8                                          @epel
epel-rpm-macros.noarch                                            8-22                                                  ol8_developer_EPEL
epel-rpm-macros.src                                               8-22                                                  ol8_developer_EPEL
lmdb-epel.src                                                     0.9.24-2.el8                                          ol8_developer_EPEL
oracle-epel-release-el8.src                                       1.0-3.el8                                             ol8_baseos_latest
oracle-epel-release-el8.x86_64                                    1.0-3.el8                                             ol8_baseos_latest

This was after installing the CentOS release RPMs with:

sudo dnf install \
  http://mirror.centos.org/centos-8/8/extras/x86_64/os/Packages/centos-release-ansible-29-1-2.el8.noarch.rpm \
  http://mirror.centos.org/centos-8/8/extras/x86_64/os/Packages/centos-release-configmanagement-1-1.el8.noarch.rpm

Well, a few manual removals of conflicting RPMs allowed a run of ā€˜dnf distro-syncā€™ to complete, which left my RPM dependencies happy, and only a few EPEL orphans, which look to be not related to Foreman/Katello.