Multiple issues trying to upgrade from 3.2 to 3.3 (Eventually 3.5)

Support
1

Problem:

Can’t upgrade from Foreman 3.2 to 3.3. See Other relevant data for more context.

Expected outcome:

To have a foreman 3.3 server after following upgrade instructions

Foreman and Proxy versions:

Foreman 3.2 w/Katello 4.4

Other relevant data:

Hello, I’m relatively new to Foreman and I’ve been tasked to upgrade our production server from 3.2 to 3.5. Reading through the documentation, I need to upgrade to 3.3 and 3.4 before I can upgrade to 3.5.

I’ve setup a development server to figure out the process, but I’ve had very little luck with anything I’ve tried. I’m running Rocky Linux 8.7 for the OS and I followed these instructions to get a clean install of Foreman 3.2/Katello 4.4…

https://docs.theforeman.org/3.2/Quickstart/index-katello.html

That installs without issue and I am able to login to my newly created instance of Foreman 3.2. From there, I proceeded to follow these instructions to upgrade to version 3.3…

https://docs.theforeman.org/3.3/Upgrading_and_Updating/index-katello.html#

I don’t have anything modified, nor do I care to create a backup of the development server, so I skip to step 6. Everything executes fine, until I get to step 10, ‘yum -y update’. It errors out with this message,

Error:

 Problem: package rubygem-qpid_proton-0.37.0-1.el8.x86_64 requires libruby.so.2.5()(64bit), but none of the providers can be installed

  - cannot install the best update candidate for package rubygem-qpid_proton-0.35.0-1.el8.x86_64

  - package ruby-libs-2.5.9-110.module+el8.6.0+992+fc951c18.x86_64 is filtered out by modular filtering

(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

I figured skipping broken wouldn’t be the best idea, so I decided to look into migrating my 3.2 server to a fresh install of Foreman 3.5. Before I got started, I wanted to backup my production server, just in case. Here, I ran into more issues. I followed the instructions from here…

https://docs.theforeman.org/3.2/Administering_Project/index-katello.html#Performing_a_Full_Backup_admin

The first issue I ran into was.

No such file or directory - /etc/foreman-installer/scenarios.d/last_scenario.yaml (Errno::ENOENT)

That was easily resolved by creating that file. However, I was greeted with more errors that required me to white-list a bunch of things that seemed critical. After running into road blocks using ‘foreman-maintain’, I found that I could make a backup with ‘foreman-rake db:dump’. But that did not seem to have the outcome I was anticipating. I am able to backup and restore on my development server of Foreman => 3.2.

So now, I am at my wits end. Can anyone tell me what I am doing wrong and how to get on the right track?

Thanks.

2

Ok, looks like I was able to update using the --nobest option. I’m not sure if that is documented anywhere, but it should be.

3

You shouldn‘t use —nobest. Many components depend on each other and require the correct version to work.

Did you reset the ruby module before the upgrade as mentioned in the docs?

4

I reset the ruby module while installing 3.2, but I don’t see any instructions to do that in the upgrade doc for 3.3.

What step in the upgrade do I need to reset the ruby module?

5

You have to check the installation docs as well

https://docs.theforeman.org/3.3/Installing_Server/index-katello.html#configuring-repositories_foreman

6

Good ol’ Appendix A. I’ll give that a try in a bit.

7

Resetting the ruby module didn’t seem to fix the issue. The instructions were kind of vague on when I am supposed to do that step.

If the module fails to enable, it can mean an incorrect module is enabled. In that case, you have to resolve dependencies manually as follows. List the enabled modules:

# dnf module list --enabled

Ruby

If the Ruby 2.5 module has already been enabled, perform a module reset:

# dnf module reset ruby

I ended up resetting ruby module before cleaning the cache, but that didn’t seem to help. Here are those results.

dnf module reset ruby

Modular dependency problem:

Problem: module foreman:el8:30320220526202658:fea1c427.x86_64 requires module(ruby:2.7), but none of the providers can be installed

  • module ruby:2.5:8060020220801093653:d63f516d.x86_64 conflicts with module(ruby:2.7) provided by ruby:2.7:8060020220913075839:d63f516d.x86_64

  • module ruby:2.7:8060020220913075839:d63f516d.x86_64 conflicts with module(ruby:2.5) provided by ruby:2.5:8060020220801093653:d63f516d.x86_64

  • conflicting requests

Dependencies resolved.

Nothing to do.

Complete!

I even tried to ensure the correct version was being used with these commands…

# dnf module reset ruby
# dnf module enable ruby:2.7
# dnf module install ruby:2.7
8

Giving up on upgrading. Going to try to migrate instead. Thanks for the help.

9

Second thought, I’ll still need to figure this out. Just realized that I wont be able to migrate from 3.2 to 3.5.

Any addition help here would be greatly appreciated.

10

So you have the installation guide and checked which modules you have to enable?

Please post the output of

# dnf module list
# dnf repolist
11

Yes. I have a clean installation using the quickstart guide for 3.2. I followed the instructions for RHEL8 since I am using Rocky Linux. I’ve edited the list to show only the most relevant info, but let me know if you feel like I am missing anything important.

These are my enabled modules at this point…

# dnf module list

postgresql	9.6
postgresql	10 [d]
postgresql	12 [e]
postgresql	13
python27	2.7 [d]
python36	3.6 [d]
python38	3.8 [d][e]
python39	3.9 [d][e]
redis	5 [d][e]
redis	6
rhn-tools	1.0 [d]
ruby	2.5 [d]
ruby	2.6
ruby	2.7 [e]
ruby	3
ruby	3.1
rust-toolset	rhel8 [d]
satellite-5-client	1.0 [d]
scala	2.10 [d]
squid	4 [d]
subversion	1.10 [d]
subversion	1.14
swig	3.0 [d]
swig	4
varnish	6 [d]
virt	rhel [d][e]
	
Rocky-PowerTools	
Name	Stream
javapackages-tools	201801
mariadb-devel	10.3
python38-devel	3.8
python39-devel	3.9
subversion-devel	1.1
virt-devel	rhel
	
Foreman 3.2	
Name	Stream
foreman	el8 [e]
	
Katello 4.4	
Name	Stream
katello	el8 [e]
	
pulpcore: Fetch, Uplo	ad, Organize,
Name	Stream
pulpcore	el8 [e]



# dnf repolist

Rocky-BaseOS |Rocky-BaseOS|
Rocky-Devel |Rocky-Devel|
Rocky-Extras|Rocky-Extras|
Rocky-HighAvailability |Rocky-HighAvailability|
Rocky-NFV |Rocky-NFV|
Rocky-Plus |Rocky-Plus|
Rocky-PowerTools |Rocky-PowerTools|
Rocky-RT |Rocky-RT|
Rocky-ResilientStorage |Rocky-ResilientStorage|
choria_release |Choria Orchestrator Releases|
devtoolset |devtoolset repo|
epel |epel|
epel-multimedia |epel-multimedia|
foreman |Foreman 3.2|
foreman-plugins |Foreman plugins 3.2|
katello |Katello 4.4|
katello-candlepin |Candlepin: an open source entitlement management system.|
pulpcore |pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.|
puppet7 |Puppet 7 Repository el 8|

After I start the upgrade process to 3.3 and before resetting the ruby module …

# dnf module list

postgresql	9.6
postgresql	10 [d]
postgresql	12 [e]
postgresql	13
python27	2.7 [d]
python36	3.6 [d]
python38	3.8 [d][e]
python39	3.9 [d][e]
redis	5 [d][e]
redis	6
rhn-tools	1.0 [d]
ruby	2.5 [d]
ruby	2.6
ruby	2.7 [e]
ruby	3
ruby	3.1
rust-toolset	rhel8 [d]
satellite-5-client	1.0 [d]
scala	2.10 [d]
squid	4 [d]
subversion	1.10 [d]
subversion	1.14
swig	3.0 [d]
swig	4
varnish	6 [d]
virt	rhel [d][e]
	
Rocky-PowerTools	
Name	Stream
javapackages-tools	201801
mariadb-devel	10.3
python38-devel	3.8
python39-devel	3.9
subversion-devel	1.1
virt-devel	rhel
	
Foreman 3.3	
Name	Stream
foreman	el8 [e]
	
Katello 4.5	
Name	Stream
katello	el8 [e]
	
pulpcore: Fetch, Uplo	ad, Organize, an
Name	Stream
pulpcore	el8 [e]


# dnf repolist

Rocky-AppStream - Rocky-AppStream
Rocky-BaseOS - Rocky-BaseOS
Rocky-Devel - Rocky-Devel
Rocky-Extras - Rocky-Extras
Rocky-HighAvailability - Rocky-HighAvailability
Rocky-NFV - Rocky-NFV
Rocky-Plus -  Rocky-Plus
Rocky-PowerTools  -  Rocky-PowerTools
Rocky-RT  -  Rocky-RT
Rocky-ResilientStorage -  Rocky-ResilientStorage
choria_release -  Choria Orchestrator Releases
devtoolset  -   devtoolset repo
epel - epel
epel-multimedia - epel-multimedia
foreman  -  Foreman 3.3
foreman-plugins  -  Foreman plugins 3.3
katello -   Katello 4.5
katello-candlepin  -  Candlepin: an open source entitlement management system.
pulpcore   -  pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
puppet7  -  Puppet 7 Repository el 8 - x86_64

And finally, here are those commands after i reset ruby…

postgresql           9.6            
postgresql           10 [d]       
postgresql           12 [e]       
postgresql           13              
python27             2.7 [d]        
python36             3.6 [d]        
python38             3.8 [d][e]    
python39             3.9 [d][e]   
redis                5 [d][e]    
redis                6     
rhn-tools            1.0 [d]    
ruby                 2.5 [d]  
ruby                 2.6         
ruby                 2.7   
ruby                 3.0            
ruby                 3.1          
rust-toolset         rhel8 [d]       
satellite-5-client   1.0 [d]      
scala                2.10 [d]    
squid                4 [d]          
subversion           1.10 [d]     
subversion           1.14         
swig                 3.0 [d]       
swig                 4.0          
varnish              6 [d]          
virt                 rhel [d][e]     

Rocky-PowerTools

javapackages-tools   201801        
mariadb-devel        10.3               
python38-devel       3.8                                 
python39-devel       3.9                                 
subversion-devel     1.10                           
virt-devel           rhel                  

Foreman 3.3
foreman              el8 [e]  

Katello 4.5
katello              el8 [e]     

pulpcore: 
pulpcore             el8 [e]                 

# dnf repolist

Rocky-BaseOS
Rocky-Devel
Rocky-Extras
Rocky-HighAvailability
Rocky-NFV
Rocky-Plus
Rocky-PowerTools
Rocky-RT
Rocky-ResilientStorage
choria_release
devtoolset
epel
epel-multimedia
foreman 3.3
foreman-plugins 3.3
katello 4.5
katello-candlepin
pulpcore
puppet7

Doesn’t seem to matter which one is enabled, I get the same outcome after running

yum/dnf -y update

12

From the guide you linked to https://docs.theforeman.org/3.2/Quickstart/index-katello.html:

“EPEL is not supported.”

…but you have EPEL enabled… I haven’t tried to recreate, but perhaps that is your issue?

13

As a sanity check, I:

  • built a new Rocky 8.7 VM
  • installed Foreman 3.2 (with Katello) - see below lines 1 through 18
  • upgraded to Foreman 3.3 - see below lines 19 through 27

…and here’s my entire bash history of that process - did not run into issues. I do think your issue is probably EPEL and I will try to test that as well:

[root@lab-vm1 ~]# history
    1  ls
    2  dnf localinstall https://yum.theforeman.org/releases/3.2/el8/x86_64/foreman-release.rpm
    3  dnf clean all
    4  dnf update
    5  dnf clean all
    6  dnf localinstall https://yum.theforeman.org/releases/3.2/el8/x86_64/foreman-release.rpm
    7  dnf localinstall https://yum.theforeman.org/releases/3.2/el8/x86_64/foreman-release.rpm -y
    8  dnf localinstall https://yum.theforeman.org/katello/4.4/katello/el8/x86_64/katello-repos-latest.rpm -y
    9  dnf localinstall https://yum.puppet.com/puppet7-release-el-8.noarch.rpm -y
   10  dnf module reset ruby -y
   11  dnf module enable ruby:2.7 -y
   12  dnf config-manager --set-enabled powertools
   13  dnf module enable postgresql:12
   14  dnf update
   15  dnf install foreman-installer-katello -y
   16  foreman-installer --scenario katello
   17  systemctl stop firewalld
   18  history
   19  foreman-rake katello:upgrade_check
   20   yum update -y
   21  dnf update -y https://yum.theforeman.org/releases/3.3/el8/x86_64/foreman-release.rpm                 https://yum.theforeman.org/katello/4.5/katello/el8/x86_64/katello-repos-latest.rpm -y
   22  dnf module enable -y katello:el8 pulpcore:el8 -y
   23  yum clean all
   24  yum -y update
   25  foreman-maintain service stop
   26  foreman-installer
   27  history
[root@lab-vm1 ~]#
14

It looks like you are following slightly different instructions, than I am. I was following RHEL 8.

Let me run through it with the commands you followed.

15

If you’re using CentOS Stream, Rocky Linux, Alma Linux, or any of the other RHEL clones, you’ll want to follow the “CentOS” section of the documentation as the RHEL repo names and commands to enable them are slightly different.

I’m still in the process of testing the install & upgrade with epel installed, but I see more clues that’s your issue with this:

[root@lab-vm1 ~]# dnf list installed | grep epel
epel-release.noarch                                8-18.el8                                   @extras           
python38-click.noarch                              8.1.2-2.el8                                @epel             
python38-dateutil.noarch                           2.8.2-2.el8                                @epel             
qpid-proton-c.x86_64                               0.37.0-1.el8                               @epel             
rubygem-gssapi.noarch                              1.3.0-2.el8                                @epel             
rubygem-locale.noarch                              2.1.2-3.el8.2                              @epel             
rubygem-mail.noarch                                2.7.1-3.el8                                @epel             
rubygem-mime-types-data.noarch                     3.2019.0331-1.el8                          @epel             
rubygem-mini_mime.noarch                           1.1.0-1.el8                                @epel             
rubygem-rack.noarch                                2.2.4-1.el8                                @epel             
rubygem-rb-inotify.noarch                          0.10.0-1.el8                               @epel             
[root@lab-vm1 ~]#

All of those packages (except for epel-release) have been installed as dependencies or through the foreman-installer process. I’m going to still continue through the install & upgrade process to see if I can get your same error… but all signs are pointing to EPEL being enabled as your issue.

16

Ok - I was able to recreate your issue. The only difference is I had enabled epel (see line 2: “dnf install epel-release”) - otherwise for the most part I copy/pasted the same install commands from my working example above:


[root@lab-vm1 ~]# yum update -y
Last metadata expiration check: 0:36:38 ago on Mon 20 Mar 2023 01:59:14 PM CDT.
Error: 
 Problem 1: package rubygem-qpid_proton-0.37.0-1.el8.x86_64 requires libruby.so.2.5()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package rubygem-qpid_proton-0.35.0-1.el8.x86_64
  - package ruby-libs-2.5.9-110.module+el8.6.0+992+fc951c18.x86_64 is filtered out by modular filtering
 Problem 2: package foreman-3.2.1-1.el8.noarch requires rubygem(net-ssh) = 4.2.0, but none of the providers can be installed
  - cannot install both rubygem-net-ssh-5.1.0-2.el8.noarch and rubygem-net-ssh-4.2.0-3.el8.noarch
  - cannot install both rubygem-net-ssh-4.2.0-3.el8.noarch and rubygem-net-ssh-5.1.0-2.el8.noarch
  - cannot install the best update candidate for package rubygem-net-ssh-4.2.0-3.el8.noarch
  - cannot install the best update candidate for package foreman-3.2.1-1.el8.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@lab-vm1 ~]# history
    1  history
    2  dnf install epel-release
    3  dnf localinstall https://yum.theforeman.org/releases/3.2/el8/x86_64/foreman-release.rpm
    4  dnf update
    5  dnf localinstall https://yum.theforeman.org/releases/3.2/el8/x86_64/foreman-release.rpm -y
    6  dnf localinstall https://yum.theforeman.org/katello/4.4/katello/el8/x86_64/katello-repos-latest.rpm -y
    7  dnf localinstall https://yum.puppet.com/puppet7-release-el-8.noarch.rpm -y
    8  dnf module reset ruby -y
    9  dnf module enable ruby:2.7 -y
   10  dnf config-manager --set-enabled powertools
   11  dnf module enable postgresql:12
   12  dnf update -y
   13  dnf install foreman-installer-katello -y
   14  foreman-installer --scenario katello
   15  foreman-rake katello:upgrade_check
   16  yum update -y
   17  history
[root@lab-vm1 ~]#

Hopefully that helps!

17

This is 100% at your own risk, but on a test VM, I was able to “recover” from the bad state with these 2 commands:

dnf remove epel-release -y
dnf downgrade $(dnf list installed | grep @epel | awk '{ print $1 }' ORS=' ') -y

and from there I was able to continue the upgrade process with:

yum update -y
dnf update -y https://yum.theforeman.org/releases/3.3/el8/x86_64/foreman-release.rpm                 https://yum.theforeman.org/katello/4.5/katello/el8/x86_64/katello-repos-latest.rpm -y
dnf module enable -y katello:el8 pulpcore:el8 -y
yum clean all
yum -y update
foreman-maintain service stop
foreman-installer
1 Like
18

Thank you so much. I haven’t run through all of this just yet, but I think you might have figured out the issue. I am going to rip epel from my image and try again. I’ll let you know if that fixed it for me.

1 Like
19

First, thank you. I really appreciate you and everyone else that helped me resolve this issue.

It was definitely the epel repo causing my problem. After disabling puppet and the epel repo, I was able to upgrade to 3.3.

Initially, I wasn’t able to install 3.2 using the steps you took. Kept giving me an error executing dnf. I decided to give the RHEL 8 instructions a try since I got it working previously. That seemed to have done the trick after a slight modification. Please let me know if that was another no-no.

dnf clean all
dnf install -y https://yum.theforeman.org/releases/3.2/el8/x86_64/foreman-release.rpm
dnf install -y https://yum.theforeman.org/katello/4.4/katello/el8/x86_64/katello-repos-latest.rpm
dnf install -y https://yum.puppet.com/puppet7-release-el-8.noarch.rpm
dnf module reset ruby -y
dnf module enable ruby:2.7 -y
dnf module enable -y katello:el8 pulpcore:el8
dnf update -y
dnf install -y foreman-installer-katello
foreman-installer --scenario katello

After it was installed, I followed your directions to upgrade to 3.3 and it seemed to have worked.

20

Just to follow up on this…

Additionally, I was able to upgrade to 3.4 and 3.5 without issues.