Yet another Foreman upgrade fail

My foreman server is registered to itself, i.e. usually I pull updates from itself. Those repositories are written into /etc/yum.repos.d/redhat.repo by subscription manager. Usually, we have the original base os repositories disabled so that all updates are delivered by our foreman server.

I didn’t disable the repos from the foreman server but actually removed the redhat.repo file to avoid them being used. In foreman, the foreman server itself had 3.10 and 3.11 repositories enabled because usually that doesn’t cause any issues, unlike this time.

During the whole procedure it shows some conflicts in regard to the postgresql module. This is to be expected because there actually is a conflict and some point. But unlike during my first attempt when 3.10 and 3.11 both were enabled and the switch simply failed, the switch works if only 3.11 is enabled.

As I posted above: the switch does show a conflict but proceeds anyway. The conflict is correct. It gets resolved by installing foreman 3.11.

I did the switch manually as suggested in the docs. foreman-installer runs another switch postgresql-13 at the beginning before upgrading the database, but that doesn’t do anything at all because postgresql-13 has already been enabled. If all necessary repositories are enabled and accessible foreman-installer should just work.

Ok thanks, but from what you are saying if all your content (served from Foreman) is in the usual redhat.repo and you’ve removed that, then you wont have anything for the OS? Base, appstream etc.

Mine is serving content directly via the internet using the repo files in /etc/yum.repos.d
Like I mentioned I usually keep al disabled until needed, like an OS update or Foreman upgrade

3.10 is definitely disabled on mine, only 3.11 enabled after installing it. The conflicts are for postgres versions via the appstream repo. The conflicts do not go away for me after installing 3.11 like you are seeing.

i have also tried enabling all repos just in case it helped, see list below (which is before I enabled and before I installed latest foreman/katello rpms)

I don’t know why my installer is unhappy then

candlepin                                                                                                                             Candlepin: an open source entitlement management system.                                                                                                                        disabled
candlepin-source                                                                                                                      Katello Candlepin source                                                                                                                                                        disabled
epel                                                                                                                                  Epel                                                                                                                                                                            disabled
foreman                                                                                                                               Foreman 3.10                                                                                                                                                                    disabled
foreman-plugins                                                                                                                       Foreman plugins 3.10                                                                                                                                                            disabled
foreman-plugins-source                                                                                                                Foreman plugins 3.10 - source                                                                                                                                                   disabled
foreman-source                                                                                                                        Foreman 3.10 - source                                                                                                                                                           disabled
katello                                                                                                                               Katello 4.12                                                                                                                                                                    disabled
katello-source                                                                                                                        Katello 4.12 Source                                                                                                                                                             disabled
ol8_MODRHCK                                                                                                                           Latest RHCK with fixes from Oracle for Oracle Linux 8 (x86_64)                                                                                                                  disabled
ol8_addons                                                                                                                            Oracle Linux 8 Addons (x86_64)                                                                                                                                                  enabled
ol8_appstream                                                                                                                         Oracle Linux 8 Application Stream (x86_64)                                                                                                                                      enabled
ol8_baseos_latest                                                                                                                     Oracle Linux 8 BaseOS Latest (x86_64)                                                                                                                                           enabled
ol8_codeready_builder                                                                                                                 Oracle Linux 8 CodeReady Builder (x86_64) - Unsupported                                                                                                                         disabled
ol8_distro_builder                                                                                                                    Oracle Linux 8 Distro Builder (x86_64) - Unsupported                                                                                                                            disabled
ol8_kvm_appstream                                                                                                                     Oracle Linux 8 KVM Application Stream (x86_64)                                                                                                                                  disabled
ol8_u0_baseos_base                                                                                                                    Oracle Linux 8 BaseOS GA (x86_64)                                                                                                                                               disabled
ol8_u1_baseos_base                                                                                                                    Oracle Linux 8.1 BaseOS (x86_64)                                                                                                                                                disabled
ol8_u2_baseos_base                                                                                                                    Oracle Linux 8.2 BaseOS (x86_64)                                                                                                                                                disabled
ol8_u3_baseos_base                                                                                                                    Oracle Linux 8.3 BaseOS (x86_64)                                                                                                                                                disabled
ol8_u4_baseos_base                                                                                                                    Oracle Linux 8.4 BaseOS (x86_64)                                                                                                                                                disabled
ol8_u4_security_validation                                                                                                            Oracle Linux 8 Update 4 (x86_64) Security Validations                                                                                                                           enabled
ol8_u5_baseos_base                                                                                                                    Oracle Linux 8.5 BaseOS (x86_64)                                                                                                                                                disabled
ol8_u6_baseos_base                                                                                                                    Oracle Linux 8.6 BaseOS (x86_64)                                                                                                                                                disabled
ol8_u7_baseos_base                                                                                                                    Oracle Linux 8.7 BaseOS (x86_64)                                                                                                                                                disabled
ol8_u8_baseos_base                                                                                                                    Oracle Linux 8.8 BaseOS (x86_64)                                                                                                                                                disabled
pulpcore                                                                                                                              pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.                                                                                                            disabled
pulpcore-source                                                                                                                       pulpcore source                                                                                                                                                                 disabled
puppet7                                                                                                                               Puppet 7 Repository el 8 - x86_64

No. I have enabled those. Just look at the repolist above. It shows all repos enabled before I started.

It’s hard to tell without seeing the output. What happened during the switch? What happened during the update, when 3.11 repos were enabled?

The conflicts must go away once 3.11 got installed because there is no conflict then. If you still have a conflict, then either the switch didn’t work or the update failed in parts.

ok. With mine, the conflict shows itself before I even run the switch. It’s flagged up when trying to disable the pulpcore module. It’s in my post above:

Then I try running the switch and it shows the same conflicts, which i get, but still allows the install of postgres13

There is no conflict during initial dnf update, it’s only there after I’ve installed foreman 3.11

Yes. But that’s O.K. The conflict is there. It will be reported. It’s expected. After the update of the foreman/katello repos to 3.11/4.13 there is only the module for 3.11/4.13 available and enabled. The foreman module for 3.11 requires postgresql-13. postgresql-12 is installed. You cannot have both at the same time. It’s just the state at that moment.

I see just the same:

# dnf module disable pulpcore
Foreman 3.11                                                                                    5.9 MB/s | 1.7 MB     00:00    
Foreman plugins 3.11                                                                            6.9 MB/s | 1.9 MB     00:00    
Katello 4.13                                                                                    1.2 MB/s | 307 kB     00:00    
Candlepin: an open source entitlement management system.                                        270 kB/s |  62 kB     00:00    
pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.                            1.2 MB/s | 308 kB     00:00    
Modular dependency problem:

 Problem: module foreman:el8:31120240718202416:34d643a2.x86_64 from foreman requires module(postgresql:13), but none of the providers can be installed
  - module postgresql:12:8090020240227082527:9edba152.x86_64 from appstream conflicts with module(postgresql:13) provided by postgresql:13:8090020240415140528:9edba152.x86_64 from appstream
  - module postgresql:13:8090020240415140528:9edba152.x86_64 from appstream conflicts with module(postgresql:12) provided by postgresql:12:8090020240227082527:9edba152.x86_64 from appstream
  - conflicting requests
Unable to resolve argument pulpcore
Error: Problems in request:
missing groups or modules: pulpcore
Modular dependency problems:

 Problem: module foreman:el8:31120240718202416:34d643a2.x86_64 from foreman requires module(postgresql:13), but none of the providers can be installed
  - module postgresql:12:8090020240227082527:9edba152.x86_64 from appstream conflicts with module(postgresql:13) provided by postgresql:13:8090020240415140528:9edba152.x86_64 from appstream
  - module postgresql:13:8090020240415140528:9edba152.x86_64 from appstream conflicts with module(postgresql:12) provided by postgresql:12:8090020240227082527:9edba152.x86_64 from appstream
  - conflicting requests

And you don’t have to disable pulpcore anyway, because you already did that for an earlier version and there is no pulpcore module anymore.

Is that error message you have posted the latest? Because it’s different and somehow odd: it is missing the foreman module, too. What modules do you have? Which repos do you have enabled? It looks to me as if you either don’t have the foreman module enabled or the foreman repo is missing.

# dnf module list foreman katello postgresql 
# dnf repolist

Before I run any changes (no update etc):

Last metadata expiration check: 0:09:30 ago on Tue 30 Jul 2024 08:28:35 UTC.
@modulefailsafe
Name                 Stream           Profiles                    Summary                                     
foreman              el8 [e]          installer                   Foreman module                              
katello              el8 [e]          installer                   Katello module                              

Oracle Linux 8 Application Stream (x86_64)
Name                 Stream           Profiles                    Summary                                     
postgresql           9.6              client, server [d]          PostgreSQL server and client module         
postgresql           10 [d]           client, server [d]          PostgreSQL server and client module         
postgresql           12 [e]           client, server [d]          PostgreSQL server and client module         
postgresql           13               client, server [d]          PostgreSQL server and client module         
postgresql           15               client, server [d]          PostgreSQL server and client module         
postgresql           16               client, server [d]          PostgreSQL server and client module 

dnf repolist will only have my base OS repos showing, nothing for katello/foreman as I disable these until/if needed.

Then once I run the dnf install Foreman/katello in the guide, I would have:

repo id                           repo name
Qualys                            Qualys
candlepin                         Candlepin: an open source entitlement management system.
foreman                           Foreman 3.11
foreman-plugins                   Foreman plugins 3.11
katello                           Katello 4.13
ol8_addons                        Oracle Linux 8 Addons (x86_64)
ol8_appstream                     Oracle Linux 8 Application Stream (x86_64)
ol8_baseos_latest                 Oracle Linux 8 BaseOS Latest (x86_64)
ol8_u4_security_validation        Oracle Linux 8 Update 4 (x86_64) Security Validations
pulpcore                          pulpcore: Fetch, Upload, Organize, and Distribute Software Packages

and modules look like:

  - module katello:el8:41320240724162450:37be364f.x86_64 from katello requires module(foreman:el8), but none of the providers can be installed
  - conflicting requests
Foreman 3.11
Name                 Stream           Profiles                    Summary                                     
foreman              el8 [e]          installer                   Foreman module                              

Katello 4.13
Name                 Stream           Profiles                    Summary                                     
katello              el8 [e]          installer                   Katello module                              

Oracle Linux 8 Application Stream (x86_64)
Name                 Stream           Profiles                    Summary                                     
postgresql           9.6              client, server [d]          PostgreSQL server and client module         
postgresql           10 [d]           client, server [d]          PostgreSQL server and client module         
postgresql           12 [e]           client, server [d]          PostgreSQL server and client module         
postgresql           13               client, server [d]          PostgreSQL server and client module         
postgresql           15               client, server [d]          PostgreSQL server and client module         
postgresql           16               client, server [d]          PostgreSQL server and client module         

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

and like I’ve mentioned - I can switch to Postrgres13 fine, albeit with the conflicts we know about. The full update in the next step is also OK. Just foreman-installer throwing errors during the postgres migration

So what happens exactly when you switch to postgresql 13?

What are your exact next steps and what is the output?

conflicts followed by dnf installing Postgresql13, no errors

exactly what the guide says:

  • Stop all services:

foreman-maintain service stop

  • Update the required packages:

dnf update

  • Run the installer:

foreman-installer

errors are as pasted above:

So what exactly is the error in the logs for this step? What is in the logs from the Migrating PostgreSQL data message until the error?

This is all in my initial opening post, above

Looks like you ran the initial installation with locale C and now you are using en_GB.UTF-8.

What locales do you use? Do you still get the same error when using locale C instead?

I’ve not changed anything with this machine setup since building it, over a year ago. It’s always been set as UTF-8 and the date of locale.conf confirms that. I’ve also upgraded several times without any issues in that time.

I’ve just re-tried after setting locale to “C” first and foreman-installer throws the same error albeit the postgres log now moans about it not being UTF8 :upside_down_face:

encodings for database "postgres" do not match:  old "UTF8", new "SQL_ASCII"
Failure, exitin

Think I might just have to cancel this upgrade and wait for the next release. Unless someone noticed an obvious issue

I don’t think this is related directly to foreman but rather the postgresql module handling and the upgrade between versions. It looks like the new database for pg13 was created with a different locale than the original one and now it cannot directly upgrade it. It’s more like an issue with postgresql-setup than foreman.

Hopefully a dev can confirm. Can’t imagine I’m the only one facing this issue. I’m sure more and more will try upgrading soon enough

If anything, use C.UTF-8 as your locale. That could explain the switch to ASCII (which we’re certainly incompatible with).

Still, the question remains why an in-place postgresql upgrade doesn’t use the locale of the old database version. Is this an issue/bug of the redhat module switch-to handling, that it doesn’t maintain the locale, or did foreman-installer at some point initialize the new database and then started the upgrade, causing a mismatch of locales?

switch-to only changes DNF modularity. The actual upgrade happens here:

The critical step that takes care of the upgrade is postgresql-setup --upgrade and I honestly don’t know exactly how it works. The code appears to call initdb and AFAIK that uses the locale from the environment to create it.

If there’s a problem with it, you should file a bug there because I don’t think we can do anything about it.

And that’s probably causing this problem. It’s not expected that the locale might have changed in the meantime.

You could check the locale of the current database and compare it to the current locale and then either stop the installer (because the upgrade will fail) or set the locale accordingly before starting the upgrade to maintain the current locale.

pg_upgrade (which is basically called in that script) also has an --check option. It would be interesting to see whether this option would find the mismatch. If so, that could be used. The postgresql-12-setup script from PostgreSQL does this. RedHat does it completely different…

Technically, it’s a redhat bug, if you want to upgrade a current database, initialize it with a different locale causing the upgrade to fail…

Agreed. I’d rather not implement all the checks the upgrade script should be doing.