Yet another Foreman upgrade fail

switch-to only changes DNF modularity. The actual upgrade happens here:

The critical step that takes care of the upgrade is postgresql-setup --upgrade and I honestly don’t know exactly how it works. The code appears to call initdb and AFAIK that uses the locale from the environment to create it.

If there’s a problem with it, you should file a bug there because I don’t think we can do anything about it.

And that’s probably causing this problem. It’s not expected that the locale might have changed in the meantime.

You could check the locale of the current database and compare it to the current locale and then either stop the installer (because the upgrade will fail) or set the locale accordingly before starting the upgrade to maintain the current locale.

pg_upgrade (which is basically called in that script) also has an --check option. It would be interesting to see whether this option would find the mismatch. If so, that could be used. The postgresql-12-setup script from PostgreSQL does this. RedHat does it completely different…

Technically, it’s a redhat bug, if you want to upgrade a current database, initialize it with a different locale causing the upgrade to fail…

Agreed. I’d rather not implement all the checks the upgrade script should be doing.

I will be holding off on this upgrade. Will try the next released version, given the issues I’ve had with locale (I’ve not changed anything on the system since building it a year ago) and the performance issues that @gvde has reported about

You’ll need to go through the PostgreSQL upgrade regardless and unless you filed a bug, nothing is going to change in that regard. The same is true for the performance issue.

This place is a good place to discuss issues and figure out if it’s user error or a real bug, but it’s a poor place for developers to track issues to work on.

The performance issues are due to the candlepin issue which has been discussed in length in Feedback for Foreman 3.11 & Katello 4.13. So that is being solved.

Regarding the database upgrade issue, I would say RedHat has to fix their upgrade script to handle this case properly. However, I don’t have this issue and I find it difficult to file a bug with RedHat to fix this or Foreman to put in a proper detection or mitigation.

I haven’t seen anyone else with your locale problem (yet) so I guess unless it turns out a major problem for many more users you’ll have to figure out yourself how to make sure the new database uses the correct locale.

Same issue here:

# foreman-installer
2024-08-06 13:54:20 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-08-06 13:54:22 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-08-06 13:54:22 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-08-06 13:54:24 [NOTICE] [pre] Performing upgrade of PostgreSQL to 13
2024-08-06 13:54:24 [NOTICE] [pre] Upgrading PostgreSQL packages
2024-08-06 13:54:28 [NOTICE] [pre] Migrating PostgreSQL data
2024-08-06 13:54:31 [ERROR ] [root] runuser -l postgres -c 'postgresql-setup --upgrade' failed! Check the output for error!

Changing the dnf module to postgresql:13 worked with warnings:

# dnf -y module switch-to postgresql:13
Foreman 3.11                                                                                                                                                                                            5.4 MB/s | 1.7 MB     00:00    
Foreman plugins 3.11                                                                                                                                                                                    6.3 MB/s | 1.9 MB     00:00    
Katello 4.13                                                                                                                                                                                            1.2 MB/s | 307 kB     00:00    
Candlepin: an open source entitlement management system.                                                                                                                                                336 kB/s |  78 kB     00:00    
pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.                                                                                                                                    1.2 MB/s | 308 kB     00:00    
Modular dependency problem:

 Problem: module foreman:el8:31120240718202416:34d643a2.x86_64 from foreman requires module(postgresql:13), but none of the providers can be installed
  - module postgresql:12:8090020240226072207:b46abd14.x86_64 from appstream conflicts with module(postgresql:13) provided by postgresql:13:8090020240226072209:b46abd14.x86_64 from appstream
  - module postgresql:13:8090020240226072209:b46abd14.x86_64 from appstream conflicts with module(postgresql:12) provided by postgresql:12:8090020240226072207:b46abd14.x86_64 from appstream
  - conflicting requests
Dependencies resolved.
========================================================================================================================================================================================================================================
 Package                                                  Architecture                                 Version                                                                    Repository                                       Size
========================================================================================================================================================================================================================================
Upgrading:
 postgresql                                               x86_64                                       13.14-1.module+el8.9.0+1736+ef8fa13a                                       appstream                                       1.5 M
 postgresql-contrib                                       x86_64                                       13.14-1.module+el8.9.0+1736+ef8fa13a                                       appstream                                       882 k
 postgresql-server                                        x86_64                                       13.14-1.module+el8.9.0+1736+ef8fa13a                                       appstream                                       5.6 M
Switching module streams:
 postgresql                                                                                            12 -> 13                                                                                                                        

Transaction Summary
========================================================================================================================================================================================================================================
Upgrade  3 Packages

Total download size: 8.0 M
Downloading Packages:
(1/3): postgresql-contrib-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64.rpm                                                                                                                               7.0 MB/s | 882 kB     00:00    
(2/3): postgresql-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64.rpm                                                                                                                                        11 MB/s | 1.5 MB     00:00    
(3/3): postgresql-server-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64.rpm                                                                                                                                 23 MB/s | 5.6 MB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                    16 MB/s | 8.0 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                1/1 
  Running scriptlet: postgresql-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                         1/1 
  Upgrading        : postgresql-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                         1/6 
  Upgrading        : postgresql-contrib-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                 2/6 
  Running scriptlet: postgresql-server-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                  3/6 
  Upgrading        : postgresql-server-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                  3/6 
  Running scriptlet: postgresql-server-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                  3/6 
  Running scriptlet: postgresql-server-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                  4/6 
  Cleanup          : postgresql-server-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                  4/6 
  Running scriptlet: postgresql-server-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                  4/6 
  Cleanup          : postgresql-contrib-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                 5/6 
  Cleanup          : postgresql-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                         6/6 
  Running scriptlet: postgresql-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                         6/6 
  Verifying        : postgresql-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                         1/6 
  Verifying        : postgresql-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                         2/6 
  Verifying        : postgresql-contrib-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                 3/6 
  Verifying        : postgresql-contrib-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                 4/6 
  Verifying        : postgresql-server-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                                                                                                                                                  5/6 
  Verifying        : postgresql-server-12.18-1.module+el8.9.0+1735+a332307b.x86_64                                                                                                                                                  6/6 

Upgraded:
  postgresql-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                  postgresql-contrib-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                  postgresql-server-13.14-1.module+el8.9.0+1736+ef8fa13a.x86_64                 

Complete!

I don’t know how to perform the Foreman 3.11 upgrade.
I had to roll back to Foreman 3.10.

Well, it could be the same issue. But what you post looks a lot like you have first tried foreman-installer and then the module switch which would be wrong. And you also left out the postgresql log containing the error which caused postgresql-setup to fail. That’s the essential piece of information whether it’s the same issue or not…

No, I first ran the module switch, and then foreman-installer.
Sorry, I have re-ordered it for relevance.
I’m not sure where I can find the postgresql log?

# journalctl -u postgresql
-- Logs begin at Tue 2024-08-06 12:54:15 CEST, end at Tue 2024-08-06 14:20:18 CEST. --
Aug 06 12:54:19 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG:  PostgreSQL 12.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit startet
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG:  erwarte Verbindungen auf IPv6-Adresse »::1«, Port 5432
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG:  erwarte Verbindungen auf IPv4-Adresse »127.0.0.1«, Port 5432
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/var/run/postgresql/.s.PGSQL.5432«
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/tmp/.s.PGSQL.5432«
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG:  Logausgabe wird an Logsammelprozess umgeleitet
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST TIPP:  Die weitere Logausgabe wird im Verzeichnis »log« erscheinen.
Aug 06 12:54:19 foreman.my.org systemd[1]: Started PostgreSQL database server.
Aug 06 12:55:22 foreman.my.org systemd[1]: Stopping PostgreSQL database server...
Aug 06 12:55:23 foreman.my.org systemd[1]: postgresql.service: Succeeded.
Aug 06 12:55:23 foreman.my.org systemd[1]: Stopped PostgreSQL database server.
Aug 06 12:55:53 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG:  PostgreSQL 12.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit startet
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG:  erwarte Verbindungen auf IPv6-Adresse »::1«, Port 5432
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG:  erwarte Verbindungen auf IPv4-Adresse »127.0.0.1«, Port 5432
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/var/run/postgresql/.s.PGSQL.5432«
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/tmp/.s.PGSQL.5432«
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG:  Logausgabe wird an Logsammelprozess umgeleitet
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST TIPP:  Die weitere Logausgabe wird im Verzeichnis »log« erscheinen.
Aug 06 12:55:53 foreman.my.org systemd[1]: Started PostgreSQL database server.
Aug 06 13:41:01 foreman.my.org systemd[1]: Stopping PostgreSQL database server...
Aug 06 13:41:01 foreman.my.org systemd[1]: postgresql.service: Succeeded.
Aug 06 13:41:01 foreman.my.org systemd[1]: Stopped PostgreSQL database server.
Aug 06 14:05:23 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG:  PostgreSQL 12.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit startet
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG:  erwarte Verbindungen auf IPv6-Adresse »::1«, Port 5432
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG:  erwarte Verbindungen auf IPv4-Adresse »127.0.0.1«, Port 5432
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/var/run/postgresql/.s.PGSQL.5432«
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG:  erwarte Verbindungen auf Unix-Socket »/tmp/.s.PGSQL.5432«
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG:  Logausgabe wird an Logsammelprozess umgeleitet
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST TIPP:  Die weitere Logausgabe wird im Verzeichnis »log« erscheinen.
Aug 06 14:05:24 foreman.my.org systemd[1]: Started PostgreSQL database server.
Aug 06 14:16:13 foreman.my.org systemd[1]: Stopping PostgreSQL database server...
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Killing process 11551 (postmaster) with signal SIGKILL.
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Succeeded.
Aug 06 14:16:14 foreman.my.org systemd[1]: Stopped PostgreSQL database server.
Aug 06 14:16:14 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 14:16:14 foreman.my.org postgresql-check-db-dir[12342]: An old version of the database format was found.
Aug 06 14:16:14 foreman.my.org postgresql-check-db-dir[12342]: Use 'postgresql-setup --upgrade' to upgrade to version '13'
Aug 06 14:16:14 foreman.my.org postgresql-check-db-dir[12342]: See /usr/share/doc/postgresql/README.rpm-dist for more information.
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Control process exited, code=exited status=1
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Failed with result 'exit-code'.
Aug 06 14:16:14 foreman.my.org systemd[1]: Failed to start PostgreSQL database server.

PostgreSQL usually writes all logs into /var/lib/pgsql or a subdirectory there.

It depends which flavor you use. If it’s plain Foreman on Enterprise Linux:
https://docs.theforeman.org/3.11/Upgrading_Project/index-foreman-el.html

If it’s Katello:
https://docs.theforeman.org/3.11/Upgrading_Project/index-katello.html

Both of these tell you to run dnf -y module switch-to postgresql:13

Thanks,
I think I have used the correct documentation for upgrading Foreman + Katello on EL.

Yes, there seems to be a problem with locales during the postgresql database version upgrade.
I should mention, that Foreman/Katello upgrades worked for years under the existing locale.
All of a sudden, the upgrade breaks, seemingly because of the locale.

# cat /var/lib/pgsql/upgrade_postgresql.log 
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
  candlepin
  foreman
  postgres
  pulpcore
  template1
                                                            ok

lc_collate values for database "postgres" do not match:  old "de_CH.utf8", new "en_US.UTF-8"
Failure, exiting
# locale 
LANG=de_CH.utf8
LC_CTYPE=de_CH.utf8
LC_NUMERIC="de_CH.utf8"
LC_TIME="de_CH.utf8"
LC_COLLATE="de_CH.utf8"
LC_MONETARY="de_CH.utf8"
LC_MESSAGES="de_CH.utf8"
LC_PAPER="de_CH.utf8"
LC_NAME="de_CH.utf8"
LC_ADDRESS="de_CH.utf8"
LC_TELEPHONE="de_CH.utf8"
LC_MEASUREMENT="de_CH.utf8"
LC_IDENTIFICATION="de_CH.utf8"
LC_ALL=

O.K. You have the same issue with the locale mismatch. It don’t think it’s an foreman-installer issue but an rhel specific postgresql issue. If you run

runuser -l postgres -c 'postgresql-setup --upgrade'

I would guess you’ll see the same error. If not, them it would be foreman-installer specific…

Yes, this command alone fails:

# runuser -l postgres -c 'postgresql-setup --upgrade'
WARNING: Note that either your data directory '/var/lib/pgsql/data' or
         the parent directory '/var/lib/pgsql'
         is a direct mountpoint.  This is usually a bad idea and your
         filesystem layout should ideally look like:
         /ROOT_OWNED_MOUNTPOINT/POSTGRES_OWNED_DIRECTORY/DATADIR.
         See the upstream documentation for more info:
         http://www.postgresql.org/docs/13/static/creating-cluster.html
 * Upgrading database.
ERROR: pg_upgrade tool failed
ERROR: Upgrade failed.
 * See /var/lib/pgsql/upgrade_postgresql.log for details.
# cat /var/lib/pgsql/upgrade_postgresql.log 
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
  candlepin
  foreman
  postgres
  pulpcore
  template1
                                                            ok

lc_collate values for database "postgres" do not match:  old "de_CH.utf8", new "en_US.UTF-8"
Failure, exiting

I will roll back to Foreman 3.10, as the upgrade is broken (once again…)
Thanks

What’s the output of localectl? I suspect that the system locale is en_US.UTF-8 and runuser discards the user’s env variables and uses the system locale. You can try this out: runuser -l postgres -c 'env | grep -E "LANG|LC_"'.

Can you add --debug to the postgresql-setup options? It should better explain what it wants to do.

Is the pg13 database already initialized? What files and directories do you have in /var/lib/pgsql/ and /var/lib/pgsql/data*

On STDOUT, I don’t see more relevant information.

# runuser -l postgres -c 'postgresql-setup --upgrade --debug'
DEBUG: using 'upgrade' confdir /etc/postgresql-setup/upgrade
DEBUG: reading config /etc/postgresql-setup/upgrade/postgresql.conf
DEBUG: mode used: upgrade
DEBUG: service name: postgresql
DEBUG: running service_configuration() for postgresql, mode: initdb
DEBUG: unit's datadir: '/var/lib/pgsql/data'
DEBUG: unit's pgport: 5432
DEBUG: running service_configuration() for postgresql, mode: upgrade
DEBUG: unit's datadir: '/var/lib/pgsql/data'
DEBUG: unit's pgport: 5432
DEBUG: postgresql.conf: /var/lib/pgsql/data/postgresql.conf
DEBUG: postgresql.conf pgport: 5432
DEBUG: final pgdata: /var/lib/pgsql/data
DEBUG: final pgport: 5432
WARNING: Note that either your data directory '/var/lib/pgsql/data' or
         the parent directory '/var/lib/pgsql'
         is a direct mountpoint.  This is usually a bad idea and your
         filesystem layout should ideally look like:
         /ROOT_OWNED_MOUNTPOINT/POSTGRES_OWNED_DIRECTORY/DATADIR.
         See the upstream documentation for more info:
         http://www.postgresql.org/docs/13/static/creating-cluster.html
DEBUG: running inplace upgrade: true
 * Upgrading database.
ERROR: pg_upgrade tool failed
ERROR: Upgrade failed.
 * See /var/lib/pgsql/upgrade_postgresql.log for details.

And in the log file, it’s just the same:

# cat /var/lib/pgsql/upgrade_postgresql.log
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
  candlepin
  foreman
  postgres
  pulpcore
  template1
                                                            ok

lc_collate values for database "postgres" do not match:  old "de_CH.utf8", new "en_US.UTF-8"
Failure, exiting

Not exactly sure what you mean by that. The older version database was running correctly with Foreman 3.10. So I would guess it should be initialized? It’s just the database version conversion that’s missing.

# ls -l /var/lib/pgsql/
total 2.2M
drwx------.  2 postgres postgres    6 Feb 26 08:43 backups
drwx------. 20 postgres postgres 4.0K Aug  7 09:45 data
-rw-------.  1 postgres postgres 1.8K Aug  7 09:45 initdb_postgresql.log
-rw-------.  1 postgres postgres 1.6K Aug  7 09:45 pg_upgrade_dump_14399.custom
-rw-------.  1 postgres postgres  496 Aug  7 09:45 pg_upgrade_dump_14399.log
-rw-------.  1 postgres postgres 242K Aug  7 09:45 pg_upgrade_dump_16385.custom
-rw-------.  1 postgres postgres  498 Aug  7 09:45 pg_upgrade_dump_16385.log
-rw-------.  1 postgres postgres 1.2M Aug  7 09:45 pg_upgrade_dump_18214.custom
-rw-------.  1 postgres postgres  494 Aug  7 09:45 pg_upgrade_dump_18214.log
-rw-------.  1 postgres postgres 2.0K Aug  7 09:45 pg_upgrade_dump_1.custom
-rw-------.  1 postgres postgres  482 Aug  7 09:45 pg_upgrade_dump_1.log
-rw-------.  1 postgres postgres 724K Aug  7 09:45 pg_upgrade_dump_23930.custom
-rw-------.  1 postgres postgres  496 Aug  7 09:45 pg_upgrade_dump_23930.log
-rw-------.  1 postgres postgres 1.4K Aug  7 09:45 pg_upgrade_dump_globals.sql
-rw-------.  1 postgres postgres 1.9K Aug  7 09:45 pg_upgrade_internal.log
-rw-------.  1 postgres postgres 3.6K Aug  7 09:45 pg_upgrade_server.log
-rw-------.  1 postgres postgres  774 Aug  7 09:45 pg_upgrade_utility.log
-rw-------.  1 postgres postgres  828 Aug  7 09:45 upgrade_postgresql.log

# ls -l /var/lib/pgsql/data/
total 64K
drwx------. 9 postgres postgres   97 Feb  3  2023 base
-rw-------. 1 postgres postgres   30 Aug  7 09:45 current_logfiles
drwx------. 2 postgres postgres 4.0K Aug  7 09:45 global
drwx------. 2 postgres postgres  188 Feb  9  2023 log
drwx------. 2 postgres postgres    6 Feb  3  2023 pg_commit_ts
drwx------. 2 postgres postgres    6 Feb  3  2023 pg_dynshmem
-rw-r-----. 1 postgres postgres  811 Jan  5  2024 pg_hba.conf
-rw-r-----. 1 postgres postgres   47 Feb  3  2023 pg_ident.conf
drwx------. 4 postgres postgres   68 Aug  7 09:45 pg_logical
drwx------. 4 postgres postgres   36 Feb  3  2023 pg_multixact
drwx------. 2 postgres postgres   18 Aug  7 09:45 pg_notify
drwx------. 2 postgres postgres    6 Feb  3  2023 pg_replslot
drwx------. 2 postgres postgres    6 Feb  3  2023 pg_serial
drwx------. 2 postgres postgres    6 Feb  3  2023 pg_snapshots
drwx------. 2 postgres postgres  143 Aug  7 09:45 pg_stat
drwx------. 2 postgres postgres    6 Aug  7 09:42 pg_stat_tmp
drwx------. 2 postgres postgres   18 Aug  7 08:03 pg_subtrans
drwx------. 2 postgres postgres    6 Feb  3  2023 pg_tblspc
drwx------. 2 postgres postgres    6 Feb  3  2023 pg_twophase
-rw-------. 1 postgres postgres    3 Feb  3  2023 PG_VERSION
drwx------. 3 postgres postgres  188 Aug  7 09:28 pg_wal
drwx------. 2 postgres postgres 4.0K Aug  6 00:58 pg_xact
-rw-------. 1 postgres postgres   88 Feb  3  2023 postgresql.auto.conf
-rw-------. 1 postgres postgres  27K Aug  7 09:45 postgresql.conf
-rw-------. 1 postgres postgres  199 Aug  7 09:45 postmaster.opts

Does it help?

O.K. It does an inplace upgrade as expected.

Can you check your locale settings in /var/lib/pgsql/data/postgresql.conf

# These settings are initialized by initdb, but they can be changed.
lc_messages = 'en_US.UTF-8' # locale for system error message
                                        # strings
lc_monetary = 'en_US.UTF-8' # locale for monetary formatting
lc_numeric = 'en_US.UTF-8' # locale for number formatting
lc_time = 'en_US.UTF-8' # locale for time formatting

What is your system locale?

# localectl status
   System Locale: LANG=en_US.UTF-8
       VC Keymap: us
      X11 Layout: us

Thanks for following up.

# grep '^lc_' /var/lib/pgsql/data/postgresql.conf
lc_messages = 'de_CH.utf8' # locale for system error message
lc_monetary = 'de_CH.utf8' # locale for monetary formatting
lc_numeric = 'de_CH.utf8' # locale for number formatting
lc_time = 'de_CH.utf8' # locale for time formatting
# localectl status
   System Locale: LANG=en_US.UTF-8
       VC Keymap: ch
      X11 Layout: ch

I should note, that I am setting this:

# cat /etc/environment 
LANGUAGE="en_US:en"
LC_ALL="de_CH.utf8"
LC_CTYPE="de_CH.utf8"
LANG="de_CH.utf8"