switch-to only changes DNF modularity. The actual upgrade happens here:
The critical step that takes care of the upgrade is postgresql-setup --upgrade and I honestly don’t know exactly how it works. The code appears to call initdb and AFAIK that uses the locale from the environment to create it.
If there’s a problem with it, you should file a bug there because I don’t think we can do anything about it.
And that’s probably causing this problem. It’s not expected that the locale might have changed in the meantime.
You could check the locale of the current database and compare it to the current locale and then either stop the installer (because the upgrade will fail) or set the locale accordingly before starting the upgrade to maintain the current locale.
pg_upgrade (which is basically called in that script) also has an --check option. It would be interesting to see whether this option would find the mismatch. If so, that could be used. The postgresql-12-setup script from PostgreSQL does this. RedHat does it completely different…
Technically, it’s a redhat bug, if you want to upgrade a current database, initialize it with a different locale causing the upgrade to fail…
I will be holding off on this upgrade. Will try the next released version, given the issues I’ve had with locale (I’ve not changed anything on the system since building it a year ago) and the performance issues that @gvde has reported about
You’ll need to go through the PostgreSQL upgrade regardless and unless you filed a bug, nothing is going to change in that regard. The same is true for the performance issue.
This place is a good place to discuss issues and figure out if it’s user error or a real bug, but it’s a poor place for developers to track issues to work on.
Regarding the database upgrade issue, I would say RedHat has to fix their upgrade script to handle this case properly. However, I don’t have this issue and I find it difficult to file a bug with RedHat to fix this or Foreman to put in a proper detection or mitigation.
I haven’t seen anyone else with your locale problem (yet) so I guess unless it turns out a major problem for many more users you’ll have to figure out yourself how to make sure the new database uses the correct locale.
# foreman-installer
2024-08-06 13:54:20 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-08-06 13:54:22 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-08-06 13:54:22 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-08-06 13:54:24 [NOTICE] [pre] Performing upgrade of PostgreSQL to 13
2024-08-06 13:54:24 [NOTICE] [pre] Upgrading PostgreSQL packages
2024-08-06 13:54:28 [NOTICE] [pre] Migrating PostgreSQL data
2024-08-06 13:54:31 [ERROR ] [root] runuser -l postgres -c 'postgresql-setup --upgrade' failed! Check the output for error!
Changing the dnf module to postgresql:13 worked with warnings:
Well, it could be the same issue. But what you post looks a lot like you have first tried foreman-installer and then the module switch which would be wrong. And you also left out the postgresql log containing the error which caused postgresql-setup to fail. That’s the essential piece of information whether it’s the same issue or not…
No, I first ran the module switch, and then foreman-installer.
Sorry, I have re-ordered it for relevance.
I’m not sure where I can find the postgresql log?
# journalctl -u postgresql
-- Logs begin at Tue 2024-08-06 12:54:15 CEST, end at Tue 2024-08-06 14:20:18 CEST. --
Aug 06 12:54:19 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG: PostgreSQL 12.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit startet
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG: erwarte Verbindungen auf IPv6-Adresse »::1«, Port 5432
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG: erwarte Verbindungen auf IPv4-Adresse »127.0.0.1«, Port 5432
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG: erwarte Verbindungen auf Unix-Socket »/var/run/postgresql/.s.PGSQL.5432«
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG: erwarte Verbindungen auf Unix-Socket »/tmp/.s.PGSQL.5432«
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST LOG: Logausgabe wird an Logsammelprozess umgeleitet
Aug 06 12:54:19 foreman.my.org postmaster[1094]: 2024-08-06 12:54:19 CEST TIPP: Die weitere Logausgabe wird im Verzeichnis »log« erscheinen.
Aug 06 12:54:19 foreman.my.org systemd[1]: Started PostgreSQL database server.
Aug 06 12:55:22 foreman.my.org systemd[1]: Stopping PostgreSQL database server...
Aug 06 12:55:23 foreman.my.org systemd[1]: postgresql.service: Succeeded.
Aug 06 12:55:23 foreman.my.org systemd[1]: Stopped PostgreSQL database server.
Aug 06 12:55:53 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG: PostgreSQL 12.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit startet
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG: erwarte Verbindungen auf IPv6-Adresse »::1«, Port 5432
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG: erwarte Verbindungen auf IPv4-Adresse »127.0.0.1«, Port 5432
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG: erwarte Verbindungen auf Unix-Socket »/var/run/postgresql/.s.PGSQL.5432«
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG: erwarte Verbindungen auf Unix-Socket »/tmp/.s.PGSQL.5432«
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST LOG: Logausgabe wird an Logsammelprozess umgeleitet
Aug 06 12:55:53 foreman.my.org postmaster[8097]: 2024-08-06 12:55:53 CEST TIPP: Die weitere Logausgabe wird im Verzeichnis »log« erscheinen.
Aug 06 12:55:53 foreman.my.org systemd[1]: Started PostgreSQL database server.
Aug 06 13:41:01 foreman.my.org systemd[1]: Stopping PostgreSQL database server...
Aug 06 13:41:01 foreman.my.org systemd[1]: postgresql.service: Succeeded.
Aug 06 13:41:01 foreman.my.org systemd[1]: Stopped PostgreSQL database server.
Aug 06 14:05:23 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG: PostgreSQL 12.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit startet
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG: erwarte Verbindungen auf IPv6-Adresse »::1«, Port 5432
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG: erwarte Verbindungen auf IPv4-Adresse »127.0.0.1«, Port 5432
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG: erwarte Verbindungen auf Unix-Socket »/var/run/postgresql/.s.PGSQL.5432«
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG: erwarte Verbindungen auf Unix-Socket »/tmp/.s.PGSQL.5432«
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST LOG: Logausgabe wird an Logsammelprozess umgeleitet
Aug 06 14:05:24 foreman.my.org postmaster[11549]: 2024-08-06 14:05:24 CEST TIPP: Die weitere Logausgabe wird im Verzeichnis »log« erscheinen.
Aug 06 14:05:24 foreman.my.org systemd[1]: Started PostgreSQL database server.
Aug 06 14:16:13 foreman.my.org systemd[1]: Stopping PostgreSQL database server...
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Killing process 11551 (postmaster) with signal SIGKILL.
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Succeeded.
Aug 06 14:16:14 foreman.my.org systemd[1]: Stopped PostgreSQL database server.
Aug 06 14:16:14 foreman.my.org systemd[1]: Starting PostgreSQL database server...
Aug 06 14:16:14 foreman.my.org postgresql-check-db-dir[12342]: An old version of the database format was found.
Aug 06 14:16:14 foreman.my.org postgresql-check-db-dir[12342]: Use 'postgresql-setup --upgrade' to upgrade to version '13'
Aug 06 14:16:14 foreman.my.org postgresql-check-db-dir[12342]: See /usr/share/doc/postgresql/README.rpm-dist for more information.
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Control process exited, code=exited status=1
Aug 06 14:16:14 foreman.my.org systemd[1]: postgresql.service: Failed with result 'exit-code'.
Aug 06 14:16:14 foreman.my.org systemd[1]: Failed to start PostgreSQL database server.
Yes, there seems to be a problem with locales during the postgresql database version upgrade.
I should mention, that Foreman/Katello upgrades worked for years under the existing locale.
All of a sudden, the upgrade breaks, seemingly because of the locale.
# cat /var/lib/pgsql/upgrade_postgresql.log
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating dump of global objects ok
Creating dump of database schemas
candlepin
foreman
postgres
pulpcore
template1
ok
lc_collate values for database "postgres" do not match: old "de_CH.utf8", new "en_US.UTF-8"
Failure, exiting
O.K. You have the same issue with the locale mismatch. It don’t think it’s an foreman-installer issue but an rhel specific postgresql issue. If you run
# runuser -l postgres -c 'postgresql-setup --upgrade'
WARNING: Note that either your data directory '/var/lib/pgsql/data' or
the parent directory '/var/lib/pgsql'
is a direct mountpoint. This is usually a bad idea and your
filesystem layout should ideally look like:
/ROOT_OWNED_MOUNTPOINT/POSTGRES_OWNED_DIRECTORY/DATADIR.
See the upstream documentation for more info:
http://www.postgresql.org/docs/13/static/creating-cluster.html
* Upgrading database.
ERROR: pg_upgrade tool failed
ERROR: Upgrade failed.
* See /var/lib/pgsql/upgrade_postgresql.log for details.
# cat /var/lib/pgsql/upgrade_postgresql.log
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating dump of global objects ok
Creating dump of database schemas
candlepin
foreman
postgres
pulpcore
template1
ok
lc_collate values for database "postgres" do not match: old "de_CH.utf8", new "en_US.UTF-8"
Failure, exiting
I will roll back to Foreman 3.10, as the upgrade is broken (once again…)
Thanks
What’s the output of localectl? I suspect that the system locale is en_US.UTF-8 and runuser discards the user’s env variables and uses the system locale. You can try this out: runuser -l postgres -c 'env | grep -E "LANG|LC_"'.
# runuser -l postgres -c 'postgresql-setup --upgrade --debug'
DEBUG: using 'upgrade' confdir /etc/postgresql-setup/upgrade
DEBUG: reading config /etc/postgresql-setup/upgrade/postgresql.conf
DEBUG: mode used: upgrade
DEBUG: service name: postgresql
DEBUG: running service_configuration() for postgresql, mode: initdb
DEBUG: unit's datadir: '/var/lib/pgsql/data'
DEBUG: unit's pgport: 5432
DEBUG: running service_configuration() for postgresql, mode: upgrade
DEBUG: unit's datadir: '/var/lib/pgsql/data'
DEBUG: unit's pgport: 5432
DEBUG: postgresql.conf: /var/lib/pgsql/data/postgresql.conf
DEBUG: postgresql.conf pgport: 5432
DEBUG: final pgdata: /var/lib/pgsql/data
DEBUG: final pgport: 5432
WARNING: Note that either your data directory '/var/lib/pgsql/data' or
the parent directory '/var/lib/pgsql'
is a direct mountpoint. This is usually a bad idea and your
filesystem layout should ideally look like:
/ROOT_OWNED_MOUNTPOINT/POSTGRES_OWNED_DIRECTORY/DATADIR.
See the upstream documentation for more info:
http://www.postgresql.org/docs/13/static/creating-cluster.html
DEBUG: running inplace upgrade: true
* Upgrading database.
ERROR: pg_upgrade tool failed
ERROR: Upgrade failed.
* See /var/lib/pgsql/upgrade_postgresql.log for details.
And in the log file, it’s just the same:
# cat /var/lib/pgsql/upgrade_postgresql.log
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating dump of global objects ok
Creating dump of database schemas
candlepin
foreman
postgres
pulpcore
template1
ok
lc_collate values for database "postgres" do not match: old "de_CH.utf8", new "en_US.UTF-8"
Failure, exiting
Not exactly sure what you mean by that. The older version database was running correctly with Foreman 3.10. So I would guess it should be initialized? It’s just the database version conversion that’s missing.
# ls -l /var/lib/pgsql/
total 2.2M
drwx------. 2 postgres postgres 6 Feb 26 08:43 backups
drwx------. 20 postgres postgres 4.0K Aug 7 09:45 data
-rw-------. 1 postgres postgres 1.8K Aug 7 09:45 initdb_postgresql.log
-rw-------. 1 postgres postgres 1.6K Aug 7 09:45 pg_upgrade_dump_14399.custom
-rw-------. 1 postgres postgres 496 Aug 7 09:45 pg_upgrade_dump_14399.log
-rw-------. 1 postgres postgres 242K Aug 7 09:45 pg_upgrade_dump_16385.custom
-rw-------. 1 postgres postgres 498 Aug 7 09:45 pg_upgrade_dump_16385.log
-rw-------. 1 postgres postgres 1.2M Aug 7 09:45 pg_upgrade_dump_18214.custom
-rw-------. 1 postgres postgres 494 Aug 7 09:45 pg_upgrade_dump_18214.log
-rw-------. 1 postgres postgres 2.0K Aug 7 09:45 pg_upgrade_dump_1.custom
-rw-------. 1 postgres postgres 482 Aug 7 09:45 pg_upgrade_dump_1.log
-rw-------. 1 postgres postgres 724K Aug 7 09:45 pg_upgrade_dump_23930.custom
-rw-------. 1 postgres postgres 496 Aug 7 09:45 pg_upgrade_dump_23930.log
-rw-------. 1 postgres postgres 1.4K Aug 7 09:45 pg_upgrade_dump_globals.sql
-rw-------. 1 postgres postgres 1.9K Aug 7 09:45 pg_upgrade_internal.log
-rw-------. 1 postgres postgres 3.6K Aug 7 09:45 pg_upgrade_server.log
-rw-------. 1 postgres postgres 774 Aug 7 09:45 pg_upgrade_utility.log
-rw-------. 1 postgres postgres 828 Aug 7 09:45 upgrade_postgresql.log
# ls -l /var/lib/pgsql/data/
total 64K
drwx------. 9 postgres postgres 97 Feb 3 2023 base
-rw-------. 1 postgres postgres 30 Aug 7 09:45 current_logfiles
drwx------. 2 postgres postgres 4.0K Aug 7 09:45 global
drwx------. 2 postgres postgres 188 Feb 9 2023 log
drwx------. 2 postgres postgres 6 Feb 3 2023 pg_commit_ts
drwx------. 2 postgres postgres 6 Feb 3 2023 pg_dynshmem
-rw-r-----. 1 postgres postgres 811 Jan 5 2024 pg_hba.conf
-rw-r-----. 1 postgres postgres 47 Feb 3 2023 pg_ident.conf
drwx------. 4 postgres postgres 68 Aug 7 09:45 pg_logical
drwx------. 4 postgres postgres 36 Feb 3 2023 pg_multixact
drwx------. 2 postgres postgres 18 Aug 7 09:45 pg_notify
drwx------. 2 postgres postgres 6 Feb 3 2023 pg_replslot
drwx------. 2 postgres postgres 6 Feb 3 2023 pg_serial
drwx------. 2 postgres postgres 6 Feb 3 2023 pg_snapshots
drwx------. 2 postgres postgres 143 Aug 7 09:45 pg_stat
drwx------. 2 postgres postgres 6 Aug 7 09:42 pg_stat_tmp
drwx------. 2 postgres postgres 18 Aug 7 08:03 pg_subtrans
drwx------. 2 postgres postgres 6 Feb 3 2023 pg_tblspc
drwx------. 2 postgres postgres 6 Feb 3 2023 pg_twophase
-rw-------. 1 postgres postgres 3 Feb 3 2023 PG_VERSION
drwx------. 3 postgres postgres 188 Aug 7 09:28 pg_wal
drwx------. 2 postgres postgres 4.0K Aug 6 00:58 pg_xact
-rw-------. 1 postgres postgres 88 Feb 3 2023 postgresql.auto.conf
-rw-------. 1 postgres postgres 27K Aug 7 09:45 postgresql.conf
-rw-------. 1 postgres postgres 199 Aug 7 09:45 postmaster.opts
Can you check your locale settings in /var/lib/pgsql/data/postgresql.conf
# These settings are initialized by initdb, but they can be changed.
lc_messages = 'en_US.UTF-8' # locale for system error message
# strings
lc_monetary = 'en_US.UTF-8' # locale for monetary formatting
lc_numeric = 'en_US.UTF-8' # locale for number formatting
lc_time = 'en_US.UTF-8' # locale for time formatting
What is your system locale?
# localectl status
System Locale: LANG=en_US.UTF-8
VC Keymap: us
X11 Layout: us