Upgrade 1.24.2 > 2.0.1 unable to install and migrate to rh-postgresql 12 on CentOS 7.8

Speertje · June 25, 2020, 8:13am

Problem:
Command: foreman-installer --scenario katello --upgrade
Desired version Foreman 2.0.1 / Katello 3.15

Error:
/Stage[main]/Postgresql::Client/Package[postgresql-client]/ensure: change from ‘purged’ to ‘present’ failed: Execution of ‘/bin/yum -d 0 -e 0 -y install rh-postgresql12-postgresql-syspaths’ returned 1: Error: rh-postgresql12-postgresql-syspaths conflicts with postgresql-9.2.24-4.el7_8.x86_64

Expected outcome:
Migration of CentOS postgresql-9.2 to rh-postgresql12-postgresql to complete without failures

Foreman and Proxy versions:
Foreman 1.24.2

Foreman and Proxy plugin versions:
Katello 3.14

Distribution and version:
CentOS 7.8

Other relevant data:
I have SCL and EPEL repos available, so packages are there.
RH-postgresql-12-syspaths package conflicts with currently installed postgresql-9.2
I cannot delete the current version, the installer needs that.

How can I fix that? <- best option
Should I migrate to DB to pg 12 before running the installer? (seems a good work-around to me.)
And how do I do that? <- second best option
pgupgrade has a zillion options, so maybe someone can give me the correct command. (db is currently in /var/lib/pgsql, and I would like the upgraded db to be in that same location as it’s a seperate filesystem)
Do I need postgresql-12-syspaths for that to work (Murphy says “Yes!”)? So can I uninstall PG9.2 and still do a pgupgrade?

Lot’s of questions, thanks for any help!

Eize Speerstra

evgeni · June 25, 2020, 8:34am

Hey,

the installer should do everything needed to get you from 9.2 to 12, including removing 9.2 bits that conflict with 12.

Can you provide a sosreport from the affected system?

cc @ehelms who mainly worked on that feature/upgrade

Speertje · June 25, 2020, 11:08am

Hi evgeni,

I rolled back a snapshot to continue using the application.
I will need to recreate the situation as it was and then create a sosreport.

Or do you think you can get any info from a current sosreport?

evgeni · June 25, 2020, 11:09am

A current one would also be helpful, yeah.

The “failed” one would contain more details, so if you could spin up a clone of the VM and re-try the upgrade, that’d be awesome.

Speertje · June 25, 2020, 1:50pm

Hi @evgeni, I dm’d you the details of the download.

reenberg · September 14, 2020, 8:44pm

I have just been in the process of upgrading an old Foreman Katello installation from 1.21, which went remarkably fine, until I got from 1.24.3 to 2.0.2, and I hit the above error trying to install the rh-postgresql12 package, due to conflicts with the postgres 9.2 package.

The server has always used an external PostgreSQL server, so I was quite surprised that it apparently had the postgres 9.2 package installed in the first place.
But as far as I remember, then some of the older Katello packages had a dependency on the postgre 9.2 package(?), which probably explains why it was installed.

However I’m not really sure why the upgrade process didn’t remove that package before it tried to install the rh-postgresql12-*-syspaths package which conflicts. But a simple manual uninstall using ‘yum remove postgresql’, and then re-running the foreman-install upgrade process made it “work”. This continued with the installation of the rh-postgresql12 package, but it failed trying to start the service afterwards.

I’m quite sure that it is some SELinux issues, as reported in other threads as well. I’ll try and debug some more, one of the days.

ekohl · September 14, 2020, 9:34pm

If the package wasn’t installed prior to running Puppet as part of the installer, it ends up with an incorrect SELinux context on /var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf and there is a known issue that it can’t correct the SELinux context on it.

Try a manual restorecon -v /var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf and see if that changed anything.

reenberg · September 14, 2020, 9:54pm

Mmm I believe I tried a recursive restore on the /opt/rh/rh-postgresql12 folder… ahh now I see why. I didn’t use the /var/opt/ path, I more or less spent 5minuttes on it, as the last thing before leaving for the day, so I might not have spotted the right path in the error logs.

However shouldn’t it be possible to not have to install a postgresql server locally on the Foreman Katello node, when I’m using an external database? Isn’t de dependency only on pg_dump and perhaps psql or something like that? Sorry this should perhaps have been save for a separate thread.

ekohl · September 14, 2020, 10:14pm

That’s something we missed for Foreman 2.0 and 2.1 does support. The problem is Pulp 3 which uses PostgreSQL as well, but we didn’t introduce parameters for external PG until later.

reenberg · September 14, 2020, 10:24pm

Ah makes perfect sense.
I was actually looking for any new parameters to control this in the --help output, and ended up assuming that it perhaps just re-used the candlepin options. Guess I didn’t wonder that much further how it would create a new database with those credentials.

Do you have a link to the issue tracking the SELinux issue? Just if it ended up being something else, then I could add the info there.

Thanks for the swift replies and spot on info.

ekohl · September 15, 2020, 5:12pm

AFAIK there is none. It’s a bit of a combination. First of all, https://tickets.puppetlabs.com/browse/PUP-10548 is the root cause.

In our installer there’s the following workaround.

github.com

theforeman/foreman-installer/blob/71f60592f33a3056f52791a86876b9ae3ce9235f/hooks/pre/32-install_selinux_packages.rb

# Working around https://tickets.puppetlabs.com/browse/PUP-10548
if facts.dig(:os, :selinux, :enabled)
  packages = []

  if el7?
    packages << 'rh-postgresql12-postgresql-server' if local_postgresql?
    packages << 'rh-redis5-redis' if local_redis?
  end

  packages << 'foreman-selinux' if foreman_server?
  packages << 'katello-selinux' if katello_enabled?
  packages << 'candlepin-selinux' if katello_enabled?
  packages << 'pulpcore-selinux' if pulpcore_enabled?
  packages << 'crane-selinux' if pulp_enabled?

  ensure_packages(packages, 'installed')
end

However, I think I’ve seen cases that indicate that if ensure_packages fails to ensure, it doesn’t always halt. Then the Puppet code would be triggered and PUP-10548 shows up.

Then there’s the matter of fixing it. Puppet will fix SELinux contexts for managed files (which is generally great), but puppetlabs-postgresql doesn’t manage postgresql.conf as a file but rather individual entries. That means it doesn’t correct the context on a second run and you’re stuck in a situation where the service doesn’t start until you manually fix it.

It’s a bit of an edge case and there are mitigations in place, but as you can see, it’s not perfect.

In hindsight, we should have also backported the external PostgreSQL for a smoother upgrade path for external database users.

reenberg · September 16, 2020, 1:42pm

Makes sense. Is it possible to have some notes added to the release notes/known issues, about potential issues when you are using an external database? This seem to foster even more issues when upgrading to 2.1

It seems that there is actually some new(?) options to control this, they are however only shown using --full-help:

--foreman-proxy-content-pulpcore-manage-postgresql = true
--foreman-proxy-content-pulpcore-postgresql-host = localhost
--foreman-proxy-content-pulpcore-postgresql-db-name = pulpcore
--foreman-proxy-content-pulpcore-postgresql-user = pulp
--foreman-proxy-content-pulpcore-postgresql-password = <random>
--foreman--proxy-content-pulpcore-postgresql-