Foreman Katello 3.14 to 3.115 upgrade cpdb

Problem:
Receiving a cpdb --upgrade error when attempting to run the forman upgrade.

/Stage[main]/Candlepin::Database::Postgresql/Exec[cpdb update]/returns: change from 'notrun' to ['0'] failed: 'cpdb --update --dbhost=localhost --dbport=5432 --database='candlepin' --user='candlepin' --password='[PASSWORD]' >> /var/log/candlepin/cpdb.log 2>&1 && touch /var/lib/candlepin/cpdb_update_done' returned 1 instead of one of [0]

Expected outcome:
There would be no error, and Foreman-katello would be able to upgrade to 3.15

Foreman and Proxy versions:
Unsure Since starting this process I’ve been unable to log into forman with my user account. Katello 3.14

Foreman and Proxy plugin versions:
Unsure Since starting this process I’ve been unable to log into forman with my user account. Katello 3.14

Distribution and version:
Unsure Since starting this process I’ve been unable to log into forman with my user account. Katello 3.14

Other relevant data:
Been trying to upgrade Foreman-Katello from 3.14 to 3.15 for a few days. Which lead me through the following articles:

@kcgaisford what are the contents of /var/log/candlepin/cpdb.log ?

Looks like the table for public.cp_consumer_type does not exsist.

> ########## ERROR ############
> Error running command: /usr/share/candlepin/liquibase.sh --driver=org.postgresql.Driver --classpath=/var/lib/tomcat/webapps/candlepin/WEB-INF/lib/postgresql-42.2.2.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-update.xml --url="jdbc:postgresql://localhost:5432/candlepin" --username=$DBUSERNAME --password=$DBPASSWORD --logLevel=severe migrate -Dcommunity=False
> Status code: 65280
> Command output: SEVERE 6/3/20 11:57 AM:liquibase: db/changelog/changelog-update.xml: db/changelog/20120416134048-insert-uebercert-consumer-type.xml::20120416134048::mstead: Change Set db/changelog/20120416134048-insert-uebercert-consumer-type.xml::20120416134048::mstead failed.  Error: Error executing SQL INSERT INTO public.cp_consumer_type (id, label, manifest) VALUES ('1005', 'uebercert', 'N'): ERROR: relation "public.cp_consumer_type" does not exist
>   Position: 13
> liquibase.exception.DatabaseException: Error executing SQL INSERT INTO public.cp_consumer_type (id, label, manifest) VALUES ('1005', 'uebercert', 'N'): ERROR: relation "public.cp_consumer_type" does not exist

It seems like my instance was running Postgres 9.6 but because of trying to upgrade this it has made me downgrade to 9.2

I no longer have 9.6 but the data should still be there. Not exactly sure on how to get that data into the 9.2 instance if that is the problem. (Would explain why I can’t log into foreman any more )

Just so I understand correctly: everything was working OK before you attempted to upgrade, and that has left you in a broken state? ie you were able to log in before attempting the upgrade? If there was a problem before upgrading then you should revert to your backup, fix any issues there, and then attempt to upgrade.

@Jonathon_Turel I almost reverted back to the snapshot I took… but i had noticed that the foreman installer made it far enough to upgrade postgres on the host to the 12.1.

Since I figured I have the snapshot to move back to I dropped the postgresql 12 data folder, and used pg_upgrade to migrate the data from the 9.6 data folder to a new postgres 12 folder. everything seems to be working. I was able to fully run the foreman --upgrade command, I can now log back into foreman see that it is upgraded. My login is working again and all my host data is back.

Great work! Thanks for sharing your success with us. Enjoy Katello 3.15 :slight_smile:

I had the same problem. I found it was because of:

  • an incompatible python2-future installed on the system
  • the installer failing to retrieve packages when http_proxy was unset
    • causing the installer to crash and not pick up where it left off after the fix
    • leading to the db upgrade not being done.

This is how I made the upgrade work. It’s probably 100% specific to my system, but it’s here in case anyone else has similar issues.

Pre-upgrade check

foreman-rake katello:upgrade_check

Upgrade OS

yum update

Remove incompatible package

yum remove python2-future

(found by trial and error in previous upgrade attempt)

Install new repos

yum update -y https://fedorapeople.org/groups/katello/releases/yum/3.15/katello/el7/x86_64/katello-repos-latest.rpm
yum update -y https://yum.theforeman.org/releases/2.0/el7/x86_64/foreman-release.rpm
yum update -y foreman-release-scl yum clean all

Pre-download packages that will be required by the upgrade

These can’t be pulled with yum because they’re incompatible with currently installed packages.

cd /var/cache/yum/x86_64/7Server/centos-sclo-rh/packages
wget http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-postgresql12-syspaths-3.4-1.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-postgresql12-postgresql-server-syspaths-12.1-2.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-postgresql12-postgresql-syspaths-12.1-2.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-postgresql12-postgresql-contrib-12.1-2.el7.x86_64.rpm
wget http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/rh-postgresql12-postgresql-contrib-syspaths-12.1-2.el7.x86_64.rpm

Pull remaining packages with yum:

yum install --downloadonly rh-postgresql12-postgresql-server rh-redis5-redis pulpcore-selinux foreman-dynflow-sidekiq python3-pulp-2to3-migration python3-pulp-container python3-pulp-file python3-future uuid

The lists of packages to download were determined by trying to run the upgrade multiple times, and catching when it fails or stalls on yum commands.

Do the upgrade

yum update

Finish upgrades

unset http_proxy https_proxy ; foreman-installer --scenario katello --upgrade

(The unset http_proxy https_proxy breaks package downloads, hence the previous steps.)

If you rely on this, I’d recommend setting the http(s) proxy in yum.conf.

With the proxy configured in yum.conf, something about our proxy breaks the certificate auth back to Foreman, so the directly-accessed external Foreman repos become accessible but the local RHEL content views break. We normally depend on no_proxy to ignore the proxy for Foreman, but I don’t think yum.conf supports that.

Also, since our Foreman is subscribed to itself for RHEL content, we’d have to pre-cache packages anyway. Just possibly not as many.

It’s mostly quirks specific to our setup and probably some self-inflicted pain. But if it helps anyone who’s ended up in the same place as the OP and myself…