Upgrade from 1.24.3 to 2.0 fails

Hi,

I’m having trouble upgrading from 1.24.3. I followed the instructions from Katello 3.15 and the first problem I encountered was that the user pulp was missing. So I did “useradd -s /sbin/nologin -g pulp pulp”, but the next test failed again: “role “pulp” does not exist” in PostgreSQL.

What could be missing here? Thanks.

Best,
Hans-Joachim

Problem:
Tried the Katello 3.15 upgrade with

foreman-installer --scenario katello --upgrade --noop --dont-save-answers --verbose

After manually adding the user pulp and repeating the above command I got:

/Stage[main]/Pulpcore::Database/Postgresql::Server::Db[pulpcore]/Postgresql::Server::Database_grant[GRANT pulp - ALL - pulpcore]/Postgresql::Server::Grant[database:GRANT pulp - ALL - pulpcore]/Postgresql_psql[grant:database:GRANT pulp - ALL - pulpcore]: Could not evaluate: Error evaluating 'unless' clause, returned pid 117217 exit 1: 'ERROR:  role "pulp" does not exist

Expected outcome:
Upgrade to Foreman 2.0.x and Katello 3.15 completed.

Foreman and Proxy versions:
Foreman 1.24.3, Katello 3.14.1

Distribution and version:
CentOS 7.7

Hi and welcome to the forum :slight_smile:

Have you stopped the foreman stack befpr running the installer command?
If yes, a run with “–noop” will always fail on some of the postgres steps, because the DB is shut down and so a connection to do some of the checks within the DB is not possible. During an actual run, the installer would ensure all the services are started before they are dealt with, but in a noop run this is obviously not the case.

Regards

This is correct, but you can (and should) start the PostgreSQL server before a noop run when you can. That makes the run much more informative.

Hi,

thanks for the welcome - I’ll write a proper introduction later when I find time :slight_smile:

I ran the installer without shutting down Foreman because from what I saw in the log it does that automatically. Now I started the installer after shutting down Foreman, and without the " --noop --dont-save-answers --verbose" options. Now I get:

Execution of ‘/usr/bin/yum -d 0 -e 0 -y install rh-postgresql12-postgresql-syspaths’ returned 1: Error: rh-postgresql12-postgresql-syspaths conflicts with postgresql-9.2.24-2.el7_7.x86_64
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

I installed rh-postgresql12-postgresql-server before running the installer. There were no conflicts or problems. I did this because when Foreman is down, yum doesn’t work anymore. This is because we use Foreman as package source, also for itself. So I have one more question: Is it an allowed configuration to have Foreman as its own package source or must this be handled differently?

Thanks,
Hans-Joachim

Allowed is a bad phrasing. It’s open source software so you’re allowed to run your server as you like it. In general it’s not recommended exactly because as you say, the server starts to depend on itself. In many cases it’ll work just fine but it if it doesn’t, recovering is more complicated. Since most admins don’t enjoy doing complicated work during a service disruption, the general advice is to avoid it.

At some point I’d like to be able to support what I call a split deployment where Foreman is deployed on one host and Pulp on another. That would probably make this easier to use. I’d like to hear from people who deploy this in production how they think about that. Back when I was an admin I always liked to separate things into smaller pieces but maybe I was in a minority with that.

I am running a self-registred Katello here for years.
I can confirm it works most of the time, but updates take a lot of preparation work in dev and testing because of the self-dependency loop.
@ekohl I would really appreciate a way to make this scenario less complicated. There are definetly some reasonable arguments for both workflows (e.g. easier upgrades vs. the benefits of Katello’s content management and staging concepts). With excessiv testing in to stages prior to production, there have not been any problems we could not work around in the last 4 years, but it is in fact complicated.

With only about half a year of experience with Foreman, it’s too early for me to understand the update workflows. Do you have some advice how to do it with a self-hosted Katello? And do you have an idea why there is a conflict between the RPM packages? I had thought the PostgreSQL 9.4 and 12 packages would have nothing in common and could be easily installed in parallel. How else could a migration from 9.4 to 12 work?

They themselves don’t, so the upgrade flow is to install them in parallel, migrate over. However, we then remove the old postgres and install rh-postgresql12-postgresql-syspaths which does install to /usr/bin/pgsql and a postgresql service. That is a conflict.

I’d recommend looking at the code itself because it’s pretty straight forward.

Perhaps the installation created /var/opt/rh/rh-postgresql12/lib/pgsql/data so the upgrade is not triggered?

Many thanks, Ewould! That hint was the breakthrough. After that, I got it all working. The necessary steps were like that:

  • Install rh-postgresql12-postgresql-contrib-syspaths rh-postgresql12-postgresql-libs rh-postgresql12-syspaths rh-postgresql12-postgresql rh-postgresql12-postgresql-server rh-postgresql12-postgresql-server-syspaths rh-postgresql12-postgresql-syspaths rh-postgresql12-runtime rh-postgresql12 rh-postgresql12-postgresql-contrib
  • If there are package conflicts, leave these packages for later
  • remove empty directory /var/opt/rh/rh-postgresql12/lib/pgsql/data
  • run installer with our customized options
  • If Installer fails because it could not install some packages start Foreman (there should be enough of the update in place so that the most necessary components can run) and install the packages
  • yum install rh-ruby25-rubygem-xmlrpc
  • re-run installer
  • If problems remain, analyze them. In our case, there were no more problems.

Thanks again for all your help!

Best,
Hans-Joachim

So this directory really is empty? If so, we can make our check more robust by verifying there are no files in it rather than just its presence. I think I’ve seen/heard a few more examples of users who ran into this.

This is Bug #30040: Centos Foreman 2.0 new install can't download xmlrpc - Packaging - Foreman and will be fixed in 2.0.2; it is fixed in 2.1.0.

Yes, for me it was indeed empty.