'/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

SarathHewage · October 27, 2023, 7:53am

grep memcache /etc/foreman-installer/scenarios.d/katello-answers.yaml
foreman::plugin::memcache: false

evgeni · October 27, 2023, 8:14am

ok, cool.

can you give the foreman service a kick (systemctl restart foreman) and do an installer run.

if that still fails, we’d need the relevant errors /var/log/foreman/production.log again

SarathHewage · October 27, 2023, 8:29am

This is the results.

systemctl restart foreman
[root@sp-flag21 ~]# foreman-installer --no-enable-foreman-plugin-memcache
2023-10-27 16:24:39 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-10-27 16:24:41 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-10-27 16:24:41 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-10-27 16:24:46 [NOTICE] [configure] Starting system configuration.
2023-10-27 16:24:56 [NOTICE] [configure] 250 configuration steps out of 1564 steps complete.
2023-10-27 16:24:59 [NOTICE] [configure] 500 configuration steps out of 1565 steps complete.
2023-10-27 16:25:02 [NOTICE] [configure] 750 configuration steps out of 1569 steps complete.
2023-10-27 16:25:02 [NOTICE] [configure] 1000 configuration steps out of 1574 steps complete.
2023-10-27 16:25:29 [NOTICE] [configure] 1250 configuration steps out of 1574 steps complete.
2023-10-27 16:25:37 [NOTICE] [configure] 1500 configuration steps out of 1574 steps complete.
2023-10-27 16:25:39 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/2: katello:correct_repositories. This may take a long while.
=============================================
Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
  Success!
  * Foreman is running at https://abc.abc.com.au
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  * Foreman Proxy is running at https://abc.abc.com.au:9090

  The full log is at /var/log/foreman-installer/katello.log

Thanks, heaps, for the help.

So the solution is:

dnf remove rubygem-foreman_memcache
systemctl restart foreman

And then run:

foreman-installer --no-enable-foreman-plugin-memcache

Is that correct ?

evgeni · October 27, 2023, 8:31am

Yes.

SarathHewage · October 27, 2023, 8:43am

OK, Thanks again for the help.
Now I will take a snap and continue with next upgrade, 3.5 and so on.
Probably I need help along the way, so will post the issue here.

SarathHewage · October 31, 2023, 1:29am

Hi, I am again.

I manage to successfully update up to 3.5.
But when doing 3.6 upgrade, following error is generated:

foreman-installer
2023-10-31 08:59:13 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-10-31 08:59:16 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-10-31 08:59:16 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-10-31 08:59:20 [NOTICE] [configure] Starting system configuration.
2023-10-31 08:59:29 [NOTICE] [configure] 250 configuration steps out of 1348 steps complete.
2023-10-31 08:59:32 [NOTICE] [configure] 500 configuration steps out of 1350 steps complete.
2023-10-31 08:59:34 [NOTICE] [configure] 750 configuration steps out of 1353 steps complete.
2023-10-31 08:59:34 [NOTICE] [configure] 1000 configuration steps out of 1357 steps complete.
2023-10-31 09:00:06 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2023-10-31 09:00:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2023-10-31 09:00:06 [NOTICE] [configure] 1250 configuration steps out of 1357 steps complete.
2023-10-31 09:00:10 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

When I check /var/log/foreman-installer/katello.log, found these lines:

2023-10-31 09:00:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return self.cursor.execute(sql, params)
2023-10-31 09:00:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: django.db.utils.ProgrammingError: permission denied to create extension "hstore"
2023-10-31 09:00:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: HINT:  Must be superuser to create this extension.
2023-10-31 09:00:06 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2023-10-31 09:00:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

Can you help me to overcome this issue, please?

evgeni · October 31, 2023, 6:21am

Uh, yes, the Pulp in 3.6 needs HStore, but we have code (via Feature #36030: Enable the HStore module in postgres for pulpcore 3.22 upgrade - Installer - Foreman / Fixes #36030 - Ensure HStore is enabled · theforeman/puppet-pulpcore@2a5b8dd · GitHub ) to enable that before Pulp tries to (as the way it does doesn’t work with our setup, as you see yourself right now).

Can you please post the output of rpm -qa |grep foreman-installer and grep -i 'hstore for pulpcore' /var/log/foreman-installer/katello.log?

SarathHewage · October 31, 2023, 8:21am

Here are the outputs of those commands:

rpm -qa |grep foreman-installer
foreman-installer-3.6.2-1.el8.noarch
foreman-installer-katello-3.6.2-1.el8.noarch

Other command gives a blank output.

sudo grep -i 'hstore for pulpcore' /var/log/foreman-installer/katello.log

evgeni · October 31, 2023, 9:05am

Oooh, wait.

Are you running this with an external DB? So PostgreSQL is installed on a different machine?

evgeni · October 31, 2023, 9:32am

If you are, install postgresql-contrib on the external DB and restart PostgreSQL, it should work afterwards.

SarathHewage · October 31, 2023, 9:50am

PostgeSQL is on a different server.

SarathHewage · November 1, 2023, 8:18am

Hi,
I am still trying download and install postgresql-contri on ‘Oracle Linux Server release 7.9’, 5.4.17-2136.315.5.el7uek.x86_64
Unable to get the correct download url
Tried in
5.4.17-2136.315.5.el7uek.x86_64
and
https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/

lumarel · November 1, 2023, 10:01am

Hey there,
This package should come directly from your OL7 repository

SarathHewage · November 2, 2023, 4:58am

Hi I ran the rpm -qa command on the PostgeSQL server and found this:

rpm -qa | grep postgresql-contrib
postgresql-contrib-9.2.24-8.el7_9.x86_64
rh-postgresql12-postgresql-contrib-12.11-1.el7.x86_64

This means, postgresql-contrib is already installed, isn’t it?

evgeni · November 2, 2023, 9:42am

Yes.

Can you try creating the extension manually please?

Something like echo 'create extension if not exists "hstore";' | sudo -u postgres psql pulpcore should do.

SarathHewage · November 3, 2023, 1:24am

Yes, it did work. Thanks you very much.

 foreman-installer
2023-11-03 09:18:40 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-11-03 09:18:43 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-11-03 09:18:43 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-11-03 09:18:47 [NOTICE] [configure] Starting system configuration.
2023-11-03 09:18:56 [NOTICE] [configure] 250 configuration steps out of 1348 steps complete.
2023-11-03 09:18:59 [NOTICE] [configure] 500 configuration steps out of 1350 steps complete.
2023-11-03 09:19:01 [NOTICE] [configure] 750 configuration steps out of 1353 steps complete.
2023-11-03 09:19:01 [NOTICE] [configure] 1000 configuration steps out of 1357 steps complete.
2023-11-03 09:19:38 [NOTICE] [configure] 1250 configuration steps out of 1357 steps complete.
2023-11-03 09:19:50 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/4: katello:correct_repositories. This may take a long while.
=============================================
Upgrade Step 2/4: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
=============================================
Upgrade Step 3/4: katello:upgrades:4.8:fix_incorrect_providers. Fixing incorrect providers
Fixed 0 incorrect providers
Cleaning Candlepin orphaned custom products for organization InfraOps
Deleted 0 Candlepin orphaned custom products for organization InfraOps
=============================================
Upgrade Step 4/4: katello:upgrades:4.8:regenerate_imported_repository_metadata. No repositories found for regeneration.
  Success!
  * Foreman is running at https://abc.abc.com.au
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  * Foreman Proxy is running at https://abs.abc.com.au:9090

  The full log is at /var/log/foreman-installer/katello.log

evgeni · November 3, 2023, 8:11am

Nice.

3.7 is next?

SarathHewage · November 22, 2023, 8:32am

Hi,
My project to upgrade Our Foreman from 3,2 to 3.8 is done.
We just completed upgrading Foreman Prod server today.
Thanks a lot for your valuable tech advice.

Damiano · January 16, 2026, 10:26am

It might be a locale issue: I followed these steps, and a fresh install on Ubuntu 22.04 and Foreman 3.17 works.

apt-get install locales

Run ‘sudo dpkg-reconfigure locales’

Check `en_US.UAT-8` in the list Locales

to be generated: 160

Default locale for the system environment: en_US.UTF-8

Reboot the server

Check if the locale is properly installed using `locale -a`

Rerun `foreman-installer`