Error upgrading to Foreman 3.13 / Katello 4.15 - 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

Support
, ,
1

Problem:
After running foreman-installer during the upgrade process, I receive the following output/error:

2025-02-24 10:11:43 [NOTICE] [root] Loading installer configuration. This will take some time.
2025-02-24 10:11:45 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2025-02-24 10:11:45 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2025-02-24 10:11:47 [NOTICE] [checks] System checks passed
2025-02-24 10:11:51 [NOTICE] [configure] Starting system configuration.
2025-02-24 10:11:58 [NOTICE] [configure] 250 configuration steps out of 1537 steps complete.
2025-02-24 10:11:59 [NOTICE] [configure] 500 configuration steps out of 1669 steps complete.
2025-02-24 10:11:59 [NOTICE] [configure] 750 configuration steps out of 1671 steps complete.
2025-02-24 10:12:02 [NOTICE] [configure] 1000 configuration steps out of 1679 steps complete.
2025-02-24 10:12:02 [NOTICE] [configure] 1250 configuration steps out of 1682 steps complete.
2025-02-24 10:12:23 [NOTICE] [configure] 1500 configuration steps out of 1682 steps complete.
2025-02-24 10:12:27 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2025-02-24 10:12:27 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2025-02-24 10:12:33 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Exec resource 'pulpcore-manager migrate --noinput' failed. Logs:
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]
    Adding autorequire relationship with File[/var/lib/pulp]
    Adding autorequire relationship with User[pulp]
    Starting to evaluate the resource (1540 of 1682)
    Evaluated in 3.45 seconds
  Exec[pulpcore-manager migrate --noinput](provider=posix)
    Executing 'pulpcore-manager migrate --noinput'
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns
    Operations to perform:
      Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, python, rpm, sessions
    Running migrations:
      Applying core.0120_get_url_removal...Traceback (most recent call last):
      File "/usr/bin/pulpcore-manager", line 33, in <module>
        sys.exit(load_entry_point('pulpcore==3.63.1', 'console_scripts', 'pulpcore-manager')())
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/pulpcore/app/manage.py", line 11, in manage
        execute_from_command_line(sys.argv)
      File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
        utility.execute()
      File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
        output = self.handle(*args, **options)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
        res = handle_func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
        post_migrate_state = executor.migrate(
                             ^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
        state = self._migrate_all_forwards(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
        state = self.apply_migration(
                ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
        state = migration.apply(state, schema_editor)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
        operation.database_forwards(
      File "/usr/lib/python3.11/site-packages/pulpcore/migrations.py", line 48, in database_forwards
        raise RuntimeError(
    RuntimeError: Incompatible versions detected (core >= 3.50.0 needed):
      - 'core'='3.49.22' with api server '54402@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with api server '54279@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with api server '54294@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with api server '54292@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with api server '54305@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54411@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54287@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54302@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54306@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54307@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54310@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54404@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54410@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with content server '54409@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with pulp worker '56410@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with pulp worker '56411@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with pulp worker '56413@sys-fore-0p03.unbc.ca'
      - 'core'='3.49.22' with pulp worker '56412@sys-fore-0p03.unbc.ca'
    Please shutdown or upgrade the outdated components before you continue the migration.
    change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/katello.log

Expected outcome:
Foreman installer completes successfully

Foreman and Proxy versions:
GUI now says I am on 3.13.0
Prior to upgrade attempt, it was 3.12.1

Foreman and Proxy plugin versions:

  • foreman-tasks 10.0.1
  • foreman_ansible 15.0.1
  • foreman_bootdisk 22.0.2
  • foreman_remote_execution 14.0.2
  • katello 4.15.0

Distribution and version:
AlmaLinux 9.5

Other relevant data:

2

Just a guess, have you set up pulpcore repositories for katello correctly ? It should point to version 3.63 in katello 4.15, version 3.49 is from katello 4.13 and 4.14.

3

I don’t recall ever manually setting up pulpcore repositories. How does one do that?

4

Well, it is done automatically through katello-repos rpm package in this step of upgrade:

dnf upgrade https://yum.theforeman.org/releases/3.13/el9/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/4.15/katello/el9/x86_64/katello-repos-latest.rpm

and it sets the correct version in /etc/yum.repos.d/katello.repo file.

Maybe you should check if you went through all the steps in Upgrading Foreman to 3.13 correctly.

5

Here’s my current /etc/yum.repos.d/katello.repo file:

# Place this file in your /etc/yum.repos.d/ directory

[katello]
name=Katello 4.15
baseurl=https://yum.theforeman.org/katello/4.15/katello/el9/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=1
gpgcheck=1

# Candlepin RPMs as supported by Katello - This is a coordinated
# copy of Candlepin's packages in order to ensure compatibility

[candlepin]
name=Candlepin: an open source entitlement management system.
baseurl=https://yum.theforeman.org/candlepin/4.4/el9/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-candlepin
enabled=1
gpgcheck=1

[pulpcore]
name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
baseurl=https://yum.theforeman.org/pulpcore/3.63/el9/$basearch/
gpgkey=https://yum.theforeman.org/pulpcore/3.63/GPG-RPM-KEY-pulpcore
enabled=1
gpgcheck=1

# source repositories

[katello-source]
name=Katello 4.15 Source
baseurl=https://yum.theforeman.org/katello/4.15/katello/el9/source/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=0
gpgcheck=1

[candlepin-source]
name=Katello Candlepin source
baseurl=https://yum.theforeman.org/candlepin/4.15/el9/source/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-candlepin
enabled=0
gpgcheck=1

[pulpcore-source]
name=pulpcore source
baseurl=https://yum.theforeman.org/pulpcore/3.63/el9/source/
gpgkey=https://yum.theforeman.org/pulpcore/3.63/GPG-RPM-KEY-pulpcore
enabled=0
gpgcheck=1

Looks like it has the correct pulpcore repo version (3.63) for Katello 4.15?

I did a bit more looking in this forum, and found this, which seems possibly relevant: Katello-nightly-rpm-pipeline 2155 failed - Infra & CI - TheForeman

6

Maybe you should check if you went through all the steps in Upgrading Foreman to 3.13 correctly.

You called it! I looked back through my command history and realized I missed step 10, which was to stop the foreman services after updating the repos, but before running the dnf upgrade command for the second time. I reverted to my pre-upgrade snapshot, did the upgrade again with all the steps, and it worked. Thank you!

2 Likes