Foreman upgrade from 3.7.1 katello 4.9 to 3.8 katello 4.10 fails with pulpcore-manager migrate error

trendyandy · December 13, 2023, 10:08am

Problem:
During the upgrade of foreman 3.7.1/4.9 to 3.8.0/4.10 pulpcore-manager migrate --noinput fails with:
ValueError: Found wrong number (0) of constraints for rpm_package(name, epoch, version, release, arch, checksum_type, pkgId)

I tried to regenerate all repositories and content views but no luck. The only thing which is special at that Foreman is that it has two content views which are imported from another foreman (import only)
I also wanted to turn off the “import only” feature that I am able to publish a new version from an upstream repo before upgrading, but the setting is read only in the web ui.

Any help is highly appreciated

Expected outcome:
Upgrade success
Foreman and Proxy versions:
3.7.1
Foreman and Proxy plugin versions:
foreman-tasks 8.1.4
foreman_ansible 12.0.6
foreman_remote_execution 10.1.1
foreman_virt_who_configure 0.5.13
katello 4.9.2

Distribution and version:
rhel 8.9
Other relevant data:
stack trace from foreman-installer:

Error 1: Puppet Exec resource 'pulpcore-manager migrate --noinput' failed. Logs:
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]
    Adding autorequire relationship with File[/var/lib/pulp]
    Adding autorequire relationship with User[pulp]
    Starting to evaluate the resource (1194 of 1340)
    Evaluated in 6.46 seconds
  Exec[pulpcore-manager migrate --noinput](provider=posix)
    Executing check 'pulpcore-manager migrate --plan | grep "No planned migration operations"'
    Executing 'pulpcore-manager migrate --noinput'
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns
    Operations to perform:
      Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, python, rpm, sessions
    Running migrations:
      Applying rpm.0051_alter_distributiontree_unique_together_and_more...Traceback (most recent call last):
      File "/usr/bin/pulpcore-manager", line 33, in <module>
        sys.exit(load_entry_point('pulpcore==3.28.19', 'console_scripts', 'pulpcore-manager')())
      File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
        execute_from_command_line(sys.argv)
      File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
        utility.execute()
      File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
        output = self.handle(*args, **options)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
        res = handle_func(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
        post_migrate_state = executor.migrate(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
        state = self._migrate_all_forwards(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
        state = self.apply_migration(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
        state = migration.apply(state, schema_editor)
      File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
        operation.database_forwards(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/models.py", line 659, in database_forwards
        alter_together(
      File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 554, in alter_unique_together
        self._delete_composed_index(
      File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 610, in _delete_composed_index
        raise ValueError(
    ValueError: Found wrong number (0) of constraints for rpm_package(name, epoch, version, release, arch, checksum_type, pkgId)
    change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

and from manual run:

root@foreman /etc/yum.repos.d (git)-[master] # PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager migrate --noinput                                                                 :(
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, python, rpm, sessions
Running migrations:
  Applying rpm.0051_alter_distributiontree_unique_together_and_more...Traceback (most recent call last):
  File "/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore==3.28.19', 'console_scripts', 'pulpcore-manager')())
  File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/models.py", line 659, in database_forwards
    alter_together(
  File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 554, in alter_unique_together
    self._delete_composed_index(
  File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 610, in _delete_composed_index
    raise ValueError(
ValueError: Found wrong number (0) of constraints for rpm_package(name, epoch, version, release, arch, checksum_type, pkgId)

iballou · December 19, 2023, 4:34pm

I’ve created a pulp_rpm issue: Migration issue pulp_rpm 3.19 -> pulp_rpm 3.22 · Issue #3368 · pulp/pulp_rpm · GitHub

CC @dralley

dralley · January 3, 2024, 5:31am

Hi @trendyandy, I’m looking into this. The first thing to figure out is why it wasn’t encountered sooner.

Do you happen to know which particular version of Katello this installation started with, and when? Was it upgraded from an old installation with Pulp 2 or is it relatively fresh? And have you performed any manual tweaks to the database indexes?

iballou · January 5, 2024, 3:39pm

Looks like another user hit this issue but on rpm_modulemd instead of rpm_package : Upgrade to Foreman 3.8 / Katello 3.10 fails "Found wrong number (0) of constraints for rpm_modulemd(name, stream, version, context, arch)"

trendyandy · January 8, 2024, 5:40am

Hi @dralley,

the installation started with 1.24 (Pulp2) in April or May 2020. I haven’t done any manual tweaks on the database indexes. I upgraded through all versions up to 3.7, including OS upgrade to rhel8 in October 2022. Let me know if you need anything else.

iballou · January 11, 2024, 8:43pm

Seems like starting on Pulp 2 and hitting this bug is a trend so far Upgrade to Foreman 3.8 / Katello 3.10 fails "Found wrong number (0) of constraints for rpm_modulemd(name, stream, version, context, arch)" - #5 by GrzellaA

dralley · January 15, 2024, 2:28pm

@trendyandy

can you check if you have a unique index on the package table containing those fields? and on the modulemd table containing the fields mentioned in the other posting?
can you check .psql_history just to be absolutely certain that the index wasn’t removed?

trendyandy · January 18, 2024, 11:11am

I do not have a .psql_history on that server, Nobody ever did some psql commands there.

Output from rpm_modulemd

pulpcore=# \d rpm_modulemd
                Table "public.rpm_modulemd"
     Column     |  Type   | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
 content_ptr_id | uuid    |           | not null |
 name           | text    |           | not null |
 stream         | text    |           | not null |
 version        | text    |           | not null |
 context        | text    |           | not null |
 arch           | text    |           | not null |
 dependencies   | jsonb   |           | not null |
 artifacts      | jsonb   |           | not null |
 static_context | boolean |           |          |
 snippet        | text    |           | not null |
 description    | text    |           | not null |
 profiles       | jsonb   |           | not null |
Indexes:
    "rpm_modulemd_pkey" PRIMARY KEY, btree (content_ptr_id)
    "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq" UNIQUE CONSTRAINT, btree (name, stream, version, context, arch)
Foreign-key constraints:
    "rpm_modulemd_content_ptr_id_17384ff7_fk_core_content_pulp_id" FOREIGN KEY (content_ptr_id) REFERENCES core_content(pulp_id) DEFERRABLE INITIALLY DEFERRED
Referenced by:
    TABLE "rpm_modulemd_packages" CONSTRAINT "rpm_modulemd_package_modulemd_id_ca6c4d74_fk_rpm_modul" FOREIGN KEY (modulemd_id) REFERENCES rpm_modulemd(content_ptr_id) DEFERRABLE INITIALLY DEFERRED

Output form rpm_package:

pulpcore=# \d rpm_package
                         Table "public.rpm_package"
      Column      |          Type          | Collation | Nullable | Default
------------------+------------------------+-----------+----------+---------
 content_ptr_id   | uuid                   |           | not null |
 name             | character varying(255) |           | not null |
 epoch            | character varying(10)  |           | not null |
 version          | character varying(255) |           | not null |
 release          | character varying(255) |           | not null |
 arch             | character varying(20)  |           | not null |
 pkgId            | text                   |           | not null |
 checksum_type    | text                   |           | not null |
 summary          | text                   |           | not null |
 description      | text                   |           | not null |
 url              | text                   |           | not null |
 changelogs       | jsonb                  |           | not null |
 files            | jsonb                  |           | not null |
 requires         | jsonb                  |           | not null |
 provides         | jsonb                  |           | not null |
 conflicts        | jsonb                  |           | not null |
 obsoletes        | jsonb                  |           | not null |
 suggests         | jsonb                  |           | not null |
 enhances         | jsonb                  |           | not null |
 recommends       | jsonb                  |           | not null |
 supplements      | jsonb                  |           | not null |
 location_base    | text                   |           | not null |
 location_href    | text                   |           | not null |
 rpm_buildhost    | text                   |           | not null |
 rpm_group        | text                   |           | not null |
 rpm_license      | text                   |           | not null |
 rpm_packager     | text                   |           | not null |
 rpm_sourcerpm    | text                   |           | not null |
 rpm_vendor       | text                   |           | not null |
 rpm_header_start | bigint                 |           |          |
 rpm_header_end   | bigint                 |           |          |
 is_modular       | boolean                |           | not null |
 size_archive     | bigint                 |           |          |
 size_installed   | bigint                 |           |          |
 size_package     | bigint                 |           |          |
 time_build       | bigint                 |           |          |
 time_file        | bigint                 |           |          |
 evr              | pulp_evr_t             |           | not null |
Indexes:
    "rpm_package_pkey" PRIMARY KEY, btree (content_ptr_id)
Foreign-key constraints:
    "rpm_package_content_ptr_id_b368ecdc_fk_core_content_pulp_id" FOREIGN KEY (content_ptr_id) REFERENCES core_content(pulp_id) DEFERRABLE INITIALLY DEFERRED
Referenced by:
    TABLE "rpm_modulemd_packages" CONSTRAINT "rpm_modulemd_package_package_id_4ca56326_fk_rpm_packa" FOREIGN KEY (package_id) REFERENCES rpm_package(content_ptr_id) DEFERRABLE INITIALLY DEFERRED
Triggers:
    pulp_evr_insert_trigger BEFORE INSERT ON rpm_package FOR EACH ROW EXECUTE FUNCTION pulp_evr_trigger()
    pulp_evr_update_trigger BEFORE UPDATE OF epoch, version, release ON rpm_package FOR EACH ROW WHEN (old.epoch::text IS DISTINCT FROM new.epoch::text OR old.version::text IS DISTINCT FROM new.version::text OR old.release::text IS DISTINCT FROM new.release::text) EXECUTE FUNCTION pulp_evr_trigger()

trendyandy · January 18, 2024, 11:41am

I found the history file on the old rhel7 server, indeed we did some manual drops:

[root@foreman pgsql]# grep drop .psql_history
alter table rpm_package drop CONSTRAINT rpm_package_name_epoch_version_relea_c9003ffa_uniq;
alter table rpm_package drop constraint "rpm_package_pkgId_key";

trendyandy · January 19, 2024, 2:20pm

Thank you all for your support. @ GrzellaA I followed your strategy and it worked.

I removed all repos which had duplicated rpm entries.
Added the constraint:
alter table rpm_package add constraint rpm_package_name_epoch_version_relea_c9003ffa_uniq" UNIQUE(name, epoch, version, release, arch, checksum_type, "pkgId");

And upgraded successfully to 3.8