Upgrade to Foreman 3.8 / Katello 3.10 fails "Found wrong number (0) of constraints for rpm_modulemd(name, stream, version, context, arch)"

Support
,
1

Upgrade to Foreman 3.8 / Katello 3.10:

Expected outcome:
Upgrade succeeds

Foreman and Proxy versions:
4.9

Distribution and version:
CentOS Stream 8

Other relevant data:

Error 1: Puppet Exec resource 'pulpcore-manager migrate --noinput' failed. Logs:
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]
    Adding autorequire relationship with File[/var/lib/pulp]
    Adding autorequire relationship with User[pulp]
    Starting to evaluate the resource (1813 of 1970)
    Failed to call refresh: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
    'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
    Evaluated in 48.22 seconds
  Exec[pulpcore-manager migrate --noinput](provider=posix)
    Executing check 'pulpcore-manager migrate --plan | grep "No planned migration operations"'
    Executing 'pulpcore-manager migrate --noinput'
    Executing check 'pulpcore-manager migrate --plan | grep "No planned migration operations"'
    Executing 'pulpcore-manager migrate --noinput'
  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns
    Operations to perform:
      Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, python, rpm, sessions
    Running migrations:
      Applying core.0099_versions_field... OK
      Applying core.0100_upstreampulp... OK
      Applying core.0101_add_domain... OK
      Applying core.0102_add_domain_relations... OK
      Applying core.0103_alter_export_task... OK
      Applying core.0104_delete_label... OK
      Applying core.0105_abstract_uuid_gen... OK
      Applying core.0106_alter_artifactdistribution_distribution_ptr_and_more... OK
      Applying ansible.0047_ansible_namespace... OK
      Applying ansible.0048_collectionversionmark... OK
      Applying ansible.0049_rbac_permissions... OK
      Applying ansible.0050_crossrepositorycollectionversionindex... OK
      Applying ansible.0051_cvindex_build... OK
      Applying ansible.0052_alter_ansiblecollectiondeprecated_content_ptr_and_more... OK
      Applying certguard.0002_alter_rhsmcertguard_contentguard_ptr_and_more... OK
      Applying container.0034_translate_signed_schema... OK
      Applying container.0035_alter_blob_content_ptr_and_more... OK
      Applying container.0036_containerpushrepository_pending_blobs_manifests... OK
      Applying core.0107_distribution_hidden... OK
      Applying deb.0021_remove_release_from_structure_types... OK
      Applying deb.0022_alter_aptdistribution_distribution_ptr_and_more... OK
      Applying deb.0023_add_default_signing_services... OK
      Applying deb.0024_add_release_fields... OK
      Applying deb.0025_merge_colliding_structure_content...
        pulp [None]: pulp_deb.app.migrations.0025_merge_colliding_structure_content:INFO: 2023-12-13 09:51:40.100101: Starting data migration!
    pulp [None]: pulp_deb.app.migrations.0025_merge_colliding_structure_content:INFO: 2023-12-13 09:51:40.102199: Data migration completed!
         OK
      Applying deb.0026_aptrepository_publish_upstream_release_fields... OK
      Applying deb.0027_alter_aptpublication_structured... OK
      Applying file.0016_add_domain... OK
      Applying file.0017_alter_filealternatecontentsource_alternatecontentsource_ptr_and_more... OK
      Applying python.0011_alter_pythondistribution_distribution_ptr_and_more... OK
      Applying rpm.0050_alter_addon_pulp_id_alter_checksum_pulp_id_and_more... OK
      Applying rpm.0051_alter_distributiontree_unique_together_and_more...Traceback (most recent call last):
      File "/usr/bin/pulpcore-manager", line 33, in <module>
        sys.exit(load_entry_point('pulpcore==3.28.19', 'console_scripts', 'pulpcore-manager')())
      File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
        execute_from_command_line(sys.argv)
      File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
        utility.execute()
      File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
        output = self.handle(*args, **options)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
        res = handle_func(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
        post_migrate_state = executor.migrate(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
        state = self._migrate_all_forwards(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
        state = self.apply_migration(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
        state = migration.apply(state, schema_editor)
      File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
        operation.database_forwards(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/models.py", line 659, in database_forwards
        alter_together(
      File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 554, in alter_unique_together
        self._delete_composed_index(
      File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 610, in _delete_composed_index
        raise ValueError(
    ValueError: Found wrong number (0) of constraints for rpm_modulemd(name, stream, version, context, arch)
    change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
    Operations to perform:
      Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, python, rpm, sessions
    Running migrations:
      Applying rpm.0051_alter_distributiontree_unique_together_and_more...Traceback (most recent call last):
      File "/usr/bin/pulpcore-manager", line 33, in <module>
        sys.exit(load_entry_point('pulpcore==3.28.19', 'console_scripts', 'pulpcore-manager')())
      File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
        execute_from_command_line(sys.argv)
      File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
        utility.execute()
      File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
        output = self.handle(*args, **options)
      File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
        res = handle_func(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
        post_migrate_state = executor.migrate(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
        state = self._migrate_all_forwards(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
        state = self.apply_migration(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
        state = migration.apply(state, schema_editor)
      File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
        operation.database_forwards(
      File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/models.py", line 659, in database_forwards
        alter_together(
      File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 554, in alter_unique_together
        self._delete_composed_index(
      File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 610, in _delete_composed_index
        raise ValueError(
    ValueError: Found wrong number (0) of constraints for rpm_modulemd(name, stream, version, context, arch)
2

This instance was initially installed as Foreman 1.24 / Katello 3.14. I upgraded it continously to Foreman 3.7 / Katello 4.9.

Table

postgres=# \c pulpcore
You are now connected to database "pulpcore" as user "postgres".
pulpcore=# \d rpm_modulemd
                Table "public.rpm_modulemd"
     Column     |  Type   | Collation | Nullable | Default
----------------+---------+-----------+----------+---------
 content_ptr_id | uuid    |           | not null |
 name           | text    |           | not null |
 stream         | text    |           | not null |
 version        | text    |           | not null |
 context        | text    |           | not null |
 arch           | text    |           | not null |
 dependencies   | jsonb   |           | not null |
 artifacts      | jsonb   |           | not null |
 static_context | boolean |           |          |
 snippet        | text    |           | not null |
 description    | text    |           | not null |
 profiles       | jsonb   |           | not null |
Indexes:
    "rpm_modulemd_pkey" PRIMARY KEY, btree (content_ptr_id)
Foreign-key constraints:
    "rpm_modulemd_content_ptr_id_17384ff7_fk_core_content_pulp_id" FOREIGN KEY (content_ptr_id) REFERENCES core_content(pulp_id) DEFERRABLE INITIALLY DEFERRED
Referenced by:
    TABLE "rpm_modulemd_packages" CONSTRAINT "rpm_modulemd_package_modulemd_id_ca6c4d74_fk_rpm_modul" FOREIGN KEY (modulemd_id) REFERENCES rpm_modulemd(content_ptr_id) DEFERRABLE INITIALLY DEFERRED
3

Looks like you are not alone: Foreman upgrade from 3.7.1 katello 4.9 to 3.8 katello 4.10 fails with pulpcore-manager migrate error

4

Do you happen to know which particular version of Katello this installation started with, and when? Was it upgraded from an old installation with Pulp 2 or is it relatively fresh? And have you performed any manual tweaks to the database indexes?

1 Like
5

@dralley , thank you for your support.
This instance was initially installed as Foreman 1.24 / Katello 3.14 with Pulp 2. I upgraded it continously to Foreman 3.7 / Katello 4.9, including the upgrade to CentOS 8 Stream.
I did not do any manual tweaks to the database indexes.

1 Like
6

Hello,
I just found this RH KB article: Upgrade to Satellite 6.12 fails on duplicate key value violates unique constraint “rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq”.

From my understanding, the index rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq was never created in my installation’s pulpcore DB.

So I’ve executed the query to identify all duplicit records violating uniq index:

su - postgres -c "psql pulpcore -c \"COPY (SELECT name,stream,version,context,arch FROM rpm_modulemd) TO STDOUT WITH (FORMAT 'csv');\"" | sort | uniq -c | sort -n | grep -v "\s*1\s"

      2 pmdk,1-fileformat-v6,8050020210407181047,b4937e53,x86_64
      2 python36,3.6,820231128143707,17efdbc7,x86_64
      8 389-ds,1.4,820231213094812,17499975,x86_64
     16 container-tools,rhel8,820231016065943,20125149,x86_64
     18 389-ds,1.4,820230615185516,17499975,x86_64
     18 389-ds,1.4,820230712181026,17499975,x86_64
     18 389-ds,1.4,820230816162424,17499975,x86_64
     18 container-tools,3.0,8070020220921123812,39077419,x86_64
     18 container-tools,4.0,8070020220830134531,39077419,x86_64
     18 container-tools,4.0,820230615083846,20125149,x86_64
     18 container-tools,4.0,820230710145546,20125149,x86_64
     18 container-tools,4.0,820230807171010,20125149,x86_64
     18 container-tools,4.0,820230828093056,20125149,x86_64
     18 container-tools,rhel8,8070020220315135406,39077419,x86_64
     18 container-tools,rhel8,8070020220830134301,39077419,x86_64
     18 container-tools,rhel8,8070020220921123537,39077419,x86_64
     18 container-tools,rhel8,820230616132605,20125149,x86_64
     18 container-tools,rhel8,820230721032921,20125149,x86_64
     18 container-tools,rhel8,820230811105938,20125149,x86_64
     18 container-tools,rhel8,820230816094934,20125149,x86_64
     18 container-tools,rhel8,820230825121312,20125149,x86_64
     18 idm,DL1,8060020220208184755,92098735,x86_64
     18 idm,DL1,8060020220315134656,92098735,x86_64
     18 subversion,1.10,8070020220512143541,78111232,x86_64
     18 subversion,1.10,8070020220711155714,78111232,x86_64
     19 idm,DL1,8060020211217220554,92098735,x86_64

@dralley, @iballou Sirs, is there a way to verify the schemas of all DBs on my installation before performing any changes?

7

Well that’s not great.

I had some inkling that the problem was likely some missing index. What doesn’t make any sense to me at all is how that index could be missing. It should have been present since the very first migration!

Yet somehow it is missing and it doesn’t seem to be a total fluke because there was a 2nd report

Perplexing.

1 Like
8

Hello @dralley, I just investigated the .psql_history file on the Katello VM.

As a bad surprise I found, that the affected index was dropped manually…
Do you maybe see a chance to recover from this situation?

1 Like
9

In the meanwhile, I identified two repos with these duplicated rpms. After deleting the repos, the numbers of the duplicit records dropped slightly.

My action plan is now this one: after the current patch cycle (starting with low business impact machines → moving to high business imapact machines) I will continue to delete repos to make this duplicit records disappear.
Than I will recreate the index with the following command:
alter table rpm_modulemd add constraint rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq" UNIQUE(name, stream, version, context, arch);

I will report progress.

1 Like
10

Are you experiencing this on multiple independent installations?

Do you think the index deletion potentially occurred as a result of trying to remediate Upgrade to Satellite 6.12 fails on duplicate key value violates unique constraint "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq" - Red Hat Customer Portal? (this seems plausible to me)

1 Like
11

@dralley : Unfortunately, I just have this one installation.

Unfortunately, I do not remember the reason, why I dropped the index. But I completely agree to your suggestion for the root cause

1 Like
12

Here the final status:
after deletion of all CVs and repos containing the packages as identified by the steps outlined in the Red Hat support article [account required] I was able to recreate that index/constraint. The upgrade succeeded now.

@dralley: thank you for your time and support!

2 Likes