Katello upgrade from 4.4 to 4.4.0.2 still fails during remove duplicate errata

Support
1

**Problem:Katello upgrade from 4.4 to 4.4.0.2 still fails during remove duplicate errata

**Expected outcome:Removal of duplicate erratas goes smoothly

**Foreman and Proxy versions:katello 4.4.0.2

Foreman and Proxy plugin versions:

**Distribution and version:RHEL7

**This problem should have been solved with Fixes #34703 - Destroy duplicate errata child records by sjha4 · Pull Request #10041 · Katello/katello · GitHub but upgrade receives another error:
2022-04-12 22:25:50 [DEBUG ] [configure] Executing with uid=foreman: ‘/usr/sbin/foreman-rake db:abort_if_pending_migrations’
2022-04-12 22:26:02 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: Run rails db:migrate to update your database then try again.
2022-04-12 22:26:02 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: You have 1 pending migration:
2022-04-12 22:26:02 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: 20220303160220 RemoveDuplicateErrata
2022-04-12 22:26:02 [DEBUG ] [configure] Execforeman-rake-db:migrate: Executing ‘/usr/sbin/foreman-rake db:migrate’
2022-04-12 22:26:02 [DEBUG ] [configure] Executing with uid=foreman: ‘/usr/sbin/foreman-rake db:migrate’
2022-04-12 22:27:02 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
2022-04-12 22:27:02 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: StandardError: An error has occurred, this and all later migrations canceled:
2022-04-12 22:27:02 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:
2022-04-12 22:27:02 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: PG::ForeignKeyViolation: ERROR: update or delete on table “katello_errata” violates foreign key constraint “katello_content_facet_errata_errata_id” on table “katello_content_facet_errata”
**

2

This seems like a miss on our part in the migration. We overlooked the katello_content_facet_errata dependency.
You can run the snippet below in foreman-rake console to clean this data for you prior to the upgrade and you should be able to proceed with the upgrade.

::Katello::Erratum.group(:errata_id).having("count(errata_id) > 1").pluck(:errata_id).each do |original_errata_id|
   errata_to_keep = ::Katello::Erratum.find_by(pulp_id: original_errata_id)      
   errata_all = ::Katello::Erratum.where(errata_id: original_errata_id)      
   dup_errata = errata_all - [errata_to_keep]      
   dup_errata_ids = dup_errata&.pluck(:id)
   if dup_errata_ids&.present?
     ::Katello::ContentFacetErratum.where(:erratum_id => dup_errata_ids).delete_all
   end  
end

Thanks for raising this and we’ll fix this in 4.4.1 when that is released.

3

Ok, thank you very much. I already moved the file 20220303160220_remove_duplicate_errata.rb to backup directory so the upgrade went out without the remove duplicate errata task. I do not know now whether I should wait for the official update to 4.4.1 or I should run your hack, move the file back, and run the installer again ?

4

I’d probably suggest waiting for 4.4.1 so we can get you a supported migration script.

The duplicate errata in ContentFacetErratum table is giving me pause here. I was wondering if you would be able to give me some sense of how you ended up with the duplicate errata in the system and what those duplicates look like to better debug the root cause. Did you upgrade this box from a pulp2 setup (version < 3.17) in the past?

An output for something like this will be useful for debugging and getting a sense of how many duplicates are there on the box and how those came to be.

::Katello::Erratum.group(:errata_id).having("count(errata_id) > 1").pluck(:errata_id).each do |original_errata_id|
   errata_to_keep = ::Katello::Erratum.find_by(pulp_id: original_errata_id)      
   errata_all = ::Katello::Erratum.where(errata_id: original_errata_id)      
   dup_errata = errata_all - [errata_to_keep]      
   p errata_to_keep, dup_errata
end

We want to make sure we don’t lose the ContentFacetErratum data if the only record tying the host to the errata uses the duplicate record. You’d probably have to run applicability regeneration if that happens and there’s probably a better way to handle that during migration.

5

Yes, the system has been upgraded gradually from 3.1x, I do not know exactly, to 3.18, then we performed pulp2to3 migration and continued gradually to 4.4.

I ran the commands you asked me for but it returned so much data that I had to break it with Ctrl+C. If you tell me how to redirect it to some file I can do it.

6

I did not realize there’d be a lot of duplicates. Just a partial output will do…Just want to know what the pulp_ids of duplicate errata looks like.

For the migration part, we’ll need to ensure the host is tied to the original errata before we delete duplicates from the ContentFacetErratum table. The snippet I had from before does not do that so I’d hold off on that or the applicability regeneration for the host would need to be run.

We have an issue filed for the migration Bug #34771: Katello upgrade from 4.4 to 4.4.0.2 still fails during remove duplicate errata - Katello - Foreman. Would love to have you test that once the PR is ready.

7

[#<Katello::Erratum id: 28814, pulp_id: “3d6bfae0-6830-425d-b00e-cd2d32c6fb57”, errata_id: “FEDORA-EPEL-2016-24b869cc40”, created_at: “2021-03-30 11:13:55”, updated_at: “2021-03-30 11:13:55”, issued: “2016-05-14”, updated: “2020-11-30”, errata_type: “bugfix”, severity: “None”, title: “ipsec-tools-0.8.2-2.el6”, solution: “”, description: “Resolves: rhbz#1251691 ifup-ipsec causes invalid I…”, summary: “ipsec-tools-0.8.2-2.el6 bugfix update”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>]
#<Katello::Erratum id: 57612, pulp_id: “RHBA-2016:2163”, errata_id: “RHBA-2016:2163”, created_at: “2022-04-12 01:07:11”, updated_at: “2022-04-12 01:07:11”, issued: “2016-11-03”, updated: “2016-11-03”, errata_type: “bugfix”, severity: “None”, title: “openldap bug fix and enhancement update”, solution: “For details on how to apply this update, refer to:…”, description: “OpenLDAP is an open-source suite of Lightweight Di…”, summary: “An update for openldap is now available for Red Ha…”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>
[#<Katello::Erratum id: 5827, pulp_id: “78101442-7105-4b87-b804-bd7361d5264a”, errata_id: “RHBA-2016:2163”, created_at: “2019-06-26 10:46:32”, updated_at: “2019-06-26 10:46:32”, issued: “2016-11-03”, updated: “2016-11-03”, errata_type: “bugfix”, severity: “None”, title: “openldap bug fix and enhancement update”, solution: “For details on how to apply this update, refer to:…”, description: “OpenLDAP is an open-source suite of Lightweight Di…”, summary: “An update for openldap is now available for Red Ha…”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>]
#<Katello::Erratum id: 47411, pulp_id: “FEDORA-EPEL-2019-ddb988e2ec”, errata_id: “FEDORA-EPEL-2019-ddb988e2ec”, created_at: “2022-04-12 01:20:10”, updated_at: “2022-04-12 01:20:10”, issued: “2019-10-25”, updated: “2019-10-07”, errata_type: “enhancement”, severity: “None”, title: “fcgi-2.4.0-36.el8”, solution: “”, description: “Rebuilt for Fedora’s Mission and Foundations :: Fedora Docs…”, summary: “fcgi-2.4.0-36.el8 enhancement update”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>
[#<Katello::Erratum id: 15433, pulp_id: “93ca37b8-63b4-4570-a902-c9697c63bc38”, errata_id: “FEDORA-EPEL-2019-ddb988e2ec”, created_at: “2020-12-11 15:14:08”, updated_at: “2021-05-14 02:23:27”, issued: “2019-10-25”, updated: “2021-05-13”, errata_type: “enhancement”, severity: “None”, title: “fcgi-2.4.0-36.el8”, solution: “”, description: “Rebuilt for Fedora’s Mission and Foundations :: Fedora Docs…”, summary: “fcgi-2.4.0-36.el8 enhancement update”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>]
#<Katello::Erratum id: 50308, pulp_id: “FEDORA-EPEL-2014-3039”, errata_id: “FEDORA-EPEL-2014-3039”, created_at: “2022-04-12 01:20:38”, updated_at: “2022-04-12 01:20:38”, issued: “2014-10-01”, updated: “2014-09-30”, errata_type: “newpackage”, severity: “None”, title: “perl-Digest-PBKDF2-0.010-1.el7”, solution: “”, description: “Initial release”, summary: “perl-Digest-PBKDF2-0.010-1.el7 newpackage update”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>
[#<Katello::Erratum id: 1685, pulp_id: “49b4bf7b-a6a4-4171-b783-95061e38110f”, errata_id: “FEDORA-EPEL-2014-3039”, created_at: “2019-06-26 10:39:43”, updated_at: “2021-05-13 02:18:00”, issued: “2014-10-01”, updated: “2021-05-12”, errata_type: “newpackage”, severity: “None”, title: “perl-Digest-PBKDF2-0.010-1.el7”, solution: “”, description: “Initial release”, summary: “perl-Digest-PBKDF2-0.010-1.el7 newpackage update”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>]
#<Katello::Erratum id: 55301, pulp_id: “RHSA-2014:0705”, errata_id: “RHSA-2014:0705”, created_at: “2022-03-31 01:03:13”, updated_at: “2022-03-31 01:03:13”, issued: “2014-06-10”, updated: “2014-06-10”, errata_type: “security”, severity: “Critical”, title: “Critical: java-1.7.1-ibm security update”, solution: “Before applying this update, make sure all previou…”, description: “IBM Java SE version 7 Release 1 includes the IBM J…”, summary: “Updated java-1.7.1-ibm packages that fix several s…”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>
[#<Katello::Erratum id: 44, pulp_id: “d41fa3ec-e5be-48b9-a785-b64889b8db15”, errata_id: “RHSA-2014:0705”, created_at: “2019-06-26 10:29:16”, updated_at: “2019-06-26 10:29:16”, issued: “2014-06-10”, updated: “2014-06-10”, errata_type: “security”, severity: “Critical”, title: “Critical: java-1.7.1-ibm security update”, solution: “Before applying this update, make sure all previou…”, description: “IBM Java SE version 7 Release 1 includes the IBM J…”, summary: “Updated java-1.7.1-ibm packages that fix several s…”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>]
#<Katello::Erratum id: 57628, pulp_id: “RHBA-2016:1302”, errata_id: “RHBA-2016:1302”, created_at: “2022-04-12 01:07:11”, updated_at: “2022-04-12 01:07:11”, issued: “2016-06-23”, updated: “2016-06-23”, errata_type: “bugfix”, severity: “None”, title: “zsh bug fix update”, solution: “Before applying this update, make sure all previou…”, description: “The zsh shell is a command interpreter usable as a…”, summary: “Updated zsh packages that fix several bugs are now…”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>
[#<Katello::Erratum id: 6275, pulp_id: “a44e7136-e7f4-419f-a8e3-61dbb450699c”, errata_id: “RHBA-2016:1302”, created_at: “2019-06-26 10:46:59”, updated_at: “2019-06-26 10:46:59”, issued: “2016-06-23”, updated: “2016-06-23”, errata_type: “bugfix”, severity: “None”, title: “zsh bug fix update”, solution: “Before applying this update, make sure all previou…”, description: “The zsh shell is a command interpreter usable as a…”, summary: “Updated zsh packages that fix several bugs are now…”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>]
#<Katello::Erratum id: 73017, pulp_id: “FEDORA-EPEL-2015-0914”, errata_id: “FEDORA-EPEL-2015-0914”, created_at: “2022-04-05 13:37:58”, updated_at: “2022-04-05 13:37:58”, issued: “2015-02-23”, updated: “2020-11-30”, errata_type: “bugfix”, severity: “None”, title: “ansifilter-1.11-1.el6”, solution: “”, description: “== ansifilter 1.11 ==\r\n\r\n-fixed bold and italic in…”, summary: “ansifilter-1.11-1.el6 bugfix update”, reboot_suggested: false, migrated_pulp3_href: nil, missing_from_migration: false, ignore_missing_from_migration: false>

1 Like
8

That helps and indeed it was caused by some changes between how we index errata between pulp2 and pulp3…

I have a pull request here: Fixes #34771 - Delete duplicate ContentFacetErratum records during migration by sjha4 · Pull Request #10063 · Katello/katello · GitHub that will be released with 4.4.1 that should take care of the content host erratum records while migrating and delete the duplicate errata records (You shouldn’t have 2 errata records with the same errata_id and errata pulp_id should be equal to errata_id).

1 Like