[ContentMigration] switchover failed: ERROR: at least one Erratum record has migrated_pulp3_href NULL value

Support
1

After I have changed my content view to work around the repository migration issue in topic [ContentMigration] switchover fails at katello:pulp3_post_migration_check I just ran into the next issue:

Now the switchover fails with this error:

Switching specified content over to pulp 3                            [FAIL]
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
 ERROR: at least one Erratum record has migrated_pulp3_href NULL value

I have checked in the foreman postgres database:

foreman=# select count(*) from katello_errata where migrated_pulp3_href is null;
 count 
-------
 12651
(1 row)

foreman=# select count(*) from katello_errata where migrated_pulp3_href is not null;
 count 
-------
     0
(1 row)

So it seems none of my errata have been migrated during the prepare run…

2

errata are a little different and you have to query a different table:

select count(*) from katello_repository_errata where erratum_pulp3_href is null;

what does this command show:

foreman-rake content migration-stats
3

I just ran another successful prepare run. Now it looks like this:

foreman=# select count(*) from katello_repository_errata where erratum_pulp3_href is null;
 count 
-------
  7989
(1 row)

foreman=# select count(*) from katello_repository_errata where erratum_pulp3_href is not null;
 count 
-------
 54646
(1 row)
foreman=# \q
[root@foreman tmp]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: 
============Migration Summary================
Migrated/Total RPMs: 140529/140529
Migrated/Total errata: 54646/62635
Migrated/Total repositories: 1335/1343

Estimated migration time based on yum content: 2 hours, 15 minutes
...
4

Here is the list of repositories with missing errata:

foreman=# select katello_repositories.id, katello_repositories.relative_path,count(*) from katello_repository_errata left join katello_repositories on katello_repositories.id = katello_repository_errata.repository_id where erratum_pulp3_href is null group by katello_repositories.id;
  id   |                           relative_path                            | count 
-------+--------------------------------------------------------------------+-------
  2093 | ORG/Library/content/dist/rhel/server/7/7Server/x86_64/optional/os |  3160
   178 | ORG/Library/custom/epel8/epel-modular                             |    11
 43067 | ORG/content_views/centos8-epel8/236.0/custom/epel8/epel           |     3
 43245 | ORG/content_views/centos8-epel8/237.0/custom/epel8/epel           |     3
 42766 | ORG/content_views/centos8-epel8/235.0/custom/epel8/epel           |    23
  2092 | ORG/Library/content/dist/rhel/server/7/7Server/x86_64/os          |  4789
(6 rows)

Looks like all RHEL7 errata and all EPEL8 Modular. EPEL7 and EPEL8 non modular are all good.

5

To compare, what has been migrated:

foreman=# select katello_repositories.id, katello_repositories.relative_path,count(*) from katello_repository_errata left join katello_repositories on katello_repositories.id = katello_repository_errata.repository_id where erratum_pulp3_href is not null group by katello_repositories.id having relative_path like '%/Library/%';
  id   |                           relative_path                            | count 
-------+--------------------------------------------------------------------+-------
 37224 | DKRZ/Library/centos8-epel8/custom/epel8/epel                       |  2952
     7 | DKRZ/Library/custom/epel7/epel_x86_64                              |  4678
   177 | DKRZ/Library/custom/epel8/epel                                     |  2953
  2092 | DKRZ/Library/content/dist/rhel/server/7/7Server/x86_64/os          |     1
 40107 | DKRZ/Library/centos7/custom/epel7/epel_x86_64                      |     8
 42928 | DKRZ/Library/centos7-epel7/custom/epel7/epel_x86_64                |  4660
  2093 | DKRZ/Library/content/dist/rhel/server/7/7Server/x86_64/optional/os |     1
 37163 | DKRZ/Library/centos8/custom/epel8/epel                             |     9
(8 rows)
7

By chance were you running the migration with Fixes #32663 - ensure unique name for each migrated repo by jlsherrill · Pull Request #9385 · Katello/katello · GitHub applied locally?

8

No. I have never applied any fixes regarding that issue, nor any others.

Instead I have removed one repository with the duplicate name from the content view, republished and removed all old versions before that run as a workaround.

(I have put back the repository since then…)

9

the output is showing:

Migrated/Total repositories: 1335/1343

indicating that ~8 repos aren’t migrated. This could point to the problem. I wonder if there are other instances of Bug #32663: Pulp3 migration switchover fails when a content view contains two repositories with the same label - Katello - Foreman ?

If you want to capture your full migration plan from your logs and upload it to the issue, i’m happy to take a look.

10

Yes, those are the 8 CV repos which don‘t get migrated. I have temporarily removed one of those repos to avoid the issue and ran migrate & switchover which then lead to the error in this topic (because switchover stops at the first error it finds instead of showing all errors). All repositories had been migrated but errata was and is incomplete.

After it failed and I have started this topic, I have put the repository back into the content view because I need it. My last prepare run and the stats are with the repository in there.

11

To be sure, I have just removed one of the two repositories with the identical name from the content view, published and promoted and removed all old versions. After another prepare it looks like this:

# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: 
============Migration Summary================
Migrated/Total RPMs: 140656/140656
Migrated/Total errata: 53010/60999
Migrated/Total repositories: 1371/1371

Estimated migration time based on yum content: 2 hours, 17 minutes

Errata migration is still not complete and unchanged to before.

12

Thanks for checking that!

There are a couple possibilities, 1) pulp isn’t properly migrating all the errata, 2) katello isn’t properly pulling all the information in.

Lets check 2) first as thats easier. Can you run the migration again (but a little differently):

foreman-rake  console

and then run:

migration = Katello::Pulp3::Migration.new(SmartProxy.pulp_primary, :reimport_all => true)
migration.import_errata
quit

note that this may take a while. Then, re-check the migration-stats command

1 Like
13

O.K. A run the commands:

output of import_errata:

=> #<struct ActiveRecord::Import::Result failed_instances=[], num_inserts=1, ids=[467433,...,3715394], results=[]>

ids containing a total of 61038 ids, which is the number of total errata. Stats:

[root@foreman k202081]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: 
============Migration Summary================
Migrated/Total RPMs: 140676/140676
Migrated/Total errata: 61038/61038
Migrated/Total repositories: 1371/1371

Estimated migration time based on yum content: 2 hours, 17 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
                                                                      [OK]
--------------------------------------------------------------------------------

I’ll make another snapshot of the vm and then try the switchover again…

14

Out of curiosity, did you ever run a migration reset? if so, you might have been hitting: Bug #32210: Pulp3 Migration after migration reset doesn't import errata info from pulp3 - Katello - Foreman

which is fixed in 3.18.3 (hopefully releasing soon). But yes, you should be good to go for a switchover.

15

Yes. Multiple times. Usually followed by a deleted_orphaned_content. So, that bug may be the reason for these problems. Switchover is successful. Now waiting for the installer rerun to complete.

17

Ran into this now as well, Katello 3.18.5. RPMs and Repos were successful, but errata was incomplete and failed. I never did a reset. Running this ‘alternate’ method of importing the data resolved it. Curious as to what this does differently than the ‘regular’ import process. Thanks!