[ContentMigration] switchover fails at katello:pulp3_post_migration_check

Support
1

Problem:

After disabling deb content (I don’t have any deb content anyway and I don’t know why it was enabled) I was finally able to get a successful migration prepare run. However, the switchover fails:

# foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content
Starting task.
2021-04-11 08:48:58 +0200: Importing migrated yum repositories: 1061/1410
Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_post_migration_check, exit status 1:
 ERROR: yum repositories with ID [40286,40803,40105,40504,40673,30638,28594] have a NULL value for version_href
--------------------------------------------------------------------------------
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

Foreman and Proxy versions: 2.3.3

Foreman and Proxy plugin versions: Katello 3.18.2.1

Distribution and version: CentOS 7.9.2009

Other relevant data:
Checking the database those ids all are related to a single repository:

  id   |                             pulp_id                             |                            relative_path                             
-------+-----------------------------------------------------------------+----------------------------------------------------------------------
 40286 | 1-centos7-epel7-v226_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/226.0/custom/foreman/2_3_el7_x86_64
 40803 | 1-centos7-epel7-v230_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/230.0/custom/foreman/2_3_el7_x86_64
 40105 | 1-centos7-epel7-v225_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/225.0/custom/foreman/2_3_el7_x86_64
 40504 | 1-centos7-epel7-v228_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/228.0/custom/foreman/2_3_el7_x86_64
 40673 | 1-centos7-epel7-Library-9423c534-6d22-4763-9fd1-6afcac70a8bb    | ORG/Library/centos7-epel7/custom/foreman/2_3_el7_x86_64
 30638 | 1-centos7-epel7-Testing-9423c534-6d22-4763-9fd1-6afcac70a8bb    | ORG/Testing/centos7-epel7/custom/foreman/2_3_el7_x86_64
 28594 | 1-centos7-epel7-Production-9423c534-6d22-4763-9fd1-6afcac70a8bb | ORG/Production/centos7-epel7/custom/foreman/2_3_el7_x86_64
(7 rows)
2

Hey @gvde!

Can you run foreman-rake katello:correct_repositories COMMIT=true and assuming no errors, give the switchover another go?

3

I cannot run that:

[root@foreman ~]# foreman-rake katello:correct_repositories COMMIT=true
enabled
{:services=>
  {:candlepin=>{:status=>"ok", :duration_ms=>"15"},
   :candlepin_auth=>{:status=>"ok", :duration_ms=>"19"},
   :foreman_tasks=>
    {:status=>"FAIL",
     :message=>
      "The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer"},
   :katello_events=>
    {:status=>"ok", :message=>"3169 Processed, 0 Failed", :duration_ms=>"0"},
   :candlepin_events=>
    {:status=>"ok", :message=>"988 Processed, 0 Failed", :duration_ms=>"0"},
   :pulp3=>{:status=>"ok", :duration_ms=>"45"},
   :pulp=>{:status=>"ok", :duration_ms=>"30"},
   :pulp_auth=>{:status=>"ok", :duration_ms=>"15"}},
 :status=>"FAIL"}
rake aborted!
Not all the services have been started. Check the status report above and try again.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/reimport.rake:10:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_repositories => katello:check_ping
(See full trace by running task with --trace)
4

Thank you. That actually highlighted an issue that we need to pull back into the next 3.18 release!

You can fix that by making this 1-line change in reimport.rake => Fixes #31725 - properly use dynflow:client with check_ping in tasks by jlsherrill · Pull Request #9113 · Katello/katello · GitHub

5

O.K. Now it ran successfully, however the switchover still has the same issue. Still all related to the foreman 2.3 repository…

6

I don’t understand what’s so special about this one repository. Yesterday, I have published a new version of the content view. After that, all the new repository versions don’t have a “version_href” in the database. (to be expected, as it’s still pulp2). After another prepare run it’s back to only the foreman repository missing the version_href:

# sudo -u postgres psql foreman -c 'select id,pulp_id,relative_path from katello_repositories where version_href is null;'
  id   |                             pulp_id                             |                            relative_path                             
-------+-----------------------------------------------------------------+----------------------------------------------------------------------
 41552 | 1-centos7-epel7-v235_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/235.0/custom/foreman/2_3_el7_x86_64
 41307 | 1-centos7-epel7-v234_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/234.0/custom/foreman/2_3_el7_x86_64
 41055 | 1-centos7-epel7-v232_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/232.0/custom/foreman/2_3_el7_x86_64
 40871 | 1-centos7-epel7-v231_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/231.0/custom/foreman/2_3_el7_x86_64
 40673 | 1-centos7-epel7-Library-9423c534-6d22-4763-9fd1-6afcac70a8bb    | ORG/Library/centos7-epel7/custom/foreman/2_3_el7_x86_64
 28594 | 1-centos7-epel7-Production-9423c534-6d22-4763-9fd1-6afcac70a8bb | ORG/Production/centos7-epel7/custom/foreman/2_3_el7_x86_64
 41239 | 1-centos7-epel7-v233_0-9423c534-6d22-4763-9fd1-6afcac70a8bb     | ORG/content_views/centos7-epel7/233.0/custom/foreman/2_3_el7_x86_64
 30638 | 1-centos7-epel7-Testing-9423c534-6d22-4763-9fd1-6afcac70a8bb    | ORG/Testing/centos7-epel7/custom/foreman/2_3_el7_x86_64
(8 rows)

I see no errors in production.log. If I grep for a id number I’ll find this for repositories which worked:

# grep -we 4155[123] production.log
2021-04-20T06:37:21 [I|aud|] Katello::Repository (41551) update event on publication_href , /pulp/api/v3/publications/rpm/rpm/39a5066f-7464-4fe6-8933-59d1721b4472/
2021-04-20T06:37:21 [I|aud|] Katello::Repository (41551) update event on version_href , /pulp/api/v3/repositories/rpm/rpm/5ca0a207-b946-450e-92ba-7fa8eb6e2fda/versions/1/
2021-04-20T06:37:22 [I|aud|] Katello::Repository (41553) update event on publication_href , /pulp/api/v3/publications/rpm/rpm/b7ef60ee-b0f5-48dc-8b8c-2a9394b18289/
2021-04-20T06:37:22 [I|aud|] Katello::Repository (41553) update event on version_href , /pulp/api/v3/repositories/rpm/rpm/78dabef7-8a36-4e73-8167-db0ebb67ae6d/versions/1/

But as you can see, the 41552 which is the number for the foreman repository is missing. I don’t see what is so special about the foreman 2.3 repository in my installation. I already have foreman 2.4 there, too, and that’s fine.

So no errors. It just doesn’t work. I don’t know where else to look. I guess I could remove the repository from the content view, etc. and delete it and set it up again, but I assume others could have similar issues with other repositories, too, and that could be a deal-breaker for the coming katello 4.0 upgrade.

7

As Katello 4.0 has been released I have tried to work around the issue: I have removed all downloaded packages from the foreman 2.3 repository, I have removed the repository from the content view “centos7-epel7”. Published a new version, promoted it to my lifecycle environments Testing and Production, remove all old version still containing the foreman 2.3 repository.

After that, I was able to do a prepare and switchover run. Everything successful. Now I am running on pulp3.

I have synced the repository to get the RPMs back in. I have added it to the content view again and published and promoted the content view.

However, the clients cannot access the repository. Any yum command gets this error:

https://foreman.example.com/pulp/repos/ORG/Production/centos7-epel7/custom/foreman/2_3_el7_x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below wiki article 

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

When I check the URL ‘http://foreman.example.com/pulp/repos/ORG/Production/centos7-epel7/custom/foreman/’ with curl I can see that the directory only contains ‘plugins_2_3_el7_x86_64’ (which is correct) but not ‘2_3_el7_x86_64’. Same is true for Testing.

If I check the Library directory ‘http://foreman.example.com/pulp/repos/ORG/Library/custom/foreman/’ it’s in there.

If I check through the content view path ‘http://foreman.example.com/pulp/repos/ORG/content_views/centos7-epel7/237.0/custom/foreman/’ it’s missing.

So it’s indeed missing. But only in the the content view. The library seems fine.

If I check in the web frontend, the content view does correctly contain the repository and if I check the latest published and promoted version I can see that the repository and the RPMs are included. Thus, from what I see there is should be in there. There must be still something broken, somewhere, which prevents the new repository to be added to the content view.

8

Well, actually the complete content view version is missing. I only see version 237.0 in http://foreman.example.com/pulp/repos/ORG/content_views/centos7-epel7/ but the latest version published and promoted is 238.0, i.e. the complete content view is missing.

9

And it gets worse: I published a new version in another CV ‘centos7’ and that also won’t appear in the content_view URL http://foreman.example.com/pulp/repos/ORG/content_views/centos7/ nor in the file system at /var/lib/pulp/published/yum/http/repos/ORG/content_views/centos7

It seems as if any new published CV won’t appear in the repositories even though katello shows it.

10

To complete this: basically the directory tree /var/lib/pulp/published/yum/http/ doesn’t change anymore. It is stuck at the point right before migration.

And if I am not mistaking, also for /var/lib/pulp/published/yum/master/yum_distributor

Looks like it doesn’t work anymore. At all…

11

After I had too many issues with my upgraded katello server, I have reverted back to a snapshot with 3.18. After fighting the .treeinfo error which prevented the prepare to complete, which is now finally working, I am actually back to this error:

Last switchover showed it again:

# foreman-maintain content switchover
Running Switch support for certain content from Pulp 2 to Pulp 3
================================================================================
Switch support for certain content from Pulp 2 to Pulp 3: 
Performing final content migration before switching content
Starting task.
2021-05-25 20:11:37 +0200: Importing migrated yum repositories: 1101/1179                
Content Migration completed successfully
Performing a check to verify everything that is needed has been migrated
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_post_migration_check, exit status 1:
 ERROR: yum repositories with ID [42880,42948,42418,42600,42238,30634,28592] have a NULL value for version_href
--------------------------------------------------------------------------------
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.

However, funny enough, now it’s not the foreman 2.3 repository but the foreman 2.3 client repository:

# sudo -u postgres psql foreman -c 'select id,pulp_id,relative_path from katello_repositories where version_href is null;'
  id   |                             pulp_id                             |                                relative_path                                
-------+-----------------------------------------------------------------+-----------------------------------------------------------------------------
 42880 | 1-centos7-epel7-v244_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/244.0/custom/foreman-client/2_3_el7_x86_64
 42948 | 1-centos7-epel7-Library-b927ce80-c073-4911-8be6-58da9a55749d    | ORG/Library/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 42418 | 1-centos7-epel7-v241_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/241.0/custom/foreman-client/2_3_el7_x86_64
 42600 | 1-centos7-epel7-v242_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/242.0/custom/foreman-client/2_3_el7_x86_64
 42238 | 1-centos7-epel7-v240_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/240.0/custom/foreman-client/2_3_el7_x86_64
 30634 | 1-centos7-epel7-Testing-b927ce80-c073-4911-8be6-58da9a55749d    | ORG/Testing/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 28592 | 1-centos7-epel7-Production-b927ce80-c073-4911-8be6-58da9a55749d | ORG/Production/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
(7 rows)

I know, I could clear this out as I don’t really need it anymore when I update to 2.4/4.0. But considering the number of problems I had after I have upgraded to 4.0 I would rather see a clean, successful switchover working than one with some workaround and hacks just to get it upgraded and starting…

12

@gvde i’m gonna dig into this more tomorrow morning and get back to you, there is quite a bit of history on this one :slight_smile:

13

@Justin_Sherrill My messages from Apr 23/24 above (no. 6-9) are obviously obsolete, as I have reverted the VM.

So the status is just like this: the switchover fails. All repositories entries in the katello_repositories table which have no version_href are for the repository foreman-client 2.3 el7 (foreman-client/2_3_el7_x86_64) in one specific CV centos7-epel7. It’s all version numbers of that CV as well as the promoted views to lifecycle environments Library, Testing and Production.

I catch up with my previous attempt: I just had to publish a new CV version. And then ran another prepare successfully. Next I ran

# foreman-rake katello:correct_repositories COMMIT=true
enabled
Processing Repository 1/1234: CentOS-7 - Base (1)
...
Processing Repository 261/1234: Foreman client 2.3 el7 (28393)
Processing Repository 267/1234: Foreman client 2.3 el7 (28591)
Processing Repository 268/1234: Foreman client 2.3 el7 (28592)
Processing Repository 293/1234: Foreman client 2.3 el7 (30574)
Processing Repository 348/1234: Foreman client 2.3 el7 (30634)
Processing Repository 555/1234: Foreman client 2.3 el7 (40157)
Processing Repository 612/1234: Foreman client 2.3 el7 (42296)
Processing Repository 675/1234: Foreman client 2.3 el7 (42418)
Processing Repository 733/1234: Foreman client 2.3 el7 (42476)
Processing Repository 798/1234: Foreman client 2.3 el7 (42600)
Processing Repository 859/1234: Foreman client 2.3 el7 (42662)
Processing Repository 948/1234: Foreman client 2.3 el7 (42880)
Processing Repository 1016/1234: Foreman client 2.3 el7 (42948)
Processing Repository 1076/1234: Foreman client 2.3 el7 (43008)
Processing Repository 1190/1234: Foreman client 2.3 el7 (43122)
...

I have grepped the foreman client 2.3 el7 messages in the output above.

Some of those don’t have version_href:

# sudo -u postgres psql foreman -c 'select id,pulp_id,relative_path from katello_repositories where version_href is null;'
  id   |                             pulp_id                             |                                relative_path
-------+-----------------------------------------------------------------+-----------------------------------------------------------------------------
 42880 | 1-centos7-epel7-v244_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/244.0/custom/foreman-client/2_3_el7_x86_64
 42948 | 1-centos7-epel7-Library-b927ce80-c073-4911-8be6-58da9a55749d    | ORG/Library/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 42418 | 1-centos7-epel7-v241_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/241.0/custom/foreman-client/2_3_el7_x86_64
 42600 | 1-centos7-epel7-v242_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/242.0/custom/foreman-client/2_3_el7_x86_64
 28592 | 1-centos7-epel7-Production-b927ce80-c073-4911-8be6-58da9a55749d | ORG/Production/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 30634 | 1-centos7-epel7-Testing-b927ce80-c073-4911-8be6-58da9a55749d    | ORG/Testing/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 43008 | 1-centos7-epel7-v245_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/245.0/custom/foreman-client/2_3_el7_x86_64
(7 rows)

So for whatever reason this one repository in this CV has a problem at any version and at any promotion.

14

Maybe I have found something. I looked at the migration plan which is logged into production.log. I have been looking for the pulp_id above and noticed that there are two repositories with the same name in the plan: “centos7-epel7-2_3_el7_x86_64”.

That’s actually correct. I have used the same repository label for my foreman 2.3 and foreman 2.3 client repository.

[root@foreman ~]# hammer repository list --label 2_3_el7_x86_64
------|------------------------|----------------|--------------|----------------------------------------------------
ID    | NAME                   | PRODUCT        | CONTENT TYPE | URL                                                
------|------------------------|----------------|--------------|----------------------------------------------------
28395 | Foreman 2.3            | Foreman        | yum          | https://yum.theforeman.org/releases/2.3/el7/x86_64/
28393 | Foreman client 2.3 el7 | Foreman Client | yum          | https://yum.theforeman.org/client/2.3/el7/x86_64/  
------|------------------------|----------------|--------------|----------------------------------------------------

It would make sense that either the foreman-client or foreman repository would get an version_href then. The foreman repository is only in my centos7-epel7 CV, i.e. only in that CV both repositories are added.

I haven’t added 2.4 to any CV yet, thus it doesn’t cause issues.

To verify, I’ll add the 2.4 repositories to the centos7-epel7 and then go publish & prepare it again. If I am right, the 2.4 repository should appear in the list with the missing version_href… I’ll report back.

15

O.K. The prepare went through. There are new repositories with missing version_href:

[root@foreman ~]# sudo -u postgres psql foreman -c 'select id,pulp_id,relative_path from katello_repositories where version_href is null;'
  id   |                             pulp_id                             |                                relative_path                                
-------+-----------------------------------------------------------------+-----------------------------------------------------------------------------
 42880 | 1-centos7-epel7-v244_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/244.0/custom/foreman-client/2_3_el7_x86_64
 42948 | 1-centos7-epel7-Library-b927ce80-c073-4911-8be6-58da9a55749d    | ORG/Library/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 42418 | 1-centos7-epel7-v241_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/241.0/custom/foreman-client/2_3_el7_x86_64
 42600 | 1-centos7-epel7-v242_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/242.0/custom/foreman-client/2_3_el7_x86_64
 43173 | 1-centos7-epel7-v246_0-7e2d627c-aaa5-4689-aeff-f34cb7163a19     | ORG/content_views/centos7-epel7/246.0/custom/foreman/2_4_el7_x86_64
 43186 | 1-centos7-epel7-v246_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/246.0/custom/foreman-client/2_3_el7_x86_64
 28592 | 1-centos7-epel7-Production-b927ce80-c073-4911-8be6-58da9a55749d | ORG/Production/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 30634 | 1-centos7-epel7-Testing-b927ce80-c073-4911-8be6-58da9a55749d    | ORG/Testing/centos7-epel7/custom/foreman-client/2_3_el7_x86_64
 43237 | 1-centos7-epel7-Library-7e2d627c-aaa5-4689-aeff-f34cb7163a19    | ORG/Library/centos7-epel7/custom/foreman/2_4_el7_x86_64
 43008 | 1-centos7-epel7-v245_0-b927ce80-c073-4911-8be6-58da9a55749d     | ORG/content_views/centos7-epel7/245.0/custom/foreman-client/2_3_el7_x86_64
(10 rows)

As expected the “foreman/2_4_el7_x86_64” has the mission version_href as there is also a “foreman-client/2_4_el7_x86_64”.

The migration plan contains the following repositories entries:

                {
                    "name": "centos7-epel7-2_4_el7_x86_64",
                    "repository_versions": [
                        {
                            "pulp2_distributor_repository_ids": [
                                "1-centos7-epel7-Library-7e2d627c-aaa5-4689-aeff-f34cb7163a19",
                                "1-centos7-epel7-v246_0-7e2d627c-aaa5-4689-aeff-f34cb7163a19"
                            ],
                            "pulp2_repository_id": "1-centos7-epel7-v246_0-7e2d627c-aaa5-4689-aeff-f34cb7163a19"
                        }
                    ]
                },
                {
                    "name": "centos7-epel7-2_4_el7_x86_64",
                    "repository_versions": [
                        {
                            "pulp2_distributor_repository_ids": [
                                "1-centos7-epel7-Library-84701efc-c3fc-4377-8c9c-29da70125868",
                                "1-centos7-epel7-v246_0-84701efc-c3fc-4377-8c9c-29da70125868"
                            ],
                            "pulp2_repository_id": "1-centos7-epel7-v246_0-84701efc-c3fc-4377-8c9c-29da70125868"
                        }
                    ]
                },

The 7e2d627c-aaa5-4689-aeff-f34cb7163a19 is the foreman 2.4 repo and the 84701efc-c3fc-4377-8c9c-29da70125868 is the foreman-client 2.4 repo. I guess, because of the order in the migration plan, the latter one will be migrated while the former one will be missing, which makes sense: when it creates objects out of this json it will create an object with the name “centos7-epel7-2_4_el7_x86_64”, the latter overwriting the former one.

Thus, having two (or more) repositories with the identical label in different products in a CV causes issues in the migration as the name set in the migration plan does not include the product.

The repositories itself are not affected as those are listed in the plan by their backend identifier, which is unique.

Now, is this a katello issue or a pulp issue?

16

After going through the code it seems it’s a katello issue. The name for the migration plan is set here: katello/migration_plan.rb at 3.18.2.1 · Katello/katello · GitHub

If have created a new bug: Bug #32663: Pulp3 migration switchover fails when a content view contains two repositories with the same label - Katello - Foreman

17

Thanks for the investigation! I agree, let me see if i can come up with a simple reproducer and i’ll see if i can come up with a patch

18

This problem raises the question whether there are more issues to be expected when two or more repositories use the same label. Or is this the only place where this happens?

In other words: should repository labels be globally unique (and thus Katello should enforce that) or is it safe and not causing any (more) problems?

19

Label does not have to be unique across products, only within a product. The migration is just not handling this properly. I’m working on a patch that will attempt to detect these duplicates and ensure that the name is unqiue. I’m not going to change the name for all repositories as it would potentially cause longer re-migrations for users that have already migrated, so this will only affect the repos with this condition.

20

I opened a pr here: Fixes #32663 - ensure unique name for each migrated repo by jlsherrill · Pull Request #9385 · Katello/katello · GitHub

its lightly tested and appears to resolve the issue, i’ll work on testing it more, but you may want to give it a try if you are feeling adventurous :slight_smile: