Content Migration fails with "414 Request-URI Too Long"

[Dirk]

Problem:
When running the content migration preparation it fails with “414 Request-URI Too Long” during “Importing migrated yum repositories”. The number of repositories show until it fails always differs, but this could be caused by a random order. In /var/log/httpd/foreman-ssl_access_ssl.log I see the request, but I have no idea how to map this to a specific repository.

172.22.87.232 - - [07/May/2021:13:01:05 +0200] "GET /pulp/api/v3/pulp2content/?pulp2_id__in=%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fcf6ae5f8-ba57-4207-864c-4a754740d57b%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F6ac7e541-0b8c-4656-8dc4-59624e243f0b%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F95a208f0-5cc6-4953-80f2-443509c610b9%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Ff764b20f-4a53-4966-9c19-a6fec56199e9%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F9f4d93bb-b1c8-4b30-a19d-5e6db961e0a1%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F4252030b-3d29-4bbb-a21d-fa72339c6cab%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fc3ec9996-f9ff-4516-bd99-13d7babdd516%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fa9aaf938-d0cf-408b-83f4-64ba2d3db262%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fc8fe5f20-9f6d-40a7-822f-ec913f202dd3%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F63abc706-6a88-4d0f-b110-e9775e46a33f%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F79e5a124-6242-4e16-b8dc-a87c04525ee8%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F6307b76f-f6d9-4721-8f6b-52e0f85db07e%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fa82cd5e6-6535-44fa-8b1a-53208eeb8b12%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F48e3d77c-e99f-4317-a23e-5a81e1bfa77e%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F332f0d5f-2ee7-4096-b49b-58076b8edf7e%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F5fddbeab-75ab-44a6-aac3-1b4d58a94e69%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F774910f8-a350-4f94-9dd0-b76dc2694d9b%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F09aeff08-5617-4bf4-94fc-d7835f7cd127%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fa321959e-aca2-49c9-a872-ef294cf80632%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F540f4784-3a3b-4711-aa23-a1ab623e2fea%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fb200e69d-e3af-4632-a420-a066b527a4fb%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F530e80d5-e8a9-4efe-b98f-9c7ff3b0fae9%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F231d2383-a900-4d91-a78c-0b14b06f18a0%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F9a77a833-16af-4b72-b32d-e69f6e6dfbee%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F9e7661bb-036e-4c4d-be0a-f055d99544b2%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F3b79f119-b147-4fc7-a9ad-8fed46f1a291%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F60e82114-b6e5-4493-ab70-4dab9346438c%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F50ecdd8b-14ba-4129-933e-43cd2685f870%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F7783d7ad-bf34-4454-aa95-a5f8bbeaa678%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F80a4b150-0c0a-48a8-b243-04ff9fef6b41%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F7e51e210-f070-493e-a9b7-09b828d807e0%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fb252e0bf-d9f7-4a70-87e4-17b70d0ebcdc%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F9896e044-45c6-44c7-87e5-e56b699f0d0e%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fdb5a4655-c963-4425-abad-fbe93b73474d%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fb8d76148-af3c-4a7e-86ff-e90e28d466e0%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fdd9e4b22-0e16-46f7-83b1-4f1437e8dfd8%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F9f043dcc-6f16-4ad0-ba36-10bfa1f217a3%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fd3fb69e7-1c11-44a8-8e3b-909337ab0e2b%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Ff71654cd-78eb-47dc-9424-0dd9800ea65b%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fa57238d4-f164-490e-8baf-49c948ffe0b6%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F1190db12-d3a0-442f-935f-8ecf55a345fd%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fb4204be4-a715-436a-8ee3-d0c7f06d2725%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Faeafe362-5b83-49cf-8e40-8e003a808800%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F09910051-29ad-4395-b321-8868323df9d9%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F2b774249-5c81-455a-90b4-511b33c336a8%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F804b9f03-f2cd-42c6-9715-901f724e5417%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fd8deadb8-fd80-48e0-a3c6-95d6c549443f%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Facc77725-c455-4bb1-b77a-0a625d1d7432%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F9ad73488-53ec-4008-8556-c989f09e93c8%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F707332f5-d839-48e2-ada6-c107806b886f%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F93b0fa63-01c3-4f19-a931-93a64653b21c%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fe3ab5dee-4e95-47ae-855d-62137f73a286%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Faab2fa1a-ae5e-4d6c-9feb-6bc2261672b8%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F535d9d2a-a7d0-4ec8-997f-14ea5ce8571c%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F1e3dc9d8-4525-496e-a7e2-13948902ddac%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F912f3add-0727-4ed5-9e6b-c5f69af1710b%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F9c0ef91b-9a40-440d-a4dc-894c25245ec1%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F171e9285-1b56-4192-a73f-2f74ebe55dba%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Febfb66fe-a2df-4402-b733-2c42a6cd9608%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fcd26ff4b-ea4a-4ec0-b5a9-aca8b1873dc7%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F13a3d854-00eb-4d13-ac87-b956b65702ac%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F2525edf8-1c0a-411b-9f83-4b7b73642276%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F4e97eb75-ca05-4519-b820-f5d6c8e6bfce%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F633829fb-5d85-43a4-9e37-1e6e9ad2d20b%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Ff4cf5987-79d8-44ab-9561-573c1351a849%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F7d2b927e-b8e5-4ff1-8f55-22239225447e%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F090030d6-8818-4a86-8cb9-a2b0e1168ffe%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F52359650-2718-4695-8f8f-55659c1811af%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F25a11f8e-dd5c-4bca-87fb-ec4d249931f9%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F3d33b1b5-7bbe-48c9-a04e-ad8d6c43bc19%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F708cfacc-f2dd-4bcf-a6cd-a3d7cc190d19%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F3d8261f2-4ec7-41d2-af2a-1d7e5724e9cd%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F2aee0fce-de6b-44a7-99cb-44ab7e268af2%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fcf07a7d0-92fe-4e65-a05a-3f7fb36cc6ff%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Ff654e1c2-af21-44c0-9fe5-7fe8a3d4bb59%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F6bcc4f60-22bd-49b8-ad54-faf570b778c0%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F059b4c71-23ac-456a-bc73-9171b7743c90%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F87923154-aa0f-4f68-b4f1-de4062cc84ed%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F2bf4fc35-b613-4b6d-8368-e5e774215ceb%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F89d3de8e-28d1-43fb-b798-187c6b4afffe%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F3cef13df-af20-4909-a883-3e5ecd3087dc%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fe98c33ea-c827-48ae-908c-9e4894739bb7%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fbc0f57bd-c2bf-4f80-b7ad-3d668bdbb68c%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2Fe861ad4f-31c2-4746-983c-cc80c1897ce9%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F5a089523-b459-4a30-a75d-f2d3895ec46d%2F%2C%2Fpulp%2Fapi%2Fv3%2Fcontent%2Frpm%2Fpackagegroups%2F06771a67-afd3-492d-" 414 248 "-" "-"

Migration stats show all data as migrated and because it is only a test setup I tried switchover but it fails with the same error message.

As a disclaimer: I am not sure what was done to migrate the data before as I was told “we already tried to migrate but something went wrong”, but we still want to migrate the test system successfully before trying in production.

I already trying to work around by increasing the limit on the apache side, but failed to get a higher limit applied.

Content varies with CentOS, SUSE and Community repositories, but as it complains about groups I looked into which repository provides many of it and it is CentOS 7 Base x86_64 with 88 groups.

Expected outcome:
Migration works.

Foreman and Proxy versions:
2.3.3

Foreman and Proxy plugin versions:
Katello 3.18.2.1

Distribution and version:
RHEL 7 (latest updates applied)

Other relevant data:
Only similar issue I found was Bug #29360: Too-large request sent to Pulp2to3MigrationClient during katello:pulp3_migration - Katello - Foreman, so perhaps @iballou can help?

[Dirk]

I tried to remove all package groups, dropped and re-created the pulpcore database and restart the prepare but it fails with the same error, so this information has to come from somewhere else than Katello or persist somewhere in Pulp2 from the failed earlier migration.

Can the migration somehow be completely undone for a fresh start?

[iballou]

Hey @Dirk,

The “pulp2” ids in the list there are Pulp 3 hrefs. This makes me think the switchover (foreman-maintain content switchover) might’ve been run at some point on your system. We’ve seen this error before when a system has already started using Pulp 3.

We can check if any content has been fully switched-over to Pulp 3 with the following command in the console:

::Katello::Rpm.where("pulp_id ilike '%/pulp/api/v3%'")

You can replace Rpm above with any other content type, like PackageGroup.

If your system does pop up with Pulp 3 indexed content, it would definitely be good to check back in your history to check for the switchover having been run before. If your machine really has started using Pulp 3, it gets pretty tricky to go back to Pulp 2 without doing a full reset.

@Dirk

Can the migration somehow be completely undone for a fresh start?

You can reset the migration with foreman-maintain content migration-reset. This only applies pre-switchover, however.