Copied my “el8” content view, published: error.
Removed all filters from the copy, published: OK.
Added one (first) include filter “etckeeper” for RPM etckeeper* on the EPEL8 repository, published: error.
So basically, once I add a filter, I’ll get an error. As I haven’t changed anything on my server for almost 2 weeks, I guess it must have something to do with the epel7 and epel8 metadata causing cv filters to break?
Another observation, although not fully verified yet: on my new server on AlmaLinux 8.6 with
foreman-3.3.0-1.el8.noarch
katello-4.5.0-1.el8.noarch
python39-pulp-rpm-3.17.7-3.el8.noarch
python39-pulpcore-3.18.5-2.el8.noarch
It doesn’t seem to happen. Old and new servers both sync EPEL around the same time, so if there is an issue with the metadata, it should be on both. On the new server I can publish the same content view with the same content and same filters without error. Of course, on the new server I didn’t have the remove orphan issue either…
How do I clean up this mess? Those two CVs become unusable. I cannot update them anymore. Is this a @katello issue or a pulpcore issue?
I dug into the databases and found that katello_package_groups has a total of 14 references in pulp_id which do not exist in the pulpcore database…
Hi @gvde,
My only guess is that, during some sync, these content units were remove from Pulp and Katello did not delete them. The history to check this is likely gone since orphan cleanup removes old repository versions.
Are the offending content units only package groups? It’ll be good to know which units are affected to search for a Katello indexing issue.
It makes sense that you can publish CVs without filters but filtering fails, since we only query for individual units during filtered CV publishing. When there are no filters, we only copy over the repository version.
Here’s what I would do:
Let me know if you want help with making console commands for any of those steps above.
I created an issue to check Katello content indexing: Bug #35457: Investigation: does Katello properly remove content units when they are removed from a Pulp repository during syncing? - Katello - Foreman
Hi @iballou,
I see only three different URIs with 404 during syncs which are the three I have posted in my second post.
That should be those three one. The others I have found via database search don’t seem cause issues so far. I am not sure what you want to delete exactly. The broken package groups are also used in the non-filtered content view repositories in the database. Thus, deleting the package group from the database would remove it from all content views as well?
I tried that in hope it would fix the situation, but it didn’t. But of course, I didn’t delete anything from the database, yet.
Yes, that would be good. I would simply delete them in the database but all those references to all published CV versions scared me off a little bit.
Deleting the broken package groups from Katello will completely remove them from all repositories and content views. I thought perhaps they were duplicated, but they aren’t. Since the package groups are broken anyway, that just means they’ll be gone from the UI.
It would be worth double checking that the package groups indeed don’t exist anymore in the upstream repository. If they do, we’ll want to make sure that they are making it into Pulp and that Katello is indexing them properly after deleting the broken ones.
Here are some commands that should help:
# Stick all of your package group IDs in an array
package_group_ids = [<ids>]
repository_ids = []
::Katello::PackageGroup.find(package_group_ids).each do |package_group|
repository_ids << package_group.repositories.pluck(:id)
package_group.destroy
end
repository_ids = repository_ids.uniq.flatten.compact
# Indexing content is optional (might take a few minutes), but it may bring back the correct package groups if they did change "pulp_hrefs"
::Katello::Repository.find(repository_ids).each { |r| r.index_content }
Not sure if it’s related at all but we did fix an issue with additive sync policy on repo not deleting packages in katello when it does so in pulp?
Well, as far as I can see:
foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path from katello_package_groups a left join katello_repository_package_groups b on a.id = b.package_group_id left join katello_repositories c on b.repository_id = c.id where (a.pulp_id like '%9a2f7d22%' or a.pulp_id like '%a18de06a%' or a.pulp_id like '%cdc38f34%') and relative_path like '%/Library/%' order by relative_path;
id | name | pulp_id | pulp_id | relative_path
------+------+------------------------------------------------------------------------------+--------------------------------------------------------------+-----------------------------------------------------
754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/custom/epel8/epel
1174 | KDE | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/custom/epel8/epel
743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | ORG/Library/custom/epel8/next
1174 | KDE | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | ORG/Library/custom/epel8/next
1174 | KDE | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 1-el8-epel8-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/el8-epel8/custom/epel8/epel
743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 1-el8-epel8-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/el8-epel8/custom/epel8/epel
743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 1-el8-epel8-Library-bf598f28-21a5-4833-be65-4b63faf9aa30 | ORG/Library/el8-epel8/custom/epel8/next
1174 | KDE | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 1-el8-epel8-Library-bf598f28-21a5-4833-be65-4b63faf9aa30 | ORG/Library/el8-epel8/custom/epel8/next
743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 1-el8-test-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/el8-test/custom/epel8/epel
1174 | KDE | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 1-el8-test-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/el8-test/custom/epel8/epel
(12 rows)
It is group Xfce for EPEL 7 and EPEL 8 and KDE for EPEL 8. Looking at the repository package group metadata for EPEL 7 and EPEL 8 a group with name “Xfce” exists in both EPEL 7 and EPEL 8, and “KDE” exists in EPEL 8. So those group still exist in upstream.
If I search the database for package groups of those names, I’ll find:
foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path from katello_package_groups a left join katello_repository_package_groups b on a.id = b.package_group_id left join katello_repositories c on b.repository_id = c.id where (a.name in ( 'Xfce', 'KDE' ) ) and relative_path like '%/Library/custom/%' order by name, c.pulp_id;
id | name | pulp_id | pulp_id | relative_path
------+------+------------------------------------------------------------------------------+--------------------------------------+-----------------------------------------
931 | KDE | /pulp/api/v3/content/rpm/packagegroups/43f79897-76df-411c-82ab-739356421d9a/ | 70cf2996-6320-4fcd-a9c5-eda568957879 | DKRZ/Library/custom/centos7/base_x86_64
3747 | KDE | /pulp/api/v3/content/rpm/packagegroups/a9f014e5-2d09-48bf-864a-b574489dbd8d/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
1174 | KDE | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
1174 | KDE | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
3747 | KDE | /pulp/api/v3/content/rpm/packagegroups/a9f014e5-2d09-48bf-864a-b574489dbd8d/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
3746 | Xfce | /pulp/api/v3/content/rpm/packagegroups/2be268c5-ff45-4ff8-96ac-17af587a9cd8/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
3746 | Xfce | /pulp/api/v3/content/rpm/packagegroups/2be268c5-ff45-4ff8-96ac-17af587a9cd8/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
3767 | Xfce | /pulp/api/v3/content/rpm/packagegroups/178d7527-4ef2-46ef-8ef4-8cefe4c9d17c/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | DKRZ/Library/custom/epel7/epel_x86_64
754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | DKRZ/Library/custom/epel7/epel_x86_64
(11 rows)
So it seems, for KDE 9a2f7d22-d2e1-45d8-98c3-60130882fc2b is the broken entry and a9f014e5-2d09-48bf-864a-b574489dbd8d is the correct entry. For Xfce there are also correct entries for each broken one.
Those package groups are duplicated in the katello database, the old ones with lower id numbers are the broken ones…
Also interesting: the newer correct entries are a month old:
foreman=# select id, name, pulp_id, created_at, updated_at from katello_package_groups where id in ( 3746, 3747, 3767 );
id | name | pulp_id | created_at | updated_at
------+------+------------------------------------------------------------------------------+----------------------------+----------------------------
3746 | Xfce | /pulp/api/v3/content/rpm/packagegroups/2be268c5-ff45-4ff8-96ac-17af587a9cd8/ | 2022-07-30 12:45:26.614919 | 2022-07-30 12:45:26.614923
3747 | KDE | /pulp/api/v3/content/rpm/packagegroups/a9f014e5-2d09-48bf-864a-b574489dbd8d/ | 2022-07-30 12:45:26.614927 | 2022-07-30 12:45:26.614931
3767 | Xfce | /pulp/api/v3/content/rpm/packagegroups/178d7527-4ef2-46ef-8ef4-8cefe4c9d17c/ | 2022-07-30 12:48:56.819516 | 2022-07-30 12:48:56.819522
(3 rows)
So I wonder, why the issue only appears now. I’ll try cleaning out later and get back…
All EPEL repos are set to additive on my server. My problems are related to package groups, though.
O.K. Those commands on the three package group ids helped somewhat. Of course, the next publish blew up on a different package group.
From the previous total of 14 packages groups without entries in the pulpcore database 11 have been left:
pulpcore=# select pulp_id,content_ptr_id from public.dblink('dbname=foreman', 'select substring(pulp_id from 40 for 36 ) from katello_package_groups') as katello_package_groups(pulp_id uuid) full outer join rpm_packagegroup on katello_package_groups.pulp_id = rpm_packagegroup.content_ptr_id where rpm_packagegroup.content_ptr_id is null;
pulp_id | content_ptr_id
--------------------------------------+----------------
fa344de9-636e-440a-9aef-7f961330cd22 |
1ca37d61-4ba1-43cb-bf65-27705bdaf592 |
190562b0-35d5-47fc-993d-834995743951 |
47f9919a-59fd-4e2a-b65e-dbc3b8fb8606 |
b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8 |
229af386-884a-41b2-af76-775f2b465ba8 |
b7cfb5f4-d603-4336-8ef0-968a157cbea1 |
a55bc85d-d557-43ad-8f3c-0801d1684b88 |
d50e7889-64ab-40a7-afe1-1b202829c00e |
c3e4825f-fe44-4dcd-a8cd-583d295fd50b |
3c91eebf-9b5f-44b5-9d02-d58559821e45 |
(11 rows)
from a variety of package groups, but all in epel 7 or 8
foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path from katello_package_groups a left join katello_repository_package_groups b on a.id = b.package_group_id left join katello_repositories c on b.repository_id = c.id where ( a.pulp_id in ( '/pulp/api/v3/content/rpm/packagegroups/fa344de9-636e-440a-9aef-7f961330cd22/', '/pulp/api/v3/content/rpm/packagegroups/1ca37d61-4ba1-43cb-bf65-27705bdaf592/', '/pulp/api/v3/content/rpm/packagegroups/190562b0-35d5-47fc-993d-834995743951/', '/pulp/api/v3/content/rpm/packagegroups/47f9919a-59fd-4e2a-b65e-dbc3b8fb8606/', '/pulp/api/v3/content/rpm/packagegroups/b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8/', '/pulp/api/v3/content/rpm/packagegroups/229af386-884a-41b2-af76-775f2b465ba8/', '/pulp/api/v3/content/rpm/packagegroups/b7cfb5f4-d603-4336-8ef0-968a157cbea1/', '/pulp/api/v3/content/rpm/packagegroups/a55bc85d-d557-43ad-8f3c-0801d1684b88/', '/pulp/api/v3/content/rpm/packagegroups/d50e7889-64ab-40a7-afe1-1b202829c00e/', '/pulp/api/v3/content/rpm/packagegroups/c3e4825f-fe44-4dcd-a8cd-583d295fd50b/', '/pulp/api/v3/content/rpm/packagegroups/3c91eebf-9b5f-44b5-9d02-d58559821e45/' ) )and relative_path like '%/Library/%' order by relative_path;
id | name | pulp_id | pulp_id | relative_path
------+---------------------------------------+------------------------------------------------------------------------------+--------------------------------------------------------------+-----------------------------------------------------
782 | Development tools | /pulp/api/v3/content/rpm/packagegroups/a55bc85d-d557-43ad-8f3c-0801d1684b88/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
784 | Cinnamon | /pulp/api/v3/content/rpm/packagegroups/d50e7889-64ab-40a7-afe1-1b202829c00e/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
766 | Milkymist | /pulp/api/v3/content/rpm/packagegroups/190562b0-35d5-47fc-993d-834995743951/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
773 | Haskell | /pulp/api/v3/content/rpm/packagegroups/229af386-884a-41b2-af76-775f2b465ba8/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
767 | MATE | /pulp/api/v3/content/rpm/packagegroups/47f9919a-59fd-4e2a-b65e-dbc3b8fb8606/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
779 | Fedora Packager | /pulp/api/v3/content/rpm/packagegroups/b7cfb5f4-d603-4336-8ef0-968a157cbea1/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
769 | LXQt Desktop | /pulp/api/v3/content/rpm/packagegroups/b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
779 | Fedora Packager | /pulp/api/v3/content/rpm/packagegroups/b7cfb5f4-d603-4336-8ef0-968a157cbea1/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
784 | Cinnamon | /pulp/api/v3/content/rpm/packagegroups/d50e7889-64ab-40a7-afe1-1b202829c00e/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
782 | Development tools | /pulp/api/v3/content/rpm/packagegroups/a55bc85d-d557-43ad-8f3c-0801d1684b88/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
773 | Haskell | /pulp/api/v3/content/rpm/packagegroups/229af386-884a-41b2-af76-775f2b465ba8/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
766 | Milkymist | /pulp/api/v3/content/rpm/packagegroups/190562b0-35d5-47fc-993d-834995743951/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
767 | MATE | /pulp/api/v3/content/rpm/packagegroups/47f9919a-59fd-4e2a-b65e-dbc3b8fb8606/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
769 | LXQt Desktop | /pulp/api/v3/content/rpm/packagegroups/b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
744 | KDE Frameworks 5 Software Development | /pulp/api/v3/content/rpm/packagegroups/fa344de9-636e-440a-9aef-7f961330cd22/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/custom/epel8/epel
752 | Fedora Packager | /pulp/api/v3/content/rpm/packagegroups/1ca37d61-4ba1-43cb-bf65-27705bdaf592/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/custom/epel8/epel
1163 | base-x | /pulp/api/v3/content/rpm/packagegroups/c3e4825f-fe44-4dcd-a8cd-583d295fd50b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/custom/epel8/epel
1169 | KDE Software Development | /pulp/api/v3/content/rpm/packagegroups/3c91eebf-9b5f-44b5-9d02-d58559821e45/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | ORG/Library/custom/epel8/epel
752 | Fedora Packager | /pulp/api/v3/content/rpm/packagegroups/1ca37d61-4ba1-43cb-bf65-27705bdaf592/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | ORG/Library/custom/epel8/next
744 | KDE Frameworks 5 Software Development | /pulp/api/v3/content/rpm/packagegroups/fa344de9-636e-440a-9aef-7f961330cd22/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | ORG/Library/custom/epel8/next
1169 | KDE Software Development | /pulp/api/v3/content/rpm/packagegroups/3c91eebf-9b5f-44b5-9d02-d58559821e45/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | ORG/Library/custom/epel8/next
So I took those 11 additional package group ids and fed them into the commands. Now they are all gone. Next, I did a complete sync on all three affected repositories (epel7, epel8, epel8-next).
Now, publishing and promoting of my filtered content views are working again!
Glad to hear it worked! We’ll try to see if we can reproduce the issue ourselves and hopefully find a root cause.
I’ll hope you figure this out. I find it rather weird. For whatever reason, it added those duplicate package groups on July 30th. I can’t see why. There were no updates whatsoever before that for more than a week.
I also don’t understand why the issue just pops up now, more than 4 weeks after the duplicates have been created. There have been some updates including pulpcore updates in that time, but nothing after August 17th.
So it puzzles me what triggered this. I guess most likely it came from the epel metadata upstream.
If the duplicate package groups were added on July 30th, are there any corresponding repository sync tasks or content view publish tasks?
The timestamps for those packages groups match the UTC time of the sync plan for the epel repositories. Tasks older than a month are removed from the history, thus I cannot check for the task itself anymore…
The publish/promote cron job runs in the morning. That is unrelated.
It happened again. This morning the publish run for my filtered view “CentOS 7” went into pause state with the same error again:
PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 404
Response headers: {"date"=>"Mon, 12 Sep 2022 02:05:55 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, HEAD, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"23", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"d9233407-5d94-4cc8-ad1b-96467b5baf52", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.dkrz.de", "connection"=>"close"}
Response body: {"detail":"Not found."}
The requested URL is a package group again:
[12/Sep/2022:04:05:55 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 404 23 "-" "OpenAPI-Generator/3.17.4/ruby"
Affected group is “Development tools” which was one of the broken ones before:
foreman=# select * from katello_package_groups where pulp_id like '%e69069bf%';
id | name | pulp_id | description | created_at | updated_at | migrated_pulp3_href | missing_from_migr
ation | ignore_missing_from_migration
------+-------------------+------------------------------------------------------------------------------+----------------------------------+----------------------------+----------------------------+---------------------+------------------
------+-------------------------------
3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | A basic development environment. | 2022-07-30 12:48:56.819566 | 2022-07-30 12:48:56.819571 | | f
| f
(1 row)
At this times it seems this is the only package group affected:
pulpcore=# select pulp_id,content_ptr_id from public.dblink('dbname=foreman', 'select substring(pulp_id from 40 for 36 ) from katello_package_groups') as katello_package_groups(pulp_id uuid) full outer join rpm_packagegroup on katello_package_groups.pulp_id = rpm_packagegroup.content_ptr_id where rpm_packagegroup.content_ptr_id is null;
pulp_id | content_ptr_id
--------------------------------------+----------------
e69069bf-6755-4977-9221-1b7a395e4890 |
(1 row)
Affected repo is epel7 again, which is the only repository with filters in this CV:
foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path
foreman-# from katello_package_groups a
foreman-# left join katello_repository_package_groups b on a.id = b.package_group_id
foreman-# left join katello_repositories c on b.repository_id = c.id
foreman-# where ( a.pulp_id in ( '/pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/' ) )
foreman-# and relative_path like '%/Library/%' order by relative_path;
id | name | pulp_id | pulp_id | relative_path
------+-------------------+------------------------------------------------------------------------------+--------------------------------------------------------------+-----------------------------------------------------
3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/custom/epel7/epel_x86_64
(2 rows)
Checking the tasks running: there is no other task with any error or warning for a couple of days now. The publish is called from a shell script using hammer. It works on all my four content views. First it promotes the version in lifecycle environment Testing to Production. After that it publishes a new version, promoted to Testing. After it runs a purge. This is the script running on Monday through Friday mornings.
#! /bin/bash
echo -n "Publishing and promoting... "
date
for i in 23 11 22 12
do
hammer content-view info --id $i --fields THIN
hammer content-view version promote --content-view-id=$i --organization ORG --from-lifecycle-environment Testing --to-lifecycle-environment Production
done
sleep 300
#for i in 4 5 11 12 17 18 19 20
for i in 23 11 22 12
do
hammer content-view info --id $i --fields THIN
time hammer --output silent content-view publish --id $i --lifecycle-environments Testing --organization ORG
hammer --output silent content-view purge --id $i
done
date
ID 11 is the “CentOS 7” content view in question. ID 23 CV is my “EL 8” content view. ID 11 and ID 23 only have a very limited overlap in repositories.
Checking other tasks overlapping with the publish task (just listing in case it might be related):
Ended at: 2022-09-12 02:04:54 UTC
Ended at: 2022-09-12 02:04:37 UTC
Started at: 2022-09-12 02:04:43 UTC
So I don’t think it’s an issue with another task running at the same time.
Creation date of the new package group is August 30th.
foreman=# select * from katello_package_groups where name = 'Development tools';
id | name | pulp_id | description | created_at | updated_at | migrated_pulp3_href | missing_from_migr
ation | ignore_missing_from_migration
------+-------------------+------------------------------------------------------------------------------+----------------------------------+----------------------------+----------------------------+---------------------+------------------
------+-------------------------------
3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | A basic development environment. | 2022-07-30 12:48:56.819566 | 2022-07-30 12:48:56.819571 | | f
| f
7237 | Development tools | /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ | A basic development environment. | 2022-08-30 18:48:28.200156 | 2022-08-30 18:48:28.200203 | | f
| f
(2 rows)
I’ll find it somewhat curious that the new package group is added again on the 30th of a month. And if that’s the true creation date it would also mean that it has been successfully published and promoted in that CV for a couple for times since then. So if I understand correctly, the package group must have been duplicated in pulpcore for two weeks and only this weekend it has been removed from there.
A grep on those two uuids in the http log reveals that both uuids worked for a while.
[root@foreman httpd]# fgrep -h -e 6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc -e e69069bf-6755-4977-9221-1b7a395e4890 foreman-ssl_access_ssl.log-20220* foreman-ssl_access_ssl.log
2001:638:70e:19::1 - - [15/Aug/2022:04:06:16 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [15/Aug/2022:04:06:16 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [16/Aug/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [16/Aug/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [17/Aug/2022:04:07:49 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [17/Aug/2022:04:07:49 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [18/Aug/2022:04:06:13 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [18/Aug/2022:04:06:14 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [19/Aug/2022:04:06:06 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [19/Aug/2022:04:06:06 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [22/Aug/2022:04:07:05 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [22/Aug/2022:04:07:05 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [23/Aug/2022:04:05:39 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [23/Aug/2022:04:05:39 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [24/Aug/2022:04:06:19 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [24/Aug/2022:04:06:19 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [25/Aug/2022:04:06:44 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [25/Aug/2022:04:06:44 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [26/Aug/2022:04:06:51 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [26/Aug/2022:04:06:51 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:26 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:27 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:27 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:27 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [12/Sep/2022:04:05:55 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 404 23 "-" "OpenAPI-Generator/3.17.4/ruby"
The 6d3f1b3b appeared the first time on 31/Aug/2022:10:08:27 +0200 which is exactly the publish run I did to fix the issue before in my post 13 above. So basically it seems after I have cleaned up before, I have published successfully but at the same time the duplicate package group popped up.
@iballou Can I get you any other information/logs?
Somehow I don’t quite understand the pulpcore database structure. I have tried to figure out if there are already duplicate package groups for the epel repository but somehow I am missing the link between the remote, the repository and the package group…
Hi @gvde,
If I’m reading this right, so the duplicate package group showed up after the first publish you did since the successful cleanup? Not after a repository sync?
We’ll have to get some more information from the Pulp team about how their package groups work. My thought is that content shouldn’t be disappearing, unless it’s like errata where updated units change uuids and the old ones get orphaned.
@dralley, the gist of this issue is that @gvde has certain package groups that are being duplicated and eventually destroyed somehow without any DELETE commands. The content view in question has filters, so content is being copied. I’m not sure if dependency solving is involved, but I’m going to guess not?
To be precise: the new uuid showed up the first time in the httpd log (i.e. it was accessed through httpd the first time) during the manual publish&promote after the cleanup. It was the first successful publish&promote. The publish before was that same morning in the cron job but that failed due to the other duplicates.
The creation date for the new package group in the pulpcore database it the day before 30th 18:48 UTC. The sync plan for the EPEL repositories is every 6 hours at hours 2,8,14,20:33 (UTC+2 currently).
Thus sync timing is:
30th 20:33 UTC+2, epel sync created 6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc.
31th 2:33 UTC+2, another sync
31th 8:33 UTC+2, another sync
31th 9:36 UTC+2, complete sync of epel7, epel8, epel8-next following
31th 10:06 UTC+2 started successful publish&promote
I cannot really tell when the new uuid showed up in pulp. If it’s right then it was during a sync and only during the successful publish after the cleanup katello accessed it the first time.
If someone tells me how to check the pulpcore database for those duplicates I could check directly. Or maybe I can find something in the pulpcore worker logs?
If I know how to check pulpcore for those duplicates I could do the whole cleanup again and check regularly to see when it appears.
On the other hand, if that timestamp is correct, it happened on the sync. Maybe EPEL rearranges something on the 30th of each month in their metadata causing pulpcore to create a duplicate?