If you can identify the affected repo this should work as a work around. If you try that, you may need to lower the orphan protection time after you deleted the repo, (or else wait around for the so created orphans to lose their orphan protection). I am not 100% sure about this though, so you can just try it without to start with.
First time I hear about orphan protection. How long is that by default, and where do I change it? Is this a new feature? I remember deleting a repo, doing delete orphans, resyncing the repo, and it did work immediately.
I wasn’t at all sure if orphan protection was going to be an issue in this case.
Background: Orphan protection is a Pulp (not Katello) feature. It can be controlled by this Pulp setting in /etc/pulp/settings.py. By default it is 24 hours (and I am not aware of Katello setting a different default). Orphan protection ensures that if users run sync and orphan cleanup jobs in parallel, the orphan job does not delete new content from the sync job before that sync has had a chance to add it to the relevant repository version (just created content is indistinguishable from old orphaned content). Since less than 24 hour old orphans being kept around doesn’t usually have much of an effect this feature is mostly “invisible” to users.
1 Like
Thanks for the update!
Take your time to release us a fix that don’t break pulp DB 
Same Issue
Action: Actions::Pulp3::OrphanCleanup::RemoveOrphans
Exception:
Katello::Errors::Pulp3Error: (“Cannot delete some instances of model ‘Content’ because they are referenced through protected foreign keys: ‘ReleaseFile.content_ptr’.”, {<RepositoryContent: pk=ee70d18c-b29b-4e03-91bf-62241095944c>, <RepositoryContent: pk=d7717620-7652-416a-b32c-8018d10e20fa>, <RepositoryContent: pk=0b66835b-da64-44a1-8c0a-0345498bca96>, <RepositoryContent: pk=8d6d06da-bce1-4cf5-8d40-df2e9d676761>, <RepositoryContent: pk=83307aef-3fd5-440b-8b97-9f6325ef6173>, <RepositoryContent: pk=eb05f1fb-56e2-4e15-bf2e-848e5ca7f410>, <RepositoryContent: pk=7d59872f-584b-4591-9c54-114bc0f6e6cd>, <RepositoryContent: pk=97caea9c-32bd-49fa-a0cc-0ed8ef989b00>, <RepositoryContent: pk=7abe6999-6044-4d38-92f3-ae803a67b6ba>})
Installed Packages
- ansible-collection-theforeman-foreman-3.9.0-1.el8.noarch
- ansible-collection-theforeman-operations-1.2.0-1.el8.noarch
- ansiblerole-foreman_scap_client-0.2.0-2.el8.noarch
- candlepin-4.2.13-1.el8.noarch
- candlepin-selinux-4.2.13-1.el8.noarch
- foreman-3.5.1-1.el8.noarch
- foreman-cli-3.5.1-1.el8.noarch
- foreman-client-release-3.5.1-1.el8.noarch
- foreman-debug-3.5.1-1.el8.noarch
- foreman-dynflow-sidekiq-3.5.1-1.el8.noarch
- foreman-installer-3.5.1-1.el8.noarch
- foreman-installer-katello-3.5.1-1.el8.noarch
- foreman-postgresql-3.5.1-1.el8.noarch
- foreman-proxy-3.5.1-1.el8.noarch
- foreman-redis-3.5.1-1.el8.noarch
- foreman-release-3.5.1-1.el8.noarch
- foreman-selinux-3.5.1-1.el8.noarch
- foreman-service-3.5.1-1.el8.noarch
- foreman-vmware-3.5.1-1.el8.noarch
- katello-4.7.3-1.el8.noarch
- katello-certs-tools-2.9.0-1.el8.noarch
- katello-client-bootstrap-1.7.9-1.el8.noarch
- katello-common-4.7.3-1.el8.noarch
- katello-debug-4.7.3-1.el8.noarch
- katello-repos-4.7.3-1.el8.noarch
- katello-selinux-4.0.2-2.el8.noarch
- pulpcore-selinux-1.3.2-1.el8.x86_64
- python39-pulp-ansible-0.15.0-1.el8.noarch
- python39-pulp-certguard-1.5.5-1.el8.noarch
- python39-pulp-cli-0.14.0-4.el8.noarch
- python39-pulp-container-2.14.3-1.el8.noarch
- python39-pulp-deb-2.20.0-1.el8.noarch
- python39-pulp-file-1.11.1-1.el8.noarch
- python39-pulp-ostree-2.0.0-0.8.a6.el8.noarch
- python39-pulp-python-3.7.2-2.el8.noarch
- python39-pulp-rpm-3.18.9-1.el8.noarch
- python39-pulpcore-3.21.5-1.el8.noarch
- qpid-proton-c-0.37.0-1.el8.x86_64
- rubygem-foreman-tasks-7.1.1-2.fm3_5.el8.noarch
- rubygem-foreman_acd-0.9.3-2.fm3_5.el8.noarch
- rubygem-foreman_ansible-10.4.1-1.fm3_5.el8.noarch
- rubygem-foreman_azure_rm-2.2.7-1.fm3_5.el8.noarch
- rubygem-foreman_bootdisk-21.0.3-1.fm3_5.el8.noarch
- rubygem-foreman_column_view-0.4.0-6.fm3_3.el8.noarch
- rubygem-foreman_discovery-22.0.2-1.fm3_5.el8.noarch
- rubygem-foreman_expire_hosts-8.1.0-1.fm3_5.el8.noarch
- rubygem-foreman_hooks-0.3.17-3.fm3_3.el8.noarch
- rubygem-foreman_host_reports-1.0.2-3.fm3_3.el8.noarch
- rubygem-foreman_kubevirt-0.1.9-5.fm3_5.el8.noarch
- rubygem-foreman_leapp-0.1.13-1.fm3_5.el8.noarch
- rubygem-foreman_maintain-1.2.4-1.el8.noarch
- rubygem-foreman_openscap-5.2.2-3.fm3_5.el8.noarch
- rubygem-foreman_remote_execution-8.2.0-1.fm3_5.el8.noarch
- rubygem-foreman_remote_execution-cockpit-8.2.0-1.fm3_5.el8.noarch
- rubygem-foreman_scap_client-0.5.0-1.el8.noarch
- rubygem-foreman_setup-8.0.1-2.fm3_3.el8.noarch
- rubygem-foreman_snapshot_management-2.0.3-1.fm3_5.el8.noarch
- rubygem-foreman_statistics-2.0.1-4.fm3_5.el8.noarch
- rubygem-foreman_templates-9.3.0-2.fm3_5.el8.noarch
- rubygem-foreman_webhooks-3.0.5-1.fm3_5.el8.noarch
- rubygem-hammer_cli-3.5.0-1.el8.noarch
- rubygem-hammer_cli_foreman-3.5.0-1.el8.noarch
- rubygem-hammer_cli_foreman_ansible-0.4.0-2.fm3_5.el8.noarch
- rubygem-hammer_cli_foreman_azure_rm-0.2.2-1.fm3_1.el8.noarch
- rubygem-hammer_cli_foreman_discovery-1.1.0-1.fm3_3.el8.noarch
- rubygem-hammer_cli_foreman_host_reports-0.1.0-1.fm3_3.el8.noarch
- rubygem-hammer_cli_foreman_kubevirt-0.1.5-1.fm3_1.el8.noarch
- rubygem-hammer_cli_foreman_openscap-0.1.13-2.fm3_5.el8.noarch
- rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
- rubygem-hammer_cli_foreman_ssh-0.0.3-1.fm3_5.el8.noarch
- rubygem-hammer_cli_foreman_tasks-0.0.18-1.fm3_5.el8.noarch
- rubygem-hammer_cli_foreman_templates-0.2.0-3.fm3_5.el8.noarch
- rubygem-hammer_cli_foreman_webhooks-0.0.4-1.fm3_5.el8.noarch
- rubygem-hammer_cli_katello-1.7.2-1.el8.noarch
- rubygem-katello-4.7.3-1.el8.noarch
- rubygem-pulp_ansible_client-0.15.0-1.el8.noarch
- rubygem-pulp_certguard_client-1.5.5-1.el8.noarch
- rubygem-pulp_container_client-2.14.2-1.el8.noarch
- rubygem-pulp_deb_client-2.20.0-1.el8.noarch
- rubygem-pulp_file_client-1.11.2-1.el8.noarch
- rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
- rubygem-pulp_python_client-3.7.3-1.el8.noarch
- rubygem-pulp_rpm_client-3.18.7-1.el8.noarch
- rubygem-pulpcore_client-3.21.2-1.el8.noarch
- rubygem-qpid_proton-0.37.0-1.el8.x86_64
- rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch
I use settings.local.py to add ORPHAN_PROTECTION_TIME=720
I try the temporary fix by @quba42
1 Like
The fix will be part of pulp_deb 2.21.0.
Once that is available, installing it will require both upgrading the RPM package and then applying the migrations either by running pulpcore-manager migrate or foreman-installer.
3 Likes
What is the upgrade path to get closer to the patched version pulp_deb 2.21.0 ?
I do not see this bug in roadmap katelllo 4.9
Update: The needed patch has now been released on the pulp_deb side (version 2.20.2 rather than 2.21.0 as previously planned): Pulp_deb 2.20.2 has been released! - Announcements - Pulp Community
This release now needs to be packaged for Katello either for a future release, or for existing releases. I am not yet quite sure if the latter will be possible, since the fix may require an updated rubygem-pulp_deb_client as well (need to run some tests), which can only be added for new Katello releases (need to coordinate with the Katello packaging team).
3 Likes
Update: I opened a PR against Katello packaging to add the fixed version: Update python-pulp-deb to 2.20.2 by quba42 · Pull Request #648 · theforeman/pulpcore-packaging · GitHub
This hopefully means the updated RPM will be available for Katello 4.9.
4 Likes
Can this be installed in katello 4.7 to?
My latest understanding is that once merged (and once the build pipeline is then run) the RPM will be available in Index of /pulpcore/3.22 which is in use for the released version 4.8, and will also be used for the soon to be released 4.9.
However, it cannot be used for Katello 4.7 and older, since those use a older pulpcore version than 3.22.
1 Like
Update: It looks like the plugin release is available in the pulpcore 2.22 repo (used for Katello >= 4.8):
http://yum.theforeman.org/pulpcore/3.22/el8/x86_64/python39-pulp-deb-2.20.2-1.el8.noarch.rpm
Note that the corresponding API bindings are not yet in the Katello 4.8 repo (this has been merged so it should be along with the next run of the Katello 4.8 build pipeline). In case this causes problems, you can already obtain the client gem from the nightly Katello repo (if you go this route make sure you only grab this one RPM from the nightly repo):
Once you have installed the two RPMs on your Katello 4.8 installation, you must re-run foreman-installer to ensure the Pulp DB migrations are applied. You can check on the status of the pulp_deb migrations in particular by running PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb.
Once your services are back up orphan cleanup will be working once more.
1 Like
Go for update
i use foreman-maintain
# foreman-maintain packages check-update
Running Check for available package updates
================================================================================
Check for available package updates:
Updating Subscription Management repositories.
This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.
Last metadata expiration check: 0:26:10 ago on Thu 11 May 2023 11:16:22 AM CEST.
python39-pulp-container.noarch 2.14.5-1.el8 pulpcore
python39-pulp-deb.noarch 2.20.2-1.el8 pulpcore
rubygem-foreman_ansible.noarch 11.2.0-1.fm3_6.el8 foreman-plugins
[OK]
--------------------------------------------------------------------------------
thx for all @quba42
# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb
deb
[X] 0001_initial
[X] 0002_auto_20190905_1000
[X] 0003_rename_package_field
[X] 0004_auto_20191014_1722
[X] 0005_rename_release_file
[X] 0006_debrepository
[X] 0007_create_metadata_models
[X] 0008_debremote_gpgkey
[X] 0009_apt_release_signing_service
[X] 0010_debpublication_signing_service
[X] 0011_rename_models_to_apt
[X] 0012_auto_20200803_1337
[X] 0013_aptremote_ignore_missing_package_indices
[X] 0014_swap_distribution_model
[X] 0015_add_custom_fields_to_packages
[X] 0016_switch_to_newer_jsonfield_model
[X] 0017_allow_longer_string_lists
[X] 0018_textfield_conversion
[X] 0019_immutable_metadata_constraints
[X] 0020_remove_fk_relations_in_packageindex_and_installerfileindex
Not work for me
i think i missing some thing
# hammer task list --search orphan
-------------------------------------|-----------------|---------|---------|---------------------|---------------------|---------------|---------------|---------------------------------------------------------------------------------
ID | ACTION | STATE | RESULT | STARTED AT | ENDED AT | DURATION | OWNER | TASK ERRORS
-------------------------------------|-----------------|---------|---------|---------------------|---------------------|---------------|---------------|---------------------------------------------------------------------------------
e62be1b6-f1f1-47de-a4a1-62ed85ec2709 | Remove orphans | paused | error | 2023/05/07 20:00:32 | | 310845.56186 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
27ec12bf-8d76-4ca9-9164-614bf831b9c1 | Remove orphans | paused | error | 2023/04/30 20:00:35 | | 915642.62886 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
90dac37e-6bfc-4dcd-8ba0-cfaa24cb40eb | Remove orphans | paused | error | 2023/04/23 20:00:37 | | 1520440.37486 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
ed7f8867-4396-4bf4-83a2-7ae0fcb48319 | Remove orphans | paused | error | 2023/04/16 20:00:33 | | 2125244.29686 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
1b9dca22-ff32-4447-9612-f65935ed8ac7 | Remove orphans | paused | error | 2023/03/26 20:00:31 | | 3939646.17886 | foreman_admin | could not write to file "base/pgsql_tmp/pgsql_tmp3898413.0.sharedfileset/i1of...
34c997c7-56a3-4a52-91a6-f46808fc994f | Remove orphans | paused | error | 2023/03/19 21:00:35 | | 4540842.36186 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
cea14877-9f11-46f0-bb06-90e321df42ec | Remove orphans | paused | error | 2023/03/12 21:00:21 | | 5145656.64586 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
e8725d9f-97a8-4cd6-9ed2-b5d5ad6b7afc | Remove orphans | stopped | warning | 2023/01/08 21:00:32 | 2023/01/13 16:42:45 | 416533.610356 | foreman_admin | Task canceled
-------------------------------------|-----------------|---------|---------|---------------------|---------------------|---------------|---------------|---------------------------------------------------------------------------------
I rerun taskID 1b9dca22-ff32-4447-9612-f65935ed8ac7
No error for step 1 and 2
At this time, task running for the last step
I just updated my test server, the task (manually launched) is now ending successfully! 
Note that it turns out that until Katello 4.8.1 is released, the repos used by Katello 4.8, are missing the new rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm to go with the patched pulp_deb version python39-pulp-deb-2.20.2-1.el8.noarch.rpm (which is available for Katello 4.8).
Apparently this causes problems during regular pulp_deb usage. See this post for a workaround.
Sorry for the bumpy rollout.
2 Likes