Remove Orphans task fails after migration and update

Problem:
The scheduled remove orphans task fails with a pulp3error.

Expected outcome:
task successful

Foreman and Proxy versions:
Foreman 3.5.1

Foreman and Proxy plugin versions:

plugins version
foreman-tasks 7.1.1
foreman_bootdisk 21.0.3
foreman_puppet 5.0.0
foreman_remote_execution 8.2.0
katello 4.7.1
puppetdb_foreman 6.0.1

Distribution and version:
Oracle Linux Server release 8.7

Other relevant data:

Last week I proceed to migrate our server from CentOS 7 with Foreman 3.3 - Katello 4.5 to a new OracleLinux 8 server. I did a full backup, installed a new server from scratch with Foreman 3.3 - Katello 4.5, restored my backup and follow the upgrade path to Foreman 3.5 - Katello 4.7.
This is the first run of the delete ophans since then.
I take a look to this topic: Remove Orphans task fails but my distribution.py file seems to be up to date.

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/ec88a399-ac2a-4ca0-a149-7e2794fbac4f/",
    "pulp_created"=>"2023-02-07T14:34:04.771+00:00",
    "state"=>"failed",
    "name"=>"pulpcore.app.tasks.orphan.orphan_cleanup",
    "logging_cid"=>"a667bb2c44f540ef893fcd4fb6771bb7",
    "started_at"=>"2023-02-07T14:34:04.837+00:00",
    "finished_at"=>"2023-02-07T14:34:08.645+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x277cc bytes=106 start="  File \"/u" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x277e0 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x277f4 bytes=99 start="  File \"/u" end="n_cleanup\n"> +
       #<Sequel::SQL::Blob:0x27808 bytes=15 content="    c.delete()\n"> +
       #<Sequel::SQL::Blob:0x2781c bytes=89 start="  File \"/u" end="in delete\n"> +
       #<Sequel::SQL::Blob:0x27830 bytes=33 start="    collec" end="el_query)\n"> +
       #<Sequel::SQL::Blob:0x27844 bytes=93 start="  File \"/u" end="n collect\n"> +
       #<Sequel::SQL::Blob:0x27858 bytes=26 start="    raise " end="tedError(\n">,
      "description"=>
       #<Sequel::SQL::Blob:0x2786c bytes=512 start="(\"Cannot d" end="ce3b56b>})">},
    "worker"=>"/pulp/api/v3/workers/df0cab6e-8da4-4848-95f5-e545184d3536/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Clean up orphan Content",
       "code"=>"clean-up.content",
       "state"=>"running",
       "total"=>105,
       "done"=>105}],
    "created_resources"=>[],
    "reserved_resources_record"=>["/pulp/api/v3/orphans/cleanup/"]}],
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>17, "failed"=>3}}

Exception

Katello::Errors::Pulp3Error: ("Cannot delete some instances of model 'Content' because
 they are referenced through protected foreign keys: 'ReleaseFile.content_ptr'.", 
{<RepositoryContent: pk=be06739d-df16-4d26-84e1-e1635adab109>, <RepositoryContent: 
pk=56781222-da5d-4551-bbb2-00d00a929b50>, <RepositoryContent: pk=79585b18-0ce4-427a-
ae37-36af20385ca2>, <RepositoryContent: pk=f64c8b94-53d3-4e4d-8dfd-5271b71dd90c>,
 <RepositoryContent: pk=8c2dab7e-f71d-4975-8bd4-a50f5056a36d>, <RepositoryContent:
 pk=53e529c6-9c55-41a3-b07d-5d08fce3b56b>})```

Thanks to anyone that could help me with this

I see the same issue after upgrading from 4.6.2 to 4.7.1 during the orphan task on the main server

Katello::Errors::Pulp3Error: ("Cannot delete some instances of model 'Content' because they are referenced through protected foreign keys: 'ReleaseFile.content_ptr'.", {<RepositoryContent: pk=b75fd537-6ff6-499d-ad65-7dba40ef7fed>})

pulp worker error:

Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]: pulp [23168f4a36804ca2b26a5a267d7d19f1]: pulpcore.tasking.pulpcore_worker:INFO: Task 31e91ff8-7ca5-478e-93d1-fd9688898464 failed (("Cannot delete some instances of model 'Content' because they are referenced through protected foreign keys: 'ReleaseFile.content_ptr'.", {<RepositoryContent: pk=b75fd537-6ff6-499d-ad65-7dba40ef7fed>}))
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]: pulp [23168f4a36804ca2b26a5a267d7d19f1]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]:     result = func(*args, **kwargs)
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]:   File "/usr/lib/python3.9/site-packages/pulpcore/app/tasks/orphan.py", line 66, in orphan_cleanup
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]:     c.delete()
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 745, in delete
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]:     collector.collect(del_query)
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]:   File "/usr/lib/python3.9/site-packages/django/db/models/deletion.py", line 302, in collect
Feb 07 17:34:12 foreman8.dkrz.de pulpcore-worker-3[26210]:     raise ProtectedError(

Pulp issue pulp orphan cleanup failure (protected foreign keys) ¡ Issue #690 ¡ pulp/pulp_deb ¡ GitHub

Thanks for the info, I hope it will be handled soon…

Also ran into this issue since, I think, 4.7.1 but still seeing it in 4.7.2, so also looking forward to a fix full fix :slight_smile:

Hi @quba42
Any news about this issue?

@hstct is actively working on the fix.

We have been able to fully reproduce the issue using our existing test fixtures, so it should be fairly straightforward to fix. Since the fix will likely involve dropping a DB relation, the bigger question is how quickly it will filter back into Katello, since it is generally not possible to backport Pulp DB migrations.

We may need to think about creating a workaround patch for older pulp_deb releases as well. The problem with that, is that this patch would likely need to be a temporary change to pulpcore. :thinking:

Which versions of Katello are affected?

@rgp This can happen in all versions of Katello irrespective of your upgrade path, it is just that the circumstances where it will happen are pretty limited: First you need a APT repo distribution, that you sync regularly where at least one component/architecture combination changes, while another component/architecture combination remains unchanged between syncs. Then Katello needs to delete the Pulp repo version from the older sync. At that point you will run into the issue.

It is also possible for the issue to resolve itself after further syncs of the affected repo (depending on how the affected upstream repo changes, and on whether Katello will keep deleting old repo versions).

I see. Hopefully that is a rare occurrence. Would deleting the repo, running remove orphan, an re-adding it, work as workaround?

If you can identify the affected repo this should work as a work around. If you try that, you may need to lower the orphan protection time after you deleted the repo, (or else wait around for the so created orphans to lose their orphan protection). I am not 100% sure about this though, so you can just try it without to start with.

First time I hear about orphan protection. How long is that by default, and where do I change it? Is this a new feature? I remember deleting a repo, doing delete orphans, resyncing the repo, and it did work immediately.

I wasn’t at all sure if orphan protection was going to be an issue in this case.

Background: Orphan protection is a Pulp (not Katello) feature. It can be controlled by this Pulp setting in /etc/pulp/settings.py. By default it is 24 hours (and I am not aware of Katello setting a different default). Orphan protection ensures that if users run sync and orphan cleanup jobs in parallel, the orphan job does not delete new content from the sync job before that sync has had a chance to add it to the relevant repository version (just created content is indistinguishable from old orphaned content). Since less than 24 hour old orphans being kept around doesn’t usually have much of an effect this feature is mostly “invisible” to users.

1 Like

Thanks for the update!
Take your time to release us a fix that don’t break pulp DB :slight_smile:

Same Issue
Action: Actions::Pulp3::OrphanCleanup::RemoveOrphans

Exception:

Katello::Errors::Pulp3Error: (“Cannot delete some instances of model ‘Content’ because they are referenced through protected foreign keys: ‘ReleaseFile.content_ptr’.”, {<RepositoryContent: pk=ee70d18c-b29b-4e03-91bf-62241095944c>, <RepositoryContent: pk=d7717620-7652-416a-b32c-8018d10e20fa>, <RepositoryContent: pk=0b66835b-da64-44a1-8c0a-0345498bca96>, <RepositoryContent: pk=8d6d06da-bce1-4cf5-8d40-df2e9d676761>, <RepositoryContent: pk=83307aef-3fd5-440b-8b97-9f6325ef6173>, <RepositoryContent: pk=eb05f1fb-56e2-4e15-bf2e-848e5ca7f410>, <RepositoryContent: pk=7d59872f-584b-4591-9c54-114bc0f6e6cd>, <RepositoryContent: pk=97caea9c-32bd-49fa-a0cc-0ed8ef989b00>, <RepositoryContent: pk=7abe6999-6044-4d38-92f3-ae803a67b6ba>})

Installed Packages

  • ansible-collection-theforeman-foreman-3.9.0-1.el8.noarch
  • ansible-collection-theforeman-operations-1.2.0-1.el8.noarch
  • ansiblerole-foreman_scap_client-0.2.0-2.el8.noarch
  • candlepin-4.2.13-1.el8.noarch
  • candlepin-selinux-4.2.13-1.el8.noarch
  • foreman-3.5.1-1.el8.noarch
  • foreman-cli-3.5.1-1.el8.noarch
  • foreman-client-release-3.5.1-1.el8.noarch
  • foreman-debug-3.5.1-1.el8.noarch
  • foreman-dynflow-sidekiq-3.5.1-1.el8.noarch
  • foreman-installer-3.5.1-1.el8.noarch
  • foreman-installer-katello-3.5.1-1.el8.noarch
  • foreman-postgresql-3.5.1-1.el8.noarch
  • foreman-proxy-3.5.1-1.el8.noarch
  • foreman-redis-3.5.1-1.el8.noarch
  • foreman-release-3.5.1-1.el8.noarch
  • foreman-selinux-3.5.1-1.el8.noarch
  • foreman-service-3.5.1-1.el8.noarch
  • foreman-vmware-3.5.1-1.el8.noarch
  • katello-4.7.3-1.el8.noarch
  • katello-certs-tools-2.9.0-1.el8.noarch
  • katello-client-bootstrap-1.7.9-1.el8.noarch
  • katello-common-4.7.3-1.el8.noarch
  • katello-debug-4.7.3-1.el8.noarch
  • katello-repos-4.7.3-1.el8.noarch
  • katello-selinux-4.0.2-2.el8.noarch
  • pulpcore-selinux-1.3.2-1.el8.x86_64
  • python39-pulp-ansible-0.15.0-1.el8.noarch
  • python39-pulp-certguard-1.5.5-1.el8.noarch
  • python39-pulp-cli-0.14.0-4.el8.noarch
  • python39-pulp-container-2.14.3-1.el8.noarch
  • python39-pulp-deb-2.20.0-1.el8.noarch
  • python39-pulp-file-1.11.1-1.el8.noarch
  • python39-pulp-ostree-2.0.0-0.8.a6.el8.noarch
  • python39-pulp-python-3.7.2-2.el8.noarch
  • python39-pulp-rpm-3.18.9-1.el8.noarch
  • python39-pulpcore-3.21.5-1.el8.noarch
  • qpid-proton-c-0.37.0-1.el8.x86_64
  • rubygem-foreman-tasks-7.1.1-2.fm3_5.el8.noarch
  • rubygem-foreman_acd-0.9.3-2.fm3_5.el8.noarch
  • rubygem-foreman_ansible-10.4.1-1.fm3_5.el8.noarch
  • rubygem-foreman_azure_rm-2.2.7-1.fm3_5.el8.noarch
  • rubygem-foreman_bootdisk-21.0.3-1.fm3_5.el8.noarch
  • rubygem-foreman_column_view-0.4.0-6.fm3_3.el8.noarch
  • rubygem-foreman_discovery-22.0.2-1.fm3_5.el8.noarch
  • rubygem-foreman_expire_hosts-8.1.0-1.fm3_5.el8.noarch
  • rubygem-foreman_hooks-0.3.17-3.fm3_3.el8.noarch
  • rubygem-foreman_host_reports-1.0.2-3.fm3_3.el8.noarch
  • rubygem-foreman_kubevirt-0.1.9-5.fm3_5.el8.noarch
  • rubygem-foreman_leapp-0.1.13-1.fm3_5.el8.noarch
  • rubygem-foreman_maintain-1.2.4-1.el8.noarch
  • rubygem-foreman_openscap-5.2.2-3.fm3_5.el8.noarch
  • rubygem-foreman_remote_execution-8.2.0-1.fm3_5.el8.noarch
  • rubygem-foreman_remote_execution-cockpit-8.2.0-1.fm3_5.el8.noarch
  • rubygem-foreman_scap_client-0.5.0-1.el8.noarch
  • rubygem-foreman_setup-8.0.1-2.fm3_3.el8.noarch
  • rubygem-foreman_snapshot_management-2.0.3-1.fm3_5.el8.noarch
  • rubygem-foreman_statistics-2.0.1-4.fm3_5.el8.noarch
  • rubygem-foreman_templates-9.3.0-2.fm3_5.el8.noarch
  • rubygem-foreman_webhooks-3.0.5-1.fm3_5.el8.noarch
  • rubygem-hammer_cli-3.5.0-1.el8.noarch
  • rubygem-hammer_cli_foreman-3.5.0-1.el8.noarch
  • rubygem-hammer_cli_foreman_ansible-0.4.0-2.fm3_5.el8.noarch
  • rubygem-hammer_cli_foreman_azure_rm-0.2.2-1.fm3_1.el8.noarch
  • rubygem-hammer_cli_foreman_discovery-1.1.0-1.fm3_3.el8.noarch
  • rubygem-hammer_cli_foreman_host_reports-0.1.0-1.fm3_3.el8.noarch
  • rubygem-hammer_cli_foreman_kubevirt-0.1.5-1.fm3_1.el8.noarch
  • rubygem-hammer_cli_foreman_openscap-0.1.13-2.fm3_5.el8.noarch
  • rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
  • rubygem-hammer_cli_foreman_ssh-0.0.3-1.fm3_5.el8.noarch
  • rubygem-hammer_cli_foreman_tasks-0.0.18-1.fm3_5.el8.noarch
  • rubygem-hammer_cli_foreman_templates-0.2.0-3.fm3_5.el8.noarch
  • rubygem-hammer_cli_foreman_webhooks-0.0.4-1.fm3_5.el8.noarch
  • rubygem-hammer_cli_katello-1.7.2-1.el8.noarch
  • rubygem-katello-4.7.3-1.el8.noarch
  • rubygem-pulp_ansible_client-0.15.0-1.el8.noarch
  • rubygem-pulp_certguard_client-1.5.5-1.el8.noarch
  • rubygem-pulp_container_client-2.14.2-1.el8.noarch
  • rubygem-pulp_deb_client-2.20.0-1.el8.noarch
  • rubygem-pulp_file_client-1.11.2-1.el8.noarch
  • rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
  • rubygem-pulp_python_client-3.7.3-1.el8.noarch
  • rubygem-pulp_rpm_client-3.18.7-1.el8.noarch
  • rubygem-pulpcore_client-3.21.2-1.el8.noarch
  • rubygem-qpid_proton-0.37.0-1.el8.x86_64
  • rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch

I use settings.local.py to add ORPHAN_PROTECTION_TIME=720

I try the temporary fix by @quba42

1 Like

The fix will be part of pulp_deb 2.21.0.
Once that is available, installing it will require both upgrading the RPM package and then applying the migrations either by running pulpcore-manager migrate or foreman-installer.

3 Likes

What is the upgrade path to get closer to the patched version pulp_deb 2.21.0 ?

I do not see this bug in roadmap katelllo 4.9

Update: The needed patch has now been released on the pulp_deb side (version 2.20.2 rather than 2.21.0 as previously planned): Pulp_deb 2.20.2 has been released! - Announcements - Pulp Community

This release now needs to be packaged for Katello either for a future release, or for existing releases. I am not yet quite sure if the latter will be possible, since the fix may require an updated rubygem-pulp_deb_client as well (need to run some tests), which can only be added for new Katello releases (need to coordinate with the Katello packaging team).

3 Likes

Update: I opened a PR against Katello packaging to add the fixed version: Update python-pulp-deb to 2.20.2 by quba42 ¡ Pull Request #648 ¡ theforeman/pulpcore-packaging ¡ GitHub

This hopefully means the updated RPM will be available for Katello 4.9.

4 Likes

Can this be installed in katello 4.7 to?