Remove Orphans task fails after migration and update

The fix will be part of pulp_deb 2.21.0.
Once that is available, installing it will require both upgrading the RPM package and then applying the migrations either by running pulpcore-manager migrate or foreman-installer.

3 Likes

What is the upgrade path to get closer to the patched version pulp_deb 2.21.0 ?

I do not see this bug in roadmap katelllo 4.9

Update: The needed patch has now been released on the pulp_deb side (version 2.20.2 rather than 2.21.0 as previously planned): Pulp_deb 2.20.2 has been released! - Announcements - Pulp Community

This release now needs to be packaged for Katello either for a future release, or for existing releases. I am not yet quite sure if the latter will be possible, since the fix may require an updated rubygem-pulp_deb_client as well (need to run some tests), which can only be added for new Katello releases (need to coordinate with the Katello packaging team).

3 Likes

Update: I opened a PR against Katello packaging to add the fixed version: Update python-pulp-deb to 2.20.2 by quba42 · Pull Request #648 · theforeman/pulpcore-packaging · GitHub

This hopefully means the updated RPM will be available for Katello 4.9.

4 Likes

Can this be installed in katello 4.7 to?

My latest understanding is that once merged (and once the build pipeline is then run) the RPM will be available in Index of /pulpcore/3.22 which is in use for the released version 4.8, and will also be used for the soon to be released 4.9.

However, it cannot be used for Katello 4.7 and older, since those use a older pulpcore version than 3.22.

1 Like

Update: It looks like the plugin release is available in the pulpcore 2.22 repo (used for Katello >= 4.8):

http://yum.theforeman.org/pulpcore/3.22/el8/x86_64/python39-pulp-deb-2.20.2-1.el8.noarch.rpm

Note that the corresponding API bindings are not yet in the Katello 4.8 repo (this has been merged so it should be along with the next run of the Katello 4.8 build pipeline). In case this causes problems, you can already obtain the client gem from the nightly Katello repo (if you go this route make sure you only grab this one RPM from the nightly repo):

http://yum.theforeman.org/katello/nightly/katello/el8/x86_64/rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm

Once you have installed the two RPMs on your Katello 4.8 installation, you must re-run foreman-installer to ensure the Pulp DB migrations are applied. You can check on the status of the pulp_deb migrations in particular by running PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb.

Once your services are back up orphan cleanup will be working once more.

1 Like

Go for update

i use foreman-maintain

# foreman-maintain packages check-update 
Running Check for available package updates
================================================================================
Check for available package updates: 
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Last metadata expiration check: 0:26:10 ago on Thu 11 May 2023 11:16:22 AM CEST.

python39-pulp-container.noarch        2.14.5-1.el8               pulpcore       
python39-pulp-deb.noarch              2.20.2-1.el8               pulpcore       
rubygem-foreman_ansible.noarch        11.2.0-1.fm3_6.el8         foreman-plugins
                                                                      [OK]
--------------------------------------------------------------------------------

:wink:

thx for all @quba42

# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb
deb
 [X] 0001_initial
 [X] 0002_auto_20190905_1000
 [X] 0003_rename_package_field
 [X] 0004_auto_20191014_1722
 [X] 0005_rename_release_file
 [X] 0006_debrepository
 [X] 0007_create_metadata_models
 [X] 0008_debremote_gpgkey
 [X] 0009_apt_release_signing_service
 [X] 0010_debpublication_signing_service
 [X] 0011_rename_models_to_apt
 [X] 0012_auto_20200803_1337
 [X] 0013_aptremote_ignore_missing_package_indices
 [X] 0014_swap_distribution_model
 [X] 0015_add_custom_fields_to_packages
 [X] 0016_switch_to_newer_jsonfield_model
 [X] 0017_allow_longer_string_lists
 [X] 0018_textfield_conversion
 [X] 0019_immutable_metadata_constraints
 [X] 0020_remove_fk_relations_in_packageindex_and_installerfileindex

Not work for me
i think i missing some thing

# hammer task list --search orphan
-------------------------------------|-----------------|---------|---------|---------------------|---------------------|---------------|---------------|---------------------------------------------------------------------------------
ID                                   | ACTION          | STATE   | RESULT  | STARTED AT          | ENDED AT            | DURATION      | OWNER         | TASK ERRORS                                                                     
-------------------------------------|-----------------|---------|---------|---------------------|---------------------|---------------|---------------|---------------------------------------------------------------------------------
e62be1b6-f1f1-47de-a4a1-62ed85ec2709 | Remove orphans  | paused  | error   | 2023/05/07 20:00:32 |                     | 310845.56186  | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
27ec12bf-8d76-4ca9-9164-614bf831b9c1 | Remove orphans  | paused  | error   | 2023/04/30 20:00:35 |                     | 915642.62886  | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
90dac37e-6bfc-4dcd-8ba0-cfaa24cb40eb | Remove orphans  | paused  | error   | 2023/04/23 20:00:37 |                     | 1520440.37486 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
ed7f8867-4396-4bf4-83a2-7ae0fcb48319 | Remove orphans  | paused  | error   | 2023/04/16 20:00:33 |                     | 2125244.29686 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
1b9dca22-ff32-4447-9612-f65935ed8ac7 | Remove orphans  | paused  | error   | 2023/03/26 20:00:31 |                     | 3939646.17886 | foreman_admin | could not write to file "base/pgsql_tmp/pgsql_tmp3898413.0.sharedfileset/i1of...
34c997c7-56a3-4a52-91a6-f46808fc994f | Remove orphans  | paused  | error   | 2023/03/19 21:00:35 |                     | 4540842.36186 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
cea14877-9f11-46f0-bb06-90e321df42ec | Remove orphans  | paused  | error   | 2023/03/12 21:00:21 |                     | 5145656.64586 | foreman_admin | ("Cannot delete some instances of model 'Content' because they are referenced...
e8725d9f-97a8-4cd6-9ed2-b5d5ad6b7afc | Remove orphans  | stopped | warning | 2023/01/08 21:00:32 | 2023/01/13 16:42:45 | 416533.610356 | foreman_admin | Task canceled                                                                   
-------------------------------------|-----------------|---------|---------|---------------------|---------------------|---------------|---------------|---------------------------------------------------------------------------------

I rerun taskID 1b9dca22-ff32-4447-9612-f65935ed8ac7
No error for step 1 and 2
image
At this time, task running for the last step

It work

After rerun all task one by one

I just updated my test server, the task (manually launched) is now ending successfully! :sunglasses:

Note that it turns out that until Katello 4.8.1 is released, the repos used by Katello 4.8, are missing the new rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm to go with the patched pulp_deb version python39-pulp-deb-2.20.2-1.el8.noarch.rpm (which is available for Katello 4.8).

Apparently this causes problems during regular pulp_deb usage. See this post for a workaround.

Sorry for the bumpy rollout.

2 Likes

Thanks for the advice, I just installed the missing update. :+1:

Hi, after upgrading to Katello 4.8.1 (OS: RHEL 8.6, Foreman: 3.6.1) i am getting this error by Remove Orphans:
There was an issue with the backend service pulp3: Connection reset by peer - SSL_connectThere was an issue with the backend service pulp3: Connection reset by peer - SSL_connect

I am not sure from the pastebin you posted, (see if you can find a pulp task backtrace/error in your failed Foreman task), but my gut feeling is that this is a different issue than the one described in this thread.

I have double checked that Katello 4.8.1 has access to every part of the pulp_deb fix. If you have successfully upgraded to this version, it is not really possible to still be getting the original orphan cleanup error from this thread (of course it is entirely possible to be hitting some different issue).

Things you can check to verify if you are relevantly upgraded:

  • Is python39-pulp-deb-2.20.2-1.el8.noarch.rpm installed on your system?
  • Is rubygem-pulp_deb_client-2.20.2-1.el8.noarch.rpm installed on your system?
  • Try running PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb and check that all pulp_deb migrations are applied.
  • Check if all services are running properly: foreman-maintain service status

I have checkt it and it is OK:
1.

dnf list installed | grep python39-pulp-deb

python39-pulp-deb.noarch 2.20.2-1.el8 pulpcore

dnf list installed | grep rubygem-pulp_deb_client

rubygem-pulp_deb_client.noarch 2.20.2-1.el8 katello

PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager showmigrations deb

deb
0001_initial
0002_auto_20190905_1000
0003_rename_package_field
0004_auto_20191014_1722
0005_rename_release_file
0006_debrepository
0007_create_metadata_models
0008_debremote_gpgkey
0009_apt_release_signing_service
0010_debpublication_signing_service
0011_rename_models_to_apt
0012_auto_20200803_1337
0013_aptremote_ignore_missing_package_indices
0014_swap_distribution_model
0015_add_custom_fields_to_packages
0016_switch_to_newer_jsonfield_model
0017_allow_longer_string_lists
0018_textfield_conversion
0019_immutable_metadata_constraints
0020_remove_fk_relations_in_packageindex_and_installerfileindex

\ All services are running [OK]

No i am getting an other error:

(“Cannot delete some instances of model ‘Content’ because they are referenced through protected foreign keys: ‘Package.content_ptr’.”, {<RepositoryContent: pk=f0f6e1cf-c3f1-41ee-ad7e-ca2f0e0f1266>})

Production log: