Problem with cleaning up the mess Foreman 3.1.2/Katello 4.3.0 created

[foreman-materna]

Hello,

followed this thread for fixing foreman and broken reposyncs:

Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2)

@iballou thank you again for your help. I finally managed to get it all back in order, this is what I did: update to the latest RPMs (which included the pulp core fix for double encrypted entries) rebooted foreman and the proxy ran the datarepair-2327 on the proxy, it repaired some entries ran it again on the proxy, this time 0 errors ran remove-orphan, which succeeded for the proxy but timed out on the foreman server discovered, that pulp-core constantly restarts on the foreman server and id…

I am now at step 14 and get this error while deleting the old content views:

update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
DETAIL:  Key (pulp_id)=(220bc134-0614-4073-b581-4fee78b14164) is still referenced from table "pulp_2to3_migration_pulp2distributor".
update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
DETAIL:  Key (pulp_id)=(69afd4f0-537f-4baf-9972-75b8ecbcbedf) is still referenced from table "pulp_2to3_migration_pulp2distributor".
update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
DETAIL:  Key (pulp_id)=(9bb89591-0e83-488b-bc2e-d02ef49f3862) is still referenced from table "pulp_2to3_migration_pulp2distributor".
update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
DETAIL:  Key (pulp_id)=(0ef1da79-4ddb-409a-9532-e4acbbdcbd28) is still referenced from table "pulp_2to3_migration_pulp2distributor".
update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
DETAIL:  Key (pulp_id)=(cd293352-3768-4dd3-8aa7-3780de29c982) is still referenced from table "pulp_2to3_migration_pulp2distributor".
update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
DETAIL:  Key (pulp_id)=(3372dd46-6e2b-4f76-9020-fedd3288ff9d) is still referenced from table "pulp_2to3_migration_pulp2distributor".
update or delete on table "core_publication" violates foreign key constraint "pulp_2to3_migration__pulp3_publication_id_221e8b1c_fk_core_publ" on table "pulp_2to3_migration_pulp2distributor"
DETAIL:  Key (pulp_id)=(f6c4ef76-947a-47d6-ac6f-0beb6c506f23) is still referenced from table "pulp_2to3_migration_pulp2distributor".

What do I have to do?
The pulp project itself just says “good luck & have fun”:
https://pulp.plan.io/issues/8351

[rbremer]

I had this problem too, it looks like after the migration from pulp 2 to 3 finished, the tables haven’t been dropped.
See this Katello 4.3.0: cannot delete repositories - #3 by gvde for more info, that did drop the tables for me.

[foreman-materna]

Thank you for clarification.
Had first to clean mongodb myself, because /opt/foreman lives on its own
drive and /var/lib/mongodb gets remounted from there.

Now I get this error:

foreman-maintain content remove-pulp2
Running Remove Pulp2 and mongodb packages and data
================================================================================
Remove pulp2:

WARNING: All pulp2 packages will be removed with the following commands:

# yum remove rh-mongodb34-*
# yum remove squid mod_wsgi

All pulp2 data will be removed.


Do you want to proceed?, [y(yes), q(quit)] y
- Removing mongo packages
| Removing additional packages
\ Dropping migration tables
/ Dropping migrations
\ Restarting pulpcore services                                        [FAIL]
undefined method `services' for nil:NilClass
--------------------------------------------------------------------------------
Scenario [Remove Pulp2 and mongodb packages and data] failed.

The following steps ended up in failing state:

  [pulp-remove]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="pulp-remove"