Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2)

Support
38

@iballou thank you again for your help.

I finally managed to get it all back in order, this is what I did:

  1. update to the latest RPMs (which included the pulp core fix for double encrypted entries)
  2. rebooted foreman and the proxy
  3. ran the datarepair-2327 on the proxy, it repaired some entries
  4. ran it again on the proxy, this time 0 errors
  5. ran remove-orphan, which succeeded for the proxy but timed out on the foreman server
  6. discovered, that pulp-core constantly restarts on the foreman server and identified the cause (md5 is enabled and 40k+ entries didn’t have md5 hashes set)
  7. did something nasty, ran SQL to set the md5 has to the value “1” for all NULL (I won’t post the SQL command here as I don’t wanna encourage anyone being as “stupidly brave”
  8. pulp-core did now restart fine
  9. tried the remove-orphan again, this time it worked on both foreman and the proxy
  10. handle-artifacts-checksum still failed with that one file which was removed from upstream
  11. did a repository sync on ALL my repositories (normal sync), as they are set to “data only” for mirroring, removed upstream content is also removed
  12. discovered, that the RPM in question is still visible in the Foreman GUI
  13. published and promoted new version of each and every content view
  14. deleted all previous version of each and every content view
  15. ran remove-orphan again, successful on both the proxy and foreman server
  16. used SQL (again boldly brave) to reset the MD5 hash to NULL where it was “1”
  17. ran handle-artifact-checksums and for the first time it completed successfully, regenerating all MD5 checksums and downloading thousands of missing files (those didn’t survive the pulp2 → pulp3 migration)
  18. ran SQL to check if I have missing MD5 checksums: 0!
  19. full sync of the proxy (still running)

So tomorrow I will check for the result of 19 and if all is fine, will be finally back with a fully functioning Foreman/Katello system.

Big thanks to everyone, who helped me throughout this process. Hopefully nobody else encounters those issues after updating too quickly from 3 to 4.

3 Likes