Upgrade to Katello 4.3 lots of problems, mainly repo sync

Problem: Upgrading from Katello 3.1.8 to Katello 4.3 caused many many problems with repos / repo syncs are broken

Expected outcome: Repos are working fine

Foreman and Proxy versions: 2.3.5 / 2.3.5

Foreman and Proxy plugin versions:

Distribution and version: CentOS 7

Other relevant data:

Hi,

After a very long struggle, I managed to upgrade our Katello 3.1.8 including 5 Linux Proxies to 4.3 yesterday. I did the pulp2 to pulp3 upgrade a while ago. This was also a struggle. :slight_smile: To check I ran:

foreman-maintain health check

on all nodes (Katello server and proxies) and there are no errors reported. However we have the issue that repo syncing is broken for all the repos. The errors we see during sync are:

undefined method `repository_href’ for nil:NilClass

The reason why we have to stick with Katello 4.3 is because we need the Foreman SCC Plugin (SUSE) content. And according to the github repo https://github.com/ATIX-AG/foreman_scc_manager/tree/v1.8.18 it only works Katello up to 4.3. What is strange is that the github page refers to version 1.18.20 while the latest version that we see on our katello server is tfm-rubygem-foreman_scc_manager-1.8.17-1.fm3_1.el7.noarch. Is there a repo where i can grab the new version?

I did a fresh install of Katello 4.3 to see if the SCC plugin works fine here, but the sync is also not working here.

404, message=‘Not Found’, url=URL(‘https://updates.suse.com/SUSE/Updates/SLE-Product-SLES/15-SP1/x86_64/update_debug/?AdwOPWCkarPHiYPZ_-_za9zbI2Fjvd3EDfB1uh-LNx-XfC8aXKSkjvgzNMl71TQz2wnL4zYrmU4si1AUQWIz9RR-VKyaCR6josojt5VxUVR3SDFIQPc7XwYXY1CZORp1zL8Plys39Fkur1qCB1DVyupa69Qct4U’)

I see a lot of these in /var/log/mesages:

Jun 23 11:08:43 katello systemd: pulp_celerybeat.service holdoff time over, scheduling restart.
Jun 23 11:08:43 katello systemd: Stopped Pulp’s Celerybeat.
Jun 23 11:08:43 katello systemd: Failed to load environment files: No such file or directory
Jun 23 11:08:43 katello systemd: pulp_celerybeat.service failed to run ‘start’ task: No such file or directory
Jun 23 11:08:43 katello systemd: Failed to start Pulp’s Celerybeat.
Jun 23 11:08:43 katello systemd: Unit pulp_celerybeat.service entered failed state.
Jun 23 11:08:43 katello systemd: pulp_celerybeat.service failed.

Any help would be highly appreciated.

Regarding the SCC plugin: I am not sure why you are looking at the tree for 1.8.18, but with 4.3 there is really only a package for 1.8.17, with 4.4 and 4.5 a release of 1.8.20 exists. I used this plugin in different environments and different versions and it was never a show stopper on upgrades for me. The last bigger change was caused by the URL format changing because of the authentication which is probably hitting you here. This was discussed in SCC plugin issues after update to katello 4.3

Hi Dirk,

According to the SCC Github Page:

For Katello 4.3 I should use 1.8.20

Compatibility

Foreman Version Katello Version Plugin Version
3.1 4.3 ~> 1.8.20*

I found the package in the nightly repo:

https://yum.theforeman.org/katello/nightly/katello/el7/x86_64/tfm-rubygem-foreman_scc_manager-1.8.20-1.fm3_4.el7.noarch.rpm

On my fresh Katello 4.2 install the upgraded package (1.8.20) solved the problem of:

404, message=‘Not Found’, url=URL…

On my production katello, the one that I upgraded I also installed it and ran foreman-installer. It seems to be fine but I see this at the end:

Repository 649 Missing
Deleting 649
Failed upgrade task: katello:correct_repositories, see logs for more information.

Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers:
Success!

  • Foreman is running at https://katello.a.space.corp

  • To install an additional Foreman proxy on separate machine continue by running:

    foreman-proxy-certs-generate --foreman-proxy-fqdn “$FOREMAN_PROXY” --certs-tar “/root/$FOREMAN_PROXY-certs.tar”

  • Foreman Proxy is running at https://katello.a.space.corp:9090

The full log is at /var/log/foreman-installer/katello.log

So I guess I will have to figure out what is going on with that missing repo…

Ok, after spending another 20+ hours. I have given up and reverted to my initial snapshot on foreman 2.3.5. As it turns out I have not successfully finished the pulp2 to pulp3 migration. I’m 100% that I had this done but the system doesn’t lie. So I’m, back now trying to migrate to pulp3 and I’m also facing serveral issues due to broken repos. I don’t really care about the repos, I can re-create them easily. I’m currently trying to remove the content-views and repos and etc. but I’m failing to remove the content-view. I was able to remove all versions from the content-view, but removing the view itself fails with an error of:

Cannot delete record because of dependent content_facets

I know I had seen this a a while ago and no my content-view is no longer linked to any activation keys or host groups. I cleared them all out. What else could this mean?