Upgrade to Katello 4.3 lots of problems, mainly repo sync

Support
1

Problem: Upgrading from Katello 3.1.8 to Katello 4.3 caused many many problems with repos / repo syncs are broken

Expected outcome: Repos are working fine

Foreman and Proxy versions: 2.3.5 / 2.3.5

Foreman and Proxy plugin versions:

Distribution and version: CentOS 7

Other relevant data:

Hi,

After a very long struggle, I managed to upgrade our Katello 3.1.8 including 5 Linux Proxies to 4.3 yesterday. I did the pulp2 to pulp3 upgrade a while ago. This was also a struggle. :slight_smile: To check I ran:

foreman-maintain health check

on all nodes (Katello server and proxies) and there are no errors reported. However we have the issue that repo syncing is broken for all the repos. The errors we see during sync are:

undefined method `repository_href’ for nil:NilClass

The reason why we have to stick with Katello 4.3 is because we need the Foreman SCC Plugin (SUSE) content. And according to the github repo https://github.com/ATIX-AG/foreman_scc_manager/tree/v1.8.18 it only works Katello up to 4.3. What is strange is that the github page refers to version 1.18.20 while the latest version that we see on our katello server is tfm-rubygem-foreman_scc_manager-1.8.17-1.fm3_1.el7.noarch. Is there a repo where i can grab the new version?

I did a fresh install of Katello 4.3 to see if the SCC plugin works fine here, but the sync is also not working here.

404, message=‘Not Found’, url=URL(‘https://updates.suse.com/SUSE/Updates/SLE-Product-SLES/15-SP1/x86_64/update_debug/?AdwOPWCkarPHiYPZ_-_za9zbI2Fjvd3EDfB1uh-LNx-XfC8aXKSkjvgzNMl71TQz2wnL4zYrmU4si1AUQWIz9RR-VKyaCR6josojt5VxUVR3SDFIQPc7XwYXY1CZORp1zL8Plys39Fkur1qCB1DVyupa69Qct4U’)

I see a lot of these in /var/log/mesages:

Jun 23 11:08:43 katello systemd: pulp_celerybeat.service holdoff time over, scheduling restart.
Jun 23 11:08:43 katello systemd: Stopped Pulp’s Celerybeat.
Jun 23 11:08:43 katello systemd: Failed to load environment files: No such file or directory
Jun 23 11:08:43 katello systemd: pulp_celerybeat.service failed to run ‘start’ task: No such file or directory
Jun 23 11:08:43 katello systemd: Failed to start Pulp’s Celerybeat.
Jun 23 11:08:43 katello systemd: Unit pulp_celerybeat.service entered failed state.
Jun 23 11:08:43 katello systemd: pulp_celerybeat.service failed.

Any help would be highly appreciated.

2

Regarding the SCC plugin: I am not sure why you are looking at the tree for 1.8.18, but with 4.3 there is really only a package for 1.8.17, with 4.4 and 4.5 a release of 1.8.20 exists. I used this plugin in different environments and different versions and it was never a show stopper on upgrades for me. The last bigger change was caused by the URL format changing because of the authentication which is probably hitting you here. This was discussed in SCC plugin issues after update to katello 4.3

3

Hi Dirk,

According to the SCC Github Page:

For Katello 4.3 I should use 1.8.20

Compatibility

Foreman Version Katello Version Plugin Version
3.1 4.3 ~> 1.8.20*

I found the package in the nightly repo:

https://yum.theforeman.org/katello/nightly/katello/el7/x86_64/tfm-rubygem-foreman_scc_manager-1.8.20-1.fm3_4.el7.noarch.rpm

On my fresh Katello 4.2 install the upgraded package (1.8.20) solved the problem of:

404, message=‘Not Found’, url=URL…

On my production katello, the one that I upgraded I also installed it and ran foreman-installer. It seems to be fine but I see this at the end:

Repository 649 Missing
Deleting 649
Failed upgrade task: katello:correct_repositories, see logs for more information.

Upgrade Step 2/2: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers:
Success!

  • Foreman is running at https://katello.a.space.corp

  • To install an additional Foreman proxy on separate machine continue by running:

    foreman-proxy-certs-generate --foreman-proxy-fqdn “$FOREMAN_PROXY” --certs-tar “/root/$FOREMAN_PROXY-certs.tar”

  • Foreman Proxy is running at https://katello.a.space.corp:9090

The full log is at /var/log/foreman-installer/katello.log

So I guess I will have to figure out what is going on with that missing repo…

4

Ok, after spending another 20+ hours. I have given up and reverted to my initial snapshot on foreman 2.3.5. As it turns out I have not successfully finished the pulp2 to pulp3 migration. I’m 100% that I had this done but the system doesn’t lie. So I’m, back now trying to migrate to pulp3 and I’m also facing serveral issues due to broken repos. I don’t really care about the repos, I can re-create them easily. I’m currently trying to remove the content-views and repos and etc. but I’m failing to remove the content-view. I was able to remove all versions from the content-view, but removing the view itself fails with an error of:

Cannot delete record because of dependent content_facets

I know I had seen this a a while ago and no my content-view is no longer linked to any activation keys or host groups. I cleared them all out. What else could this mean?

5

@jeremylenz Do you know what this refers to? (I have no idea)

6

Content facet is the content-related attributes of a host. Are any hosts associated to the content view?

cc @sajha

7

Hi,

sorry for getting back to you this late but I finally managed to do the upgrade to Katello 4.3 successfully. More or Less I still have the issue with the content-view not deleting…

There are no more hosts assigned to the content-view. I also re/unassigned all activation keys and host groups using the content-view. I don’t know what else could be linked to it…

8

Hostgroups also have content facets. If the content view is still attached to a hostgroup, that could explain it…

9

Is it still the same error message?

10

Hi,

yes still the same error message. The content-view is not linked to any hostgroup either.

11

In foreman-rake console, if you find the content view

cv = Katello::ContentView.find_by(name: "My content view")

and then query

cv.content_facets
cv.hostgroup_content_facets

what is the result?

(Now that I’m looking at the code, Host content_facet is the only one with :dependent => :restrict_with_exception, so that’s probably what’s causing the problem.)

12

Hi,

here is the output for both content-views

CentOS Stream 8

[root@katello ~]# foreman-rake console
Rubocop not loaded.
Rubocop not loaded.
Loading production environment (Rails 6.0.4.7)
irb(main):001:0> cv = Katello::ContentView.find_by(name: "Comp Linux CentOS 8 Stream x86_64")
=> #<Katello::ContentView id: 36, name: "Comp Linux CentOS 8 Stream x86_64", label: "Comp_Linux_CentOS_8_Stream_x86_64", description: nil, organization_id: 2, default: false, created_at: "2021-06-1...

irb(main):002:0> cv.content_facets
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::Host::ContentFacet id: 635, host_id: 658, uuid: "7e5c4caf-7da4-43f8-856d-70fbf0190d44", content_view_id: 36, lifecycle_environment_id: 5, kickstart_repository_id: nil, content_source_id: nil, installable_security_errata_count: 0, installable_enhancement_errata_count: 0, installable_bugfix_errata_count: 0, applicable_rpm_count: 0, upgradable_rpm_count: 0, applicable_module_stream_count: 0, upgradable_module_stream_count: 0, applicable_deb_count: 0, upgradable_deb_count: 0>]>

irb(main):003:0> cv.hostgroup_content_facets
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::Hostgroup::ContentFacet id: 84, hostgroup_id: 114, kickstart_repository_id: nil, content_source_id: 5, content_view_id: 36, lifecycle_environment_id: 5>, #<Katello::Hostgroup::ContentFacet id: 75, hostgroup_id: 105, kickstart_repository_id: nil, content_source_id: 1, content_view_id: 36, lifecycle_environment_id: 5>, #<Katello::Hostgroup::ContentFacet id: 76, hostgroup_id: 106, kickstart_repository_id: nil, content_source_id: 1, content_view_id: 36, lifecycle_environment_id: 5>, #<Katello::Hostgroup::ContentFacet id: 82, hostgroup_id: 112, kickstart_repository_id: nil, content_source_id: 1, content_view_id: 36, lifecycle_environment_id: 5>, #<Katello::Hostgroup::ContentFacet id: 114, hostgroup_id: 140, kickstart_repository_id: nil, content_source_id: 6, content_view_id: 36, lifecycle_environment_id: 5>, #<Katello::Hostgroup::ContentFacet id: 123, hostgroup_id: 149, kickstart_repository_id: nil, content_source_id: 5, content_view_id: 36, lifecycle_environment_id: 5>, #<Katello::Hostgroup::ContentFacet id: 124, hostgroup_id: 150, kickstart_repository_id: nil, content_source_id: 9, content_view_id: 36, lifecycle_environment_id: 5>]>

CentOS 7

irb(main):001:0> cv = Katello::ContentView.find_by(name: "Comp Linux CentOS 7 x86_64")
=> #<Katello::ContentView id: 12, name: "Comp Linux CentOS 7 x86_64", label: "Comp_Linux_CentOS_7_x86_64", description: nil, organization_id: 2, default: false, created_at: "2019-06-13 09:39:25", u...
irb(main):002:0> cv.content_facets
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::Host::ContentFacet id: 652, host_id: 676, uuid: "8a2c89cb-b1cd-44a9-9ccd-bda55958c28f", content_view_id: 12, lifecycle_environment_id: 1, kickstart_repository_id: nil, content_source_id: nil, installable_security_errata_count: 0, installable_enhancement_errata_count: 0, installable_bugfix_errata_count: 0, applicable_rpm_count: 0, upgradable_rpm_count: 0, applicable_module_stream_count: 0, upgradable_module_stream_count: 0, applicable_deb_count: 0, upgradable_deb_count: 0>]>
irb(main):003:0> cv.hostgroup_content_facets
=> #<ActiveRecord::Associations::CollectionProxy []>

I was not able to find both hosts with id 658 (stream) and 676 (centos 7):

[root@katello ~]# hammer host list | grep 658
[root@katello ~]# hammer host list | grep 676

Cheers,
Oliver

13

Not sure how it’s happened, but you seem to have an orphaned content facet. Removing it from the content view manually should allow you to delete the content view:

cv = Katello::ContentView.find_by(name: "Comp Linux CentOS 8 Stream x86_64")
cv.content_facets = []

# confirm the content facet is no longer associated
cv.content_facets
# should get back an empty array [] or empty <ActiveRecord::Associations::CollectionProxy []>
14

Hi,

sorry, doesn’t seem to be working:

[root@katello ~]# foreman-rake console
Rubocop not loaded.
Rubocop not loaded.
Loading production environment (Rails 6.0.4.7)
irb(main):001:0> cv = Katello::ContentView.find_by(name: "Comp Linux CentOS 8 Stream x86_64")
=> #<Katello::ContentView id: 36, name: "Comp Linux CentOS 8 Stream x86_64", label: "Comp_Linux_CentOS_8_Stream_x86_64", description: nil, organization_id: 2, default: false, created_at: "2021-06-13 18:43:0...
irb(main):002:0> cv.content_facets = []
Traceback (most recent call last):
        2: from lib/tasks/console.rake:5:in `block in <top (required)>'
        1: from (irb):2
ActiveRecord::NotNullViolation (PG::NotNullViolation: ERROR:  null value in column "content_view_id" violates not-null constraint)
DETAIL:  Failing row contains (635, 658, 7e5c4caf-7da4-43f8-856d-70fbf0190d44, null, 5, null, null, 0, 0, 0, 0, 0, 0, 0, 0, 0).
irb(main):003:0> cv.content_facets
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::Host::ContentFacet id: 635, host_id: 658, uuid: "7e5c4caf-7da4-43f8-856d-70fbf0190d44", content_view_id: 36, lifecycle_environment_id: 5, kickstart_repository_id: nil, content_source_id: nil, installable_security_errata_count: 0, installable_enhancement_errata_count: 0, installable_bugfix_errata_count: 0, applicable_rpm_count: 0, upgradable_rpm_count: 0, applicable_module_stream_count: 0, upgradable_module_stream_count: 0, applicable_deb_count: 0, upgradable_deb_count: 0>]>

Best Regards,
Oliver

15

Try first destroying the content facet itself:

cv = Katello::ContentView.find_by(name: "Comp Linux CentOS 8 Stream x86_64")
cv.content_facets.first.destroy
16

Thanks a lot. This did the trick. :slight_smile:

1 Like