Katello-nightly-rpm-pipeline 2097 failed

jenkins · August 14, 2024, 11:34pm

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/2097/

foreman-pipeline-katello-rpm-nightly (failed) (remote job)

ekohl · August 15, 2024, 10:45am

[2024-08-14T23:04:24.637Z]     2024-08-14 23:03:43 [ERROR ] [configure] Error making POST request to Foreman at https://pipe-katello-server-nightly-almalinux8.example.com/api/v2/hosts/facts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on pipe-katello-server-nightly-almalinux8.example.com for detailed information
[2024-08-14T23:04:24.637Z]     2024-08-14 23:03:43 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-pipe-katello-server-nightly-almalinux8.example.com]/ensure: change from 'absent' to 'present' failed: Error making POST request to Foreman at https://pipe-katello-server-nightly-almalinux8.example.com/api/v2/hosts/facts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on pipe-katello-server-nightly-almalinux8.example.com for detailed information

I didn’t look at the details, but I’m going to guess it’s the same as Foreman-nightly-deb-pipeline 1558 failed - #2 by ekohl.

evgeni · August 15, 2024, 1:25pm

The TL;DR is: WTF

The longer WTF is:
For some reason (probably because the fact upload influences the org taxonomy of the host) an empty SubscriptionFacet gets created and because the org id is flagged as “changed” Katello is super unhappy (rightfully, one should not move hosts between katello orgs that have subs).
The odd thing: when I patch out the “has sub facet” check – there are really no subs on that host, the host gets destroyed by some action which I couldn’t trace properly.

Interestingly, this doesn’t always happen as I did test the change yesterday on an EL8toEL9 upgrade and there it correctly updated the OS and just the OS.

Therefor I am going to revert that change for now:

ekohl · August 15, 2024, 4:18pm

On Matrix we traced it back to this change:

github.com/Katello/katello

Fixes #37696 - Add convert2rhel env fact to host subscription facet

Katello:master ← chris1984:convert2rhel

opened 03:18PM - 01 Aug 24 UTC

chris1984

+36 -1

Convert2rhel creates a custom fact with an env variable, if it got converted thr…ough Satellite or through the Customer Portal. It looks like this: ```bash [root@toledo8 ~]# cat /etc/rhsm/facts/convert2rhel.facts { "conversions.version": "1", "conversions.activity": "conversion", "conversions.packages.0.nevra": "convert2rhel-0:2.0.0-1.el8.noarch", "conversions.packages.0.signature": "RSA/SHA256, Thu May 30 13:31:33 2024, Key ID 199e2f91fd431d51", "conversions.executed": "/usr/bin/convert2rhel", "conversions.success": true, "conversions.activity_started": "2024-07-11T17:28:54.281664Z", "conversions.activity_ended": "2024-07-11T17:48:47.026664Z", "conversions.source_os.id": "Cerulean Leopard", "conversions.source_os.name": "AlmaLinux", "conversions.source_os.version": "8.10", "conversions.target_os.id": "Ootpa", "conversions.target_os.name": "Red Hat Enterprise Linux", "conversions.target_os.version": "8.10", "conversions.env.CONVERT2RHEL_THROUGH_FOREMAN": "1", <- what we care about "conversions.run_id": "null" } ``` The env part of the fact is getting filtered out due to https://github.com/theforeman/foreman/blob/55d2a0d22a0fe738342804bcc4987fc24a3b0917/app/registries/foreman/settings/facts.rb#L23 We can see that the fact with this PR is present on the system when looking with pry: ```bash [12] pry(main)> foo.subscription_facet => #<HostFacets::ReportedDataFacet:0x00007f6a30b9fb30 id: 19, host_id: 18, boot_time: Thu, 11 Jul 2024 17:49:26.000000000 UTC +00:00, created_at: Mon, 29 Jul 2024 18:36:51.167876000 UTC +00:00, updated_at: Mon, 29 Jul 2024 19:30:23.632255000 UTC +00:00, virtual: true, sockets: 1, cores: 1, ram: 7953, disks_total: nil, kernel_version: "4.18.0-553.8.1.el8_10.x86_64", bios_vendor: nil, bios_release_date: nil, bios_version: nil, convert2rhel: 1> ``` The convert2rhel team wants that fact reported through RH cloud so they know if they should keep working on enhancing this process and to see how many people are using it. Once this is in, I will make a PR to RH Cloud to add that into the report we send off. To test this: * Apply PR * Run DB Migration * On one host add the custom fact to subscription-manager in /etc/rhsm/facts * On that host then run `subscription-manager facts --update` * On a second host without the fact run `subscription-manager facts --update` * Verify you see in the DB or rails console that the host with the convert2rhel fact has the value to set to 1/true and the host without the fact has a `nil` value for the fact

Both changes are needed to trigger the problem.

system · August 22, 2024, 4:19pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.