Foreman centos7 errata processing

George · January 19, 2022, 12:40pm

Problem:
I think there are some issues how foreman processes the errata.

I used this python script to process the errata.xml file.GitHub - vmfarms/generate_updateinfo
then proceeded to make a local repository on the host and synced the repo to foreman. All this works just fine.

The issue however seems to be with how foreman handles that information. CESA_2021__5192 for example. According to foreman there are no hosts that this applies to. While according to spacewalk this errata affects hosts that are registered to foreman.

Package installed currently on the host.
samba-common-4.10.16-15.el7_9.noarch
Package that should contain fix for the security issue.
samba-common-4.10.16-17.el7_9.noarch

So I believe that foreman does not correctly inform the admin about what needs to be updated. AFAIK the packages where the version is lower than what is listed on the updated packages here Red Hat Customer Portal - Access to 24x7 support and knowledge is affected.

Foreman does see that the package needs to be updated but doesn’t inform that there is some errata to be applied to the host even though the errata is synced to the host.

I’m curious wheter this is an issue with how foreman handles this specific errata file or something else. ATM I have not updated the package yet, but I’m guessing that when the package matches the package version on the errata updated packaged foreman might inform that there is errata that need to be applied. I’m currently using this errata file. https://cefs.steve-meier.de/errata.latest.xml.bz2
Expected outcome:
Foreman show errata correctly on which packages are affected.
Foreman and Proxy versions:
foreman version 3.0.1
Foreman and Proxy plugin versions:
foreman-tasks 5.1.1
foreman_puppet 1.0.5
foreman_remote_execution 4.8.0
katello 4.2.1

Other relevant data:

tvziem · January 20, 2022, 3:09pm

Having exactly the same problem :’(

George · April 8, 2022, 12:14pm

So seems that there are has been some crucial changes between the major versions or some but etc. in 3.0.1 that causes this not to work. Fired up an older test foreman with 2.5.3 and it just worked.

Will have to dig deeper on this issue later.

tedevil · April 20, 2022, 7:16am

Actually just created an errata repo for CentOS 7 and 8 and it links fine to the packages on the hosts that I subscribed these repos to. Just took a while before it worked, maybe some index job had to run…

George · April 20, 2022, 10:41am

Seems that there might be something weird happening with my installation.

For example some of the hosts with around 200-300 packages should have a lot of errata to be applied. I certainly would expect that a host which has 600 packages needing updates would also have some errata to be applied to it.

Some index job etc. might explain these issue if it does not run properly always and therefore cannot associate errata with packages properly.

George · April 20, 2022, 11:16am

Like these 2 hosts are identical regarding installed packages etc. but only 1 of them shows errata which need to be applied.

George · April 20, 2022, 1:50pm

So yeah seems that the issue is preciding with hammer in this case.

Those 2 hosts mentioned on the previous comment are identical in every even with every installed packages.

#hammer host errata list --host-id 71
---|------------|------|-------|------------
ID | ERRATUM ID | TYPE | TITLE | INSTALLABLE
---|------------|------|-------|------------
#hammer host errata list --host-id 72
------|-----------------|----------|------------------------------------------|------------
ID    | ERRATUM ID      | TYPE     | TITLE                                    | INSTALLABLE
------|-----------------|----------|------------------------------------------|------------
15211 | CEBA_2022__1193 | bugfix   | CentOS microcode_ctl Update              | true
15209 | CEBA_2022__1201 | bugfix   | CentOS libpcap Update                    | true
15208 | CEBA_2022__1203 | bugfix   | CentOS rsyslog Update                    | true
15206 | CEBA_2022__1202 | bugfix   | CentOS net-snmp Update                   | true
15205 | CESA_2022__1198 | security | Important CentOS kernel Update           | true
15123 | CESA_2022__1066 | security | Important CentOS openssl Security Update | true
15122 | CESA_2022__1069 | security | Important CentOS expat Security Update   | true
15103 | CEBA_2022__1032 | bugfix   | CentOS tzdata Update                     | true
------|-----------------|----------|------------------------------------------|------------
#hammer host errata recalculate --host-id 71
Could not recalculate errata:
  Error: Please mention appropriate attribute value

  See: 'hammer host errata recalculate --help'.
#hammer host errata recalculate --host-id 71 --async
Errata recalculated with task %{id}.
#hammer host errata list --host-id 71
---|------------|------|-------|------------
ID | ERRATUM ID | TYPE | TITLE | INSTALLABLE
---|------------|------|-------|------------

The logs even say that the task was successfully executed.

2022-04-20T16:37:23 [I|bac|215288c9] Task {label: Actions::Katello::Applicability::Hosts::BulkGenerate, id: 0b1a2c58-c14a-4c04-ba92-dc47ab2ea337, execution_plan_id: 89ed4d0e-3c0e-46cc-8065-0de20341961e} state changed: stopped  result: success
2022-04-20T16:37:23 [I|bac|215288c9] Task {label: Actions::Katello::Applicability::Hosts::BulkGenerate, id: 0b1a2c58-c14a-4c04-ba92-dc47ab2ea337, execution_plan_id: 89ed4d0e-3c0e-46cc-8065-0de20341961e} state changed: stopped  result: success

Also this bug is atleast affecting here where hammer command doesn’t return the task id. Also the task does nothing here.

I recently updated my foreman installation but that didn’t help. Next i’m going to try to upgrade this to a 3.1 maybe that will solve this issue. Currently installed packages.


Installed Packages

    candlepin-4.1.7-1.el8.noarch
    candlepin-selinux-4.1.7-1.el8.noarch
    foreman-3.0.2-1.el8.noarch
    foreman-cli-3.0.2-1.el8.noarch
    foreman-debug-3.0.2-1.el8.noarch
    foreman-dynflow-sidekiq-3.0.2-1.el8.noarch
    foreman-installer-3.0.2-2.el8.noarch
    foreman-installer-katello-3.0.2-2.el8.noarch
    foreman-postgresql-3.0.2-1.el8.noarch
    foreman-proxy-3.0.2-1.el8.noarch
    foreman-release-3.0.2-1.el8.noarch
    foreman-selinux-3.0.2-1.el8.noarch
    foreman-service-3.0.2-1.el8.noarch
    foreman.palvelu.local-apache-1.0-2.noarch
    foreman.palvelu.local-foreman-client-1.0-1.noarch
    foreman.palvelu.local-foreman-proxy-1.0-2.noarch
    foreman.palvelu.local-foreman-proxy-client-1.0-1.noarch
    foreman.palvelu.local-puppet-client-1.0-1.noarch
    katello-4.2.2-1.el8.noarch
    katello-ca-consumer-foreman.palvelu.local-1.0-2.noarch
    katello-certs-tools-2.8.2-1.el8.noarch
    katello-client-bootstrap-1.7.7-1.el8.noarch
    katello-common-4.2.2-1.el8.noarch
    katello-debug-4.2.2-1.el8.noarch
    katello-default-ca-1.0-1.noarch
    katello-repos-4.2.2-1.el8.noarch
    katello-selinux-4.0.2-1.el8.noarch
    katello-server-ca-1.0-2.noarch
    pulp-client-1.0-1.noarch
    pulpcore-selinux-1.2.7-1.el8.x86_64
    python3-pulp-ansible-0.9.0-2.el8.noarch
    python3-pulp-certguard-1.4.0-3.el8.noarch
    python3-pulp-container-2.8.4-0.1.el8.noarch
    python3-pulp-deb-2.14.1-2.el8.noarch
    python3-pulp-file-1.8.2-2.el8.noarch
    python3-pulp-rpm-3.14.12-1.el8.noarch
    python3-pulpcore-3.14.15-1.el8.noarch
    qpid-proton-c-0.32.0-3.el8.x86_64
    rubygem-foreman-tasks-5.1.1-1.fm3_0.el8.noarch
    rubygem-foreman_maintain-0.8.21-1.el8.noarch
    rubygem-foreman_puppet-1.0.5-1.fm3_0.el8.noarch
    rubygem-foreman_remote_execution-4.8.0-1.fm3_0.el8.noarch
    rubygem-hammer_cli-3.0.2-1.el8.noarch
    rubygem-hammer_cli_foreman-3.0.0-1.el8.noarch
    rubygem-hammer_cli_foreman_puppet-0.0.3-1.fm3_0.el8.noarch
    rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
    rubygem-hammer_cli_foreman_tasks-0.0.16-1.fm3_0.el8.noarch
    rubygem-hammer_cli_katello-1.1.1-0.1.pre.master.20210804141838gitece0b63.el8.noarch
    rubygem-katello-4.2.2-1.el8.noarch
    rubygem-pulp_ansible_client-0.8.0-1.el8.noarch
    rubygem-pulp_certguard_client-1.4.0-1.el8.noarch
    rubygem-pulp_container_client-2.7.0-1.el8.noarch
    rubygem-pulp_deb_client-2.13.0-1.el8.noarch
    rubygem-pulp_file_client-1.8.1-1.el8.noarch
    rubygem-pulp_python_client-3.4.0-1.el8.noarch
    rubygem-pulp_rpm_client-3.13.3-1.el8.noarch
    rubygem-pulpcore_client-3.14.1-1.el8.noarch
    rubygem-qpid_proton-0.32.0-3.el8.x86_64
    rubygem-smart_proxy_pulp-3.1.0-1.fm2_6.el8.noarch

tedevil · April 20, 2022, 7:42pm

I am on a new installed Foreman 3.2 and Katello 4.4.0.2 and do not see the issues you have.

George · April 21, 2022, 11:09am

I upgraded from 3.0 to 3.1 then to 3.2. Did not solve the issue for me. Don’t really know what is causing this since the tasks are completed successfully but errata isn’t still calculated correctly for most of the hosts.
Maybe some @katello developers can give some insight on what might be happening in this case.

iballou · April 21, 2022, 3:59pm

Hi @George,

I wasn’t able to reproduce the issue with a couple of CentOS 7 hosts, so I’ll need to ask some debugging questions. Many of them will require use of the foreman console, which you can access with foreman-rake console.

For any debugging instructions that involve a host, please select one that has applicable RPMs but missing applicable errata.

Does installing a new package (or running katello-package-upload -f) fix the issue?
Do you have any duplicate errata? You can check with the following code:

::Katello::Erratum.group(:errata_id).having("count(errata_id) > 1").pluck(:errata_id)

Are your host’s “bound repositories” correct? Those are the repositories that the host thinks it’s consuming from. I have a feeling that maybe the errata-import repo is missing from your affected hosts’ bound repositories. You can check with the following:

bound_repos = ::Host.find_by(name: "enter host name here").content_facet.bound_repositories.collect do |repo|
  repo.library_instance_id.nil? ? repo.id : repo.library_instance_id
end  

::Katello::Repository.find(bound_repos)

The repositories returned should be the ones that your host is getting RPMs and errata from. If any repositories are missing, let me know.

Let’s take a closer look at the applicability calculations. The applicability calculation first calculates the applicable errata, then imports them into the DB. Maybe the DB import step is going wrong. Are any errata being calculated as applicable for the host? You can find that with the following, using the bound_repos calculation from above:

::Katello::Applicability::ApplicableContentHelper.new(::Host.find_by(name: "name goes here").content_facet, ::Katello::Erratum, bound_repos).fetch_errata_content_ids.count

If there is a number there, then the errata is likely not being imported correctly for the host.
You can take a closer look at the errata IDs from that calculation with the following:

::Katello::Erratum.find(::Katello::Applicability::ApplicableContentHelper.new(::Host.last.content_facet, ::Katello::Erratum, bound_repos).fetch_errata_content_ids).pluck(:errata_id)

George · April 22, 2022, 6:32am

Yeah seems that the host which has the working errata has that errata repo bound to it. Whilst the not working one doesn’t have.

Loading production environment (Rails 6.0.3.7)
irb(main):001:0> ::Katello::Erratum.group(:errata_id).having("count(errata_id) > 1").pluck(:errata_id)
=> []
irb(main):002:1* bound_repos = ::Host.find_by(name: "errata-not-working-host.name.domain").content_facet.bound_repositories.collect do |repo|
irb(main):003:1*
irb(main):004:1*   repo.library_instance_id.nil? ? repo.id : repo.library_instance_id
irb(main):005:1*
irb(main):006:0> end
=> [5, 7, 6]
irb(main):007:1* bound_repos = ::Host.find_by(name: "working-host.name.domain").content_facet.bound_repositories.collect do |repo|
irb(main):008:1*
irb(main):009:1*   repo.library_instance_id.nil? ? repo.id : repo.library_instance_id
irb(main):010:1*
irb(main):011:0> end
=> [11, 5, 7, 6]
irb(main):012:0>
irb(main):007:0> ::Katello::Repository.find(bound_repos)
=> [#<Katello::Repository id: 5, pulp_id: "927242c5-7fdd-41d3-a199-90eb534cfd37", library_instance_id: nil, content_view_version_id: 1, relative_path: "Company/Library/custom/Centos7/extras_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 5, remote_href: "/pulp/api/v3/remotes/rpm/rpm/e8ff9395-65a0-4b0c-82...", publication_href: "/pulp/api/v3/publications/rpm/rpm/eba67d6e-cdb6-42...", version_href: "/pulp/api/v3/repositories/rpm/rpm/b3cbbe16-bc34-4b...", last_contents_changed: "2022-03-04 12:00:15", last_applicability_regen: "2022-03-04 12:07:42", last_indexed: "2022-04-21 11:22:25">, 
#<Katello::Repository id: 7, pulp_id: "eba5f2b6-2b21-4ed3-b9e6-8973f7a1d098", library_instance_id: nil, content_view_version_id: 1, relative_path: "Company/Library/custom/Centos7/updates_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 7, remote_href: "/pulp/api/v3/remotes/rpm/rpm/031db795-0c68-450d-a0...", publication_href: "/pulp/api/v3/publications/rpm/rpm/bef78f50-6ede-42...", version_href: "/pulp/api/v3/repositories/rpm/rpm/7feda14a-1e2a-45...", last_contents_changed: "2022-04-16 11:01:44", last_applicability_regen: "2022-04-16 11:07:43", last_indexed: "2022-04-21 11:22:43">, 
#<Katello::Repository id: 6, pulp_id: "bd722086-e668-492c-b7bc-f3dea81dd8ee", library_instance_id: nil, content_view_version_id: 1, relative_path: "Company/Library/custom/Centos7/os_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: "7", distribution_arch: "x86_64", distribution_bootable: true, distribution_family: "CentOS", distribution_variant: "CentOS", container_repository_name: nil, root_id: 6, remote_href: "/pulp/api/v3/remotes/rpm/rpm/1d982e1c-9bde-4747-a4...", publication_href: "/pulp/api/v3/publications/rpm/rpm/467e12c0-5782-4c...", version_href: "/pulp/api/v3/repositories/rpm/rpm/ec8b8ab5-e2fe-42...", last_contents_changed: "2022-04-21 11:15:30", last_applicability_regen: "2022-04-21 11:25:24", last_indexed: "2022-04-21 11:23:37">, 
#<Katello::Repository id: 11, pulp_id: "4d9b2f74-f050-447a-acbe-402c84825b14", library_instance_id: nil, content_view_version_id: 1, relative_path: "Company/Library/custom/Centos7/centos-7-errata", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 11, remote_href: "/pulp/api/v3/remotes/rpm/rpm/c8be6a59-657a-4271-ac...", publication_href: "/pulp/api/v3/publications/rpm/rpm/e63ca2a4-12de-4c...", version_href: "/pulp/api/v3/repositories/rpm/rpm/84e3db6e-07b2-4c...", last_contents_changed: "2022-04-15 05:00:26", last_applicability_regen: "2022-04-15 05:01:08", last_indexed: "2022-04-21 11:01:01">]
irb(main):008:0>

# hammer repository list
11 | centos-7-errata        | Centos7 | yum          |

I also checked the other working hosts which have the centos errata working and they too have the repo 11 listed as bound.

So how can the bound repositories be fixed?

iballou · April 25, 2022, 6:40pm

@George,

Do that hosts that are missing errata have any special “repository sets” configuration? If you never touched that I’ll guess not.

In Katello, the bound repositories are reported by the host via a “profile upload”. Assuming my question above is false, you can trigger the bound repositories update by running katello-package-upload -f on EL7 hosts. Installing, removing, and updating packages will also cause the profile to be uploaded. There are likely other yum actions that trigger the upload as well.

It sounds like something may have changed regarding the bound repositories update recently in Katello. I’ll make an issue to investigate it.

George · April 26, 2022, 7:16am

No. All centos7 host should be identical on repo sets.

We don’t have katello-agents installed on our hosts since it’s deprecated.

But yes thanks for the help. It helped a lot finding the cause for this issue as you were correct on your analysis. Running subscription-manager repos updated the bound repos and now the hosts show errata properly. This sort of makes sense as the errata repo was added later to the repository set and the working hosts had apparently updated their repos from subscription-manager whilst the not working hosts had not updated.

George · April 26, 2022, 7:18am

Run subscription-manager repos on the host. This worked for me.

iballou · April 26, 2022, 1:40pm

Great to hear @George. You mentioned that on older Katello (4.1 I think?) this didn’t seem to be an issue. Knowing now that the profile upload was needed on some hosts, do you still think that is the case? We’ll be investigating to ensure there isn’t a regression.

George · April 26, 2022, 1:45pm

I don’t think that the katello version had anything to do with it. I think on the older katello version i had made the product for centos7 with the errata before registering any centos7 hosts to foreman which would explain why it worked on the older version as it bound the correct repositories from the start.