Installable errata suddenly shows up in hosts content view, but not installable via content view

[caseybea]

Katello 4.3.1-1, Foreman 3.1.2.

In my content hosts view, I suddenly have an “installable errata” item show up for my Rocky8 hosts.

This is confusing, because this update is NOT installable in my content view. My current content view is almost a month old. So my content hosts view should be unchanged.

Katello says it’s installable, but when I do a yum-check or yum update on my client hosts, of course nothing is updated-- because I have not yet published a new content view.

Did something uh, “leak” through somehow? I have enclosed relevant screenshots.

[iballou]

Hi @caseybea,

Does this show all of the hosts that see the errata issue?

 ::Katello::Host::ContentFacet.joins_installable_errata.where("katello_errata.errata_id = 'RLSA-2021:3666'")

Also, if you pick one of the hosts that show this wrong errata, and run the following, do all of the linked (“bound”) repositories make sense? These are the repositories that Katello thinks your hosts are consuming from:

::Host.find(<host id>).content_facet.bound_repositories

[caseybea]

The first command shows 10 hosts (but in my content hosts view there are more than that). But maybe there’s more and it can’t list them? The output ends with “…]>”

The repositories listed make sense.

Example host (of the 10)

#<Katello::Host::ContentFacet id: 9, host_id: 10, uuid: “7bbbd0a0-3247-4d1c-ae3c-6d0fac640424”, content_view_id: 3, lifecycle_environment_id: 3, kickstart_repository_id: nil, content_source_id: nil, installable_security_errata_count: 1, installable_enhancement_errata_count: 0, installable_bugfix_errata_count: 0, applicable_rpm_count: 311, upgradable_rpm_count: 0, applicable_module_stream_count: 2, upgradable_module_stream_count: 0, applicable_deb_count: 0, upgradable_deb_count: 0>,

Example repo (of the ones listed which look correct)

#<Katello::Repository id: 54, pulp_id: “1-Rocky8-Production-c45cf342-b2dc-427c-89c5-02d0ff…”, library_instance_id: 8, content_view_version_id: 32, relative_path: “CTSI/Production/Rocky8/custom/Rocky8/rocky8_appstr…”, environment_id: 3, saved_checksum_type: nil, distribution_version: “8”, distribution_arch: “x86_64”, distribution_bootable: false, distribution_family: “Rocky Linux”, distribution_variant: “AppStream”, container_repository_name: nil, root_id: 8, remote_href: nil, publication_href: “/pulp/api/v3/publications/rpm/rpm/32d4dd3c-f5a6-43…”, version_href: “/pulp/api/v3/repositories/rpm/rpm/1db9644e-4c09-42…”, last_contents_changed: “2021-11-16 17:19:12”, last_applicability_regen: “1970-01-01 00:00:00”, last_indexed: “1970-01-01 00:00:00”>,

[iballou]

Thanks for the info. Is this erratum one that should be applicable, but not installable? Or should it not be applicable at all to these hosts?

To clarify, an erratum should be applicable if its packages are also applicable.

[caseybea]

As far as I can tell, these are APPLICABLE; the following is the package list from the errata. All of the nodes have c-ares so that package makes the hosts applicable.

But somehow they’re showing up as installable, which per above makes no sense until I create a current content view.

c-ares-1.13.0-6.el8.i686
c-ares-1.13.0-6.el8.x86_64
c-ares-devel-1.13.0-6.el8.i686
c-ares-devel-1.13.0-6.el8.x86_64
nodejs-12.22.5-1.module+el8.4.0+647+e905fa21.x86_64
nodejs-14.17.5-1.module+el8.4.0+654+17ff1d38.x86_64
nodejs-devel-12.22.5-1.module+el8.4.0+647+e905fa21.x86_64
nodejs-devel-14.17.5-1.module+el8.4.0+654+17ff1d38.x86_64
nodejs-docs-12.22.5-1.module+el8.4.0+647+e905fa21.noarch
nodejs-docs-14.17.5-1.module+el8.4.0+654+17ff1d38.noarch
nodejs-full-i18n-12.22.5-1.module+el8.4.0+647+e905fa21.x86_64
nodejs-full-i18n-14.17.5-1.module+el8.4.0+654+17ff1d38.x86_64
npm-6.14.14-1.12.22.5.1.module+el8.4.0+647+e905fa21.x86_64
npm-6.14.14-1.14.17.5.1.module+el8.4.0+654+17ff1d38.x86_64

[iballou]

To other @katello devs, this sounds like it’s not an applicability problem necessarily. Rather it seems like an installable vs applicable errata query problem (which is interesting because that code is pretty old and solid). Source: katello/pulp_database_unit.rb at master · Katello/katello · GitHub

@caseybea, does the following code also return the bad erratum for one of the hosts?

::Katello::Erratum.installable_for_content_facet(::Host.find(<id>), ::Host.find(<id>).lifecycle_environment, ::Host.find(<id>).content_view)

Also, ::Host.find(<id>).lifecycle_environment and ::Host.find(<id>).content_view returns what you’d expect?

[iballou]

Also @caseybea, have you tried searching your content view version for RLSA-2021:3666? There aren’t many ways for the code there to go wrong as far as I can tell. I wonder if somehow that errata is associated with one of the repositories in your hosts’ bound_repositories unexpectedly.

[caseybea]

Yes, these come up as you indicate. The environments are correct. We have a pretty simple setup… Library–>Developent–>Produciton

irb(main):001:0> ::Katello::Erratum.installable_for_content_facet(::Host.find(42), ::Host.find(42).lifecycle_environment, ::Host.find(42).content_view)
=> #<ActiveRecord::AssociationRelation [#<Katello::Erratum id: 16418, pulp_id: “RLSA-2021:3666”, errata_id: “RLSA-2021:3666”, created_at: “2021-11-17 14:02:58”, updated_at: “2021-11-17 14:02:58”, issued: “2021-10-01”, updated: “2021-09-27”, errata_type: “security”, severity: “Important”, title: “Important: nodejs:14 security and bug fix update”, solution: “”, description: “For more information visit https://errata.rockylin…”, summary: “An update for the nodejs:14 module is now availabl…”, reboot_suggested: false, migrated_pulp3_href: nil>]>

irb(main):002:0> ::Host.find(42).lifecycle_environment and ::Host.find(42).content_view
=> #<Katello::ContentView id: 3, name: “Rocky8”, label: “Rocky8”, description: “Rocky Linux 8”, organization_id: 1, default: false, created_at: “2021-11-15 17:04:34”, updated_at: “2022-05-06 16:59:04”, composite: false, next_version: 11, force_puppet_environment: false, auto_publish: false, solve_dependencies: false, import_only: false>

[caseybea]

@iballou

RLSA-2021:3666

Searching the content view does come up with the errata.

[caseybea]

If you look at the errata detail,. there’s a bunch of packages that are straight up, but ALSO many (nodejs things) that are MODULES.

Although I know this is a totally different case and section of the code, this feels somewat like a larger issue we all once had with EL8 modules altogether, many of us were having package updates “show up and installable” that well, just couldn’t be installed because of the module “path” in plpace for the client (hard to describe).

Same behavior here, but now it’s in the errata section and not the package updates portion. Hope I’m explaining that properly.

[iballou]

Okay, so the erratum being in the content view version partially explains it. In that case it doesn’t sound like an applicability issue, rather a CV one. Next up we’d need to figure out why the erratum is in your CV.

Is this an erratum that should have explicitly been filtered out? Do any of the erratum’s RPMs exist in the content view? Or, are any related modules streams being marked as “upgradable”?

Our copying logic should exclude an erratum if it doesn’t have all of its RPMs (unless the missing RPMs don’t even exist in the Library repositories).

It is strange that you’re reporting that this occurred suddenly, as in the erratum suddenly showed up in the content view version. Were there any CV-related actions performed around the same time that the new erratum popped up?

@caseybea

Although I know this is a totally different case and section of the code, this feels somewat like a larger issue we all once had with EL8 modules altogether, many of us were having package updates “show up and installable” that well, just couldn’t be installed because of the module “path” in plpace for the client (hard to describe).

If they’re modular RPMs, then they should only be marked as applicable if there’s an update available within the module stream that’s active. If that’s not the case (which it sounds like not), then yeah it’s probably a scenario that Katello couldn’t know about. Modular applicability is tricky.

[caseybea]

So of the entire package list (it’s listed above) for the errata, I checked the CONTENT VIEWS that were published 25 days ago. Shortly after I had updated those servers on patch night, everything was clean/zero.

Then a week or so later, these errata updates showed up.

Anyway, in checking the content views: The standard package RPM’s (c-ares-xxxxx) are not there. But all the rest (which are MODULE updates for nodejs, npm) are there in the package list for all of the content views (development, production, for Rocky 8).

Hope that makes sense.

In about a week, per my monthly schedule, I will be creating/promoting all new content views. (development gets everything current… production gets what development had previously).

I suggest it might be worth waiting to see what the envirionemt looks like after I publish those. I am not in a rush, as this issue isn’t breaking anything per se, it’s jjust telling me there’s stuff to update that isn’t actually applicable.

[iballou]

Hey @caseybea,

That sounds good, I’d like to hear how things look after your next batch of CV publishes.

No module streams are marked as upgradable in Katello on the content host’s module stream page?

[iballou]

I reproduced the issue and I think I know what’s going on.

Rocky Linux is doing something a little strange and is including modular RPMs in the list of normal RPMs. On top of that, they included this RLSA-2021:3666 erratum that we singled out in both BaseOS and AppStream. Your content view filtered out the c-ares RPMs from BaseOS, so the erratum didn’t make it into there. However, the erratum did make it into AppStream because the c-ares RPMs don’t exist there at all.

Katello marked the erratum as applicable because the c-ares RPM is indeed applicable, and then it marked the erratum as installable because the erratum exists in your content view version.

Essentially it boils down to the relationship between AppStream and BaseOS, plus Rocky Linux seemingly doing something unexpected.

I think there is something here to be improved, regardless of whether or not Rocky Linux is doing something weird (I’m not 100% sure). Either we consider all of a CV’s repositories when deciding if an erratum should be thrown out (katello/yum.rb at KATELLO-4.3 · Katello/katello · GitHub) or we improve the “installable errata” query to not just look at which errata are in the host’s LCE. We’d also need to check if the errata’s RPMs are all available. I’ll make an issue.

Anyway, good catch @caseybea !

[caseybea]

HA! I figured it was something along those lines - thoigh I’m not a developer so I don’t posess your coding skills.

I am however, good at noticing weird behavior, so there’s that.

Thank you so much for filing the issue @iballou , it’s really appreciated. You guys rock!

[iballou]

Here’s the issue: Bug #35016: Not-installable errata can appear as installable - Katello - Foreman

No problem, we really appreciate the feedback!

[lumarel]

To add on here, @iballou the weird behavior you are talking about in some of the Rocky Errata, is a long known flaw of the current Errata Generator over there, which is planned to be fixed in the next version of that, which will hopefully fix all (all known) the issues for the future ^^
(it’s basically that it currently doesn’t handle Module package Errata that well)
Thank you for looking into that here as well, I also reported it now another time, and there is also a tracking issue now there
Hopefully such weirdness will be gone as soon as Peridot is in production.

[iballou]

I’m happy to hear my suspicions are confirmed! Thanks for that @lumarel. Module streams & the EL8 repositories are such a tricky subject that I’m often left wondering if a bug is due to the repo or Katello. Seems today it’s a little of both.

[caseybea]

Side note/followup:

So 2 days from now is patch day, but I usually get a head start patching and rebooting SOME servers ahead of time when I can. Yesterday I published new content views, promoted everything to the next level, and did some updates.

Servers running Rocky 8 that are “development” severs (ie, they get the latest updates of everything), no longer show the uninstallable errata. (More specifically, those servers had a “dnf update” performed after the content view and promotion).

I believe that’s because the c-ares packages (part of the affected errata) made it into the mix. Servers classified as “production” (ie, the content view is now 1 month old, yesterday’s development hosts…), still have the issue.

So, eventualy (that is, next month), this example of “uninstallable errata” in my situation will resolve itself because all of the packages will actually BE in the content views I have.

That does not mean there isn’t an issue-- there is— but that has been identified, reproduced, and fix(es) added to a future version of Foreman/Katello as outlined above ^^^.

But I did promise an update after I published a new content view set.

-kcb