Upgrade calculation for nginx module stream packages incorrect


I recently noticed that dnf shows me a different upgrade result than the host details UI,
only saw it for EL 8 systems so far, because I especially kept an eye on the nginx packages.
I think after investigation it has to do with a module stream metadata mismatch on the Rocky side, but shouldn’t Katello ignore the packages like dnf does (here the 1.20 stream is enabled)?

compared to dnf:

# dnf update
Updating Subscription Management repositories.
subscription-manager plugin disabled 3 system repositories with respect of configuration in /etc/dnf/plugins/subscription-manager.conf
Zabbix RHEL 8 - Non-Supported                                                                                                  37 kB/s | 1.5 kB     00:00    
Zabbix RHEL 8 - Supported                                                                                                      37 kB/s | 1.5 kB     00:00    
Foreman Client EL8                                                                                                             44 kB/s | 1.8 kB     00:00    
Extra Packages for Enterprise Linux 8                                                                                          49 kB/s | 2.3 kB     00:00    
Puppet EL 8                                                                                                                    37 kB/s | 1.5 kB     00:00    
Rocky Linux 8 - PowerTools                                                                                                     63 kB/s | 2.6 kB     00:00    
Rocky Linux 8 - AppStream                                                                                                      60 kB/s | 2.6 kB     00:00    
Rocky Linux 8 - BaseOS                                                                                                         54 kB/s | 2.3 kB     00:00    
Extra Packages for Enterprise Linux 8 Modular                                                                                  54 kB/s | 2.3 kB     00:00    
Dependencies resolved.
 Package                          Architecture            Version                                      Repository                                        Size
 libnghttp2                       x86_64                  1.33.0-5.el8_8                               LUMA_rockycdn8_rockycdn8_baseos                   77 k
 platform-python                  x86_64                  3.6.8-51.el8_8.2.rocky.0                     LUMA_rockycdn8_rockycdn8_baseos                   86 k
 python3-libs                     x86_64                  3.6.8-51.el8_8.2.rocky.0                     LUMA_rockycdn8_rockycdn8_baseos                  7.8 M

Transaction Summary
Upgrade  3 Packages

Total download size: 8.0 M
Is this ok [y/N]:

Which made the module metadata issue apparent to me was, if I disable the module at all I get some updates.
Or is Katello dealing with this a bit differently and only on the Rocky side the change has to be made?

Foreman and Proxy versions:

Foreman 3.8
Katello 4.10

Foreman and Proxy plugin versions:

Snapshot Management

Distribution and version:

Rocky Linux 8.8

Hey there @lumarel ,

This is a great question, we’ll need to investigate it!

Which version of nginx does your machine have installed on it?

1 Like

Hey @iballou,
thank you for taking a look :slightly_smiling_face:

It’s the nginx:1.20 stream this machine is using, but theoretically it should happen with any nginx stream, even 1.22, the newly built artifacts are just not part of the application stream metadata (which makes them regular packages)

1 Like

Btw the missing metadata already got added now, so hmm only for old syncs now (good generally bad for reproducibility) :slightly_smiling_face:

1 Like

(if needed I can also help with my old states, or prop up a dummy for reproduction)

1 Like

We should be able to reproduce this with our test repositories that we carry in the Katello codebase. It is odd though because we have logic checking if the installed RPMs exist in a module stream and to limit the search to that module stream if so. So I’m thinking, in this case, the modular RPMs might have been reported as “modular” by Pulp even though they existed in no module stream.

It’s not possible that the newer packages were added to the 1.20 stream by Rocky accidentally?


I made an issue: Bug #36879: Broken upstream modular RPMs reported as upgradable to - Katello - Foreman

1 Like

Tbh I’m currently trying to figure out where nginx 1.20.1-6.module_el8+12928+992082b2.x86_64 was coming from (the currently installed version), because Rocky never built that version, I suspect it was once part of the EPEL 8 application streams.
(=> oh yeah that’s really what happened, this version was coming from EPEL, glory to dnf history!)
But that’s also basically irrelevant (at least for dnf), the application stream config for dnf is set, so it also counts for the later appearing nginx:1.20 stream.
Only this exact package version/artifact is not present in the module metadata anymore, since EPEL removed that. Hrm maybe it’s that, the case, that the metadata for 1.20 got removed in the EPEL repo.

I’m having a pretty close eye to the Rocky module metadata somehow (something something qa), so pretty sure everything was in there until last 2 weeks where the new version was built, where the new metadata was missing at first and now got added :+1:

1 Like

(and yes there is also a new build of 1.20 now, which also got added to the metadata now, but as both 1.20 and 1.22 were not recognized as module packages, it wanted to upgrade to 1.22)

1 Like

Gotcha, so it really should be a missing metadata case rather than a “1.22 packages made it into 1.20” case. Your case about the EPEL module metadata would certainly attest to that.

1 Like

Yes I think so too :+1:

More interesting things happened!

my system synced the correct metadata now (aka. default library, no CVs), but the host already shows that there is an update

So, the current state is:

  • correct metadata in default library
  • incorrect metadata built for used CV (and promoted to all LEs)
  • packages available in default library and CV (all LEs)

Looks like update status gets compiled with the default library metadata instead of the one from the assigned CV :slightly_smiling_face:

And at the end, dnf doesn’t see that new metadata up to now, so doesn’t install the updates :+1:

1 Like

Without doing a self-refresher on applicability that makes sense logically to me – the applicable content should be based on what’s in Library / Default Content View because, if it’s needed but not in the CVV, you can pull it in via incremental update or a new CV publish.

So it looks like in your photo above the applicability is correct now? Since it’s 1.20.1 → 1.20.1 instead of 1.20.1 → impossible newer version. Seems those packages are actually all in the same module stream now.

And then dnf doesn’t pick it up because the packages are applicable but not installable.

So, rambles aside, it sounds like with the correct metadata in Rocky everything is looking better?

We’ve put that issue on our backlog, hopefully we can get to it soon.

Took me a moment as well right now :sweat_smile: (actually multiple moments… sorry I didn’t want to scratch all of it again, because there is valuable information…)

(basically tl;dr;) I think the key is, applicable (errata/package available in the top library) vs. installable/upgradable (errata/package available directly to the host by the assigned CV), as far as I understood it up to now the packages (upgradable filter) and errata tabs in the new host ui show installable/upgradable. (and beside this metadata thing that broke out of that scheme it also is correct on my systems to that assumption)
So, as dnf only know the end result, it also only shows installable/upgradable.

Examples, that show the behavior (uh... found another thing, but minor)

i.e. cloud-utils-0.33-3.el8 here:

it’s only applicable up to now, so it doesn’t show up on the hosts package and errata page:

but looking at the old content host page it shows up if I select the other LE:

just only noticed that this feature is missing in the new host ui :slightly_smiling_face:

and i.e. for packages overall:

Unfortunately not, at the time I shot that picture, the packages only were applicable, not upgradable. (and dnf only saw the built metadata so could only show the correct state → not having nginx update)

Yes now as everything promoted through Rocky’s and my staging everything is correct and (seemingly) flawless again :slight_smile:

But okay, once again thank you (and sorry) for taking the time!

1 Like

Good catch there, I totally hadn’t realized that yet either! @jeremylenz you might be interested in what was found above, we used to be able to filter a host’s applicable errata by different lifecycle environments.

Interesting, so the installability of a package it determined by the repositories that are “bound” to the host. If a package is applicable to a host but not in the bound repositories, then it’s just applicable. If the package is in the bound repositories, then it’s installable. That means then that those nginx packages were in your host’s bound repositories somehow.

If this pops up again, you can try this:

# foreman-rake console
::Host.find(<host id here>).content_facet.bound_repositories

That will return what repositories Katello thinks your host is consuming from. If it’s out of date, a subscription-manager repos call will update them. We also have a new feature where you can trigger a remote execution job to send over the new bound repositories from the host details page. I can’t remember if that made it into Katello 4.10…

1 Like

For errata, if the host is assigned to a non-default content view and/or lifecycle environment, the new host UI will have a toggle with two options. “Applicable” is equivalant to “Library Synced Content”, while “Installable” will show the installable errata in the host’s environment(s). My memory isn’t perfect, but if I had to guess I’d say we probably decided that displaying errata based on other environments that are not the host’s, isn’t relevant to that host, so isn’t needed on that page.

1 Like

Ah yeah that totally makes sense, thank you for the in depth look :slight_smile:

Okay I will see if it shows up once again, thanks as well! :+1: