Katello 3.15 doesn't show available package updates anymore

Problem:
I have pretty new Katello 3.15 installation on CentOS 7. Recently it doesn’t show me any available/installable updates on the content host page anymore. “yum check-update” on the host itself shows the available update via katello, so it has been published into the content view.

This issue seems to coincide with a stuck (for >20h) sync of two repositories after which I have restarted the whole service. After the restart, I manually started the sync plan and that went through just fine for all repositories.

Expected outcome:
Show the correct count of available/installable updates.

Foreman and Proxy versions:

Foreman 2.0.0

Foreman and Proxy plugin versions:

Katello 3.15

Distribution and version:

CentOS 7.8.2003

Hi gvde,

This is a known issue and relates to an incompatibility with the latest Pulp 2 version. This will be fixed if you update to Katello 3.15.1.1

2 Likes

I have trouble finding the issue for this problem. Do you know the issue #?

Here’s the issue: Bug #29488: Applicable errata not showing up for module stream errata - Katello - Foreman

Here’s the commit: https://github.com/Katello/katello/commit/90201afa98b5c5e9bc0d6a741bbef11ae10f00b1

1 Like

I saw that one but thought it wasn’t applicable for CentOS 7. My contents are CentOS 7 repositories and others which don’t have errata nor support modules…

The change there targets modules but actually affects all types of Yum content. The bug you’re seeing is the result of Katello not sending the modules in the package profile to Pulp correctly (even if you have none), which causes applicability to break in Pulp. The change above that’s in 3.15.1.1 fixes how we’re sending the modules to Pulp in response to this.

I have updated to katello-3.15.1-1.el7.noarch. So far, it doesn’t show available updates. Even worse: when I look to the list of installed packages for a CentOS 7 content host, I only see a few packages. Only 77. Far from the correct number. The package which can be updated is missing in the list of installed packages.

I have tried a “katello-package-upload -f” on the content host. Doesn’t make a difference.

There are some error messages in the logs:

2020-05-28T19:12:49 [W|app|] Mismatch found in installed package insertion, expected 494 but only could find 0. This is most likley a bug.
2020-05-28T19:12:49 [W|app|] System client.example.com (15) requested binding to unknown repo /pulp/repos/ORG/Library/Standard_CentOS_7_Server/custom/centos7/base_x86_64
2020-05-28T19:12:49 [W|app|] System client.example.com (15) requested binding to unknown repo /pulp/repos/ORG/Library/Standard_CentOS_7_Server/custom/centos7/updates_x86_64
2020-05-28T19:12:49 [W|app|] System client.example.com (15) requested binding to unknown repo > /pulp/repos/ORG/Library/Standard_CentOS_7_Server/custom/puppet/puppet6_el7_x86_64
2020-05-28T19:12:49 [W|app|] System client.example.com (15) requested binding to unknown repo /pulp/repos/ORG/Library/Standard_CentOS_7_Server/custom/foreman-client/2_0_el7_x86_64
2020-05-28T19:12:49 [W|app|] System client.example.com (15) requested binding to unknown repo /pulp/repos/ORG/Library/Standard_CentOS_7_Server/custom/epel7/epel_x86_64
2020-05-28T19:12:49 [E|bac|] invalid argument: nil. (ArgumentError)

@gvde have you seen the those “unknown repo” errors before your update to 3.15.1.1?

Also looking into the “Mismatch” error, that is related to new changes you’ve installed but shouldn’t be occurring. Can we see your production.log?

The “unknown repo” was already there before the update. The Mismatch and ArgumentError messages are only since the update. They only appear for the content hosts which don’t show all installed packages in the list. Some other hosts seem fine, though.

A few questions:

Did you update your Katello with the foreman-installer --scenario katello --upgrade command? I’m curious if all of the database migrations have run.

Are you correctly seeing applicable packages for the hosts that seem okay?

Are you able to try deleting + un-registering one of your problematic hosts and re-registering it?

I snuck in my last comment before you replied, but let us know if you’re able to share your whole production.log.

I did not run the installer. It was a fresh 3.15 installation before. Do I have to run the installer again after simple rpm updates? I thought that is only necessary for version upgrades, e.g. from 3.15 to 3.16.

I see the applicable packages count if it’s a package that is shown in the installed packages list. Just tested with sqlite-libs which is one of the packages listed on the content host which has a incomplete list. Downgraded and it shows 1 applicable update. Upgraded and it’s back to 0.

So the count itself seems to be working now for those package it knows to be installed. But for those it’s missing in the list of installed packages it’s not working.

I unregistered and deleted a host (with an available update which was not shown in the applicable packages) from katello and then registered again. Now it correctly shows an applicable update and clicking on the number even shows that correct package. However, the list of installed packages is still 77 and the package with the update is still not part of that list.

I can provide you the production.log. What’s the best and secure way to share that?

The installer upgrade will perform things like DB migrations and config updates which is why we recommend it. Can you run foreman-rake db:migrate on your Katello server?

Let’s try that before you send over your production logs since I’m pretty confident that should fix it.

O.K. I can db:migrate:

API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
== 20200429153103 InstalledPackageBadNvrea: migrating =========================
== 20200429153103 InstalledPackageBadNvrea: migrated (1.0905s) ================

== 20200501155054 InstalledPackageUniqueNvrea: migrating ======================
– change_column(:katello_installed_packages, :nvrea, :string, {:null=>false})
-> 0.0085s
– add_index(“katello_installed_packages”, [:nvrea], {:unique=>true})
-> 0.0334s
– remove_index(“katello_installed_packages”, [:nvra])
-> 0.0064s
== 20200501155054 InstalledPackageUniqueNvrea: migrated (0.0973s) =============

Installed package list was still short after that. Running katello-package-upload -f on a centos 7 content host seems to fix that. For CentOS 8 (no katello-package-upload available) I have (re)installed or updated any package and that seems to update that either.

katello-package-upload -f still causes the exception:

2020-05-29T06:29:04 [E|bac|] invalid argument: nil. (ArgumentError)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/spawn_methods.rb:37:in merge' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:46:in upload_modules_to_pulp’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:70:in import_module_streams' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:114:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:113:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:113:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:563:in `block (3 levels) in execute_run’

Now it looks good for all content hosts and no updates pending. I’ll see what happens when the next updates appear in the repositories. At the moment it looks good.

Something is still off. Tonight there came a few foreman updates tfm-rubygem-multi_json.noarch, tfm-runtime.x86_64 and some others. Published the content view. yum check-update on the content host shows available updates.

In Katello, though, it shows 0 applicable updates. katello-package-upload -f doesn’t help either. Even a manual downgrade of an installed package doesn’t show up anymore.

I guess it’s related to my other thread Synchronize repository task hanging

There is a sync task hanging for 2.5h now. I guess if I restart the katello service now it might solve the counters…

Was there any progress after restarting Katello? I wouldn’t expect the hanging sync task to have an effect on applicability.

Do you have any failed tasks that are stopped or paused relating to applicability or package profile uploads? From the traceback you sent I see the UploadProfiles task is failing (which I would expect to also show up in the tasks browser), which might explain applicability failing if the profile isn’t reaching Pulp 2 properly.

I’ll look into that UploadProfiles error.

On the centos 8 box where you run this command can you share us your
/var/lib/rhsm/cache/profile.json

Also send us the output of rpm -qa|grep sub

I didn’t restart before since the hanging sync task. I just did.

I can’t say for sure if it’s really related but the timeline fits. It worked fine. When I returned two hours later it did. In those two hours the hanging sync task started.

Now even after the katello restart it still won’t show applicable updates, even after katello-package-upload -f.

In the task list I see Combined Profile Update tasks ended stopped in warning whenever I did that package upload, i.e. action “Actions::Katello::Host::UploadProfiles”, exception “ArgumentError: invalid argument: nil.”. That seems to be each time I run katello-package-upload.

Other than that all tasks stopped successfully except the sync plan, of course, to which that hanging sync belongs to.

There is no katello-package-upload on CentOS 8. At the moment I only have CentOS 7 hosts which have issues. I downgraded a package on a CentOS 8 host and that showed up as applicable. Did the same on a CentOS 7 and that did not show any applicables and threw the exception and showed that warning stopped task.

But this time only a CentOS 7 sync task was hanging unlike before when the EPEL8 sync was also hanging…

On CentOS 7 host:

subscription-manager-rhsm-certificates-1.24.26-3.el7.centos.x86_64
subscription-manager-rhsm-1.24.26-3.el7.centos.x86_64
subscription-manager-1.24.26-3.el7.centos.x86_64