A few questions:
Did you update your Katello with the foreman-installer --scenario katello --upgrade command? I’m curious if all of the database migrations have run.
Are you correctly seeing applicable packages for the hosts that seem okay?
Are you able to try deleting + un-registering one of your problematic hosts and re-registering it?
I snuck in my last comment before you replied, but let us know if you’re able to share your whole production.log.
I did not run the installer. It was a fresh 3.15 installation before. Do I have to run the installer again after simple rpm updates? I thought that is only necessary for version upgrades, e.g. from 3.15 to 3.16.
I see the applicable packages count if it’s a package that is shown in the installed packages list. Just tested with sqlite-libs which is one of the packages listed on the content host which has a incomplete list. Downgraded and it shows 1 applicable update. Upgraded and it’s back to 0.
So the count itself seems to be working now for those package it knows to be installed. But for those it’s missing in the list of installed packages it’s not working.
I unregistered and deleted a host (with an available update which was not shown in the applicable packages) from katello and then registered again. Now it correctly shows an applicable update and clicking on the number even shows that correct package. However, the list of installed packages is still 77 and the package with the update is still not part of that list.
I can provide you the production.log. What’s the best and secure way to share that?
The installer upgrade will perform things like DB migrations and config updates which is why we recommend it. Can you run foreman-rake db:migrate on your Katello server?
Let’s try that before you send over your production logs since I’m pretty confident that should fix it.
O.K. I can db:migrate:
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
== 20200429153103 InstalledPackageBadNvrea: migrating =========================
== 20200429153103 InstalledPackageBadNvrea: migrated (1.0905s) ================== 20200501155054 InstalledPackageUniqueNvrea: migrating ======================
– change_column(:katello_installed_packages, :nvrea, :string, {:null=>false})
-> 0.0085s
– add_index(“katello_installed_packages”, [:nvrea], {:unique=>true})
-> 0.0334s
– remove_index(“katello_installed_packages”, [:nvra])
-> 0.0064s
== 20200501155054 InstalledPackageUniqueNvrea: migrated (0.0973s) =============
Installed package list was still short after that. Running katello-package-upload -f on a centos 7 content host seems to fix that. For CentOS 8 (no katello-package-upload available) I have (re)installed or updated any package and that seems to update that either.
katello-package-upload -f still causes the exception:
2020-05-29T06:29:04 [E|bac|] invalid argument: nil. (ArgumentError)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/spawn_methods.rb:37:inmerge' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:46:inupload_modules_to_pulp’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:70:inimport_module_streams' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:114:inblock in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:113:ineach' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:113:inrun’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:563:in `block (3 levels) in execute_run’
…
Now it looks good for all content hosts and no updates pending. I’ll see what happens when the next updates appear in the repositories. At the moment it looks good.
Something is still off. Tonight there came a few foreman updates tfm-rubygem-multi_json.noarch, tfm-runtime.x86_64 and some others. Published the content view. yum check-update on the content host shows available updates.
In Katello, though, it shows 0 applicable updates. katello-package-upload -f doesn’t help either. Even a manual downgrade of an installed package doesn’t show up anymore.
I guess it’s related to my other thread Synchronize repository task hanging
There is a sync task hanging for 2.5h now. I guess if I restart the katello service now it might solve the counters…
Was there any progress after restarting Katello? I wouldn’t expect the hanging sync task to have an effect on applicability.
Do you have any failed tasks that are stopped or paused relating to applicability or package profile uploads? From the traceback you sent I see the UploadProfiles task is failing (which I would expect to also show up in the tasks browser), which might explain applicability failing if the profile isn’t reaching Pulp 2 properly.
I’ll look into that UploadProfiles error.
On the centos 8 box where you run this command can you share us your
/var/lib/rhsm/cache/profile.json
Also send us the output of rpm -qa|grep sub
I didn’t restart before since the hanging sync task. I just did.
I can’t say for sure if it’s really related but the timeline fits. It worked fine. When I returned two hours later it did. In those two hours the hanging sync task started.
Now even after the katello restart it still won’t show applicable updates, even after katello-package-upload -f.
In the task list I see Combined Profile Update tasks ended stopped in warning whenever I did that package upload, i.e. action “Actions::Katello::Host::UploadProfiles”, exception “ArgumentError: invalid argument: nil.”. That seems to be each time I run katello-package-upload.
Other than that all tasks stopped successfully except the sync plan, of course, to which that hanging sync belongs to.
There is no katello-package-upload on CentOS 8. At the moment I only have CentOS 7 hosts which have issues. I downgraded a package on a CentOS 8 host and that showed up as applicable. Did the same on a CentOS 7 and that did not show any applicables and threw the exception and showed that warning stopped task.
But this time only a CentOS 7 sync task was hanging unlike before when the EPEL8 sync was also hanging…
On CentOS 7 host:
subscription-manager-rhsm-certificates-1.24.26-3.el7.centos.x86_64
subscription-manager-rhsm-1.24.26-3.el7.centos.x86_64
subscription-manager-1.24.26-3.el7.centos.x86_64
So one thing on rhel8/centos8 you would use
dnf uploadprofile instead of katello-packages-upload -f
Same thing: at the moment CentOS 8 seems to be fine. Now exception in the logs. The combined profile update tasks end in success state.
I have a PR ready that fixes the nil error which in turn will hopefully sort out your applicability issues. Would you like to try it out? It will also be available in Katello 3.15.2.
I have patched upload_profiles.rb. The task is successful, too.
However, it doesn’t affect the applicability count for the centos 7 content hosts. It is still at 0, even though there should be three. katello-package-upload -f doesn’t help. Downgrading a package doesn’t help, either. (Downgrading increases the count on CentOS 8 content hosts, though).
So far the patch doesn’t seem to make a difference on how it is working at the moment. The exception is gone, the count still doesn’t work…
Can we see your /var/lib/rhsm/cache/profile.json on one of the troublesome CentOS 7 clients?
What’s the best way to share that here?
You could upload the file directly to the forum in your reply (Upload button) or private message it to me. If you’d rather it not be on the forum you could email it to me at iballou@redhat.com
I have just sent you a PM with the file.
I downgraded sqlite, i.e. currently it shows an available update to 3.7.17-8.el7_7.1 with yum check-update. Katello shows no applicable updates, though.
On Content - Packages I can see for the 3.7.17-8.el7_7.1 version of sqlite the correct count of installed on hosts, but also 0 applicable & upgradable hosts.
I had the same issue. After upgrade the updates view didn’t show any package on a host. So after I read the thread I did a yum update on foreman and then a foreman-rake db:migrate
Now if I click on Hosts > Content hosts I got an empty page. No hosts is listed. I don’t see any particular error in the logs but I may just miss something.
I also did a apipie:cache after the update as suggested by the db:migrate output. Then I cleared all cookies and stored data in the browser to clear any remnants from the previous version. After that everything worked fine again.