Ah! Thanks for letting me know, I wasn’t seeing where this bug would pop up under normal circumstances.
I will ask the Pulp team about this in case they have any ideas.
It’s really odd. Now for days, it doesn’t show the applicable updates for that one particular host. I can’t tell under which circumstances the problem would “move” to another host…
@iballou Do you have any feedback from the pulp team?
@gvde I’m getting back to them now. In the meantime, can you re-do the steps I wrote here for your problematic host and show me the output? Katello 3.15 doesn't show available package updates anymore
I want to double check that the host’s Pulp consumer is properly subscribed to your repositories.
Output:
irb(main):001:0> uuid = ::Host.find_by(name: "host.example.com").content_facet.uuid
=> "93cb9847-57d7-41b6-a6cc-83c145fbd2a2"
irb(main):002:0> ::Katello::Repository.where(pulp_id: ::Katello::Pulp::Consumer.new(uuid).bound_yum_repositories)
=> #<ActiveRecord::Relation [#<Katello::Repository id: 2, pulp_id: "d4b5ef45-b48a-4d09-bccd-1159f448b916", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/centos7/updates_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 2, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 7, pulp_id: "ff8674d5-eed1-4476-b103-1f31d9d2cc6b", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/epel7/epel_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 7, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 1, pulp_id: "70cf2996-6320-4fcd-a9c5-eda568957879", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/centos7/base_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: "7", distribution_arch: "x86_64", distribution_bootable: true, distribution_family: "CentOS", distribution_variant: "", distribution_uuid: "b71ebdf3-9070-48bd-971c-254bd1635439", container_repository_name: nil, root_id: 1, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 36, pulp_id: "f6303b53-8f37-412c-b8d9-d8da561dd61a", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/foreman-client/2_0_el7_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 8, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 4052, pulp_id: "d2e6c54c-d7c9-4f12-ab75-9393bdcbf2b8", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/opennms/opennms-repo-stable-co...", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 62, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 56, pulp_id: "59a32c7e-f8f4-4f05-8cd8-2a982dede276", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/puppet/puppet6_el7_x86_64", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 10, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 4053, pulp_id: "a15137e5-0716-4df1-82e1-3004c3919367", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/opennms/opennms-repo-stable-rh...", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 63, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 77, pulp_id: "2b3a6d0e-a8d4-4c82-9681-c695d3003ac0", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/postgresql/pgdg-common-el7", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 11, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 80, pulp_id: "6ff7e34e-29ac-43a8-9371-87da5dbe1b96", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/postgresql/pgdg11-el7", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 14, remote_href: nil, publication_href: nil, version_href: nil>, #<Katello::Repository id: 4884, pulp_id: "bf5d1f89-364d-4938-87b7-bbc2b0992afe", library_instance_id: nil, content_view_version_id: 1, relative_path: "ORG/Library/custom/dkrz/el7", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, container_repository_name: nil, root_id: 65, remote_href: nil, publication_href: nil, version_href: nil>]>
irb(main):003:0> ::Katello::Pulp::Consumer.new(uuid).applicable_ids('rpm')
=> []
The host has 5 updates pending according to yum check-updates:
selinux-policy.noarch 3.13.1-266.el7_8.1 ORG_centos7_updates_x86_64
selinux-policy-targeted.noarch 3.13.1-266.el7_8.1 ORG_centos7_updates_x86_64
systemd.x86_64 219-73.el7_8.8 ORG_centos7_updates_x86_64
systemd-libs.x86_64 219-73.el7_8.8 ORG_centos7_updates_x86_64
systemd-sysv.x86_64 219-73.el7_8.8 ORG_centos7_updates_x86_64
::Katello.pulp_server.resources.consumer.retrieve_profile(uuid, "rpm") correctly lists the currently installed versions:
{"vendor"=>"CentOS", "name"=>"selinux-policy", "epoch"=>0, "version"=>"3.13.1", "release"=>"266.el7", "arch"=>"noarch"},
{"vendor"=>"CentOS", "name"=>"selinux-policy-targeted", "epoch"=>0, "version"=>"3.13.1", "release"=>"266.el7", "arch"=>"noarch"},
{"vendor"=>"CentOS", "name"=>"systemd", "epoch"=>0, "version"=>"219", "release"=>"73.el7_8.6", "arch"=>"x86_64"},
{"vendor"=>"CentOS", "name"=>"systemd-libs", "epoch"=>0, "version"=>"219", "release"=>"73.el7_8.6", "arch"=>"x86_64"},
{"vendor"=>"CentOS", "name"=>"systemd-sysv", "epoch"=>0, "version"=>"219", "release"=>"73.el7_8.6", "arch"=>"x86_64"},Thanks. So the bound repositories made it to the Pulp consumer, which means the Katello side of things are likely functioning correctly.
I created an issue on the Pulp bug tracker. Pulp 2.21.3 has reached its dev freeze, the list of bugs that will be included are here: https://pulp.plan.io/issues?query_id=59&sort=cf_5%2Cpriority%3Adesc%2Cstatus
I do see one bug that might be related: https://pulp.plan.io/issues/6724
This line may interest us: “10th and 20th consumers will always get 0 applicability because the calculations are skipped.”
The related PR is small, perhaps those of you having this problem (@gvde and @ppascher) can try patching it in. PR: https://github.com/pulp/pulp/pull/3986
Pulp PR 3986 doesn’t seem to make a difference for me. Updated the file. Restarted all services. Ran ForemanTasks.sync_task(::Actions::Katello::Host::GenerateApplicability, Host.all), ran katello-package-upload -f, ran GenerateApplicability again. No change. That one host still doesn’t show any applicable updates.
O.K. Tonight there came some applicable updates for that particular host in a different repository then the one with the missing applicables. This morning it shows 5 applicable updates (the missing ones) and in the Content - Packages list I can see that those 5 are applicable & upgradable and the 3 new ones are applicable (as they are not yet in the content view for that host).
So basically, those new updates in another repository of that host fixed the applicable count for that host. If I remember correctly, those were the first updates for that host since the problem occurred.
At the moment, all other hosts don’t have any updates pending thus I cannot tell, if the issue moved to another host now…
Ok, let us know if you see any more of the missing applicability behavior. For anyone following along, the original information for the single-host-only applicability problem is as follows:
Sorry for the late reply.
For me it was a CentOS8 Host which was currently affected. Bound_yum_repositories looked OK but applicable IDs were empty .
I installed Pulp 2.21.3 yesterday and today the host is showing applicable updates again. I will have to check when there are new updates available again whether the problem only moved to a different host or if it is solved now.
Thanks for the reply, let us know if you see the issues again.
So, I’m seeing a more minor issue, after having updated to the latest Katello 3.15 RPMs, bringing us to 3.15.3, where hosts show as having applicable updates, when yum tells us they don’t…
and
vs.
$ sudo yum check-update
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
Our_Org_CentOS_CentOS-7-x86_64-extras | 2.1 kB 00:00:00
Our_Org_CentOS_CentOS-7-x86_64-os | 2.1 kB 00:00:00
Our_Org_CentOS_CentOS-7-x86_64-scl-rh | 2.1 kB 00:00:00
Our_Org_CentOS_CentOS-7-x86_64-scl-sclo | 2.1 kB 00:00:00
Our_Org_CentOS_CentOS-7-x86_64-updates | 2.1 kB 00:00:00
Our_Org_CentOS_EPEL-7-x86_64 | 2.8 kB 00:00:00
Our_Org_CentOS_Elasticsearch-7x | 2.1 kB 00:00:00
Our_Org_CentOS_Foreman-Client-2_0-EL7-x86_64 | 2.1 kB 00:00:00
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
$ rpm -q dbus dbus-libs
dbus-1.10.24-13.el7_6.x86_64
dbus-libs-1.10.24-13.el7_6.x86_64
The above even after we’ve done “sudo rm -fr /var/cache/yum/x86_64”
In this particular case, it seems like the installed RPM versions come from the “CentOS-7-x86_64-os” repository, and the applicable RPM versions come from “CentOS-7-x86_64-updates” but don’t end up being offered to the host.
Indeed the repodata that yum has downloaded looks to be stale:
$ ls -l /var/cache/yum/x86_64/7/Our_Org_CentOS_CentOS-7-x86_64-updates
total 1.7M
-rw-r--r--. 1 root root 124 Jul 1 00:08 a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
-rw-r--r--. 1 root root 1.7M Jul 1 00:08 aa4b4b1638db5db3bcecee48c86ab77db9be96c87d4a9f30f7edeb9f67d21270-primary.xml.gz
-rw-r--r--. 1 root root 92 Jul 1 00:08 b398b289c739f2f2ed3956981eeffcbb524fc6e226f03196f2d87007da15f932-updateinfo.xml.gz
-rw-r--r--. 1 root root 0 Jul 21 16:41 cachecookie
drwxr-xr-x. 2 root root 70 Jul 21 16:39 gen/
drwxr-xr-x. 2 root root 6 Jul 21 16:39 packages/
-rw-r--r--. 1 root root 2.2K Jul 6 13:22 repomd.xmlI have not followed the complete thread, but just to ensure how is your setting for the comparison? You can change the calculation from Library to Content View in the settings (can not remember setting name and would require to boot my demo system, but it is on the Content tab). In most environments I changed the default as comparing with Library gets you this confusing result.
