New synced/published/promoted packages are not discovered in the content host

Support
1

Problem:

Hi,

Yesterday I successfully synced my Ceph Nautilus product. Packages for version 14.2.9 have been added.
Also, for both the product content view as well the composite view a new version has been successfully published and promoted. The number of packages is increased based on these new Ceph-packages and they are well listed in the package list.

But, for the content-hosts linked with this repo (Activation key), the new package is not discovered.

When I search for these package updates on the host, the are not listed, even not after a subscription-manager refresh.

However, after I cleaned the yum-cache (centos 7) on the host, the updates are then available, but still not in Foreman / content hosts.

Expected outcome:

Available package updates should be mentioned/listed in the content host overview so they can be applied.

No idea what I may have done wrong, so thanks in advance for any help.

Foreman and Proxy versions:

Foreman: 1.24.2
Katello: 3.14.1

On host:
katello-agent: 3.5.1

Foreman and Proxy plugin versions:

Distribution and version:

CentOS 7

Other relevant data:

2

Hi @langesmalle,

You said that the updates are available after clearing the yum-cache on the host.

Does clearing the cache also work on other hosts? Or is the problem just with a single host?

If the problem is just with the single host, can you compare the contents of the files in /etc/yum.repos.d/redhat.repo on each host, especially the baseurl line?

3

Hi @jeremylenz,

A manual cache clean on 2 hosts helped to get the latest version available.
But, as described, the available package updates are not visible in the content hosts overview for these 2 hosts.

The redhat.repo is correct on all the systems linked to the composite view containing these new packages.

These systems were initially installed with ceph version 13.2.x.
In orde to upgrade them to version 14.2.8 I removed the content view for 13.2.x from the concerning content hosts subscription and added the new 14.2.x on them. I was able to upgrade the 14.2.8 version of the packages on the concerning systems.
However, yesterday I saw a newer version was available, so I published/promoted a new version of the content view and composite view.
So I was hoping the available upgrades would be visible.
No idea what I may have done wrong.

Just thinking about following, the activation key used for these hosts is still linked with the initial content view of ceph version 13.2.x. Might this be the reason and should I let them point to the content view of ceph 14.2.x?

Thanks in advance for your reply.

4

I think the activation key shouldn’t be a problem.

Are the host and content view assigned to the same lifecycle environment?

Also you can try this: In Content Views -> (your content view) in the Actions column, click the dropdown and then “Regenerate Repository Metadata” and see if that helps.

5

@jeremylenz,

Yes, the hosts and (composite) views are in the same lifecycle environment.
In the lifecycle overview, the number of content hosts and content views is correct.
When I go into the concerned lifecycle I can see the updated “Last promoted” date of May 4th for the concerned updated (promoted) content views.
Also, the new packages are well listed for both the content view of the product, as well the composite view.

I have executed the “Regenerate Repository Metadata” for both the content view and composite view version, but that does not solved it.

Even a forced “subscription-manager register” with the necessary parameters on one of the hosts does not help.

6

@langesmalle,

hmm, ok. What if you try creating a new composite content view, with the same component content views inside?

Could you grab the repo metadata and provide it?

7

Hi Jeremy,

Foreman is relative new for me.
Currently I’m working with it on a testlab, but the idea is to apply it soon for production as it almost works as expected, beside this specific issue.

Do you mean the metadata of the new created composite view, which i already created, published and promoted?
How exactly can I grab this metadata within foreman? And if it is not possible, how can I grab it by command line?
Or do you mean the repomd.xml files in the repodata directory for each component view attached to the composite view, which are located in de folder /var/lib/pulp/published/yum/master/yum_distributor?

Thnx in advance.

8

Sorry for the confusion… The repo metadata in /var/lib/pulp/published/yum/master/yum_distributor/, but for the composite.

9


jeremylenz,

Here the content of the new composite view.

10

https://www.transfernow.net/Oh70SS052020

11

@langesmalle Thanks for sending. I do see the ceph-base-14.2.9-0.el7.x86_64 package in the metadata. Can you confirm if the client sees that one or not?

12

Also, can you clarify what you mean when you say the updates are not in “Foreman / content hosts”? Can you provide a screenshot?

13

@jeremylenz,

Here a link for a tar.gz file that contains a few screenshots based on my initial composite view “All” which contains also the latest version of the ceph-package.

https://www.transfernow.net/IvTNkg052020

In the content-host overview you will see there is no notification for any applicable package upgrade, however in the console session, after applying “yum clean all”, the new version is shown when retrieving the yum-info for one of the packages.

And yes, that version of the ceph-base package is visible as available package on the system/client.

Thnx

14

As the latest package version is visible in the console with example “yum info ceph-common” (only after executing"yum clean all"), I would have expected that the number of available package upgrades were visbible in the content-host overview, like:

image
image1164Ă—50 12 KB

This is an overview based on version 1 of the composite view “All” containing content view “Ceph_mimic_13.x”.
The system was initially installed with ceph-version 13.2.6, so the client has seen there are newer versions available.

image

image
image448Ă—620 46.9 KB

When changing from Ceph_Minic_13.x" to “Ceph_Nautilus_14.x” everything goes fine, it makes use of the latest promoted version 2 of composite view “All”:

image
image489Ă—673 51.4 KB

On the host:

image
image951Ă—656 130 KB

Latest version for Ceph Nautilus is seen:

image
image953Ă—834 176 KB

New number of available package upgrades:

image
image1158Ă—46 11 KB

image
image369Ă—716 58.4 KB

This all went fine as well on the other hosts, when the content view for “Ceph_Nautilus_14.x” and the composite view “All” contained initially the packages till version 14.2.8 in the promoted version 1.

I was able to install/upgrade the hosts with these latest package version.

However, after the upgrade, I saw a newer version 14.2.9 was available in the official Ceph_Nautilus repo, so I synced the product, and published/promoted it into a new version “2” for both the content view “Ceph_Nautilus_14.x” as well for the composite view “All”. Both having, as expected, more packages.

And from that moment on, for those systems already upgraded with version 14.2.8, the latest packages for version 14.2.9 were not discovered anymore.
The number of available package upgrades remains “0”.

On the host, only after performing a “yum clean all” I was able to see the latest package-versions, but in the content host, the number of available package upgrades remains “0”.

Also, after performing subscription-manager refresh, forced register, unregister/register, … on the host, it does not result in a new number of applicable package updates, it remains “0”.

Hope this makes it more clear.

Thnx in advance.

15

Based on discussion with @iballou and @Justin_Sherrill we think that for some reason Pulp doesn’t view the update as “applicable.” We are currently looking at why this may be happening and will update here hopefully soon.

16

@jeremylenz and co,

Thanks for having a look at it. Hope you guys will find something and may also have a solution for it.

Good to know, when I reinstall a system with this promoted version of the composite view, the latest version 14.2.9 is well installed.
So, as you mention it might be pulp that doesn’t view the update as applicable.

I haven’t tried to test this out by syncing another product like Centos/Epel? Please let me know if that might be useful to perform and keep you guys informed about the result?

17

@jeremylenz and co,

This morning I have added a new repo to our product OpenNebula_5.10.x in order to have hotfixes.
As where for Ceph, the new version was made available in the initial existing repo, for OpenNebula it is provided in a new separate repo, which I have added to the product.

image
image1129Ă—471 55.9 KB

So, once synced, I added the repo to the content-view for OpenNebula_5.10.x, published and promoted a new version for it.
Also for the composite view “all” a new version has been published and promoted.

In this specific case, with a new separate, the available package upgrades for this product are visible in the “Content Host” overview.

Those for Ceph are still not visible with the new promoted “composite view”.

image
image953Ă—158 29.4 KB

However, I had to manually execute “subscription-manager repos” on these hosts before the number of available package upgrades became visible in the overview.

I guess this should go all automatically, not?

Nevertheless, seems that for a new version of a package for Ceph within the same product-repo something is not working as expected. Hope this info might be helpful.

18

@jeremylenz and co,

Did you guys found something in Pulp?
Can anyone provide me some feedback?

Thnx in advance.

19

We’re figuring out some similar issues on newer versions of Katello: Katello 3.15 doesn't show available package updates anymore

The issue here is related to the version of Pulp, what version is your Pulp server and RPM plugin?

20

@iballou,

Sorry for the late respons, was very busy with an upgrade.

Keep in mind we still run under Katello 3.14.1 and Foreman 1.24.2.

For pulp, most versions are 2.21.0-1.el7, but here the complete list of installed packages for pulp:

~]# rpm -qa | grep pulp

pulp-rpm-admin-extensions-2.21.0-1.el7.noarch
python-pulp-streamer-2.21.0-1.el7.noarch
rubygem-smart_proxy_pulp-1.5.0-1.el7.noarch
tfm-rubygem-pulp_file_client-0.0.1b10.dev.1557779852-1.el7.noarch
python-pulp-puppet-common-2.21.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.0.0b7.dev01570381057-1.el7.noarch
python-pulp-bindings-2.21.0-1.el7.noarch
pulp-consumer-client-2.21.0-1.el7.noarch
pulp-puppet-tools-2.21.0-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
pulp-server-2.21.0-1.el7.noarch
pulp-docker-plugins-3.2.5-1.el7.noarch
pulp-admin-client-2.21.0-1.el7.noarch
pulp-rpm-yumplugins-2.21.0-1.el7.noarch
python-pulp-oid_validation-2.21.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.2.0b1.dev0.1560866833-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-rpm-plugins-2.21.0-1.el7.noarch
python-pulp-deb-common-1.10.0-1.el7.noarch
python-pulp-client-lib-2.21.0-1.el7.noarch
python-pulp-repoauth-2.21.0-1.el7.noarch
pulp-client-1.0-1.noarch
tfm-rubygem-pulp_2to3_migration_client-0.0.1a1.dev01570560174-1.el7.noarch
pulp-rpm-consumer-extensions-2.21.0-1.el7.noarch
pulp-puppet-plugins-2.21.0-1.el7.noarch
python-pulp-common-2.21.0-1.el7.noarch
python-pulp-docker-common-3.2.5-1.el7.noarch
python-pulp-rpm-common-2.21.0-1.el7.noarch
pulp-selinux-2.21.0-1.el7.noarch
pulp-deb-plugins-1.10.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.0.0rc2.dev.1557772471-1.el7.noarch

When can we expect a fix for this issue, and for which version(s) will it become available?

Thnx in advance.