Katello Ubuntu server no installation candidate

Support
#1

I have a Katello content view configured for Ubuntu 20.04. Most things work fine. However - if I go to install the meta package sssd and actually quite a few other packages are not showing up as available to install on the client.

/usr/local/bin# apt-get install sssd
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package sssd is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'sssd' has no installation candidate

But - If I go to the content view that is applied to this host in Foreman - it does show that the packages are there.

image
image2222×252 36.1 KB

I’ve tried re-attaching the client, cleaning the apt cache, etc. Nothing seems to help here.

Other systems that are subscribed to the same Content View are not getting updates after doing an apt update. E.g., the UI says there should be 151 packages for upgrade, but the client shows 4 and they are all older versions then are in the Content View?

Any thoughts? Never had this much issue with rpm based management.

#2

How are your clients attached to the content view? Are you using the clients from http://apt.atix.de/?

#3

Maybe a stupid question if you are experienced to use Katello, but are the servers only attached to the CV or do they also have subscriptions for the corresponding products? Are all your repositories listed under “Repository sets”?

#4

Yes, using the Aptix subscription manager

#5

Yes, they are only attached to my composite content view and they have the correct repository sets enabled. They are subscribed to the correct repositories in subscription manager.

#6

Can you check the repository-file of an attached ubuntu client for the subscriptionmanager?
It should be placed under /etc/apt/sources.list.d/rhsm.sources

Do you have a proxy between your client and your katello?

Have you checked if there are any errors in /var/log/rhsm/rhsm.log?

#7

Just for giggles, I did just upgrade Foreman/Katello to the latest release and that didn’t help.

Can you check the repository-file of an attached ubuntu client for the subscriptionmanager?
It should be placed under /etc/apt/sources.list.d/rhsm.sources

Not sure what you're looking for here - it's definitely subscribed to the correct repositories.

Here is the entry for focal-security where the sssd package resides:

name: focal-security
baseurl: https://hq-1pforeman.internal.domain.com/pulp/deb/Default_Organization/DevPortal/Ubuntu_20_04_Compsite/custom/Ubuntu_20_04/focal-security
enabled: 1
gpgcheck: 0
sslverify: 1
sslcacert: /etc/rhsm/ca/katello-server-ca.pem
sslclientkey: /etc/pki/entitlement/5260617178678886184-key.pem
sslclientcert: /etc/pki/entitlement/5260617178678886184.pem
metadata_expire: 1
enable_metadata: 0
arches: none
enabled_metadata: 1
Types: deb
URIs: katello://5260617178678886184@hq-1pforeman.internal.domain.com/pulp/deb/Default_Organization/DevPortal/Ubuntu_20_04_Compsite/custom/Ubuntu_20_04/focal-security
Suites: default
Components: all
Trusted: yes
id: Default_Organization_Ubuntu_20_04_focal-security

Do you have a proxy between your client and your katello?

No proxy

Have you checked if there are any errors in /var/log/rhsm/rhsm.log?

Nothing of significance here, the warnings repeat, the updates in the rest of the entries are 0 as nothing has been added to the content view.

2022-02-08 15:44:53,638 [WARNING] subscription-manager:3591:MainThread @__init__.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @__init__.py:141 - Exiting plugin
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @__init__.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @__init__.py:141 - Exiting plugin
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @__init__.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @__init__.py:141 - Exiting plugin
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @__init__.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @__init__.py:141 - Exiting plugin
2022-02-08 15:44:53,639 [WARNING] subscription-manager:3591:MainThread @config.py:117 - /etc/ostree/remotes.d does not exist, so unable to save /etc/ostree/remote>
2022-02-08 15:44:55,054 [WARNING] subscription-manager:3591:MainThread @repolib.py:124 - Configuration file of dnf plugin: "/etc/dnf/plugins/subscription-manager.>
2022-02-08 15:44:55,055 [WARNING] subscription-manager:3591:MainThread @repolib.py:124 - Configuration file of dnf plugin: "/etc/dnf/plugins/product-id.conf" cann>
2022-02-08 16:28:09,307 [INFO] subscription-manager:7028:MainThread @entcertlib.py:132 - certs updated:
Total updates: 4
Found (local) serial# []
Expected (UEP) serial# [3458859113961921236, 1993736400662520074, 4192335943816619221, 6778812690503264061]
Added (new)
  [sn:3458859113961921236 (Wazuh,) @ /etc/pki/entitlement/3458859113961921236.pem]
  [sn:1993736400662520074 (Zabbix,) @ /etc/pki/entitlement/1993736400662520074.pem]
  [sn:4192335943816619221 (Ubuntu 20.04,) @ /etc/pki/entitlement/4192335943816619221.pem]
  [sn:6778812690503264061 (Atix,) @ /etc/pki/entitlement/6778812690503264061.pem]
Deleted (rogue):
  <NONE>
#8

Example of an issue - here are the top 3 packages that Foreman is saying require updating on this machine - I’m going to look at apport as one example.

image
image2154×310 72 KB

On the console of the machine, after doing a subscription-manager refresh, apt update

root@max-ubuntu:/etc/apt# apt list apport -a
Listing... Done
apport/unknown,unknown,now 2.20.11-0ubuntu27.18 all [installed,automatic]
apport/unknown 2.20.11-0ubuntu27 all

Clearly 2.20.11-0ubuntu27.21 is newer than 2.20.11-0ubuntu27.18

Foreman says that there are 148 packages that need updating, the ubuntu machine says that there are 0.

#9

I have no idea what changed on one system. I un-registered it, re-registered it multiple times, all of a sudden it started receiving updates. A second system with the same issue is still not receiving updates. Something is definitely wrong.

subscription-manager unregister
apt clean
apt purge
subscription-manager register --org=“Default_Organization” --activationkey=“DevPortal-Ubuntu”

#10

Now the second system is working. Again working the following. Probably about 3 minutes between the unregister and re-register.

Something is preventing the systems from updating without major work. I can’t do this semi-random process for each server.

59 sudo subscription-manager unregister
60 sudo apt clean all
61 sudo apt purge
62 sudo apt-get
63 sudo apt-cache list
64 sudo apt-get
65 sudo apt-get update
66 sudo subscription-manager register --org=“Default_Organization” --activationkey=“DevPortal-Ubuntu”
67 sudo apt update

#11

Thanks for all the extra information you are adding, please keep doing that with anything you do find/come up with. If this is an intermittent/timing related issue, it will probably be difficult for us to reproduce it. If we can’t reproduce this ourselves, the best we can do right now is probably to keep an eye on this thread as well as keep on the look out for the issue in our own deployments. I will pass this on to our QA testers one more time.

#12

Can you have a look at /etc/yum.repos.d/redhat.repo and /var/lib/rhsm/repo_server_val/rhsm.sources if these files have differences?

diff /etc/apt/sources.list.d/rhsm.sources /var/lib/rhsm/repo_server_val/rhsm.sources

Did you try to do a “subscription-manager refresh --force” if this is the “only” command to run to get it working again?

#13

Before doing anything:

diff /etc/apt/sources.list.d/rhsm.sources /var/lib/rhsm/repo_server_val/rhsm.sources
20,21c20,21
< sslclientkey: /etc/pki/entitlement/1155631218222210547-key.pem
< sslclientcert: /etc/pki/entitlement/1155631218222210547.pem
---
> sslclientkey: /etc/pki/entitlement/7464288580996207129-key.pem
> sslclientcert: /etc/pki/entitlement/7464288580996207129.pem
37,38c37,38
< sslclientkey: /etc/pki/entitlement/1155631218222210547-key.pem
< sslclientcert: /etc/pki/entitlement/1155631218222210547.pem
---
> sslclientkey: /etc/pki/entitlement/7464288580996207129-key.pem
> sslclientcert: /etc/pki/entitlement/7464288580996207129.pem
55,56c55,56
< sslclientkey: /etc/pki/entitlement/4672733234345866-key.pem
< sslclientcert: /etc/pki/entitlement/4672733234345866.pem
---
> sslclientkey: /etc/pki/entitlement/933470687186359263-key.pem
> sslclientcert: /etc/pki/entitlement/933470687186359263.pem
72,73c72,73
< sslclientkey: /etc/pki/entitlement/1155631218222210547-key.pem
< sslclientcert: /etc/pki/entitlement/1155631218222210547.pem
---
> sslclientkey: /etc/pki/entitlement/7464288580996207129-key.pem
> sslclientcert: /etc/pki/entitlement/7464288580996207129.pem
90,91c90,91
< sslclientkey: /etc/pki/entitlement/4035655559503194575-key.pem
< sslclientcert: /etc/pki/entitlement/4035655559503194575.pem
---
> sslclientkey: /etc/pki/entitlement/5902345535042827072-key.pem
> sslclientcert: /etc/pki/entitlement/5902345535042827072.pem
108,109c108,109
< sslclientkey: /etc/pki/entitlement/5763354279374614677-key.pem
< sslclientcert: /etc/pki/entitlement/5763354279374614677.pem
---
> sslclientkey: /etc/pki/entitlement/4805172231685807871-key.pem
> sslclientcert: /etc/pki/entitlement/4805172231685807871.pem

There is no option for force for this command

sudo subscription-manager refresh --force
Usage: subscription-manager refresh [OPTIONS]

subscription-manager: error: no such option: --force

sudo subscription-manager refresh
4 local certificates have been deleted.
All local data refreshed

diff /etc/apt/sources.list.d/rhsm.sources /var/lib/rhsm/repo_server_val/rhsm.sources
20,21c20,21
< sslclientkey: /etc/pki/entitlement/167943493715475653-key.pem
< sslclientcert: /etc/pki/entitlement/167943493715475653.pem
---
> sslclientkey: /etc/pki/entitlement/7464288580996207129-key.pem
> sslclientcert: /etc/pki/entitlement/7464288580996207129.pem
37,38c37,38
< sslclientkey: /etc/pki/entitlement/167943493715475653-key.pem
< sslclientcert: /etc/pki/entitlement/167943493715475653.pem
---
> sslclientkey: /etc/pki/entitlement/7464288580996207129-key.pem
> sslclientcert: /etc/pki/entitlement/7464288580996207129.pem
55,56c55,56
< sslclientkey: /etc/pki/entitlement/1033279582955578697-key.pem
< sslclientcert: /etc/pki/entitlement/1033279582955578697.pem
---
> sslclientkey: /etc/pki/entitlement/933470687186359263-key.pem
> sslclientcert: /etc/pki/entitlement/933470687186359263.pem
72,73c72,73
< sslclientkey: /etc/pki/entitlement/167943493715475653-key.pem
< sslclientcert: /etc/pki/entitlement/167943493715475653.pem
---
> sslclientkey: /etc/pki/entitlement/7464288580996207129-key.pem
> sslclientcert: /etc/pki/entitlement/7464288580996207129.pem
90,91c90,91
< sslclientkey: /etc/pki/entitlement/3744431905807390335-key.pem
< sslclientcert: /etc/pki/entitlement/3744431905807390335.pem
---
> sslclientkey: /etc/pki/entitlement/5902345535042827072-key.pem
> sslclientcert: /etc/pki/entitlement/5902345535042827072.pem
108,109c108,109
< sslclientkey: /etc/pki/entitlement/5366247819297891607-key.pem
< sslclientcert: /etc/pki/entitlement/5366247819297891607.pem
---
> sslclientkey: /etc/pki/entitlement/4805172231685807871-key.pem
> sslclientcert: /etc/pki/entitlement/4805172231685807871.pem

Still not working.

sudo subscription-manager unregister
Unregistering from: hq-1pforeman.internal.ieeeglobalspec.com:443/rhsm
System has been unregistered.

sudo subscription-manager register --org="Default_Organization" --activationkey="DevPortal-Ubuntu"
The system has been registered with ID: 81d04c64-c0a6-43bb-9223-cc28fa22d065
The registered system name is: max-ufromt
No products installed.

sudo apt update
...
Reading package lists... Done
Building dependency tree
Reading state information... Done
146 packages can be upgraded. Run 'apt list --upgradable' to see them.

It seem this process works every time - but shouldn’t need to be done:

subscription-manager unregister
Wait a few minutes, I'm guessing for Katello to remove the registration.
subscription-manager register --org="Default_Organization" --activationkey="DevPortal-Ubuntu"
apt update
#14

Would it be possible, to test if the issue can be fixed by

rm -f /etc/apt/sources.list.d/rhsm.sources /var/lib/rhsm/repo_server_val/rhsm.sources
subscription-manager refresh

Which subscription-manager version are you using?

#15

I may have run out of hosts to test with. I need to promote some content views to see if it affects systems in production as well.

I’m using the aptix repo.
python3-subscription-manager/unknown,now 1.28.24-1 amd64

I can say that after doing this, the rhsm.sources file remains empty and no repos are available.

rm -f /etc/apt/sources.list.d/rhsm.sources /var/lib/rhsm/repo_server_val/rhsm.sources
subscription-manager refresh

[/quote]