Unable to apply updates from Foreman

gchamberlain · February 3, 2023, 5:02pm

Problem: Unable to apply updates from Foreman

Expected outcome: Expect to see installable updates listed under Content Hosts and be able to apply them from Foreman.

Foreman and Proxy versions: Foreman 3.4.1 (built-in proxy only)

Foreman and Proxy plugin versions:

foreman-tasks 7.0.0
foreman_remote_execution 8.0.0
katello 4.6.2.1

Distribution and version: AlmaLinux 8.7 (Stone Smilodon)

Other relevant data:

Hi guys,

I inherited a half-finished Foreman/Katello server running AlmaLinux 8 that is intended to replace our Spacewalk server. Its main role is to manage system updates across a cluster of mostly CentOS 7 plus a few Debian 11/bullseye containers.

My goal is just to have an overview of package updates available across a LAN of a few dozen hosts and bulk apply them from Foreman.

I have a Product for CentOS 7 with one repository which successfully syncs from the upstream mirror. By generating a registration command from the web interface, I was able to register a few hosts as clients, although I got the warning below (we do not have a Red Hat subscription).

This system has already been registered with Red Hat using RHN
Classic.

Your system is being registered again using Red Hat Subscription
Management. Red Hat recommends that customers only register once.

The registered hosts show up in the Content Hosts page with the correct OS and recent check-in times. However, the installable updates are all zeroes even when I know there should be available updates. I know there are updates available because I can run yum check-update on the client and see them listed — I think it is pulling those directly from upstream though.

[gchamberlain@web3 ~]$ yum check-update
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager, tmprepo
Loading mirror speeds from cached hostfile
 * base: mirrors.coreix.net
 * epel: ftp.uni-bayreuth.de
 * extras: mirrors.vinters.com
 * updates: centos.mirrors.nublue.co.uk

httpd.x86_64                                                      2.4.6-98.el7.centos.6                                  updates
httpd-devel.x86_64                                                2.4.6-98.el7.centos.6                                  updates
httpd-tools.x86_64                                                2.4.6-98.el7.centos.6                                  updates
java-1.8.0-openjdk.x86_64                                         1:1.8.0.362.b08-1.el7_9                                updates
java-1.8.0-openjdk-headless.x86_64                                1:1.8.0.362.b08-1.el7_9                                updates
kernel-debug-devel.x86_64                                         3.10.0-1160.83.1.el7                                   updates
kernel-headers.x86_64                                             3.10.0-1160.83.1.el7                                   updates
libXpm.x86_64                                                     3.5.12-2.el7_9                                         updates
mod_ssl.x86_64                                                    1:2.4.6-98.el7.centos.6                                updates
sudo.x86_64                                                       1.8.23-10.el7_9.3                                      updates

I am trying to understand how subscription-manager works.

[gchamberlain@web3 ~]$ sudo subscription-manager list --available
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Debian 11/bullseye
[...]

Subscription Name:   CentOS 7
Provides:
SKU:                 739249773278
Contract:
Pool ID:             8a8b8a8e8438d31201843ce804810047
Provides Management: No
Available:           Unlimited
Suggested:           1
Service Type:
Roles:
Service Level:
Usage:
Add-ons:
Subscription Type:   Standard
Starts:              03/11/22
Ends:                01/12/49
Entitlement Type:    Physical

Subscription Name:   AlmaLinux 8
[...]

[gchamberlain@web3 ~]$ sudo subscription-manager attach --pool=8a8b8a8e8438d31201843ce804810047
Successfully attached a subscription for: CentOS 7

Great, I’ve attached a subscription!

[gchamberlain@web3 ~]$ sudo subscription-manager repos
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   Default_Organization_CentOS_7_CentOS_7_Main
Repo Name: CentOS 7 Main
Repo URL:  https://foreman.domain.tld/pulp/content/Default_Organization/Library/custom/CentOS_7/CentOS_7_Main
Enabled:   1

[gchamberlain@web3 ~]$ cat /etc/yum.repos.d/foreman.repo 
[foreman-base]
name=Foreman Base Repo
baseurl=https://foreman.lan/pulp/content/Default_Organization/Library/custom/CentOS_7/CentOS_7_Main/
gpgcheck=0
enabled=1
sslverify=0

Things I have tried:

create “Operating Systems” in Foreman
add content credentials in the form of RPM-GPG-KEY-CentOS-7 and assign to the repo
create new activation key and re-register the clients
publish a Content View
using a “Production” environment
disable simple content access for the organization
update the Foreman host system and reboot
suggestions on similar posts here (some kinda outdated)

Under Content Hosts, for “Installable Updates” all I see is zeroes:

My CentOS 7 repo:

I configured my firewall according to the Foreman and Katello docs, but it’s possible I made a mistake there.

Am I missing something? How can I update my client machines from Foreman?

With thanks,

Greg

gvde · February 3, 2023, 5:14pm

Where is that file coming from? That doesn’t belong there.

Run

# yum repolist

to see all active repositories.

iballou · February 6, 2023, 3:06pm

Hi @gchamberlain ,

So it sounds like you’re able to consume content from your Foreman/Katello setup, but applicability (the mechanism for showing you needed updates) isn’t working.

Firstly, some corrections to help clear things up:

You don’t need to worry about Foreman Operating Systems for machine patching.
Only use what you need from the whole Content View / Lifecycle environment workflow. Some users use LCEs even if they don’t need them, which could add unneeded complexity.
You don’t need to disable SCA especially if you’re using content views to isolate different media like CentOS/Debian/etc.
- SCA will become the default at some point and the page for managing subscriptions is now “legacy”.

Since you disabled SCA, take a look on the “legacy” content host page’s Subscriptions tab to ensure that your host actually has subscriptions. You can get there from the kebab drop-down menu on the host’s details page.

If you can confirm that your host has subscriptions, please show us the result of the following commands:

# foreman-rake console
=> ::Host.find_by(name: "name of your host").content_facet.bound_repositories.pluck(:relative_path)

Do the returned repositories make sense?

gchamberlain · February 6, 2023, 3:59pm

Hi @iballou, thanks for your help.

# foreman-rake console
Loading production environment (Rails 6.1.6.1)
irb(main):001:0> ::Host.find_by(name: "web3").content_facet.bound_repositories.pluck(:relative_path)
=> ["Default_Organization/Library/custom/CentOS_7/CentOS_7_Main"]

Looks good to me. My web3 is a CentOS 7 system, and CentOS_7_Main is the correct repository.

I am not convinced content consumption is working, because I’m not able to apply updates from another CentOS 7 server which out outside my DMZ; it has HTTP(S) access to Foreman but not directly to the Internet.

[gchamberlain@ansible ~]$ curl -k https://foreman.lan/pulp/content/Default_Organization/Library/custom/CentOS_7/CentOS_7_Main/
<!DOCTYPE html>
  <html>
    <body>
...

Although, without -k it does not accept the certificate.

gchamberlain · February 6, 2023, 4:05pm

I don’t know where it’s coming from, I just found it and thought it might be relevant.

[gchamberlain@web3 ~]$ sudo yum repolist
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager, tmprepo
1 local certificate has been deleted.
Loading mirror speeds from cached hostfile
 * base: mirror.cov.ukservers.com
 * epel: mirrors.20i.com
 * extras: mirrors.melbourne.co.uk
 * updates: mirror.cov.ukservers.com
repo id                                                 repo name                                                 status
!Default_Organization_CentOS_7_CentOS_7_Main            CentOS 7 Main                                             10,072
!base/7/x86_64                                          CentOS-7 - Base                                           10,072
!epel/x86_64                                            Extra Packages for Enterprise Linux 7 - x86_64            13,744
!extras/7/x86_64                                        CentOS-7 - Extras                                            515
!foreman-base                                           Foreman Base Repo                                         10,072
!spacewalk-client/x86_64                                spacewalk project                                             95
!updates/7/x86_64                                       CentOS-7 - Updates                                         4,691
repolist: 49,261

CentOS 7 Main and Foreman Base Repo come from Foreman. I guess most or all of the others come from Spacewalk.

iballou · February 7, 2023, 3:57pm

@gchamberlain interesting, if your machine cannot yum install from your Foreman/Katello then I would think that it also isn’t capable of pushing up the installed package information. If you install a package on your machine, do you see that in the list of installed packages on the host’s page in Foreman?

Also curious if you see any logs in the Foreman production.log that look like Started PUT "/rhsm/consumers/ ... when you run yum actions on your host. That is, if you can yum install something from Foreman.

To aid in your debugging, subscription-manger is responsible for sending a “profile” to Foreman that contains your host’s installed packages. Once Foreman has that profile, we use the installed package information paired with the “bound repositories” to calculate your necessary updates.

gchamberlain · February 9, 2023, 5:12pm

OK, I just ran yum install nmap from ansible (a server with no direct Internet access) and it looks like it did indeed pull it down from Foreman/Katello and I can now see that package listed as installed on the host in Foreman. Also, when I run yum install, yum update etc. I can watch production.log and see the requests as they come through.

I see one line from today that looks like Started PUT "/rhsm/consumers/... and other such lines in logs from previous days.

iballou · February 10, 2023, 3:31pm

Okay nice, so it sounds like all those mechanisms are working. Awesome.

One thing to keep in mind is that if you change lifecycle environment and content view for a host, you won’t see the updates until the host sends another profile to Katello. So it’ll work the next time you do some yum action, for example. I’m not sure if that might be why you see missing updates but I figured I’d mention it.

Anyway, do you still see a case of a package update that isn’t being reported on your host? If so, let’s isolate it to one package. For a sanity check, I’d pick that package and then search for it in your CentOS_7_Main repository to check that the updated version is indeed in that repo.

gchamberlain · February 20, 2023, 4:14pm

I noticed one of the minimum requirements of a Foreman server is full forward and reverse DNS resolution using a FQDN. My cluster doesn’t have internal nameservers. We have always relied on updating /etc/hosts on each system – could that have something to do with this? Or does this requirement just refer to global DNS?

iballou · February 21, 2023, 7:23pm

If your host can hit the Foreman server I think you should be all set. The fact that you were able to send installed package info to the Foreman means things are working.

I’m not sure what else would be broken due to that DNS requirement, I’ll see if anyone else has any ideas.

From the following:

it sounds like everything is working. Is something else broken still?

wbclark · February 21, 2023, 8:27pm

Hey there, a couple observations and things to consider:

The machine web3 is still getting some repos from spacewalk, and some other repos from outside your network it appears. Maybe that’s intended at this stage of the migration, but I want to make sure we’re all on the same page.
I don’t see a katello-host-tools yum plugin, which may be necessary for package profile uploads and applicability if your subscription-manager is an older version. You can check sudo rpm -q subscription-manager to find out… or just update it to the latest version
After upgrading subscription-manager, you can either wait for the next rhsmcertd check-in to upload the package profile, or enable the package_profile_on_trans setting in /etc/rhsm/rhsm.conf to have yum/dnf do this immediately on any transaction (it’s disabled by default because it can generate a lot of load when patching a large # of servers simultaneously)
In general, the Red Hat knowledge base is very useful for troubleshooting these kinds of issues. For example Why and when is katello-host-tools required in Satellite hosts? - Red Hat Customer Portal and How to remove/unregister a System from Red Hat Satellite 5.x? - Red Hat Customer Portal would be immediately relevant in your situation, probably some others too. Even if you don’t have any paid RH subscription, you can get a No-cost RHEL individual developer subscription which should provide access to the knowledge base, see No-cost Red Hat Enterprise Linux Individual Developer Subscription: FAQs | Red Hat Developer for details

wbclark · February 21, 2023, 8:37pm

To answer this question, some functionality may not work without full forward and reverse name resolution in your network. If you have static IPs and edited /etc/hosts on each machine you can probably get basic stuff working… I don’t see any reason why it should cause this particular problem for you, for instance. NOTE: There are DNS plugins available for foreman-proxy, you may find those useful in the future, as situations like yours where you want to use Foreman but haven’t reached a scale where you need a separate dedicated nameserver are basically the intended use case. See: List of Smart-Proxy Plugins - Foreman for more information

gchamberlain · March 20, 2023, 3:24pm

Sorry for the late reply. I am beginning to wrap my head around all this now.

Spacewalk was configured with a few additional repositories that had more recent releases of many packages, meaning it had updated them before Foreman had a chance I guess. After replicating those additional repositories, Foreman now allows me to apply updates to my machines.

Thanks for your suggestions