No package katello-host-tools-tracer available

ToniF · February 9, 2023, 11:57pm

Problem:

From the Foreman Admin GUI, in a Content Host document, select “Enable Traces” under “Traces”.
Submit the Remote Execution job.
The job fails to run:

Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager, susemanagerplugin, tmprepo
Loading mirror speeds from cached hostfile
No package katello-host-tools-tracer available.
Error: Nothing to do
Package action failed, exiting...
Exit status: 1
StandardError: Job execution failed

Expected outcome:
The embedded function should just work, and enable Traces.

Foreman and Proxy versions:
3.5.1

Foreman and Proxy plugin versions:
Katello 4.7.1

Distribution and version:
Rocky Linux 8.7

Other relevant data:

Remote Execution is configured to use SSH, and not katello-agent, as Foreman’s documentation states that katello-agent should not be used anymore, as it will be deprecated.
This issue is reproducible on several Content Hosts and several OS platforms (e.g. CentOS 7, Rock Linux 8).
The issue is reproducible when running the command directly on the Content Host (i.e. yum install katello-host-tools-tracer)

Is something missing in my configuration?

Thanks

ToniF · February 10, 2023, 12:55am

Ok, I found the following documentation (didn’t see previously):
https://docs.theforeman.org/nightly/Managing_Hosts/index-katello.html#enabling-tracer-on-a-host_managing-hosts

It appears that the Foreman-Client yum repository needs to be added to the Content View, in order for the Content Host to be able to install a Foreman agent software.

This seems to break the “agent-less” pragma of SSH Remote Execution.
The whole point of SSH management is to avoid client agents.

Are there ambitions by the Foreman community to replace katello-host-tools-tracer with an SSH script?

ToniF · February 10, 2023, 1:18am

I’m still struggling.
yum fails to install the packet, because packets in the nightly repo don’t appear to be signed with a GPG key.
What is the proper way to deal with it?
I don’t like to install unsigned packages, as it opens to the threat of supply-chain attacks.

Package katello-host-tools-3.5.7-5.el8.noarch.rpm is not signed
Package katello-host-tools-tracer-3.5.7-5.el8.noarch.rpm is not signed
Package python3-psutil-5.7.2-2.el8.x86_64.rpm is not signed
Package python3-tracer-0.7.8-1.el8.noarch.rpm is not signed
Package tracer-common-0.7.8-1.el8.noarch.rpm is not signed
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED

thanks

evgeni · February 10, 2023, 7:28am

Install them from a release repo (like 3.5 instead of nightly), then they are signed

maximilian · February 10, 2023, 7:32am

Hi @ToniF

katello-host-tools-tracer is not required to do REX using SSH or Ansible. It is optional and reports if services (like httpd) or the host itself (after a Linux Kernel update) needs to be rebooted.

katello-agent is deprecated and has been replaced with an pull-based approach using MQTT and yggdrasil. For more information, see Transport Modes for Remote Execution.

ToniF · February 10, 2023, 9:52am

Many thanks for your answer.

Did you see that the official documentation (link above) states explicitly that the nightly repo is a pre-requisite?
My issue may have been caused by the documentation.

I tried to use latest (3.5) instead, but I am still getting errors, because it seems that the signature of packages does not match the GPG key.

I found this GPG key in the repo:
https://yum.theforeman.org/RPM-GPG-KEY-foreman

What I did so far:

I had deleted all old “nightly” repos, and created new ones from scratch for “latest”, and re-added them to the corresponding Content View.
/etc/yum.repos.d/redhat.repo was updated correctly on the Content Hosts.
Tried yum clean all

Error output:

GPG key at https://foreman.company.com/katello/api/v2/repositories/276/gpg_key_content (0x1AA043B8) is already installed
The GPG keys listed for the "foreman-client-el8" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: katello-host-tools-3.5.7-5.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/276/gpg_key_content
Public key for katello-host-tools-tracer-3.5.7-5.el8.noarch.rpm is not installed. Failing package is: katello-host-tools-tracer-3.5.7-5.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/276/gpg_key_content
Public key for python3-psutil-5.7.2-2.el8.x86_64.rpm is not installed. Failing package is: python3-psutil-5.7.2-2.el8.x86_64
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/276/gpg_key_content
Public key for python3-tracer-0.7.8-1.el8.noarch.rpm is not installed. Failing package is: python3-tracer-0.7.8-1.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/276/gpg_key_content
Public key for tracer-common-0.7.8-1.el8.noarch.rpm is not installed. Failing package is: tracer-common-0.7.8-1.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/276/gpg_key_content
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED

evgeni · February 10, 2023, 9:54am

You linked to the nightly documentation, so it talks about nightly

https://docs.theforeman.org/3.5/Managing_Hosts/index-katello.html#enabling-tracer-on-a-host_managing-hosts talks about 3.5 (but I am afraid we’re currently not linking those up properly on the index page)

ToniF · February 10, 2023, 10:30am

Hi, many thanks!
Oh there is a dedicated documentation for nightly?
First time I see in any product.
OK, my bad, I picked the wrong documentation.

Are you saying that latest is not a symlink pointing to 3.5?

I deleted all repos and recreated them.
Instead of:

https://yum.theforeman.org/client/latest/el8/x86_64/

I am now using:

https://yum.theforeman.org/client/3.5/el8/x86_64/

But still the same error:

GPG key at https://foreman.company.com/katello/api/v2/repositories/332/gpg_key_content (0x1AA043B8) is already installed
The GPG keys listed for the "foreman-client-el8" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: katello-host-tools-3.5.7-5.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/332/gpg_key_content
Public key for katello-host-tools-tracer-3.5.7-5.el8.noarch.rpm is not installed. Failing package is: katello-host-tools-tracer-3.5.7-5.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/332/gpg_key_content
Public key for python3-psutil-5.7.2-2.el8.x86_64.rpm is not installed. Failing package is: python3-psutil-5.7.2-2.el8.x86_64
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/332/gpg_key_content
Public key for python3-tracer-0.7.8-1.el8.noarch.rpm is not installed. Failing package is: python3-tracer-0.7.8-1.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/332/gpg_key_content
Public key for tracer-common-0.7.8-1.el8.noarch.rpm is not installed. Failing package is: tracer-common-0.7.8-1.el8.noarch
 GPG Keys are configured as: https://foreman.company.com/katello/api/v2/repositories/332/gpg_key_content
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED

And yum clean all doesn’t help.

evgeni · February 10, 2023, 10:36am

It should be, but the creation/update is a bit broken and we sometimes forget to clean that up (humans are terrible at repetitive tasks, huh?)

Where is that 0x1AA043B8 coming from?

When I install the 3.5 client bits here locally, I get the following key:

Importing GPG key 0x8F5CA95B:
 Userid     : "Foreman Automatic Signing Key (3.5) <packages@theforeman.org>"
 Fingerprint: 51B9 D1D1 BADA 3F00 1C87 7404 6F7F 9C61 8F5C A95B
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-foreman-client

Looking at Foreman :: Security 0x1AA043B8 is a super old key that we used in 2016

ToniF · February 10, 2023, 10:43am

As mentioned above, I searched the repo, and all I was able to find within the repo is this:
https://yum.theforeman.org/RPM-GPG-KEY-foreman

Is this the wrong GPG key, and where do I find the correct one?

evgeni · February 10, 2023, 10:49am

https://yum.theforeman.org/releases/3.5/RPM-GPG-KEY-foreman is the right key for the 3.5 release.

ToniF · February 10, 2023, 10:57am

Yes, this works now!
Maybe the GPG URL is worth mentioning in the documentation?

P.S. I would be interested in a broader discussion, i.e. why agent software is (kind of) required, when SSH Remote Execution allows agent-less systems management. Potentially a contradictory strategy? And why MQTT is used, when the protocol is inherently insecure. What’s the best formal way to start such a discussion?

evgeni · February 10, 2023, 11:01am

It’s not required. Tracer is additional functionality, that needs an agent-like software. You don’t have to use that

What’s insecure about MQTT?

But it’s probably best to start new discussions in the “Development” section for that.

gvde · February 10, 2023, 11:05am

If you want to know which hosts need to reboot or restart services after an update you need the tracer yum/dnf plugin. It automatically runs during yum/dnf and uploads the information to the foreman server. If you don’t want that, you don’t need it. It only runs on the client not the server. It has nothing to do with remote execution from the server on the client…

ToniF · February 10, 2023, 11:12am

I know - but in real world systems operation, the operator must have the information whether a system requires a reboot or not. In my eyes, it is too relevant to not use it. Hence I say “kind of” required.

And we don’t want a step backwards from Spacewalk, do we?

Thanks, I’ll give it a try.

evgeni · February 10, 2023, 11:16am

Sure, but you also want to know if someone did a local dnf upgrade and stuff needs restarting.
And that will be reported by Tracer (because it is “an agent” (well, really, a plugin to dnf)), which a REX-based thing would not do.

ToniF · February 10, 2023, 11:17am

Thanks evgeni, very well explained!