Traces don't recognize reboot

Support
1

Problem:
There are a few managed hosts in Foreman, where Foreman tells me that a reboot is required.
Foreman obviously relies on “Traces” (which are installed on both Foreman and managed hosts).
When I use “Traces” in Foreman to resolve and reboot, the system reboots, but after the reboot, the status is still the same - Reboot Required.
This used to work in Foreman 3.5, and the problem got introduced after migrating to Foreman 3.7.

I also tried to delete the host from Foreman, and re-register it. Same result.

Expected outcome:
After reboot of the managed host, Foreman (or Traces) should recognize that the host was rebooted.

Foreman and Proxy versions:
3.7.0

Foreman and Proxy plugin versions:
foreman-tasks 8.1.1
foreman_ansible 12.0.6
foreman_remote_execution 10.0.7
katello4.9.1

Distribution and version:
AlmaLinux 8.8

Other relevant data:

image
image1138×450 46.7 KB

2

Hi @ToniF

On the Foreman instance can you run the following command

foreman-tail > traces

On the client run

katello-tracer-upload

And post the contents of the traces file in this thread for me?

1 Like
3

When the host in question reboots, does it actually boot up to the newly
installed kernel or is it booting into an older kernel version?

1 Like
4

this is what I get:

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
10.11.12.13 - - [07/Sep/2023:21:26:11 +0200] "PUT /rhsm/consumers/ebd7540f-48d5-4f27-875e-e1c7e4358c60/tracer HTTP/1.1" 200 57 "-" "RHSM/1.0 (cmd=katello-tracer-upload)"

==> /var/log/foreman/production.log <==
2023-09-07T21:26:11 [I|app|a04e45a1] Started PUT "/rhsm/consumers/ebd7540f-48d5-4f27-875e-e1c7e4358c60/tracer" for 10.11.12.13 at 2023-09-07 21:26:11 +0200
2023-09-07T21:26:11 [I|app|a04e45a1] Processing by Katello::Api::Rhsm::CandlepinProxiesController#upload_tracer_profile as JSON
2023-09-07T21:26:11 [I|app|a04e45a1]   Parameters: {"traces"=>{"kernel"=>{"type"=>"static", "helper"=>"You will have to reboot your computer"}}, "id"=>"ebd7540f-48d5-4f27-875e-e1c7e4358c60"}
2023-09-07T21:26:11 [I|app|a04e45a1] Completed 200 OK in 41ms (Views: 0.1ms | ActiveRecord: 9.5ms | Allocations: 14829)

Many thanks!

5

Hi,
I have indeed noticed that the OS does not seem to boot into the latest kernel.
The systems that have the problem are all CentOS 7 (but not all CentOS 7 systems have the problem).

sh-4.2# rpm -q kernel
kernel-3.10.0-1160.83.1.el7.x86_64
kernel-3.10.0-1160.90.1.el7.x86_64
kernel-3.10.0-1160.92.1.el7.x86_64
kernel-3.10.0-1160.95.1.el7.x86_64
kernel-3.10.0-1160.88.1.el7.x86_64
sh-4.2# uname -sr
Linux 3.10.0-1160.88.1.el7.x86_64

I also tried to yum reinstall kernel, which did not help.
Also, running grub2-mkconfig -o /boot/grub2/grub.cfg did not change the behavior of booting kernels.

Looking at CentOS systems where Tracer does not complain, they seem to be booting the latest kernel.

sh-4.2# rpm -q kernel
kernel-3.10.0-1160.83.1.el7.x86_64
kernel-3.10.0-1160.88.1.el7.x86_64
kernel-3.10.0-1160.90.1.el7.x86_64
kernel-3.10.0-1160.92.1.el7.x86_64
kernel-3.10.0-1160.95.1.el7.x86_64
sh-4.2# uname -sr
Linux 3.10.0-1160.95.1.el7.x86_64

So it’s most likely a problem with the remote hosts, rather than a problem with Foreman/Tracer.

But I have yet to find a way to fix the systems…

Thank you @cintrix84

6

OK, it seems that reinstalling the latest version of the kernel fixes the issue.

List all kernel versions that are installed:

sh-4.2# rpm -q kernel
kernel-3.10.0-1160.95.1.el7.x86_64
kernel-3.10.0-1160.90.1.el7.x86_64
kernel-3.10.0-1160.92.1.el7.x86_64
kernel-3.10.0-1160.88.1.el7.x86_64
kernel-3.10.0-1160.83.1.el7.x86_64

The latest version of the kernel appears to be kernel-3.10.0-1160.95.1.el7.x86_64.
Notice the order, in which the kernels appear.
This system was booting the kernel kernel-3.10.0-1160.83.1.el7.x86_64 - the last in this list, rather than the latest kernel version.

It seems that reinstalling the latest version of the kernel will move it to the end of the list:

sh-4.2# yum reinstall kernel-3.10.0-1160.95.1.el7.x86_64
[ ..... ]
sh-4.2# rpm -q kernel
kernel-3.10.0-1160.90.1.el7.x86_64
kernel-3.10.0-1160.92.1.el7.x86_64
kernel-3.10.0-1160.88.1.el7.x86_64
kernel-3.10.0-1160.83.1.el7.x86_64
kernel-3.10.0-1160.95.1.el7.x86_64

After system reboot, the system will reboot with the latest kernel (i.e. the last in the list), and Tracer will notice, and will be happy.

Thanks to everyone!

1 Like