Trace status incorrect

gvde · July 8, 2024, 5:11am

Problem:

When checking in the GUI for traces I’ll get a lot of hosts with process_restart_needed. However, when I check on the hosts there is nothing to do:

# needs-restarting -s
# needs-restarting -r
# needs-restarting
# tracer

all report nothing. Even after I run

# /sbin/katello-tracer-upload

it still reports traces in the GUI. For instance, I have just updated openssh on my hosts. I have run /sbin/katello-tracer-upload on all. Still, the GUI shows I have to restart sshd. But I even checked with lsof | fgrep -e DEL -e deleted: there is nothing which indicate that sshd needs to be restarted. It has been already restarted during the update.

Only after I restart the service on a host and run katello-tracer-upload again, it shows ‘No applications to restart’.

Expected outcome:
No traces.

Foreman and Proxy versions:
Foreman 3.10 with Katello 4.12, latest updates on AlmaLinux 8.

gvde · July 18, 2024, 6:28pm

I am really puzzled how katello determines the traces and which command I have to use to either see the traces on the host and update the traces to foreman. The way it is now it’s kind of useless if I have to restart everything one additional time.

jeremylenz · July 18, 2024, 6:50pm

Are you searching for hosts based on trace status? That’s broken in 4.12: Bug #37354: Trace_status = reboot_needed not working after upgrade to 4.12 - Katello - Foreman
It should be fixed in 4.13.0.

Individual traces should still show correctly.

Traces are uploaded to Katello (by default) after any dnf or yum package action, if katello-host-tools-tracer is installed. You will see this:

Installed products updated.
Uploading Tracer Profile

To force this upload, you can run

katello-tracer-upload

The traces you see in the web UI for the host should match what you see when you run

tracer

.

To confirm a tracer upload on the server, you can check in /var/log/foreman/production.log for

Started PUT "/rhsm/consumers/c6a16ade-ff65-4545-9e10-19955cf053a1/tracer"

(with your host’s consumer ID there).

gvde · July 23, 2024, 5:21am

di38rut
di38rut/sd9Pvr3L

I know the searching bug. It has nothing to do with that. It’s the status that is shown incorrectly on the host page.

For instance, I have a client host with docker updates pending. Trace status on the host page is O.K. That matches the status on the host:

[root@runner9 ~]# needs-restarting 
Updating Subscription Management repositories.
[root@runner9 ~]# needs-restarting -s
Updating Subscription Management repositories.
[root@runner9 ~]# needs-restarting -r
Updating Subscription Management repositories.
No core libraries or services have been updated since boot-up.
Reboot should not be necessary.
[root@runner9 ~]# tracer
[root@runner9 ~]# lsof | fgrep -e DEL -e deleted | fgrep -v -e ' /var/lib/sss/mc/' -e ' /memfd:'
[root@runner9 ~]# katello-tracer-upload 
Updating Subscription Management repositories.

Now I run the update. docker.service has been restarted during the update:

[root@runner9 ~]# systemctl status docker.service
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: d>
     Active: active (running) since Tue 2024-07-23 07:09:23 CEST; 12s ago
TriggeredBy: ● docker.socket
…
Jul 23 07:09:23 runner9.example.com systemd[1]: Started Docker Application Containe>

On the client host everything looks O.K:

[root@runner9 ~]# needs-restarting
Updating Subscription Management repositories.
[root@runner9 ~]# needs-restarting -s
Updating Subscription Management repositories.
[root@runner9 ~]# needs-restarting -r
Updating Subscription Management repositories.
No core libraries or services have been updated since boot-up.
Reboot should not be necessary.
[root@runner9 ~]# tracer
[root@runner9 ~]# lsof | fgrep -e DEL -e deleted | fgrep -v -e ' /var/lib/sss/mc/' -e ' /memfd:'
[root@runner9 ~]#

However, the trace status in foreman on the client host page says that docker needs to be restarted.

[root@foreman8 ~]# hammer host traces list --host runner9.example.com
---------|-------------|-------------------------------|-------
TRACE ID | APPLICATION | HELPER                        | TYPE  
---------|-------------|-------------------------------|-------
193703   | docker      | sudo systemctl restart docker | daemon
---------|-------------|-------------------------------|-------

Even if I run katello-trace-upload on the client again:

[root@runner9 ~]# katello-tracer-upload 
Updating Subscription Management repositories.
[root@runner9 ~]#

It still says to restart docker on foreman.

So no option to needs-restarting nor tracer says anything about a service restart. The service has been restarted during the update. Everything is fine on the client. Yet, foreman still thinks the service needs to be restarted.

Only after I restart docker.service again and run katello-tracer-upload, the trace status in foreman will be back to O.K. So it must be something else which determines why foreman thinks the service needs to be restarted…

jeremylenz · July 23, 2024, 3:15pm

If you have a host that exhibits this issue, can you go into foreman-rake console and try this:

[2] pry(main)> myhost = ::Host.find_by(name: 'runner9.example.com')

[3] pry(main)> myhost.host_traces
=> []
[4] pry(main)> myhost.host_traces.reload
=> []

Do you see the errant trace(s) on the first host_traces call?
Does the .reload fix it? (e.g. are the results different?)

gvde · July 23, 2024, 3:35pm

Unfortunately, it makes no difference:

# foreman-rake console
Loading production environment (Rails 6.1.7.8)
irb(main):001:0> myhost = ::Host.find_by(name: 'runner9.example.com')
=> #<Host::Managed id: 341, name: "runner9.example.com", last_compile: "2024-07...
irb(main):002:0> myhost.host_traces
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::HostTracer id: 193703, host_id: 341, application: "docker", helper: "sudo systemctl restart docker", app_type: "daemon">]>
irb(main):003:0> myhost.host_traces.reload
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::HostTracer id: 193703, host_id: 341, application: "docker", helper: "sudo systemctl restart docker", app_type: "daemon">]>
irb(main):004:0> myhost.host_traces
=> #<ActiveRecord::Associations::CollectionProxy [#<Katello::HostTracer id: 193703, host_id: 341, application: "docker", helper: "sudo systemctl restart docker", app_type: "daemon">]>
irb(main):005:0>

jeremylenz · July 23, 2024, 3:52pm

Hm, I thought it may have had something to do with preloading the records. But it appears not.

Does this happen only with docker traces, or with any trace?

gvde · July 23, 2024, 3:59pm

I have seen different services. I think httpd was one of them. But I can check the next updates to see when it happens. I have to wait for an update which automatically restarts the service unit…

Didn’t katello-tracer-upload used to have a ‘force’ option? I thought a “forced” upload might help but it doesn’t have that option (anymore?)…

gvde · July 26, 2024, 5:07am

O.K. Today, I installed updates of puppet-agent and it shows the same issue: the puppet.service has been restarted during the update. tracer and needs-restarting show nothing to do. katello-tracer-upload doesn’t change the trace status on the foreman server.

However, restarting and uploading does update the status:

# systemctl restart puppet
# katello-tracer-upload 
Updating Subscription Management repositories.
#

Now the trace status on foreman is good.

iballou · July 26, 2024, 2:09pm

Hi @gvde ,

I’m going to give reproducing this a try. In the meantime, would you like to try our new version of katello-host-tools-tracer that uses dnf needs-restarting instead of tracer? It might solve the problem, perhaps worth a try. Index of /client/nightly/el8/x86_64

iballou · July 26, 2024, 2:58pm

This part might be important – are you using a dnf plugin to achieve this? The normal workflow is to see the status in Foreman and then trigger the update yourself via REX through the UI. I wonder if the restart is happening in a way such that Tracer isn’t picking up that the service was restarted.

iballou · July 26, 2024, 3:02pm

Ah, perhaps via Changes/Restart services at end of rpm transaction - Fedora Project Wiki …

gvde · July 26, 2024, 3:07pm

No. It’s standard that an rpm which installs the service unit restarts the unit during update. docker, httpd, puppet-agent, sssd, etc. all install the service units and restart them automatically during update. Anything else would be surprising, I’d say.

Only when updating only libraries used by services there would be some services which need to be restarted.

But then shouldn’t tracer tell me what to do?

O.K. I’ll give it a try but actually I thought it would already do so. When I run katello-tracer-upload on AL9:

# katello-tracer-upload 
Updating Subscription Management repositories.

and adding option --help shows that it’s a dnf command dnf katello-tracer-upload.

iballou · July 26, 2024, 3:17pm

I just tested this myself via puppet-agent. After upgrading puppet-agent, tracer is reporting that puppet-agent needs a restart, but dnf needs-restarting does not. After upgrading to the newer katello-host-tools-tracer that uses dnf needs-restarting, I stopped seeing the issue.

I also realized this issue is why we switched to using dnf needs-restarting: tracer wants me to unnecessarily restart services · Issue #126 · FrostyX/tracer · GitHub

gvde · July 26, 2024, 3:19pm

O.K. It seems 4.4.0 version fixes the problem.

I have installed

on an Alma 9 server which currently has a trace to restart puppet.service.

With 4.4.0 installed I did

# katello-tracer-upload 
Updating Subscription Management repositories.

and now the trace status of the host is O.K. as it should be.

And then downgraded back to 4.2.3, ran katello-tracer-upload again and the puppet trace pops up again.

So with 4.2.3 it wants to restart puppet.service. With 4.4.0 it doesn’t which is correct.

I still don’t quite understand why tracer doesn’t show anything. I have looked through all the options to tracer and found that tracer -e or tracer -r does in fact list puppet:

# tracer -r
You should restart:
  * Some applications using:
      sudo systemctl restart puppet
# tracer -e
You should restart:
  * Some applications using:
      sudo systemctl restart puppet

So I guess katello-tracer-upload runs tracer with one of these options?

iballou · July 26, 2024, 3:25pm

I think not including -e or -r runs tracer within the context of the user running it, perhaps that’s why it’s not showing for you. In my case, I ran tracer as root.

Thanks for testing the nightly version, I’m glad it seems to help. We can look into getting it backported to the older client repos.

gvde · July 26, 2024, 3:27pm

In all my tests above I ran tracer as root. So that’s not it. But it’s good to know that this is going to be fixed in the foreseeable future.