When trying to use Katello for package updates: Host did not respond within 20 seconds

Support
1

Problem:
Can’t push updates to content hosts

Expected outcome:
Be able to push updates to content hosts

Foreman and Proxy versions:
foreman 1.23

Foreman and Proxy plugin versions:
katello 3.13

Distribution and version:
Katello-Foreman on Centos 7
Content host on Fedora 30

Other relevant data:
On the content host have the following installed:
katello-agent-3.5.1-1
katello-host-tools-tracer-3.5.1
katello-host-tools-3.5.1
katello-ca-consumer-… package
python3-gofer-proton-3.0.0.0.2
gofer-3.0.0.0-0.2
python3-gofer (same version)

Based on what I’ve googled - I checked and the CPU doesn’t spike (saw that as a reason for the 20 second issue)

Also, I’ve tried turning off SELinux and the firewall.

They’re both VMs (they can reach each other - the host is providing puppet feedback to Katello-Foreman) so I tried turning off firewall and VM on the host: no change.

The only thing that looks like maybe it could point to some kind of symptom:

[WARNING][MainThread] gofer.agent.plugin:642 - plugin:demo, DISABLED

Shows up in the systemctl status for goferd. Everything else looks like it should be good:

Dec 01 10:14:28 redyoshi.mushroomkingdom systemd[1]: Started Gofer Agent.
Dec 01 10:14:29 redyoshi.mushroomkingdom goferd[847]: [WARNING][MainThread] gofer.agent.plugin:642 - plugin:demo, DISABLED
Dec 01 10:14:29 redyoshi.mushroomkingdom goferd[847]: [INFO][Thread-1] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/demo
Dec 01 10:14:29 redyoshi.mushroomkingdom goferd[847]: [INFO][Thread-2] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/katello
Dec 01 10:14:30 redyoshi.mushroomkingdom goferd[847]: [INFO][MainThread] gofer.agent.plugin:687 - plugin:katello loaded using: katello.agent.goferd.plugin
Dec 01 10:14:35 redyoshi.mushroomkingdom goferd[847]: [INFO][MainThread] katello.agent.goferd.plugin:162 - Using /etc/rhsm/ca/katello-default-ca.pem as the ca cert for qpid connection
Dec 01 10:14:35 redyoshi.mushroomkingdom goferd[847]: [INFO][MainThread] gofer.agent.main:87 - agent started.

And in Katello-Foreman I can see the packages, errata, etc that need to be applied.

Ideas? Been trying various things for about 2 hours now, but it seems that each person I google has an idiosyncratic issue going on that doesn’t apply to my situation.

Thanks!

2

Hi @djotaku,

It does look like the client gofer logs are OK. Is everything looking alright in foreman-maintain service status especially with respect to pulp?

Can you tell me exactly how you are trying to push content to the host? Also, let me know any pertinent details from the record of trying to push content from the Monitor -> Tasks list.

3

Thanks for the help. On both the Katello-Foreman computer and the Content host I typed:

systemctl status foreman-maintain

And got a service not found error. Is that the command I was supposed to run? On which computer?

As for how I tried to push content. First I went to content host. On there it shows I have a bunch of installable updates - split by category. So I clicked the check box next to that content host. Then I clicked “Select Action”. Then “manage errata”. Click the checkbox on one of those. Then click “install selected” then “yes”.

Then it moves to the Task details screen.

Then…

I get state: “stopped” and result “warning”.

When I click on more details:

There’s where I would get the error. This time it just said “a subtask failed”

If instead of what I did above, I click on the link of all the errata, updates, etc. Then click on “Errata”, select the package and tell it to update ->

Then I get the error about the host not responding within 20 seconds. Is katello-agent installed and goferd running?

The pastebin of the backtrace is at https://pastebin.com/zhfXmang

4

You should run foreman-maintain service status on your foreman server itself.

Thank you for sharing the task details. It looks like the task created in the Pulp backend which issues/tracks the client system update isn’t completing. I recommend doing a foreman-maintain service restart regardless if the status output looks OK just to give everything a fresh boot. Let me know if anything changes.

5

I did a restart and it didn’t make any difference. Still getting the same error.

Attached is the output of the foreman-maintain service status command.

foremainmaintainstatus.log (264.6 KB)

two potential things I see that might help you:

Dec 04 20:27:49 katello.mushroomkingdom qdrouterd[12619]: 2019-12-04 20:27:49.882403 -0500 SERVER (info) [C4] Accepted connection to :5647 from 192.168.1.5:60948
Dec 04 20:27:49 katello.mushroomkingdom qdrouterd[12619]: 2019-12-04 20:27:49.920374 -0500 ROUTER (info) [C4] Connection Opened: dir=in host=192.168.1.5:60948 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=c3f68993-44ca-4069-b598-f812301af436 props=
Dec 04 20:27:49 katello.mushroomkingdom qdrouterd[12619]: 2019-12-04 20:27:49.923127 -0500 ROUTER (info) [C4][L14] Link attached: dir=out source={pulp.agent.3b7dec3d-12ba-41c7-8ef2-65eda75786ab expire:sess} target={ expire:sess}

Why is target none?

and

ec 04 20:26:31 katello.mushroomkingdom systemd[1]: Starting An AMQP message broker daemon…
Dec 04 20:26:32 katello.mushroomkingdom qpidd[12622]: 2019-12-04 20:26:32 [System] error Error reading socket: Encountered end of file [-5938]
Dec 04 20:26:32 katello.mushroomkingdom qpidd[12622]: 2019-12-04 20:26:32 [System] error Error reading socket: Encountered end of file [-5938]

6

Don’t know if this helps or not in terms of figuring out what’s up, but from the machine that I’m not able to push updates to - that machine is able to pull updates from Katello-Foreman via dnf update.

7

I’m spinning up a Fedora 30 client right now to see if I can reproduce :slight_smile:

8

Another data point to help → I installed all the packages needed on the Katello Server (running CentOS 7) and that successfully can push from Katello Foreman. In case that was unclear - it can push to itself.

katello%20works%20on%20itself
katello%20works%20on%20itself1366×768 87.3 KB

9

Very useful. Thanks for testing with a different OS!

I was able to reproduce the problem straight away on my fedora 30 client which is kind of reassuring. The katello-agent mechanism uses a queue , and the client will pull messages from that queue. I found that these messages are just sitting on the queue and not being read by the client. You can probably see the same on your end .

qpid-stat --ssl-certificate=/etc/pki/katello/qpid_router_client.crt --ssl-key=/etc/pki/katello/qpid_router_client.key -b amqps://localhost:5671 -q

I saw this:

  queue                                                                                  dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
  ======================================================================================================================================================================
<snip>
pulp.agent.4b72bfbb-4059-4e7f-ab72-8447cc4c500b                                        Y                      6     6      0    3.53k  3.53k       0         0     1

Six messages received, but 0 messages out. The pulp.agent.x pattern represents the UUID of the client:

[root@fedora30 vagrant]# subscription-manager identity
system identity: 4b72bfbb-4059-4e7f-ab72-8447cc4c500b

The katello-agent is a plugin to a larger system called goferd and Fedora is using a newer version - 3.0.0 compared to CentOS which is 2.12.x and I suspect that there is problem in gofer 3.x and/or its interaction with the katello-agent plugin. I need to reach out to the goferd developers for some insight.

10

You’re welcome! It’s not my first rodeo when it comes to trying to get good help from a forum so I always try and give as much info as I can. (As someone who supports some software, I certainly appreciate it when bug reports are similarly helpful)

Well, that’s, as you said, reassuring. I thought maybe I’d made some huge mistake in configuration. ALSO, thanks for reaching out to the goferd devs!

Cheers!

2 Likes
11

I’ve found the root of the problem and have a PR open with the fix. It could be a little while (1+ weeks) before the fix is merged and a new release it out, so you may be interested in manually patching your client.

See the changes to plugin.py here https://github.com/Katello/katello-host-tools/pull/104/files

On a Fedora 30 system the file to patch would be /usr/lib/python3.7/site-packages/katello/agent/goferd/plugin.py

12

Thank you very much for that. I’m OK waiting. What package is getting the patch, gofer?

13

You’re welcome! It’ll be a new version of katello-agent