Problem:
When I try to apply an errata to one of my host I get the following error: (foreman) Host did not respond within 20 sec. The task has been cancelled. Is katello-agent installed and is goferd running on the host.
I ran a tcpdump and nc -vvv katello.server 5647 to verify that the host and server had communication on port 5647.
Is see on the (client) that there is a goferd error qd: no-route-to-dest. The server goferd is also showing the same error. I’ve been digging around but am unsure were to go from here. I wondered if anyone had run into this and had any thoughts. The network seems good so it seems that it might be an issue with this Katello server but this is only a guess.
Expected outcome:
That I be able to apply packages and errata patches with out error.
Foreman and Proxy versions:
Foreman version 1.16
IPA 1.16.2
Katello_agent 3.1.0-1
Foreman and Proxy plugin versions:
Other relevant data:
Backend system status was ok on the following:
candlepin
candlepin_auth
foreman_task
pulp
pulp_auth
The specific goferd error is:
goferd.messaging.adapter.proton.reliabitity:47 qd: no-route-to-dest
I am get the following error from the goferd agent:
gofer-messaging.adapter.connect:33 - connect+amqps://server:5647 : failed Connection amaps://server:5647 disconnected: condition (‘proton:io’, 'recv:connection refused)
The katello server is running on a VM. We have a very flat network and I believe the switch is wide open.
the goferd error is appearing on the client, katello server and VM host.
Is the qpid router daemon on the server side up and running? Are you able to connect via telnet from client to the qpid router process on your foreman server from?
the qpidd is running, however I see the following error:
server:dq couldn’t setup next timer firing: -45818147ns
Telnet is disabled however I have tried ncat (nc -vvv servername 5647)
From both the server and the client with no luck.
also getting the following:
connection qpid.127.0.0.1:5671-127.0.0.1:52108 time out closing.
Not using satellite but foreman
Also I can not see the RH knowledge base solution, I don’t have a subscription.
The error is appearing in the qpidd logs for the service. I appears that qpidd cannot run or is not functioning.
I wondered how I could test?
Per the bugzilla about the timing errors “This sequence of events does not affect the operation of the timer in any way, and is benign (other than the confusing error messages that show up in the broker log file).” Your problem is probably elsewhere.
Is qdrouterd service running? (systemctl status qdrouterd) Do you have a Firewall enabled blocking connections?
If you don’t have the telnet client utility available you can try something like this: openssl s_client -connect example.com:5647 to see if you can connect to the port.
You were right qdrouterd was down. I tried a test patch and got the following:
There was an issue with the backend server PULP: Not all necessary Pulp workers running.
Ruth
You have pulp services that aren’t running, check katello-service status, and restart whatever ones that show an error. How much memory does your Katello server have?
OK, I tested the connection with openssl and it appears that the client can speck to the katello server. However going back the other way it appears that the connection is blocked. I was wondering about the correct iptables setttings for the client. I see that they are set on the server but the client has nothing.
When I test a errata update I get a subtask failed error. Digging on down I see that the process has failed when it trying to connect to katello and goferd.
The qpidd process is throwing a an error on attach:no source specified.
The total memory (RAM) is 16267032m free space is 960476m free
There are a couple of other things running on the server.
The pulp workers all appear to be running.
I wo[quote=“RuthBishop, post:15, topic:10524, full:true”]
The total memory (RAM) is 16267032m free space is 960476m free
There are a couple of other things running on the server.
The pulp workers all appear to be running.
[/quote]
I would go with a firewall issue. We are running SElinux, however I turned off hte firewalld so that should not be an issue. Iptables on the client not configured so should be wide open.
I shut off firewalld and iptables and got the same error. It looks like its failing at:
Actions:: Plup::consumer::contentinstall
Ruth
I now have connectivity both ways using openssl, the problem was in the iptables configuration. I still, however have an issue with the errata timing out.
The error is still that the host did not respond within 20 seconds…
Process is failing at Actions::Katello:Host:Erratum::ApplicationErrataInstall
Error is still subtask failed.
I have found the solution to the issue. We were running Fedoria 26 which I believe is no longer supported. There were some dependency conflicts which were not allowing the errata to apply. Please see RHBA-2018:0681 - Bug Fix Advisory. I was unable to find a patch that would resolve the issue on Fedoria 26 so we upgrade to Fedoria 28 which solved the problem. The dependancy was between python-rhsm and subscription-manager.
Thanks for the help.
Ruth
