My server is;
LSB Version:
:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.6 (Final)
Release: 6.6
Codename: Final
Katello is @ 2.2.2
Foreman is @ Version 1.8.2
I first noticed that Task queue filling up with these tasks and they ar
egoing into suspend status on dynflow;
ActionStateResult▲ Started at
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks?order=started_at+ASC>
UserListen on candlepin events {"locale"=>"en"}
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/76500b51-2ab6-4241-9287-4f37500eb1d1>
runningpending2015-07-16 14:51:36 -0600Listen on candlepin events
{"locale"=>"en"}
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/fe0e77d4-d75a-467b-bffb-57e2ed7ffd73>
stoppedsuccess2015-07-16 14:39:18 -0600Listen on candlepin events
{"locale"=>"en"}
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/1b9ee3bf-4299-4ee4-8ded-4a8fed61f889>
pausedpending2015-07-16 12:56:55 -0600Update package ; system
'ms-linux-test'; organization 'ms-linux'
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/df512049-d3c1-4116-a1a2-7d8cbdab5a83>
pausedpending2015-07-16 12:42:41 -0600adminListen on candlepin events
{"locale"=>"en"}
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/013f0ddf-c210-4a91-8e88-8f27b98addb0>
stoppedsuccess2015-07-16 12:38:36 -0600Listen on candlepin events
{"locale"=>"en"}
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/ddf63f92-abaf-4cbf-966f-74815b6cf2a7>
stoppedsuccess2015-07-16 12:07:38 -0600Generate applicability
{"system_ids"=>[5], "locale"=>"en"}
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/1de81b28-3b7a-498d-a635-2291d40e9b46>
stoppedsuccess2015-07-16 11:35:53 -0600foreman_adminUpdate system
'ms-linux-test'; organization 'ms-linux'
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/abe4f552-f940-4c7d-9c11-17eae96166b0>
stoppedsuccess2015-07-16 11:35:43 -0600foreman_adminListen on candlepin
events {"locale"=>"en"}
<https://ms-linux-tools.msimanagedservices.local/foreman_tasks/tasks/39f03de7-7860-4717-967b-11d64be92f07>
pausedpending2015-07-16 10:52:58 -0600
katello-service status shows qpidd failed;
ms-linux-tools:root# katello-service status
qdrouterd (pid 7539) is running…
mongod (pid 8151) is running…
tomcat6 (pid 8005) is running… [ OK ]
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
pulp_celerybeat (pid 7732) is running.
celery init v10.0.
Using config script: /etc/default/pulp_workers
node reserved_resource_worker-0 (pid 7654) is running…
foreman-proxy (pid 7693) is running…
celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
node resource_manager (pid 8227) is running…
elasticsearch (pid 7786) is running…
httpd (pid 8289) is running…
dynflow_executor is running.
dynflow_executor_monitor is running.
Some services failed: qpidd
ms-linux-tools:root#
I found these error messages in /var/log/candlepin/error.log;
2015-07-16 16:47:48,013 [=, org=] WARN
org.apache.qpid.transport.network.security.ssl.SSLUtil - Exception
received while trying to verify hostname
2015-07-16 16:47:52,225 [=, org=] WARN org.hibernate.id.UUIDHexGenerator -
HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate
IETF RFC 4122 compliant UUID values; consider using
org.hibernate.id.UUIDGenerator instead
Also found similar messages in gutterball's log;
2015-07-16 16:48:34,897 [IoReceiver - MYFQDN/10.0.0.11:5671] WARN
org.apache.qpid.transport.network.security.ssl.SSLUtil - Exception
received while trying to verify hostname
2015-07-16 16:48:35,558 [gutterball-qpid-connect] INFO
org.candlepin.gutterball.receiver.EventReceiver - Receiver init complete
I found these two bugs;
https://bugzilla.redhat.com/show_bug.cgi?id=1193547
https://bugzilla.redhat.com/show_bug.cgi?id=1190040
Although I am not sure they apply to Katello 2.2.2. I verified my host can
resolve its fqdn and it also matches the output of hostname -f. Also
checked all the instances of my FQDN in /etc/candlepin/candlepin.conf
and /etc/gutterball/gutterball.conf to make sure they are correct.
This is basically stopping content host updates. I am not sure what to
look at next or that I am even looking at the right thing. Any
help/suggestions are greatly appreciated
Thanks for taking the time to read this,
Michael Bay
(not the director, just another linux engineer)