celery.worker.consumer:ERROR consumer: Cannot connect to qpid://foreman.example.com:5671//: [Errno 111] Connection refused

katello_user · February 11, 2018, 7:28pm

Hi all,

I’m having an issue with katello.

I installed everything and created a product and synced the base centos7 repository. However qrouterd was not running. Later I found that is because of the issue with qdrouterd installation problem as can be seen here - QPID Broken Default Configuration - I used the /etc/qpid-dispatch/qdrouterd.conf file given by @stbenjam and changed the hostname sections in the file.

I noticed that there are errors like this:

Feb 11 10:20:10 foreman.example.com pulp[4442]: gofer.messaging.adapter.connect:ERROR: connect: qpid+ssl://localhost:5671, failed: [Errno 113] No route to host

Then I stumbled around one Red Hat kcs which told to edit the /etc/pulp/server.conf file as well.
Now, I am getting errors like this:

Feb 11 10:30:02 foreman pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Feb 11 10:30:02 foreman pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Feb 11 10:30:02 foreman pulp: celery.worker.consumer:ERROR: (6231-35712) consumer: Cannot connect to qpid://foreman.example.com:5671//: [Errno 111] Connection refused.
Feb 11 10:30:02 foreman pulp: celery.worker.consumer:ERROR: (6231-35712) Trying again in 32.00 seconds...
Feb 11 10:30:02 foreman pulp: celery.worker.consumer:ERROR: (6231-35712)
Feb 11 10:30:02 foreman pulp: celery.worker.consumer:ERROR: (6256-29856) consumer: Cannot connect to qpid://foreman.example.com:5671//: [Errno 111] Connection refused.
Feb 11 10:30:02 foreman pulp: celery.worker.consumer:ERROR: (6256-29856) Trying again in 32.00 seconds...
Feb 11 10:30:02 foreman pulp: celery.worker.consumer:ERROR: (6256-29856)
Feb 11 10:30:02 foreman pulp: celery.beat:ERROR: beat: Connection error: [Errno 111] Connection refused. Trying again in 32.0 seconds...
Feb 11 10:30:07 foreman pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Feb 11 10:30:07 foreman pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.

And when I check the content page in the web UI I am getting:

 Oops, we're sorry but something went wrong Katello::Resources::Candlepin::Owner: 503 Service Unavailable {"displayMessage":"Candlepin is in Suspend mode, please check /status resource to get more details","requestUuid":"b6d62b45-84f7-47ea-b356-092e2a2b8b49"} (GET /candlepin/owners/Default_Organization)

This looks like something easy that I am not understanding. There are quite a few moving parts in the entire setup and I thought I would try to get any answer here. I can blow this and start from scratch but it will be great to fix this as I have already synced a full repository.

Thanks!

katello_user · February 11, 2018, 7:29pm

I could not post the config files in the earlier reply due to new user limitation:

[root@foreman ~]# grep -v "#" /etc/pulp/server.conf

[database]
name: pulp_database
seeds: localhost:27017
ssl: false
unsafe_autoretry: false

[server]
server_name: foreman.example.com
key_url: /pulp/gpg
ks_url: /pulp/ks
default_login: admin
default_password: xxxx
debugging_mode: false
log_level: INFO

[authentication]
rsa_key = /etc/pki/pulp/rsa.key
rsa_pub = /etc/pki/pulp/rsa_pub.key

[security]
cacert: /etc/pki/pulp/ca.crt
cakey: /etc/pki/pulp/ca.key
ssl_ca_certificate: /etc/pki/pulp/ca.crt
user_cert_expiration: 7
consumer_cert_expiration: 3650
serial_number_path: /var/lib/pulp/sn.dat

[consumer_history]
lifetime: 180

[data_reaping]
reaper_interval: 0.25
archived_calls: 0.5
consumer_history: 60
repo_sync_history: 60
repo_publish_history: 60
repo_group_publish_history: 60
task_status_history: 7
task_result_history: 3

[oauth]
enabled: true
oauth_key: katello
oauth_secret: xxxxxx

[messaging]
url: ssl://foreman.example.com:5671
transport: qpid
auth_enabled: false
cacert: /etc/pki/katello/certs/katello-default-ca.crt
clientcert: /etc/pki/katello/qpid_client_striped.crt
topic_exchange: 'amq.topic'
event_notifications_enabled: false

[tasks]
broker_url: qpid://foreman.example.com:5671
celery_require_ssl: true
cacert: /etc/pki/katello/certs/katello-default-ca.crt
keyfile: /etc/pki/katello/qpid_client_striped.crt
certfile: /etc/pki/katello/qpid_client_striped.crt

[email]
host: localhost
port: 25
from: no-reply@example.com
enabled: false

[lazy]
redirect_host: foreman.example.com
redirect_path: /streamer/
https_retrieval: false
download_interval: 10
download_concurrency: 5

[profiling]
enable: false
directory: /var/lib/pulp/c_profiles
[root@foreman ~]#

And this is the qdrouterd.conf file:

[root@foreman ~]# cat /etc/qpid-dispatch/qdrouterd.conf


router {
id: centos7-katello-3-5.zpm.example.com
mode: interior
worker-threads: 2
}

ssl-profile {
name: client
cert-db: /etc/pki/katello/certs/katello-default-ca.crt
cert-file: /etc/pki/katello/qpid_router_client.crt
key-file: /etc/pki/katello/qpid_router_client.key
}

ssl-profile {
name: server
cert-db: /etc/pki/katello/certs/katello-default-ca.crt
cert-file: /etc/pki/katello/qpid_router_server.crt
key-file: /etc/pki/katello/qpid_router_server.key
}

connector {
name: broker
host: foreman.example.com
port: 5671
sasl-mechanisms: ANONYMOUS
role: route-container
ssl-profile: client
idle-timeout-seconds: 0
}

linkRoute {
prefix: pulp.
dir: out
connection: broker
}
linkRoute {
prefix: pulp.task
dir: in
connection: broker
}
linkRoute {
prefix: qmf.
dir: in
connection: broker
}
linkRoute {
prefix: qmf.
dir: out
connection: broker
}
listener {
host: 0.0.0.0
port: 5647
sasl-mechanisms: ANONYMOUS
ssl-profile: server
}

listener {
host: 0.0.0.0
port: 5646
sasl-mechanisms: ANONYMOUS
role: inter-router
ssl-profile: server
}

log {
module: DEFAULT
enable: info+
timestamp: true
output: /var/log/qdrouterd/qdrouterd.log
}
address {
prefix: closest
distribution: closest
}

address {
prefix: multicast
distribution: multicast
}

address {
prefix: unicast
distribution: closest
}

address {
prefix: exclusive
distribution: closest
}

address {
prefix: broadcast
distribution: multicast
}

bcochofel · February 14, 2018, 10:15am

Hi, thanks for the quick answer.

This configuration seens to be for the server, right? My server is ok, the proxy it’s the one that doesn’t start qdrouterd.

Thanks,
Bruno

bcochofel · February 14, 2018, 10:34am

Hi,

Sorry, seens I’ve replied on the wrong thread.

Bruno

imphocused · February 14, 2018, 1:13pm

You’ve confirmed the service is running on that port?

[root@katello ~]# lsof -Pi:5671 |grep LISTEN
qpidd     31628     qpidd   25u  IPv4  70005      0t0  TCP localhost:5671 (LISTEN)
qpidd     31628     qpidd   27u  IPv6  70008      0t0  TCP localhost:5671 (LISTEN)
    
[root@katello ~]# systemctl status qpidd
    ● qpidd.service - An AMQP message broker daemon.
       Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
       Active: active (running) since Wed 2018-02-14 02:34:17 EST; 5h 36min ago
         Docs: man:qpidd(1)
               http://qpid.apache.org/
     Main PID: 31628 (qpidd)
       CGroup: /system.slice/qpidd.service
               └─31628 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

    Feb 14 02:34:17 katello.domain.com systemd[1]: Started An AMQP message broker daemon..
    Feb 14 02:34:17 katello.domain.com systemd[1]: Starting An AMQP message broker daemon....
    [root@katello ~]#

In the replacement ‘qdrouterd.conf’ file the only line necessary to change is the hostname on line 24:

router {
    id: katello.domain.com
    mode: interior
    worker-threads: 2
}

ruzzetto · February 14, 2018, 1:26pm

Hi,
I’m facing the same issue. My client is trying to communicate over tcp 5647 port but it seems no sockets are listening on that port. But on 5671 port i have a lot of opened socket.

katello_user · February 15, 2018, 3:05pm

Hi, thanks for the reply. Yes, to test the functionality I made the change in the replacement /etc/qpid-dispatch/qdrouterd.conf file.

The entire output is captured here: https://paste.fedoraproject.org/paste/gBa-mZdDCCfSavsY4Z11iQ

FWIW,

[root@foremankatellodemo log]# hostname
foremankatellodemo.example.com
[root@foremankatellodemo log]#

[root@foremankatellodemo log]# egrep -v ‘^(^#|$)’ /etc/qpid-dispatch/qdouterd.conf
router {
id: foremankatellodemo.example.com
mode: interior
worker-threads: 2
}
ssl-profile {
name: client
cert-db: /etc/pki/katello/certs/katello-default-ca.crt
cert-file: /etc/pki/katello/qpid_router_client.crt
key-file: /etc/pki/katello/qpid_router_client.key
}
ssl-profile {
name: server
cert-db: /etc/pki/katello/certs/katello-default-ca.crt
cert-file: /etc/pki/katello/qpid_router_server.crt
key-file: /etc/pki/katello/qpid_router_server.key
}
connector {
name: broker
host: localhost
port: 5671
sasl-mechanisms: ANONYMOUS
role: route-container
ssl-profile: client
idle-timeout-seconds: 0
}
linkRoute {
prefix: pulp.
dir: out
connection: broker
}
linkRoute {
prefix: pulp.task
dir: in
connection: broker
}
linkRoute {
prefix: qmf.
dir: in
connection: broker
}
linkRoute {
prefix: qmf.
dir: out
connection: broker
}
listener {
host: 0.0.0.0
port: 5647
sasl-mechanisms: ANONYMOUS
ssl-profile: server
}
listener {
host: 0.0.0.0
port: 5646
sasl-mechanisms: ANONYMOUS
role: inter-router
ssl-profile: server
}
log {
module: DEFAULT
enable: info+
timestamp: true
output: /var/log/qdrouterd/qdrouterd.log
}
address {
prefix: closest
distribution: closest
}
address {
prefix: multicast
distribution: multicast
}
address {
prefix: unicast
distribution: closest
}
address {
prefix: exclusive
distribution: closest
}
address {
prefix: broadcast
distribution: multicast
}
[root@foremankatellodemo log]#

Version of components:

[root@foremankatellodemo log]# rpm -qa | grep -i qpid
qpid-cpp-server-linearstore-1.37.0-1.el7.x86_64
qpid-dispatch-router-1.0.0-1.el7.x86_64
qpid-qmf-1.37.0-1.el7.x86_64
qpid-proton-c-0.18.1-1.el7.x86_64
python-qpid-proton-0.18.1-1.el7.x86_64
qpid-cpp-server-1.37.0-1.el7.x86_64
qpid-cpp-client-1.37.0-1.el7.x86_64
tfm-rubygem-qpid_messaging-0.34.1-1.el7.x86_64
qpid-cpp-client-devel-1.37.0-1.el7.x86_64
python-qpid-qmf-1.37.0-1.el7.x86_64
foremankatellodemo.example.com-qpid-router-server-1.0-1.noarch
python-gofer-qpid-2.7.6-1.el7.noarch
qpid-tools-1.37.0-1.el7.noarch
foremankatellodemo.example.com-qpid-router-client-1.0-1.noarch
foremankatellodemo.example.com-qpid-client-cert-1.0-1.noarch
python-qpid-1.37.0-1.el7.noarch
foremankatellodemo.example.com-qpid-broker-1.0-1.noarch
[root@foremankatellodemo log]#

[root@foremankatellodemo log]# rpm -qa | grep -i katello
katello-installer-base-3.4.5-1.el7.noarch
foremankatellodemo.example.com-foreman-client-1.0-1.noarch
foremankatellodemo.example.com-puppet-client-1.0-1.noarch
katello-repos-3.4.0-3.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
foremankatellodemo.example.com-foreman-proxy-client-1.0-1.noarch
katello-service-3.4.5-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
katello-server-ca-1.0-1.noarch
tfm-rubygem-katello-3.4.5-1.el7.noarch
foremankatellodemo.example.com-qpid-router-server-1.0-1.noarch
foremankatellodemo.example.com-tomcat-1.0-1.noarch
katello-selinux-3.0.2-1.el7.noarch
katello-debug-3.4.5-1.el7.noarch
katello-common-3.4.5-1.el7.noarch
katello-3.4.5-1.el7.noarch
foremankatellodemo.example.com-apache-1.0-1.noarch
katello-client-bootstrap-1.4.0-2.el7.noarch
foremankatellodemo.example.com-qpid-router-client-1.0-1.noarch
foremankatellodemo.example.com-qpid-client-cert-1.0-1.noarch
foremankatellodemo.example.com-foreman-proxy-1.0-1.noarch
foremankatellodemo.example.com-qpid-broker-1.0-1.noarch
foreman-installer-katello-3.4.5-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.10.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
[root@foremankatellodemo log]#

Logs:

[root@foremankatellodemo log]# journalctl -u qdrouterd
– Logs begin at Thu 2018-02-15 08:26:52 UTC, end at Thu 2018-02-15 14:50:57 UTC. –
Feb 15 08:27:03 foremankatellodemo.example.com systemd[1]: Started Qpid Dispatch router daemon.
Feb 15 08:27:03 foremankatellodemo.example.com systemd[1]: Starting Qpid Dispatch router daemon…
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: 2018-02-15 08:27:08.455174 +0000 ERROR (error) Python: Exception: Cannot load configura
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: 2018-02-15 08:27:08.478911 +0000 ERROR (error) Traceback (most recent call last):
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/config.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: config = Config(filename)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/config.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: self.load(filename, raw_json)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/config.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: self.load(f, raw_json)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/config.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: self.schema.validate_all(entities)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/schema.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: self.validate_add(a, entities)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/qdrouter.py”, lin
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: super(QdSchema, self).validate_add(attributes, entities)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/schema.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: self.validate_entity(attributes)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/schema.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: entity_type = self.entity_type(attributes[‘type’])
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/schema.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: return self._lookup(self.entity_types, name, “No such entity type ‘%s’”, error)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: File “/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/schema.py”, line
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: raise ValidationError(message % name)
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: Exception: Cannot load configuration file /etc/qpid-dispatch/qdrouterd.conf: No such en
Feb 15 08:27:08 foremankatellodemo.example.com qdrouterd[656]: 2018-02-15 08:27:08.482662 +0000 MAIN (critical) Router start-up failed: Python: Except
Feb 15 08:27:08 foremankatellodemo.example.com systemd[1]: qdrouterd.service: main process exited, code=exited, status=1/FAILURE
Feb 15 08:27:08 foremankatellodemo.example.com systemd[1]: Unit qdrouterd.service entered failed state.
Feb 15 08:27:08 foremankatellodemo.example.com systemd[1]: qdrouterd.service failed.
Feb 15 08:35:27 foremankatellodemo.example.com systemd[1]: Started Qpid Dispatch router daemon.
Feb 15 08:35:27 foremankatellodemo.example.com systemd[1]: Starting Qpid Dispatch router daemon…
Feb 15 08:35:27 foremankatellodemo.example.com systemd[1]: qdrouterd.service: main process exited, code=exited, status=1/FAILURE
Feb 15 08:35:27 foremankatellodemo.example.com systemd[1]: Unit qdrouterd.service entered failed state.
Feb 15 08:35:27 foremankatellodemo.example.com systemd[1]: qdrouterd.service failed.
Feb 15 14:46:52 foremankatellodemo.example.com systemd[1]: Started Qpid Dispatch router daemon.
Feb 15 14:46:52 foremankatellodemo.example.com systemd[1]: Starting Qpid Dispatch router daemon…
Feb 15 14:46:53 foremankatellodemo.example.com systemd[1]: qdrouterd.service: main process exited, code=exited, status=1/FAILURE
Feb 15 14:46:53 foremankatellodemo.example.com systemd[1]: Unit qdrouterd.service entered failed state.
Feb 15 14:46:53 foremankatellodemo.example.com systemd[1]: qdrouterd.service failed.
Feb 15 14:47:22 foremankatellodemo.example.com systemd[1]: Started Qpid Dispatch router daemon.
Feb 15 14:47:22 foremankatellodemo.example.com systemd[1]: Starting Qpid Dispatch router daemon…
Feb 15 14:47:22 foremankatellodemo.example.com qdrouterd[6386]: 2018-02-15 14:47:22.801640 +0000 ERROR (error) Not found: Configuration file could not
Feb 15 14:47:22 foremankatellodemo.example.com qdrouterd[6386]: 2018-02-15 14:47:22.801676 +0000 MAIN (critical) Router start-up failed: Not found: Co
Feb 15 14:47:22 foremankatellodemo.example.com systemd[1]: qdrouterd.service: main process exited, code=exited, status=1/FAILURE
Feb 15 14:47:22 foremankatellodemo.example.com systemd[1]: Unit qdrouterd.service entered failed state.
Feb 15 14:47:22 foremankatellodemo.example.com systemd[1]: qdrouterd.service failed.
Feb 15 14:48:45 foremankatellodemo.example.com systemd[1]: Started Qpid Dispatch router daemon.
Feb 15 14:48:45 foremankatellodemo.example.com systemd[1]: Starting Qpid Dispatch router daemon…
Feb 15 14:48:45 foremankatellodemo.example.com systemd[1]: qdrouterd.service: main process exited, code=exited, status=1/FAILURE
Feb 15 14:48:45 foremankatellodemo.example.com systemd[1]: Unit qdrouterd.service entered failed state.
Feb 15 14:48:45 foremankatellodemo.example.com systemd[1]: qdrouterd.service failed.
Feb 15 14:50:57 foremankatellodemo.example.com systemd[1]: Started Qpid Dispatch router daemon.
Feb 15 14:50:57 foremankatellodemo.example.com systemd[1]: Starting Qpid Dispatch router daemon…
Feb 15 14:50:57 foremankatellodemo.example.com systemd[1]: qdrouterd.service: main process exited, code=exited, status=1/FAILURE
Feb 15 14:50:57 foremankatellodemo.example.com systemd[1]: Unit qdrouterd.service entered failed state.
Feb 15 14:50:57 foremankatellodemo.example.com systemd[1]: qdrouterd.service failed.
[root@foremankatellodemo log]#

Any thoughts?

James_Shewey · February 15, 2018, 4:56pm

Is the firewalld service running?