[Katello 2.2] goferd: disconnected

[JC1]

The katello-agent and subscription manager install on a the (CentOS6)
client, and the client can use yum to update itself from Katello, but tasks
triggered from Katello fail.

Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the Host?

I've implemented:

http://projects.theforeman.org/issues/10350
/ https://bugzilla.redhat.com/show_bug.cgi?id=1217828

ll /etc/pki/katello/qpid_router*

-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32
/etc/pki/katello/qpid_router_client.crt
-rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31
/etc/pki/katello/qpid_router_client.key
-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31
/etc/pki/katello/qpid_router_server.crt
-rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31
/etc/pki/katello/qpid_router_server.key

But don't see an improvement. I'm also not using ipv6*, so haven't
modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did).

*Well the 'advanced' section for the content host within Katello shows ::1
(localhost).

Of course I do see, on the Katello server, 5647 being listened to and the
firewall is open (and turned off for testing on the client):

netstat -pntl|grep 5647

tcp 0 0 0.0.0.0:5647 0.0.0.0:*
LISTEN 48646/qdrouterd

iptables -nL | grep 5647

ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 multiport
dports 22,80,443,5647,5671,8140

tcpdump shows a short conversation between server and client:

IP myserver.50035 > mytestclient.5647: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 304
IP mytestclient.5647 > myserver.50035: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 3665
IP myserver.50035 > mytestclient.5647: tcp 0

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://myserver.mydomain:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 - connecting
to myserver.mydomain:5647…

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 -
Disconnected

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
amqps://myserver.mydomain:5647

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call
last):

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py",
line 102, in open

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - self._impl =
BlockingConnection(url, ssl_domain=domain)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 200, in init

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - msg="Opening
connection")

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 231, in wait

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - self.container.process()

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3729, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
ev.dispatch(self.handler)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3654, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - result =
dispatch(handler, type.method, self)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3543, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - return m(*args)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in
on_transport_tail_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
self.on_transport_closed(event)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in
on_transport_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise
ConnectionException("Connection %s disconnected" % self.url);

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - ConnectionException:
Connection amqps://myserver.mydomain:5647 disconnected

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 29 seconds

Can anyone help? Many thanks.

[JC1]

Is anyone able to help?

Having got the Katello server to the brink of deployment into production,
it's frustrating to not be able to proceed.

···

On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote: > > The katello-agent and subscription manager install on a the (CentOS6) > client, and the client can use yum to update itself from Katello, but tasks > triggered from Katello fail. > > Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the Host? > > > I've implemented: > > http://projects.theforeman.org/issues/10350 / > https://bugzilla.redhat.com/show_bug.cgi?id=1217828 > > # ll /etc/pki/katello/qpid_router* > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32 > /etc/pki/katello/qpid_router_client.crt > -rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31 > /etc/pki/katello/qpid_router_client.key > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31 > /etc/pki/katello/qpid_router_server.crt > -rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31 > /etc/pki/katello/qpid_router_server.key > > But don't see an improvement. I'm also not using ipv6*, so haven't > modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did). > > *Well the 'advanced' section for the content host within Katello shows ::1 > (localhost). > > Of course I do see, on the Katello server, 5647 being listened to and the > firewall is open (and turned off for testing on the client): > > # netstat -pntl|grep 5647 > tcp 0 0 0.0.0.0:5647 0.0.0.0:* > LISTEN 48646/qdrouterd > > # iptables -nL | grep 5647 > ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport > dports 22,80,443,5647,5671,8140 > > tcpdump shows a short conversation between server and client: > > IP myserver.50035 > mytestclient.5647: tcp 0 > IP mytestclient.5647 > myserver.50035: tcp 0 > IP myserver.50035 > mytestclient.5647: tcp 0 > IP myserver.50035 > mytestclient.5647: tcp 304 > IP mytestclient.5647 > myserver.50035: tcp 0 > IP mytestclient.5647 > myserver.50035: tcp 3665 > IP myserver.50035 > mytestclient.5647: tcp 0 > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > amqps://myserver.mydomain:5647|SSL: ca: > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > /etc/pki/consumer/bundle.pem|host-validation: None > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 - > connecting to myserver.mydomain:5647... > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 - > Disconnected > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > amqps://myserver.mydomain:5647 > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call > last): > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 102, in open > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - self._impl = > BlockingConnection(url, ssl_domain=domain) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 200, in __init__ > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - msg="Opening > connection") > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 231, in wait > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - self.container.process() > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3729, in > dispatch > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > ev.dispatch(self.handler) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3654, in > dispatch > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - result = > dispatch(handler, type.method, self) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3543, in > dispatch > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - return m(*args) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in > on_transport_tail_closed > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > self.on_transport_closed(event) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in > on_transport_closed > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - raise > ConnectionException("Connection %s disconnected" % self.url); > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - ConnectionException: > Connection amqps://myserver.mydomain:5647 disconnected > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:108 - retry in 29 seconds > > > Can anyone help? Many thanks. >

[JC1]

OK, strace showed a problem writing to the log file. Perms changed and now
we have some output…

cat /var/log/qdrouterd.log

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/DISPATCH, identity=log/DISPATCH,
type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_HELLO)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/SERVER,
identity=log/SERVER, type=org.apache.qpid.dispatch.log, enable=default,
module=SERVER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONTAINER, identity=log/CONTAINER,
type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/AGENT,
identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default,
module=AGENT)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ERROR,
identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default,
module=ERROR)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ROUTER,
identity=log/ROUTER, type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/MESSAGE,
identity=log/MESSAGE, type=org.apache.qpid.dispatch.log, enable=default,
module=MESSAGE)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/CONFIG,
identity=log/CONFIG, type=org.apache.qpid.dispatch.log, enable=default,
module=CONFIG)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=container/katello.server, identity=container/katello.server,
type=org.apache.qpid.dispatch.container, containerName=katello.server,
workerThreads=2)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4,
helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60,
routerId=katello.server, raInterval=30, mode=interior, mobileAddrMaxAge=60)

Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server

Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior mode,
area=0 id=katello.server

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16,
typeSize=2104, transferBatchSize=16, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_log_entry_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=16,
localFreeListMax=32)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=88, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=32, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_item_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_node_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=136, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_bitmask_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_timer_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=216, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_address_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=16, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0,
deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False,
deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0,
deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False,
deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0,
deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False,
deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdrouter

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdrouter.ma

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdhello

Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated:
id=katello.server instance=1439983963 max_routers=1024

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/0, identity=fixedAddress/0,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/closest,
bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/closest
phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/1, identity=fixedAddress/1,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/unicast,
bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/unicast
phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/2, identity=fixedAddress/2,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single,
prefix=/exclusive, bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/3, identity=fixedAddress/3,
type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple,
prefix=/multicast, bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/multicast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/4, identity=fixedAddress/4,
type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple,
prefix=/broadcast, bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/broadcast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/5, identity=fixedAddress/5,
type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/,
bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ phase=0
fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=listener/0.0.0.0:5647, identity=listener/0.0.0.0:5647,
type=org.apache.qpid.dispatch.listener, requirePeerAuth=True,
allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS,
maxFrameSize=65536, role=normal,
certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False,
certFile=/etc/pki/katello/qpid_router_server.crt,
keyFile=/etc/pki/katello/qpid_router_server.key, port=5647)

Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647
role=normal

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=listener/0.0.0.0:5646, identity=listener/0.0.0.0:5646,
type=org.apache.qpid.dispatch.listener,
certFile=/etc/pki/katello/qpid_router_server.crt, addr=0.0.0.0,
saslMechanisms=ANONYMOUS, maxFrameSize=65536, requirePeerAuth=True,
certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False,
allowNoSasl=False, role=inter-router,
keyFile=/etc/pki/katello/qpid_router_server.key, port=5646)

Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5646
role=inter-router

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=broker,
identity=connector/katello.server:5671,
type=org.apache.qpid.dispatch.connector,
certFile=/etc/pki/katello/qpid_router_client.crt, addr=katello.server,
allowRedirect=True, saslMechanisms=ANONYMOUS, maxFrameSize=65536,
role=on-demand, certDb=/etc/pki/katello/certs/katello-default-ca.crt,
keyFile=/etc/pki/katello/qpid_router_client.key, port=5671)

Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured on-demand connector:
katello.server:5671 name=broker

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=linkRoutePattern/0, identity=linkRoutePattern/0,
type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker,
prefix=pulp.)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern:
prefix=pulp. connector=broker

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=linkRoutePattern/1, identity=linkRoutePattern/1,
type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker,
prefix=qmf.)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern:
prefix=qmf. connector=broker

Wed Aug 19 12:32:43 2015 SERVER (info) Operational, 2 Threads Running

Wed Aug 19 12:32:43 2015 CONN_MGR (info) Starting on-demand connector:
broker

Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'pulp.' for routed
links to 'broker'

Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'qmf.' for routed
links to 'broker'

Running a tcpdump capture on the client and katello server after asking for
Katello to 'update-all' or install one package shows no traffic.

/var/log/messages only shows:

Aug 19 11:24:44 katello.server pulp:
gofer.messaging.adapter.qpid.connection:INFO: connecting: URL:
ssl://katello.server|SSL: ca:
/etc/pki/katello/certs/katello-default-ca.crt|key: None|certificate:
/etc/pki/katello/qpid_client_striped.crt|host-validation: False
Aug 19 11:24:44 katello.server pulp:
gofer.messaging.adapter.qpid.connection:INFO: connected:
ssl://katello.server

When asked to install one package the job moves (in 20 seconds) to:

Host did not respond within 20 seconds. Is katello-agent installed and
goferd running on the Host?

When asked to 'update-all' the task moves immediately to 50% and the
poll_attempt count increases:

Action:

Actions::Pulp::Consumer::ContentUpdate

Input:

{"consumer_uuid"=>"90a34ad1-d153-4905-b57b-c1c9442003a7",
"type"=>"rpm",
"args"=>[],
"remote_user"=>"admin-e8886651",
"remote_cp_user"=>"admin",
"locale"=>"en-GB"}

Output:

{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>nil,
"_href"=>"/pulp/api/v2/tasks/f78143cb-3be5-44e6-825d-037e0ef1c363/",
"task_id"=>"f78143cb-3be5-44e6-825d-037e0ef1c363",
"tags"=>
["pulp:consumer:90a34ad1-d153-4905-b57b-c1c9442003a7",
"pulp:action:unit_update"],
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"agent.dq",
"state"=>"waiting",
"worker_name"=>"agent",
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"55d4596c388e58985eff62b7"},
"id"=>"55d4596c388e58985eff62b7"}],
"poll_attempts"=>{"total"=>242, "failed"=>0}}

···

On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote: > > The katello-agent and subscription manager install on a the (CentOS6) > client, and the client can use yum to update itself from Katello, but tasks > triggered from Katello fail. > > Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the Host? > > > I've implemented: > > http://projects.theforeman.org/issues/10350 / > https://bugzilla.redhat.com/show_bug.cgi?id=1217828 > > # ll /etc/pki/katello/qpid_router* > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32 > /etc/pki/katello/qpid_router_client.crt > -rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31 > /etc/pki/katello/qpid_router_client.key > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31 > /etc/pki/katello/qpid_router_server.crt > -rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31 > /etc/pki/katello/qpid_router_server.key > > But don't see an improvement. I'm also not using ipv6*, so haven't > modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did). > > *Well the 'advanced' section for the content host within Katello shows ::1 > (localhost). > > Of course I do see, on the Katello server, 5647 being listened to and the > firewall is open (and turned off for testing on the client): > > # netstat -pntl|grep 5647 > tcp 0 0 0.0.0.0:5647 0.0.0.0:* > LISTEN 48646/qdrouterd > > # iptables -nL | grep 5647 > ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport > dports 22,80,443,5647,5671,8140 > > tcpdump shows a short conversation between server and client: > > IP myserver.50035 > mytestclient.5647: tcp 0 > IP mytestclient.5647 > myserver.50035: tcp 0 > IP myserver.50035 > mytestclient.5647: tcp 0 > IP myserver.50035 > mytestclient.5647: tcp 304 > IP mytestclient.5647 > myserver.50035: tcp 0 > IP mytestclient.5647 > myserver.50035: tcp 3665 > IP myserver.50035 > mytestclient.5647: tcp 0 > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > amqps://myserver.mydomain:5647|SSL: ca: > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > /etc/pki/consumer/bundle.pem|host-validation: None > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 - > connecting to myserver.mydomain:5647... > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 - > Disconnected > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > amqps://myserver.mydomain:5647 > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call > last): > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 102, in open > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - self._impl = > BlockingConnection(url, ssl_domain=domain) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 200, in __init__ > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - msg="Opening > connection") > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 231, in wait > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - self.container.process() > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3729, in > dispatch > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > ev.dispatch(self.handler) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3654, in > dispatch > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - result = > dispatch(handler, type.method, self) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3543, in > dispatch > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - return m(*args) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in > on_transport_tail_closed > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > self.on_transport_closed(event) > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in > on_transport_closed > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - raise > ConnectionException("Connection %s disconnected" % self.url); > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - ConnectionException: > Connection amqps://myserver.mydomain:5647 disconnected > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:108 - retry in 29 seconds > > > Can anyone help? Many thanks. >

[stbenjam]

Is there a corresponding error on the Katello when you get the
disconnected message?

You can enable debug logging on qpid dispatch router, which might help:
https://gist.github.com/stbenjam/226f6939e981f1bca156

···

On Tue, Aug 18, 2015 at 05:29:10AM -0700, JC wrote: > Is anyone able to help? > > Having got the Katello server to the brink of deployment into production, > it's frustrating to not be able to proceed. > > On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote: > > > > The katello-agent and subscription manager install on a the (CentOS6) > > client, and the client can use yum to update itself from Katello, but tasks > > triggered from Katello fail. > > > > Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the Host? > > > > > > I've implemented: > > > > http://projects.theforeman.org/issues/10350 / > > https://bugzilla.redhat.com/show_bug.cgi?id=1217828 > > > > # ll /etc/pki/katello/qpid_router* > > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32 > > /etc/pki/katello/qpid_router_client.crt > > -rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31 > > /etc/pki/katello/qpid_router_client.key > > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31 > > /etc/pki/katello/qpid_router_server.crt > > -rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31 > > /etc/pki/katello/qpid_router_server.key > > > > But don't see an improvement. I'm also not using ipv6*, so haven't > > modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did). > > > > *Well the 'advanced' section for the content host within Katello shows ::1 > > (localhost). > > > > Of course I do see, on the Katello server, 5647 being listened to and the > > firewall is open (and turned off for testing on the client): > > > > # netstat -pntl|grep 5647 > > tcp 0 0 0.0.0.0:5647 0.0.0.0:* > > LISTEN 48646/qdrouterd > > > > # iptables -nL | grep 5647 > > ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport > > dports 22,80,443,5647,5671,8140 > > > > tcpdump shows a short conversation between server and client: > > > > IP myserver.50035 > mytestclient.5647: tcp 0 > > IP mytestclient.5647 > myserver.50035: tcp 0 > > IP myserver.50035 > mytestclient.5647: tcp 0 > > IP myserver.50035 > mytestclient.5647: tcp 304 > > IP mytestclient.5647 > myserver.50035: tcp 0 > > IP mytestclient.5647 > myserver.50035: tcp 3665 > > IP myserver.50035 > mytestclient.5647: tcp 0 > > > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] > > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > > amqps://myserver.mydomain:5647|SSL: ca: > > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > > /etc/pki/consumer/bundle.pem|host-validation: None > > > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 - > > connecting to myserver.mydomain:5647... > > > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 - > > Disconnected > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - > > amqps://myserver.mydomain:5647 > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call > > last): > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > line 102, in open > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - self._impl = > > BlockingConnection(url, ssl_domain=domain) > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 200, in __init__ > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - msg="Opening > > connection") > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 231, in wait > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - self.container.process() > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3729, in > > dispatch > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - > > ev.dispatch(self.handler) > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3654, in > > dispatch > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - result = > > dispatch(handler, type.method, self) > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3543, in > > dispatch > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - return m(*args) > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in > > on_transport_tail_closed > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - > > self.on_transport_closed(event) > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in > > on_transport_closed > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - raise > > ConnectionException("Connection %s disconnected" % self.url); > > > > Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - ConnectionException: > > Connection amqps://myserver.mydomain:5647 disconnected > > > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] > > gofer.messaging.adapter.proton.connection:108 - retry in 29 seconds > > > > > > Can anyone help? Many thanks. > > > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

[JC1]

Ahhh, right, on the client (/var/log/messages) I see:

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - amqps://katello.server:5647

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call
last):

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py",
line 101, in open

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - domain =
self.ssl_domain(connector)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py",
line 57, in ssl_domain

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
domain.set_trusted_ca_db(connector.ssl.ca_certificate)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3386, in
set_trusted_ca_db

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - certificate_db) )

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3376, in
_check

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise exc("SSL
failure.")

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - SSLException: SSL failure.

Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://katello.server:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

And on the server I see (ignore the times, grabbed from later in
/var/log/qdrouterd.log):

Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection from
katello.client:44791 to 0.0.0.0:5647

I've previously updated the Foreman UI cert using:

$ katello-installer --certs-server-cert "katello.cer"
–certs-server-cert-req "katello.csr" --certs-server-key "katello.key"
–certs-server-ca-cert "RootAll.cer" --certs-update-server
–certs-update-server-ca

Did I miss something then? The UI looks happy…

OK, anyone know the solution?

···

On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > OK, strace showed a problem writing to the log file. Perms changed and > now we have some output... > > # cat /var/log/qdrouterd.log > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/DISPATCH, identity=log/DISPATCH, > type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_HELLO) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/SERVER, > identity=log/SERVER, type=org.apache.qpid.dispatch.log, enable=default, > module=SERVER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/CONTAINER, identity=log/CONTAINER, > type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/AGENT, > identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default, > module=AGENT) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ERROR, > identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default, > module=ERROR) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ROUTER, > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/MESSAGE, identity=log/MESSAGE, > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/CONFIG, > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, enable=default, > module=CONFIG) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=container/katello.server, identity=container/katello.server, > type=org.apache.qpid.dispatch.container, containerName=katello.server, > workerThreads=2) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > routerId=katello.server, raInterval=30, mode=interior, mobileAddrMaxAge=60) > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior mode, > area=0 id=katello.server > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=16, > localFreeListMax=32) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_node_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_address_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter.ma > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdhello > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > id=katello.server instance=1439983963 max_routers=1024 > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/0, identity=fixedAddress/0, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/closest, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/closest > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/1, identity=fixedAddress/1, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/unicast, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/unicast > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/2, identity=fixedAddress/2, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/exclusive, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/3, identity=fixedAddress/3, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/multicast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/multicast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/4, identity=fixedAddress/4, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/broadcast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/broadcast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/5, identity=fixedAddress/5, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5647, identity=listener/0.0.0.0:5647, > type=org.apache.qpid.dispatch.listener, requirePeerAuth=True, > allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS, > maxFrameSize=65536, role=normal, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > certFile=/etc/pki/katello/qpid_router_server.crt, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5647) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647 > role=normal > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5646, identity=listener/0.0.0.0:5646, > type=org.apache.qpid.dispatch.listener, > certFile=/etc/pki/katello/qpid_router_server.crt, addr=0.0.0.0, > saslMechanisms=ANONYMOUS, maxFrameSize=65536, requirePeerAuth=True, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > allowNoSasl=False, role=inter-router, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5646) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5646 > role=inter-router > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=broker, > identity=connector/katello.server:5671, > type=org.apache.qpid.dispatch.connector, > certFile=/etc/pki/katello/qpid_router_client.crt, addr=katello.server, > allowRedirect=True, saslMechanisms=ANONYMOUS, maxFrameSize=65536, > role=on-demand, certDb=/etc/pki/katello/certs/katello-default-ca.crt, > keyFile=/etc/pki/katello/qpid_router_client.key, port=5671) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured on-demand connector: > katello.server:5671 name=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/0, identity=linkRoutePattern/0, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=pulp.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=pulp. connector=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/1, identity=linkRoutePattern/1, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=qmf.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=qmf. connector=broker > > Wed Aug 19 12:32:43 2015 SERVER (info) Operational, 2 Threads Running > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Starting on-demand connector: > broker > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'pulp.' for > routed links to 'broker' > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'qmf.' for routed > links to 'broker' > > Running a tcpdump capture on the client and katello server after asking > for Katello to 'update-all' or install one package shows *no* traffic. > > /var/log/messages only shows: > > Aug 19 11:24:44 katello.server pulp: > gofer.messaging.adapter.qpid.connection:INFO: connecting: URL: > ssl://katello.server|SSL: ca: > /etc/pki/katello/certs/katello-default-ca.crt|key: None|certificate: > /etc/pki/katello/qpid_client_striped.crt|host-validation: False > Aug 19 11:24:44 katello.server pulp: > gofer.messaging.adapter.qpid.connection:INFO: connected: > ssl://katello.server > > When asked to install one package the job moves (in 20 seconds) to: > > Host did not respond within 20 seconds. Is katello-agent installed and > goferd running on the Host? > > When asked to 'update-all' the task moves immediately to 50% and the > poll_attempt count increases: > > Action: > > Actions::Pulp::Consumer::ContentUpdate > > Input: > > {"consumer_uuid"=>"90a34ad1-d153-4905-b57b-c1c9442003a7", > "type"=>"rpm", > "args"=>[], > "remote_user"=>"admin-e8886651", > "remote_cp_user"=>"admin", > "locale"=>"en-GB"} > > Output: > > {"pulp_tasks"=> > [{"exception"=>nil, > "task_type"=>nil, > "_href"=>"/pulp/api/v2/tasks/f78143cb-3be5-44e6-825d-037e0ef1c363/", > "task_id"=>"f78143cb-3be5-44e6-825d-037e0ef1c363", > "tags"=> > ["pulp:consumer:90a34ad1-d153-4905-b57b-c1c9442003a7", > "pulp:action:unit_update"], > "finish_time"=>nil, > "_ns"=>"task_status", > "start_time"=>nil, > "traceback"=>nil, > "spawned_tasks"=>[], > "progress_report"=>{}, > "queue"=>"agent.dq", > "state"=>"waiting", > "worker_name"=>"agent", > "result"=>nil, > "error"=>nil, > "_id"=>{"$oid"=>"55d4596c388e58985eff62b7"}, > "id"=>"55d4596c388e58985eff62b7"}], > "poll_attempts"=>{"total"=>242, "failed"=>0}} > > > > On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote: >> >> The katello-agent and subscription manager install on a the (CentOS6) >> client, and the client can use yum to update itself from Katello, but tasks >> triggered from Katello fail. >> >> Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the Host? >> >> >> I've implemented: >> >> http://projects.theforeman.org/issues/10350 / >> https://bugzilla.redhat.com/show_bug.cgi?id=1217828 >> >> # ll /etc/pki/katello/qpid_router* >> -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32 >> /etc/pki/katello/qpid_router_client.crt >> -rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31 >> /etc/pki/katello/qpid_router_client.key >> -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31 >> /etc/pki/katello/qpid_router_server.crt >> -rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31 >> /etc/pki/katello/qpid_router_server.key >> >> But don't see an improvement. I'm also not using ipv6*, so haven't >> modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did). >> >> *Well the 'advanced' section for the content host within Katello shows >> ::1 (localhost). >> >> Of course I do see, on the Katello server, 5647 being listened to and the >> firewall is open (and turned off for testing on the client): >> >> # netstat -pntl|grep 5647 >> tcp 0 0 0.0.0.0:5647 0.0.0.0:* >> LISTEN 48646/qdrouterd >> >> # iptables -nL | grep 5647 >> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport >> dports 22,80,443,5647,5671,8140 >> >> tcpdump shows a short conversation between server and client: >> >> IP myserver.50035 > mytestclient.5647: tcp 0 >> IP mytestclient.5647 > myserver.50035: tcp 0 >> IP myserver.50035 > mytestclient.5647: tcp 0 >> IP myserver.50035 > mytestclient.5647: tcp 304 >> IP mytestclient.5647 > myserver.50035: tcp 0 >> IP mytestclient.5647 > myserver.50035: tcp 3665 >> IP myserver.50035 > mytestclient.5647: tcp 0 >> >> Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] >> gofer.messaging.adapter.proton.connection:100 - connecting: URL: >> amqps://myserver.mydomain:5647|SSL: ca: >> /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: >> /etc/pki/consumer/bundle.pem|host-validation: None >> >> Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 - >> connecting to myserver.mydomain:5647... >> >> Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 - >> Disconnected >> >> Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] >> gofer.messaging.adapter.proton.connection:106 - >> amqps://myserver.mydomain:5647 >> >> Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] >> gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call >> last): >> >> Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1] >> gofer.messaging.adapter.proton.connection:106 - File >> "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", >> line 102, in open >> >> Aug 13 12:50:20 mytestclient goferd: [ERROR][Thre

[JC1]

Thanks for getting back to me Stephen and the suggestion.

So that configuration is appended to the end of
/etc/qpidd-dispatch/qdrouterd.conf?

···

On 18 August 2015 at 15:07, Stephen Benjamin wrote:

Is there a corresponding error on the Katello when you get the
disconnected message?

You can enable debug logging on qpid dispatch router, which might help:
https://gist.github.com/stbenjam/226f6939e981f1bca156

On Tue, Aug 18, 2015 at 05:29:10AM -0700, JC wrote:

Is anyone able to help?

Having got the Katello server to the brink of deployment into production,
it’s frustrating to not be able to proceed.

On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote:

The katello-agent and subscription manager install on a the (CentOS6)
client, and the client can use yum to update itself from Katello, but
tasks

triggered from Katello fail.

Host did not respond within 20 seconds. Is katello-agent installed and
goferd running on the Host?

I’ve implemented:

Bug #10350: qpid-dispatch-0.4-4 runs under non-root account and has no access to PKI files - Katello - Foreman /
https://bugzilla.redhat.com/show_bug.cgi?id=1217828

ll /etc/pki/katello/qpid_router*

-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32
/etc/pki/katello/qpid_router_client.crt
-rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31
/etc/pki/katello/qpid_router_client.key
-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31
/etc/pki/katello/qpid_router_server.crt
-rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31
/etc/pki/katello/qpid_router_server.key

But don’t see an improvement. I’m also not using ipv6*, so haven’t
modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did).

*Well the ‘advanced’ section for the content host within Katello shows
::1

(localhost).

Of course I do see, on the Katello server, 5647 being listened to and
the

firewall is open (and turned off for testing on the client):

netstat -pntl|grep 5647

tcp 0 0 0.0.0.0:5647 0.0.0.0:*
LISTEN 48646/qdrouterd

iptables -nL | grep 5647

ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 multiport
dports 22,80,443,5647,5671,8140

tcpdump shows a short conversation between server and client:

IP myserver.50035 > mytestclient.5647: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 304
IP mytestclient.5647 > myserver.50035: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 3665
IP myserver.50035 > mytestclient.5647: tcp 0

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://myserver.mydomain:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 -
connecting to myserver.mydomain:5647…

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 -
Disconnected

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
amqps://myserver.mydomain:5647

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most recent
call

last):

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 102, in open

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - self._impl =
BlockingConnection(url, ssl_domain=domain)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 200, in
init

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - msg=“Opening
connection”)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 231, in wait

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
self.container.process()

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3729, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
ev.dispatch(self.handler)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3654, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - result =
dispatch(handler, type.method, self)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3543, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - return m(*args)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in
on_transport_tail_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
self.on_transport_closed(event)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in
on_transport_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise
ConnectionException(“Connection %s disconnected” % self.url);

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - ConnectionException:
Connection amqps://myserver.mydomain:5647 disconnected

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 29 seconds

Can anyone help? Many thanks.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/rXbUvcCTuf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

[JC1]

Sorry for all the postings, but perhaps I'm getting somewhere…

# grep "cert=" /etc/gofer/plugins/katelloplugin.conf
cacert=/etc/rhsm/ca/candlepin-local.pem
clientcert=/etc/pki/consumer/bundle.pem
# file /etc/rhsm/ca/candlepin-local.pem
/etc/rhsm/ca/candlepin-local.pem: cannot open
`/etc/rhsm/ca/candlepin-local.pem' (No such file or directory)
# file /etc/pki/consumer/bundle.pem
/etc/pki/consumer/bundle.pem: ASCII text

$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text

shows a cert for katello.server signed by RH, rather than my CA.

Is that relevant? Can you tell that I don't really understand how the
certs are used?

···

On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > Ahhh, right, on the client (/var/log/messages) I see: > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - amqps://katello.server:5647 > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call > last): > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 101, in open > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - domain = > self.ssl_domain(connector) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 57, in ssl_domain > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3386, in > set_trusted_ca_db > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - certificate_db) ) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3376, in > _check > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > failure.") > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > failure.* > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > amqps://katello.server:5647|SSL: ca: > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > /etc/pki/consumer/bundle.pem|host-validation: None > > And on the server I see (ignore the times, grabbed from later in > /var/log/qdrouterd.log): > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection from > katello.client:44791 to 0.0.0.0:5647 > > I've previously updated the Foreman UI cert using: > > $ katello-installer --certs-server-cert "katello.cer" > --certs-server-cert-req "katello.csr" --certs-server-key "katello.key" > --certs-server-ca-cert "RootAll.cer" --certs-update-server > --certs-update-server-ca > > Did I miss something then? The UI looks happy... > > OK, anyone know the solution? > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > OK, strace showed a problem writing to the log file. Perms changed and > now we have some output... > > # cat /var/log/qdrouterd.log > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/DISPATCH, identity=log/DISPATCH, > type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_HELLO) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/SERVER, > identity=log/SERVER, type=org.apache.qpid.dispatch.log, enable=default, > module=SERVER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/CONTAINER, identity=log/CONTAINER, > type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/AGENT, > identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default, > module=AGENT) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ERROR, > identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default, > module=ERROR) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ROUTER, > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/MESSAGE, identity=log/MESSAGE, > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/CONFIG, > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, enable=default, > module=CONFIG) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=container/katello.server, identity=container/katello.server, > type=org.apache.qpid.dispatch.container, containerName=katello.server, > workerThreads=2) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > routerId=katello.server, raInterval=30, mode=interior, mobileAddrMaxAge=60) > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior mode, > area=0 id=katello.server > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=16, > localFreeListMax=32) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_node_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_address_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter.ma > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdhello > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > id=katello.server instance=1439983963 max_routers=1024 > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/0, identity=fixedAddress/0, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/closest, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/closest > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/1, identity=fixedAddress/1, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/unicast, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/unicast > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/2, identity=fixedAddress/2, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/exclusive, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/3, identity=fixedAddress/3, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/multicast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/multicast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/4, identity=fixedAddress/4, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/broadcast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/broadcast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/5, identity=fixedAddress/5, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5647, identity=listener/0.0.0.0:5647, > type=org.apache.qpid.dispatch.listener, requirePeerAuth=True, > allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS, > maxFrameSize=65536, role=normal, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > certFile=/etc/pki/katello/qpid_router_server.crt, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5647) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647 > role=normal > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5646, identity=listener/0.0.0.0:5646, > type=org.apache.qpid.dispatch.listener, > certFile=/etc/pki/katello/qpid_router_server.crt, addr=0.0.0.0, > saslMechanisms=ANONYMOUS, maxFrameSize=65536, requirePeerAuth=True, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > allowNoSasl=False, role=inter-router, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5646) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5646 > role=inter-router > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=broker, > identity=connector/katello.server:5671, > type=org.apache.qpid.dispatch.connector, > certFile=/etc/pki/katello/qpid_router_client.crt, addr=katello.server, > allowRedirect=True, saslMechanisms=ANONYMOUS, maxFrameSize=65536, > role=on-demand, certDb=/etc/pki/katello/certs/katello-default-ca.crt, > keyFile=/etc/pki/katello/qpid_router_client.key, port=5671) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured on-demand connector: > katello.server:5671 name=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/0, identity=linkRoutePattern/0, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=pulp.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=pulp. connector=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/1, identity=linkRoutePattern/1, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=qmf.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=qmf. connector=broker > > Wed Aug 19 12:32:43 2015 SERVER (info) Operational, 2 Threads Running > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Starting on-demand connector: > broker > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'pulp.' for > routed links to 'broker' > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'qmf.' for routed > links to 'broker' > > Running a tcpdump capture on the client and katello server after asking > for Katello to 'update-all' or install one package shows *no* traffic. > > /var/log/messages only shows: > > Aug 19 11:24:44 katello.server pulp: > gofer.messaging.adapter.qpid.connection:INFO: connecting: URL: > ssl://katello.server|SSL: ca: > /etc/pki/katello/certs/katello-default-ca.crt|key: None|certificate: > /etc/pki/katello/qpid_client_striped.crt|host-validation: False > Aug 19 11:24:44 katello.server pulp: > gofer.messaging.adapter.qpid.connection:INFO: connected: > ssl://katello.server > > When asked to install one package the job moves (in 20 seconds) to: > > Host did not respond within 20 seconds. Is katello-agent installed and > goferd running on the Host? > > When asked to 'update-all' the task moves immediately to 50% and the > poll_attempt count increases: > > Action: > > Actions::Pulp::Consumer::ContentUpdate > > Input: > > {"consumer_uuid"=>"90a34ad1-d153-4905-b57b-c1c9442003a7", > "type"=>"rpm", > "args"=>[], > "remote_user"=>"admin-e8886651", > "remote_cp_user"=>"admin", > "locale"=>"en-GB"} > > Output: > > {"pulp_tasks"=> > [{"exception"=>nil, > "task_type"=>nil, > "_href"=>"/pulp/api/v2/tasks/f78143cb-3be5-44e6-825d-037e0ef1c363/", > "task_id"=>"f78143cb-3be5-44e6-825d-037e0ef1c363", > "tags"=> > ["pulp:consumer:90a34ad1-d153-4905-b57b-c1c9442003a7", > "pulp:action:unit_update"], > "finish_time"=>nil, > "_ns"=>"task_status", > "start_time"=>nil, > "traceback"=>nil, > "spawned_tasks"=>[], > "progress_report"=>{}, > "queue"=>"agent.dq", > "state"=>"waiting", > "worker_name"=>"agent", > "result"=>nil, > "error"=>nil, > "_id"=>{"$oid"=>"55d4596c388e58985eff62b7"}, > "id"=>"55d4596c388e58985eff62b7"}], > "poll_attempts"=>{"total"=>242, "failed"=>0}} > > > > On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote: > > The katello-agent and subscription manager install on a the (CentOS6) > client, and the client can use yum to update itself from Katello, but tasks > triggered from Katello fail. > > Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the Host? > > > I've implemented: > > http://projects.theforeman.org/issues/10350 / > https://bugzilla.redhat.com/show_bug.cgi?id=1217828 > > # ll /etc/pki/katello/qpid_router* > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32 > /etc/pki/katello/qpid_router_client.crt > -rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31 > /etc/pki/katello/qpid_router_client.key > -rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31 > /etc/pki/katello/qpid_router_server.crt > -rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31 > /etc/pki/katello/qpid_router_server.key > > But don't see an improvement. I'm also not using ipv6*, so haven't > modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did). > > *Well the 'advanced' section for the content host within Katello shows ::1 > (localhost). > > Of course I do see, on the Katello server, 5647 being listened to and the > firewall is open (and turned off for testing on the client): > > # netstat -pntl|grep 5647 > tcp 0 0 0.0.0.0:5647 0.0.0.0:* > LISTEN 48646/qdrouterd > > # iptables -nL | grep 5647 > ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport > dports 22,80,443,5647,5671,8140 > > tcpdump shows a short conversation between server and client: > > IP myserver.50035 > mytestclient.5647: tcp 0 > IP mytestclient.5647 > myserver.50035: tcp 0 > IP myserver.50035 > mytestclient.5647: tcp 0 > IP myserver.50035 > mytestclient.5647: tcp 304 > IP mytestclient.5647 > myserver.50035: tcp 0 > IP mytestclient.5647 > myserver.50035: tcp 3665 > IP myserver.50035 > mytestclient.5647: tcp 0 > > Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > amqps://myserver.mydomain:5647|SSL: ca: > /etc/rhsm/ca/katello-server-ca.pem|key: None

[stbenjam]

> Thanks for getting back to me Stephen and the suggestion.
>
> So that configuration is appended to the end of
> /etc/qpidd-dispatch/qdrouterd.conf?

Yup, I should've mentioned where to put it.

And restart qdrouterd after.

···

On Tue, Aug 18, 2015 at 03:21:28PM +0100, Jamie Caldwell wrote:

On 18 August 2015 at 15:07, Stephen Benjamin stephen@redhat.com wrote:

Is there a corresponding error on the Katello when you get the
disconnected message?

You can enable debug logging on qpid dispatch router, which might help:
https://gist.github.com/stbenjam/226f6939e981f1bca156

On Tue, Aug 18, 2015 at 05:29:10AM -0700, JC wrote:

Is anyone able to help?

Having got the Katello server to the brink of deployment into production,
it’s frustrating to not be able to proceed.

On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote:

The katello-agent and subscription manager install on a the (CentOS6)
client, and the client can use yum to update itself from Katello, but
tasks

triggered from Katello fail.

Host did not respond within 20 seconds. Is katello-agent installed and
goferd running on the Host?

I’ve implemented:

Bug #10350: qpid-dispatch-0.4-4 runs under non-root account and has no access to PKI files - Katello - Foreman /
https://bugzilla.redhat.com/show_bug.cgi?id=1217828

ll /etc/pki/katello/qpid_router*

-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32
/etc/pki/katello/qpid_router_client.crt
-rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31
/etc/pki/katello/qpid_router_client.key
-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31
/etc/pki/katello/qpid_router_server.crt
-rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31
/etc/pki/katello/qpid_router_server.key

But don’t see an improvement. I’m also not using ipv6*, so haven’t
modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did).

*Well the ‘advanced’ section for the content host within Katello shows
::1

(localhost).

Of course I do see, on the Katello server, 5647 being listened to and
the

firewall is open (and turned off for testing on the client):

netstat -pntl|grep 5647

tcp 0 0 0.0.0.0:5647 0.0.0.0:*
LISTEN 48646/qdrouterd

iptables -nL | grep 5647

ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 multiport
dports 22,80,443,5647,5671,8140

tcpdump shows a short conversation between server and client:

IP myserver.50035 > mytestclient.5647: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 304
IP mytestclient.5647 > myserver.50035: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 3665
IP myserver.50035 > mytestclient.5647: tcp 0

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://myserver.mydomain:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 -
connecting to myserver.mydomain:5647…

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 -
Disconnected

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
amqps://myserver.mydomain:5647

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most recent
call

last):

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 102, in open

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - self._impl =
BlockingConnection(url, ssl_domain=domain)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 200, in
init

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - msg=“Opening
connection”)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 231, in wait

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
self.container.process()

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3729, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
ev.dispatch(self.handler)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3654, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - result =
dispatch(handler, type.method, self)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3543, in
dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - return m(*args)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in
on_transport_tail_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
self.on_transport_closed(event)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in
on_transport_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise
ConnectionException(“Connection %s disconnected” % self.url);

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - ConnectionException:
Connection amqps://myserver.mydomain:5647 disconnected

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 29 seconds

Can anyone help? Many thanks.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/rXbUvcCTuf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

[JC1]

Thanks for confirming. Nothing I'm afraid. The file wasn't created, so I
touched it. But still nought. ?

···

On 18 August 2015 at 15:55, Stephen Benjamin wrote:

On Tue, Aug 18, 2015 at 03:21:28PM +0100, Jamie Caldwell wrote:

Thanks for getting back to me Stephen and the suggestion.

So that configuration is appended to the end of
/etc/qpidd-dispatch/qdrouterd.conf?

Yup, I should’ve mentioned where to put it.

And restart qdrouterd after.

On 18 August 2015 at 15:07, Stephen Benjamin stephen@redhat.com wrote:

Is there a corresponding error on the Katello when you get the
disconnected message?

You can enable debug logging on qpid dispatch router, which might help:
https://gist.github.com/stbenjam/226f6939e981f1bca156

On Tue, Aug 18, 2015 at 05:29:10AM -0700, JC wrote:

Is anyone able to help?

Having got the Katello server to the brink of deployment into
production,

it’s frustrating to not be able to proceed.

On Thursday, 13 August 2015 13:35:59 UTC+1, JC wrote:

The katello-agent and subscription manager install on a the
(CentOS6)

client, and the client can use yum to update itself from Katello,
but

tasks

triggered from Katello fail.

Host did not respond within 20 seconds. Is katello-agent installed
and

goferd running on the Host?

I’ve implemented:

Bug #10350: qpid-dispatch-0.4-4 runs under non-root account and has no access to PKI files - Katello - Foreman /
https://bugzilla.redhat.com/show_bug.cgi?id=1217828

ll /etc/pki/katello/qpid_router*

-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:32
/etc/pki/katello/qpid_router_client.crt
-rw-r-----. 1 qdrouterd qdrouterd 1679 Jul 28 15:31
/etc/pki/katello/qpid_router_client.key
-rw-r-----. 1 qdrouterd qdrouterd 5579 Jul 28 15:31
/etc/pki/katello/qpid_router_server.crt
-rw-r-----. 1 qdrouterd qdrouterd 1675 Jul 28 15:31
/etc/pki/katello/qpid_router_server.key

But don’t see an improvement. I’m also not using ipv6*, so haven’t
modified /etc/qpid-dispatch/qdrouterd.conf (as David LeVene did).

*Well the ‘advanced’ section for the content host within Katello
shows

::1

(localhost).

Of course I do see, on the Katello server, 5647 being listened to
and

the

firewall is open (and turned off for testing on the client):

netstat -pntl|grep 5647

tcp 0 0 0.0.0.0:5647 0.0.0.0:*
LISTEN 48646/qdrouterd

iptables -nL | grep 5647

ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0
multiport

dports 22,80,443,5647,5671,8140

tcpdump shows a short conversation between server and client:

IP myserver.50035 > mytestclient.5647: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 0
IP myserver.50035 > mytestclient.5647: tcp 304
IP mytestclient.5647 > myserver.50035: tcp 0
IP mytestclient.5647 > myserver.50035: tcp 3665
IP myserver.50035 > mytestclient.5647: tcp 0

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://myserver.mydomain:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:481 -
connecting to myserver.mydomain:5647…

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1] root:521 -
Disconnected

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
amqps://myserver.mydomain:5647

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most
recent

call

last):

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 102, in open

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - self._impl =
BlockingConnection(url, ssl_domain=domain)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 200, in
init

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - msg=“Opening
connection”)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 231, in
wait

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
self.container.process()

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line
3729, in

dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
ev.dispatch(self.handler)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line
3654, in

dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - result =
dispatch(handler, type.method, self)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line
3543, in

dispatch

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - return m(*args)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in
on_transport_tail_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
self.on_transport_closed(event)

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in
on_transport_closed

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise
ConnectionException(“Connection %s disconnected” % self.url);

Aug 13 12:50:20 mytestclient goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
ConnectionException:

Connection amqps://myserver.mydomain:5647 disconnected

Aug 13 12:50:20 mytestclient goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 29 seconds

Can anyone help? Many thanks.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it,
send

an email to foreman-users+unsubscribe@googlegroups.com.

To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/foreman-users/rXbUvcCTuf0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Best Regards,

Stephen Benjamin
Red Hat Engineering

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/rXbUvcCTuf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

[JC1]

Correction, the issuer is katello.server (with some RH text to fill in the
cert), and it is for my client!

Certificate:

<snip>

•    Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, 
  

OU=SomeOrgUnit, CN=katello.server*

    Validity

        Not Before: Aug 18 14:25:23 2015 GMT

        Not After : Aug 18 14:25:23 2031 GMT

<snip>

            Exponent: 65257 (0x10001)

    X509v3 extensions:

        Netscape Cert Type:

            SSL Client, S/MIME

        X509v3 Key Usage:

            Digital Signature, Key Encipherment, Data Encipherment

        X509v3 Authority Key Identifier:

            keyid:EA:37:47:…..2E:71:EC:91

•            DirName:/C=US/ST=North 
  

Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server*

            serial:A7:A2:3C:……….:65:B1

        X509v3 Subject Key Identifier:

            FB:A7:9C:E6:7……………………………..0E:78:B8:C8

        X509v3 Extended Key Usage:

            TLS Web Client Authentication

•        X509v3 Subject Alternative Name:*
  

•            URI:CN=katello.client*
  

  Signature Algorithm: sha1WithRSAEncryption

     7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07:
  
     <snip>
  

Now, from the client:

# openssl s_client -connect katello.server:5647 -showcerts ## same for
port 5446
CONNECTED(00000003)
depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU =
SomeOrgUnit, CN = katello.server
verify error:num=19:self signed certificate in certificate chain
verify return:0
140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert
handshake failure:s3_pkt.c:1259:SSL alert number 40
140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
failure:s23_lib.c:184:

···

On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote: > > Sorry for all the postings, but perhaps I'm getting somewhere... > > *# grep "cert=" /etc/gofer/plugins/katelloplugin.conf* > cacert=/etc/rhsm/ca/candlepin-local.pem > clientcert=/etc/pki/consumer/bundle.pem > *# file /etc/rhsm/ca/candlepin-local.pem* > /etc/rhsm/ca/candlepin-local.pem: cannot open > `/etc/rhsm/ca/candlepin-local.pem' (No such file or directory) > *# file /etc/pki/consumer/bundle.pem* > /etc/pki/consumer/bundle.pem: ASCII text > > > *$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text* > # shows a cert for katello.server signed by RH, rather than my CA. > > Is that relevant? Can you tell that I don't really understand how the > certs are used? ;-) > > On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > Ahhh, right, on the client (/var/log/messages) I see: > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - amqps://katello.server:5647 > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call > last): > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 101, in open > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - domain = > self.ssl_domain(connector) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 57, in ssl_domain > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3386, in > set_trusted_ca_db > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - certificate_db) ) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3376, in > _check > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > failure.") > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > failure.* > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > amqps://katello.server:5647|SSL: ca: > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > /etc/pki/consumer/bundle.pem|host-validation: None > > And on the server I see (ignore the times, grabbed from later in > /var/log/qdrouterd.log): > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection from > katello.client:44791 to 0.0.0.0:5647 > > I've previously updated the Foreman UI cert using: > > $ katello-installer --certs-server-cert "katello.cer" > --certs-server-cert-req "katello.csr" --certs-server-key "katello.key" > --certs-server-ca-cert "RootAll.cer" --certs-update-server > --certs-update-server-ca > > Did I miss something then? The UI looks happy... > > OK, anyone know the solution? > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > OK, strace showed a problem writing to the log file. Perms changed and > now we have some output... > > # cat /var/log/qdrouterd.log > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/DISPATCH, identity=log/DISPATCH, > type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_HELLO) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/SERVER, > identity=log/SERVER, type=org.apache.qpid.dispatch.log, enable=default, > module=SERVER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/CONTAINER, identity=log/CONTAINER, > type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/AGENT, > identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default, > module=AGENT) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ERROR, > identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default, > module=ERROR) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ROUTER, > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/MESSAGE, identity=log/MESSAGE, > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/CONFIG, > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, enable=default, > module=CONFIG) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=container/katello.server, identity=container/katello.server, > type=org.apache.qpid.dispatch.container, containerName=katello.server, > workerThreads=2) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > routerId=katello.server, raInterval=30, mode=interior, mobileAddrMaxAge=60) > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior mode, > area=0 id=katello.server > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=16, > localFreeListMax=32) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_node_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_address_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter.ma > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdhello > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > id=katello.server instance=1439983963 max_routers=1024 > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/0, identity=fixedAddress/0, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/closest, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/closest > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/1, identity=fixedAddress/1, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/unicast, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/unicast > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/2, identity=fixedAddress/2, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/exclusive, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/3, identity=fixedAddress/3, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/multicast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/multicast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/4, identity=fixedAddress/4, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/broadcast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/broadcast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/5, identity=fixedAddress/5, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5647, identity=listener/0.0.0.0:5647, > type=org.apache.qpid.dispatch.listener, requirePeerAuth=True, > allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS, > maxFrameSize=65536, role=normal, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > certFile=/etc/pki/katello/qpid_router_server.crt, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5647) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647 > role=normal > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5646, identity=listener/0.0.0.0:5646, > type=org.apache.qpid.dispatch.listener, > certFile=/etc/pki/katello/qpid_router_server.crt, addr=0.0.0.0, > saslMechanisms=ANONYMOUS, maxFrameSize=65536, requirePeerAuth=True, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > allowNoSasl=False, role=inter-router, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5646) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5646 > role=inter-router > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=broker, > identity=connector/katello.server:5671, > type=org.apache.qpid.dispatch.connector, > certFile=/etc/pki/katello/qpid_router_client.crt, addr=katello.server, > allowRedirect=True, saslMechanisms=ANONYMOUS, maxFrameSize=65536, > role=on-demand, certDb=/etc/pki/katello/certs/katello-default-ca.crt, > keyFile=/etc/pki/katello/qpid_router_client.key, port=5671) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured on-demand connector: > katello.server:5671 name=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/0, identity=linkRoutePattern/0, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=pulp.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=pulp. connector=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/1, identity=linkRoutePattern/1, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=qmf.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=qmf. connector=broker > > Wed Aug 19 12:32:43 2015 SERVER (info) Operational, 2 Threads Running > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Starting on-demand connector: > broker > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'pulp.' for > routed links to 'broker' > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'qmf.' for routed > links to 'broker' > > Running a tcpdump capture on the client and katello server after asking > for Katello to 'update-all' or install one package shows *no* traffic. > > /var/log/messages only shows: > > Aug 19 11:24:44 katello.server pulp: > gofer.messaging.adapter.qpid.conne

[stbenjam]

The host you installed katello-agent on is actually registered to the
katello and not Red Hat CDN?

What version of katello-agent are you using? What version of Katello are
you running?

Are you using custom certificates?

···

On Wed, Aug 19, 2015 at 05:49:54AM -0700, JC wrote: > Sorry for all the postings, but perhaps I'm getting somewhere... > > *# grep "cert=" /etc/gofer/plugins/katelloplugin.conf* > cacert=/etc/rhsm/ca/candlepin-local.pem > clientcert=/etc/pki/consumer/bundle.pem > *# file /etc/rhsm/ca/candlepin-local.pem* > /etc/rhsm/ca/candlepin-local.pem: cannot open > `/etc/rhsm/ca/candlepin-local.pem' (No such file or directory) > *# file /etc/pki/consumer/bundle.pem* > /etc/pki/consumer/bundle.pem: ASCII text > > > *$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text* > # shows a cert for katello.server signed by RH, rather than my CA. > > Is that relevant? Can you tell that I don't really understand how the > certs are used? ;-) > > On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > > > Ahhh, right, on the client (/var/log/messages) I see: > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - amqps://katello.server:5647 > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call > > last): > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > line 101, in open > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - domain = > > self.ssl_domain(connector) > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > line 57, in ssl_domain > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - > > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3386, in > > set_trusted_ca_db > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - certificate_db) ) > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - File > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3376, in > > _check > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > > failure.") > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > > failure.* > > > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > > gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds > > > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > > amqps://katello.server:5647|SSL: ca: > > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > > /etc/pki/consumer/bundle.pem|host-validation: None > > > > And on the server I see (ignore the times, grabbed from later in > > /var/log/qdrouterd.log): > > > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection from > > katello.client:44791 to 0.0.0.0:5647 > > > > I've previously updated the Foreman UI cert using: > > > > $ katello-installer --certs-server-cert "katello.cer" > > --certs-server-cert-req "katello.csr" --certs-server-key "katello.key" > > --certs-server-ca-cert "RootAll.cer" --certs-update-server > > --certs-update-server-ca > > > > Did I miss something then? The UI looks happy... > > > > OK, anyone know the solution? > > > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > > > OK, strace showed a problem writing to the log file. Perms changed and > > now we have some output... > > > > # cat /var/log/qdrouterd.log > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=log/DISPATCH, identity=log/DISPATCH, > > type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_HELLO) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/SERVER, > > identity=log/SERVER, type=org.apache.qpid.dispatch.log, enable=default, > > module=SERVER) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=log/CONTAINER, identity=log/CONTAINER, > > type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/AGENT, > > identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default, > > module=AGENT) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ERROR, > > identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default, > > module=ERROR) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ROUTER, > > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, enable=default, > > module=ROUTER) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=log/MESSAGE, identity=log/MESSAGE, > > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/CONFIG, > > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, enable=default, > > module=CONFIG) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=container/katello.server, identity=container/katello.server, > > type=org.apache.qpid.dispatch.container, containerName=katello.server, > > workerThreads=2) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > > routerId=katello.server, raInterval=30, mode=interior, mobileAddrMaxAge=60) > > > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior mode, > > area=0 id=katello.server > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=16, > > localFreeListMax=32) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > > localFreeListMax=128) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > > localFreeListMax=128) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_node_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > > localFreeListMax=128) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > > localFreeListMax=128) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > > localFreeListMax=128) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_address_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > > localFreeListMax=128) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > > localFreeListMax=128) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > > deliveriesToContainer=0) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > > deliveriesToContainer=0) > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > > deliveriesToContainer=0) > > > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on > > $management > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > $management > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > $management > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > qdrouter > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > qdrouter.ma > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > qdhello > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > > id=katello.server instance=1439983963 max_routers=1024 > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=fixedAddress/0, identity=fixedAddress/0, > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/closest, > > bias=closest) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/closest > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=fixedAddress/1, identity=fixedAddress/1, > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/unicast, > > bias=closest) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/unicast > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=fixedAddress/2, identity=fixedAddress/2, > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > > prefix=/exclusive, bias=closest) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=fixedAddress/3, identity=fixedAddress/3, > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > > prefix=/multicast, bias=closest) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > prefix=/multicast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=fixedAddress/4, identity=fixedAddress/4, > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > > prefix=/broadcast, bias=closest) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > prefix=/broadcast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=fixedAddress/5, identity=fixedAddress/5, > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/, > > bias=closest) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > > 0.0.0.0:5647, identity=listener/0.0.0.0:5647, > > type=org.apache.qpid.dispatch.listener, requirePeerAuth=True, > > allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS, > > maxFrameSize=65536, role=normal, > > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > > certFile=/etc/pki/katello/qpid_router_server.crt, > > keyFile=/etc/pki/katello/qpid_router_server.key, port=5647) > > > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647 > > role=normal > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > > 0.0.0.0:5646, identity=listener/0.0.0.0:5646, > > type=org.apache.qpid.dispatch.listener, > > certFile=/etc/pki/katello/qpid_router_server.crt, addr=0.0.0.0, > > saslMechanisms=ANONYMOUS, maxFrameSize=65536, requirePeerAuth=True, > > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > > allowNoSasl=False, role=inter-router, > > keyFile=/etc/pki/katello/qpid_router_server.key, port=5646) > > > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5646 > > role=inter-router > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=broker, > > identity=connector/katello.server:5671, > > type=org.apache.qpid.dispatch.connector, > > certFile=/etc/pki/katello/qpid_router_client.crt, addr=katello.server, > > allowRedirect=True, saslMechanisms=ANONYMOUS, maxFrameSize=65536, > > role=on-demand, certDb=/etc/pki/katello/certs/katello-default-ca.crt, > > keyFile=/etc/pki/katello/qpid_router_client.key, port=5671) > > > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured on-demand connector: > > katello.server:5671 name=broker > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=linkRoutePattern/0, identity=linkRoutePattern/0, > > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > > prefix=pulp.) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > > prefix=pulp. connector=broker > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > Entity(name=linkRoutePattern/1, identity=linkRoutePattern/1, > > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > > prefix=qmf.) > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > > prefix=qmf. connector=broker > > > > Wed Aug 19 12:32:43 2015 SERVER (info) Operational, 2 Threads Running > > > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Starting on-demand connector: > > broker > > > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'pulp.' for > > routed links to 'broker' > > > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'qmf.' for routed > > links to 'broker' > > > > Running a tcpdump capture on the client and katello server after asking > > for Katello to 'update-all' or install one package shows *no* traffic. > > > > /var/log/messages only shows: > > > > Aug 19 11:24:44 katello.server pulp: > > gofer.messaging.adapter.qpid.connection:INFO: connecting: URL: > > ssl://katello.server|SSL: ca: > > /etc/pki/katello/certs/katello-default-ca.crt|key: None|certificate: > > /etc/pki/katello/qpid_client_striped.crt|host-validation: False > > Aug 19 11:24:44 katello.server pulp: > > gofer.messaging.adapter.qpid.connection:INFO: connected: > > ssl://katello.server > > > > When asked to install one package the job moves (in 20 seconds) to: > > > > Host did not respond within 20 seconds. Is katello-agent installed and > > goferd running on the Host? > > > > When asked to 'update-all' the task moves immediately to 50% and the > > poll_attempt count increases: > > > > Action: > > > > Actions::Pulp::Consumer::ContentUpdate > > > > Input: > > > > {"consumer_uuid"=>"90a34ad1-d153-4905-b57b-c1c9442003a7", > > "type"=>"rpm", > > "args"=>[], > > "remote_user"=>"admin-e8886651", > > "remote_cp_user"=>"admin", > > "locale"=>"en-GB"} > > > > Output: > > > > {"pulp_tasks"=> > > [{"exception"=>n

[JC1]

Hi Stephen,

The host you installed katello-agent on is actually registered to
thekatello* and not Red Hat CDN?*
Yes, registered to katello.server. subscription-manager repo happily shows
the repos katello.server is serving and I can make use of those repos from
the client

What version of katello-agent are you using? What version of Katello
areyou running?
from the client

rpm -qa --last | head -15

python-qpid-proton-0.9-3.el6.x86_64 Tue 18 Aug 2015 15:25:15 BST
python-gofer-proton-2.5.3-1.el6.noarch Tue 18 Aug 2015 15:25:15 BST
katello-agent-2.2.1-1.el6.noarch Tue 18 Aug 2015 15:25:15 BST
qpid-proton-c-0.9-3.el6.x86_64 Tue 18 Aug 2015 15:25:09 BST
python-pulp-rpm-common-2.6.0-1.el6.noarch Tue 18 Aug 2015 15:25:09 BST
python-pulp-common-2.6.0-1.el6.noarch Tue 18 Aug 2015 15:25:09 BST
python-pulp-agent-lib-2.6.0-1.el6.noarch Tue 18 Aug 2015 15:25:09 BST
python-isodate-0.5.0-4.pulp.el6.noarch Tue 18 Aug 2015 15:25:09 BST
pulp-rpm-handlers-2.6.0-1.el6.noarch Tue 18 Aug 2015 15:25:09 BST
gofer-2.5.3-1.el6.noarch Tue 18 Aug 2015 15:25:09 BST
python-gofer-2.5.3-1.el6.noarch Tue 18 Aug 2015 15:25:08 BST
gpg-pubkey-0608b895-4bd22942 Tue 18 Aug 2015 15:25:07 BST
subscription-manager-1.13.12-1.el6.x86_64 Tue 18 Aug 2015 15:25:04 BST
python-rhsm-1.13.8-1.el6.x86_64 Tue 18 Aug 2015 15:25:04 BST
katello-ca-consumer-katello.server-1.0-2.noarch Tue 18 Aug 2015 15:25:04 BST

from the Katello server UI:

• candlepin-0.9.45-1.el6.noarch
• candlepin-common-1.0.22-1.el6.noarch
• candlepin-selinux-0.9.45-1.el6.noarch
• candlepin-tomcat6-0.9.45-1.el6.noarch
• elasticsearch-0.90.10-7.el6.noarch
• katello-2.2.1-0.el6.noarch
• katello-certs-tools-2.0.1-1.el6.noarch
• katello-common-2.2.1-0.el6.noarch
• katello-debug-2.2.1-0.el6.noarch
• katello-default-ca-1.0-1.noarch
• katello-installer-2.2.2-1.el6.noarch
• katello-installer-base-2.2.2-1.el6.noarch
• katello-repos-2.2.1-1.el6.noarch
• katello-selinux-2.2.1-1.el6.noarch
• katello-server-ca-1.0-3.noarch
• katello-service-2.2.1-0.el6.noarch
• libqpid-dispatch-0.4-4.el6.x86_64
• m2crypto-0.21.1.pulp-8.el6.x86_64
• mod_wsgi-3.4-2.pulp.el6.x86_64
• katello.server-qpid-broker-1.0-1.noarch
• katello.server-qpid-client-cert-1.0-1.noarch
• katello.server-qpid-router-client-1.0-1.noarch
• katello.server-qpid-router-server-1.0-1.noarch
• pulp-admin-client-2.6.0-1.el6.noarch
• pulp-docker-plugins-0.2.2-1.el6.noarch
• pulp-katello-0.4-1.el6.noarch
• pulp-nodes-common-2.6.0-1.el6.noarch
• pulp-nodes-parent-2.6.0-1.el6.noarch
• pulp-puppet-plugins-2.6.0-1.el6.noarch
• pulp-puppet-tools-2.6.0-1.el6.noarch
• pulp-rpm-plugins-2.6.0-1.el6.noarch
• pulp-selinux-2.6.0-1.el6.noarch
• pulp-server-2.6.0-1.el6.noarch
• python-gofer-qpid-2.5.3-1.el6.noarch
• python-isodate-0.5.0-4.pulp.el6.noarch
• python-kombu-3.0.24-5.pulp.el6.noarch
• python-pulp-bindings-2.6.0-1.el6.noarch
• python-pulp-client-lib-2.6.0-1.el6.noarch
• python-pulp-common-2.6.0-1.el6.noarch
• python-pulp-docker-common-0.2.2-1.el6.noarch
• python-pulp-puppet-common-2.6.0-1.el6.noarch
• python-pulp-rpm-common-2.6.0-1.el6.noarch
• python-qpid-0.30-7.el6.noarch
• python-qpid-qmf-0.30-5.el6.x86_64
• python-rhsm-1.8.0-2.pulp.el6.x86_64
• qpid-cpp-client-0.30-7.proton.0.9.el6.x86_64
• qpid-cpp-client-devel-0.30-7.proton.0.9.el6.x86_64
• qpid-cpp-server-0.30-7.proton.0.9.el6.x86_64
• qpid-cpp-server-linearstore-0.30-7.proton.0.9.el6.x86_64
• qpid-dispatch-router-0.4-4.el6.x86_64
• qpid-proton-c-0.9-2.el6.x86_64
• qpid-qmf-0.30-5.el6.x86_64
• qpid-tools-0.30-4.el6.noarch
• ruby193-rubygem-katello-2.2.2-2.el6.noarch
• ruby193-rubygem-qpid_messaging-0.30.0-1.el6.x86_64
• rubygem-hammer_cli_katello-0.0.14-1.el6.noarch
• rubygem-smart_proxy_pulp-1.0.1-1.el6.noarch

• Are you using custom certificates?*

The UI cert was signed by our internal CA.

···

On Wednesday, 19 August 2015 14:32:10 UTC+1, stephen wrote: > > The host you installed katello-agent on is actually registered to the > *katello* and not Red Hat CDN? > > What version of katello-agent are you using? What version of Katello are > you running? > > Are you using custom certificates? > > On Wed, Aug 19, 2015 at 05:49:54AM -0700, JC wrote: > > Sorry for all the postings, but perhaps I'm getting somewhere... > > > > *# grep "cert=" /etc/gofer/plugins/katelloplugin.conf* > > cacert=/etc/rhsm/ca/candlepin-local.pem > > clientcert=/etc/pki/consumer/bundle.pem > > *# file /etc/rhsm/ca/candlepin-local.pem* > > /etc/rhsm/ca/candlepin-local.pem: cannot open > > `/etc/rhsm/ca/candlepin-local.pem' (No such file or directory) > > *# file /etc/pki/consumer/bundle.pem* > > /etc/pki/consumer/bundle.pem: ASCII text > > > > > > *$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text* > > # shows a cert for katello.server signed by RH, rather than my CA. > > > > Is that relevant? Can you tell that I don't really understand how the > > certs are used? ;-) > > > > On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > > > > > Ahhh, right, on the client (/var/log/messages) I see: > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - > amqps://katello.server:5647 > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent > call > > > last): > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > line 101, in open > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - domain = > > > self.ssl_domain(connector) > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > line 57, in ssl_domain > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - > > > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3386, in > > > set_trusted_ca_db > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - certificate_db) ) > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3376, in > > > _check > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > > > failure.") > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > > > failure.* > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > > > gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds > > > > > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > > > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > > > amqps://katello.server:5647|SSL: ca: > > > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > > > /etc/pki/consumer/bundle.pem|host-validation: None > > > > > > And on the server I see (ignore the times, grabbed from later in > > > /var/log/qdrouterd.log): > > > > > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection > from > > > katello.client:44791 to 0.0.0.0:5647 > > > > > > I've previously updated the Foreman UI cert using: > > > > > > $ katello-installer --certs-server-cert "katello.cer" > > > --certs-server-cert-req "katello.csr" --certs-server-key "katello.key" > > > --certs-server-ca-cert "RootAll.cer" --certs-update-server > > > --certs-update-server-ca > > > > > > Did I miss something then? The UI looks happy... > > > > > > OK, anyone know the solution? > > > > > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > > > > > OK, strace showed a problem writing to the log file. Perms changed > and > > > now we have some output... > > > > > > # cat /var/log/qdrouterd.log > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > > > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > > > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/DISPATCH, identity=log/DISPATCH, > > > type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > > > type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER_HELLO) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/SERVER, > > > identity=log/SERVER, type=org.apache.qpid.dispatch.log, > enable=default, > > > module=SERVER) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/CONTAINER, identity=log/CONTAINER, > > > type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/AGENT, > > > identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default, > > > module=AGENT) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ERROR, > > > identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default, > > > module=ERROR) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER, > > > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, > enable=default, > > > module=ROUTER) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/MESSAGE, identity=log/MESSAGE, > > > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/CONFIG, > > > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, > enable=default, > > > module=CONFIG) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=container/katello.server, > identity=container/katello.server, > > > type=org.apache.qpid.dispatch.container, containerName=katello.server, > > > workerThreads=2) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > > > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > > > routerId=katello.server, raInterval=30, mode=interior, > mobileAddrMaxAge=60) > > > > > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior > mode, > > > area=0 id=katello.server > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > > > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=16, > > > localFreeListMax=32) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_node_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_address_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router.address, > subscriberCount=0, > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > inProcess=False, > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > > > deliveriesToContainer=0) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router.address, > subscriberCount=0, > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > inProcess=False, > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > > > deliveriesToContainer=0) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router.address, > subscriberCount=0, > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > inProcess=False, > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > > > deliveriesToContainer=0) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on > > > $management > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > $management > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > $management > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > qdrouter > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > qdrouter.ma > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > qdhello > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > > > id=katello.server instance=1439983963 max_routers=1024 > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/0, identity=fixedAddress/0, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/closest, > > > bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/closest > > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/1, identity=fixedAddress/1, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/unicast, > > > bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/unicast > > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/2, identity=fixedAddress/2, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > > > prefix=/exclusive, bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > > prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/3, identity=fixedAddress/3, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > > > prefix=/multicast, bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > > prefix=/multicast phase=0 > fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/4, identity=fixedAddress/4, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > > > prefix=/broadcast, bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > > prefix=/broadcast phase=0 > fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/5, identity=fixedAddress/5, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/, > > > bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ > > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=listener/ > > > 0.0.0.0:5647, identity=listener/0.0.0.0:5647, > > > type=org.apache.qpid.dispatch.listener, requirePeerAuth=True, > > > allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS, > > > maxFrameSize=65536, role=normal, > > > certDb=/etc/pki/katello/certs/katello-default-ca.crt, > allowUnsecured=False, > > > certFile=/etc/pki/katello/qpid_router_server.crt, > > > keyFile=/etc/pki/katello/qpid_router_server.key, port=5647) > > > > > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: > 0.0.0.0:5647 > > > role=normal > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:

[JC1]

FYI: the self signed certificate in certificate chain looks to be a red
herring.

···

On Wednesday, 19 August 2015 14:34:00 UTC+1, JC wrote: > > Correction, the issuer is katello.server (with some RH text to fill in the > cert), and it is for my client! > > Certificate: > > > > * Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, > OU=SomeOrgUnit, CN=katello.server* > > Validity > > Not Before: Aug 18 14:25:23 2015 GMT > > Not After : Aug 18 14:25:23 2031 GMT > > > > Exponent: 65257 (0x10001) > > X509v3 extensions: > > Netscape Cert Type: > > SSL Client, S/MIME > > X509v3 Key Usage: > > Digital Signature, Key Encipherment, Data Encipherment > > X509v3 Authority Key Identifier: > > keyid:EA:37:47:…..2E:71:EC:91 > > * DirName:/C=US/ST=North > Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server* > > serial:A7:A2:3C:……….:65:B1 > > X509v3 Subject Key Identifier: > > FB:A7:9C:E6:7……………………………..0E:78:B8:C8 > > X509v3 Extended Key Usage: > > TLS Web Client Authentication > > * X509v3 Subject Alternative Name:* > > * URI:CN=katello.client* > > Signature Algorithm: sha1WithRSAEncryption > > 7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07: > > > > Now, from the client: > > *# openssl s_client -connect katello.server:5647 -showcerts ## same for > port 5446* > CONNECTED(00000003) > depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = > SomeOrgUnit, CN = katello.server > verify error:num=19:self signed certificate in certificate chain > verify return:0 > 140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert > handshake failure:s3_pkt.c:1259:SSL alert number 40 > 140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake > failure:s23_lib.c:184: > > > > On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote: > > Sorry for all the postings, but perhaps I'm getting somewhere... > > *# grep "cert=" /etc/gofer/plugins/katelloplugin.conf* > cacert=/etc/rhsm/ca/candlepin-local.pem > clientcert=/etc/pki/consumer/bundle.pem > *# file /etc/rhsm/ca/candlepin-local.pem* > /etc/rhsm/ca/candlepin-local.pem: cannot open > `/etc/rhsm/ca/candlepin-local.pem' (No such file or directory) > *# file /etc/pki/consumer/bundle.pem* > /etc/pki/consumer/bundle.pem: ASCII text > > > *$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text* > # shows a cert for katello.server signed by RH, rather than my CA. > > Is that relevant? Can you tell that I don't really understand how the > certs are used? ;-) > > On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > Ahhh, right, on the client (/var/log/messages) I see: > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - amqps://katello.server:5647 > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call > last): > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 101, in open > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - domain = > self.ssl_domain(connector) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > line 57, in ssl_domain > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3386, in > set_trusted_ca_db > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - certificate_db) ) > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3376, in > _check > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > failure.") > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > failure.* > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > amqps://katello.server:5647|SSL: ca: > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > /etc/pki/consumer/bundle.pem|host-validation: None > > And on the server I see (ignore the times, grabbed from later in > /var/log/qdrouterd.log): > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection from > katello.client:44791 to 0.0.0.0:5647 > > I've previously updated the Foreman UI cert using: > > $ katello-installer --certs-server-cert "katello.cer" > --certs-server-cert-req "katello.csr" --certs-server-key "katello.key" > --certs-server-ca-cert "RootAll.cer" --certs-update-server > --certs-update-server-ca > > Did I miss something then? The UI looks happy... > > OK, anyone know the solution? > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > OK, strace showed a problem writing to the log file. Perms changed and > now we have some output... > > # cat /var/log/qdrouterd.log > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/DISPATCH, identity=log/DISPATCH, > type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_HELLO) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/SERVER, > identity=log/SERVER, type=org.apache.qpid.dispatch.log, enable=default, > module=SERVER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/CONTAINER, identity=log/CONTAINER, > type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/AGENT, > identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default, > module=AGENT) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ERROR, > identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default, > module=ERROR) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ROUTER, > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/MESSAGE, identity=log/MESSAGE, > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/CONFIG, > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, enable=default, > module=CONFIG) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=container/katello.server, identity=container/katello.server, > type=org.apache.qpid.dispatch.container, containerName=katello.server, > workerThreads=2) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > routerId=katello.server, raInterval=30, mode=interior, mobileAddrMaxAge=60) > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior mode, > area=0 id=katello.server > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=16, > localFreeListMax=32) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_node_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_address_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64, > localFreeListMax=128) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0, > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False, > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > deliveriesToContainer=0) > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > $management > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdrouter.ma > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > qdhello > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > id=katello.server instance=1439983963 max_routers=1024 > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/0, identity=fixedAddress/0, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/closest, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/closest > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/1, identity=fixedAddress/1, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/unicast, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/unicast > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/2, identity=fixedAddress/2, > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/exclusive, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/3, identity=fixedAddress/3, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/multicast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/multicast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/4, identity=fixedAddress/4, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > prefix=/broadcast, bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/broadcast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=fixedAddress/5, identity=fixedAddress/5, > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/, > bias=closest) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5647, identity=listener/0.0.0.0:5647, > type=org.apache.qpid.dispatch.listener, requirePeerAuth=True, > allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS, > maxFrameSize=65536, role=normal, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > certFile=/etc/pki/katello/qpid_router_server.crt, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5647) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647 > role=normal > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/ > 0.0.0.0:5646, identity=listener/0.0.0.0:5646, > type=org.apache.qpid.dispatch.listener, > certFile=/etc/pki/katello/qpid_router_server.crt, addr=0.0.0.0, > saslMechanisms=ANONYMOUS, maxFrameSize=65536, requirePeerAuth=True, > certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False, > allowNoSasl=False, role=inter-router, > keyFile=/etc/pki/katello/qpid_router_server.key, port=5646) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5646 > role=inter-router > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=broker, > identity=connector/katello.server:5671, > type=org.apache.qpid.dispatch.connector, > certFile=/etc/pki/katello/qpid_router_client.crt, addr=katello.server, > allowRedirect=True, saslMechanisms=ANONYMOUS, maxFrameSize=65536, > role=on-demand, certDb=/etc/pki/katello/certs/katello-default-ca.crt, > keyFile=/etc/pki/katello/qpid_router_client.key, port=5671) > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured on-demand connector: > katello.server:5671 name=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/0, identity=linkRoutePattern/0, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=pulp.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=pulp. connector=broker > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=linkRoutePattern/1, identity=linkRoutePattern/1, > type=org.apache.qpid.dispatch.linkRoutePattern, connector=broker, > prefix=qmf.) > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Link-route-pattern: > prefix=qmf. connector=broker > > Wed Aug 19 12:32:43 2015 SERVER (info) Operational, 2 Threads Running > > Wed Aug 19 12:32:43 2015 CONN_MGR (info) Starting on-demand connector: > broker > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activating Prefix 'pulp.' for > routed links to 'broker' > > Wed Aug 19 12:32:44 2015 ROUTER (info) Activ

[stbenjam]

> Correction, the issuer is katello.server (with some RH text to fill in the
> cert), and it is for my client!
>
> Certificate:
>
> <snip>
>
> * Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello,
> OU=SomeOrgUnit, CN=katello.server*
>
> Validity
>
> Not Before: Aug 18 14:25:23 2015 GMT
>
> Not After : Aug 18 14:25:23 2031 GMT
>
> <snip>
>
> Exponent: 65257 (0x10001)
>
> X509v3 extensions:
>
> Netscape Cert Type:
>
> SSL Client, S/MIME
>
> X509v3 Key Usage:
>
> Digital Signature, Key Encipherment, Data Encipherment
>
> X509v3 Authority Key Identifier:
>
> keyid:EA:37:47:……2E:71:EC:91
>
> * DirName:/C=US/ST=North
> Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server*
>
> serial:A7:A2:3C:……….:65:B1
>
> X509v3 Subject Key Identifier:
>
> FB:A7:9C:E6:7………………………………0E:78:B8:C8
>
> X509v3 Extended Key Usage:
>
> TLS Web Client Authentication
>
> * X509v3 Subject Alternative Name:*
>
> * URI:CN=katello.client*
>
> Signature Algorithm: sha1WithRSAEncryption
>
> 7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07:
>
> <snip>
>
> Now, from the client:
>
> # openssl s_client -connect katello.server:5647 -showcerts ## same for
> port 5446
> CONNECTED(00000003)
> depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU =
> SomeOrgUnit, CN = katello.server
> verify error:num=19:self signed certificate in certificate chain
> verify return:0
> 140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert
> handshake failure:s3_pkt.c:1259:SSL alert number 40
> 140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
> failure:s23_lib.c:184:

That's expected, we don't use the UI cert for qpid.

You have the latest katello-ca-consumer-latest from the Katello server
installed on the client? (it's in /pub on the web server)

···

On Wed, Aug 19, 2015 at 06:33:59AM -0700, JC wrote:

On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote:

Sorry for all the postings, but perhaps I’m getting somewhere…

# grep “cert=” /etc/gofer/plugins/katelloplugin.conf
cacert=/etc/rhsm/ca/candlepin-local.pem
clientcert=/etc/pki/consumer/bundle.pem
# file /etc/rhsm/ca/candlepin-local.pem
/etc/rhsm/ca/candlepin-local.pem: cannot open
`/etc/rhsm/ca/candlepin-local.pem’ (No such file or directory)
# file /etc/pki/consumer/bundle.pem
/etc/pki/consumer/bundle.pem: ASCII text

$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text

shows a cert for katello.server signed by RH, rather than my CA.

Is that relevant? Can you tell that I don’t really understand how the
certs are used?

On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote:

Ahhh, right, on the client (/var/log/messages) I see:

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - amqps://katello.server:5647

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most recent call
last):

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py",
line 101, in open

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - domain =
self.ssl_domain(connector)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py",
line 57, in ssl_domain

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
domain.set_trusted_ca_db(connector.ssl.ca_certificate)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3386, in
set_trusted_ca_db

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - certificate_db) )

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3376, in
_check

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise exc(“SSL
failure.”)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - SSLException: SSL
failure.

Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://katello.server:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

And on the server I see (ignore the times, grabbed from later in
/var/log/qdrouterd.log):

Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection from
katello.client:44791 to 0.0.0.0:5647

I’ve previously updated the Foreman UI cert using:

$ katello-installer --certs-server-cert “katello.cer”
–certs-server-cert-req “katello.csr” --certs-server-key “katello.key”
–certs-server-ca-cert “RootAll.cer” --certs-update-server
–certs-update-server-ca

Did I miss something then? The UI looks happy…

OK, anyone know the solution?

On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote:

OK, strace showed a problem writing to the log file. Perms changed and
now we have some output…

cat /var/log/qdrouterd.log

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/DISPATCH, identity=log/DISPATCH,
type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_HELLO)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/SERVER,
identity=log/SERVER, type=org.apache.qpid.dispatch.log, enable=default,
module=SERVER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONTAINER, identity=log/CONTAINER,
type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/AGENT,
identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default,
module=AGENT)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ERROR,
identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default,
module=ERROR)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/ROUTER,
identity=log/ROUTER, type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/MESSAGE, identity=log/MESSAGE,
type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=log/CONFIG,
identity=log/CONFIG, type=org.apache.qpid.dispatch.log, enable=default,
module=CONFIG)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=container/katello.server, identity=container/katello.server,
type=org.apache.qpid.dispatch.container, containerName=katello.server,
workerThreads=2)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4,
helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60,
routerId=katello.server, raInterval=30, mode=interior, mobileAddrMaxAge=60)

Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server

Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior mode,
area=0 id=katello.server

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16,
typeSize=2104, transferBatchSize=16, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_log_entry_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=16,
localFreeListMax=32)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=88, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=32, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_item_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_node_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=136, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_bitmask_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_timer_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=216, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_address_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=16, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0, totalAllocFromHeap=64,
localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0,
deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False,
deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0,
deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False,
deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address, subscriberCount=0,
deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, inProcess=False,
deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdrouter

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdrouter.ma

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdhello

Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated:
id=katello.server instance=1439983963 max_routers=1024

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/0, identity=fixedAddress/0,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/closest,
bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/closest
phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/1, identity=fixedAddress/1,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single, prefix=/unicast,
bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/unicast
phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/2, identity=fixedAddress/2,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single,
prefix=/exclusive, bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/3, identity=fixedAddress/3,
type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple,
prefix=/multicast, bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/multicast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/4, identity=fixedAddress/4,
type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple,
prefix=/broadcast, bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/broadcast phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/5, identity=fixedAddress/5,
type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/,
bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/
phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/
0.0.0.0:5647, identity=listener/0.0.0.0:5647,
type=org.apache.qpid.dispatch.listener, requirePeerAuth=True,
allowNoSasl=False, addr=0.0.0.0, saslMechanisms=ANONYMOUS,
maxFrameSize=65536, role=normal,
certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False,
certFile=/etc/pki/katello/qpid_router_server.crt,
keyFile=/etc/pki/katello/qpid_router_server.key, port=5647)

Wed Aug 19 12:32:43 2015 CONN_MGR (info) Configured Listener: 0.0.0.0:5647
role=normal

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: Entity(name=listener/
0.0.0.0:5646, identity=listener/0.0.0.0:5646,
type=org.apache.qpid.dispatch.listener,
certFile=/etc/pki/katello/qpid_router_server.crt, addr=0.0.0.0,
saslMechanisms=ANONYMOUS, maxFrameSize=65536, requirePeerAuth=True,
certDb=/etc/pki/katello/certs/katello-default-ca.crt, allowUnsecured=False,
allowNoSasl=False, role=inter-router,
keyFile=/etc/pki/katello/qpid_route

[JC1]

Morning Stephen,

Yes, they match. I have checked the date/time stamps on
https://katello.server/pub/katello-ca-consumer-katello.server-1.0-2.noarch.rpm
with that on the client (using rpm to review the build date).

Installed on the client, it provides:

/etc/rhsm/ca/katello-server-ca.pem

Examining this cert shows it to be issued by our internal CA.

···

On Wednesday, 19 August 2015 17:43:42 UTC+1, stephen wrote: > > On Wed, Aug 19, 2015 at 06:33:59AM -0700, JC wrote: > > Correction, the issuer is katello.server (with some RH text to fill in > the > > cert), and it is for my client! > > > > Certificate: > > > > > > > > * Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, > > OU=SomeOrgUnit, CN=katello.server* > > > > Validity > > > > Not Before: Aug 18 14:25:23 2015 GMT > > > > Not After : Aug 18 14:25:23 2031 GMT > > > > > > > > Exponent: 65257 (0x10001) > > > > X509v3 extensions: > > > > Netscape Cert Type: > > > > SSL Client, S/MIME > > > > X509v3 Key Usage: > > > > Digital Signature, Key Encipherment, Data Encipherment > > > > X509v3 Authority Key Identifier: > > > > keyid:EA:37:47:…..2E:71:EC:91 > > > > * DirName:/C=US/ST=North > > Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server* > > > > serial:A7:A2:3C:……….:65:B1 > > > > X509v3 Subject Key Identifier: > > > > FB:A7:9C:E6:7……………………………..0E:78:B8:C8 > > > > X509v3 Extended Key Usage: > > > > TLS Web Client Authentication > > > > * X509v3 Subject Alternative Name:* > > > > * URI:CN=katello.client* > > > > Signature Algorithm: sha1WithRSAEncryption > > > > 7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07: > > > > > > > > Now, from the client: > > > > *# openssl s_client -connect katello.server:5647 -showcerts ## same for > > port 5446* > > CONNECTED(00000003) > > depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = > > SomeOrgUnit, CN = katello.server > > verify error:num=19:self signed certificate in certificate chain > > verify return:0 > > 140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert > > handshake failure:s3_pkt.c:1259:SSL alert number 40 > > 140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake > > failure:s23_lib.c:184: > > > That's expected, we don't use the UI cert for qpid. > > You have the latest katello-ca-consumer-latest from the Katello server > installed on the client? (it's in /pub on the web server) > > > > > > > > > On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote: > > > > > > Sorry for all the postings, but perhaps I'm getting somewhere... > > > > > > *# grep "cert=" /etc/gofer/plugins/katelloplugin.conf* > > > cacert=/etc/rhsm/ca/candlepin-local.pem > > > clientcert=/etc/pki/consumer/bundle.pem > > > *# file /etc/rhsm/ca/candlepin-local.pem* > > > /etc/rhsm/ca/candlepin-local.pem: cannot open > > > `/etc/rhsm/ca/candlepin-local.pem' (No such file or directory) > > > *# file /etc/pki/consumer/bundle.pem* > > > /etc/pki/consumer/bundle.pem: ASCII text > > > > > > > > > *$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text* > > > # shows a cert for katello.server signed by RH, rather than my CA. > > > > > > Is that relevant? Can you tell that I don't really understand how the > > > certs are used? ;-) > > > > > > On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > > > > > Ahhh, right, on the client (/var/log/messages) I see: > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - > amqps://katello.server:5647 > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - Traceback (most recent > call > > > last): > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > line 101, in open > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - domain = > > > self.ssl_domain(connector) > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > line 57, in ssl_domain > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - > > > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3386, in > > > set_trusted_ca_db > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - certificate_db) ) > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - File > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3376, in > > > _check > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > > > failure.") > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > > > failure.* > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > > > gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds > > > > > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > > > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > > > amqps://katello.server:5647|SSL: ca: > > > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > > > /etc/pki/consumer/bundle.pem|host-validation: None > > > > > > And on the server I see (ignore the times, grabbed from later in > > > /var/log/qdrouterd.log): > > > > > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection > from > > > katello.client:44791 to 0.0.0.0:5647 > > > > > > I've previously updated the Foreman UI cert using: > > > > > > $ katello-installer --certs-server-cert "katello.cer" > > > --certs-server-cert-req "katello.csr" --certs-server-key "katello.key" > > > --certs-server-ca-cert "RootAll.cer" --certs-update-server > > > --certs-update-server-ca > > > > > > Did I miss something then? The UI looks happy... > > > > > > OK, anyone know the solution? > > > > > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > > > > > OK, strace showed a problem writing to the log file. Perms changed > and > > > now we have some output... > > > > > > # cat /var/log/qdrouterd.log > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > > > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > > > type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/DISPATCH, identity=log/DISPATCH, > > > type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > > > type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER_HELLO) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/SERVER, > > > identity=log/SERVER, type=org.apache.qpid.dispatch.log, > enable=default, > > > module=SERVER) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/CONTAINER, identity=log/CONTAINER, > > > type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/AGENT, > > > identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default, > > > module=AGENT) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ERROR, > > > identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default, > > > module=ERROR) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/ROUTER, > > > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, > enable=default, > > > module=ROUTER) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/MESSAGE, identity=log/MESSAGE, > > > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > Entity(name=log/CONFIG, > > > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, > enable=default, > > > module=CONFIG) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=container/katello.server, > identity=container/katello.server, > > > type=org.apache.qpid.dispatch.container, containerName=katello.server, > > > workerThreads=2) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > > > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > > > routerId=katello.server, raInterval=30, mode=interior, > mobileAddrMaxAge=60) > > > > > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior > mode, > > > area=0 id=katello.server > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > > > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=16, > > > localFreeListMax=32) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_node_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_address_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > > > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > totalAllocFromHeap=64, > > > localFreeListMax=128) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router.address, > subscriberCount=0, > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > inProcess=False, > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > > > deliveriesToContainer=0) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router.address, > subscriberCount=0, > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > inProcess=False, > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > > > deliveriesToContainer=0) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(type=org.apache.qpid.dispatch.router.address, > subscriberCount=0, > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > inProcess=False, > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > > > deliveriesToContainer=0) > > > > > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on > > > $management > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > $management > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > $management > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > qdrouter > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > qdrouter.ma > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered: > > > qdhello > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > > > id=katello.server instance=1439983963 max_routers=1024 > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/0, identity=fixedAddress/0, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/closest, > > > bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/closest > > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/1, identity=fixedAddress/1, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > prefix=/unicast, > > > bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > prefix=/unicast > > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/2, identity=fixedAddress/2, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > > > prefix=/exclusive, bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > > prefix=/exclusive phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE > > > bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/3, identity=fixedAddress/3, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > > > prefix=/multicast, bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > > prefix=/multicast phase=0 > fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/4, identity=fixedAddress/4, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, > > > prefix=/broadcast, bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: > > > prefix=/broadcast phase=0 > fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=fixedAddress/5, identity=fixedAddress/5, > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=multiple, prefix=/, > > > bias=closest) > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address: prefix=/ > > > phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_MULTIPLE > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entit

[stbenjam]

> Morning Stephen,
>
> Yes, they match. I have checked the date/time stamps on
> https://katello.server/pub/katello-ca-consumer-katello.server-1.0-2.noarch.rpm
> with that on the client (using rpm to review the build date).
>
> Installed on the client, it provides:
>
> /etc/rhsm/ca/katello-server-ca.pem

It doesn't provide katello-default-ca.pem?

That is the one from the self-signed cert and the one that katello-agent
needs to use to connect to qpid.

···

On Thu, Aug 20, 2015 at 12:36:37AM -0700, JC wrote:

Examining this cert shows it to be issued by our internal CA.

On Wednesday, 19 August 2015 17:43:42 UTC+1, stephen wrote:

On Wed, Aug 19, 2015 at 06:33:59AM -0700, JC wrote:

Correction, the issuer is katello.server (with some RH text to fill in
the
cert), and it is for my client!

Certificate:

•    Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello,
  

OU=SomeOrgUnit, CN=katello.server*

    Validity

        Not Before: Aug 18 14:25:23 2015 GMT

        Not After : Aug 18 14:25:23 2031 GMT

            Exponent: 65257 (0x10001)

    X509v3 extensions:

        Netscape Cert Type:

            SSL Client, S/MIME

        X509v3 Key Usage:

            Digital Signature, Key Encipherment, Data Encipherment

        X509v3 Authority Key Identifier:

            keyid:EA:37:47:…..2E:71:EC:91

•            DirName:/C=US/ST=North
  

Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server*

            serial:A7:A2:3C:……….:65:B1

        X509v3 Subject Key Identifier:

            FB:A7:9C:E6:7……………………………..0E:78:B8:C8

        X509v3 Extended Key Usage:

            TLS Web Client Authentication

•        X509v3 Subject Alternative Name:*
  

•            URI:CN=katello.client*
  

  Signature Algorithm: sha1WithRSAEncryption

     7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07:
  
     <snip>
  

Now, from the client:

# openssl s_client -connect katello.server:5647 -showcerts ## same for
port 5446
CONNECTED(00000003)
depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU =
SomeOrgUnit, CN = katello.server
verify error:num=19:self signed certificate in certificate chain
verify return:0
140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert
handshake failure:s3_pkt.c:1259:SSL alert number 40
140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
failure:s23_lib.c:184:

That’s expected, we don’t use the UI cert for qpid.

You have the latest katello-ca-consumer-latest from the Katello server
installed on the client? (it’s in /pub on the web server)

On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote:

Sorry for all the postings, but perhaps I’m getting somewhere…

# grep “cert=” /etc/gofer/plugins/katelloplugin.conf
cacert=/etc/rhsm/ca/candlepin-local.pem
clientcert=/etc/pki/consumer/bundle.pem
# file /etc/rhsm/ca/candlepin-local.pem
/etc/rhsm/ca/candlepin-local.pem: cannot open
`/etc/rhsm/ca/candlepin-local.pem’ (No such file or directory)
# file /etc/pki/consumer/bundle.pem
/etc/pki/consumer/bundle.pem: ASCII text

$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text

shows a cert for katello.server signed by RH, rather than my CA.

Is that relevant? Can you tell that I don’t really understand how the
certs are used?

On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote:

Ahhh, right, on the client (/var/log/messages) I see:

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
amqps://katello.server:5647

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most recent
call

last):

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 101, in open

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - domain =
self.ssl_domain(connector)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 57, in ssl_domain

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
domain.set_trusted_ca_db(connector.ssl.ca_certificate)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3386, in
set_trusted_ca_db

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - certificate_db) )

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line 3376, in
_check

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise exc(“SSL
failure.”)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - SSLException: SSL
failure.

Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://katello.server:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

And on the server I see (ignore the times, grabbed from later in
/var/log/qdrouterd.log):

Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming connection
from

katello.client:44791 to 0.0.0.0:5647

I’ve previously updated the Foreman UI cert using:

$ katello-installer --certs-server-cert “katello.cer”
–certs-server-cert-req “katello.csr” --certs-server-key “katello.key”
–certs-server-ca-cert “RootAll.cer” --certs-update-server
–certs-update-server-ca

Did I miss something then? The UI looks happy…

OK, anyone know the solution?

On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote:

OK, strace showed a problem writing to the log file. Perms changed
and

now we have some output…

cat /var/log/qdrouterd.log

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_LS)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA,
type=org.apache.qpid.dispatch.log, enable=default, module=ROUTER_MA)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/DISPATCH, identity=log/DISPATCH,
type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO,
type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER_HELLO)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/SERVER,

identity=log/SERVER, type=org.apache.qpid.dispatch.log,
enable=default,

module=SERVER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONTAINER, identity=log/CONTAINER,
type=org.apache.qpid.dispatch.log, enable=default, module=CONTAINER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/AGENT,

identity=log/AGENT, type=org.apache.qpid.dispatch.log, enable=default,
module=AGENT)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ERROR,

identity=log/ERROR, type=org.apache.qpid.dispatch.log, enable=default,
module=ERROR)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER,

identity=log/ROUTER, type=org.apache.qpid.dispatch.log,
enable=default,

module=ROUTER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/MESSAGE, identity=log/MESSAGE,
type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONFIG,

identity=log/CONFIG, type=org.apache.qpid.dispatch.log,
enable=default,

module=CONFIG)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=container/katello.server,
identity=container/katello.server,

type=org.apache.qpid.dispatch.container, containerName=katello.server,
workerThreads=2)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4,
helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60,
routerId=katello.server, raInterval=30, mode=interior,
mobileAddrMaxAge=60)

Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: katello.server

Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior
mode,

area=0 id=katello.server

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16,
typeSize=2104, transferBatchSize=16, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_log_entry_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=16,

localFreeListMax=32)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=88, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=32, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_item_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_node_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=136, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_bitmask_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_timer_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=216, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_address_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=16, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address,
subscriberCount=0,

deliveriesEgress=0, deliveriesIngress=0, remoteCount=0,
inProcess=False,

deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address,
subscriberCount=0,

deliveriesEgress=0, deliveriesIngress=0, remoteCount=0,
inProcess=False,

deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address,
subscriberCount=0,

deliveriesEgress=0, deliveriesIngress=0, remoteCount=0,
inProcess=False,

deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent on
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
$management

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdrouter

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdrouter.ma

Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address Registered:
qdhello

Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated:
id=katello.server instance=1439983963 max_routers=1024

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/0, identity=fixedAddress/0,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single,
prefix=/closest,

bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/closest

phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/1, identity=fixedAddress/1,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single,
prefix=/unicast,

bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/unicast

phase=0 fanout=QD_SCHEMA_FIXEDADDRESS_FANOUT_SINGLE
bias=QD_SCHEMA_FIXEDADDRESS_BIAS_CLOSEST

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=fixedAddress/2, identity=fixedAddress/2,
type=org.apache.qpid.dispatch.fixedAddress, fanout=single,
prefix=/exclusive, bias=closest)

Wed Aug 19 12:32:43 2015 ROUTER (info) Configured Address:
prefix=/exclusive phase=0 fanout=

[ehelms]

JC,

Trying to read back, I think I understand that you are using custom
certificates and hitting a katello-agent issue. A user on IRC, tamarin,
made me realize in discussion today that I never backported to 2.2 a fix
for custom certificates and katello-agent whereby we aren't laying down the
katello-default-ca.crt (this is fixed in 2.3). You can see here –

theforeman/puppet-certs/blob/master/manifests/katello.pp

this exact comment which sounds like your issue. If you need this fixed in
2.2 vs just upgrading to 2.3 (and all the excellent work and bugs fixes
that went into it) please let me know.

Eric

···

On Thu, Aug 20, 2015 at 8:25 AM, Stephen Benjamin wrote:

On Thu, Aug 20, 2015 at 12:36:37AM -0700, JC wrote:

Morning Stephen,

Yes, they match. I have checked the date/time stamps on

https://katello.server/pub/katello-ca-consumer-katello.server-1.0-2.noarch.rpm

with that on the client (using rpm to review the build date).

Installed on the client, it provides:

/etc/rhsm/ca/katello-server-ca.pem

It doesn’t provide katello-default-ca.pem?

That is the one from the self-signed cert and the one that katello-agent
needs to use to connect to qpid.

Examining this cert shows it to be issued by our internal CA.

On Wednesday, 19 August 2015 17:43:42 UTC+1, stephen wrote:

On Wed, Aug 19, 2015 at 06:33:59AM -0700, JC wrote:

Correction, the issuer is katello.server (with some RH text to fill
in

the

cert), and it is for my client!

Certificate:

•    Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello,
  

OU=SomeOrgUnit, CN=katello.server*

    Validity

        Not Before: Aug 18 14:25:23 2015 GMT

        Not After : Aug 18 14:25:23 2031 GMT

            Exponent: 65257 (0x10001)

    X509v3 extensions:

        Netscape Cert Type:

            SSL Client, S/MIME

        X509v3 Key Usage:

            Digital Signature, Key Encipherment, Data

Encipherment

        X509v3 Authority Key Identifier:

            keyid:EA:37:47:…..2E:71:EC:91

•            DirName:/C=US/ST=North
  

Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server*

            serial:A7:A2:3C:……….:65:B1

        X509v3 Subject Key Identifier:

            FB:A7:9C:E6:7……………………………..0E:78:B8:C8

        X509v3 Extended Key Usage:

            TLS Web Client Authentication

•        X509v3 Subject Alternative Name:*
  

•            URI:CN=katello.client*
  

  Signature Algorithm: sha1WithRSAEncryption

     7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07:
  
     <snip>
  

Now, from the client:

*# openssl s_client -connect katello.server:5647 -showcerts ## same
for

port 5446*
CONNECTED(00000003)
depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU =
SomeOrgUnit, CN = katello.server
verify error:num=19:self signed certificate in certificate chain
verify return:0
140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3
alert

handshake failure:s3_pkt.c:1259:SSL alert number 40
140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
failure:s23_lib.c:184:

That’s expected, we don’t use the UI cert for qpid.

You have the latest katello-ca-consumer-latest from the Katello server
installed on the client? (it’s in /pub on the web server)

On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote:

Sorry for all the postings, but perhaps I’m getting somewhere…

# grep “cert=” /etc/gofer/plugins/katelloplugin.conf
cacert=/etc/rhsm/ca/candlepin-local.pem
clientcert=/etc/pki/consumer/bundle.pem
# file /etc/rhsm/ca/candlepin-local.pem
/etc/rhsm/ca/candlepin-local.pem: cannot open
`/etc/rhsm/ca/candlepin-local.pem’ (No such file or directory)
# file /etc/pki/consumer/bundle.pem
/etc/pki/consumer/bundle.pem: ASCII text

$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text

shows a cert for katello.server signed by RH, rather than my CA.

Is that relevant? Can you tell that I don’t really understand how
the

certs are used?

On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote:

Ahhh, right, on the client (/var/log/messages) I see:

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
amqps://katello.server:5647

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most
recent

call

last):

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 101, in open

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - domain =
self.ssl_domain(connector)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 57, in ssl_domain

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
domain.set_trusted_ca_db(connector.ssl.ca_certificate)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line
3386, in

set_trusted_ca_db

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
certificate_db) )

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line
3376, in

_check

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise exc(“SSL
failure.”)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - SSLException: SSL
failure.

Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 106
seconds

Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://katello.server:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

And on the server I see (ignore the times, grabbed from later in
/var/log/qdrouterd.log):

Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming
connection

from

katello.client:44791 to 0.0.0.0:5647

I’ve previously updated the Foreman UI cert using:

$ katello-installer --certs-server-cert “katello.cer”
–certs-server-cert-req “katello.csr” --certs-server-key
"katello.key"

–certs-server-ca-cert “RootAll.cer” --certs-update-server
–certs-update-server-ca

Did I miss something then? The UI looks happy…

OK, anyone know the solution?

On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote:

OK, strace showed a problem writing to the log file. Perms changed
and

now we have some output…

cat /var/log/qdrouterd.log

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS,
type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER_LS)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA,
type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER_MA)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/DISPATCH, identity=log/DISPATCH,
type=org.apache.qpid.dispatch.log, enable=default, module=DISPATCH)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO,
type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER_HELLO)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/SERVER,

identity=log/SERVER, type=org.apache.qpid.dispatch.log,
enable=default,

module=SERVER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONTAINER, identity=log/CONTAINER,
type=org.apache.qpid.dispatch.log, enable=default,
module=CONTAINER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/AGENT,

identity=log/AGENT, type=org.apache.qpid.dispatch.log,
enable=default,

module=AGENT)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ERROR,

identity=log/ERROR, type=org.apache.qpid.dispatch.log,
enable=default,

module=ERROR)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER,

identity=log/ROUTER, type=org.apache.qpid.dispatch.log,
enable=default,

module=ROUTER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/MESSAGE, identity=log/MESSAGE,
type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONFIG,

identity=log/CONFIG, type=org.apache.qpid.dispatch.log,
enable=default,

module=CONFIG)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=container/katello.server,
identity=container/katello.server,

type=org.apache.qpid.dispatch.container,
containerName=katello.server,

workerThreads=2)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4,
helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60,
routerId=katello.server, raInterval=30, mode=interior,
mobileAddrMaxAge=60)

Wed Aug 19 12:32:43 2015 SERVER (info) Container Name:
katello.server

Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior
mode,

area=0 id=katello.server

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16,
typeSize=2104, transferBatchSize=16, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_log_entry_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=16,

localFreeListMax=32)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=88, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=32, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_item_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_node_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=136, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_bitmask_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_timer_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=216, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_address_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=16, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address,
subscriberCount=0,

deliveriesEgress=0, deliveriesIngress=0, remoteCount=0,
inProcess=False,

deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address,
subscriberCount=0,

deliveriesEgress=0, deliveriesIngress=0, remoteCount=0,
inProcess=False,

deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma,
deliveriesToContainer=0)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router.address,
subscriberCount=0,

deliveriesEgress=0, deliveriesIngress=0, remoteCount=0,
inProcess=False,

deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello,
deli

[JC1]

Folks, any further thoughts on this please? Our deployment of Katello has
now stalled… Many thanks.

···

On Friday, 21 August 2015 13:41:40 UTC+1, JC wrote: > > OK, I'm now running 2.3. > > > -------------------------------------------------------------------------------------------------------------------------------------------------------- > The installer told me something went wrong. The installer log says: > > [ERROR 2015-08-21 12:27:22 main] Repeating errors encountered during run: > > [ERROR 2015-08-21 12:27:22 main] nil > > [ERROR 2015-08-21 12:27:22 main] httpd is stopped > > [ERROR 2015-08-21 12:27:22 main] *does not exist at > /usr/share/katello-installer/modules/certs/manifests/init.pp:115* on node > ptnousvapp100.rnet.dstl.gov.uk > > But as already[1] seen, it doesn't seem to have prevented things from > working. > 1. https://bugzilla.redhat.com/show_bug.cgi?id=1252932 > > -------------------------------------------------------------------------------------------------------------------------------------------------------- > > Crucially, I *still* get the same problem with goferd, when trying to > push out an update to clients (Host did not respond within 20 seconds. Is > katello-agent installed and goferd running on the Host?). Gutted. On a > new CentOS6 client (in /var/log/messages): > > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3654, in > dispatch > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - result = > dispatch(handler, type.method, self) > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line 3543, in > dispatch > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - return m(*args) > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 257, in > on_transport_tail_closed > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - > self.on_transport_closed(event) > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - File > "/usr/lib64/python2.6/site-packages/proton/utils.py", line 261, in > on_transport_closed > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - raise > ConnectionException("Connection %s disconnected" % self.url); > Aug 21 13:14:28 katello.client goferd: [ERROR][MainThread] > gofer.messaging.adapter.proton.connection:106 - ConnectionException: > Connection amqps://katello.server:5647 disconnected > Aug 21 13:14:28 katello.client goferd: [INFO][MainThread] > gofer.messaging.adapter.proton.connection:108 - retry in 74 seconds > > /var/log/qdrouter.log on the katello server reports: > > Fri Aug 21 13:17:12 2015 SERVER (debug) Accepting incoming connection from > katello.client:37399 to 0.0.0.0:5647 > > Going back to one of my earlier posts, on the client, shouldn't 'cacert=' > in /etc/gofer/plugins/katelloplugin.conf put to a file that exists? My > conf file points to /etc/rhsm/ca/candlepin-local.pem (which isn't there). > The clientcert= looks healthier, pointing to /etc/pki/consumer/bundle.pem. > > On Thursday, 20 August 2015 13:25:04 UTC+1, stephen wrote: > > On Thu, Aug 20, 2015 at 12:36:37AM -0700, JC wrote: > > Morning Stephen, > > > > Yes, they match. I have checked the date/time stamps on > > > https://katello.server/pub/katello-ca-consumer-katello.server-1.0-2.noarch.rpm > > with that on the client (using rpm to review the build date). > > > > Installed on the client, it provides: > > > > /etc/rhsm/ca/katello-server-ca.pem > > It doesn't provide katello-default-ca.pem? > > That is the one from the self-signed cert and the one that katello-agent > needs to use to connect to qpid. > > > > > Examining this cert shows it to be issued by our internal CA. > > > > On Wednesday, 19 August 2015 17:43:42 UTC+1, stephen wrote: > > > > > > On Wed, Aug 19, 2015 at 06:33:59AM -0700, JC wrote: > > > > Correction, the issuer is katello.server (with some RH text to fill > in > > > the > > > > cert), and it is for my client! > > > > > > > > Certificate: > > > > > > > > > > > > > > > > * Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, > > > > OU=SomeOrgUnit, CN=katello.server* > > > > > > > > Validity > > > > > > > > Not Before: Aug 18 14:25:23 2015 GMT > > > > > > > > Not After : Aug 18 14:25:23 2031 GMT > > > > > > > > > > > > > > > > Exponent: 65257 (0x10001) > > > > > > > > X509v3 extensions: > > > > > > > > Netscape Cert Type: > > > > > > > > SSL Client, S/MIME > > > > > > > > X509v3 Key Usage: > > > > > > > > Digital Signature, Key Encipherment, Data > Encipherment > > > > > > > > X509v3 Authority Key Identifier: > > > > > > > > keyid:EA:37:47:…..2E:71:EC:91 > > > > > > > > * DirName:/C=US/ST=North > > > > Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server* > > > > > > > > serial:A7:A2:3C:……….:65:B1 > > > > > > > > X509v3 Subject Key Identifier: > > > > > > > > FB:A7:9C:E6:7……………………………..0E:78:B8:C8 > > > > > > > > X509v3 Extended Key Usage: > > > > > > > > TLS Web Client Authentication > > > > > > > > * X509v3 Subject Alternative Name:* > > > > > > > > * URI:CN=katello.client* > > > > > > > > Signature Algorithm: sha1WithRSAEncryption > > > > > > > > 7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07: > > > > > > > > > > > > > > > > Now, from the client: > > > > > > > > *# openssl s_client -connect katello.server:5647 -showcerts ## same > for > > > > port 5446* > > > > CONNECTED(00000003) > > > > depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = > > > > SomeOrgUnit, CN = katello.server > > > > verify error:num=19:self signed certificate in certificate chain > > > > verify return:0 > > > > 140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 > alert > > > > handshake failure:s3_pkt.c:1259:SSL alert number 40 > > > > 140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl > handshake > > > > failure:s23_lib.c:184: > > > > > > > > > That's expected, we don't use the UI cert for qpid. > > > > > > You have the latest katello-ca-consumer-latest from the Katello server > > > installed on the client? (it's in /pub on the web server) > > > > > > > > > > > > > > > > > > > On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote: > > > > > > > > > > Sorry for all the postings, but perhaps I'm getting somewhere... > > > > > > > > > > *# grep "cert=" /etc/gofer/plugins/katelloplugin.conf* > > > > > cacert=/etc/rhsm/ca/candlepin-local.pem > > > > > clientcert=/etc/pki/consumer/bundle.pem > > > > > *# file /etc/rhsm/ca/candlepin-local.pem* > > > > > /etc/rhsm/ca/candlepin-local.pem: cannot open > > > > > `/etc/rhsm/ca/candlepin-local.pem' (No such file or directory) > > > > > *# file /etc/pki/consumer/bundle.pem* > > > > > /etc/pki/consumer/bundle.pem: ASCII text > > > > > > > > > > > > > > > *$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text* > > > > > # shows a cert for katello.server signed by RH, rather than my CA. > > > > > > > > > > Is that relevant? Can you tell that I don't really understand how > the > > > > > certs are used? ;-) > > > > > > > > > > On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > > > > > > > > > Ahhh, right, on the client (/var/log/messages) I see: > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - > > > amqps://katello.server:5647 > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - Traceback (most > recent > > > call > > > > > last): > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > > > > > > line 101, in open > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - domain = > > > > > self.ssl_domain(connector) > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > > > > > > line 57, in ssl_domain > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - > > > > > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line > 3386, in > > > > > set_trusted_ca_db > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - > certificate_db) ) > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line > 3376, in > > > > > _check > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > > > > > failure.") > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > > > > > failure.* > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:108 - retry in 106 > seconds > > > > > > > > > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > > > > > amqps://katello.server:5647|SSL: ca: > > > > > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > > > > > /etc/pki/consumer/bundle.pem|host-validation: None > > > > > > > > > > And on the server I see (ignore the times, grabbed from later in > > > > > /var/log/qdrouterd.log): > > > > > > > > > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming > connection > > > from > > > > > katello.client:44791 to 0.0.0.0:5647 > > > > > > > > > > I've previously updated the Foreman UI cert using: > > > > > > > > > > $ katello-installer --certs-server-cert "katello.cer" > > > > > --certs-server-cert-req "katello.csr" --certs-server-key > "katello.key" > > > > > --certs-server-ca-cert "RootAll.cer" --certs-update-server > > > > > --certs-update-server-ca > > > > > > > > > > Did I miss something then? The UI looks happy... > > > > > > > > > > OK, anyone know the solution? > > > > > > > > > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > > > > > > > > > OK, strace showed a problem writing to the log file. Perms > changed > > > and > > > > > now we have some output... > > > > > > > > > > # cat /var/log/qdrouterd.log > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER_LS) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER_MA) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/DISPATCH, identity=log/DISPATCH, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=DISPATCH) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > > > module=ROUTER_HELLO) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/SERVER, > > > > > identity=log/SERVER, type=org.apache.qpid.dispatch.log, > > > enable=default, > > > > > module=SERVER) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/CONTAINER, identity=log/CONTAINER, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=CONTAINER) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/AGENT, > > > > > identity=log/AGENT, type=org.apache.qpid.dispatch.log, > enable=default, > > > > > module=AGENT) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ERROR, > > > > > identity=log/ERROR, type=org.apache.qpid.dispatch.log, > enable=default, > > > > > module=ERROR) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER, > > > > > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, > > > enable=default, > > > > > module=ROUTER) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/MESSAGE, identity=log/MESSAGE, > > > > > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/CONFIG, > > > > > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, > > > enable=default, > > > > > module=CONFIG) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=container/katello.server, > > > identity=container/katello.server, > > > > > type=org.apache.qpid.dispatch.container, > containerName=katello.server, > > > > > workerThreads=2) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > > > > > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > > > > > routerId=katello.server, raInterval=30, mode=interior, > > > mobileAddrMaxAge=60) > > > > > > > > > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: > katello.server > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior > > > mode, > > > > > area=0 id=katello.server > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > > > > > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=16, > > > > > localFreeListMax=32) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_node_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0,

[JC1]

Stephen, on the client:

rpm -q --whatprovides /etc/rhsm/ca/katello-server-ca.pem

katello-ca-consumer-katello.server-1.0-2.noarch

Which is the RPM I've installed from
https://katello/pub/katello-ca-consumer-katello.server-1.0-2.noarch.rpm.

There isn't a katello-default-ca.pem.

···

On Thursday, 20 August 2015 13:25:04 UTC+1, stephen wrote: > > On Thu, Aug 20, 2015 at 12:36:37AM -0700, JC wrote: > > Morning Stephen, > > > > Yes, they match. I have checked the date/time stamps on > > > https://katello.server/pub/katello-ca-consumer-katello.server-1.0-2.noarch.rpm > > with that on the client (using rpm to review the build date). > > > > Installed on the client, it provides: > > > > /etc/rhsm/ca/katello-server-ca.pem > > It doesn't provide katello-default-ca.pem? > > That is the one from the self-signed cert and the one that katello-agent > needs to use to connect to qpid. > > > > > Examining this cert shows it to be issued by our internal CA. > > > > On Wednesday, 19 August 2015 17:43:42 UTC+1, stephen wrote: > > > > > > On Wed, Aug 19, 2015 at 06:33:59AM -0700, JC wrote: > > > > Correction, the issuer is katello.server (with some RH text to fill > in > > > the > > > > cert), and it is for my client! > > > > > > > > Certificate: > > > > > > > > > > > > > > > > * Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, > > > > OU=SomeOrgUnit, CN=katello.server* > > > > > > > > Validity > > > > > > > > Not Before: Aug 18 14:25:23 2015 GMT > > > > > > > > Not After : Aug 18 14:25:23 2031 GMT > > > > > > > > > > > > > > > > Exponent: 65257 (0x10001) > > > > > > > > X509v3 extensions: > > > > > > > > Netscape Cert Type: > > > > > > > > SSL Client, S/MIME > > > > > > > > X509v3 Key Usage: > > > > > > > > Digital Signature, Key Encipherment, Data > Encipherment > > > > > > > > X509v3 Authority Key Identifier: > > > > > > > > keyid:EA:37:47:…..2E:71:EC:91 > > > > > > > > * DirName:/C=US/ST=North > > > > Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server* > > > > > > > > serial:A7:A2:3C:……….:65:B1 > > > > > > > > X509v3 Subject Key Identifier: > > > > > > > > FB:A7:9C:E6:7……………………………..0E:78:B8:C8 > > > > > > > > X509v3 Extended Key Usage: > > > > > > > > TLS Web Client Authentication > > > > > > > > * X509v3 Subject Alternative Name:* > > > > > > > > * URI:CN=katello.client* > > > > > > > > Signature Algorithm: sha1WithRSAEncryption > > > > > > > > 7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07: > > > > > > > > > > > > > > > > Now, from the client: > > > > > > > > *# openssl s_client -connect katello.server:5647 -showcerts ## same > for > > > > port 5446* > > > > CONNECTED(00000003) > > > > depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = > > > > SomeOrgUnit, CN = katello.server > > > > verify error:num=19:self signed certificate in certificate chain > > > > verify return:0 > > > > 140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 > alert > > > > handshake failure:s3_pkt.c:1259:SSL alert number 40 > > > > 140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl > handshake > > > > failure:s23_lib.c:184: > > > > > > > > > That's expected, we don't use the UI cert for qpid. > > > > > > You have the latest katello-ca-consumer-latest from the Katello server > > > installed on the client? (it's in /pub on the web server) > > > > > > > > > > > > > > > > > > > On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote: > > > > > > > > > > Sorry for all the postings, but perhaps I'm getting somewhere... > > > > > > > > > > *# grep "cert=" /etc/gofer/plugins/katelloplugin.conf* > > > > > cacert=/etc/rhsm/ca/candlepin-local.pem > > > > > clientcert=/etc/pki/consumer/bundle.pem > > > > > *# file /etc/rhsm/ca/candlepin-local.pem* > > > > > /etc/rhsm/ca/candlepin-local.pem: cannot open > > > > > `/etc/rhsm/ca/candlepin-local.pem' (No such file or directory) > > > > > *# file /etc/pki/consumer/bundle.pem* > > > > > /etc/pki/consumer/bundle.pem: ASCII text > > > > > > > > > > > > > > > *$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text* > > > > > # shows a cert for katello.server signed by RH, rather than my CA. > > > > > > > > > > Is that relevant? Can you tell that I don't really understand how > the > > > > > certs are used? ;-) > > > > > > > > > > On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote: > > > > > > > > > > Ahhh, right, on the client (/var/log/messages) I see: > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - > > > amqps://katello.server:5647 > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - Traceback (most > recent > > > call > > > > > last): > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > > > > > > line 101, in open > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - domain = > > > > > self.ssl_domain(connector) > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > > > > > "/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py", > > > > > > > > > line 57, in ssl_domain > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - > > > > > domain.set_trusted_ca_db(connector.ssl.ca_certificate) > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line > 3386, in > > > > > set_trusted_ca_db > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - > certificate_db) ) > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - File > > > > > "/usr/lib64/python2.6/site-packages/proton/__init__.py", line > 3376, in > > > > > _check > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - raise exc("SSL > > > > > failure.") > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:106 - SSLException: *SSL > > > > > failure.* > > > > > > > > > > Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:108 - retry in 106 > seconds > > > > > > > > > > Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1] > > > > > gofer.messaging.adapter.proton.connection:100 - connecting: URL: > > > > > amqps://katello.server:5647|SSL: ca: > > > > > /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: > > > > > /etc/pki/consumer/bundle.pem|host-validation: None > > > > > > > > > > And on the server I see (ignore the times, grabbed from later in > > > > > /var/log/qdrouterd.log): > > > > > > > > > > Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming > connection > > > from > > > > > katello.client:44791 to 0.0.0.0:5647 > > > > > > > > > > I've previously updated the Foreman UI cert using: > > > > > > > > > > $ katello-installer --certs-server-cert "katello.cer" > > > > > --certs-server-cert-req "katello.csr" --certs-server-key > "katello.key" > > > > > --certs-server-ca-cert "RootAll.cer" --certs-update-server > > > > > --certs-update-server-ca > > > > > > > > > > Did I miss something then? The UI looks happy... > > > > > > > > > > OK, anyone know the solution? > > > > > > > > > > On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote: > > > > > > > > > > OK, strace showed a problem writing to the log file. Perms > changed > > > and > > > > > now we have some output... > > > > > > > > > > # cat /var/log/qdrouterd.log > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER_LS) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=ROUTER_MA) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/DISPATCH, identity=log/DISPATCH, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=DISPATCH) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > > > module=ROUTER_HELLO) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/SERVER, > > > > > identity=log/SERVER, type=org.apache.qpid.dispatch.log, > > > enable=default, > > > > > module=SERVER) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/CONTAINER, identity=log/CONTAINER, > > > > > type=org.apache.qpid.dispatch.log, enable=default, > module=CONTAINER) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/AGENT, > > > > > identity=log/AGENT, type=org.apache.qpid.dispatch.log, > enable=default, > > > > > module=AGENT) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ERROR, > > > > > identity=log/ERROR, type=org.apache.qpid.dispatch.log, > enable=default, > > > > > module=ERROR) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/ROUTER, > > > > > identity=log/ROUTER, type=org.apache.qpid.dispatch.log, > > > enable=default, > > > > > module=ROUTER) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=log/MESSAGE, identity=log/MESSAGE, > > > > > type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > Entity(name=log/CONFIG, > > > > > identity=log/CONFIG, type=org.apache.qpid.dispatch.log, > > > enable=default, > > > > > module=CONFIG) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=container/katello.server, > > > identity=container/katello.server, > > > > > type=org.apache.qpid.dispatch.container, > containerName=katello.server, > > > > > workerThreads=2) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4, > > > > > helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60, > > > > > routerId=katello.server, raInterval=30, mode=interior, > > > mobileAddrMaxAge=60) > > > > > > > > > > Wed Aug 19 12:32:43 2015 SERVER (info) Container Name: > katello.server > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior > > > mode, > > > > > area=0 id=katello.server > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16, > > > > > typeSize=2104, transferBatchSize=16, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_log_entry_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=16, > > > > > localFreeListMax=32) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=88, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=32, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_hash_item_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_node_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=136, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_bitmask_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=56, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_timer_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=216, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_address_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64, > > > > > typeSize=16, transferBatchSize=64, globalFreeListMax=0, > > > > > batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t, > > > > > batchesRebalancedToThreads=0, totalFreeToHeap=0, > > > totalAllocFromHeap=64, > > > > > localFreeListMax=128) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.router.address, > > > subscriberCount=0, > > > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > > > inProcess=False, > > > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter, > > > > > deliveriesToContainer=0) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.router.address, > > > subscriberCount=0, > > > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > > > inProcess=False, > > > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdrouter.ma, > > > > > deliveriesToContainer=0) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(type=org.apache.qpid.dispatch.router.address, > > > subscriberCount=0, > > > > > deliveriesEgress=0, deliveriesIngress=0, remoteCount=0, > > > inProcess=False, > > > > > deliveriesFromContainer=0, deliveriesTransit=0, key=Lqdhello, > > > > > deliveriesToContainer=0) > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (info) Activating management agent > on > > > > > $management > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address > Registered: > > > > > $management > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address > Registered: > > > > > $management > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address > Registered: > > > > > qdrouter > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address > Registered: > > > > > qdrouter.ma > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) In-Process Address > Registered: > > > > > qdhello > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTER (info) Router Engine Instantiated: > > > > > id=katello.server instance=1439983963 max_routers=1024 > > > > > > > > > > Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity: > > > > > Entity(name=fixedAddress/0, identity=fixedAddress/0, > > > > > type=org.apache.qpid.dispatch.fixedAddress, fanout=single, > > > prefix=/closest, > > > > > bias=closest) > > > > > > > > > > Wed Aug 19 12:32:43 2015 ROUTE

[JC1]

Cheers Eric. That does sound like it might be it. I tried 2.3 earlier,
but found it too problematic (so I blew away the install and went back to
2.2). As my install is looking pretty sweet now, I'm apprehensive about
the upgrade, but of course I shouldn't be!!

Does the katello-installer --upgrade preserve my certs, for example?
Similarly, where I've changed other files, do these get saved? Perhaps in
some rpm -U fashion? I've upgraded before, but wasn't so bothered about
what I've already done.

Stephen - I wanted to get back to you earlier, but couldn't login to our
virtual environment to validate my early post.

···

On 20 August 2015 at 20:21, Eric D Helms wrote:

JC,

Trying to read back, I think I understand that you are using custom
certificates and hitting a katello-agent issue. A user on IRC, tamarin,
made me realize in discussion today that I never backported to 2.2 a fix
for custom certificates and katello-agent whereby we aren’t laying down the
katello-default-ca.crt (this is fixed in 2.3). You can see here –
https://github.com/Katello/puppet-certs/blob/master/manifests/katello.pp#L42
this exact comment which sounds like your issue. If you need this fixed in
2.2 vs just upgrading to 2.3 (and all the excellent work and bugs fixes
that went into it) please let me know.

Eric

On Thu, Aug 20, 2015 at 8:25 AM, Stephen Benjamin stephen@redhat.com > wrote:

On Thu, Aug 20, 2015 at 12:36:37AM -0700, JC wrote:

Morning Stephen,

Yes, they match. I have checked the date/time stamps on

https://katello.server/pub/katello-ca-consumer-katello.server-1.0-2.noarch.rpm

with that on the client (using rpm to review the build date).

Installed on the client, it provides:

/etc/rhsm/ca/katello-server-ca.pem

It doesn’t provide katello-default-ca.pem?

That is the one from the self-signed cert and the one that katello-agent
needs to use to connect to qpid.

Examining this cert shows it to be issued by our internal CA.

On Wednesday, 19 August 2015 17:43:42 UTC+1, stephen wrote:

On Wed, Aug 19, 2015 at 06:33:59AM -0700, JC wrote:

Correction, the issuer is katello.server (with some RH text to fill
in

the

cert), and it is for my client!

Certificate:

•    Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello,
  

OU=SomeOrgUnit, CN=katello.server*

    Validity

        Not Before: Aug 18 14:25:23 2015 GMT

        Not After : Aug 18 14:25:23 2031 GMT

            Exponent: 65257 (0x10001)

    X509v3 extensions:

        Netscape Cert Type:

            SSL Client, S/MIME

        X509v3 Key Usage:

            Digital Signature, Key Encipherment, Data

Encipherment

        X509v3 Authority Key Identifier:

            keyid:EA:37:47:…..2E:71:EC:91

•            DirName:/C=US/ST=North
  

Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=katello.server*

            serial:A7:A2:3C:……….:65:B1

        X509v3 Subject Key Identifier:

            FB:A7:9C:E6:7……………………………..0E:78:B8:C8

        X509v3 Extended Key Usage:

            TLS Web Client Authentication

•        X509v3 Subject Alternative Name:*
  

•            URI:CN=katello.client*
  

  Signature Algorithm: sha1WithRSAEncryption

     7d:80:dd:6f:d5:b……………….2d:a8:a2:86:07:
  
     <snip>
  

Now, from the client:

*# openssl s_client -connect katello.server:5647 -showcerts ##
same for

port 5446*
CONNECTED(00000003)
depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU =
SomeOrgUnit, CN = katello.server
verify error:num=19:self signed certificate in certificate chain
verify return:0
140220054890312:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3
alert

handshake failure:s3_pkt.c:1259:SSL alert number 40
140220054890312:error:140790E5:SSL routines:SSL23_WRITE:ssl
handshake

failure:s23_lib.c:184:

That’s expected, we don’t use the UI cert for qpid.

You have the latest katello-ca-consumer-latest from the Katello server
installed on the client? (it’s in /pub on the web server)

On Wednesday, 19 August 2015 13:49:54 UTC+1, JC wrote:

Sorry for all the postings, but perhaps I’m getting somewhere…

# grep “cert=” /etc/gofer/plugins/katelloplugin.conf
cacert=/etc/rhsm/ca/candlepin-local.pem
clientcert=/etc/pki/consumer/bundle.pem
# file /etc/rhsm/ca/candlepin-local.pem
/etc/rhsm/ca/candlepin-local.pem: cannot open
`/etc/rhsm/ca/candlepin-local.pem’ (No such file or directory)
# file /etc/pki/consumer/bundle.pem
/etc/pki/consumer/bundle.pem: ASCII text

$ openssl x509 -in /etc/pki/consumer/bundle.pem -noout -text

shows a cert for katello.server signed by RH, rather than my CA.

Is that relevant? Can you tell that I don’t really understand
how the

certs are used?

On Wednesday, 19 August 2015 13:35:52 UTC+1, JC wrote:

Ahhh, right, on the client (/var/log/messages) I see:

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
amqps://katello.server:5647

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - Traceback (most
recent

call

last):

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 101, in open

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - domain =
self.ssl_domain(connector)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File

“/usr/lib/python2.6/site-packages/gofer/messaging/adapter/proton/connection.py”,

line 57, in ssl_domain

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
domain.set_trusted_ca_db(connector.ssl.ca_certificate)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line
3386, in

set_trusted_ca_db

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 -
certificate_db) )

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - File
"/usr/lib64/python2.6/site-packages/proton/init.py", line
3376, in

_check

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - raise exc(“SSL
failure.”)

Aug 19 13:18:02 ptnoc61 goferd: [ERROR][Thread-1]
gofer.messaging.adapter.proton.connection:106 - SSLException: SSL
failure.

Aug 19 13:18:02 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:108 - retry in 106
seconds

Aug 19 13:19:49 ptnoc61 goferd: [INFO][Thread-1]
gofer.messaging.adapter.proton.connection:100 - connecting: URL:
amqps://katello.server:5647|SSL: ca:
/etc/rhsm/ca/katello-server-ca.pem|key: None|certificate:
/etc/pki/consumer/bundle.pem|host-validation: None

And on the server I see (ignore the times, grabbed from later in
/var/log/qdrouterd.log):

Wed Aug 19 13:32:23 2015 SERVER (debug) Accepting incoming
connection

from

katello.client:44791 to 0.0.0.0:5647

I’ve previously updated the Foreman UI cert using:

$ katello-installer --certs-server-cert “katello.cer”
–certs-server-cert-req “katello.csr” --certs-server-key
"katello.key"

–certs-server-ca-cert “RootAll.cer” --certs-update-server
–certs-update-server-ca

Did I miss something then? The UI looks happy…

OK, anyone know the solution?

On Wednesday, 19 August 2015 12:57:20 UTC+1, JC wrote:

OK, strace showed a problem writing to the log file. Perms
changed

and

now we have some output…

cat /var/log/qdrouterd.log

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_LS, identity=log/ROUTER_LS,
type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER_LS)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_MA, identity=log/ROUTER_MA,
type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER_MA)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/DISPATCH, identity=log/DISPATCH,
type=org.apache.qpid.dispatch.log, enable=default,
module=DISPATCH)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER_HELLO, identity=log/ROUTER_HELLO,
type=org.apache.qpid.dispatch.log, enable=default,
module=ROUTER_HELLO)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/SERVER,

identity=log/SERVER, type=org.apache.qpid.dispatch.log,
enable=default,

module=SERVER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONTAINER, identity=log/CONTAINER,
type=org.apache.qpid.dispatch.log, enable=default,
module=CONTAINER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/AGENT,

identity=log/AGENT, type=org.apache.qpid.dispatch.log,
enable=default,

module=AGENT)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ERROR,

identity=log/ERROR, type=org.apache.qpid.dispatch.log,
enable=default,

module=ERROR)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/ROUTER,

identity=log/ROUTER, type=org.apache.qpid.dispatch.log,
enable=default,

module=ROUTER)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/MESSAGE, identity=log/MESSAGE,
type=org.apache.qpid.dispatch.log, enable=default, module=MESSAGE)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=log/CONFIG,

identity=log/CONFIG, type=org.apache.qpid.dispatch.log,
enable=default,

module=CONFIG)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(name=container/katello.server,
identity=container/katello.server,

type=org.apache.qpid.dispatch.container,
containerName=katello.server,

workerThreads=2)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.router, raIntervalFlux=4,
helloInterval=1, area=0, helloMaxAge=3, remoteLsMaxAge=60,
routerId=katello.server, raInterval=30, mode=interior,
mobileAddrMaxAge=60)

Wed Aug 19 12:32:43 2015 SERVER (info) Container Name:
katello.server

Wed Aug 19 12:32:43 2015 ROUTER (info) Router started in Interior
mode,

area=0 id=katello.server

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=16,
typeSize=2104, transferBatchSize=16, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_log_entry_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=16,

localFreeListMax=32)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=88, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_field_iterator_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=32, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_item_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_node_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=136, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_bitmask_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=56, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_timer_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=216, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_address_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,
totalAllocFromHeap=64,

localFreeListMax=128)

Wed Aug 19 12:32:43 2015 AGENT (debug) Add entity:
Entity(type=org.apache.qpid.dispatch.allocator, heldByThreads=64,
typeSize=16, transferBatchSize=64, globalFreeListMax=0,
batchesRebalancedToGlobal=0, typeName=qd_hash_handle_t,
batchesRebalancedToThreads=0, totalFreeToHeap=0,