Qdrouterd Utilizing More Memory Overtime

Support
1

Problem:
I’ve recently noticed that the qrouterd process on my Foreman server appears to be consuming more and more memory overtime.

Expected outcome:
That the qrouterd process maintains the around same amount of system memory usage overtime.

Foreman and Proxy versions: 2.2.3

Foreman and Proxy plugin versions: Katello 3.17.3 - needs updating!

Distribution and version: CentOS 7

Other relevant data:
Below is a link to an image of an “Available memory” graph taken from my Zabbix monitoring server, illustrating how the available memory usage on the systems has been steadily dropping.

Notice that available memory was alarmingly low in the beginning of the graph, however after restarting all Foreman related services (using foreman-maintain service restart), memory usage returned back to normal, but it appears to be continuing to drop. At that time, the qrouterd service was the process being the memory hog.

Here are the top 10 processes on the system at the time of this writing, sorted by memory usage:

# ps aux --sort -pmem | head -n 10
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
qdroute+ 29844  1.1 16.1 11078036 10635296 ?   Ssl   2021 115:26 /usr/sbin/qdrouterd -c /etc/qpid-dispatch/qdrouterd.conf
root      4727  0.4 10.8 9429308 7124328 ?     Sl    2021 199:29 /var/IBM/ITM/lx8266/lz/bin/klzagent
mongodb  29783  0.4  9.2 7136312 6060892 ?     Sl    2021  44:54 /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run
puppet   30469  3.7  2.7 6058884 1838204 ?     Sl    2021 387:25 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -XX:OnOutOfMemoryError=kill -9 %p -XX:ErrorFile=/var/log/puppetlabs/puppetserver/puppetserver_err_pid%p.log -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter
tomcat   30285  1.1  2.0 8134484 1339672 ?     Ssl   2021 118:31 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
foreman  31025  1.1  1.3 1316044 913624 ?      Sl    2021 122:37 puma: cluster worker 0: 30291 [foreman]
foreman  31028  1.0  1.3 1274080 887740 ?      Sl    2021 106:18 puma: cluster worker 1: 30291 [foreman]
foreman  30975  0.1  1.2 1181876 815276 ?      Ssl   2021  13:44 sidekiq 5.2.7  [0 of 5 busy]
qpidd    29847  0.1  1.0 1037476 722992 ?      Ssl   2021  14:08 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

Memory usage of the qdrouterd process is now at 16.1%, however it was at 2.7% on 12/27/2021 after the service restart.

Here is my memory usage summary at this current moment:

# free
              total        used        free      shared  buff/cache   available
Mem:       65808684    36419088    13389048     1966852    16000548    26890864
Swap:       2097148      332220     1764928

I honestly don’t know much about qdrouterd, but I’ll try to share some information about its state on my system.

Here is an example of its open files:

qdrouterd 29844 qdrouterd *685u     IPv4          239610516       0t0       TCP chluxforeman01p:5647->10.52.41.175:59780 (ESTABLISHED)
qdrouterd 29844 qdrouterd *686u     IPv4          239612198       0t0       TCP chluxforeman01p:5647->10.52.41.175:59786 (ESTABLISHED)
qdrouterd 29844 qdrouterd *687u     IPv4          239613711       0t0       TCP chluxforeman01p:5647->10.52.169.24:38442 (ESTABLISHED)
qdrouterd 29844 qdrouterd *688u     IPv4          239614649       0t0       TCP chluxforeman01p:5647->10.52.169.24:38444 (ESTABLISHED)
qdrouterd 29844 qdrouterd *689u     IPv4          239614671       0t0       TCP chluxforeman01p:5647->10.52.169.21:58244 (ESTABLISHED)
qdrouterd 29844 qdrouterd *690u     IPv4          239613740       0t0       TCP chluxforeman01p:5647->10.52.169.21:58246 (ESTABLISHED)
qdrouterd 29844 qdrouterd *691u     IPv4          239613887       0t0       TCP chluxforeman01p:5647->10.43.181.23:48614 (ESTABLISHED)
qdrouterd 29844 qdrouterd *692u     IPv4          239615650       0t0       TCP chluxforeman01p:5647->10.41.172.23:47430 (ESTABLISHED)
qdrouterd 29844 qdrouterd *693u     IPv4          239617291       0t0       TCP chluxforeman01p:5647->10.41.172.23:47432 (ESTABLISHED)
qdrouterd 29844 qdrouterd *694u     IPv4          239624274       0t0       TCP chluxforeman01p:5647->10.36.153.25:59414 (ESTABLISHED)
qdrouterd 29844 qdrouterd *695u     IPv4          239624397       0t0       TCP chluxforeman01p:5647->10.43.181.24:38264 (ESTABLISHED)
qdrouterd 29844 qdrouterd *696u     IPv4          239627538       0t0       TCP chluxforeman01p:5647->10.43.181.21:54388 (ESTABLISHED)
qdrouterd 29844 qdrouterd *697u     IPv4          239624960       0t0       TCP chluxforeman01p:5647->10.43.181.23:48630 (ESTABLISHED)
qdrouterd 29844 qdrouterd *698u     IPv4          239631376       0t0       TCP chluxforeman01p:5647->10.52.169.22:45832 (ESTABLISHED)
qdrouterd 29844 qdrouterd *699u     IPv4          239630992       0t0       TCP chluxforeman01p:5647->10.52.169.22:45834 (ESTABLISHED)
qdrouterd 29844 qdrouterd *700u     IPv4          239631269       0t0       TCP chluxforeman01p:5647->10.36.153.25:59426 (ESTABLISHED)
qdrouterd 29844 qdrouterd *701u     IPv4          239633860       0t0       TCP chluxforeman01p:5647->10.43.181.24:38276 (ESTABLISHED)
qdrouterd 29844 qdrouterd *702u     IPv4          239636729       0t0       TCP chluxforeman01p:5647->10.43.181.21:54406 (ESTABLISHED)
qdrouterd 29844 qdrouterd *703u     IPv4          239637289       0t0       TCP chluxforeman01p:5647->10.43.181.23:48642 (ESTABLISHED)
qdrouterd 29844 qdrouterd *704u     IPv4          239640731       0t0       TCP chluxforeman01p:5647->10.36.153.25:59438 (ESTABLISHED)
qdrouterd 29844 qdrouterd *705u     IPv4          239639869       0t0       TCP chluxforeman01p:5647->10.43.181.24:38288 (ESTABLISHED)
qdrouterd 29844 qdrouterd *706u     IPv4          239641080       0t0       TCP chluxforeman01p:5647->10.43.181.21:54420 (ESTABLISHED)

I didn’t paste all the output because it is 11759 lines of these established connections, which to my untrained eye, seems to be a lot as I only have around 900 systems being managed.

Here are logs taken from journald showing some recent logs. Note sure if at all useful:

Jan 03 13:07:55 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44097] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:07:55 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24559][L44096] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:07:55 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44095] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:07:55 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24559][L44094] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:07:55 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24560] Accepted connection to :5647 from 10.41.172.23:47432
Jan 03 13:07:56 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24560] Connection Opened: dir=in host=10.41.172.23:47432 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=5e8afff0-309e-4462-87ed-a1c53933d31a props=
Jan 03 13:07:56 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24560][L44098] Link attached: dir=out source={pulp.agent.c1ac4915-4808-4aa2-a0b5-afb3a32dbe86 expire:sess} target={<none> expire:sess}
Jan 03 13:08:14 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24561] Accepted connection to :5647 from 10.36.153.25:59414
Jan 03 13:08:35 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24562] Accepted connection to :5647 from 10.43.181.24:38264
Jan 03 13:08:39 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24563] Accepted connection to :5647 from 10.43.181.21:54388
Jan 03 13:09:32 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24564] Accepted connection to :5647 from 10.43.181.23:48630
Jan 03 13:09:47 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24565] Accepted connection to :5647 from 10.52.169.22:45832
Jan 03 13:09:47 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24565] Connection Opened: dir=in host=10.52.169.22:45832 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=d5622947-e367-42d7-a4cd-c906b9484662 props=
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24565][L44100] Link attached: dir=out source={(dyn)<none> expire:sess props:{:"x-opt-qd.address"="qmf.default.direct"}} target={<none> expire:sess}
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24565][L44102] Link attached: dir=in source={<none> expire:sess} target={qmf.default.direct expire:sess}
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44103] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24565][L44102] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44101] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24565][L44100] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24566] Accepted connection to :5647 from 10.52.169.22:45834
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24566] Connection Opened: dir=in host=10.52.169.22:45834 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=5f724874-b787-4f3d-8082-4aeda6250250 props=
Jan 03 13:09:48 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24566][L44104] Link attached: dir=out source={pulp.agent.fb8abaf6-675b-46e6-8353-08d03bcce31e expire:sess} target={<none> expire:sess}
Jan 03 13:10:12 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24567] Accepted connection to :5647 from 10.36.153.25:59426
Jan 03 13:10:32 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24568] Accepted connection to :5647 from 10.43.181.24:38276
Jan 03 13:10:37 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24569] Accepted connection to :5647 from 10.43.181.21:54406
Jan 03 13:11:30 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24570] Accepted connection to :5647 from 10.43.181.23:48642
Jan 03 13:12:09 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24571] Accepted connection to :5647 from 10.36.153.25:59438
Jan 03 13:12:29 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24572] Accepted connection to :5647 from 10.43.181.24:38288
Jan 03 13:12:34 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24573] Accepted connection to :5647 from 10.43.181.21:54420
Jan 03 13:13:27 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24574] Accepted connection to :5647 from 10.43.181.23:48660
Jan 03 13:14:06 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24575] Accepted connection to :5647 from 10.36.153.25:59460
Jan 03 13:14:19 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24368] Connection from 10.36.186.22:44990 (to :5647) failed: amqp:connection:framing-error SSL Failure: Unknown error
Jan 03 13:14:19 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24368][L43970] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:14:19 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24368] Connection Closed
Jan 03 13:14:19 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L43971] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:14:27 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24576] Accepted connection to :5647 from 10.43.181.24:38312
Jan 03 13:14:31 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24577] Accepted connection to :5647 from 10.43.181.21:54438
Jan 03 13:15:24 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24578] Accepted connection to :5647 from 10.43.181.23:48672
Jan 03 13:16:04 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24579] Accepted connection to :5647 from 10.36.153.25:59472
Jan 03 13:16:24 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24580] Accepted connection to :5647 from 10.43.181.24:38324
Jan 03 13:16:29 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24581] Accepted connection to :5647 from 10.43.181.21:54450
Jan 03 13:17:05 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23232] Connection from 10.38.170.21:58748 (to :5647) failed: amqp:connection:framing-error SSL Failure: Unknown error
Jan 03 13:17:05 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23232][L41848] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:05 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23232] Connection Closed
Jan 03 13:17:05 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L41849] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:06 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23344] Connection from 10.38.170.22:55636 (to :5647) failed: amqp:connection:framing-error SSL Failure: Unknown error
Jan 03 13:17:06 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23344][L42178] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:06 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23344] Connection Closed
Jan 03 13:17:06 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L42179] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:07 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23554] Connection from 10.38.170.24:49750 (to :5647) failed: amqp:connection:framing-error SSL Failure: Unknown error
Jan 03 13:17:07 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23554][L42802] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:07 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23554] Connection Closed
Jan 03 13:17:07 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L42803] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:09 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23454] Connection from 10.38.170.23:49620 (to :5647) failed: amqp:connection:framing-error SSL Failure: Unknown error
Jan 03 13:17:09 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23454][L42502] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:09 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23454] Connection Closed
Jan 03 13:17:09 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L42503] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:10 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24582] Accepted connection to :5647 from 10.38.170.24:53476
Jan 03 13:17:10 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24582] Connection Opened: dir=in host=10.38.170.24:53476 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=757ae00f-20b4-45ca-8b98-c6eda68b087d props=
Jan 03 13:17:11 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24582][L44106] Link attached: dir=out source={pulp.agent.d33d59f7-f4fb-4073-a521-e3043336c2b9 expire:sess} target={<none> expire:sess}
Jan 03 13:17:13 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24583] Accepted connection to :5647 from 10.38.170.21:34226
Jan 03 13:17:13 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24583] Connection Opened: dir=in host=10.38.170.21:34226 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=101179cb-7da6-4aeb-8769-7575140bae5c props=
Jan 03 13:17:13 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24583][L44108] Link attached: dir=out source={pulp.agent.2fd6a689-ca0c-467f-a9c3-21229c591b55 expire:sess} target={<none> expire:sess}
Jan 03 13:17:15 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24584] Accepted connection to :5647 from 10.38.170.23:53434
Jan 03 13:17:16 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24584] Connection Opened: dir=in host=10.38.170.23:53434 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=c1d84835-e5e4-4c9a-82d0-56a076d13328 props=
Jan 03 13:17:16 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24585] Accepted connection to :5647 from 10.38.170.22:59418
Jan 03 13:17:16 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24584][L44110] Link attached: dir=out source={pulp.agent.8f228331-6a4f-4b33-b3d3-12894e9fdeef expire:sess} target={<none> expire:sess}
Jan 03 13:17:16 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24585] Connection Opened: dir=in host=10.38.170.22:59418 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=ba5101cb-4373-405d-a781-4de1c067db2e props=
Jan 03 13:17:16 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24585][L44112] Link attached: dir=out source={pulp.agent.0496536a-b669-47b9-bae0-3c14300c6921 expire:sess} target={<none> expire:sess}
Jan 03 13:17:22 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24586] Accepted connection to :5647 from 10.43.181.23:48682
Jan 03 13:17:29 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23721] Connection from 10.46.160.21:37872 (to :5647) failed: amqp:connection:framing-error SSL Failure: Unknown error
Jan 03 13:17:29 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23721][L43300] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:17:29 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C23721] Connection Closed
Jan 03 13:17:29 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L43301] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:18:01 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24587] Accepted connection to :5647 from 10.36.153.25:59492
Jan 03 13:18:22 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24588] Accepted connection to :5647 from 10.43.181.24:38336
Jan 03 13:18:26 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24589] Accepted connection to :5647 from 10.43.181.21:54470
Jan 03 13:18:49 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24590] Accepted connection to :5647 from 10.36.186.22:49238
Jan 03 13:18:49 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24590] Connection Opened: dir=in host=10.36.186.22:49238 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=df0fbe72-7ae1-4758-986c-6df5d2a11613 props=
Jan 03 13:18:49 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24590][L44114] Link attached: dir=out source={(dyn)<none> expire:sess props:{:"x-opt-qd.address"="qmf.default.direct"}} target={<none> expire:sess}
Jan 03 13:18:50 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24590][L44116] Link attached: dir=in source={<none> expire:sess} target={qmf.default.direct expire:sess}
Jan 03 13:18:50 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44117] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:18:50 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24590][L44116] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:18:50 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44115] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:18:50 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24590][L44114] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:18:50 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24591] Accepted connection to :5647 from 10.36.186.22:49240
Jan 03 13:18:51 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24591] Connection Opened: dir=in host=10.36.186.22:49240 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=3846004c-2414-42b3-b844-49f6aaebb17c props=
Jan 03 13:18:51 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24591][L44118] Link attached: dir=out source={pulp.agent.b064a8ba-ad93-4102-8289-3dac33629668 expire:sess} target={<none> expire:sess}
Jan 03 13:19:19 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24592] Accepted connection to :5647 from 10.43.181.23:48722
Jan 03 13:19:58 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24593] Accepted connection to :5647 from 10.36.153.25:59506
Jan 03 13:20:19 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24594] Accepted connection to :5647 from 10.43.181.24:38352
Jan 03 13:20:24 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24595] Accepted connection to :5647 from 10.43.181.21:54486
Jan 03 13:20:30 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24596] Accepted connection to :5647 from 10.46.160.21:36604
Jan 03 13:20:30 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24596] Connection Opened: dir=in host=10.46.160.21:36604 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=1617b952-048a-495e-a5e1-2c7b065795d4 props=
Jan 03 13:20:30 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24596][L44120] Link attached: dir=out source={(dyn)<none> expire:sess props:{:"x-opt-qd.address"="qmf.default.direct"}} target={<none> expire:sess}
Jan 03 13:20:30 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24596][L44122] Link attached: dir=in source={<none> expire:sess} target={qmf.default.direct expire:sess}
Jan 03 13:20:31 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44123] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:20:31 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24596][L44122] Link detached: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:20:31 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C4][L44121] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:20:31 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24596][L44120] Link detached: del=1 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Jan 03 13:20:31 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24597] Accepted connection to :5647 from 10.46.160.21:36606
Jan 03 13:20:32 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24597] Connection Opened: dir=in host=10.46.160.21:36606 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=43283639-37c3-4ea6-a39d-30bd8c5b8f2e props=
Jan 03 13:20:32 chluxforeman01p.na.odcorp.net qdrouterd[29844]: ROUTER_CORE (info) [C24597][L44124] Link attached: dir=out source={pulp.agent.fa7e14de-1138-4869-bd02-2b81701d717f expire:sess} target={<none> expire:sess}
Jan 03 13:21:17 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24598] Accepted connection to :5647 from 10.43.181.23:48734
Jan 03 13:21:56 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24599] Accepted connection to :5647 from 10.36.153.25:59514
Jan 03 13:22:16 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24600] Accepted connection to :5647 from 10.43.181.24:38364
Jan 03 13:22:21 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24601] Accepted connection to :5647 from 10.43.181.21:54502
Jan 03 13:23:14 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24602] Accepted connection to :5647 from 10.43.181.23:48750
Jan 03 13:23:53 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24603] Accepted connection to :5647 from 10.36.153.25:59532
Jan 03 13:24:14 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24604] Accepted connection to :5647 from 10.43.181.24:38404
Jan 03 13:24:19 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24605] Accepted connection to :5647 from 10.43.181.21:54530
Jan 03 13:25:12 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24606] Accepted connection to :5647 from 10.43.181.23:48766
Jan 03 13:25:50 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C24607] Accepted connection to :5647 from 10.36.153.25:59542

I also see a lot of this logged:

Jan 03 04:15:08 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23216] Connection from 10.28.189.23:52348 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 04:15:08 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23198] Connection from 10.28.189.23:52174 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 04:15:08 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23202] Connection from 10.28.189.23:52206 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 04:15:08 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23225] Connection from 10.28.189.23:52480 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 04:15:08 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23219] Connection from 10.28.189.23:52388 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 04:15:08 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23336] Connection from 10.28.189.23:52518 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 04:15:08 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23353] Connection from 10.28.189.23:52544 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 07:58:15 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C22966] Connection from 10.42.157.23:47598 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672 (SSL Failure: Unknown error)
Jan 03 08:34:57 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C23129] Connection from 10.30.147.25:60992 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672 (SSL Failure: Unknown error)
Jan 03 10:48:39 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C20211] Connection from 10.43.181.22:48836 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 13:00:22 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C20719] Connection from 10.43.181.22:49760 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672
Jan 03 13:03:27 chluxforeman01p.na.odcorp.net qdrouterd[29844]: SERVER (info) [C20737] Connection from 10.43.181.22:49788 (to :5647) failed: proton:io Connection reset by peer - disconnected :5672

I would be so very grateful for any advice on how to approach this. Please let me know of any other information I can provide which would be helpful. Perhaps a good first step would be getting Katello and Foreman up-to-date?

Thanks!

2

@fresh-pie out of curiosity, which version of qpid-dispatch is installed on your server? I’m not aware of any issues around memory usage but maybe upgrading qpid-dispatch would resolve it.

Also, starting with Katello 4.1 we don’t ship qpid or qdrouterd by default in new installations and unless you are using katello-agent in your infrastructure it’s not required to re-enable them in 4.1 and newer. So, if you’re not using katello agent I high recommend upgrading to 4.1+ as soon as possible.

1 Like
3

Hey Jonathon!

Looks like qpid-dispatch-router-1.17.0-1 is the version that I’m running on my Katello server.

# rpm -qa | grep qpid-dispatch
qpid-dispatch-router-1.17.0-1.el7.x86_64

# yum info qpid-dispatch-router
Installed Packages
Name        : qpid-dispatch-router
Arch        : x86_64
Version     : 1.17.0
Release     : 1.el7
Size        : 3.3 M
Repo        : installed
From repo   : epel
Summary     : The Qpid Dispatch Router executable
URL         : http://qpid.apache.org/
License     : ASL 2.0
Description : The Qpid Dispatch Router executable.

Regarding the use of katello-agent, it is installed on my end-points, but I don’t believe I’m actually using it at all. After a quick review, I think it is getting installed per the redhat_register provisioning template in Foreman.

Please correct me if I am wrong, but my understanding is that katello-agent used to be the client which enabled one to deploy content from Katello to the end-point, correct? If that is so, I don’t believe I need it. In my environment, I of course host content via Katello and my end-points are subscribed, however I use Ansible to install packages from the Katello hosted repositories on my clients via a yum upgrade.

It’s looking more and more like I should just get Katello up to version 4 as soon as possible!

Thanks Jonathon for your insight :slight_smile:

4

Thanks for the info!

Your understanding of what katello-agent does is correct. Today you can do the same with Remote Execution, Ansible, or your tool of choice. It’s true that redhat_register will install the katello-agent package although you can prevent it from doing that by setting the host parameter redhat_install_agent to false. It sounds like you’re not using katello-agent and if Ansible is working well for your use case (which is great to hear) then you may try removing katello-agent from your content hosts if now isn’t a good time to upgrade to a newer Katello. I’d be curious to know if the qdrouter memory usage improves for the time being.

FYI when you do upgrade 4.1+ you’ll want to run the installer with --foreman-proxy-content-enable-katello-agent=false to remove qpid, qdrouterd, etc from the Katello server.

1 Like
5

Awesome, thanks for clearing that up for me. Great to know about the host parameter and the installer option too.

We’re in a “freeze” right now in my organization, so I’ll spend that time keeping an eye on memory usage and put together a plan to either remove katello-agent or upgrade to 4.1+. Once one or the other is done, I’ll report back.

Honestly, my initial desire is just to take the upgrade route. I feel like I’m falling behind in versions and don’t want to get left in the dust with unsupported software! :slight_smile:

1 Like