Constant qpidd errors in journal

System journal is full of errors from qpidd like:

Dec 01 14:36:51 qpidd[1804367]: 2021-12-01 14:36:51 [System] error Error reading socket: Encountered end of file [-5938]
Dec 01 14:36:51 qpidd[1804367]: 2021-12-01 14:36:51 [System] error Error reading socket: Encountered end of file [-5938]
Dec 01 14:37:06 qpidd[1804367]: 2021-12-01 14:37:06 [System] error Error reading socket: Encountered end of file [-5938]
Dec 01 14:37:06 qpidd[1804367]: 2021-12-01 14:37:06 [System] error Error reading socket: Encountered end of file [-5938]

I’ve done some searching, and can only find some very old issues which don’t seem to match the versions I have installed.
Expected outcome:
No errors in the journal for normal operation
Foreman and Proxy versions:
Foreman and Proxy plugin versions:
Distribution and version:
Oracle Linux 8.5
Other relevant data:

# systemctl status qpidd
_ qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/qpidd.service.d
           └─90-limits.conf, wait-for-port.conf
   Active: active (running) since Wed 2021-11-24 11:54:29 UTC; 1 weeks 0 days ago
     Docs: man:qpidd(1)
 Main PID: 1804367 (qpidd)
    Tasks: 10 (limit: 216020)
   Memory: 15.5M
   CGroup: /system.slice/qpidd.service
           └─1804367 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

I believe you will see these errors when a client is opening and closing connections quickly or you have a client(s) with SSL issues.

qpidd is only listening on localhost…

tcp        0      0*               LISTEN      1804367/qpidd
tcp        0      0*               LISTEN      1804367/qpidd

Clients talk to qdrouterd which routes connections to the qpidd service running on localhost so you can still end up with this situation if clients are rapidly opening and closing connections or have SSL issues. An example of the latter could be if you changed SSL certificates and a client with old ones is still out there trying to connect.

Ah, OK - given this relaying behaviour, how would I identify those clients?

As far as I can tell, no clients are accessing qdrouterd at all - we uninstalled katello-agent throughout, prior to its removal.

Well, the solution (of sorts) from another thread I’ve just performed is:

foreman-install --foreman-proxy-content-enable-katello-agent=false

Edit: “Of sorts” as you don’t get errors from qpidd when it (and qdrouterd) are no longer running

So, only just discovered this today, when I noticed some python cron jobs were failing - running the installer like above ended up removing a lot of core packages!

2021-12-07T16:28:53+0000 SUBDEBUG Erase: qpid-tools-1.39.0-7.el8.noarch
2021-12-07T16:28:53+0000 SUBDEBUG Erase: python2-qpid-1.37.0-6.el8.noarch
2021-12-07T16:28:53+0000 SUBDEBUG Erase: python2-qpid-qmf-1.39.0-7.el8.x86_64
2021-12-07T16:28:53+0000 SUBDEBUG Erase: python2-saslwrapper-0.22-6.el8.x86_64
2021-12-07T16:28:53+0000 SUBDEBUG Erase: python2-2.7.18-7.0.1.module+el8.5.0+20361+8a9d3d27.x86_64
2021-12-07T16:28:53+0000 SUBDEBUG Erase: python2-libs-2.7.18-7.0.1.module+el8.5.0+20361+8a9d3d27.x86_64
2021-12-07T16:28:54+0000 SUBDEBUG Erase: python2-pip-wheel-9.0.3-18.module+el8.3.0+7833+4aaf98ce.noarch
2021-12-07T16:28:54+0000 SUBDEBUG Erase: python2-setuptools-wheel-39.0.1-13.module+el8.4.0+20050+79c7b4ee.noarch
2021-12-07T16:28:54+0000 SUBDEBUG Erase: saslwrapper-0.22-6.el8.x86_64
2021-12-07T16:28:54+0000 SUBDEBUG Erase: qpid-qmf-1.39.0-7.el8.x86_64
2021-12-07T16:28:57+0000 INFO --- logging initialized ---
2021-12-07T16:28:59+0000 SUBDEBUG Erase: qpid-cpp-server-linearstore-1.39.0-7.el8.x86_64
2021-12-07T16:28:59+0000 SUBDEBUG Erase: qpid-cpp-server-1.39.0-7.el8.x86_64
2021-12-07T16:28:59+0000 INFO warning: file /var/run/qpidd: remove failed: No such file or directory
warning: /etc/qpid/qpidd.conf saved as /etc/qpid/qpidd.conf.rpmsave

2021-12-07T16:29:00+0000 SUBDEBUG Erase: qpid-cpp-client-1.39.0-7.el8.x86_64
2021-12-07T16:29:00+0000 SUBDEBUG Erase: boost-filesystem-1.66.0-10.el8.x86_64
2021-12-07T16:29:00+0000 SUBDEBUG Erase: boost-system-1.66.0-10.el8.x86_64
2021-12-07T16:29:00+0000 SUBDEBUG Erase: boost-program-options-1.66.0-10.el8.x86_64
2021-12-07T16:29:00+0000 SUBDEBUG Erase: libdb-cxx-5.3.28-42.el8_4.x86_64
2021-12-07T16:29:03+0000 INFO --- logging initialized ---
2021-12-07T16:29:04+0000 INFO --- logging initialized ---
2021-12-07T16:29:09+0000 INFO --- logging initialized ---
2021-12-07T16:29:12+0000 INFO --- logging initialized ---
2021-12-07T16:29:14+0000 SUBDEBUG Erase: qpid-dispatch-router-1.14.0-1.el8.x86_64
2021-12-07T16:29:14+0000 INFO warning: /etc/qpid-dispatch/qdrouterd.conf saved as /etc/qpid-dispatch/qdrouterd.conf.rpmsave

2021-12-07T16:29:14+0000 SUBDEBUG Erase: python3-qpid-proton-0.32.0-3.el8.x86_64
2021-12-07T16:29:15+0000 SUBDEBUG Erase: python36-3.6.8-38.module+el8.5.0+20329+5c5719bc.x86_64
2021-12-07T16:29:15+0000 SUBDEBUG Erase: python3-pip-9.0.3-20.el8.noarch
2021-12-07T16:29:15+0000 SUBDEBUG Erase: libwebsockets-2.4.2-2.el8.x86_64

From dnf.log itself:

2021-12-07T16:29:13+0000 INFO Dependencies resolved.
2021-12-07T16:29:13+0000 INFO ============================================================================================
 Package                Arch    Version                                Repository       Size
 qpid-dispatch-router   x86_64  1.14.0-1.el8                           @katello        3.0 M
Removing unused dependencies:
 libwebsockets          x86_64  2.4.2-2.el8                            @katello        243 k
 python3-pip            noarch  9.0.3-20.el8                           @ol8_appstream  2.8 k
 python3-qpid-proton    x86_64  0.32.0-3.el8                           @katello        2.5 M
 python36               x86_64  3.6.8-38.module+el8.5.0+20329+5c5719bc @ol8_appstream   13 k

Transaction Summary
Remove  5 Packages

Worth raising a bug about?

The other removal:

2021-12-07T16:28:49+0000 INFO Dependencies resolved.
2021-12-07T16:28:49+0000 INFO ====================================================================================================
 Package                    Arch    Version                                    Repository       Size
 qpid-tools                 noarch  1.39.0-7.el8                               @katello        281 k
Removing unused dependencies:
 python2                    x86_64  2.7.18-7.0.1.module+el8.5.0+20361+8a9d3d27 @ol8_appstream   80 k
 python2-libs               x86_64  2.7.18-7.0.1.module+el8.5.0+20361+8a9d3d27 @ol8_appstream   25 M
 python2-pip-wheel          noarch  9.0.3-18.module+el8.3.0+7833+4aaf98ce      @ol8_appstream  1.1 M
 python2-qpid               noarch  1.37.0-6.el8                               @PA_Repo        1.8 M
 python2-qpid-qmf           x86_64  1.39.0-7.el8                               @katello        308 k
 python2-saslwrapper        x86_64  0.22-6.el8                                 @katello         76 k
 python2-setuptools-wheel   noarch  39.0.1-13.module+el8.4.0+20050+79c7b4ee    @ol8_appstream  337 k
 qpid-qmf                   x86_64  1.39.0-7.el8                               @katello        500 k
 saslwrapper                x86_64  0.22-6.el8                                 @katello         49 k

Transaction Summary
Remove  10 Packages

OK, I figured why the above didn’t break Pulp at least…

# rpm -q --requires python3-pulpcore
python(abi) = 3.6