Qpid-stat command fails after OS upgrade in place with leapp

Support
1

Problem:
Hi everyone,
after upgrading OS from RHEL7 to RHEL8 following
https://docs.theforeman.org/3.3/Upgrading_and_Updating/index-katello.html#upgrading-foreman-or-proxy-in-place-using-leapp_upgrade-guide

qpid-stat command fail with error

# qpid-stat --ssl-certificate=/etc/pki/pulp/qpid/client.crt -b "amqps://localhost:5671" -q katello.agent
Failed: AuthenticationFailure - Error in sasl_client_start (-4) SASL(-4): no mechanism available:

Expected outcome:
for command to show queue stats like the following

Properties:
  Name           Durable  AutoDelete  Exclusive  FlowStopped  FlowStoppedCount  Consumers  Bindings
  ===================================================================================================
  katello.agent  Y        N           N          N            0                 1          1

Optional Properties:
  Property      Value
  =====================
  arguments     {}
  alt-exchange

Statistics:
  Statistic                   Messages  Bytes
  =============================================
  queue-depth                 0         0
  total-enqueues              0         0
  total-dequeues              0         0
  persistent-enqueues         0         0
  persistent-dequeues         0         0
  transactional-enqueues      0         0
  transactional-dequeues      0         0
  flow-to-disk-depth          0         0
  flow-to-disk-enqueues       0         0
  flow-to-disk-dequeues       0         0
  acquires                    0
  releases                    0
  discards-ttl-expired        0
  discards-limit-overflow     0
  discards-ring-overflow      0
  discards-lvq-replace        0
  discards-subscriber-reject  0
  discards-purged             0
  reroutes                    0

Foreman and Proxy versions:
Foreman 3.3.1 / Katello 4.5.1

Foreman and Proxy plugin versions:

Installed Packages

    ansible-collection-theforeman-foreman-3.4.0-1.el8.noarch
    candlepin-4.1.11-1.el8.noarch
    candlepin-selinux-4.1.11-1.el8.noarch
    foreman-3.3.1-1.el8.noarch
    foreman-cli-3.3.1-1.el8.noarch
    foreman-debug-3.3.1-1.el8.noarch
    foreman-dynflow-sidekiq-3.3.1-1.el8.noarch
    foreman-installer-3.3.1-2.el8.noarch
    foreman-installer-katello-3.3.1-2.el8.noarch
    foreman-postgresql-3.3.1-1.el8.noarch
    foreman-proxy-3.3.1-1.el8.noarch
    foreman-release-3.3.1-1.el8.noarch
    foreman-selinux-3.3.1-1.el8.noarch
    foreman-service-3.3.1-1.el8.noarch
    foreman-vmware-3.3.1-1.el8.noarch
    katello-4.5.1-1.el8.noarch
    katello-certs-tools-2.9.0-1.el8.noarch
    katello-client-bootstrap-1.7.9-1.el8.noarch
    katello-common-4.5.1-1.el8.noarch
    katello-debug-4.5.1-1.el8.noarch
    katello-default-ca-1.0-1.noarch
    katello-repos-4.5.1-1.el8.noarch
    katello-selinux-4.0.2-1.el8.noarch
    katello-server-ca-1.0-1.noarch
    pulp-client-1.0-7.noarch
    pulpcore-selinux-1.3.2-1.el8.x86_64
    python2-qpid-1.37.0-1.el8.noarch
    python2-qpid-qmf-1.39.0-7.el8.x86_64
    python3-qpid-proton-0.37.0-1.el8.x86_64
    python39-pulp-ansible-0.13.5-1.el8.noarch
    python39-pulp-certguard-1.5.2-3.el8.noarch
    python39-pulp-cli-0.14.0-4.el8.noarch
    python39-pulp-container-2.10.12-1.el8.noarch
    python39-pulp-deb-2.18.3-1.el8.noarch
    python39-pulp-file-1.10.2-2.el8.noarch
    python39-pulp-python-3.7.1-1.el8.noarch
    python39-pulp-rpm-3.18.13-1.el8.noarch
    python39-pulpcore-3.18.25-1.el8.noarch
    qpid-cpp-client-1.39.0-7.el8.x86_64
    qpid-cpp-client-devel-1.39.0-7.el8.x86_64
    qpid-cpp-server-1.39.0-7.el8.x86_64
    qpid-cpp-server-linearstore-1.39.0-7.el8.x86_64
    qpid-dispatch-router-1.14.0-1.el8.x86_64
    qpid-proton-c-0.37.0-1.el8.x86_64
    qpid-qmf-1.39.0-7.el8.x86_64
    qpid-tools-1.39.0-7.el8.noarch
    qpid_router_katello_agent-qpid-router-client-1.0-1.noarch
    rubygem-foreman-tasks-6.0.2-1.fm3_3.el8.noarch
    rubygem-foreman_ansible-7.1.4-1.fm3_3.el8.noarch
    rubygem-foreman_maintain-1.1.10-1.el8.noarch
    rubygem-foreman_puppet-4.0.4-1.fm3_3.el8.noarch
    rubygem-foreman_remote_execution-7.2.2-1.fm3_3.el8.noarch
    rubygem-foreman_snapshot_management-2.0.1-1.fm2_6.el8.noarch
    rubygem-foreman_virt_who_configure-0.5.8-2.fm3_3.el8.noarch
    rubygem-foreman_vmwareannotations-0.0.1-6.fm3_3.el8.noarch
    rubygem-foreman_wreckingball-4.0.0-2.fm3_3.el8.noarch
    rubygem-hammer_cli-3.3.0-1.el8.noarch
    rubygem-hammer_cli_foreman-3.3.0-1.el8.noarch
    rubygem-hammer_cli_foreman_ansible-0.3.4-1.fm3_0.el8.noarch
    rubygem-hammer_cli_foreman_bootdisk-0.3.0-2.el8.noarch
    rubygem-hammer_cli_foreman_puppet-0.0.6-1.fm3_3.el8.noarch
    rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
    rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
    rubygem-hammer_cli_foreman_virt_who_configure-0.0.8-1.el8.noarch
    rubygem-hammer_cli_katello-1.5.3-1.el8.noarch
    rubygem-katello-4.5.1-1.el8.noarch
    rubygem-pulp_ansible_client-0.13.1-1.el8.noarch
    rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
    rubygem-pulp_container_client-2.10.3-1.el8.noarch
    rubygem-pulp_deb_client-2.18.0-1.el8.noarch
    rubygem-pulp_file_client-1.10.0-1.el8.noarch
    rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
    rubygem-pulp_python_client-3.6.0-1.el8.noarch
    rubygem-pulp_rpm_client-3.17.4-1.el8.noarch
    rubygem-pulpcore_client-3.18.5-1.el8.noarch
    rubygem-qpid_proton-0.37.0-1.el8.x86_64
    rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch

Distribution and version:
OS upgraded from RHEL7 to RHEL8 (7.9 to 8.9) using leapp

Additional details
hosts connected to Foreman server successfully connect via goferd and we can install packages remotely via katello-agents.

regards

2

If you can perform package actions via katello-agent, why is it a problem that qpid-stat fails?

I would recommend upgrading Katello to a supported version and switching to remote execution pull mode. We removed katello-agent a while ago so we don’t have a lot of resources to test and support it anymore.

3

Hi,
it’s not a problem, but I was wondering what could have been broken/changed during the upgrade.
I know it’s not strictly related to Foreman either but maybe someone might have an idea or faced the same or similar issue before.

We have the same qpid and cyrus-sasl packages installed pre and post upgrade (obviously different version) and the same loaded configurations (/etc/qpid/qpidd.conf, /etc/sasl2/qdrouterd.conf, /etc/sasl2/qpidd.conf).

qpid conf pre and post upgrade

# cat /etc/qpid/qpidd.conf | grep -v '^#' | grep -v '^$'
acl-file=/etc/qpid/qpid.acl
data-dir=/var/lib/qpidd
log-enable=error+
log-to-syslog=yes
auth=yes
require-encryption=yes
ssl-require-client-authentication=yes
ssl-port=5671
ssl-cert-db=/etc/pki/katello/nssdb
ssl-cert-password-file=/etc/pki/katello/nss_db_password-file
ssl-cert-name=broker
interface=lo
wcache-page-size=4

sasl qpid conf pre and post upgrade

# cat /etc/sasl2/qpidd.conf | grep -v '^#' | grep -v '^$'
pwcheck_method: auxprop
auxprop_plugin: sasldb
sasldb_path: /var/lib/qpidd/qpidd.sasldb
mech_list: ANONYMOUS DIGEST-MD5 EXTERNAL PLAIN
sql_select: dummy select

sasl qdrouter conf pre and post upgrade

# cat /etc/sasl2/qdrouterd.conf | grep -v '^#' | grep -v '^$'
pwcheck_method: auxprop
auxprop_plugin: sasldb
sasldb_path: /var/lib/qdrouterd/qdrouterd.sasldb
mech_list: ANONYMOUS DIGEST-MD5 EXTERNAL PLAIN
sql_select: dummy select

installed packages PRE upgrade

$ rpm -qa | grep cyrus
cyrus-sasl-lib-2.1.26-24.el7_9.x86_64
cyrus-sasl-2.1.26-24.el7_9.x86_64
cyrus-sasl-gssapi-2.1.26-24.el7_9.x86_64
cyrus-sasl-plain-2.1.26-24.el7_9.x86_64

$ rpm -qa | grep qpid
qpid-cpp-client-1.39.0-1.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python2-qpid-qmf-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
qpid-proton-c-0.37.0-1.el7.x86_64
python36-qpid-proton-0.37.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.19.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
tfm-rubygem-qpid_proton-0.37.0-1.el7.x86_64
python2-qpid-1.37.0-5.el7.noarch

installed packages POST upgrade

# rpm -qa | grep cyrus
cyrus-sasl-lib-2.1.27-6.el8_5.i686
cyrus-sasl-lib-2.1.27-6.el8_5.x86_64
cyrus-sasl-plain-2.1.27-6.el8_5.x86_64
cyrus-sasl-gssapi-2.1.27-6.el8_5.x86_64
cyrus-sasl-2.1.27-6.el8_5.x86_64

# rpm -qa | grep qpid
qpid-cpp-client-1.39.0-7.el8.x86_64
qpid-proton-c-0.37.0-1.el8.x86_64
qpid_router_katello_agent-qpid-router-client-1.0-1.noarch
qpid-cpp-server-1.39.0-7.el8.x86_64
qpid-tools-1.39.0-7.el8.noarch
python3-qpid-proton-0.37.0-1.el8.x86_64
python2-qpid-qmf-1.39.0-7.el8.x86_64
python2-qpid-1.37.0-1.el8.noarch
qpid-cpp-client-devel-1.39.0-7.el8.x86_64
qpid-qmf-1.39.0-7.el8.x86_64
qpid-cpp-server-linearstore-1.39.0-7.el8.x86_64
rubygem-qpid_proton-0.37.0-1.el8.x86_64
qpid-dispatch-router-1.14.0-1.el8.x86_64

So the message ‘no mechanism available’ is baffling me.

Anyway we are planning to upgrade to Foreman 3.7 - Katello 4.9 and then prepare to switch to remote execution and upgrade to Foreman 3.9 - Katello 4.11 or later depending on version available.

Regards,
Fabio

1 Like
4

I have an update,
if I add to the qpid-stat command the parameter ‘–sasl-mechanism ANONYMOUS’ it works.

It seems that now the sasl mechanism that does not work locally on the server is the EXTERNAL one,
but without it the client hosts do not connect though goferd.

So we have a workaround.

Regards,
Fabio