Foreman and SSL certificates for Qpid

dj_kill · February 4, 2020, 8:17am

Hello!
Can I ask for some support with SSL?

Problem:
Foreman and most of the components cannot connect to QPID and this case different problems.

pulp_auth stays in error state with

SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca

Thousands of failed tasks ‘Listen on candlepin events’ with traces like:
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in block in assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in tap’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in match_value’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in block in match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in match’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.3/lib/dynflow/execution_plan/steps/error.rb:12:in `new’

So, I’ve replaced /etc/pki/katello/qpid_client_striped.crt to correct one and now foreman-debug start showing qpid stats. But rest of the stuff still doesn’t work.

Expected outcome:
All related tasks are working

Foreman and Proxy versions:
Foreman 1.22.2

Foreman and Proxy plugin versions:
katello 3.12.3-1.el7
candlepin 2.6.8-1.el7
pulp-katello 1.0.2-1.el7
qpid-proton-c.x86_64 0.29.0-1.el7

Distribution and version:
CentOS Linux release 7.6.1810 (Core)

Other relevant data:
Console:
foreman-rake console
Loading production environment (Rails 5.2.1)
irb(main):001:0> conn = Qpid::Messaging::Connection.new(:url => “amqp:ssl:localhost:5671”, :options => {:transport => ‘ssl’})
=> #<Qpid::Messaging::Connection:0x00000000099a5638 @url=“amqp:ssl:localhost:5671”, @options={“transport”=>“ssl”}, @connection_impl=#<Cqpid::Connection:0x00000000099a54a8 @swigtype="_p_qpid__messaging__Connection">>
irb(main):002:0> conn.open
terminate called after throwing an instance of ‘qpid::types::Exception’
what(): Failed to initialise SSL: Failed: NSS error [-8015] (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/sys/ssl/util.cpp:100) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/client/SslConnector.cpp:149)
/tmp/tmp.udui85xPle: line 1: 29196 Aborted rake console
[root@bdapmgmtsbx01 private]# foreman-debug

Foreman logs:
Starting Rails environment
Starting dynflow with the following options: {:rails_root=>"/usr/share/foreman", :process_name=>“dynflow_executor”, :pid_dir=>"/usr/share/foreman/tmp/pids", :log_dir=>"/usr/share/foreman/log", :wait_attempts=>300, :wait_sleep=>1, :executors_count=>1, :memory_limit=>0, :memory_init_delay=>7200, :memory_polling_interval=>60}
Everything ready for world: 16338e89-ac72-43a1-9b42-07d733011383
terminate called after throwing an instance of ‘qpid::types::Exception’
what(): Failed to initialise SSL: Failed: NSS error [-8015] (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/sys/ssl/util.cpp:100) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/client/SslConnector.cpp:149)
Starting Rails environment
Starting dynflow with the following options: {:rails_root=>"/usr/share/foreman", :process_name=>“dynflow_executor”, :pid_dir=>"/usr/share/foreman/tmp/pids", :log_dir=>"/usr/share/foreman/log", :wait_attempts=>300, :wait_sleep=>1, :executors_count=>1, :memory_limit=>0, :memory_init_delay=>7200, :memory_polling_interval=>60}

ehelms · February 4, 2020, 12:33pm

Hello,

Can you provide some background on how you installed and whether this started failing after some change you made or on a fresh install?

dj_kill · February 4, 2020, 1:07pm

Hello,

This is fresh install.
Installed all necessary packages via yum and than run:

/sbin/foreman-installer --scenario katello --puppet-server=false --puppet-server-ca=false
–no-enable-puppet --foreman-proxy-content-puppet

In the Foreman->About all except pulp_auth is Ok:

hammer ping
candlepin:
    Status:          ok
    Server Response: Duration: 23ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 12ms
pulp:
    Status:          ok
    Server Response: Duration: 93ms
pulp_auth:
    Status: FAIL
foreman_tasks:
    Status:          ok
    Server Response: Duration: 3ms

All services are up and running:

Redirecting to 'foreman-maintain service'
Running Service List
================================================================================
List applicable services:
dynflowd.service                              enabled
foreman-proxy.service                         enabled
httpd.service                                 enabled
pulp_celerybeat.service                       enabled
pulp_resource_manager.service                 enabled
pulp_streamer.service                         enabled
pulp_workers.service                          enabled
puppetserver.service                          enabled
qdrouterd.service                             enabled
qpidd.service                                 enabled
rh-mongodb34-mongod.service                   enabled
squid.service                                 enabled
tomcat.service                                enabled

All services listed                                                   [OK]
--------------------------------------------------------------------------------

With trace+ logging of qpidd I cannot see any obvious errors or even warnings.

2 tasks constantly failing: Actions::Katello::EventQueue::Monitor and Actions::Candlepin::ListenOnCandlepinEvents but in Dynaflow all flows and plans are valid.

ehelms · February 4, 2020, 1:09pm

Since this is a fresh install, may I suggest doing so with Foreman 1.24/Katello 3.14 as this is the latest and most supported version? If the issue still occurs with that version, I can spin up and attempt the same install myself.

dj_kill · February 5, 2020, 12:14pm

After upgrade to 1.24.2 with katello 3.14 and pulp 2.21 issue still persists.

But now I can upload RH license manifest to Foreman (which is significant improvement).
Also I see 2 new components that may be related:
katello_events FAIL Not running
candlepin_events FAIL Not running

And if I try to register RH with licenses on Foreman I’ve got (even with Insecure=True)

2020-02-05 12:57:11,525 [INFO] subscription-manager:32749:MainThread @connection.py:905 - Connection built: host=bdapmgmtsbx01.bdap port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2020-02-05 12:57:11,535 [INFO] subscription-manager:32749:MainThread @managercli.py:462 - X-Correlation-ID: f18ba53e897447c28d206d7303880ea1
2020-02-05 12:57:11,536 [INFO] subscription-manager:32749:MainThread @managercli.py:351 - Client Versions: {'subscription-manager': '1.24.13-3.el7.centos'}
2020-02-05 12:57:11,536 [INFO] subscription-manager:32749:MainThread @connection.py:905 - Connection built: host=bdapmgmtsbx01.bdap port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2020-02-05 12:57:11,536 [INFO] subscription-manager:32749:MainThread @connection.py:905 - Connection built: host=bdapmgmtsbx01.bdap port=443 handler=/rhsm auth=none
2020-02-05 12:57:11,536 [INFO] subscription-manager:32749:MainThread @managercli.py:351 - Client Versions: {'subscription-manager': '1.24.13-3.el7.centos'}
2020-02-05 12:57:11,537 [INFO] subscription-manager:32749:MainThread @managercli.py:327 - Consumer Identity name=None uuid=None
2020-02-05 12:57:11,537 [INFO] subscription-manager:32749:MainThread @managercli.py:327 - Consumer Identity name=None uuid=None
2020-02-05 12:57:15,785 [INFO] subscription-manager:32749:MainThread @connection.py:905 - Connection built: host=bdapmgmtsbx01.bdap port=443 handler=/rhsm auth=basic username=admin
2020-02-05 12:57:15,899 [INFO] subscription-manager:32749:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/users/admin/owners"
2020-02-05 12:57:16,943 [INFO] subscription-manager:32749:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/"
2020-02-05 12:57:17,108 [INFO] subscription-manager:32749:MainThread @connection.py:618 - Response: status=200, request="GET /rhsm/owners/Default_Organization/environments"
2020-02-05 12:57:17,128 [INFO] subscription-manager:32749:MainThread @dmiinfo.py:76 - Using dmidecode dump file: /dev/mem
2020-02-05 12:57:21,792 [INFO] subscription-manager:32749:MainThread @connection.py:618 - Response: status=500, request="POST /rhsm/environments/64c2cce3716a5bf67b0413fac95e5daa/consumers"
2020-02-05 12:57:21,792 [ERROR] subscription-manager:32749:MainThread @managercli.py:1335 - HTTP error (500 - Internal Server Error): SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 1332, in _do_command
    type=self.options.consumertype
  File "/usr/lib64/python2.7/site-packages/rhsmlib/services/register.py", line 91, in register
    usage=usage
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1011, in registerConsumer
    return self.conn.request_post(url, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 729, in request_post
    return self._request("POST", method, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 627, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 689, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: HTTP error (500 - Internal Server Error): SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca
2020-02-05 13:02:11,625 [INFO] subscription-manager:947:MainThread @connection.py:905 - Connection built: host=bdapmgmtsbx01.bdap port=443 handler=

Since it 500 I think it’s also related to Foreman and not rhsm itself.

And now apache log of foreman full of:

[Wed Feb 05 12:59:41.054324 2020] [ssl:warn] [pid 7440] [client 172.18.2.5:34646] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Wed Feb 05 12:59:41.066684 2020] [ssl:warn] [pid 7435] [client 172.18.2.5:34648] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Wed Feb 05 12:59:41.217945 2020] [ssl:error] [pid 7433] [client 172.18.2.5:34654] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain

ehelms · February 5, 2020, 12:57pm

This is probably still a related case given you upgraded. Since the original was a fresh install that was broken, are you able to install 1.24 entirely new rather than upgrade? If not, that’s fine. You can try resetting all certificates through the installer.

foreman-installer --certs-update-all

dj_kill · February 5, 2020, 1:17pm

Last step was:

/sbin/foreman-installer --scenario katello --puppet-server=false --puppet-server-ca=false --no-enable-puppet --foreman-proxy-content-puppet false --certs-update-all

But I’ve ran foreman-installer --certs-update-all one more time:

foreman-installer --certs-update-all
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/java-client for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-tomcat for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-qpid-broker for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-foreman-proxy-client for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-qpid-client-cert for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-apache for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-foreman-client for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/pulp-client for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-foreman-proxy for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-qpid-router-server for update
Marking certificate /root/ssl-build/bdapmgmtsbx01.bdap/bdapmgmtsbx01.bdap-qpid-router-client for update
Preparing installation Debug: Class[Foreman::Service]: The container Clas

Preparing installation Done
  Success!
  * Katello is running at https://bdapmgmtsbx01.bdap
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  The full log is at /var/log/foreman-installer/katello.log

No changes at all…
And I cannot perform 100% clean install. But this “upgrade” was pretty close to clean install started from yum erase…

ehelms · February 5, 2020, 1:28pm

Are you still getting errors?

dj_kill · February 5, 2020, 1:48pm

Yes. No general changes in system behavior.

foreman-rake console

Loading production environment (Rails 5.2.1)
irb(main):001:0> conn = Qpid::Messaging::Connection.new(:url => "amqp:ssl:localhost:5671", :options => {:transport => 'ssl'})
=> #<Qpid::Messaging::Connection:0x00000000086854b8 @url="amqp:ssl:localhost:5671", @options={"transport"=>"ssl"}, @connection_impl=#<Cqpid::Connection:0x0000000008685440 @__swigtype__="_p_qpid__messaging__Connection">>
irb(main):002:0> conn.open
terminate called after throwing an instance of 'qpid::types::Exception'
  what():  Fail

dj_kill · February 5, 2020, 1:54pm

Actual problem I’m dealing with is:

subscription-manager register --username=admin --password=******* --org=Default_Organization
Registering to: bdapmgmtsbx01.bdap:443/rhsm
HTTP error (500 - Internal Server Error): SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca

I’m thinking these issues are related to Qpid and certificates (cause also I see failing tasks). But not 1000% sure

ehelms · February 5, 2020, 2:45pm

This registration error is happening on a client host? Can you reinstall the latest katello-consumer RPM on it?

dj_kill · February 5, 2020, 3:26pm

Yes, on client.
Here is fresh install:

[root@bdaposmaster1sbx01 ~]# rpm -e katello-ca-consumer-bdapmgmtsbx01.bdap-1.0-1.noarch
[root@bdaposmaster1sbx01 ~]# rpm -Uvh https://bdapmgmtsbx01.bdap/pub/katello-ca-consumer-latest.noarch.rpm
Retrieving https://bdapmgmtsbx01.bdap/pub/katello-ca-consumer-latest.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:katello-ca-consumer-bdapmgmtsbx01################################# [100%]
[root@bdaposmaster1sbx01 ~]# subscription-manager register --username=admin --password=***** --org=Default_Organization
Registering to: bdapmgmtsbx01.bdap:443/rhsm
HTTP error (500 - Internal Server Error): SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca
[root@bdaposmaster1sbx01 ~]# date
Wed Feb  5 16:24:21 CET 2020
[root@bdaposmaster1sbx01 ~]#

dj_kill · February 5, 2020, 4:14pm

One more thing to add (and that’s why I’m thinking this is not client-side problem): when I 'm trying to add repository to product I’ve got:

There was an issue with the backend service pulp_auth: SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca

production.log at that moment:

2020-02-05T16:50:53 [I|aud|] Katello::RootRepository (2) create event on http_proxy_id
2020-02-05T16:50:54 [E|kat|] GET: https://bdapmgmtsbx01.bdap/pulp/api/v2/users/: {"content_type"=>"application/json", "accept"=>"application/json"}
SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca:
2020-02-05T16:50:54 [W|app|] SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in `connect_nonblock'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in `ssl_socket_connect'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:985:in `connect'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:920:in `do_start'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:909:in `start'
...

ehelms · February 5, 2020, 5:36pm

Hrmm, something is not getting updated and it’s not clear why. If you want the full blown route since you’d re-install clean.

rm -rf /root/ssl-build
rm -rf /etc/pki/katello-certs-tools
rm -rf /var/www/html/pub/katello-ca-consumer-pipeline-upgrade-katello-nightly-centos7.war.example.com-1.0-1.noarch.rpm

Now re-run the installer with the --certs-update-all flag

dj_kill · February 6, 2020, 7:40am

Not a big improvement.

[root@bdapmgmtsbx01 ~]# rm -rf /root/ssl-build
[root@bdapmgmtsbx01 ~]# rm -rf /etc/pki/katello-certs-tools
[root@bdapmgmtsbx01 ~]# rm -rf /var/www/html/pub/katello-ca-consumer-pipeline-upgrade-katello-nightly-centos7.war.example.com-1.0-1.noarch.rpm
[root@bdapmgmtsbx01 ~]# foreman-installer --certs-update-all
Preparing installation Done
  Success!
  * Katello is running at https://bdapmgmtsbx01.bdap
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  The full log is at /var/log/foreman-installer/katello.log
[root@bdapmgmtsbx01 ~]# date
Thu Feb  6 08:37:31 CET 2020
[root@bdapmgmtsbx01 ~]# foreman-rake console

Loading production environment (Rails 5.2.1)
irb(main):001:0> conn = Qpid::Messaging::Connection.new(:url => "amqp:ssl:localhost:5671", :options => {:transport => 'ssl'})
=> #<Qpid::Messaging::Connection:0x000000000e8f1930 @url="amqp:ssl:localhost:5671", @options={"transport"=>"ssl"}, @connection_impl=#<Cqpid::Connection:0x000000000e8f18b8 @__swigtype__="_p_qpid__messaging__Connection">>
irb(main):002:0> conn.open
terminate called after throwing an instance of 'qpid::types::Exception'
  what():  Failed to initialise SSL: Failed: NSS error [-8015] (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/sys/ssl/util.cpp:100) (/builddir/build/BUILD/qpid-cpp-1.39.0/src/qpid/client/SslConnector.cpp:149)
Aborted
[root@bdapmgmtsbx01 ~]# date
Thu Feb  6 08:38:16 CET 2020
[root@bdapmgmtsbx01 ~]#

How can I check what certificates used for what connections? How it’s possible that foreman-rake using different cert than foreman-debug? Can I trace this somehow?

dj_kill · February 6, 2020, 7:55am

Oh! Just noticed!
After last certificate clean-up process status in interface looks worse:

pulp	OK	
pulp_auth	FAIL	SSL_connect returned=1 errno=0 state=error: tlsv1 alert unknown ca
candlepin	FAIL	SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
candlepin_auth	FAIL	Katello::Resources::Candlepin::CandlepinPing: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (GET /candlepin/status)
foreman_tasks	OK	
katello_events	FAIL	Not running
candlepin_events	FAIL	Not running

Now candlepin also fails…

dj_kill · February 6, 2020, 8:04am

And one more thing not broken:

[root@bdapmgmtsbx01 foreman-debug-Y8Nk7]# qpid-stat --ssl-certificate=/etc/pki/pulp/qpid/client.crt -b amqps://localhost:5671 -q katello_event_queue
Properties:
  Name                 Durable  AutoDelete  Exclusive  FlowStopped  FlowStoppedCount  Consumers  Bindings
  =========================================================================================================
  katello_event_queue  Y        N           N          N            0                 0          7

Optional Properties:
  Property      Value
  =====================
  arguments     {}
  alt-exchange

Statistics:
  Statistic                   Messages  Bytes
  ==============================================
  queue-depth                 48        37,455
  total-enqueues              48        37,455
  total-dequeues              0         0
  persistent-enqueues         48        37,455
  persistent-dequeues         0         0
  transactional-enqueues      0         0
  transactional-dequeues      0         0
  flow-to-disk-depth          0         0
  flow-to-disk-enqueues       0         0
  flow-to-disk-dequeues       0         0
  acquires                    0
  releases                    0
  discards-ttl-expired        0
  discards-limit-overflow     0
  discards-ring-overflow      0
  discards-lvq-replace        0
  discards-subscriber-reject  0
  discards-purged             0
  reroutes                    0
[root@bdapmgmtsbx01 foreman-debug-Y8Nk7]# qpid-stat --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 -q katello_event_queue
Failed: ConnectError - [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:618)
[root@bdapmgmtsbx01 foreman-debug-Y8Nk7]#

dj_kill · February 6, 2020, 11:30am

And one more difference in certificates:

[root@bdapmgmtsbx01 certs]# keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN

Your keystore contains 1 entry

amqp-client, Jan 31, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1): FB:19:FB:69:C3:C0:72:C1:A4:85:BB:3E:37:40:58:77:25:39:FE:5E

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/candlepin/certs/amqp/candlepin.jks -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -deststoretype pkcs12".

[root@bdapmgmtsbx01 certs]# openssl x509 -noout -fingerprint -sha1 -inform pem -in /etc/pki/pulp/qpid/client.crt
SHA1 Fingerprint=F0:F3:4B:F2:B4:60:2D:EF:BB:2A:18:A0:ED:44:53:72:25:9C:FD:19

This actually means that certificate in candlepin keystore is different than required for Qpid.