After yum upgrade, running foreman-installer --scenario katello --upgrade throws an SSL error. To my knowledge, nothing has changed on the system wrt certs. Has candlepin suddenly become stricter?
Upgrade Step: clean_backend_objects (this may take a while) ...
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
rake aborted!
RestClient::SSLCertificateNotVerified: Katello::Resources::Candlepin::Consumer: SSL_connect returned=1 errno=0 state=error: certificate verify failed (GET /candlepin/consumers/?owner=COMPANY&include=uuid&per_page=15000&page=1)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:758:in `rescue in transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:642:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/http_resource.rb:84:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:22:in `block in get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin.rb:65:in `block in fetch_paged'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin.rb:63:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin.rb:63:in `fetch_paged'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:21:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:10:in `block in all_uuids'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:9:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:9:in `all_uuids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/lib/katello/tasks/clean_backend_objects.rake:13:in `populate!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/lib/katello/tasks/clean_backend_objects.rake:93:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/http_resource.rb:84:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:22:in `block in get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin.rb:65:in `block in fetch_paged'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin.rb:63:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin.rb:63:in `fetch_paged'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:21:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:10:in `block in all_uuids'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:9:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/app/lib/katello/resources/candlepin/consumer.rb:9:in `all_uuids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/lib/katello/tasks/clean_backend_objects.rake:13:in `populate!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.0/lib/katello/tasks/clean_backend_objects.rake:93:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:clean_backend_objects
(See full trace by running task with --trace)
foreman-rake katello:clean_backend_objects COMMIT=true failed! Check the output for error!
Upgrade step clean_backend_objects failed. Check logs for more information.
Expected outcome:
Error doesn’t happen
Better error message suggesting a solution?
Foreman and Proxy versions:
foreman 1.19.0
Foreman and Proxy plugin versions:
katello 3.8.0
candlepin-2.4.4-1.el7
I think I’ve rediscovered this bug but I don’t know how to solve the problem - download the diff and apply it to /usr/share/foreman-installer-katello/bin/katello-certs-check?
Downloaded the file, found it was the same as /usr/sbin. Replaced /usr/share/foreman-installer/katello/bin but that didn’t work as a solution. Might just comment out the function check-cert-san (lines 139-148) and the calling of it (line 165) for the moment.
2018-09-12 12:36:23,748 [thread=localhost-startStop-1] [=, org=, csid=] ERROR org.candlepin.audit.ActiveMQContextListener - Error starting
AMQP client
javax.jms.JMSException: Error creating connection: General SSLEngine problem
at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
at org.candlepin.audit.QpidConnection.newConnection(QpidConnection.java:170)
at org.candlepin.audit.QpidConnection.connect(QpidConnection.java:151)
at org.candlepin.audit.ActiveMQContextListener.setupAmqp(ActiveMQContextListener.java:204)
at org.candlepin.audit.ActiveMQContextListener.contextInitialized(ActiveMQContextListener.java:169)
at org.candlepin.guice.CandlepinContextListener.withInjector(CandlepinContextListener.java:182)
at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletC
ontextListener.java:57)
at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:144)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.qpid.AMQConnectionFailureException: General SSLEngine problem
at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:520)
at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
... 20 common frames omitted
Caused by: org.apache.qpid.transport.SenderException: SSL, Error occurred while encrypting data
at org.apache.qpid.transport.network.security.ssl.SSLSender.send(SSLSender.java:165)
at org.apache.qpid.transport.network.security.ssl.SSLSender.send(SSLSender.java:35)
at org.apache.qpid.transport.network.Disassembler.init(Disassembler.java:160)
at org.apache.qpid.transport.network.Disassembler.init(Disassembler.java:48)
at org.apache.qpid.transport.ProtocolHeader.delegate(ProtocolHeader.java:110)
at org.apache.qpid.transport.network.Disassembler.send(Disassembler.java:73)
at org.apache.qpid.transport.network.Disassembler.send(Disassembler.java:48)
at org.apache.qpid.transport.Connection.send(Connection.java:420)
at org.apache.qpid.transport.Connection.connect(Connection.java:259)
at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:222)
at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:664)
at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:444)
... 21 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.qpid.transport.network.security.ssl.SSLSender.send(SSLSender.java:157)
... 32 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at org.apache.qpid.transport.network.security.ssl.SSLReceiver.doTasks(SSLReceiver.java:209)
at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:165)
at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:36)
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161)
... 1 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
... 11 common frames omitted
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:154)
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
... 17 common frames omitted
2018-09-12 12:36:24,023 [thread=Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@65042107)] [=, org=, csid=] WARN org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings
2018-09-12 12:36:24,024 [thread=Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@65042107)] [=, org=, csid=] WARN org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings
I restored an old snapshot of this VM. Tried the foreman-installer --scenario katello --upgrade incantation on the previous version of Katello(3.7.0)/Foreman(1.18.1) and got the same result.
It’s a very specific error message and I have all the information at hand except what’s gone wrong and how do I fix it.
Any tips on how to either fix our route around would be appreciated.
Unfortunately the IT dept moves too slowly to make this check workable - recovering VM snapshots from tape at this point - between the lag here and the lag there, I moved on to just replacing the certs.
@Chris_Duryee the answer to that question was a solid “didn’t know” last week. Now, I can definitively answer yes - moved out of the way and replaced and returned a couple of times.
What problems can that lead to and how would you recommend solving them?
Today I’ve made two attempts to solve this problem - once using openssl and once using katello-ssl-tools.
With katello-ssl-tool, the same problem is encountered. It doesn’t work atm.
With openssl, and lots of reading and dancing around, I finally got a ca set built, and a server set built. When I run katello-certs-check -c certs/katello-server.crt -k private/katello-server.key -b certs/katello-new-ca.crt the output looks great:
Checking server certificate encoding:
[OK]
Checking expiration of certificate:
[OK]
Checking expiration of CA bundle:
[OK]
Checking if server certificate has CA:TRUE flag
[OK]
Checking to see if the private key matches the certificate:
[OK]
Checking CA bundle against the certificate file:
[OK]
Checking Subject Alt Name on certificate
[OK]
Checking Key Usage extension on certificate for Key Encipherment
[OK]
Validation succeeded
To install the Katello main server with the custom certificates, run:
foreman-installer --scenario katello\
--certs-server-cert "/root/ca/certs/katello-server.crt"\
--certs-server-key "/root/ca/private/katello-server.key"\
--certs-server-ca-cert "/root/ca/certs/katello-new-ca.crt"
To update the certificates on a currently running Katello installation, run:
foreman-installer --scenario katello\
--certs-server-cert "/root/ca/certs/katello-server.crt"\
--certs-server-key "/root/ca/private/katello-server.key"\
--certs-server-ca-cert "/root/ca/certs/katello-new-ca.crt"\
--certs-update-server --certs-update-server-ca
I’ve had to update the cnf file - I don’t know why, maybe some of these options are hidden defaults. I originally copied the cnf from ssl-build, added the subjectAltName that is required. Added some other love - made the req_distinguished_name a bit more personal and localized. I also increase the default_bits to 4096
These were the options that I was asked to add by foreman-installer:
[ CA_default ]
new_certs_dir =
default_md =
unique_subject = no
database =
serial =
default_days =
This was an unpleasant experience. Might be worth whipping up a little example on how it could be done and/or making sure that katello-ssl-tool works.
