Gutterball and candlepin not working after upgrade

Jorick_Astrego · June 16, 2015, 8:48am

Hi,

A week ago I upgraded from 2.1 to 2.2.1 and although everything seemed
ok, I didn't have time to check everything. Yesterday, I noticed the
sync task was not running and I am unable to run any task now.

Checking various logs, I can see gutterball and candelpin are giving
connection erros to QPID (longer log excerpt at the bottom):

2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
org.candlepin.gutterball.receiver.EventReceiver - Unable to
initialize connection to QPID.
java.lang.RuntimeException: javax.jms.JMSException:*Error creating
connection: Keystore was tampered with, or password was incorrect*

2015-06-16 10:02:30,946 [=, org=] ERROR
org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
AMQPBusProvider:
javax.jms.JMSException: Error creating connection: Connection refused

Then I checked the installer log and I can see errors with the certificates:

[DEBUG 2015-06-10 16:37:05 main]  Exec[[import client certificate
into gutterball keystore] signal import if pair has not been
imported](provider=posix): Executing check &#39;keytool -list -keystore
/etc/gutterball/certs/amqp//gutterball.jks -storepass
eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball&#39;
[DEBUG 2015-06-10 16:37:05 main]  Executing &#39;keytool -list -keystore
/etc/gutterball/certs/amqp//gutterball.jks -storepass
eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball&#39;
[DEBUG 2015-06-10 16:37:05 main] 
/Stage[main]/Certs::Gutterball/Certs::Ssltools::Keytool::Import_keypair[import
client certificate into gutterball keystore]/Exec[[import client
certificate into gutterball keystore] signal import if pair has not
been imported]/unless: keytool error: java.io.IOException: Keystore
was tampered with, or password was incorrect
[DEBUG 2015-06-10 16:37:05 main]  Exec[[import client certificate
into gutterball keystore] signal import if pair has not been
imported](provider=posix): Executing &#39;echo importing keypair&#39;
[DEBUG 2015-06-10 16:37:05 main]  Executing &#39;echo importing keypair&#39;


[DEBUG 2015-06-10 16:37:07 main]  Exec[import client certificate
into Candlepin keystore](provider=posix): Executing check &#39;keytool
-list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca&#39;
[DEBUG 2015-06-10 16:37:07 main]  Executing &#39;keytool -list -keystore
/etc/candlepin/certs/amqp/candlepin.jks -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca&#39;
[DEBUG 2015-06-10 16:37:07 main] 
/Stage[main]/Certs::Candlepin/Exec[import client certificate into
Candlepin keystore]/unless: keytool error: java.lang.Exception:
Alias &lt;katello-default-ca&gt; does not exist
[DEBUG 2015-06-10 16:37:07 main]  Exec[import client certificate
into Candlepin keystore](provider=posix): Executing &#39;openssl pkcs12
-export -name amqp-client -in /etc/pki/katello/certs/java-client.crt
-inkey /etc/pki/katello/private/java-client.key -out
/tmp/keystore.p12 -passout
file:/etc/pki/katello/keystore_password-file &amp;&amp; keytool
-importkeystore -destkeystore
/etc/candlepin/certs/amqp/candlepin.jks -srckeystore
/tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt &amp;&amp; rm /tmp/keystore.p12&#39;
[DEBUG 2015-06-10 16:37:07 main]  Executing &#39;openssl pkcs12 -export
-name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey
/etc/pki/katello/private/java-client.key -out /tmp/keystore.p12
-passout file:/etc/pki/katello/keystore_password-file &amp;&amp; keytool
-importkeystore -destkeystore
/etc/candlepin/certs/amqp/candlepin.jks -srckeystore
/tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt &amp;&amp; rm /tmp/keystore.p12&#39;
[ WARN 2015-06-10 16:37:08 main] 
/Stage[main]/Certs::Candlepin/Exec[import client certificate into
Candlepin keystore]/returns: executed successfully

When I try it manually, I also get an error:

keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks
-storepass ********* -alias katello-default-ca

keytool error: java.lang.Exception: Alias &lt;katello-default-ca&gt; does
not exist

Anything I can try to fix this? Any idea on how this could have happened?

From the logs:

/var/log/gutterball/gutterball.log

2015-06-15 13:42:42,223 [main] WARN 
org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator
- HHH000208: org.hibernate.connection.C3P0ConnectionProvider has
been deprecated in favor of
org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider;
that provider will be used instead.
2015-06-15 13:42:42,304 [main] WARN 
com.mchange.v2.c3p0.management.ActiveManagementCoordinator - A
C3P0Registry mbean is already registered. This probably means that
an application using c3p0 was undeployed, but not all
PooledDataSources were closed prior to undeployment. This may lead
to resource leaks over time. Please take care to close all
PooledDataSources.
2015-06-15 13:42:43,022 [main] WARN 
org.hibernate.id.UUIDHexGenerator - HHH000409: Using
org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC
4122 compliant UUID values; consider using
org.hibernate.id.UUIDGenerator instead
2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
org.candlepin.gutterball.receiver.EventReceiver - Unable to
initialize connection to QPID.
java.lang.RuntimeException: javax.jms.JMSException:*Error creating
connection: Keystore was tampered with, or password was incorrect*
    at
org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:179)
~[EventReceiver$QpidConnectionThread.class:na]
javax.jms.JMSException: Error creating connection: Keystore was
tampered with, or password was incorrect
    at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
~[qpid-client-0.30.jar:0.30]
    at
org.candlepin.gutterball.receiver.EventReceiver.init(EventReceiver.java:82)
~[EventReceiver.class:na]
    at
org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:176)
~[EventReceiver$QpidConnectionThread.class:na]
Caused by: org.apache.qpid.AMQConnectionFailureException: Keystore
was tampered with, or password was incorrect
    at
org.apache.qpid.client.AMQConnection.&lt;init&gt;(AMQConnection.java:503)
~[qpid-client-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
~[qpid-client-0.30.jar:0.30]
    ... 2 common frames omitted
Caused by: org.apache.qpid.transport.TransportException: Error
creating SSL Context
    at
org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.&lt;init&gt;(SecurityLayerFactory.java:91)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.transport.network.security.SecurityLayerFactory.newInstance(SecurityLayerFactory.java:51)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.transport.Connection.connect(Connection.java:232)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
~[qpid-client-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647)
~[qpid-client-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnection.&lt;init&gt;(AMQConnection.java:429)
~[qpid-client-0.30.jar:0.30]
    ... 3 common frames omitted
Caused by: java.io.IOException: Keystore was tampered with, or
password was incorrect
    at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
~[na:1.7.0_79]
    at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
~[na:1.7.0_79]
    at java.security.KeyStore.load(KeyStore.java:1214) ~[na:1.7.0_79]
    at
org.apache.qpid.transport.network.security.ssl.SSLUtil.getInitializedKeyStore(SSLUtil.java:172)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.ssl.SSLContextFactory.buildContext(SSLContextFactory.java:92)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.ssl.SSLContextFactory.buildClientContext(SSLContextFactory.java:62)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.&lt;init&gt;(SecurityLayerFactory.java:78)
~[qpid-common-0.30.jar:0.30]
    ... 8 common frames omitted
Caused by: java.security.UnrecoverableKeyException: Password
verification failed
    at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
~[na:1.7.0_79]
    ... 14 common frames omitted
2015-06-16 09:59:49,416 [main] INFO 
org.candlepin.gutterball.servlet.GutterballContextListener -
Destroying gutterball context
2015-06-16 09:59:49,416 [main] INFO 
org.candlepin.gutterball.receiver.EventReceiver - Closing QPID
connection


/var/log/candlepin/error.log

2015-06-16 10:02:30,946 [=, org=] ERROR
org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
AMQPBusProvider:
javax.jms.JMSException: Error creating connection: Connection refused
    at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
~[qpid-client-0.30.jar:0.30]
    at
org.candlepin.guice.AMQPBusPubProvider.&lt;init&gt;(AMQPBusPubProvider.java:81)
~[AMQPBusPubProvider.class:na]
    at
org.candlepin.guice.AMQPBusPubProvider$$FastClassByGuice$$ef75a35a.newInstance(&lt;generated&gt;)
[guice-3.0.jar:na]
    at
com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
[guice-3.0.jar:na]
    at
com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
[guice-3.0.jar:na]
    at
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
[guice-3.0.jar:na]
    at
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
[guice-3.0.jar:na]
    at
com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
[guice-3.0.jar:na]
    at
com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
[guice-3.0.jar:na]
    at
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
[guice-3.0.jar:na]
    at com.google.inject.Scopes$1$1.get(Scopes.java:65)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
[guice-3.0.jar:na]
    at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
[guice-3.0.jar:na]
    at com.google.inject.Guice.createInjector(Guice.java:95)
[guice-3.0.jar:na]
    at
org.candlepin.guice.CandlepinContextListener.getInjector(CandlepinContextListener.java:146)
[CandlepinContextListener.class:na]
    at
org.candlepin.guice.CandlepinGuiceResteasyBootstrap.contextInitialized(CandlepinGuiceResteasyBootstrap.java:57)
[CandlepinGuiceResteasyBootstrap.class:na]
    at
org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:114)
[CandlepinContextListener.class:na]
    at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526) [catalina-6.0.24.jar:na]
    at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) [catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) [catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.StandardService.start(StandardService.java:516)
[catalina-6.0.24.jar:na]
    at
org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
[catalina-6.0.24.jar:na]
    at org.apache.catalina.startup.Catalina.start(Catalina.java:593)
[catalina-6.0.24.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_79]
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_79]
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_79]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
    at
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
[bootstrap-6.0.24.jar:na]
    at
org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
[bootstrap-6.0.24.jar:na]
Caused by: org.apache.qpid.AMQConnectionFailureException: Connection
refused
    at
org.apache.qpid.client.AMQConnection.&lt;init&gt;(AMQConnection.java:503)
~[qpid-client-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
~[qpid-client-0.30.jar:0.30]
    ... 46 common frames omitted
Caused by: org.apache.qpid.transport.TransportException: Error
connecting to broker
    at
org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:89)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.transport.Connection.connect(Connection.java:243)
~[qpid-common-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
~[qpid-client-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647)
~[qpid-client-0.30.jar:0.30]
    at
org.apache.qpid.client.AMQConnection.&lt;init&gt;(AMQConnection.java:429)
~[qpid-client-0.30.jar:0.30]
    ... 47 common frames omitted
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
~[na:1.7.0_79]
    at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
~[na:1.7.0_79]
    at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
~[na:1.7.0_79]
    at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
~[na:1.7.0_79]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[na:1.7.0_79]
    at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_79]
    at
org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:85)
~[qpid-common-0.30.jar:0.30]
    ... 51 common frames omitted
2015-06-16 10:02:34,595 [=, org=] WARN 
org.hibernate.id.UUIDHexGenerator - HHH000409: Using
org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC
4122 compliant UUID values; consider using
org.hibernate.id.UUIDGenerator instead
2015-06-16 10:02:36,405 [=, org=] ERROR
org.candlepin.guice.CandlepinGuiceResteasyBootstrap - Could not
create Guice injector.
com.google.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.RuntimeException:
javax.jms.JMSException: Error creating connection: Connection refuse

ehelms · June 16, 2015, 12:23pm

Can you give us some more background on your deployment of 2.1 (OS,
anything special about the deployment) and then what steps you took in
upgrading? The candlepin and gutterball services were working without issue
in your 2.1 setup?

···

On Tue, Jun 16, 2015 at 4:48 AM, Jorick Astrego wrote:

Hi,

A week ago I upgraded from 2.1 to 2.2.1 and although everything seemed ok,
I didn’t have time to check everything. Yesterday, I noticed the sync task
was not running and I am unable to run any task now.

Checking various logs, I can see gutterball and candelpin are giving
connection erros to QPID (longer log excerpt at the bottom):

2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
org.candlepin.gutterball.receiver.EventReceiver - Unable to initialize
connection to QPID.
java.lang.RuntimeException: javax.jms.JMSException:* Error creating
connection: Keystore was tampered with, or password was incorrect*

2015-06-16 10:02:30,946 [=, org=] ERROR
org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
AMQPBusProvider:
javax.jms.JMSException: Error creating connection: Connection refused

Then I checked the installer log and I can see errors with the
certificates:

[DEBUG 2015-06-10 16:37:05 main] Exec[import client certificate into
gutterball keystore] signal import if pair has not been
imported: Executing check ‘keytool -list -keystore
/etc/gutterball/certs/amqp//gutterball.jks -storepass
eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball’
[DEBUG 2015-06-10 16:37:05 main] Executing ‘keytool -list -keystore
/etc/gutterball/certs/amqp//gutterball.jks -storepass
eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball’
[DEBUG 2015-06-10 16:37:05 main]
/Stage[main]/Certs::Gutterball/Certs::Ssltools::Keytool::Import_keypair[import
client certificate into gutterball keystore]/Exec[[import client
certificate into gutterball keystore] signal import if pair has not been
imported]/unless: keytool error: java.io.IOException: Keystore was tampered
with, or password was incorrect
[DEBUG 2015-06-10 16:37:05 main] Exec[import client certificate into
gutterball keystore] signal import if pair has not been
imported: Executing ‘echo importing keypair’
[DEBUG 2015-06-10 16:37:05 main] Executing ‘echo importing keypair’

[DEBUG 2015-06-10 16:37:07 main] Execimport client certificate into
Candlepin keystore: Executing check ‘keytool -list
-keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca’
[DEBUG 2015-06-10 16:37:07 main] Executing ‘keytool -list -keystore
/etc/candlepin/certs/amqp/candlepin.jks -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca’
[DEBUG 2015-06-10 16:37:07 main]
/Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin
keystore]/unless: keytool error: java.lang.Exception: Alias
does not exist
[DEBUG 2015-06-10 16:37:07 main] Execimport client certificate into
Candlepin keystore: Executing ‘openssl pkcs12 -export
-name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey
/etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout
file:/etc/pki/katello/keystore_password-file && keytool -importkeystore
-destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore
/tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm /tmp/keystore.p12’
[DEBUG 2015-06-10 16:37:07 main] Executing ‘openssl pkcs12 -export -name
amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey
/etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout
file:/etc/pki/katello/keystore_password-file && keytool -importkeystore
-destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore
/tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm /tmp/keystore.p12’
[ WARN 2015-06-10 16:37:08 main]
/Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin
keystore]/returns: executed successfully

When I try it manually, I also get an error:

keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass
********* -alias katello-default-ca

keytool error: java.lang.Exception: Alias does not
exist

Anything I can try to fix this? Any idea on how this could have happened?

From the logs:

/var/log/gutterball/gutterball.log

2015-06-15 13:42:42,223 [main] WARN
org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator

HHH000208: org.hibernate.connection.C3P0ConnectionProvider has been
deprecated in favor of
org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider;
that provider will be used instead.
2015-06-15 13:42:42,304 [main] WARN
com.mchange.v2.c3p0.management.ActiveManagementCoordinator - A C3P0Registry
mbean is already registered. This probably means that an application using
c3p0 was undeployed, but not all PooledDataSources were closed prior to
undeployment. This may lead to resource leaks over time. Please take care
to close all PooledDataSources.
2015-06-15 13:42:43,022 [main] WARN org.hibernate.id.UUIDHexGenerator -
HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate
IETF RFC 4122 compliant UUID values; consider using
org.hibernate.id.UUIDGenerator instead
2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
org.candlepin.gutterball.receiver.EventReceiver - Unable to initialize
connection to QPID.
java.lang.RuntimeException: javax.jms.JMSException:* Error creating
connection: Keystore was tampered with, or password was incorrect*
at
org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:179)
~[EventReceiver$QpidConnectionThread.class:na]
javax.jms.JMSException: Error creating connection: Keystore was tampered
with, or password was incorrect
at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
~[qpid-client-0.30.jar:0.30]
at
org.candlepin.gutterball.receiver.EventReceiver.init(EventReceiver.java:82)
~[EventReceiver.class:na]
at
org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:176)
~[EventReceiver$QpidConnectionThread.class:na]
Caused by: org.apache.qpid.AMQConnectionFailureException: Keystore was
tampered with, or password was incorrect
at org.apache.qpid.client.AMQConnection.(AMQConnection.java:503)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
~[qpid-client-0.30.jar:0.30]
… 2 common frames omitted
Caused by: org.apache.qpid.transport.TransportException: Error creating
SSL Context
at
org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.(SecurityLayerFactory.java:91)
~[qpid-common-0.30.jar:0.30]
at
org.apache.qpid.transport.network.security.SecurityLayerFactory.newInstance(SecurityLayerFactory.java:51)
~[qpid-common-0.30.jar:0.30]
at org.apache.qpid.transport.Connection.connect(Connection.java:232)
~[qpid-common-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQConnection.(AMQConnection.java:429)
~[qpid-client-0.30.jar:0.30]
… 3 common frames omitted
Caused by: java.io.IOException: Keystore was tampered with, or password
was incorrect
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
~[na:1.7.0_79]
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
~[na:1.7.0_79]
at java.security.KeyStore.load(KeyStore.java:1214) ~[na:1.7.0_79]
at
org.apache.qpid.transport.network.security.ssl.SSLUtil.getInitializedKeyStore(SSLUtil.java:172)
~[qpid-common-0.30.jar:0.30]
at
org.apache.qpid.ssl.SSLContextFactory.buildContext(SSLContextFactory.java:92)
~[qpid-common-0.30.jar:0.30]
at
org.apache.qpid.ssl.SSLContextFactory.buildClientContext(SSLContextFactory.java:62)
~[qpid-common-0.30.jar:0.30]
at
org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.(SecurityLayerFactory.java:78)
~[qpid-common-0.30.jar:0.30]
… 8 common frames omitted
Caused by: java.security.UnrecoverableKeyException: Password verification
failed
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
~[na:1.7.0_79]
… 14 common frames omitted
2015-06-16 09:59:49,416 [main] INFO
org.candlepin.gutterball.servlet.GutterballContextListener - Destroying
gutterball context
2015-06-16 09:59:49,416 [main] INFO
org.candlepin.gutterball.receiver.EventReceiver - Closing QPID connection

/var/log/candlepin/error.log

2015-06-16 10:02:30,946 [=, org=] ERROR
org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
AMQPBusProvider:
javax.jms.JMSException: Error creating connection: Connection refused
at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
~[qpid-client-0.30.jar:0.30]
at
org.candlepin.guice.AMQPBusPubProvider.(AMQPBusPubProvider.java:81)
~[AMQPBusPubProvider.class:na]
at
org.candlepin.guice.AMQPBusPubProvider$$FastClassByGuice$$ef75a35a.newInstance()
[guice-3.0.jar:na]
at
com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
[guice-3.0.jar:na]
at
com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
[guice-3.0.jar:na]
at
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
[guice-3.0.jar:na]
at
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
[guice-3.0.jar:na]
at
com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
[guice-3.0.jar:na]
at
com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
[guice-3.0.jar:na]
at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
[guice-3.0.jar:na]
at
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
[guice-3.0.jar:na]
at com.google.inject.Scopes$1$1.get(Scopes.java:65) [guice-3.0.jar:na]
at
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
[guice-3.0.jar:na]
at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
[guice-3.0.jar:na]
at com.google.inject.Guice.createInjector(Guice.java:95)
[guice-3.0.jar:na]
at
org.candlepin.guice.CandlepinContextListener.getInjector(CandlepinContextListener.java:146)
[CandlepinContextListener.class:na]
at
org.candlepin.guice.CandlepinGuiceResteasyBootstrap.contextInitialized(CandlepinGuiceResteasyBootstrap.java:57)
[CandlepinGuiceResteasyBootstrap.class:na]
at
org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:114)
[CandlepinContextListener.class:na]
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)
[catalina-6.0.24.jar:na]
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
[catalina-6.0.24.jar:na]
at org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.StandardService.start(StandardService.java:516)
[catalina-6.0.24.jar:na]
at
org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
[catalina-6.0.24.jar:na]
at org.apache.catalina.startup.Catalina.start(Catalina.java:593)
[catalina-6.0.24.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_79]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_79]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
[bootstrap-6.0.24.jar:na]
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
[bootstrap-6.0.24.jar:na]
Caused by: org.apache.qpid.AMQConnectionFailureException: Connection
refused
at org.apache.qpid.client.AMQConnection.(AMQConnection.java:503)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
~[qpid-client-0.30.jar:0.30]
… 46 common frames omitted
Caused by: org.apache.qpid.transport.TransportException: Error connecting
to broker
at
org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:89)
~[qpid-common-0.30.jar:0.30]
at org.apache.qpid.transport.Connection.connect(Connection.java:243)
~[qpid-common-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQConnection.(AMQConnection.java:429)
~[qpid-client-0.30.jar:0.30]
… 47 common frames omitted
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_79]
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
~[na:1.7.0_79]
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
~[na:1.7.0_79]
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
~[na:1.7.0_79]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[na:1.7.0_79]
at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_79]
at
org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:85)
~[qpid-common-0.30.jar:0.30]
… 51 common frames omitted
2015-06-16 10:02:34,595 [=, org=] WARN org.hibernate.id.UUIDHexGenerator

HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not
generate IETF RFC 4122 compliant UUID values; consider using
org.hibernate.id.UUIDGenerator instead
2015-06-16 10:02:36,405 [=, org=] ER

Cliff_Perry · June 16, 2015, 9:50am

> Hi,
>
> A week ago I upgraded from 2.1 to 2.2.1 and although everything seemed
> ok, I didn't have time to check everything. Yesterday, I noticed the
> sync task was not running and I am unable to run any task now.
>
> Checking various logs, I can see gutterball and candelpin are giving
> connection erros to QPID (longer log excerpt at the bottom):
>
> 2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
> org.candlepin.gutterball.receiver.EventReceiver - Unable to
> initialize connection to QPID.
> java.lang.RuntimeException: javax.jms.JMSException:Error creating
> connection: Keystore was tampered with, or password was incorrect
>
> 2015-06-16 10:02:30,946 [=, org=] ERROR
> org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
> AMQPBusProvider:
> javax.jms.JMSException: Error creating connection: Connection refused
>
> Then I checked the installer log and I can see errors with the certificates:
>
>
>
> [DEBUG 2015-06-10 16:37:05 main] Exec[import client certificate
> into gutterball keystore] signal import if pair has not been
> imported: Executing check 'keytool -list -keystore
> /etc/gutterball/certs/amqp//gutterball.jks -storepass
> eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball'
> [DEBUG 2015-06-10 16:37:05 main] Executing 'keytool -list -keystore
> /etc/gutterball/certs/amqp//gutterball.jks -storepass
> eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball'
> [DEBUG 2015-06-10 16:37:05 main]
> /Stage[main]/Certs::Gutterball/Certs::Ssltools::Keytool::Import_keypair[import
> client certificate into gutterball keystore]/Exec[[import client
> certificate into gutterball keystore] signal import if pair has not
> been imported]/unless: keytool error: java.io.IOException: Keystore
> was tampered with, or password was incorrect
> [DEBUG 2015-06-10 16:37:05 main] Exec[import client certificate
> into gutterball keystore] signal import if pair has not been
> imported: Executing 'echo importing keypair'
> [DEBUG 2015-06-10 16:37:05 main] Executing 'echo importing keypair'
>
>
> [DEBUG 2015-06-10 16:37:07 main] Execimport client certificate
> into Candlepin keystore: Executing check 'keytool
> -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass
> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca'
> [DEBUG 2015-06-10 16:37:07 main] Executing 'keytool -list -keystore
> /etc/candlepin/certs/amqp/candlepin.jks -storepass
> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca'
> [DEBUG 2015-06-10 16:37:07 main]
> /Stage[main]/Certs::Candlepin/Exec[import client certificate into
> Candlepin keystore]/unless: keytool error: java.lang.Exception:
> Alias <katello-default-ca> does not exist
> [DEBUG 2015-06-10 16:37:07 main] Execimport client certificate
> into Candlepin keystore: Executing 'openssl pkcs12
> -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt
> -inkey /etc/pki/katello/private/java-client.key -out
> /tmp/keystore.p12 -passout
> file:/etc/pki/katello/keystore_password-file && keytool
> -importkeystore -destkeystore
> /etc/candlepin/certs/amqp/candlepin.jks -srckeystore
> /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass
> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm /tmp/keystore.p12'
> [DEBUG 2015-06-10 16:37:07 main] Executing 'openssl pkcs12 -export
> -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey
> /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12
> -passout file:/etc/pki/katello/keystore_password-file && keytool
> -importkeystore -destkeystore
> /etc/candlepin/certs/amqp/candlepin.jks -srckeystore
> /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass
> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm /tmp/keystore.p12'
> [ WARN 2015-06-10 16:37:08 main]
> /Stage[main]/Certs::Candlepin/Exec[import client certificate into
> Candlepin keystore]/returns: executed successfully
>
>
> When I try it manually, I also get an error:
>
> keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks
> -storepass ********* -alias katello-default-ca
>
> keytool error: java.lang.Exception: Alias <katello-default-ca> does
> not exist
>
>
> Anything I can try to fix this? Any idea on how this could have happened?
>
>
> From the logs:
>
> /var/log/gutterball/gutterball.log
>
> 2015-06-15 13:42:42,223 [main] WARN
> org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator
> - HHH000208: org.hibernate.connection.C3P0ConnectionProvider has
> been deprecated in favor of
> org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider;
> that provider will be used instead.
> 2015-06-15 13:42:42,304 [main] WARN
> com.mchange.v2.c3p0.management.ActiveManagementCoordinator - A
> C3P0Registry mbean is already registered. This probably means that
> an application using c3p0 was undeployed, but not all
> PooledDataSources were closed prior to undeployment. This may lead
> to resource leaks over time. Please take care to close all
> PooledDataSources.
> 2015-06-15 13:42:43,022 [main] WARN
> org.hibernate.id.UUIDHexGenerator - HHH000409: Using
> org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC
> 4122 compliant UUID values; consider using
> org.hibernate.id.UUIDGenerator instead
> 2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
> org.candlepin.gutterball.receiver.EventReceiver - Unable to
> initialize connection to QPID.
> java.lang.RuntimeException: javax.jms.JMSException:Error creating
> connection: Keystore was tampered with, or password was incorrect
> at
> org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:179)
> ~[EventReceiver$QpidConnectionThread.class:na]
> javax.jms.JMSException: Error creating connection: Keystore was
> tampered with, or password was incorrect
> at
> org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.candlepin.gutterball.receiver.EventReceiver.init(EventReceiver.java:82)
> ~[EventReceiver.class:na]
> at
> org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:176)
> ~[EventReceiver$QpidConnectionThread.class:na]
> Caused by: org.apache.qpid.AMQConnectionFailureException: Keystore
> was tampered with, or password was incorrect
> at
> org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:503)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
> ~[qpid-client-0.30.jar:0.30]
> … 2 common frames omitted
> Caused by: org.apache.qpid.transport.TransportException: Error
> creating SSL Context
> at
> org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.<init>(SecurityLayerFactory.java:91)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.transport.network.security.SecurityLayerFactory.newInstance(SecurityLayerFactory.java:51)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.transport.Connection.connect(Connection.java:232)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:429)
> ~[qpid-client-0.30.jar:0.30]
> … 3 common frames omitted
> Caused by: java.io.IOException: Keystore was tampered with, or
> password was incorrect
> at
> sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
> ~[na:1.7.0_79]
> at
> sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
> ~[na:1.7.0_79]
> at java.security.KeyStore.load(KeyStore.java:1214) ~[na:1.7.0_79]
> at
> org.apache.qpid.transport.network.security.ssl.SSLUtil.getInitializedKeyStore(SSLUtil.java:172)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.ssl.SSLContextFactory.buildContext(SSLContextFactory.java:92)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.ssl.SSLContextFactory.buildClientContext(SSLContextFactory.java:62)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.<init>(SecurityLayerFactory.java:78)
> ~[qpid-common-0.30.jar:0.30]
> … 8 common frames omitted
> Caused by: java.security.UnrecoverableKeyException: Password
> verification failed
> at
> sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
> ~[na:1.7.0_79]
> … 14 common frames omitted
> 2015-06-16 09:59:49,416 [main] INFO
> org.candlepin.gutterball.servlet.GutterballContextListener -
> Destroying gutterball context
> 2015-06-16 09:59:49,416 [main] INFO
> org.candlepin.gutterball.receiver.EventReceiver - Closing QPID
> connection
>
>
> /var/log/candlepin/error.log
>
> 2015-06-16 10:02:30,946 [=, org=] ERROR
> org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
> AMQPBusProvider:
> javax.jms.JMSException: Error creating connection: Connection refused
> at
> org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.candlepin.guice.AMQPBusPubProvider.<init>(AMQPBusPubProvider.java:81)
> ~[AMQPBusPubProvider.class:na]
> at
> org.candlepin.guice.AMQPBusPubProvider$$FastClassByGuice$$ef75a35a.newInstance(<generated>)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> [guice-3.0.jar:na]
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
> [guice-3.0.jar:na]
> at
> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
> [guice-3.0.jar:na]
> at com.google.inject.Guice.createInjector(Guice.java:95)
> [guice-3.0.jar:na]
> at
> org.candlepin.guice.CandlepinContextListener.getInjector(CandlepinContextListener.java:146)
> [CandlepinContextListener.class:na]
> at
> org.candlepin.guice.CandlepinGuiceResteasyBootstrap.contextInitialized(CandlepinGuiceResteasyBootstrap.java:57)
> [CandlepinGuiceResteasyBootstrap.class:na]
> at
> org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:114)
> [CandlepinContextListener.class:na]
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526) [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.StandardService.start(StandardService.java:516)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> [catalina-6.0.24.jar:na]
> at
> org.apache.catalina.startup.Catalina.start(Catalina.java:593)
> [catalina-6.0.24.jar:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_79]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_79]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_79]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
> at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> [bootstrap-6.0.24.jar:na]
> at
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> [bootstrap-6.0.24.jar:na]
> Caused by: org.apache.qpid.AMQConnectionFailureException: Connection
> refused
> at
> org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:503)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
> ~[qpid-client-0.30.jar:0.30]
> … 46 common frames omitted
> Caused by: org.apache.qpid.transport.TransportException: Error
> connecting to broker
> at
> org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:89)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.transport.Connection.connect(Connection.java:243)
> ~[qpid-common-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:429)
> ~[qpid-client-0.30.jar:0.30]
> … 47 common frames omitted
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> ~[na:1.7.0_79]
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> ~[na:1.7.0_79]
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> ~[na:1.7.0_79]
> at
> java.net.Abstrac

···

On 16/06/15 09:48, Jorick Astrego wrote:

Jorick_Astrego · June 16, 2015, 1:16pm

> Can you give us some more background on your deployment of 2.1 (OS,
> anything special about the deployment) and then what steps you took in
> upgrading? The candlepin and gutterball services were working without
> issue in your 2.1 setup?
>

Sure. I had 2.1 installed on Centos 6.6 for a couple of months now.
Nothing special, just a normal installation. I followed the steps on
"http://www.katello.org/docs/upgrade/index.html". I also recently ran
the Manual steps to try to fix things without any effect.

After upgrading the rpm's, it was the end of the day and I wasn't around
to run the "katello-installer --upgrade" until a couple days later.

First the upgrade failed because there were a couple of locked tasks
that prevented the upgrade. One "Actions::Katello::System::Create" and a
"Actions::Katello::ContentView::Publish". Somehow these were unable to
resume, so I had to force unlock to continue.

When I check this first katello-installer.log, there are the exact same
"katello-default-ca" and keytool errors as I have now.

I have now checked older logs and the gutterball errors appear in all of
them except for the couple of days after installation :-&

So that appears to be unrelated to the upgrade and I never noticed
anything not working because of it.

The candlepin errors did appear after the upgrade though and I guess
this is the cause of my katello install not running any tasks anymore.

The only error in candlepin before the upgade is this one:

2015-05-07 10:37:51,312 [req=7ceec278-e898-430d-83ec-0326d4b18b98,
org=Netbulae] ERROR
org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper -
Runtime Error null at
org.candlepin.model.AbstractHibernateCurator.delete:326
java.lang.NullPointerException: null
        at
org.candlepin.model.AbstractHibernateCurator.delete(AbstractHibernateCurator.java:326)
~[AbstractHibernateCurator.class:na]
        at
com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58)
~[guice-persist-3.0.jar:na]
        at
org.candlepin.resource.EnvironmentResource.demoteContent(EnvironmentResource.java:228)
~[EnvironmentResource.class:na]
        at
org.candlepin.guice.TransactionalInvoker.invoke(TransactionalInvoker.java:34)
~[TransactionalInvoker.class:na]
        at
com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66)
~[guice-persist-3.0.jar:na]
        at
org.candlepin.guice.CandlepinResourceTxnInterceptor.invoke(CandlepinResourceTxnInterceptor.java:33)
~[CandlepinResourceTxnInterceptor.class:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) ~[na:1.7.0_79]
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_79]
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606)
~[na:1.7.0_79]
        at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
~[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269)
~[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227)
~[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:216)
~[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
[resteasy-jaxrs-2.3.7.Final.jar:na]
        at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
[resteasy-jaxrs-2.3.7.Final.jar:na]

Kind regards,

Jorick Astrego

···

On 06/16/2015 02:23 PM, Eric D Helms wrote:

On Tue, Jun 16, 2015 at 4:48 AM, Jorick Astrego <j.astrego@netbulae.eu > mailto:j.astrego@netbulae.eu> wrote:

Hi,

A week ago I upgraded from 2.1 to 2.2.1 and although everything
seemed ok, I didn't have time to check everything. Yesterday, I
noticed the sync task was not running and I am unable to run any
task now.

Checking various logs, I can see gutterball and candelpin are
giving connection erros to QPID (longer log excerpt at the bottom):

    2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
    org.candlepin.gutterball.receiver.EventReceiver - Unable to
    initialize connection to QPID.
    java.lang.RuntimeException: javax.jms.JMSException:*Error
    creating connection: Keystore was tampered with, or password
    was incorrect*

    2015-06-16 10:02:30,946 [=, org=] ERROR
    org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
    AMQPBusProvider:
    javax.jms.JMSException: Error creating connection: Connection
    refused

Then I checked the installer log and I can see errors with the
certificates:



    [DEBUG 2015-06-10 16:37:05 main]  Exec[[import client
    certificate into gutterball keystore] signal import if pair
    has not been imported](provider=posix): Executing check
    'keytool -list -keystore
    /etc/gutterball/certs/amqp//gutterball.jks -storepass
    eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball'
    [DEBUG 2015-06-10 16:37:05 main]  Executing 'keytool -list
    -keystore /etc/gutterball/certs/amqp//gutterball.jks
    -storepass eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball'
    [DEBUG 2015-06-10 16:37:05 main] 
    /Stage[main]/Certs::Gutterball/Certs::Ssltools::Keytool::Import_keypair[import
    client certificate into gutterball keystore]/Exec[[import
    client certificate into gutterball keystore] signal import if
    pair has not been imported]/unless: keytool error:
    java.io.IOException: Keystore was tampered with, or password
    was incorrect
    [DEBUG 2015-06-10 16:37:05 main]  Exec[[import client
    certificate into gutterball keystore] signal import if pair
    has not been imported](provider=posix): Executing 'echo
    importing keypair'
    [DEBUG 2015-06-10 16:37:05 main]  Executing 'echo importing
    keypair'


    [DEBUG 2015-06-10 16:37:07 main]  Exec[import client
    certificate into Candlepin keystore](provider=posix):
    Executing check 'keytool -list -keystore
    /etc/candlepin/certs/amqp/candlepin.jks -storepass
    gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca'
    [DEBUG 2015-06-10 16:37:07 main]  Executing 'keytool -list
    -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass
    gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca'
    [DEBUG 2015-06-10 16:37:07 main] 
    /Stage[main]/Certs::Candlepin/Exec[import client certificate
    into Candlepin keystore]/unless: keytool error:
    java.lang.Exception: Alias <katello-default-ca> does not exist
    [DEBUG 2015-06-10 16:37:07 main]  Exec[import client
    certificate into Candlepin keystore](provider=posix):
    Executing 'openssl pkcs12 -export -name amqp-client -in
    /etc/pki/katello/certs/java-client.crt -inkey
    /etc/pki/katello/private/java-client.key -out
    /tmp/keystore.p12 -passout
    file:/etc/pki/katello/keystore_password-file && keytool
    -importkeystore -destkeystore
    /etc/candlepin/certs/amqp/candlepin.jks -srckeystore
    /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client
    -storepass gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
    gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm
    /tmp/keystore.p12'
    [DEBUG 2015-06-10 16:37:07 main]  Executing 'openssl pkcs12
    -export -name amqp-client -in
    /etc/pki/katello/certs/java-client.crt -inkey
    /etc/pki/katello/private/java-client.key -out
    /tmp/keystore.p12 -passout
    file:/etc/pki/katello/keystore_password-file && keytool
    -importkeystore -destkeystore
    /etc/candlepin/certs/amqp/candlepin.jks -srckeystore
    /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client
    -storepass gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass
    gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm
    /tmp/keystore.p12'
    [ WARN 2015-06-10 16:37:08 main] 
    /Stage[main]/Certs::Candlepin/Exec[import client certificate
    into Candlepin keystore]/returns: executed successfully


When I try it manually, I also get an error:

    keytool -list -keystore
    /etc/candlepin/certs/amqp/candlepin.jks -storepass *********
    -alias katello-default-ca

    keytool error: java.lang.Exception: Alias <katello-default-ca>
    does not exist


Anything I can try to fix this? Any idea on how this could have
happened?


From the logs:

    /var/log/gutterball/gutterball.log

    2015-06-15 13:42:42,223 [main] WARN 
    org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator
    - HHH000208: org.hibernate.connection.C3P0ConnectionProvider
    has been deprecated in favor of
    org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider;
    that provider will be used instead.
    2015-06-15 13:42:42,304 [main] WARN 
    com.mchange.v2.c3p0.management.ActiveManagementCoordinator - A
    C3P0Registry mbean is already registered. This probably means
    that an application using c3p0 was undeployed, but not all
    PooledDataSources were closed prior to undeployment. This may
    lead to resource leaks over time. Please take care to close
    all PooledDataSources.
    2015-06-15 13:42:43,022 [main] WARN 
    org.hibernate.id.UUIDHexGenerator - HHH000409: Using
    org.hibernate.id.UUIDHexGenerator which does not generate IETF
    RFC 4122 compliant UUID values; consider using
    org.hibernate.id.UUIDGenerator instead
    2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR
    org.candlepin.gutterball.receiver.EventReceiver - Unable to
    initialize connection to QPID.
    java.lang.RuntimeException: javax.jms.JMSException:*Error
    creating connection: Keystore was tampered with, or password
    was incorrect*
        at
    org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:179)
    ~[EventReceiver$QpidConnectionThread.class:na]
    javax.jms.JMSException: Error creating connection: Keystore
    was tampered with, or password was incorrect
        at
    org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
    ~[qpid-client-0.30.jar:0.30]
        at
    org.candlepin.gutterball.receiver.EventReceiver.init(EventReceiver.java:82)
    ~[EventReceiver.class:na]
        at
    org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:176)
    ~[EventReceiver$QpidConnectionThread.class:na]
    Caused by: org.apache.qpid.AMQConnectionFailureException:
    Keystore was tampered with, or password was incorrect
        at
    org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:503)
    ~[qpid-client-0.30.jar:0.30]
        at
    org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
    ~[qpid-client-0.30.jar:0.30]
        ... 2 common frames omitted
    Caused by: org.apache.qpid.transport.TransportException: Error
    creating SSL Context
        at
    org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.<init>(SecurityLayerFactory.java:91)
    ~[qpid-common-0.30.jar:0.30]
        at
    org.apache.qpid.transport.network.security.SecurityLayerFactory.newInstance(SecurityLayerFactory.java:51)
    ~[qpid-common-0.30.jar:0.30]
        at
    org.apache.qpid.transport.Connection.connect(Connection.java:232)
    ~[qpid-common-0.30.jar:0.30]
        at
    org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
    ~[qpid-client-0.30.jar:0.30]
        at
    org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647)
    ~[qpid-client-0.30.jar:0.30]
        at
    org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:429)
    ~[qpid-client-0.30.jar:0.30]
        ... 3 common frames omitted
    Caused by: java.io.IOException: Keystore was tampered with, or
    password was incorrect
        at
    sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
    ~[na:1.7.0_79]
        at
    sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
    ~[na:1.7.0_79]
        at java.security.KeyStore.load(KeyStore.java:1214)
    ~[na:1.7.0_79]
        at
    org.apache.qpid.transport.network.security.ssl.SSLUtil.getInitializedKeyStore(SSLUtil.java:172)
    ~[qpid-common-0.30.jar:0.30]
        at
    org.apache.qpid.ssl.SSLContextFactory.buildContext(SSLContextFactory.java:92)
    ~[qpid-common-0.30.jar:0.30]
        at
    org.apache.qpid.ssl.SSLContextFactory.buildClientContext(SSLContextFactory.java:62)
    ~[qpid-common-0.30.jar:0.30]
        at
    org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.<init>(SecurityLayerFactory.java:78)
    ~[qpid-common-0.30.jar:0.30]
        ... 8 common frames omitted
    Caused by: java.security.UnrecoverableKeyException: Password
    verification failed
        at
    sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
    ~[na:1.7.0_79]
        ... 14 common frames omitted
    2015-06-16 09:59:49,416 [main] INFO 
    org.candlepin.gutterball.servlet.GutterballContextListener -
    Destroying gutterball context
    2015-06-16 09:59:49,416 [main] INFO 
    org.candlepin.gutterball.receiver.EventReceiver - Closing QPID
    connection


    /var/log/candlepin/error.log

    2015-06-16 10:02:30,946 [=, org=] ERROR
    org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate
    AMQPBusProvider:
    javax.jms.JMSException: Error creating connection: Connection
    refused
        at
    org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128)
    ~[qpid-client-0.30.jar:0.30]
        at
    org.candlepin.guice.AMQPBusPubProvider.<init>(AMQPBusPubProvider.java:81)
    ~[AMQPBusPubProvider.class:na]
        at
    org.candlepin.guice.AMQPBusPubProvider$$FastClassByGuice$$ef75a35a.newInstance(<generated>)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    [guice-3.0.jar:na]
        at com.google.inject.Scopes$1$1.get(Scopes.java:65)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
    [guice-3.0.jar:na]
        at
    com.google.inject.internal.InjectorImpl.callInContext(I

Jorick_Astrego · June 18, 2015, 10:49am

Hi,

Got things going again so I don't need to recover from backups.

What I did:

I merged some parameters from /etc/qpid/qpidd.conf.rpmnew in

/etc/qpid/qpidd.conf

ran: katello-installer --certs-update-all
ran: katello-installer

Resumed all paused sync tasks and cancelled them from the

"sync_management" view.

I do not know if all the steps were required but it fixed it at least

Regards,

Jorick

···

On 06/16/2015 03:16 PM, Jorick Astrego wrote: > > > On 06/16/2015 02:23 PM, Eric D Helms wrote: >> Can you give us some more background on your deployment of 2.1 (OS, >> anything special about the deployment) and then what steps you took >> in upgrading? The candlepin and gutterball services were working >> without issue in your 2.1 setup? >> > > Sure. I had 2.1 installed on Centos 6.6 for a couple of months now. > Nothing special, just a normal installation. I followed the steps on > "http://www.katello.org/docs/upgrade/index.html". I also recently ran > the Manual steps to try to fix things without any effect. > > After upgrading the rpm's, it was the end of the day and I wasn't > around to run the "katello-installer --upgrade" until a couple days later. > > First the upgrade failed because there were a couple of locked tasks > that prevented the upgrade. One "Actions::Katello::System::Create" and > a "Actions::Katello::ContentView::Publish". Somehow these were unable > to resume, so I had to force unlock to continue. > > When I check this first katello-installer.log, there are the exact > same "katello-default-ca" and keytool errors as I have now. > > I have now checked older logs and the gutterball errors appear in all > of them except for the couple of days after installation :-& > > So that appears to be unrelated to the upgrade and I never noticed > anything not working because of it. > > The candlepin errors did appear after the upgrade though and I guess > this is the cause of my katello install not running any tasks anymore. > > The only error in candlepin before the upgade is this one: > > 2015-05-07 10:37:51,312 [req=7ceec278-e898-430d-83ec-0326d4b18b98, > org=Netbulae] ERROR > org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - > Runtime Error null at > org.candlepin.model.AbstractHibernateCurator.delete:326 > java.lang.NullPointerException: null > at > org.candlepin.model.AbstractHibernateCurator.delete(AbstractHibernateCurator.java:326) > ~[AbstractHibernateCurator.class:na] > at > com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) > ~[guice-persist-3.0.jar:na] > at > org.candlepin.resource.EnvironmentResource.demoteContent(EnvironmentResource.java:228) > ~[EnvironmentResource.class:na] > at > org.candlepin.guice.TransactionalInvoker.invoke(TransactionalInvoker.java:34) > ~[TransactionalInvoker.class:na] > at > com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) > ~[guice-persist-3.0.jar:na] > at > org.candlepin.guice.CandlepinResourceTxnInterceptor.invoke(CandlepinResourceTxnInterceptor.java:33) > ~[CandlepinResourceTxnInterceptor.class:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[na:1.7.0_79] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > ~[na:1.7.0_79] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.7.0_79] > at java.lang.reflect.Method.invoke(Method.java:606) > ~[na:1.7.0_79] > at > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) > ~[resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269) > ~[resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227) > ~[resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:216) > ~[resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542) > [resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524) > [resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126) > [resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) > [resteasy-jaxrs-2.3.7.Final.jar:na] > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) > [resteasy-jaxrs-2.3.7.Final.jar:na] > > Kind regards, > > Jorick Astrego > >> >> On Tue, Jun 16, 2015 at 4:48 AM, Jorick Astrego >> <j.astrego@netbulae.eu > wrote: >> >> Hi, >> >> A week ago I upgraded from 2.1 to 2.2.1 and although everything >> seemed ok, I didn't have time to check everything. Yesterday, I >> noticed the sync task was not running and I am unable to run any >> task now. >> >> Checking various logs, I can see gutterball and candelpin are >> giving connection erros to QPID (longer log excerpt at the bottom): >> >> 2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR >> org.candlepin.gutterball.receiver.EventReceiver - Unable to >> initialize connection to QPID. >> java.lang.RuntimeException: javax.jms.JMSException:*Error >> creating connection: Keystore was tampered with, or password >> was incorrect* >> >> 2015-06-16 10:02:30,946 [=, org=] ERROR >> org.candlepin.guice.AMQPBusPubProvider - Unable to >> instantiate AMQPBusProvider: >> javax.jms.JMSException: Error creating connection: Connection >> refused >> >> Then I checked the installer log and I can see errors with the >> certificates: >> >> >> >> [DEBUG 2015-06-10 16:37:05 main] Exec[[import client >> certificate into gutterball keystore] signal import if pair >> has not been imported](provider=posix): Executing check >> 'keytool -list -keystore >> /etc/gutterball/certs/amqp//gutterball.jks -storepass >> eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball' >> [DEBUG 2015-06-10 16:37:05 main] Executing 'keytool -list >> -keystore /etc/gutterball/certs/amqp//gutterball.jks >> -storepass eS8hZrxxf6tcZfHEHkN2hYY2BXN9NnXc -alias gutterball' >> [DEBUG 2015-06-10 16:37:05 main] >> /Stage[main]/Certs::Gutterball/Certs::Ssltools::Keytool::Import_keypair[import >> client certificate into gutterball keystore]/Exec[[import >> client certificate into gutterball keystore] signal import if >> pair has not been imported]/unless: keytool error: >> java.io.IOException: Keystore was tampered with, or password >> was incorrect >> [DEBUG 2015-06-10 16:37:05 main] Exec[[import client >> certificate into gutterball keystore] signal import if pair >> has not been imported](provider=posix): Executing 'echo >> importing keypair' >> [DEBUG 2015-06-10 16:37:05 main] Executing 'echo importing >> keypair' >> >> >> [DEBUG 2015-06-10 16:37:07 main] Exec[import client >> certificate into Candlepin keystore](provider=posix): >> Executing check 'keytool -list -keystore >> /etc/candlepin/certs/amqp/candlepin.jks -storepass >> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca' >> [DEBUG 2015-06-10 16:37:07 main] Executing 'keytool -list >> -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass >> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -alias katello-default-ca' >> [DEBUG 2015-06-10 16:37:07 main] >> /Stage[main]/Certs::Candlepin/Exec[import client certificate >> into Candlepin keystore]/unless: keytool error: >> java.lang.Exception: Alias does not exist >> [DEBUG 2015-06-10 16:37:07 main] Exec[import client >> certificate into Candlepin keystore](provider=posix): >> Executing 'openssl pkcs12 -export -name amqp-client -in >> /etc/pki/katello/certs/java-client.crt -inkey >> /etc/pki/katello/private/java-client.key -out >> /tmp/keystore.p12 -passout >> file:/etc/pki/katello/keystore_password-file && keytool >> -importkeystore -destkeystore >> /etc/candlepin/certs/amqp/candlepin.jks -srckeystore >> /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client >> -storepass gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass >> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm >> /tmp/keystore.p12' >> [DEBUG 2015-06-10 16:37:07 main] Executing 'openssl pkcs12 >> -export -name amqp-client -in >> /etc/pki/katello/certs/java-client.crt -inkey >> /etc/pki/katello/private/java-client.key -out >> /tmp/keystore.p12 -passout >> file:/etc/pki/katello/keystore_password-file && keytool >> -importkeystore -destkeystore >> /etc/candlepin/certs/amqp/candlepin.jks -srckeystore >> /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client >> -storepass gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -srcstorepass >> gttJyejWhbMVgcBkVwqsX6g9nSAb76WN -noprompt && rm >> /tmp/keystore.p12' >> [ WARN 2015-06-10 16:37:08 main] >> /Stage[main]/Certs::Candlepin/Exec[import client certificate >> into Candlepin keystore]/returns: executed successfully >> >> >> When I try it manually, I also get an error: >> >> keytool -list -keystore >> /etc/candlepin/certs/amqp/candlepin.jks -storepass ********* >> -alias katello-default-ca >> >> keytool error: java.lang.Exception: Alias >> does not exist >> >> >> Anything I can try to fix this? Any idea on how this could have >> happened? >> >> >> From the logs: >> >> /var/log/gutterball/gutterball.log >> >> 2015-06-15 13:42:42,223 [main] WARN >> org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator >> - HHH000208: org.hibernate.connection.C3P0ConnectionProvider >> has been deprecated in favor of >> org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider; >> that provider will be used instead. >> 2015-06-15 13:42:42,304 [main] WARN >> com.mchange.v2.c3p0.management.ActiveManagementCoordinator - >> A C3P0Registry mbean is already registered. This probably >> means that an application using c3p0 was undeployed, but not >> all PooledDataSources were closed prior to undeployment. This >> may lead to resource leaks over time. Please take care to >> close all PooledDataSources. >> 2015-06-15 13:42:43,022 [main] WARN >> org.hibernate.id.UUIDHexGenerator - HHH000409: Using >> org.hibernate.id.UUIDHexGenerator which does not generate >> IETF RFC 4122 compliant UUID values; consider using >> org.hibernate.id.UUIDGenerator instead >> 2015-06-15 13:45:12,289 [gutterball-qpid-connect] ERROR >> org.candlepin.gutterball.receiver.EventReceiver - Unable to >> initialize connection to QPID. >> java.lang.RuntimeException: javax.jms.JMSException:*Error >> creating connection: Keystore was tampered with, or password >> was incorrect* >> at >> org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:179) >> ~[EventReceiver$QpidConnectionThread.class:na] >> javax.jms.JMSException: Error creating connection: Keystore >> was tampered with, or password was incorrect >> at >> org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.candlepin.gutterball.receiver.EventReceiver.init(EventReceiver.java:82) >> ~[EventReceiver.class:na] >> at >> org.candlepin.gutterball.receiver.EventReceiver$QpidConnectionThread.run(EventReceiver.java:176) >> ~[EventReceiver$QpidConnectionThread.class:na] >> Caused by: org.apache.qpid.AMQConnectionFailureException: >> Keystore was tampered with, or password was incorrect >> at >> org.apache.qpid.client.AMQConnection.(AMQConnection.java:503) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124) >> ~[qpid-client-0.30.jar:0.30] >> ... 2 common frames omitted >> Caused by: org.apache.qpid.transport.TransportException: >> Error creating SSL Context >> at >> org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.(SecurityLayerFactory.java:91) >> ~[qpid-common-0.30.jar:0.30] >> at >> org.apache.qpid.transport.network.security.SecurityLayerFactory.newInstance(SecurityLayerFactory.java:51) >> ~[qpid-common-0.30.jar:0.30] >> at >> org.apache.qpid.transport.Connection.connect(Connection.java:232) >> ~[qpid-common-0.30.jar:0.30] >> at >> org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:647) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.apache.qpid.client.AMQConnection.(AMQConnection.java:429) >> ~[qpid-client-0.30.jar:0.30] >> ... 3 common frames omitted >> Caused by: java.io.IOException: Keystore was tampered with, >> or password was incorrect >> at >> sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) >> ~[na:1.7.0_79] >> at >> sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) >> ~[na:1.7.0_79] >> at java.security.KeyStore.load(KeyStore.java:1214) >> ~[na:1.7.0_79] >> at >> org.apache.qpid.transport.network.security.ssl.SSLUtil.getInitializedKeyStore(SSLUtil.java:172) >> ~[qpid-common-0.30.jar:0.30] >> at >> org.apache.qpid.ssl.SSLContextFactory.buildContext(SSLContextFactory.java:92) >> ~[qpid-common-0.30.jar:0.30] >> at >> org.apache.qpid.ssl.SSLContextFactory.buildClientContext(SSLContextFactory.java:62) >> ~[qpid-common-0.30.jar:0.30] >> at >> org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.(SecurityLayerFactory.java:78) >> ~[qpid-common-0.30.jar:0.30] >> ... 8 common frames omitted >> Caused by: java.security.UnrecoverableKeyException: Password >> verification failed >> at >> sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770) >> ~[na:1.7.0_79] >> ... 14 common frames omitted >> 2015-06-16 09:59:49,416 [main] INFO >> org.candlepin.gutterball.servlet.GutterballContextListener - >> Destroying gutterball context >> 2015-06-16 09:59:49,416 [main] INFO >> org.candlepin.gutterball.receiver.EventReceiver - Closing >> QPID connection >> >> >> /var/log/candlepin/error.log >> >> 2015-06-16 10:02:30,946 [=, org=] ERROR >> org.candlepin.guice.AMQPBusPubProvider - Unable to >> instantiate AMQPBusProvider: >> javax.jms.JMSException: Error creating connection: Connection >> refused >> at >> org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.candlepin.guice.AMQPBusPubProvider.(AMQPBusPubProvider.java:81) >> ~[AMQPBusPubProvider.class:na] >> at >> org.candlepin.guice.AMQPBusPubProvider$$FastClassByGuice$$ef75a35a.newInstance() >> [guice-3.0.jar:na] >> at >> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) >> [guice-3.0.jar:na] >> at >> com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) >> [guice-3.0.jar:na] >> at >> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) >> [guice-3.0.jar:na] >> at >> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) >> [guice-3.0.jar:na] >> at >> com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53) >> [guice-3.0.jar:na] >> at >> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) >> [guice-3.0.jar:na] >> at >>