Katello 2.3 RC repo sync issue

Duncan_Innes · July 27, 2015, 8:03am

Not sure how I got into this mess, but I appear to have messed up my fresh
Katello 2.3 RC install with very little in the way of poking and prodding.

OS - CentOS 7.1 (running in a libvirt guest)
Katello 2.3 RC & Foreman 1.9 RC 2

Created Product for CentOS

Created repo for CentOS 7
Created repo for CentOS 7 Updates
Created repo for CentOS 7 Extras

Started synching all 3 repos and things seemed to be going well (Sync
Status page was showing progress on all 3 syncs). During the sync, laptop
started running low on battery, so I stuck it in docking station and
switched it's wired network on and turned off the wireless network. I
think this network switch may be the start of the problems. All 3 repo
syncs then stopped and gave errors (perhaps not surprising).

I've managed to poke the CentOS 7 repo into a state where I could start
another sync.

I thought I'd done the same thing with the Updates and Extras repos, but
there are locks that I cannot find out how to remove. I'm also seeing
failures in the About page:

ComponentStatusMessagecandlepinFAIL404 Resource Not Foundcandlepin_authFAILKatello::Resources::Candlepin::CandlepinPing:
404 Resource Not Found (GET /candlepin/status)elasticsearchOKforeman_tasks
FAILuninitialized constant Dynflow::Executors::RemoteViaSocketpulpOK
pulp_authOK

There are currently pending syncs for the Extras and Updates repos which I
can't stop, resume, cancel, delete, etc. I also can't see the Product page
as the spinner just sits in the centre of the page and never produces and
data to see.

Any help greatly appreciated. Don't mind wiping the setup and starting
again, but I feel that I've got into this state rather too easily and that
I can't recover the system simply. If this happened on a production
system, I wouldn't be willing to wipe and reset.

iNecas · July 27, 2015, 9:00am

Seems like some issue with the candlepin service: Eric?

Ad. locked tasks: you need to resume the tasks that are causing the locks (the details for the locks are in the error message),
http://www.katello.org/troubleshooting/index.html has some details on how it works.

But first, you would need to fix the candlepin: please check /var/log/tomcat6/catalina.out for anything interesting… especially,
when restarting tomcat6 service).

The "uninitialized constant Dynflow::Executors::RemoteViaSocket" has already an issue filed and patch sent
http://projects.theforeman.org/issues/11215

– Ivan

···

----- Original Message -----

Not sure how I got into this mess, but I appear to have messed up my fresh
Katello 2.3 RC install with very little in the way of poking and prodding.

OS - CentOS 7.1 (running in a libvirt guest)
Katello 2.3 RC & Foreman 1.9 RC 2

Created Product for CentOS

Created repo for CentOS 7

Created repo for CentOS 7 Updates

Created repo for CentOS 7 Extras

Started synching all 3 repos and things seemed to be going well (Sync Status
page was showing progress on all 3 syncs). During the sync, laptop started
running low on battery, so I stuck it in docking station and switched it’s
wired network on and turned off the wireless network. I think this network
switch may be the start of the problems. All 3 repo syncs then stopped and
gave errors (perhaps not surprising).

I’ve managed to poke the CentOS 7 repo into a state where I could start
another sync.

I thought I’d done the same thing with the Updates and Extras repos, but
there are locks that I cannot find out how to remove. I’m also seeing
failures in the About page:

Component Status Message
candlepin FAIL 404 Resource Not Found
candlepin_auth FAIL Katello::Resources::Candlepin::CandlepinPing: 404
Resource Not Found (GET /candlepin/status)
elasticsearch OK
foreman_tasks FAIL uninitialized constant
Dynflow::Executors::RemoteViaSocket
pulp OK
pulp_auth OK

There are currently pending syncs for the Extras and Updates repos which I
can’t stop, resume, cancel, delete, etc. I also can’t see the Product page
as the spinner just sits in the centre of the page and never produces and
data to see.

Any help greatly appreciated. Don’t mind wiping the setup and starting again,
but I feel that I’ve got into this state rather too easily and that I can’t
recover the system simply. If this happened on a production system, I
wouldn’t be willing to wipe and reset.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com .
To post to this group, send email to foreman-users@googlegroups.com .
Visit this group at http://groups.google.com/group/foreman-users .
For more options, visit https://groups.google.com/d/optout .

Duncan_Innes · July 27, 2015, 9:16am

Thanks Ivan, I'll look into the troubleshooting page. From what I noticed
myself, though, the tasks that are flagged as locking my repos don't allow
me to click the resume button. I'll investigate more tonight though.

I noticed the DynFlow problem had been raised while I was writing my
message. Will leave that part of it for now.

Duncan

···

On Monday, 27 July 2015 10:00:59 UTC+1, Ivan Necas wrote: > > Seems like some issue with the candlepin service: Eric? > > Ad. locked tasks: you need to resume the tasks that are causing the locks > (the details for the locks are in the error message), > http://www.katello.org/troubleshooting/index.html has some details on how > it works. > > But first, you would need to fix the candlepin: please check > /var/log/tomcat6/catalina.out for anything interesting… especially, > when restarting tomcat6 service). > > The "uninitialized constant Dynflow::Executors::RemoteViaSocket" has > already an issue filed and patch sent > http://projects.theforeman.org/issues/11215 > > -- Ivan > > ------------------------------ > > Not sure how I got into this mess, but I appear to have messed up my fresh > Katello 2.3 RC install with very little in the way of poking and prodding. > > OS - CentOS 7.1 (running in a libvirt guest) > Katello 2.3 RC & Foreman 1.9 RC 2 > > Created Product for CentOS > - Created repo for CentOS 7 > - Created repo for CentOS 7 Updates > - Created repo for CentOS 7 Extras > > Started synching all 3 repos and things seemed to be going well (Sync > Status page was showing progress on all 3 syncs). During the sync, laptop > started running low on battery, so I stuck it in docking station and > switched it's wired network on and turned off the wireless network. I > think this network switch may be the start of the problems. All 3 repo > syncs then stopped and gave errors (perhaps not surprising). > > I've managed to poke the CentOS 7 repo into a state where I could start > another sync. > > I thought I'd done the same thing with the Updates and Extras repos, but > there are locks that I cannot find out how to remove. I'm also seeing > failures in the About page: > > ComponentStatusMessagecandlepinFAIL404 Resource Not Foundcandlepin_auth > FAILKatello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found > (GET /candlepin/status)elasticsearchOKforeman_tasksFAILuninitialized > constant Dynflow::Executors::RemoteViaSocketpulpOKpulp_authOK > > > There are currently pending syncs for the Extras and Updates repos which I > can't stop, resume, cancel, delete, etc. I also can't see the Product page > as the spinner just sits in the centre of the page and never produces and > data to see. > > Any help greatly appreciated. Don't mind wiping the setup and starting > again, but I feel that I've got into this state rather too easily and that > I can't recover the system simply. If this happened on a production > system, I wouldn't be willing to wipe and reset. > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-user...@googlegroups.com . > To post to this group, send email to forema...@googlegroups.com > . > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > > >

iNecas · July 27, 2015, 9:20am

> Thanks Ivan, I'll look into the troubleshooting page. From what I noticed
> myself, though, the tasks that are flagged as locking my repos don't allow
> me to click the resume button. I'll investigate more tonight though.

If the state of the tasks is 'running', look at the 'runnings steps' tab: the details from there will be valuable to find
out what's going on and what't not working… My guess is some pulp task not getting stareted, but I will wait for more details.

– Ivan

···

----- Original Message -----

I noticed the DynFlow problem had been raised while I was writing my message.
Will leave that part of it for now.

Duncan

On Monday, 27 July 2015 10:00:59 UTC+1, Ivan Necas wrote:

Seems like some issue with the candlepin service: Eric?

Ad. locked tasks: you need to resume the tasks that are causing the locks
(the details for the locks are in the error message),

http://www.katello.org/troubleshooting/index.html has some details on how
it
works.

But first, you would need to fix the candlepin: please check
/var/log/tomcat6/catalina.out for anything interesting… especially,

when restarting tomcat6 service).

The “uninitialized constant Dynflow::Executors::RemoteViaSocket” has
already
an issue filed and patch sent

Bug #11215: ping causing "uninitialized constant Dynflow::Executors::RemoteViaSocket". - Katello - Foreman

– Ivan

Not sure how I got into this mess, but I appear to have messed up my
fresh
Katello 2.3 RC install with very little in the way of poking and
prodding.

OS - CentOS 7.1 (running in a libvirt guest)

Katello 2.3 RC & Foreman 1.9 RC 2

Created Product for CentOS

Created repo for CentOS 7

Created repo for CentOS 7 Updates

Created repo for CentOS 7 Extras

Started synching all 3 repos and things seemed to be going well (Sync
Status
page was showing progress on all 3 syncs). During the sync, laptop
started
running low on battery, so I stuck it in docking station and switched
it’s
wired network on and turned off the wireless network. I think this
network
switch may be the start of the problems. All 3 repo syncs then stopped
and
gave errors (perhaps not surprising).

I’ve managed to poke the CentOS 7 repo into a state where I could start
another sync.

I thought I’d done the same thing with the Updates and Extras repos, but
there are locks that I cannot find out how to remove. I’m also seeing
failures in the About page:

Component Status Message

candlepin FAIL 404 Resource Not Found

candlepin_auth FAIL Katello::Resources::Candlepin::CandlepinPing: 404
Resource Not Found (GET /candlepin/status)

elasticsearch OK

foreman_tasks FAIL uninitialized constant
Dynflow::Executors::RemoteViaSocket

pulp OK

pulp_auth OK

There are currently pending syncs for the Extras and Updates repos which
I
can’t stop, resume, cancel, delete, etc. I also can’t see the Product
page
as the spinner just sits in the centre of the page and never produces and
data to see.

Any help greatly appreciated. Don’t mind wiping the setup and starting
again,
but I feel that I’ve got into this state rather too easily and that I
can’t
recover the system simply. If this happened on a production system, I
wouldn’t be willing to wipe and reset.

–

You received this message because you are subscribed to the Google Groups
"Foreman users" group.

To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-user...@googlegroups.com .

To post to this group, send email to forema...@googlegroups.com .

Visit this group at http://groups.google.com/group/foreman-users .

For more options, visit https://groups.google.com/d/optout .

Duncan_Innes · July 30, 2015, 10:50am

Sorry for delay - got distracted by my daytime job.

It this any help?

[root@foreman tomcat]# ll
total 164
-rw-r–r--. 1 tomcat tomcat 4401 Jul 24 20:53 catalina.2015-07-24.log
-rw-r–r--. 1 tomcat tomcat 51879 Jul 25 21:20 catalina.2015-07-25.log
-rw-r–r--. 1 tomcat tomcat 19719 Jul 28 17:17 catalina.2015-07-28.log
-rw-r–r--. 1 tomcat tomcat 15851 Jul 29 13:28 catalina.2015-07-29.log
-rw-r–r--. 1 tomcat tomcat 14871 Jul 30 11:45 catalina.2015-07-30.log
-rw-rw----. 1 tomcat tomcat 0 Jul 26 03:13 catalina.out
-rw-rw----. 1 tomcat tomcat 48 May 12 21:08 catalina.out-20150726.gz
-rw-r–r--. 1 tomcat tomcat 0 Jul 24 20:49 host-manager.2015-07-24.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 25 10:09 host-manager.2015-07-25.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 28 07:52 host-manager.2015-07-28.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 29 13:22 host-manager.2015-07-29.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 30 11:42 host-manager.2015-07-30.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 24 20:49 localhost.2015-07-24.log
-rw-r–r--. 1 tomcat tomcat 22817 Jul 25 21:15 localhost.2015-07-25.log
-rw-r–r--. 1 tomcat tomcat 7605 Jul 28 09:47 localhost.2015-07-28.log
-rw-r–r--. 1 tomcat tomcat 7605 Jul 29 13:23 localhost.2015-07-29.log
-rw-r–r--. 1 tomcat tomcat 7607 Jul 30 11:45 localhost.2015-07-30.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 24 20:49 manager.2015-07-24.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 25 10:09 manager.2015-07-25.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 28 07:52 manager.2015-07-28.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 29 13:22 manager.2015-07-29.log
-rw-r–r--. 1 tomcat tomcat 0 Jul 30 11:42 manager.2015-07-30.log
[root@foreman tomcat]# cat catalina.2015-07-30.log
Jul 30, 2015 11:42:44 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path:
/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jul 30, 2015 11:42:44 AM org.apache.catalina.startup.SetAllPropertiesRule
begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property
'sslProtocols' to 'TLSv1.2,TLSv1.1,TLSv1' did not find a matching property.
Jul 30, 2015 11:42:45 AM org.apache.tomcat.util.digester.SetPropertiesRule
begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property
'xmlValidation' to 'false' did not find a matching property.
Jul 30, 2015 11:42:45 AM org.apache.tomcat.util.digester.SetPropertiesRule
begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property
'xmlNamespaceAware' to 'false' did not find a matching property.
Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8443"]
Jul 30, 2015 11:42:49 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 18332 ms
Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardService
startInternal
INFO: Starting service Catalina
Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardEngine
startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.54
Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/gutterball
Jul 30, 2015 11:42:58 AM org.apache.catalina.loader.WebappClassLoader
validateJarFile
INFO:
validateJarFile(/usr/share/tomcat/webapps/gutterball/WEB-INF/lib/servlet-api-2.5.jar)

jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class:
javax/servlet/Servlet.class
Jul 30, 2015 11:44:58 AM org.apache.catalina.util.SessionIdGenerator
createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using
[SHA1PRNG] took [107,531] milliseconds.
Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
/var/lib/tomcat/webapps/gutterball has finished in 134,231 ms
Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/candlepin
Jul 30, 2015 11:45:14 AM org.apache.catalina.loader.WebappClassLoader
validateJarFile
INFO:
validateJarFile(/usr/share/tomcat/webapps/candlepin/WEB-INF/lib/servlet-api-2.5.jar)
jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class:
javax/servlet/Servlet.class
Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [protected org.candlepin.model.AbstractHibernateObject
org.candlepin.resource.ConsumerContentOverrideResource.findParentById(java.lang.String)]
is synthetic and is being intercepted by
[org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This could
indicate a bug. The method may be intercepted twice, or may not be
intercepted at all.
Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [protected org.candlepin.model.AbstractHibernateObject
org.candlepin.resource.ActivationKeyContentOverrideResource.findParentById(java.lang.String)]
is synthetic and is being intercepted by
[org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This could
indicate a bug. The method may be intercepted twice, or may not be
intercepted at all.
Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext
startInternal
SEVERE: Error listenerStart
Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext
startInternal
SEVERE: Context [/candlepin] startup failed due to previous errors
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesJdbc
SEVERE: The web application [/candlepin] registered the JDBC driver
[org.postgresql.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [com.google.inject.internal.util.$Finalizer] but has failed to stop
it. This is very likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [IoSender - foreman.rhel7.local/192.168.122.129:5671] but has failed
to stop it. This is very likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [IoReceiver - foreman.rhel7.local/192.168.122.129:5671] but has
failed to stop it. This is very likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [ack-flusher] but has failed to stop it. This is very likely to
create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [Timer-1] but has failed to stop it. This is very likely to create a
memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but
has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] but
has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2] but
has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-1] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-2] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-3] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-4] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-5] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-6] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-7] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-8] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-9] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-10] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-11] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-12] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-13] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-14] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_Worker-15] but has failed to stop it. This is very
likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread
named [QuartzScheduler_QuartzSchedulerThread] but has failed to stop it.
This is very likely to create a memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/candlepin] created a ThreadLocal with key of
type [org.apache.qpid.transport.network.Disassembler$1] (value
[org.apache.qpid.transport.network.Disassembler$1@639179fe]) and a value of
type [org.apache.qpid.transport.codec.BBEncoder] (value
[org.apache.qpid.transport.codec.BBEncoder@63e045c3]) but failed to remove
it when the web application was stopped. Threads are going to be renewed
over time to try and avoid a probable memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/candlepin] created a ThreadLocal with key of
type [org.apache.qpid.util.Strings$1] (value
[org.apache.qpid.util.Strings$1@1904386f]) and a value of type [char[]]
(value [[C@64150c3d]) but failed to remove it when the web application was
stopped. Threads are going to be renewed over time to try and avoid a
probable memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/candlepin] created a ThreadLocal with key of
type [com.google.inject.internal.InjectorImpl$1] (value
[com.google.inject.internal.InjectorImpl$1@21d6ac20]) and a value of type
[java.lang.Object[]] (value [[Ljava.lang.Object;@14be7a3b]) but failed to
remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/candlepin] created a ThreadLocal with key of
type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@654ac220]) and a
value of type [org.hibernate.ejb.EntityManagerImpl] (value
[org.hibernate.ejb.EntityManagerImpl@f0ef6d4]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/candlepin] created a ThreadLocal with key of
type [org.apache.qpid.transport.network.Assembler$1] (value
[org.apache.qpid.transport.network.Assembler$1@1f6ee005]) and a value of
type [org.apache.qpid.transport.codec.BBDecoder] (value
[org.apache.qpid.transport.codec.BBDecoder@28357862]) but failed to remove
it when the web application was stopped. Threads are going to be renewed
over time to try and avoid a probable memory leak.
Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/candlepin] created a ThreadLocal with key of
type [org.apache.qpid.transport.network.Disassembler$1] (value
[org.apache.qpid.transport.network.Disassembler$1@639179fe]) and a value of
type [org.apache.qpid.transport.codec.BBEncoder] (value
[org.apache.qpid.transport.codec.BBEncoder@4e727e8d]) but failed to remove
it when the web application was stopped. Threads are going to be renewed
over time to try and avoid a probable memory leak.
Jul 30, 2015 11:45:44 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
/var/lib/tomcat/webapps/candlepin has finished in 40,426 ms
Jul 30, 2015 11:45:44 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Jul 30, 2015 11:45:44 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8443"]
Jul 30, 2015 11:45:44 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Jul 30, 2015 11:45:44 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 175051 ms
[root@foreman tomcat]#

I see lots of SEVERE in there, but don't know enough tomcat to know why.

Duncan

···

On Monday, 27 July 2015 10:00:59 UTC+1, Ivan Necas wrote: > > Seems like some issue with the candlepin service: Eric? > > Ad. locked tasks: you need to resume the tasks that are causing the locks > (the details for the locks are in the error message), > http://www.katello.org/troubleshooting/index.html has some details on how > it works. > > But first, you would need to fix the candlepin: please check > /var/log/tomcat6/catalina.out for anything interesting… especially, > when restarting tomcat6 service). > > The "uninitialized constant Dynflow::Executors::RemoteViaSocket" has > already an issue filed and patch sent > http://projects.theforeman.org/issues/11215 > > -- Ivan > > ------------------------------ > > Not sure how I got into this mess, but I appear to have messed up my fresh > Katello 2.3 RC install with very little in the way of poking and prodding. > > OS - CentOS 7.1 (running in a libvirt guest) > Katello 2.3 RC & Foreman 1.9 RC 2 > > Created Product for CentOS > - Created repo for CentOS 7 > - Created repo for CentOS 7 Updates > - Created repo for CentOS 7 Extras > > Started synching all 3 repos and things seemed to be going well (Sync > Status page was showing progress on all 3 syncs). During the sync, laptop > started running low on battery, so I stuck it in docking station and > switched it's wired network on and turned off the wireless network. I > think this network switch may be the start of the problems. All 3 repo > syncs then stopped and gave errors (perhaps not surprising). > > I've managed to poke the CentOS 7 repo into a state where I could start > another sync. > > I thought I'd done the same thing with the Updates and Extras repos, but > there are locks that I cannot find out how to remove. I'm also seeing > failures in the About page: > > ComponentSta

Duncan_Innes · July 27, 2015, 8:00pm

No time to investigate properly at the moment, but the second task in the
entire history is still showing as Pending:

Listen on candlepin events {"locale"=>"en"}
State: running
Result: Pending
Started at: 2015-07-24 20:00:14 UTC

The only event before it is the "Create organisation 'Default
Organization'" (odd mix of UK English and US English there with
organi(s|z)ation BTW)

The next event is me starting to create the Product and repositories 20
minutes later.

Don't remember looking at this before, but should I expect the "Listen on
candlepin events" task to be in this state? Perhaps my network switch
isn't what caused my problem?

D

···

On Monday, 27 July 2015 10:20:15 UTC+1, Ivan Necas wrote: > > > > > ------------------------------ > > Thanks Ivan, I'll look into the troubleshooting page. From what I noticed > myself, though, the tasks that are flagged as locking my repos don't allow > me to click the resume button. I'll investigate more tonight though. > > > If the state of the tasks is 'running', look at the 'runnings steps' tab: > the details from there will be valuable to find > out what's going on and what't not working… My guess is some pulp task not > getting stareted, but I will wait for more details. > > -- Ivan > > > I noticed the DynFlow problem had been raised while I was writing my > message. Will leave that part of it for now. > > Duncan > > On Monday, 27 July 2015 10:00:59 UTC+1, Ivan Necas wrote: >> >> Seems like some issue with the candlepin service: Eric? >> >> Ad. locked tasks: you need to resume the tasks that are causing the locks >> (the details for the locks are in the error message), >> http://www.katello.org/troubleshooting/index.html has some details on >> how it works. >> >> But first, you would need to fix the candlepin: please check >> /var/log/tomcat6/catalina.out for anything interesting… especially, >> when restarting tomcat6 service). >> >> The "uninitialized constant Dynflow::Executors::RemoteViaSocket" has >> already an issue filed and patch sent >> http://projects.theforeman.org/issues/11215 >> >> -- Ivan >> >> ------------------------------ >> >> Not sure how I got into this mess, but I appear to have messed up my >> fresh Katello 2.3 RC install with very little in the way of poking and >> prodding. >> >> OS - CentOS 7.1 (running in a libvirt guest) >> Katello 2.3 RC & Foreman 1.9 RC 2 >> >> Created Product for CentOS >> - Created repo for CentOS 7 >> - Created repo for CentOS 7 Updates >> - Created repo for CentOS 7 Extras >> >> Started synching all 3 repos and things seemed to be going well (Sync >> Status page was showing progress on all 3 syncs). During the sync, laptop >> started running low on battery, so I stuck it in docking station and >> switched it's wired network on and turned off the wireless network. I >> think this network switch may be the start of the problems. All 3 repo >> syncs then stopped and gave errors (perhaps not surprising). >> >> I've managed to poke the CentOS 7 repo into a state where I could start >> another sync. >> >> I thought I'd done the same thing with the Updates and Extras repos, but >> there are locks that I cannot find out how to remove. I'm also seeing >> failures in the About page: >> >> ComponentStatusMessagecandlepinFAIL404 Resource Not Foundcandlepin_auth >> FAILKatello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found >> (GET /candlepin/status)elasticsearchOKforeman_tasksFAILuninitialized >> constant Dynflow::Executors::RemoteViaSocketpulpOKpulp_authOK >> >> >> There are currently pending syncs for the Extras and Updates repos which >> I can't stop, resume, cancel, delete, etc. I also can't see the Product >> page as the spinner just sits in the centre of the page and never produces >> and data to see. >> >> Any help greatly appreciated. Don't mind wiping the setup and starting >> again, but I feel that I've got into this state rather too easily and that >> I can't recover the system simply. If this happened on a production >> system, I wouldn't be willing to wipe and reset. >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-user...@googlegroups.com . >> To post to this group, send email to forema...@googlegroups.com >> . >> Visit this group at http://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> >> >> >

iNecas · July 30, 2015, 11:24am

That was very helpful, the important error message here is "Error listenerStart",
I was able to reproduce the behavior, when I installed jdk 1.6 and made
that the default, using alternatives.

The solution for this issue should be making sure java-1.7.0-openjdk is installed,
making it a default using alternatives --config java, restarting
the tomcat service, checking with hammer ping that it helped and proceeding with the upgrade procedure.

– Ivan

···

----- Original Message ----- > Sorry for delay - got distracted by my daytime job. > > It this any help? > > [root@foreman tomcat]# ll > total 164 > -rw-r--r--. 1 tomcat tomcat 4401 Jul 24 20:53 catalina.2015-07-24.log > -rw-r--r--. 1 tomcat tomcat 51879 Jul 25 21:20 catalina.2015-07-25.log > -rw-r--r--. 1 tomcat tomcat 19719 Jul 28 17:17 catalina.2015-07-28.log > -rw-r--r--. 1 tomcat tomcat 15851 Jul 29 13:28 catalina.2015-07-29.log > -rw-r--r--. 1 tomcat tomcat 14871 Jul 30 11:45 catalina.2015-07-30.log > -rw-rw----. 1 tomcat tomcat 0 Jul 26 03:13 catalina.out > -rw-rw----. 1 tomcat tomcat 48 May 12 21:08 catalina.out-20150726.gz > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 host-manager.2015-07-24.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 25 10:09 host-manager.2015-07-25.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 28 07:52 host-manager.2015-07-28.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 29 13:22 host-manager.2015-07-29.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 30 11:42 host-manager.2015-07-30.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 localhost.2015-07-24.log > -rw-r--r--. 1 tomcat tomcat 22817 Jul 25 21:15 localhost.2015-07-25.log > -rw-r--r--. 1 tomcat tomcat 7605 Jul 28 09:47 localhost.2015-07-28.log > -rw-r--r--. 1 tomcat tomcat 7605 Jul 29 13:23 localhost.2015-07-29.log > -rw-r--r--. 1 tomcat tomcat 7607 Jul 30 11:45 localhost.2015-07-30.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 manager.2015-07-24.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 25 10:09 manager.2015-07-25.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 28 07:52 manager.2015-07-28.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 29 13:22 manager.2015-07-29.log > -rw-r--r--. 1 tomcat tomcat 0 Jul 30 11:42 manager.2015-07-30.log > [root@foreman tomcat]# cat catalina.2015-07-30.log > Jul 30, 2015 11:42:44 AM org.apache.catalina.core.AprLifecycleListener init > INFO: The APR based Apache Tomcat Native library which allows optimal > performance in production environments was not found on the > java.library.path: > /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > Jul 30, 2015 11:42:44 AM org.apache.catalina.startup.SetAllPropertiesRule > begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property > 'sslProtocols' to 'TLSv1.2,TLSv1.1,TLSv1' did not find a matching property. > Jul 30, 2015 11:42:45 AM org.apache.tomcat.util.digester.SetPropertiesRule > begin > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property > 'xmlValidation' to 'false' did not find a matching property. > Jul 30, 2015 11:42:45 AM org.apache.tomcat.util.digester.SetPropertiesRule > begin > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property > 'xmlNamespaceAware' to 'false' did not find a matching property. > Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["http-bio-8080"] > Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["http-bio-8443"] > Jul 30, 2015 11:42:49 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["ajp-bio-8009"] > Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.Catalina load > INFO: Initialization processed in 18332 ms > Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardService > startInternal > INFO: Starting service Catalina > Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardEngine > startInternal > INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 > Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.HostConfig > deployDirectory > INFO: Deploying web application directory /var/lib/tomcat/webapps/gutterball > Jul 30, 2015 11:42:58 AM org.apache.catalina.loader.WebappClassLoader > validateJarFile > INFO: > validateJarFile(/usr/share/tomcat/webapps/gutterball/WEB-INF/lib/servlet-api-2.5.jar) > - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: > javax/servlet/Servlet.class > Jul 30, 2015 11:44:58 AM org.apache.catalina.util.SessionIdGenerator > createSecureRandom > INFO: Creation of SecureRandom instance for session ID generation using > [SHA1PRNG] took [107,531] milliseconds. > Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig > deployDirectory > INFO: Deployment of web application directory > /var/lib/tomcat/webapps/gutterball has finished in 134,231 ms > Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig > deployDirectory > INFO: Deploying web application directory /var/lib/tomcat/webapps/candlepin > Jul 30, 2015 11:45:14 AM org.apache.catalina.loader.WebappClassLoader > validateJarFile > INFO: > validateJarFile(/usr/share/tomcat/webapps/candlepin/WEB-INF/lib/servlet-api-2.5.jar) > - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: > javax/servlet/Servlet.class > Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory > WARNING: Method [protected org.candlepin.model.AbstractHibernateObject > org.candlepin.resource.ConsumerContentOverrideResource.findParentById(java.lang.String)] > is synthetic and is being intercepted by > [org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This could > indicate a bug. The method may be intercepted twice, or may not be > intercepted at all. > Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory > WARNING: Method [protected org.candlepin.model.AbstractHibernateObject > org.candlepin.resource.ActivationKeyContentOverrideResource.findParentById(java.lang.String)] > is synthetic and is being intercepted by > [org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This could > indicate a bug. The method may be intercepted twice, or may not be > intercepted at all. > Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext > startInternal > SEVERE: Error listenerStart > Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext > startInternal > SEVERE: Context [/candlepin] startup failed due to previous errors > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesJdbc > SEVERE: The web application [/candlepin] registered the JDBC driver > [org.postgresql.Driver] but failed to unregister it when the web > application was stopped. To prevent a memory leak, the JDBC Driver has been > forcibly unregistered. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [com.google.inject.internal.util.$Finalizer] but has failed to stop > it. This is very likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [IoSender - foreman.rhel7.local/192.168.122.129:5671] but has failed > to stop it. This is very likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [IoReceiver - foreman.rhel7.local/192.168.122.129:5671] but has > failed to stop it. This is very likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [ack-flusher] but has failed to stop it. This is very likely to > create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [Timer-1] but has failed to stop it. This is very likely to create a > memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but > has failed to stop it. This is very likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] but > has failed to stop it. This is very likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2] but > has failed to stop it. This is very likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-1] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-2] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-3] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-4] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-5] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-6] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-7] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-8] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-9] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-10] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-11] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-12] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-13] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-14] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_Worker-15] but has failed to stop it. This is very > likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/candlepin] appears to have started a thread > named [QuartzScheduler_QuartzSchedulerThread] but has failed to stop it. > This is very likely to create a memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/candlepin] created a ThreadLocal with key of > type [org.apache.qpid.transport.network.Disassembler$1] (value > [org.apache.qpid.transport.network.Disassembler$1@639179fe]) and a value of > type [org.apache.qpid.transport.codec.BBEncoder] (value > [org.apache.qpid.transport.codec.BBEncoder@63e045c3]) but failed to remove > it when the web application was stopped. Threads are going to be renewed > over time to try and avoid a probable memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/candlepin] created a ThreadLocal with key of > type [org.apache.qpid.util.Strings$1] (value > [org.apache.qpid.util.Strings$1@1904386f]) and a value of type [char[]] > (value [[C@64150c3d]) but failed to remove it when the web application was > stopped. Threads are going to be renewed over time to try and avoid a > probable memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/candlepin] created a ThreadLocal with key of > type [com.google.inject.internal.InjectorImpl$1] (value > [com.google.inject.internal.InjectorImpl$1@21d6ac20]) and a value of type > [java.lang.Object[]] (value [[Ljava.lang.Object;@14be7a3b]) but failed to > remove it when the web application was stopped. Threads are going to be > renewed over time to try and avoid a probable memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/candlepin] created a ThreadLocal with key of > type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@654ac220]) and a > value of type [org.hibernate.ejb.EntityManagerImpl] (value > [org.hibernate.ejb.EntityManagerImpl@f0ef6d4]) but failed to remove it when > the web application was stopped. Threads are going to be renewed over time > to try and avoid a probable memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/candlepin] created a ThreadLocal with key of > type [org.apache.qpid.transport.network.Assembler$1] (value > [org.apache.qpid.transport.network.Assembler$1@1f6ee005]) and a value of > type [org.apache.qpid.transport.codec.BBDecoder] (value > [org.apache.qpid.transport.codec.BBDecoder@28357862]) but failed to remove > it when the web application was stopped. Threads are going to be renewed > over time to try and avoid a probable memory leak. > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/candlepin] created a ThreadLocal with key of > type [org.apache.qpid.transport.network.Disassembler$1] (value > [org.apache.qpid.transport.network.Disassembler$1@639179fe]) and a value of > type [org.apache.qpid.transport.codec.BBEncoder] (value > [org.apache.qpid.transport.codec.BBEncoder@4e727e8d]) but failed to remove > it when the web application was stopped. Threads are going to be renewed > over time to try and avoid a probable memory leak. > Jul 30, 2015 11:45:44 AM org.apache.catalina.startup.HostConfig > deployDirectory > INFO: Deployment of web application directory > /var/lib/tomcat/webapps/candlepin has finished in 40,426 ms > Jul 30, 2015 11:45:44 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["http-bio-8080"] > Jul 30, 2015 11:45:44 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["http-bio-8443"] > Jul 30, 2015 11:45:44 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["ajp-bio-8009"] > Jul 30, 2015 11:45:44 AM org.apache.catalina.startup.Catalina start > INFO: Server startup in 175051 ms > [root@foreman tomcat]# > > I see lots of SEVERE in there, but don't know enough tomcat to know why. > > Duncan > > On Monday, 27 July 2015 10:00:59 UTC+1, Ivan Necas wrote: > > > > Seems like some i

iNecas · July 28, 2015, 7:59am

The listen on candlepin action is a long-running tasks, that is expected to be in this state and
it itself is not indication of any error

– Ivan

···

----- Original Message ----- > No time to investigate properly at the moment, but the second task in the > entire history is still showing as Pending: > > Listen on candlepin events {"locale"=>"en"} > State: running > Result: Pending > Started at: 2015-07-24 20:00:14 UTC > > The only event before it is the "Create organisation 'Default > Organization'" (odd mix of UK English and US English there with > organi(s|z)ation BTW) > > The next event is me starting to create the Product and repositories 20 > minutes later. > > Don't remember looking at this before, but should I expect the "Listen on > candlepin events" task to be in this state? Perhaps my network switch > isn't what caused my problem? > > D > > On Monday, 27 July 2015 10:20:15 UTC+1, Ivan Necas wrote: > > > > > > > > > > ------------------------------ > > > > Thanks Ivan, I'll look into the troubleshooting page. From what I noticed > > myself, though, the tasks that are flagged as locking my repos don't allow > > me to click the resume button. I'll investigate more tonight though. > > > > > > If the state of the tasks is 'running', look at the 'runnings steps' tab: > > the details from there will be valuable to find > > out what's going on and what't not working… My guess is some pulp task not > > getting stareted, but I will wait for more details. > > > > -- Ivan > > > > > > I noticed the DynFlow problem had been raised while I was writing my > > message. Will leave that part of it for now. > > > > Duncan > > > > On Monday, 27 July 2015 10:00:59 UTC+1, Ivan Necas wrote: > >> > >> Seems like some issue with the candlepin service: Eric? > >> > >> Ad. locked tasks: you need to resume the tasks that are causing the locks > >> (the details for the locks are in the error message), > >> http://www.katello.org/troubleshooting/index.html has some details on > >> how it works. > >> > >> But first, you would need to fix the candlepin: please check > >> /var/log/tomcat6/catalina.out for anything interesting… especially, > >> when restarting tomcat6 service). > >> > >> The "uninitialized constant Dynflow::Executors::RemoteViaSocket" has > >> already an issue filed and patch sent > >> http://projects.theforeman.org/issues/11215 > >> > >> -- Ivan > >> > >> ------------------------------ > >> > >> Not sure how I got into this mess, but I appear to have messed up my > >> fresh Katello 2.3 RC install with very little in the way of poking and > >> prodding. > >> > >> OS - CentOS 7.1 (running in a libvirt guest) > >> Katello 2.3 RC & Foreman 1.9 RC 2 > >> > >> Created Product for CentOS > >> - Created repo for CentOS 7 > >> - Created repo for CentOS 7 Updates > >> - Created repo for CentOS 7 Extras > >> > >> Started synching all 3 repos and things seemed to be going well (Sync > >> Status page was showing progress on all 3 syncs). During the sync, laptop > >> started running low on battery, so I stuck it in docking station and > >> switched it's wired network on and turned off the wireless network. I > >> think this network switch may be the start of the problems. All 3 repo > >> syncs then stopped and gave errors (perhaps not surprising). > >> > >> I've managed to poke the CentOS 7 repo into a state where I could start > >> another sync. > >> > >> I thought I'd done the same thing with the Updates and Extras repos, but > >> there are locks that I cannot find out how to remove. I'm also seeing > >> failures in the About page: > >> > >> ComponentStatusMessagecandlepinFAIL404 Resource Not Foundcandlepin_auth > >> FAILKatello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found > >> (GET /candlepin/status)elasticsearchOKforeman_tasksFAILuninitialized > >> constant Dynflow::Executors::RemoteViaSocketpulpOKpulp_authOK > >> > >> > >> There are currently pending syncs for the Extras and Updates repos which > >> I can't stop, resume, cancel, delete, etc. I also can't see the Product > >> page as the spinner just sits in the centre of the page and never produces > >> and data to see. > >> > >> Any help greatly appreciated. Don't mind wiping the setup and starting > >> again, but I feel that I've got into this state rather too easily and that > >> I can't recover the system simply. If this happened on a production > >> system, I wouldn't be willing to wipe and reset. > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Foreman users" group. > >> To unsubscribe from this group and stop receiving emails from it, send an > >> email to foreman-user...@googlegroups.com . > >> To post to this group, send email to forema...@googlegroups.com > >> . > >> Visit this group at http://groups.google.com/group/foreman-users. > >> For more options, visit https://groups.google.com/d/optout. > >> > >> > >> > >

Duncan_Innes · July 30, 2015, 11:44am

This system doesn't have java-1.6 installed:

[root@foreman tomcat]# rpm -qa | grep java | sort
java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64
java-1.7.0-openjdk-headless-1.7.0.85-2.6.1.2.el7_1.x86_64
java-client-1.0-1.noarch
javamail-1.4.6-8.el7.noarch
javapackages-tools-3.4.1-6.el7_0.noarch
python-javapackages-3.4.1-6.el7_0.noarch
snappy-java-1.0.4-1.el7.x86_64
tzdata-java-2015e-1.el7.noarch
[root@foreman tomcat]#

···

On Thursday, 30 July 2015 12:24:53 UTC+1, Ivan Necas wrote: > > That was very helpful, the important error message here is "Error > listenerStart", > I was able to reproduce the behavior, when I installed jdk 1.6 and made > that the default, using `alternatives`. > > The solution for this issue should be making sure java-1.7.0-openjdk is > installed, > making it a default using `alternatives --config java`, restarting > the tomcat service, checking with `hammer ping` that it helped and > proceeding with the upgrade procedure. > > -- Ivan > > > > ----- Original Message ----- > > Sorry for delay - got distracted by my daytime job. > > > > It this any help? > > > > [root@foreman tomcat]# ll > > total 164 > > -rw-r--r--. 1 tomcat tomcat 4401 Jul 24 20:53 catalina.2015-07-24.log > > -rw-r--r--. 1 tomcat tomcat 51879 Jul 25 21:20 catalina.2015-07-25.log > > -rw-r--r--. 1 tomcat tomcat 19719 Jul 28 17:17 catalina.2015-07-28.log > > -rw-r--r--. 1 tomcat tomcat 15851 Jul 29 13:28 catalina.2015-07-29.log > > -rw-r--r--. 1 tomcat tomcat 14871 Jul 30 11:45 catalina.2015-07-30.log > > -rw-rw----. 1 tomcat tomcat 0 Jul 26 03:13 catalina.out > > -rw-rw----. 1 tomcat tomcat 48 May 12 21:08 catalina.out-20150726.gz > > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 > host-manager.2015-07-24.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 25 10:09 > host-manager.2015-07-25.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 28 07:52 > host-manager.2015-07-28.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 29 13:22 > host-manager.2015-07-29.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 30 11:42 > host-manager.2015-07-30.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 localhost.2015-07-24.log > > -rw-r--r--. 1 tomcat tomcat 22817 Jul 25 21:15 localhost.2015-07-25.log > > -rw-r--r--. 1 tomcat tomcat 7605 Jul 28 09:47 localhost.2015-07-28.log > > -rw-r--r--. 1 tomcat tomcat 7605 Jul 29 13:23 localhost.2015-07-29.log > > -rw-r--r--. 1 tomcat tomcat 7607 Jul 30 11:45 localhost.2015-07-30.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 manager.2015-07-24.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 25 10:09 manager.2015-07-25.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 28 07:52 manager.2015-07-28.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 29 13:22 manager.2015-07-29.log > > -rw-r--r--. 1 tomcat tomcat 0 Jul 30 11:42 manager.2015-07-30.log > > [root@foreman tomcat]# cat catalina.2015-07-30.log > > Jul 30, 2015 11:42:44 AM org.apache.catalina.core.AprLifecycleListener > init > > INFO: The APR based Apache Tomcat Native library which allows optimal > > performance in production environments was not found on the > > java.library.path: > > /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > Jul 30, 2015 11:42:44 AM > org.apache.catalina.startup.SetAllPropertiesRule > > begin > > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting > property > > 'sslProtocols' to 'TLSv1.2,TLSv1.1,TLSv1' did not find a matching > property. > > Jul 30, 2015 11:42:45 AM > org.apache.tomcat.util.digester.SetPropertiesRule > > begin > > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting > property > > 'xmlValidation' to 'false' did not find a matching property. > > Jul 30, 2015 11:42:45 AM > org.apache.tomcat.util.digester.SetPropertiesRule > > begin > > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting > property > > 'xmlNamespaceAware' to 'false' did not find a matching property. > > Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init > > INFO: Initializing ProtocolHandler ["http-bio-8080"] > > Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init > > INFO: Initializing ProtocolHandler ["http-bio-8443"] > > Jul 30, 2015 11:42:49 AM org.apache.coyote.AbstractProtocol init > > INFO: Initializing ProtocolHandler ["ajp-bio-8009"] > > Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.Catalina load > > INFO: Initialization processed in 18332 ms > > Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardService > > startInternal > > INFO: Starting service Catalina > > Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardEngine > > startInternal > > INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 > > Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.HostConfig > > deployDirectory > > INFO: Deploying web application directory > /var/lib/tomcat/webapps/gutterball > > Jul 30, 2015 11:42:58 AM org.apache.catalina.loader.WebappClassLoader > > validateJarFile > > INFO: > > > validateJarFile(/usr/share/tomcat/webapps/gutterball/WEB-INF/lib/servlet-api-2.5.jar) > > > - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: > > javax/servlet/Servlet.class > > Jul 30, 2015 11:44:58 AM org.apache.catalina.util.SessionIdGenerator > > createSecureRandom > > INFO: Creation of SecureRandom instance for session ID generation using > > [SHA1PRNG] took [107,531] milliseconds. > > Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig > > deployDirectory > > INFO: Deployment of web application directory > > /var/lib/tomcat/webapps/gutterball has finished in 134,231 ms > > Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig > > deployDirectory > > INFO: Deploying web application directory > /var/lib/tomcat/webapps/candlepin > > Jul 30, 2015 11:45:14 AM org.apache.catalina.loader.WebappClassLoader > > validateJarFile > > INFO: > > > validateJarFile(/usr/share/tomcat/webapps/candlepin/WEB-INF/lib/servlet-api-2.5.jar) > > > - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: > > javax/servlet/Servlet.class > > Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory > > WARNING: Method [protected org.candlepin.model.AbstractHibernateObject > > > org.candlepin.resource.ConsumerContentOverrideResource.findParentById(java.lang.String)] > > > is synthetic and is being intercepted by > > [org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This > could > > indicate a bug. The method may be intercepted twice, or may not be > > intercepted at all. > > Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory > > WARNING: Method [protected org.candlepin.model.AbstractHibernateObject > > > org.candlepin.resource.ActivationKeyContentOverrideResource.findParentById(java.lang.String)] > > > is synthetic and is being intercepted by > > [org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This > could > > indicate a bug. The method may be intercepted twice, or may not be > > intercepted at all. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext > > startInternal > > SEVERE: Error listenerStart > > Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext > > startInternal > > SEVERE: Context [/candlepin] startup failed due to previous errors > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesJdbc > > SEVERE: The web application [/candlepin] registered the JDBC driver > > [org.postgresql.Driver] but failed to unregister it when the web > > application was stopped. To prevent a memory leak, the JDBC Driver has > been > > forcibly unregistered. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [com.google.inject.internal.util.$Finalizer] but has failed to > stop > > it. This is very likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [IoSender - foreman.rhel7.local/192.168.122.129:5671] but has > failed > > to stop it. This is very likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [IoReceiver - foreman.rhel7.local/192.168.122.129:5671] but has > > failed to stop it. This is very likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [ack-flusher] but has failed to stop it. This is very likely to > > create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [Timer-1] but has failed to stop it. This is very likely to create > a > > memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] > but > > has failed to stop it. This is very likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] > but > > has failed to stop it. This is very likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2] > but > > has failed to stop it. This is very likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-1] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-2] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-3] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-4] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-5] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-6] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-7] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-8] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-9] but has failed to stop it. This is very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-10] but has failed to stop it. This is > very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-11] but has failed to stop it. This is > very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-12] but has failed to stop it. This is > very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-13] but has failed to stop it. This is > very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-14] but has failed to stop it. This is > very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_Worker-15] but has failed to stop it. This is > very > > likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreads > > SEVERE: The web application [/candlepin] appears to have started a > thread > > named [QuartzScheduler_QuartzSchedulerThread] but has failed to stop it. > > This is very likely to create a memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > checkThreadLocalMapForLeaks > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > of > > type [org.apache.qpid.transport.network.Disassembler$1] (value > > [org.apache.qpid.transport.network.Disassembler$1@639179fe]) and a value > of > > type [org.apache.qpid.transport.codec.BBEncoder] (value > > [org.apache.qpid.transport.codec.BBEncoder@63e045c3]) but failed to > remove > > it when the web application was stopped. Threads are going to be renewed > > over time to try and avoid a probable memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > checkThreadLocalMapForLeaks > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > of > > type [org.apache.qpid.util.Strings$1] (value > > [org.apache.qpid.util.Strings$1@1904386f]) and a value of type [char[]] > > (value [[C@64150c3d]) but failed to remove it when the web application > was > > stopped. Threads are going to be renewed over time to try and avoid a > > probable memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > checkThreadLocalMapForLeaks > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > of > > type [com.google.inject.internal.InjectorImpl$1] (value > > [com.google.inject.internal.InjectorImpl$1@21d6ac20]) and a value of > type > > [java.lang.Object[]] (value [[Ljava.lang.Object;@14be7a3b]) but failed > to > > remove it when the web application was stopped. Threads are going to be > > renewed over time to try and avoid a probable memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > checkThreadLocalMapForLeaks > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > of > > type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@654ac220]) > and a > > value of type [org.hibernate.ejb.EntityManagerImpl] (value > > [org.hibernate.ejb.EntityManagerImpl@f0ef6d4]) but failed to remove it > when > > the web application was stopped. Threads are going to be renewed over > time > > to try and avoid a probable memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > checkThreadLocalMapForLeaks > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > of > > type [org.apache.qpid.transport.network.Assembler$1] (value > > [org.apache.qpid.transport.network.Assembler$1@1f6ee005]) and a value of > > type [org.apache.qpid.transport.codec.BBDecoder] (value > > [org.apache.qpid.transport.codec.BBDecoder@28357862]) but failed to > remove > > it when the web application was stopped. Threads are going to be renewed > > over time to try and avoid a probable memory leak. > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > chec

iNecas · July 30, 2015, 12:28pm

Oh, we need to dig deeper than (calling for some Candlepin devels to help with debugging).

In the meantime, what about /var/log/candlepin/{candlepin.log,error.log}, anything interesting there?

– Ivan

···

----- Original Message ----- > This system doesn't have java-1.6 installed: > > [root@foreman tomcat]# rpm -qa | grep java | sort > java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64 > java-1.7.0-openjdk-headless-1.7.0.85-2.6.1.2.el7_1.x86_64 > java-client-1.0-1.noarch > javamail-1.4.6-8.el7.noarch > javapackages-tools-3.4.1-6.el7_0.noarch > python-javapackages-3.4.1-6.el7_0.noarch > snappy-java-1.0.4-1.el7.x86_64 > tzdata-java-2015e-1.el7.noarch > [root@foreman tomcat]# > > > On Thursday, 30 July 2015 12:24:53 UTC+1, Ivan Necas wrote: > > > > That was very helpful, the important error message here is "Error > > listenerStart", > > I was able to reproduce the behavior, when I installed jdk 1.6 and made > > that the default, using `alternatives`. > > > > The solution for this issue should be making sure java-1.7.0-openjdk is > > installed, > > making it a default using `alternatives --config java`, restarting > > the tomcat service, checking with `hammer ping` that it helped and > > proceeding with the upgrade procedure. > > > > -- Ivan > > > > > > > > ----- Original Message ----- > > > Sorry for delay - got distracted by my daytime job. > > > > > > It this any help? > > > > > > [root@foreman tomcat]# ll > > > total 164 > > > -rw-r--r--. 1 tomcat tomcat 4401 Jul 24 20:53 catalina.2015-07-24.log > > > -rw-r--r--. 1 tomcat tomcat 51879 Jul 25 21:20 catalina.2015-07-25.log > > > -rw-r--r--. 1 tomcat tomcat 19719 Jul 28 17:17 catalina.2015-07-28.log > > > -rw-r--r--. 1 tomcat tomcat 15851 Jul 29 13:28 catalina.2015-07-29.log > > > -rw-r--r--. 1 tomcat tomcat 14871 Jul 30 11:45 catalina.2015-07-30.log > > > -rw-rw----. 1 tomcat tomcat 0 Jul 26 03:13 catalina.out > > > -rw-rw----. 1 tomcat tomcat 48 May 12 21:08 catalina.out-20150726.gz > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 > > host-manager.2015-07-24.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 25 10:09 > > host-manager.2015-07-25.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 28 07:52 > > host-manager.2015-07-28.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 29 13:22 > > host-manager.2015-07-29.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 30 11:42 > > host-manager.2015-07-30.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 localhost.2015-07-24.log > > > -rw-r--r--. 1 tomcat tomcat 22817 Jul 25 21:15 localhost.2015-07-25.log > > > -rw-r--r--. 1 tomcat tomcat 7605 Jul 28 09:47 localhost.2015-07-28.log > > > -rw-r--r--. 1 tomcat tomcat 7605 Jul 29 13:23 localhost.2015-07-29.log > > > -rw-r--r--. 1 tomcat tomcat 7607 Jul 30 11:45 localhost.2015-07-30.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 24 20:49 manager.2015-07-24.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 25 10:09 manager.2015-07-25.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 28 07:52 manager.2015-07-28.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 29 13:22 manager.2015-07-29.log > > > -rw-r--r--. 1 tomcat tomcat 0 Jul 30 11:42 manager.2015-07-30.log > > > [root@foreman tomcat]# cat catalina.2015-07-30.log > > > Jul 30, 2015 11:42:44 AM org.apache.catalina.core.AprLifecycleListener > > init > > > INFO: The APR based Apache Tomcat Native library which allows optimal > > > performance in production environments was not found on the > > > java.library.path: > > > /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > > Jul 30, 2015 11:42:44 AM > > org.apache.catalina.startup.SetAllPropertiesRule > > > begin > > > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting > > property > > > 'sslProtocols' to 'TLSv1.2,TLSv1.1,TLSv1' did not find a matching > > property. > > > Jul 30, 2015 11:42:45 AM > > org.apache.tomcat.util.digester.SetPropertiesRule > > > begin > > > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting > > property > > > 'xmlValidation' to 'false' did not find a matching property. > > > Jul 30, 2015 11:42:45 AM > > org.apache.tomcat.util.digester.SetPropertiesRule > > > begin > > > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting > > property > > > 'xmlNamespaceAware' to 'false' did not find a matching property. > > > Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init > > > INFO: Initializing ProtocolHandler ["http-bio-8080"] > > > Jul 30, 2015 11:42:46 AM org.apache.coyote.AbstractProtocol init > > > INFO: Initializing ProtocolHandler ["http-bio-8443"] > > > Jul 30, 2015 11:42:49 AM org.apache.coyote.AbstractProtocol init > > > INFO: Initializing ProtocolHandler ["ajp-bio-8009"] > > > Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.Catalina load > > > INFO: Initialization processed in 18332 ms > > > Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardService > > > startInternal > > > INFO: Starting service Catalina > > > Jul 30, 2015 11:42:49 AM org.apache.catalina.core.StandardEngine > > > startInternal > > > INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 > > > Jul 30, 2015 11:42:49 AM org.apache.catalina.startup.HostConfig > > > deployDirectory > > > INFO: Deploying web application directory > > /var/lib/tomcat/webapps/gutterball > > > Jul 30, 2015 11:42:58 AM org.apache.catalina.loader.WebappClassLoader > > > validateJarFile > > > INFO: > > > > > validateJarFile(/usr/share/tomcat/webapps/gutterball/WEB-INF/lib/servlet-api-2.5.jar) > > > > > - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: > > > javax/servlet/Servlet.class > > > Jul 30, 2015 11:44:58 AM org.apache.catalina.util.SessionIdGenerator > > > createSecureRandom > > > INFO: Creation of SecureRandom instance for session ID generation using > > > [SHA1PRNG] took [107,531] milliseconds. > > > Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig > > > deployDirectory > > > INFO: Deployment of web application directory > > > /var/lib/tomcat/webapps/gutterball has finished in 134,231 ms > > > Jul 30, 2015 11:45:03 AM org.apache.catalina.startup.HostConfig > > > deployDirectory > > > INFO: Deploying web application directory > > /var/lib/tomcat/webapps/candlepin > > > Jul 30, 2015 11:45:14 AM org.apache.catalina.loader.WebappClassLoader > > > validateJarFile > > > INFO: > > > > > validateJarFile(/usr/share/tomcat/webapps/candlepin/WEB-INF/lib/servlet-api-2.5.jar) > > > > > - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: > > > javax/servlet/Servlet.class > > > Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory > > > WARNING: Method [protected org.candlepin.model.AbstractHibernateObject > > > > > org.candlepin.resource.ConsumerContentOverrideResource.findParentById(java.lang.String)] > > > > > is synthetic and is being intercepted by > > > [org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This > > could > > > indicate a bug. The method may be intercepted twice, or may not be > > > intercepted at all. > > > Jul 30, 2015 11:45:31 AM com.google.inject.internal.ProxyFactory > > > WARNING: Method [protected org.candlepin.model.AbstractHibernateObject > > > > > org.candlepin.resource.ActivationKeyContentOverrideResource.findParentById(java.lang.String)] > > > > > is synthetic and is being intercepted by > > > [org.candlepin.guice.CandlepinResourceTxnInterceptor@5abb6d06]. This > > could > > > indicate a bug. The method may be intercepted twice, or may not be > > > intercepted at all. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext > > > startInternal > > > SEVERE: Error listenerStart > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.core.StandardContext > > > startInternal > > > SEVERE: Context [/candlepin] startup failed due to previous errors > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesJdbc > > > SEVERE: The web application [/candlepin] registered the JDBC driver > > > [org.postgresql.Driver] but failed to unregister it when the web > > > application was stopped. To prevent a memory leak, the JDBC Driver has > > been > > > forcibly unregistered. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [com.google.inject.internal.util.$Finalizer] but has failed to > > stop > > > it. This is very likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [IoSender - foreman.rhel7.local/192.168.122.129:5671] but has > > failed > > > to stop it. This is very likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [IoReceiver - foreman.rhel7.local/192.168.122.129:5671] but has > > > failed to stop it. This is very likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [ack-flusher] but has failed to stop it. This is very likely to > > > create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [Timer-1] but has failed to stop it. This is very likely to create > > a > > > memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] > > but > > > has failed to stop it. This is very likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] > > but > > > has failed to stop it. This is very likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2] > > but > > > has failed to stop it. This is very likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-1] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-2] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-3] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-4] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-5] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-6] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-7] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-8] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-9] but has failed to stop it. This is very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-10] but has failed to stop it. This is > > very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-11] but has failed to stop it. This is > > very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-12] but has failed to stop it. This is > > very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-13] but has failed to stop it. This is > > very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-14] but has failed to stop it. This is > > very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_Worker-15] but has failed to stop it. This is > > very > > > likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > clearReferencesThreads > > > SEVERE: The web application [/candlepin] appears to have started a > > thread > > > named [QuartzScheduler_QuartzSchedulerThread] but has failed to stop it. > > > This is very likely to create a memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > checkThreadLocalMapForLeaks > > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > > of > > > type [org.apache.qpid.transport.network.Disassembler$1] (value > > > [org.apache.qpid.transport.network.Disassembler$1@639179fe]) and a value > > of > > > type [org.apache.qpid.transport.codec.BBEncoder] (value > > > [org.apache.qpid.transport.codec.BBEncoder@63e045c3]) but failed to > > remove > > > it when the web application was stopped. Threads are going to be renewed > > > over time to try and avoid a probable memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > checkThreadLocalMapForLeaks > > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > > of > > > type [org.apache.qpid.util.Strings$1] (value > > > [org.apache.qpid.util.Strings$1@1904386f]) and a value of type [char[]] > > > (value [[C@64150c3d]) but failed to remove it when the web application > > was > > > stopped. Threads are going to be renewed over time to try and avoid a > > > probable memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > checkThreadLocalMapForLeaks > > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > > of > > > type [com.google.inject.internal.InjectorImpl$1] (value > > > [com.google.inject.internal.InjectorImpl$1@21d6ac20]) and a value of > > type > > > [java.lang.Object[]] (value [[Ljava.lang.Object;@14be7a3b]) but failed > > to > > > remove it when the web application was stopped. Threads are going to be > > > renewed over time to try and avoid a probable memory leak. > > > Jul 30, 2015 11:45:43 AM org.apache.catalina.loader.WebappClassLoader > > > checkThreadLocalMapForLeaks > > > SEVERE: The web application [/candlepin] created a ThreadLocal with key > > of > > > ty

Peter_Verbist · July 29, 2015, 1:30pm

I'm having the same thing. Just installed it brand new as explained on
http://www.katello.org/docs/2.2/installation/index.html running centos7.1

After default installation, logged in and added 1 small repos (puppet
dependencies). Sync works fine.

foreman-rake katello:clean_backend_objects

–> runs fine

katello-service stop

katello-service start

–> checking and Product page spinner just never stops

foreman-rake katello:clean_backend_objects

Performance on MRI may be improved with the concurrent-ruby-ext gem. Please
see http://concurrent-ruby.com

rake aborted!

Katello::Resources::Candlepin::Consumer: 404 Resource Not Found (GET
/candlepin/consumers/?)

Tasks: TOP => katello:clean_backend_objects

(See full trace by running task with --trace)

hammer product list --organization-id 1

Katello::Resources::Candlepin::Product: 404 Resource Not Found (GET
/candlepin/products/1438174202256)

Snippet from /var/log/foreman/production.log

2015-07-29 14:55:29 [app] [E] exception when talking to a remote client:
Katello::Resources::Candlepin::Product: 404 Resource Not Found (GET
/candlepin/products/1438174202256) RestClient::ResourceNotFound:
Katello::Resources::Candlepin::Product: 404 Resource Not Found (GET
/candlepin/products/1438174202256)

> Body:

>
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
`return!'

>
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in
`process_result'

>
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.35/lib/restclient_ext/request.rb:50:in
`block in transmit'

…

Checked with selinux enabled and disabled

···

On Tuesday, July 28, 2015 at 9:59:37 AM UTC+2, Ivan Necas wrote: > > The listen on candlepin action is a long-running tasks, that is expected > to be in this state and > it itself is not indication of any error > > -- Ivan > > ----- Original Message ----- > > No time to investigate properly at the moment, but the second task in > the > > entire history is still showing as Pending: > > > > Listen on candlepin events {"locale"=>"en"} > > State: running > > Result: Pending > > Started at: 2015-07-24 20:00:14 UTC > > > > The only event before it is the "Create organisation 'Default > > Organization'" (odd mix of UK English and US English there with > > organi(s|z)ation BTW) > > > > The next event is me starting to create the Product and repositories 20 > > minutes later. > > > > Don't remember looking at this before, but should I expect the "Listen > on > > candlepin events" task to be in this state? Perhaps my network switch > > isn't what caused my problem? > > > > D > > > > On Monday, 27 July 2015 10:20:15 UTC+1, Ivan Necas wrote: > > > > > > > > > > > > > > > ------------------------------ > > > > > > Thanks Ivan, I'll look into the troubleshooting page. From what I > noticed > > > myself, though, the tasks that are flagged as locking my repos don't > allow > > > me to click the resume button. I'll investigate more tonight though. > > > > > > > > > If the state of the tasks is 'running', look at the 'runnings steps' > tab: > > > the details from there will be valuable to find > > > out what's going on and what't not working… My guess is some pulp task > not > > > getting stareted, but I will wait for more details. > > > > > > -- Ivan > > > > > > > > > I noticed the DynFlow problem had been raised while I was writing my > > > message. Will leave that part of it for now. > > > > > > Duncan > > > > > > On Monday, 27 July 2015 10:00:59 UTC+1, Ivan Necas wrote: > > >> > > >> Seems like some issue with the candlepin service: Eric? > > >> > > >> Ad. locked tasks: you need to resume the tasks that are causing the > locks > > >> (the details for the locks are in the error message), > > >> http://www.katello.org/troubleshooting/index.html has some details > on > > >> how it works. > > >> > > >> But first, you would need to fix the candlepin: please check > > >> /var/log/tomcat6/catalina.out for anything interesting… especially, > > >> when restarting tomcat6 service). > > >> > > >> The "uninitialized constant Dynflow::Executors::RemoteViaSocket" has > > >> already an issue filed and patch sent > > >> http://projects.theforeman.org/issues/11215 > > >> > > >> -- Ivan > > >> > > >> ------------------------------ > > >> > > >> Not sure how I got into this mess, but I appear to have messed up my > > >> fresh Katello 2.3 RC install with very little in the way of poking > and > > >> prodding. > > >> > > >> OS - CentOS 7.1 (running in a libvirt guest) > > >> Katello 2.3 RC & Foreman 1.9 RC 2 > > >> > > >> Created Product for CentOS > > >> - Created repo for CentOS 7 > > >> - Created repo for CentOS 7 Updates > > >> - Created repo for CentOS 7 Extras > > >> > > >> Started synching all 3 repos and things seemed to be going well (Sync > > >> Status page was showing progress on all 3 syncs). During the sync, > laptop > > >> started running low on battery, so I stuck it in docking station and > > >> switched it's wired network on and turned off the wireless network. > I > > >> think this network switch may be the start of the problems. All 3 > repo > > >> syncs then stopped and gave errors (perhaps not surprising). > > >> > > >> I've managed to poke the CentOS 7 repo into a state where I could > start > > >> another sync. > > >> > > >> I thought I'd done the same thing with the Updates and Extras repos, > but > > >> there are locks that I cannot find out how to remove. I'm also > seeing > > >> failures in the About page: > > >> > > >> ComponentStatusMessagecandlepinFAIL404 Resource Not > Foundcandlepin_auth > > >> FAILKatello::Resources::Candlepin::CandlepinPing: 404 Resource Not > Found > > >> (GET /candlepin/status)elasticsearchOKforeman_tasksFAILuninitialized > > >> constant Dynflow::Executors::RemoteViaSocketpulpOKpulp_authOK > > >> > > >> > > >> There are currently pending syncs for the Extras and Updates repos > which > > >> I can't stop, resume, cancel, delete, etc. I also can't see the > Product > > >> page as the spinner just sits in the centre of the page and never > produces > > >> and data to see. > > >> > > >> Any help greatly appreciated. Don't mind wiping the setup and > starting > > >> again, but I feel that I've got into this state rather too easily and > that > > >> I can't recover the system simply. If this happened on a production > > >> system, I wouldn't be willing to wipe and reset. > > >> -- > > >> You received this message because you are subscribed to the Google > Groups > > >> "Foreman users" group. > > >> To unsubscribe from this group and stop receiving emails from it, > send an > > >> email to foreman-user...@googlegroups.com . > > > >> To post to this group, send email to forema...@googlegroups.com > > >> . > > >> Visit this group at http://groups.google.com/group/foreman-users. > > >> For more options, visit https://groups.google.com/d/optout. > > >> > > >> > > >> > > > >

Duncan_Innes · July 30, 2015, 12:47pm

candlepin.log

2015-07-30 11:45:29,847 [=, org=] INFO
org.candlepin.guice.CandlepinContextListener - Candlepin initializing
context.
2015-07-30 11:45:29,854 [=, org=] INFO
org.candlepin.guice.CandlepinContextListener - Candlepin reading
configuration.
2015-07-30 11:45:29,875 [=, org=] INFO
org.candlepin.common.config.EncryptedConfiguration - No secret file
provided.
2015-07-30 11:45:30,097 [=, org=] INFO
org.candlepin.guice.CustomizableModules - Found custom module
module.config.adapter_module
2015-07-30 11:45:33,691 [=, org=] INFO
org.candlepin.guice.AMQPBusPubProvider - AMQP connection factory created.
2015-07-30 11:45:34,432 [=, org=] WARN
org.apache.qpid.transport.network.security.ssl.SSLUtil - Exception
received while trying to verify hostname
2015-07-30 11:45:34,577 [=, org=] INFO
org.candlepin.guice.AMQPBusPubProvider - AMQP session created
successfully…
2015-07-30 11:45:36,148 [=, org=] WARN
com.mchange.v2.c3p0.management.ActiveManagementCoordinator - A
C3P0Registry mbean is already registered. This probably means that an
application using c3p0 was undeployed, but not all PooledDataSources were
closed prior to undeployment. This may lead to resource leaks over time.
Please take care to close all PooledDataSources.
2015-07-30 11:45:39,259 [=, org=] WARN org.hibernate.id.UUIDHexGenerator -
HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate
IETF RFC 4122 compliant UUID values; consider using
org.hibernate.id.UUIDGenerator instead
2015-07-30 11:45:43,670 [=, org=] ERROR
org.candlepin.guice.CandlepinGuiceResteasyBootstrap - Could not create
Guice injector.
com.google.inject.CreationException: Guice creation errors:

Error in custom provider, java.lang.RuntimeException:
javax.jms.JMSException: Error creating producer
while locating org.candlepin.guice.AMQPBusPubProvider
at
org.candlepin.guice.CandlepinModule.configureAmqp(CandlepinModule.java:349)
while locating org.candlepin.audit.AMQPBusPublisher

1 error
at
com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:183)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
~[guice-3.0.jar:na]
at com.google.inject.Guice.createInjector(Guice.java:95) ~[guice-3.0.jar:na]
at
org.candlepin.guice.CandlepinContextListener.getInjector(CandlepinContextListener.java:146)
[CandlepinContextListener.class:na]
at
org.candlepin.guice.CandlepinGuiceResteasyBootstrap.contextInitialized(CandlepinGuiceResteasyBootstrap.java:57)
~[CandlepinGuiceResteasyBootstrap.class:na]
at
org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:114)
[CandlepinContextListener.class:na]
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
[catalina.jar:7.0.54]
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
[catalina.jar:7.0.54]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
[catalina.jar:7.0.54]
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
[catalina.jar:7.0.54]
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
[catalina.jar:7.0.54]
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
[catalina.jar:7.0.54]
at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1247)
[catalina.jar:7.0.54]
at
org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1898)
[catalina.jar:7.0.54]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_85]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_85]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
Caused by: java.lang.RuntimeException: javax.jms.JMSException: Error
creating producer
at org.candlepin.guice.AMQPBusPubProvider.get(AMQPBusPubProvider.java:168)
~[AMQPBusPubProvider.class:na]
at org.candlepin.guice.AMQPBusPubProvider.get(AMQPBusPubProvider.java:50)
~[AMQPBusPubProvider.class:na]
at
com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
~[guice-3.0.jar:na]
at
com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
~[guice-3.0.jar:na]
at
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
~[guice-3.0.jar:na]
at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
~[guice-3.0.jar:na]
… 18 common frames omitted
Caused by: javax.jms.JMSException: Error creating producer
at org.apache.qpid.client.AMQSession.toJMSException(AMQSession.java:3719)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:696)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:69)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession$7.execute(AMQSession.java:2791)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession$7.execute(AMQSession.java:2782)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnectionDelegate_0_10.executeRetrySupport(AMQConnectionDelegate_0_10.java:371)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnection.executeRetrySupport(AMQConnection.java:652)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.failover.FailoverRetrySupport.execute(FailoverRetrySupport.java:96)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession.createProducerImpl(AMQSession.java:2780)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession.createProducer(AMQSession.java:1316)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession.createPublisher(AMQSession.java:1323)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQTopicSessionAdaptor.createPublisher(AMQTopicSessionAdaptor.java:46)
~[qpid-client-0.30.jar:0.30]
at
org.candlepin.guice.AMQPBusPubProvider.storeTopicProducer(AMQPBusPubProvider.java:196)
~[AMQPBusPubProvider.class:na]
at
org.candlepin.guice.AMQPBusPubProvider.buildAllTopicPublishers(AMQPBusPubProvider.java:184)
~[AMQPBusPubProvider.class:na]
at org.candlepin.guice.AMQPBusPubProvider.get(AMQPBusPubProvider.java:164)
~[AMQPBusPubProvider.class:na]
… 30 common frames omitted
Caused by: org.apache.qpid.AMQException: Exception occured while verifying
destination
at
org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:96)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.BasicMessageProducer.<init>(BasicMessageProducer.java:139)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.BasicMessageProducer_0_10.<init>(BasicMessageProducer_0_10.java:65)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:691)
~[qpid-client-0.30.jar:0.30]
… 43 common frames omitted
Caused by: org.apache.qpid.AMQException: The name 'event' supplied in the
address doesn't resolve to an exchange or a queue
at org.apache.qpid.client.AMQSession.resolveAddress(AMQSession.java:722)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:90)
~[qpid-client-0.30.jar:0.30]
… 46 common frames omitted

error.log

2015-07-30 11:45:34,432 [=, org=] WARN
org.apache.qpid.transport.network.security.ssl.SSLUtil - Exception
received while trying to verify hostname
2015-07-30 11:45:36,148 [=, org=] WARN
com.mchange.v2.c3p0.management.ActiveManagementCoordinator - A
C3P0Registry mbean is already registered. This probably means that an
application using c3p0 was undeployed, but not all PooledDataSources were
closed prior to undeployment. This may lead to resource leaks over time.
Please take care to close all PooledDataSources.
2015-07-30 11:45:39,259 [=, org=] WARN org.hibernate.id.UUIDHexGenerator -
HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate
IETF RFC 4122 compliant UUID values; consider using
org.hibernate.id.UUIDGenerator instead
2015-07-30 11:45:43,670 [=, org=] ERROR
org.candlepin.guice.CandlepinGuiceResteasyBootstrap - Could not create
Guice injector.
com.google.inject.CreationException: Guice creation errors:

Error in custom provider, java.lang.RuntimeException:
javax.jms.JMSException: Error creating producer
while locating org.candlepin.guice.AMQPBusPubProvider
at
org.candlepin.guice.CandlepinModule.configureAmqp(CandlepinModule.java:349)
while locating org.candlepin.audit.AMQPBusPublisher

1 error
at
com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:183)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
~[guice-3.0.jar:na]
at com.google.inject.Guice.createInjector(Guice.java:95) ~[guice-3.0.jar:na]
at
org.candlepin.guice.CandlepinContextListener.getInjector(CandlepinContextListener.java:146)
[CandlepinContextListener.class:na]
at
org.candlepin.guice.CandlepinGuiceResteasyBootstrap.contextInitialized(CandlepinGuiceResteasyBootstrap.java:57)
~[CandlepinGuiceResteasyBootstrap.class:na]
at
org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:114)
[CandlepinContextListener.class:na]
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
[catalina.jar:7.0.54]
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
[catalina.jar:7.0.54]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
[catalina.jar:7.0.54]
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
[catalina.jar:7.0.54]
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
[catalina.jar:7.0.54]
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
[catalina.jar:7.0.54]
at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1247)
[catalina.jar:7.0.54]
at
org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1898)
[catalina.jar:7.0.54]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_85]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_85]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
Caused by: java.lang.RuntimeException: javax.jms.JMSException: Error
creating producer
at org.candlepin.guice.AMQPBusPubProvider.get(AMQPBusPubProvider.java:168)
~[AMQPBusPubProvider.class:na]
at org.candlepin.guice.AMQPBusPubProvider.get(AMQPBusPubProvider.java:50)
~[AMQPBusPubProvider.class:na]
at
com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
~[guice-3.0.jar:na]
at
com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
~[guice-3.0.jar:na]
at
com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
~[guice-3.0.jar:na]
at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
~[guice-3.0.jar:na]
at
com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
~[guice-3.0.jar:na]
… 18 common frames omitted
Caused by: javax.jms.JMSException: Error creating producer
at org.apache.qpid.client.AMQSession.toJMSException(AMQSession.java:3719)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:696)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:69)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession$7.execute(AMQSession.java:2791)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession$7.execute(AMQSession.java:2782)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnectionDelegate_0_10.executeRetrySupport(AMQConnectionDelegate_0_10.java:371)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQConnection.executeRetrySupport(AMQConnection.java:652)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.failover.FailoverRetrySupport.execute(FailoverRetrySupport.java:96)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession.createProducerImpl(AMQSession.java:2780)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession.createProducer(AMQSession.java:1316)
~[qpid-client-0.30.jar:0.30]
at org.apache.qpid.client.AMQSession.createPublisher(AMQSession.java:1323)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQTopicSessionAdaptor.createPublisher(AMQTopicSessionAdaptor.java:46)
~[qpid-client-0.30.jar:0.30]
at
org.candlepin.guice.AMQPBusPubProvider.storeTopicProducer(AMQPBusPubProvider.java:196)
~[AMQPBusPubProvider.class:na]
at
org.candlepin.guice.AMQPBusPubProvider.buildAllTopicPublishers(AMQPBusPubProvider.java:184)
~[AMQPBusPubProvider.class:na]
at org.candlepin.guice.AMQPBusPubProvider.get(AMQPBusPubProvider.java:164)
~[AMQPBusPubProvider.class:na]
… 30 common frames omitted
Caused by: org.apache.qpid.AMQException: Exception occured while verifying
destination
at
org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:96)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.BasicMessageProducer.<init>(BasicMessageProducer.java:139)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.BasicMessageProducer_0_10.<init>(BasicMessageProducer_0_10.java:65)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:691)
~[qpid-client-0.30.jar:0.30]
… 43 common frames omitted
Caused by: org.apache.qpid.AMQException: The name 'event' supplied in the
address doesn't resolve to an exchange or a queue
at org.apache.qpid.client.AMQSession.resolveAddress(AMQSession.java:722)
~[qpid-client-0.30.jar:0.30]
at
org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:90)
~[qpid-client-0.30.jar:0.30]
… 46 common frames omitted

···

On Thursday, 30 July 2015 13:28:13 UTC+1, Ivan Necas wrote: > > Oh, we need to dig deeper than (calling for some Candlepin devels to help > with debugging). > > In the meantime, what about /var/log/candlepin/{candlepin.log,error.log}, > anything interesting there? > > -- Ivan > > ----- Original Message ----- > > This system doesn't have java-1.6 installed: > > > > [root@foreman tomcat]# rpm -qa | grep java | sort > > java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64 > > java-1.7.0-openjdk-headless-1.7.0.85-2.6.1.2.el7_1.x86_64 > > java-client-1.0-1.noarch > > javamail-1.4.6-8.el7.noarch > > javapackages-tools-3.4.1-6.el7_0.noarch > > python-javapackages-3.4.1-6.el7_0.noarch > > snappy-java-1.0.4-1.el7.x86_64 > > tzdata-java-2015e-1.el7.noarch > > [root@foreman tomcat]# > > > > > > On Thursday, 30 July 2015 12:24:53 UTC+1, Ivan Necas wrote: > > > > > > That was very helpful, the important error message here is "Error > > > listenerStart", > > > I was able to reproduce the behavior, when I installed jdk 1.6 and > made > > > that the default, using `alternatives`. > > > > > > The solution for this issue should be making sure java-1.7.0-openjdk > is > > > installed, > > > making it a default using `alternatives --config java`, restarting > > > the tomcat service, checking with `hammer ping` that it helped and > > > proceeding with the upgrade procedure. > > > > > > -- Ivan > > > > > > > > > > > > ----- Original Message ----- > > > > Sorry for delay - got distracted by my daytime job. > > > > > > > > It this any help? > > > > > > > > [root@foreman tomcat]# ll > > > > total 164 > > > > -rw-r--r--. 1 tomcat

Alex_Wood · July 30, 2015, 5:48pm

>
> Caused by: org.apache.qpid.AMQException: Exception occured while verifying
> destination
> at
> org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:96)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.BasicMessageProducer.<init>(BasicMessageProducer.java:139)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.BasicMessageProducer_0_10.<init>(BasicMessageProducer_0_10.java:65)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:691)
> ~[qpid-client-0.30.jar:0.30]
> … 43 common frames omitted
> Caused by: org.apache.qpid.AMQException: The name 'event' supplied in the
> address doesn't resolve to an exchange or a queue
> at org.apache.qpid.client.AMQSession.resolveAddress(AMQSession.java:722)
> ~[qpid-client-0.30.jar:0.30]
> at
> org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:90)
> ~[qpid-client-0.30.jar:0.30]
> … 46 common frames omitted
>
>
To me, that message has the hallmarks of a missing QPid exchange. Would
you mind running a few commands for me?

$ sudo yum install qpid-tools
$ sudo qpid-config -b amqps://localhost:5671 --ssl-certificate
/etc/pki/katello/certs/java-client.crt --ssl-key
/etc/pki/katello/private/java-client.key exchanges

In that output do you see an exchange named 'event' with the '–durable'
attribute set? If not, then something has gone awry. If you are just
interested in moving forward, you can create the exchange by hand like so

$ sudo qpid-config -b amqps://localhost:5671 --ssl-certificate
/etc/pki/katello/certs/java-client.crt --ssl-key
/etc/pki/katello/private/java-client.key add exchange topic "event"
–durable

But the installer should have handled this for you. If you find anything
in the installer logs (under /var/log/katello-installer) about failing to
create the exchange or anything similar, that would be useful for us.

Once the exchange is created, restarting Tomcat should be sufficient to get
Candlepin online.

If the qpid-config commands fail with something like

Failed: ConnectError: [Errno 1] _ssl.c:492: error:14094412:SSL
routines:SSL3_READ_BYTES:sslv3 alert bad certificate

then there is a problem with the certificates that Katello generated and
we'll have to do some more investigation.

···

---------- Regards, Alex

River_Senyurt · July 29, 2015, 6:51pm

Hi all,

Same is happening to me./
Fresh install. 2.3 on Centos 7.
Added one repo (puppetforge Which apparently failes to sync even following
katello instructions)
Now after reboot
candlepin FAIL 404 Resource Not Found candlepin_auth
FAIL Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found
(GET /candlepin/status)
pulp_celerybeat.service loaded failed failed Pulp's Celerybeat
pulp_resource_manager.service loaded failed failed Pulp Resource Manager
pulp_worker-0.service loaded failed failed Pulp Worker #0
pulp_worker-1.service loaded failed failed Pulp Worker #1

Assure you i did nothing . just fresh install add small repo and boom.

Anyone has any ideas ?
And

Duncan_Innes · July 31, 2015, 8:37am

[root@foreman ~]# qpid-config -b amqps://localhost:5671 --ssl-certificate
/etc/pki/katello/certs/java-client.crt --ssl-key
/etc/pki/katello/private/java-client.key exchanges
Type Exchange Name Attributes

···

On Thursday, 30 July 2015 18:58:16 UTC+1, aw...@redhat.com wrote: > > Caused by: org.apache.qpid.AMQException: Exception occured while verifying >> destination >> at >> org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:96) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.apache.qpid.client.BasicMessageProducer.(BasicMessageProducer.java:139) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.apache.qpid.client.BasicMessageProducer_0_10.(BasicMessageProducer_0_10.java:65) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.apache.qpid.client.AMQSession_0_10.createMessageProducer(AMQSession_0_10.java:691) >> ~[qpid-client-0.30.jar:0.30] >> ... 43 common frames omitted >> Caused by: org.apache.qpid.AMQException: The name 'event' supplied in the >> address doesn't resolve to an exchange or a queue >> at org.apache.qpid.client.AMQSession.resolveAddress(AMQSession.java:722) >> ~[qpid-client-0.30.jar:0.30] >> at >> org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:90) >> ~[qpid-client-0.30.jar:0.30] >> ... 46 common frames omitted >> >> > To me, that message has the hallmarks of a missing QPid exchange. Would > you mind running a few commands for me? > > $ sudo yum install qpid-tools > $ sudo qpid-config -b amqps://localhost:5671 --ssl-certificate > /etc/pki/katello/certs/java-client.crt --ssl-key > /etc/pki/katello/private/java-client.key exchanges > > In that output do you see an exchange named 'event' with the '--durable' > attribute set? If not, then something has gone awry. >

==================================================
direct --replicate=none
direct C.dq --durable
direct amq.direct --durable --replicate=none
fanout amq.fanout --durable --replicate=none
headers amq.match --durable --replicate=none
topic amq.topic --durable --replicate=none
direct celery --durable
fanout celery.pidbox
topic celeryev --durable
direct qmf.default.direct --replicate=none
topic qmf.default.topic --replicate=none
topic qpid.management --replicate=none
direct resource_manager --durable
[root@foreman ~]#

If you are just interested in moving forward, you can create the exchange
by hand like so

$ sudo qpid-config -b amqps://localhost:5671 --ssl-certificate
/etc/pki/katello/certs/java-client.crt --ssl-key
/etc/pki/katello/private/java-client.key add exchange topic “event”
–durable

[root@foreman ~]# qpid-config -b amqps://localhost:5671 --ssl-certificate
/etc/pki/katello/certs/java-client.crt --ssl-key
/etc/pki/katello/private/java-client.key add exchange topic “event”
–durable
[root@foreman ~]# qpid-config -b amqps://localhost:5671 --ssl-certificate
/etc/pki/katello/certs/java-client.crt --ssl-key
/etc/pki/katello/private/java-client.key exchanges
Type Exchange Name Attributes

direct --replicate=none
direct C.dq --durable
direct amq.direct --durable --replicate=none
fanout amq.fanout --durable --replicate=none
headers amq.match --durable --replicate=none
topic amq.topic --durable --replicate=none
direct celery --durable
fanout celery.pidbox
topic celeryev --durable
topic event --durable
direct qmf.default.direct --replicate=none
topic qmf.default.topic --replicate=none
topic qpid.management --replicate=none
direct resource_manager --durable
[root@foreman ~]#

But the installer should have handled this for you. If you find anything
in the installer logs (under /var/log/katello-installer) about failing to
create the exchange or anything similar, that would be useful for us.

I’ll have a look, but not sure what I’m looking for. I posted the log
files to:

/var/log/katello-installer/katello-installer.log -
http://pastebin.com/bj6AqyFe
/var/log/katello-installer/katello-installer.1.log -
http://pastebin.com/1xAqPexp
/var/log/katello-installer/katello-installer.2.log - zero length file

Once the exchange is created, restarting Tomcat should be sufficient to get

Candlepin online.

The Foreman/about page now shows Candlepin as status OK in the Backend
System Status section. Can’t untangle the mess of various tasks that I’ve
started, cancelled, pending, etc, so removing the repositories and creating
them again.

If the qpid-config commands fail with something like

Failed: ConnectError: [Errno 1] _ssl.c:492: error:14094412:SSL
routines:SSL3_READ_BYTES:sslv3 alert bad certificate

then there is a problem with the certificates that Katello generated and
we’ll have to do some more investigation.

Regards,
Alex

Cheers,

Duncan

Duncan_Innes · August 1, 2015, 7:54pm

I got this to replicate yesterday. Fresh CentOS 7 install, repo installs
etc done during kickstart. katello-installer run in ssh session as soon as
system booted.

Once installed, created product, created CentOS 7 repo, started sync, sync
started, sync stopped. I did put the latpop back into the docking station
which switched it from wifi access to RJ45 access to the router - the same
as I did the first time. Couldn't get the repo to start it's sync again
under any circumstances.

Am now reinstalling again to try and narrow down where the issue might be.
Some of the repos (or my ISP) are playing up. So I had to install the
repos and yum -y install katello via my first SSH session. One thing I
noticed this time is a bunch of warnings during the gutterball install:

Installing : gutternall-1.0.16-1.el7.x86_64 279/431
warning: user tomcat does not exist - using root
warning: group tomcat does not exist - using root
(repeated around 180 times)

Never noticed it during the previous installs as this normally happens
during my kickstart. I don't check the log from the kickstart file - just
run katello-installer when I log in.

I'll see how this install goes and then report back.

D

Duncan_Innes · August 4, 2015, 1:03pm

Hi,

I've replicated this over the weekend and have another vanilla install
which has hung on the repo sync. This time, however, I'm pretty sure there
were no network changes on the host laptop until the repos has run their
first sync. I ran a second sync because the first one didn't grab all the
packages. A third sync has seen 2 of the repos failing to do anything.
The jobs are just sitting there.

I ran the qpid command to list the exchanges and am getting:
[root@foreman ~]# qpid-config -b amqps://localhost:5671 --ssl-certificate
/etc/pki/katello/certs/java-client.crt --ssl-key
/etc/pki/katello/private/java-client.key exchanges
Type Exchange Name Attributes

···

================================================== direct --replicate=none direct C.dq --durable direct amq.direct --durable --replicate=none fanout amq.fanout --durable --replicate=none headers amq.match --durable --replicate=none topic amq.topic --durable --replicate=none direct celery --durable fanout celery.pidbox topic celeryev --durable direct qmf.default.direct --replicate=none topic qmf.default.topic --replicate=none topic qpid.management --replicate=none direct resource_manager --durable [root@foreman ~]#

i.e. the ‘event’ one you mentioned previously is missing.

Duncan

Alex_Wood · August 4, 2015, 1:53pm

Duncan,

Thanks for doing the legwork on this. The issue you're hitting
definitely sounds like a bug to me.

Would you mind filing a bug at
Foreman and just
filling in the basic details? (If you don't have an account, you can
just email me the report directly and I'll file it. Mostly we just
need to know the Katello version, how the problem manifests itself,
and an attachment of the installer log under /var/log/katello-installer.)

Thanks for your patience with this.

···

On 08/04/2015 09:03 AM, Duncan Innes wrote: > Hi, > > I've replicated this over the weekend and have another vanilla > install which has hung on the repo sync. This time, however, I'm > pretty sure there were no network changes on the host laptop until > the repos has run their first sync. I ran a second sync because > the first one didn't grab all the packages. A third sync has seen > 2 of the repos failing to do anything. The jobs are just sitting > there. > > I ran the qpid command to list the exchanges and am getting: > [root@foreman ~]# qpid-config -b amqps://localhost:5671 > --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key > /etc/pki/katello/private/java-client.key exchanges Type > Exchange Name Attributes > ================================================== direct > --replicate=none direct C.dq --durable direct > amq.direct --durable --replicate=none fanout amq.fanout > --durable --replicate=none headers amq.match --durable > --replicate=none topic amq.topic --durable > --replicate=none direct celery --durable fanout > celery.pidbox topic celeryev --durable direct > qmf.default.direct --replicate=none topic qmf.default.topic > --replicate=none topic qpid.management --replicate=none > direct resource_manager --durable [root@foreman ~]# > > i.e. the 'event' one you mentioned previously is missing. > > Duncan

Regards,
Alex

–
Senior Software Engineer
Red Hat
919.754.4445

Duncan_Innes · August 4, 2015, 2:37pm

Alex,

Ticket created: Bug #11285: Katello 2.3 Repo Sync issue - Katello - Foreman

I put as much information as I thought was relevant. Will keep this virt
guest around to collect more details if necessary.

Duncan

Katello 2.3 RC repo sync issue

foreman-rake katello:clean_backend_objects

katello-service stop

katello-service start

foreman-rake katello:clean_backend_objects

hammer product list --organization-id 1

then there is a problem with the certificates that Katello generated and we’ll have to do some more investigation.

then there is a problem with the certificates that Katello generated and
we’ll have to do some more investigation.