[Katello] 2.1 create new product does not finish

Alvin_Johnson · February 19, 2015, 6:18pm

I been running into problem for the last week.
I have install the new Katello 2.1.
And in two or my katello installation both install have the same problem.
I have create a new Product Centos Server Linux
I create the Product after that have url to do repo discovery.
It never complete the pulp_task,
It just said state => waiting.
below that task is Listen on candlepin events {"locale"=>"en"}
The process never completes.

Below is the set that has been waiting.
4cup
12 GB ram
500GB storage.

I have create a foreman-debug for you to see.
foreman-debug-HAq0L.tar.xz
I am having the same problem in my new AWS installation Katello 2.1 doing
the same thing.

Thanks Al

Jorick_Astrego · February 20, 2015, 9:54am

Any errors in "/var/log/candlepin/error.log" or "/var/log/messages"?

Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts

···

On 02/19/2015 07:18 PM, Alvin Johnson wrote: > I been running into problem for the last week. > I have install the new Katello 2.1. > And in two or my katello installation both install have the same problem. > I have create a new Product Centos Server Linux > I create the Product after that have url to do repo discovery. > It never complete the pulp_task, > It just said state => waiting. > below that task is Listen on candlepin events {"locale"=>"en"} > The process never completes. > > Below is the set that has been waiting. > 4cup > 12 GB ram > 500GB storage. > > I have create a foreman-debug for you to see. > foreman-debug-HAq0L.tar.xz > I am having the same problem in my new AWS installation Katello 2.1 > doing the same thing. > > Thanks Al > ----------------

Tel: 053 20 30 270 	info@netbulae.eu 	Staalsteden 4-3A 	KvK 08198180
Fax: 053 20 30 271 	www.netbulae.eu 	7547 TA Enschede 	BTW NL821234584B01

Alvin_Johnson · February 20, 2015, 8:34pm

No error in Messages but here the out from candleping/error.log
2015-02-17 19:23:21,773 [=, org=] ERROR
org.apache.qpid.client.AMQConnectionDelegate_0_10 - connection exception:
conn:62e561a7
org.apache.qpid.transport.ConnectionException: connection aborted
at org.apache.qpid.transport.Connection.closed(Connection.java:563)
~[qpid-common-0.22.jar:na]
at
org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112)
[qpid-common-0.22.jar:na]
at
org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:204)
[qpid-common-0.22.jar:na]
at
org.apache.qpid.transport.network.security.ssl.SSLReceiver.closed(SSLReceiver.java:66)
[qpid-common-0.22.jar:na]
at
org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:222)
[qpid-common-0.22.jar:na]

···

On Thursday, February 19, 2015 at 12:18:27 PM UTC-6, Alvin Johnson wrote: > > I been running into problem for the last week. > I have install the new Katello 2.1. > And in two or my katello installation both install have the same problem. > I have create a new Product Centos Server Linux > I create the Product after that have url to do repo discovery. > It never complete the pulp_task, > It just said state => waiting. > below that task is Listen on candlepin events {"locale"=>"en"} > The process never completes. > > Below is the set that has been waiting. > 4cup > 12 GB ram > 500GB storage. > > I have create a foreman-debug for you to see. > foreman-debug-HAq0L.tar.xz > I am having the same problem in my new AWS installation Katello 2.1 doing > the same thing. > > Thanks Al > >

Alvin_Johnson · February 23, 2015, 9:59pm

Jorick,

Still not fixed at all here is the latest output.

Started at: 2015-02-23 21:39:55 UTC

Ended at: 2015-02-23 21:59:05 UTC

Real time: 1150.02s

Execution time (excluding suspended state): 8.83s

Input:

···

--- pulp_id: AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64 distributor_type_id: yum_distributor source_pulp_id: dependency: remote_user: ajohnson2-58a818a8 remote_cp_user: ajohnson2 locale: en

Output:

pulp_tasks:- exception:
task_type: pulp.server.managers.repo.publish.publish
_href: /pulp/api/v2/tasks/00a33deb-6b63-4fd7-b013-ddf39aebdacb/
task_id: 00a33deb-6b63-4fd7-b013-ddf39aebdacb
tags:

pulp:repository:AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64
pulp:action:publish
finish_time:
start_time:
traceback:
spawned_tasks: []
progress_report: {}
queue: reserved_resource_worker-1@til-vm-swk6-11.aircell.prod.dq
state: waiting
result:
error:
_id:
$oid: 54eb9e2c9431b63575bec426
id: 54eb9e2ce4fd854da18c3d6d
poll_attempts:
total: 91
failed: 0

On Thursday, February 19, 2015 at 12:18:27 PM UTC-6, Alvin Johnson wrote:

I been running into problem for the last week.
I have install the new Katello 2.1.
And in two or my katello installation both install have the same problem.
I have create a new Product Centos Server Linux
I create the Product after that have url to do repo discovery.
It never complete the pulp_task,
It just said state => waiting.
below that task is Listen on candlepin events {“locale”=>“en”}
The process never completes.

Below is the set that has been waiting.
4cup
12 GB ram
500GB storage.

I have create a foreman-debug for you to see.
foreman-debug-HAq0L.tar.xz
I am having the same problem in my new AWS installation Katello 2.1 doing
the same thing.

Thanks Al

Jorick_Astrego · February 20, 2015, 11:50pm

I've been experiencing these too. Doing a "service tomcat6 restart"
fixes it for a while.

Caused by: org.apache.qpid.transport.TransportException: Error
connecting to broker
    at
org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:86)
~[qpid-common-0.22.jar:na]
    at
org.apache.qpid.transport.Connection.connect(Connection.java:232)
~[qpid-common-0.22.jar:na]
    at
org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221)
~[qpid-client-0.22.jar:na]
    at
org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:619)
~[qpid-client-0.22.jar:na]
    at
org.apache.qpid.client.AMQConnection.&lt;init&gt;(AMQConnection.java:398)
~[qpid-client-0.22.jar:na]
    ... 47 common frames omitted
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
~[na:1.7.0_75]
    at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
~[na:1.7.0_75]
    at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
~[na:1.7.0_75]
    at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
~[na:1.7.0_75]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[na:1.7.0_75]
    at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_75]
    at
org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:82)
~[qpid-common-0.22.jar:na]
    ... 51 common frames omitted
2015-02-19 12:54:26,477 [=, org=] ERROR
org.apache.qpid.client.AMQConnectionDelegate_0_10 - connection
exception: conn:63d8fdbe
org.apache.qpid.transport.ConnectionException: connection aborted
    at
org.apache.qpid.transport.Connection.closed(Connection.java:563)
~[qpid-common-0.22.jar:na]
    at
org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112)
[qpid-common-0.22.jar:na]
    at
org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:204)
[qpid-common-0.22.jar:na]
    at
org.apache.qpid.transport.network.security.ssl.SSLReceiver.closed(SSLReceiver.java:66)
[qpid-common-0.22.jar:na]
    at
org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:222)
[qpid-common-0.22.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

···

On 02/20/2015 09:34 PM, Alvin Johnson wrote: > No error in Messages but here the out from candleping/error.log > 2015-02-17 19:23:21,773 [=, org=] ERROR > org.apache.qpid.client.AMQConnectionDelegate_0_10 - connection > exception: conn:62e561a7 > org.apache.qpid.transport.ConnectionException: connection aborted > at > org.apache.qpid.transport.Connection.closed(Connection.java:563) > ~[qpid-common-0.22.jar:na] > at > org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112) > [qpid-common-0.22.jar:na] > at > org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:204) > [qpid-common-0.22.jar:na] > at > org.apache.qpid.transport.network.security.ssl.SSLReceiver.closed(SSLReceiver.java:66) > [qpid-common-0.22.jar:na] > at > org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:222) > [qpid-common-0.22.jar:na] > > > > > > On Thursday, February 19, 2015 at 12:18:27 PM UTC-6, Alvin Johnson wrote: > > I been running into problem for the last week. > I have install the new Katello 2.1. > And in two or my katello installation both install have the same > problem. > I have create a new Product Centos Server Linux > I create the Product after that have url to do repo discovery. > It never complete the pulp_task, > It just said state => waiting. > below that task is Listen on candlepin events {"locale"=>"en"} > The process never completes. > > Below is the set that has been waiting. > 4cup > 12 GB ram > 500GB storage. > > I have create a foreman-debug for you to see. > foreman-debug-HAq0L.tar.xz > I am having the same problem in my new AWS installation Katello > 2.1 doing the same thing. > > Thanks Al > > -- > You received this message because you are subscribed to the Google > Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-users+unsubscribe@googlegroups.com > . > To post to this group, send email to foreman-users@googlegroups.com > . > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts

Tel: 053 20 30 270 	info@netbulae.eu 	Staalsteden 4-3A 	KvK 08198180
Fax: 053 20 30 271 	www.netbulae.eu 	7547 TA Enschede 	BTW NL821234584B01

Justin_Sherrill · February 24, 2015, 5:56pm

> Jorick,
>
> Still not fixed at all here is the latest output.

Alvin,

If you do a katello-service restart and then go into the dynflow console
and resume the task does it go any further?

It looks like the pulp tasks are getting stuck for some reason.

-Justin

···

On 02/23/2015 04:59 PM, Alvin Johnson wrote: > > *Started at:* 2015-02-23 21:39:55 UTC > > *Ended at:* 2015-02-23 21:59:05 UTC > > *Real time:* 1150.02s > > *Execution time (excluding suspended state):* 8.83s > > *Input:* > > --- > pulp_id: AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64 > distributor_type_id: yum_distributor > source_pulp_id: > dependency: > remote_user: ajohnson2-58a818a8 > remote_cp_user: ajohnson2 > locale: en > > *Output:* > > --- > pulp_tasks: > - exception: > task_type: pulp.server.managers.repo.publish.publish > _href: /pulp/api/v2/tasks/00a33deb-6b63-4fd7-b013-ddf39aebdacb/ > task_id: 00a33deb-6b63-4fd7-b013-ddf39aebdacb > tags: > - pulp:repository:AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64 > - pulp:action:publish > finish_time: > start_time: > traceback: > spawned_tasks: [] > progress_report: {} > queue: reserved_resource_worker-1@til-vm-swk6-11.aircell.prod.dq > state: waiting > result: > error: > _id: > $oid: 54eb9e2c9431b63575bec426 > id: 54eb9e2ce4fd854da18c3d6d > poll_attempts: > total: 91 > failed: 0 > > On Thursday, February 19, 2015 at 12:18:27 PM UTC-6, Alvin Johnson wrote: > > I been running into problem for the last week. > I have install the new Katello 2.1. > And in two or my katello installation both install have the same > problem. > I have create a new Product Centos Server Linux > I create the Product after that have url to do repo discovery. > It never complete the pulp_task, > It just said state => waiting. > below that task is Listen on candlepin events {"locale"=>"en"} > The process never completes. > Below is the set that has been waiting. > 4cup > 12 GB ram > 500GB storage. > > I have create a foreman-debug for you to see. > foreman-debug-HAq0L.tar.xz > I am having the same problem in my new AWS installation Katello > 2.1 doing the same thing. > > Thanks Al > > -- > You received this message because you are subscribed to the Google > Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-users+unsubscribe@googlegroups.com > . > To post to this group, send email to foreman-users@googlegroups.com > . > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

Jorick_Astrego · February 24, 2015, 1:15pm

Mmm, I don't know.

I had some problems before because of heavy diskIO lots of tasks got hung.

Had to skip them using https://<hostname>/foreman_tasks/dynflow/ to get
things going again.

But it became pretty unusable. We upgraded to SSD storage because of
this and now all tasks finish fine.

Could you check iowait while running the task?

Regards,

Jorick

···

On 02/23/2015 10:59 PM, Alvin Johnson wrote: > Jorick, > > Still not fixed at all here is the latest output. > > *Started at:* 2015-02-23 21:39:55 UTC > > *Ended at:* 2015-02-23 21:59:05 UTC > > *Real time:* 1150.02s > > *Execution time (excluding suspended state):* 8.83s > > *Input:* > > --- > pulp_id: AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64 > distributor_type_id: yum_distributor > source_pulp_id: > dependency: > remote_user: ajohnson2-58a818a8 > remote_cp_user: ajohnson2 > locale: en > > *Output:* > > --- > pulp_tasks: > - exception: > task_type: pulp.server.managers.repo.publish.publish > _href: /pulp/api/v2/tasks/00a33deb-6b63-4fd7-b013-ddf39aebdacb/ > task_id: 00a33deb-6b63-4fd7-b013-ddf39aebdacb > tags: > - pulp:repository:AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64 > - pulp:action:publish > finish_time: > start_time: > traceback: > spawned_tasks: [] > progress_report: {} > queue: reserved_resource_worker-1@til-vm-swk6-11.aircell.prod.dq > state: waiting > result: > error: > _id: > $oid: 54eb9e2c9431b63575bec426 > id: 54eb9e2ce4fd854da18c3d6d > poll_attempts: > total: 91 > failed: 0 > > On Thursday, February 19, 2015 at 12:18:27 PM UTC-6, Alvin Johnson wrote: > > I been running into problem for the last week. > I have install the new Katello 2.1. > And in two or my katello installation both install have the same > problem. > I have create a new Product Centos Server Linux > I create the Product after that have url to do repo discovery. > It never complete the pulp_task, > It just said state => waiting. > below that task is Listen on candlepin events {"locale"=>"en"} > The process never completes. > > Below is the set that has been waiting. > 4cup > 12 GB ram > 500GB storage. > > I have create a foreman-debug for you to see. > foreman-debug-HAq0L.tar.xz > I am having the same problem in my new AWS installation Katello > 2.1 doing the same thing. > > Thanks Al > > -- > You received this message because you are subscribed to the Google > Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-users+unsubscribe@googlegroups.com > . > To post to this group, send email to foreman-users@googlegroups.com > . > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts

Tel: 053 20 30 270 	info@netbulae.eu 	Staalsteden 4-3A 	KvK 08198180
Fax: 053 20 30 271 	www.netbulae.eu 	7547 TA Enschede 	BTW NL821234584B01

Jorick_Astrego · February 24, 2015, 1:21pm

Spoke to soon, there are some NodeMetadataGenerate tasks stuck and a lot
of ListenOnCandlepinEvents.

Id 	Action 	State
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/?order_by=state&amp;desc&gt;
	Result
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/?order_by=result&amp;desc&gt;
	▼ Started at
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/?order_by=started_at&amp;desc&gt;
	
fb585fd1-6c7a-41b1-a2ea-37dbaa87354b
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-24 11:28:32 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/fb585fd1-6c7a-41b1-a2ea-37dbaa87354b&gt;

60cbdd6a-85b1-45fe-922a-517a17a3749d
Actions::Katello::ContentView::NodeMetadataGenerate 	paused 	error
2015-02-23 14:45:12 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/60cbdd6a-85b1-45fe-922a-517a17a3749d&gt;

9b07b149-65fb-4b10-9cea-34e4db47aa4e
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-23 12:48:47 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/9b07b149-65fb-4b10-9cea-34e4db47aa4e&gt;

90cda926-1709-4cec-be4f-de295cc4ecd5
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-23 12:41:30 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/90cda926-1709-4cec-be4f-de295cc4ecd5&gt;

286c4448-a6a8-4f3d-b7ec-a6e7907541e6
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-23 12:09:16 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/286c4448-a6a8-4f3d-b7ec-a6e7907541e6&gt;

7e7910a5-73be-4af2-9917-3dc86e0f6e04
Actions::Katello::ContentView::NodeMetadataGenerate 	paused 	error
2015-02-23 10:57:04 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/7e7910a5-73be-4af2-9917-3dc86e0f6e04&gt;

e5bc368b-54bd-4967-b492-14c0c520fbc7
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-23 09:12:36 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/e5bc368b-54bd-4967-b492-14c0c520fbc7&gt;

fae1e798-092e-413b-9b60-5235b58c0dbb
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-20 12:20:47 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/fae1e798-092e-413b-9b60-5235b58c0dbb&gt;

56eb3a58-1618-4662-986a-b3feccd081c5
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-18 14:55:59 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/56eb3a58-1618-4662-986a-b3feccd081c5&gt;

edb30ad5-bb10-427d-8aad-a9b87d346726
Actions::Candlepin::ListenOnCandlepinEvents 	paused 	error
2015-02-18 12:50:04 UTC 	Show
&lt;https://deployen.netbulae.mgmt/foreman_tasks/dynflow/edb30ad5-bb10-427d-8aad-a9b87d346726&gt;

I'm thinking of dowgrading to Katello 2.0 and foreman 1.6, the latest
versions are giving me a headache

···

On 02/24/2015 02:15 PM, Jorick Astrego wrote: > Mmm, I don't know. > > I had some problems before because of heavy diskIO lots of tasks got hung. > > Had to skip them using https:///foreman_tasks/dynflow/ to > get things going again. > > But it became pretty unusable. We upgraded to SSD storage because of > this and now all tasks finish fine. > > Could you check iowait while running the task? > > Regards, > > Jorick > > > On 02/23/2015 10:59 PM, Alvin Johnson wrote: >> Jorick, >> >> Still not fixed at all here is the latest output. >> >> *Started at:* 2015-02-23 21:39:55 UTC >> >> *Ended at:* 2015-02-23 21:59:05 UTC >> >> *Real time:* 1150.02s >> >> *Execution time (excluding suspended state):* 8.83s >> >> *Input:* >> >> --- >> pulp_id: AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64 >> distributor_type_id: yum_distributor >> source_pulp_id: >> dependency: >> remote_user: ajohnson2-58a818a8 >> remote_cp_user: ajohnson2 >> locale: en >> >> *Output:* >> >> --- >> pulp_tasks: >> - exception: >> task_type: pulp.server.managers.repo.publish.publish >> _href: /pulp/api/v2/tasks/00a33deb-6b63-4fd7-b013-ddf39aebdacb/ >> task_id: 00a33deb-6b63-4fd7-b013-ddf39aebdacb >> tags: >> - pulp:repository:AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64 >> - pulp:action:publish >> finish_time: >> start_time: >> traceback: >> spawned_tasks: [] >> progress_report: {} >> queue: reserved_resource_worker-1@til-vm-swk6-11.aircell.prod.dq >> state: waiting >> result: >> error: >> _id: >> $oid: 54eb9e2c9431b63575bec426 >> id: 54eb9e2ce4fd854da18c3d6d >> poll_attempts: >> total: 91 >> failed: 0 >> >> On Thursday, February 19, 2015 at 12:18:27 PM UTC-6, Alvin Johnson >> wrote: >> >> I been running into problem for the last week. >> I have install the new Katello 2.1. >> And in two or my katello installation both install have the same >> problem. >> I have create a new Product Centos Server Linux >> I create the Product after that have url to do repo discovery. >> It never complete the pulp_task, >> It just said state => waiting. >> below that task is Listen on candlepin events {"locale"=>"en"} >> The process never completes. >> >> Below is the set that has been waiting. >> 4cup >> 12 GB ram >> 500GB storage. >> >> I have create a foreman-debug for you to see. >> foreman-debug-HAq0L.tar.xz >> I am having the same problem in my new AWS installation Katello >> 2.1 doing the same thing. >> >> Thanks Al >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, >> send an email to foreman-users+unsubscribe@googlegroups.com >> . >> To post to this group, send email to foreman-users@googlegroups.com >> . >> Visit this group at http://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. > > > > > Met vriendelijke groet, With kind regards, > > Jorick Astrego* > > Netbulae Virtualization Experts * > ------------------------------------------------------------------------ > Tel: 053 20 30 270 info@netbulae.eu Staalsteden 4-3A KvK 08198180 > Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 > > > ------------------------------------------------------------------------ > > -- > You received this message because you are subscribed to the Google > Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-users+unsubscribe@googlegroups.com > . > To post to this group, send email to foreman-users@googlegroups.com > . > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts

Tel: 053 20 30 270 	info@netbulae.eu 	Staalsteden 4-3A 	KvK 08198180
Fax: 053 20 30 271 	www.netbulae.eu 	7547 TA Enschede 	BTW NL821234584B01

Justin_Sherrill · February 24, 2015, 5:57pm

> Spoke to soon, there are some NodeMetadataGenerate tasks stuck and a
> lot of ListenOnCandlepinEvents.
>
> Id Action State
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/?order_by=state&desc>
> Result
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/?order_by=result&desc>
> ▼ Started at
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/?order_by=started_at&desc>
>
> fb585fd1-6c7a-41b1-a2ea-37dbaa87354b
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-24 11:28:32 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/fb585fd1-6c7a-41b1-a2ea-37dbaa87354b>
>
> 60cbdd6a-85b1-45fe-922a-517a17a3749d
> Actions::Katello::ContentView::NodeMetadataGenerate paused
> error 2015-02-23 14:45:12 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/60cbdd6a-85b1-45fe-922a-517a17a3749d>
>
> 9b07b149-65fb-4b10-9cea-34e4db47aa4e
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-23 12:48:47 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/9b07b149-65fb-4b10-9cea-34e4db47aa4e>
>
> 90cda926-1709-4cec-be4f-de295cc4ecd5
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-23 12:41:30 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/90cda926-1709-4cec-be4f-de295cc4ecd5>
>
> 286c4448-a6a8-4f3d-b7ec-a6e7907541e6
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-23 12:09:16 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/286c4448-a6a8-4f3d-b7ec-a6e7907541e6>
>
> 7e7910a5-73be-4af2-9917-3dc86e0f6e04
> Actions::Katello::ContentView::NodeMetadataGenerate paused
> error 2015-02-23 10:57:04 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/7e7910a5-73be-4af2-9917-3dc86e0f6e04>
>
> e5bc368b-54bd-4967-b492-14c0c520fbc7
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-23 09:12:36 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/e5bc368b-54bd-4967-b492-14c0c520fbc7>
>
> fae1e798-092e-413b-9b60-5235b58c0dbb
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-20 12:20:47 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/fae1e798-092e-413b-9b60-5235b58c0dbb>
>
> 56eb3a58-1618-4662-986a-b3feccd081c5
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-18 14:55:59 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/56eb3a58-1618-4662-986a-b3feccd081c5>
>
> edb30ad5-bb10-427d-8aad-a9b87d346726
> Actions::Candlepin::ListenOnCandlepinEvents paused error
> 2015-02-18 12:50:04 UTC Show
> <https://deployen.netbulae.mgmt/foreman_tasks/dynflow/edb30ad5-bb10-427d-8aad-a9b87d346726>
>
>
>
> I'm thinking of dowgrading to Katello 2.0 and foreman 1.6, the latest
> versions are giving me a headache
>

Hey Jorick

Are there any errors under the Node metadata generate tasks?

We're tracking an issue related with the ListenOnCandlepin
http://projects.theforeman.org/issues/9068

-Justin

···

On 02/24/2015 08:21 AM, Jorick Astrego wrote:

On 02/24/2015 02:15 PM, Jorick Astrego wrote:
Mmm, I don’t know.

I had some problems before because of heavy diskIO lots of tasks got
hung.

Had to skip them using https:///foreman_tasks/dynflow/ to
get things going again.

But it became pretty unusable. We upgraded to SSD storage because of
this and now all tasks finish fine.

Could you check iowait while running the task?

Regards,

Jorick

On 02/23/2015 10:59 PM, Alvin Johnson wrote:
Jorick,

Still not fixed at all here is the latest output.

Started at: 2015-02-23 21:39:55 UTC

Ended at: 2015-02-23 21:59:05 UTC

Real time: 1150.02s

Execution time (excluding suspended state): 8.83s

Input:

pulp_id: AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64
distributor_type_id: yum_distributor
source_pulp_id:
dependency:
remote_user: ajohnson2-58a818a8
remote_cp_user: ajohnson2
locale: en

Output:

pulp_tasks:

exception:
task_type: pulp.server.managers.repo.publish.publish
_href: /pulp/api/v2/tasks/00a33deb-6b63-4fd7-b013-ddf39aebdacb/
task_id: 00a33deb-6b63-4fd7-b013-ddf39aebdacb
tags:

pulp:repository:AIRCELL_PROD-Puppet_Labs_Rpms-pupept6_dependencies_x86_64

pulp:action:publish
finish_time:
start_time:
traceback:
spawned_tasks: []
progress_report: {}
queue: reserved_resource_worker-1@til-vm-swk6-11.aircell.prod.dq
state: waiting
result:
error:
_id:
$oid: 54eb9e2c9431b63575bec426
id: 54eb9e2ce4fd854da18c3d6d
poll_attempts:
total: 91
failed: 0

On Thursday, February 19, 2015 at 12:18:27 PM UTC-6, Alvin Johnson >>> wrote:
I been running into problem for the last week.
I have install the new Katello 2.1.
And in two or my katello installation both install have the same
problem.
I have create a new Product Centos Server Linux
I create the Product after that have url to do repo discovery.
It never complete the pulp_task,
It just said state => waiting.
below that task is Listen on candlepin events {"locale"=>"en"}
The process never completes.
Below is the set that has been waiting.
4cup
12 GB ram
500GB storage.

I have create a foreman-debug for you to see.
foreman-debug-HAq0L.tar.xz
I am having the same problem in my new AWS installation Katello
2.1 doing the same thing.

Thanks Al
–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it,
send an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Met vriendelijke groet, With kind regards,

Jorick Astrego*

Netbulae Virtualization Experts *

Tel: 053 20 30 270 info@netbulae.eu Staalsteden 4-3A KvK 08198180
Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW
NL821234584B01

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it,
send an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Met vriendelijke groet, With kind regards,

Jorick Astrego*

Netbulae Virtualization Experts *

Tel: 053 20 30 270 info@netbulae.eu Staalsteden 4-3A KvK 08198180
Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.