mbassler
gwmngilfen: I have hundreds of tasks titled either "Update" or "Update
system 'lily.example.com'; organization 'ORG'" that are sitting in a
planning pending state. When I try to access any of the tasks I get a
warning message
"Warning! plan_self has to be invoked before being able to reference the
output"
I can seem to get any additional information about the task at all besides
that.
mbassler
This was grinding my instance to a halt. I have since unregistered the node
lily and removed any katello related packages from that node and I am not
seeing performance hit anymore but I still have hundreds on tasks sitting
in this state.
jsherrill
mbassler: is foreman-tasks running?
I was pulled away after asking this question on IRC, and didnt get a chance
to reply.
Justin, to answer your question:
foreman-tasks was/is running and I can confirm that plenty of other tasks
ran and completed during the period where this was going on. When this
started our web interface basically became unusable noticed that Passenger
processes would max out and loading anything in the web UI would take
minutes. We ended up tuning the passenger configs for Max pool size which
helped. But we than started to hit max process issues where we would get
errors like "can't create Thread (11)". As I mentioned I've removed the
node lily and things are now stable, but I would like to clean up the tasks
that are still in planning pending state as well as I'd like to get an
understanding of what happened so it doesn't happen again. It scares me a
little that something wrong in a registration or katello-agent could cause
things to go south.
Thank You
-Michael
> mbassler
> gwmngilfen: I have hundreds of tasks titled either "Update" or "Update
> system 'lily.example.com'; organization 'ORG'" that are sitting in a
> planning pending state. When I try to access any of the tasks I get a
> warning message
> "Warning! plan_self has to be invoked before being able to reference the
> output"
> I can seem to get any additional information about the task at all
> besides that.
> mbassler
> This was grinding my instance to a halt. I have since unregistered the
> node lily and removed any katello related packages from that node and I
> am not seeing performance hit anymore but I still have hundreds on tasks
> sitting in this state.
> jsherrill
> mbassler: is foreman-tasks running?
>
> I was pulled away after asking this question on IRC, and didnt get a
> chance to reply.
>
> Justin, to answer your question:
> foreman-tasks was/is running and I can confirm that plenty of other
> tasks ran and completed during the period where this was going on. When
> this started our web interface basically became unusable noticed that
> Passenger processes would max out and loading anything in the web UI
> would take minutes. We ended up tuning the passenger configs for Max
> pool size which helped. But we than started to hit max process issues
> where we would get errors like "can't create Thread (11)". As I
> mentioned I've removed the node lily and things are now stable, but I
> would like to clean up the tasks that are still in planning pending
> state as well as I'd like to get an understanding of what happened so it
> doesn't happen again. It scares me a little that something wrong in a
> registration or katello-agent could cause things to go south.
Few questions for you:
- How many clients do you have registered?
- What are the hardware specs of your system running katello/foreman?
- Can you detail exactly the config change you made?
- Was there anything special about this node lilly? (Was it running
virt-who, did it have its checkin time increased)
You can likely delete the tasks by running (these are instructions for
katello 2.3 and 2.4, 3.0 would differ slightly):
service foreman-tasks stop
foreman-rake console
> ForemanTasks::Task.where("state != 'stopped'").where(:label =>
'Actions::Katello::System::Update').destroy_all
> quit
service foreman-tasks start
Note that this way of destroying tasks can be quite dangerous depending
on the task, but in this case it is perfectly safe.
-Justin
···
On 05/04/2016 07:33 PM, Michael Bassler wrote:
Thank You
-Michael
–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com
mailto:foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com
mailto:foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Currently we have 906 registered nodes but we will eventually want to grow
to ~6000-7000.I should note that we have no plans to use puppet on our
nodes as we run Chef.
We are running on a reclaimed hypervisor that we forgot to reclaim memory
from. So 48 cores @ 2.7Ghz, 256GB RAM with 450GB SSD storage, We have off
loaded our pulp storage to network storage but we have bonded 10GB
networking.
I believe that the registration of lily happened early this year back in
January. (That is when I start seeing "Update" tasks) At some point we
upgraded to Katello 2.4. But initially it did not result in unresponsive
webUI. May 2nd katello-agent on lily was upgrade from 1.1.2-1.el6 to
2.4.0-3.el6 (We do not normally install 1.1.2 from EPEL) I believe that is
related to what caused things to go south. At that point web interface
became extremely sluggish and would eventually become unusable. At that
point I found an older thread on here talking about PassenderMaxPoolSize. I
added 'PassengerMaxPoolSize 24' to passenger.conf and 05-foreman.conf,
after this change is when we started to see the 'can't create threads'
errors.
Lily is running RHEVM, so it is running virt-who.
the console delete was successful.
Thank You
···
On Thursday, May 5, 2016 at 6:50:23 AM UTC-7, jsherril wrote:
>
> On 05/04/2016 07:33 PM, Michael Bassler wrote:
> > mbassler
> > gwmngilfen: I have hundreds of tasks titled either "Update" or "Update
> > system 'lily.example.com'; organization 'ORG'" that are sitting in a
> > planning pending state. When I try to access any of the tasks I get a
> > warning message
> > "Warning! plan_self has to be invoked before being able to reference the
> > output"
> > I can seem to get any additional information about the task at all
> > besides that.
> > mbassler
> > This was grinding my instance to a halt. I have since unregistered the
> > node lily and removed any katello related packages from that node and I
> > am not seeing performance hit anymore but I still have hundreds on tasks
> > sitting in this state.
> > jsherrill
> > mbassler: is foreman-tasks running?
> >
> > I was pulled away after asking this question on IRC, and didnt get a
> > chance to reply.
> >
> > Justin, to answer your question:
> > foreman-tasks was/is running and I can confirm that plenty of other
> > tasks ran and completed during the period where this was going on. When
> > this started our web interface basically became unusable noticed that
> > Passenger processes would max out and loading anything in the web UI
> > would take minutes. We ended up tuning the passenger configs for Max
> > pool size which helped. But we than started to hit max process issues
> > where we would get errors like "can't create Thread (11)". As I
> > mentioned I've removed the node lily and things are now stable, but I
> > would like to clean up the tasks that are still in planning pending
> > state as well as I'd like to get an understanding of what happened so it
> > doesn't happen again. It scares me a little that something wrong in a
> > registration or katello-agent could cause things to go south.
>
> Few questions for you:
>
> * How many clients do you have registered?
> * What are the hardware specs of your system running katello/foreman?
> * Can you detail exactly the config change you made?
> * Was there anything special about this node lilly? (Was it running
> virt-who, did it have its checkin time increased)
>
>
> You can likely delete the tasks by running (these are instructions for
> katello 2.3 and 2.4, 3.0 would differ slightly):
>
> # service foreman-tasks stop
> # foreman-rake console
>
> > ForemanTasks::Task.where("state != 'stopped'").where(:label =>
> 'Actions::Katello::System::Update').destroy_all
> > quit
>
> # service foreman-tasks start
>
>
> Note that this way of destroying tasks can be quite dangerous depending
> on the task, but in this case it is perfectly safe.
>
> -Justin
>
>
>
> >
> > Thank You
> > -Michael
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Foreman users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to foreman-user...@googlegroups.com
> > <mailto:foreman-users+unsubscribe@googlegroups.com >.
> > To post to this group, send email to forema...@googlegroups.com
>
> > <mailto:forema...@googlegroups.com >.
> > Visit this group at https://groups.google.com/group/foreman-users.
> > For more options, visit https://groups.google.com/d/optout.
>
>
