Greetings,
I have a foreman/katello server up and running and I had been having
pretty good luck with it until yesterday. Suddenly I'm getting
messages about how tasks can't be accepted and that the core is
terminating.
I'm not sure how to troubleshoot this. There's a nice stack trace in
the logs, or the full trace I can see via the error on the web page,
but I'm not seeing a cause for the error. Where do I begin to
troubleshoot this?
These errors are occurring for a number of different tasks. For
instance, I have a task that errored out with a message about a
temporary failure in name resolution. In the past, if I just resume
these tasks, they fix themselves. Now I'm getting this core
termination error. So, as an example, here's the full trace from the
error :
Dynflow::Error
cannot accept execution_plan_id:4ce3cdb0-af8a-413f-a953-4a2166737f29
core is terminating
app/controllers/concerns/application_shared.rb:13:in set_timezone' app/models/concerns/foreman/thread_session.rb:32:inclear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call'
And here's the stack trace from the foreman production log :
2015-06-25 10:42:18 [E] cannot accept
execution_plan_id:4ce3cdb0-af8a-413f-a953-4a2166737f29 core is
terminating (Dynflow::Error)
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/execu
tors/parallel/core.rb:54:in
track_execution_plan' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/execu tors/parallel/core.rb:23:inblock in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.r
b:859:in
block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.r b:858:intap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.r
b:858:in
assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.r b:138:inmatch_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.r
b:116:in
block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.r b:115:ineach'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.r
b:115:in
match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/execu tors/parallel/core.rb:21:inon_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro
_actor.rb:80:in
block in on_envelope' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/futur e.rb:75:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/futur
e.rb:75:in
evaluate_to' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro _actor.rb:80:inon_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro
_actor.rb:72:in
receive' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro _actor.rb:99:inblock (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro
_actor.rb:99:in
loop' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro _actor.rb:99:inblock in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro
_actor.rb:99:in
catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro _actor.rb:99:inrun'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro
_actor.rb:13:in
block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagn ostic_context.rb:323:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagn
ostic_context.rb:323:in
`block in create_with_logging_context'
Jason 'XenoPhage' Frisvold
xenophage@godshell.com
"A common mistake that people make when trying to design something
completely foolproof is to underestimate the ingenuity of complete
fools."
-
- The Hitchhikers Guide to the Galaxy