[Katello 2.4] Viewing paused task leads to internal server error

I have a task stuck in "state=planning&result=error"

When I go to the task console and try to click the dynflow console button I
just get "Internal Server Error" on a totally white page.

I have tried restarting the server to address the "no pulp workers running"
error but it never goes away, even on startup.

The task tab in the tasks console gives me the following :

Id: 51578312-605d-42ee-9c15-d0a5e7b4ad6e
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Execution type: Immediate
Start at: -
Start before: -
Started at: 2015-12-16 16:58:49 UTC
Ended at:
State: planning
Result: error
Params: repository 'CentoS 5 Updates'; product 'CentOS 5'; organization 'My
Company'
0% Complete
0%
Output:

key not found: 2 (KeyError)
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:in
fetch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:inblock in planned_steps'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:in
map' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:inplanned_steps'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/action.rb:196:in
planned_actions' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/app/lib/actions/katello/repository/sync.rb:49:inpresenter'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/app/lib/actions/helpers/presenter.rb:9:in
humanized_output' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/models/foreman_tasks/task/dynflow_task.rb:94:inget_humanized'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/models/foreman_tasks/task/dynflow_task.rb:70:in
humanized' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/tasks/_details.html.erb:92:in_14f12df51818b491a48d397caa42867d'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:145:in
block in render' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:125:ininstrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:143:in
render' /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.0.1/lib/deface/action_view_extensions.rb:41:inrender'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/partial_renderer.rb:265:in
render_partial' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/partial_renderer.rb:238:inblock in render'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:inblock in instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:ininstrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/partial_renderer.rb:237:inrender'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:41:in
render_partial' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:15:inrender'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/rendering_helper.rb:24:in
render' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/tasks/show.html.erb:96:inblock in _3a96b90fef3a95097e8901e60de73fdf'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/capture_helper.rb:40:in
block in capture' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/capture_helper.rb:187:inwith_output_buffer'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/capture_helper.rb:40:in
capture' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/form_helper.rb:607:infields_for'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/form_helper.rb:378:in
form_for' /usr/share/foreman/app/helpers/layout_helper.rb:400:inform_for'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/tasks/show.html.erb:84:in
_3a96b90fef3a95097e8901e60de73fdf' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:145:inblock in render'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:125:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:143:inrender'
/opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.0.1/lib/deface/action_view_extensions.rb:41:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:47:inblock (2 levels) in render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:inblock in instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:ininstrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:46:inblock in render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:54:in
render_with_layout' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:45:inrender_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:18:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:36:inrender_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:17:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:110:in_render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/streaming.rb:225:in
_render_template' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:103:inrender_to_body'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/renderers.rb:28:in
render_to_body' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/compatibility.rb:50:inrender_to_body'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:88:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:16:inrender'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:40:in
block (2 levels) in render' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/core_ext/benchmark.rb:5:inblock in ms'
/opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in realtime' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/core_ext/benchmark.rb:5:inms'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:40:in
block in render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:83:incleanup_view_runtime'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:24:in
cleanup_view_runtime' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:39:inrender'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:10:in
default_render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:5:insend_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in
process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:inprocess_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in
block in process_action' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:638:inblock (4 levels) in
_run__821916532020871237__process_action__4507901181148516982__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in
block in _conditional_callback_around_11870' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:inaround'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in
_callback_around_2502' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in_conditional_callback_around_11870'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:626:in
block (3 levels) in _run__821916532020871237__process_action__4507901181148516982__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:inblock in _conditional_callback_around_11869'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:13:in
set_timezone' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in_conditional_callback_around_11869'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:625:in
block (2 levels) in _run__821916532020871237__process_action__4507901181148516982__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:inblock in _conditional_callback_around_11868'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in
clear_thread' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in_conditional_callback_around_11868'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:448:in
block in _run__821916532020871237__process_action__4507901181148516982__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:inblock in _conditional_callback_around_11867'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in
around' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in_callback_around_13'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in
_conditional_callback_around_11867' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:414:in_run__821916532020871237__process_action__4507901181148516982__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in
__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in_run_process_action_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in
run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:inprocess_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in
process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:inblock in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in
block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:ininstrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:inprocess_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in
process_action' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:18:inprocess_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in
process' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:inprocess'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in
dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:indispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in
block in action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in
dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in
block in call' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:ineach'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:incall'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.2.6/lib/apipie/extractor/recorder.rb:97:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in
call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.2.6/lib/apipie/middleware/checksum_in_headers.rb:27:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:25:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in
call' /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:incontext'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/query_cache.rb:64:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:473:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in
block in call' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in_run__181790860792438380__call__884820464455613324__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in
__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in_run_call_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in
run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:incall_app'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:inforward'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in
fetch' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:inlookup'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in
call!' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:incall'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in
method_missing' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/builder.rb:134:incall'
/opt/rh/ruby193/root/usr/share/gems/

Nathan,

try restarting the pulp services this way

sudo systemctl stop pulp_resource_manager.service pulp_celerybeat.service
pulp_workers.service
sudo systemctl start pulp_resource_manager.service pulp_celerybeat.service
pulp_workers.service

for some reason stopping them completely and starting them together
sometimes fixes these types of issues

also try restarting qpidd

If those suggestions don't work, please comment back with any errors you
see in /var/log/messages.

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Mon, Mar 7, 2016 at 1:00 PM, Nathan Peters wrote:

I have a task stuck in “state=planning&result=error”

When I go to the task console and try to click the dynflow console button
I just get “Internal Server Error” on a totally white page.

I have tried restarting the server to address the “no pulp workers
running” error but it never goes away, even on startup.

The task tab in the tasks console gives me the following :

Id: 51578312-605d-42ee-9c15-d0a5e7b4ad6e
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Execution type: Immediate
Start at: -
Start before: -
Started at: 2015-12-16 16:58:49 UTC
Ended at:
State: planning
Result: error
Params: repository ‘CentoS 5 Updates’; product ‘CentOS 5’; organization
’My Company’
0% Complete
0%
Output:

key not found: 2 (KeyError)
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:in
fetch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:inblock in planned_steps’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:in
map' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/execution_plan/steps/plan_step.rb:27:inplanned_steps’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.9/lib/dynflow/action.rb:196:in
planned_actions' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/app/lib/actions/katello/repository/sync.rb:49:inpresenter’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-2.4.0/app/lib/actions/helpers/presenter.rb:9:in
humanized_output' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/models/foreman_tasks/task/dynflow_task.rb:94:inget_humanized’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/models/foreman_tasks/task/dynflow_task.rb:70:in
humanized' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/tasks/_details.html.erb:92:in_14f12df51818b491a48d397caa42867d’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:145:in
block in render' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:125:ininstrument’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:143:in
render' /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.0.1/lib/deface/action_view_extensions.rb:41:inrender’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/partial_renderer.rb:265:in
render_partial' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/partial_renderer.rb:238:inblock in render’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:inblock in instrument’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:ininstrument’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/partial_renderer.rb:237:inrender’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:41:in
render_partial' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:15:inrender’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/rendering_helper.rb:24:in
render' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/tasks/show.html.erb:96:inblock in _3a96b90fef3a95097e8901e60de73fdf’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/capture_helper.rb:40:in
block in capture' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/capture_helper.rb:187:inwith_output_buffer’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/capture_helper.rb:40:in
capture' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/form_helper.rb:607:infields_for’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/helpers/form_helper.rb:378:in
form_for' /usr/share/foreman/app/helpers/layout_helper.rb:400:inform_for’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.11/app/views/foreman_tasks/tasks/show.html.erb:84:in
_3a96b90fef3a95097e8901e60de73fdf' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:145:inblock in render’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:125:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:143:inrender’
/opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.0.1/lib/deface/action_view_extensions.rb:41:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:47:inblock (2 levels) in render_template’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:inblock in instrument’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:ininstrument’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:46:inblock in render_template’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:54:in
render_with_layout' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:45:inrender_template’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:18:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:36:inrender_template’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:17:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:110:in_render_template’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/streaming.rb:225:in
_render_template' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:103:inrender_to_body’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/renderers.rb:28:in
render_to_body' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/compatibility.rb:50:inrender_to_body’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:88:in
render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:16:inrender’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:40:in
block (2 levels) in render' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/core_ext/benchmark.rb:5:inblock in ms’
/opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in realtime' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/core_ext/benchmark.rb:5:inms’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:40:in
block in render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:83:incleanup_view_runtime’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:24:in
cleanup_view_runtime' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:39:inrender’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:10:in
default_render' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:5:insend_action’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in
process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:inprocess_action’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in
block in process_action' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:638:inblock (4 levels) in
_run__821916532020871237__process_action__4507901181148516982__callbacks’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in
block in _conditional_callback_around_11870' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:inaround’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in
_callback_around_2502' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in_conditional_callback_around_11870’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:626:in
block (3 levels) in _run__821916532020871237__process_action__4507901181148516982__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:inblock in _conditional_callback_around_11869’
/usr/share/foreman/app/controllers/concerns/application_shared.rb:13:in
set_timezone' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in_conditional_callback_around_11869’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:625:in
block (2 levels) in _run__821916532020871237__process_action__4507901181148516982__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:inblock in _conditional_callback_around_11868’
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in
clear_thread' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in_conditional_callback_around_11868’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:448:in
block in _run__821916532020871237__process_action__4507901181148516982__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:inblock in _conditional_callback_around_11867’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in
around' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in_callback_around_13’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in
_conditional_callback_around_11867' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:414:in_run__821916532020871237__process_action__4507901181148516982__callbacks’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in
__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in_run_process_action_callbacks’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in
run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:inprocess_action’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in
process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:inblock in process_action’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in
block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:ininstrument’
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in
instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:inprocess_action’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in
process_action' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:18:inprocess_action’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in
process' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:inprocess’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in
dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:indispatch’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in
block in action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in
dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in
block in call' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:ineach’
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:incall’
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.2.6/lib/apipie/extractor/recorder.rb:97:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in
call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.2.6/lib/apipie/middleware/checksum_in_headers.rb:27:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:25:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in
call' /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:incontext’
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in
call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:incall’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/query_cache.rb:64:in
`call’
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abst