Random ISE on host details page: undefined method `parent_task' for nil:NilClass

Support
1

Problem:
Sometimes, while a REX Task is running on a host, the host details page for that host shows the error message Failure: undefined method 'parent_task' for nil:NilClass instead of the content of the properties tab. production.log suggests a internal server error, probably while fetching the remote execution status. Log is attached below. I am not able to reproduce this reliably, and I also looked at the tasks for the affected hosts both in the UI and the DB and can not find any weird stuff going on.
Expected outcome:
Properties and status information displayed as intended.
Foreman and Proxy versions:
2.5.4
Foreman and Proxy plugin versions:

Plugins:
 - foreman-tasks 4.1.5
 - foreman_expire_hosts 7.0.4
 - foreman_hooks 0.3.17
 - foreman_remote_execution 4.5.6
 - foreman_scc_manager 1.8.10
 - foreman_snapshot_management 2.0.1
 - foreman_templates 9.1.0
 - katello 4.1.4
 - puppetdb_foreman 5.0.0

Distribution and version:
RHEL 7
Other relevant data:

# foreman-rake errors:fetch_log request_id=05e6af92
Rubocop not loaded.
Rubocop not loaded.
2021-12-03T14:49:40 [I|app|05e6af92] Started GET "/hosts/host.domain/overview" for 10.129.6.100 at 2021-12-03 14:49:40 +0100
2021-12-03T14:49:40 [I|app|05e6af92] Processing by HostsController#overview as HTML
2021-12-03T14:49:40 [I|app|05e6af92]   Parameters: {"id"=>"host.domain"}
2021-12-03T14:49:40 [I|app|05e6af92]   Rendered hosts/_overview.html.erb (Duration: 23.7ms | Allocations: 14094)
2021-12-03T14:49:40 [W|app|05e6af92] Failed to fetch overview information
2021-12-03T14:49:40 [I|app|05e6af92] Backtrace for 'Failed to fetch overview information' error (ActionView::Template::Error): undefined method `parent_task' for nil:NilClass
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.5.6/app/models/host_status/execution_status.rb:53:in `status_link'
 05e6af92 | /usr/share/foreman/app/helpers/host_description_helper.rb:56:in `block in host_detailed_status_list'
 05e6af92 | /usr/share/foreman/app/helpers/host_description_helper.rb:51:in `map'
 05e6af92 | /usr/share/foreman/app/helpers/host_description_helper.rb:51:in `host_detailed_status_list'
 05e6af92 | /usr/share/foreman/app/helpers/host_description_helper.rb:44:in `base_status_overview_fields'
 05e6af92 | /usr/share/foreman/app/helpers/hosts_helper.rb:253:in `block in overview_fields'
 05e6af92 | /usr/share/foreman/app/helpers/hosts_helper.rb:252:in `each'
 05e6af92 | /usr/share/foreman/app/helpers/hosts_helper.rb:252:in `overview_fields'
 05e6af92 | /usr/share/foreman/app/views/hosts/_overview.html.erb:8:in `_83cd38600a8745b8797f5f9cae1d1a9a'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/base.rb:274:in `_run'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:185:in `block in render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:182:in `instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:385:in `instrument_render_template'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:183:in `render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/partial_renderer.rb:357:in `block in render_partial'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:87:in `instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/partial_renderer.rb:346:in `render_partial'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/partial_renderer.rb:317:in `render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:65:in `render_partial_to_object'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:27:in `render_to_object'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:117:in `block in _render_template'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/base.rb:304:in `in_rendering_context'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:116:in `_render_template'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/streaming.rb:218:in `_render_template'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:103:in `render_to_body'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/rendering.rb:25:in `render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:36:in `render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 05e6af92 | /opt/rh/rh-ruby27/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:43:in `render'
 05e6af92 | /usr/share/foreman/app/controllers/hosts_controller.rb:258:in `overview'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in `process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 05e6af92 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 05e6af92 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 05e6af92 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in `process'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in `process'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in `dispatch'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in `dispatch'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `each'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.4/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.4/lib/katello/middleware/event_daemon.rb:10:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 05e6af92 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 05e6af92 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in `run_callbacks'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 05e6af92 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.4/lib/katello/prevent_json_parsing.rb:12:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
 05e6af92 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-12-03T14:49:40 [I|app|05e6af92]   Rendered common/_ajax_error.html.erb (Duration: 2.2ms | Allocations: 299)
2021-12-03T14:49:40 [I|app|05e6af92] Completed 500 Internal Server Error in 41ms (Views: 3.9ms | ActiveRecord: 11.6ms | Allocations: 20093)

If any more info would be helpfull in analyzing this, I will try and get it.

Regards

2

This is rather odd. Have you run any task cleanup recently?

3

We ran some task cleanups last weekend during our upgrade from 2.0 to 2.5 to get rid of some tasks in a broken state that were failing the take upgradecheck. But we have not performed any manual cleanups since then and this occured on a host installed today during the first REX execution on that host, so it should not be affected by anything we did last weekend I assume.

4

During the execution? As in just as the job was running? And it sorted itself out eventually?

5

Yes, it only occured while the REX job was running and the error message went away as soon as the Job finished. It was a long-running job on a freshly installed VM, my coworkers catched this. We had a similar case earlier this week, on another rather freshly installed server (I just checked the times). Maybe this only occures during the first REX execution per host? At least that would explain why I was unable to reproduce this on any test system.
I’m off for the weekend now, but I will try and test this further next week.

6

I found a way to reproduce this reliably today. It is in fact consistently occuring on the first job for each host after creation and does not reoccure on further executions.

Steps to reproduce:

  1. Create a new host and wait for it to finish installing
  2. Run a REX job on that host that takes enough time for you to refresh the host status page, e.g. sleep 100
  3. Refresh the hosts status page while the job is still running

I do not have a more current Foreman version than 2.5.4 at hand, so I cannot test if this is fixed in the current release. Since it occures reliably in both our testing and production environment, I doubt something broke on our end.

7

Closing the loop on this one, a fix for this[1] possibly went in today.

[1] - https://github.com/theforeman/foreman_remote_execution/pull/742

1 Like
8

Thanks for the heads up. I just tested the patch and it got rid of the error message.
Instead the status page now show “Last execution succeeded” as soon as you start the first job on a new host, which is technically wrong I guess but should not hurt anyone.