Upgrade Foreman 3.3/Katello 4.5 errors

Problem:
Upgrade F3.3/K4.5 errors:

  • $ foreman-installer

    “2022-07-05 14:28:04 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[cpt-infra05l-p.acceo.com]/features: change from [“Dynflow”, “Logs”, “Pulpcore”, “Puppet”, “Puppet CA”, “Registration”, “TFTP”, “Templates”] to [“Dynflow”, “Logs”, “Pulpcore”, “Puppet”, “Puppet CA”, “Registration”, “Script”, “TFTP”, “Templates”] failed: Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors”

Remote execution on clients error:
“Oops, we’re sorry but something went wrong** undefined method `template_inputs’ for nil:NilClass”

Expected outcome:
Upgrade without errors, and Remote execution (via SSH) functional

Foreman and Proxy versions:
Foreman version: 3.3.0

Foreman and Proxy plugin versions:

  • foreman-tasks 6.0.2
  • foreman_ansible 7.1.0
  • foreman_docker 5.0.0
  • foreman_puppet 4.0.1
  • foreman_remote_execution 7.1.0
  • katello 4.5.0.rc2

Distribution and version:
CentOS Linux release 7.9.2009

Other relevant data:

  • $ foreman-rake errors:fetch_log request_id=379927f5
    2022-07-05T14:41:40 [I|app|379927f5] Processing by JobInvocationsController#show as HTML
    2022-07-05T14:41:40 [I|app|379927f5] Parameters: {“id”=>“287”}
    2022-07-05T14:41:40 [I|app|379927f5] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-7.1.0/app/views/job_invocations/show.html.erb within layouts/application
    2022-07-05T14:41:40 [I|app|379927f5] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-7.1.0/app/views/job_invocations/show.html.erb within layouts/application (Duration: 14.1ms | Allocations: 5453)
    2022-07-05T14:41:40 [W|app|379927f5] undefined method template_inputs' for nil:NilClass 2022-07-05T14:41:40 [I|app|379927f5] Backtrace for 'undefined method template_inputs’ for nil:NilClass’ error (ActionView::Template::Error): undefined method template_inputs' for nil:NilClass 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-7.1.0/app/helpers/remote_execution_helper.rb:249:in job_report_template’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-7.1.0/app/helpers/remote_execution_helper.rb:67:in job_invocation_task_buttons' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-7.1.0/app/views/job_invocations/show.html.erb:12:in _7eff5b6f155a0fbd0280849f7f3b71ab’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/base.rb:274:in _run' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/template.rb:185:in block in render’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications.rb:182:in instrument' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/template.rb:385:in instrument_render_template’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/template.rb:183:in render' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in render’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/template_renderer.rb:58:in block (2 levels) in render_template' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/abstract_renderer.rb:88:in block in instrument’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications.rb:180:in block in instrument' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications/instrumenter.rb:24:in instrument’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications.rb:180:in instrument' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/abstract_renderer.rb:87:in instrument’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/template_renderer.rb:57:in block in render_template' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/template_renderer.rb:65:in render_with_layout’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/template_renderer.rb:56:in render_template' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/template_renderer.rb:13:in render’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/renderer.rb:61:in render_template_to_object' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/renderer/renderer.rb:29:in render_to_object’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/rendering.rb:117:in block in _render_template' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/base.rb:304:in in_rendering_context’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/rendering.rb:116:in _render_template' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/streaming.rb:218:in _render_template’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/rendering.rb:103:in render_to_body' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/rendering.rb:52:in render_to_body’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/renderers.rb:142:in render_to_body' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/rendering.rb:25:in render’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/rendering.rb:36:in render' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:44:in block (2 levels) in render’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/core_ext/benchmark.rb:14:in block in ms' 379927f5 | /opt/rh/rh-ruby27/root/usr/share/ruby/benchmark.rb:308:in realtime’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/core_ext/benchmark.rb:14:in ms' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:44:in block in render’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:84:in cleanup_view_runtime' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/controller_runtime.rb:34:in cleanup_view_runtime’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:43:in render' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/implicit_render.rb:35:in default_render’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/basic_implicit_render.rb:6:in block in send_action' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/basic_implicit_render.rb:6:in tap’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/basic_implicit_render.rb:6:in send_action' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/base.rb:195:in process_action’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/rendering.rb:30:in process_action' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/callbacks.rb:42:in block in process_action’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:112:in block in run_callbacks' 379927f5 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 379927f5 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in clear_thread’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 379927f5 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in block in run_callbacks' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:139:in run_callbacks’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/callbacks.rb:41:in process_action' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/rescue.rb:22:in process_action’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:33:in block in process_action' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications.rb:180:in block in instrument’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications/instrumenter.rb:24:in instrument' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/notifications.rb:180:in instrument’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/instrumentation.rb:32:in process_action' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal/params_wrapper.rb:245:in process_action’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/controller_runtime.rb:27:in process_action' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/abstract_controller/base.rb:136:in process’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.4.7/lib/action_view/rendering.rb:39:in process' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal.rb:190:in dispatch’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_controller/metal.rb:254:in dispatch' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/route_set.rb:50:in dispatch’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/route_set.rb:33:in serve' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:49:in block in serve’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:32:in each' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/journey/router.rb:32:in serve’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/routing/route_set.rb:834:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0.rc2/lib/katello/middleware/organization_created_enforcer.rb:18:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0.rc2/lib/katello/middleware/event_daemon.rb:10:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call’
    379927f5 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in call' 379927f5 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/http/content_security_policy.rb:18:in call’
    379927f5 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in context’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/cookies.rb:654:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/callbacks.rb:27:in block in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:101:in run_callbacks’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/callbacks.rb:26:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/rack/logger.rb:37:in call_app' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/rack/logger.rb:28:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call' 379927f5 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/remote_ip.rb:81:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/request_id.rb:27:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0.rc2/lib/katello/prevent_json_parsing.rb:12:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/executor.rb:14:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/static.rb:126:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/ssl.rb:74:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.4.7/lib/action_dispatch/middleware/host_authorization.rb:97:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/engine.rb:527:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/railtie.rb:190:in public_send’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/railtie.rb:190:in method_missing' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/configuration.rb:252:in call' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/request.rb:77:in block in handle_request’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/thread_pool.rb:340:in with_force_shutdown' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/request.rb:76:in handle_request’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/server.rb:441:in process_client' 379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.6.2/lib/puma/thread_pool.rb:147:in block in spawn_thread’
    379927f5 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’
    2022-07-05T14:41:40 [I|app|379927f5] Rendering common/500.html.erb within layouts/application
    2022-07-05T14:41:40 [I|app|379927f5] Rendered common/500.html.erb within layouts/application (Duration: 1.8ms | Allocations: 812)
    2022-07-05T14:41:40 [I|app|379927f5] Rendered layouts/_application_content.html.erb (Duration: 2.8ms | Allocations: 5695)
    2022-07-05T14:41:40 [I|app|379927f5] Rendering layouts/base.html.erb
    2022-07-05T14:41:40 [I|app|379927f5] Rendered layouts/base.html.erb (Duration: 5.3ms | Allocations: 8012)
    2022-07-05T14:41:40 [I|app|379927f5] Completed 500 Internal Server Error in 53ms (Views: 15.1ms | ActiveRecord: 5.1ms | Allocations: 35524)

Foreman version: 3.3.0
Plugins:

  • foreman-tasks 6.0.2
  • foreman_ansible 7.1.0
  • foreman_docker 5.0.0
  • foreman_puppet 4.0.1
  • foreman_remote_execution 7.1.0
  • katello 4.5.0.rc2

I’ve seen this error and I’m pretty sure there is a fix for it in foreman_remote_execution, but @aruzicka will know better…

As a workaround in my dev environment I changed this line

to

template if template&.template_inputs.where(name: 'job_id').exists?

Thank you very much @jeremylenz

This workaround did not work in my case :frowning:
the new error was then: undefined method `where’ for nil:NilClass
40dce9c7 | Did you mean? when

I tried too, modify remote_execution_helper.rb, with a copie of https://github.com/theforeman/foreman_remote_execution/blob/291df12dee06d1a15852e3531191bbb869a300d6/app/helpers/remote_execution_helper.rb#L249-L250
if by the way it is more recent than the one I got via rpm, but without success.

this problem appeared to me, since the upgrade to F3.2
The remote tasks(the jobs) are executed successfully; the problem is to show the page of each job.

I even uninstalled the chef and docker plugin so that foreman-installer wouldn’t give me errors.

Well, the weird thing here is the template should always exist, or at least the default one so it feels as if the seeds did not run properly or something else went awry along the way.

In any case, let’s not assume the template will always be there, here’s an issue and an associated fix Bug #35172: Assume job report template may not exist - Foreman Remote Execution - Foreman

Thank you very much @aruzicka

Is this other error(from foreman-installer) related to the same thing ?

2022-07-07 10:59:49 [ERROR ] [configure] Proxy … has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-07 10:59:49 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[…]/features: change from [“Dynflow”, “Logs”, “Pulpcore”, “Puppet”, “Puppet CA”, “Registration”, “TFTP”, “Templates”] to [“Dynflow”, “Logs”, “Pulpcore”, “Puppet”, “Puppet CA”, “Registration”, "Script", “TFTP”, “Templates”] failed: Proxy … has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-07 10:59:50 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[…]: Failed to call refresh: Proxy … has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-07 10:59:50 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[…]: Proxy … has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors

Hard to say from such a short excerpt. What does /var/log/foreman-proxy/proxy.log say?

Nothing explicit for me, but could it be related to the change of module/plugin: remo-execution-ssh for remote-execution-script?

$ cat /var/log/foreman-proxy/proxy.log
...
2022-07-08T10:26:04  [I] start terminating delayed_executor...
2022-07-08T10:26:04  [I] start terminating throttle_limiter...
2022-07-08T10:26:04  [I] start terminating executor...
2022-07-08T10:26:04  [I] shutting down Core ...
2022-07-08T10:26:04  [I] ... Dynflow core terminated.
2022-07-08T10:26:04  [I] start terminating executor dispatcher...
2022-07-08T10:26:04  [I] start terminating client dispatcher...
2022-07-08T10:26:04  [I] stop listening for new events...
2022-07-08T10:26:04  [I] start terminating clock...
2022-07-08T10:26:04  [I] going to shutdown ...
2022-07-08T10:26:04  [I] WEBrick::HTTPServer#start done.
2022-07-08T10:26:06  [I] Successfully initialized 'pulpcore'
2022-07-08T10:26:06  [W] Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.
2022-07-08T10:26:06  [I] Execution plan cleaner removing 0 execution plans.
2022-07-08T10:26:06  [I] Successfully initialized 'dynflow'
2022-07-08T10:26:06  [I] Successfully initialized 'script'
2022-07-08T10:26:06  [I] Successfully initialized 'foreman_proxy'
2022-07-08T10:26:06  [I] Successfully initialized 'templates'
2022-07-08T10:26:06  [I] Successfully initialized 'tftp'
2022-07-08T10:26:06  [I] Successfully initialized 'puppetca_http_api'
2022-07-08T10:26:06  [I] Successfully initialized 'puppetca_hostname_whitelisting'
2022-07-08T10:26:06  [I] Successfully initialized 'puppetca'
2022-07-08T10:26:06  [I] Successfully initialized 'puppet_proxy_puppet_api'
2022-07-08T10:26:06  [I] Successfully initialized 'puppet'
2022-07-08T10:26:06  [I] Successfully initialized 'logs'
2022-07-08T10:26:06  [I] Successfully initialized 'registration'
2022-07-08T10:26:06  [I] WEBrick 1.6.1
2022-07-08T10:26:06  [I] ruby 2.7.4 (2021-07-07) [x86_64-linux]
2022-07-08T10:26:06  [I]
Certificate:
    Data:
        Version: 3 (0x2)
        ...
        Validity
            Not Before: Dec  7 22:13:18 2020 GMT
            Not After : Dec  5 22:13:18 2030 GMT
    ...
2022-07-08T10:26:06  [I] WEBrick::HTTPServer#start: pid=124799 port=9090
2022-07-08T10:26:06  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2022-07-08T10:32:20 22f5da30 [I] Started GET /features
2022-07-08T10:32:20 22f5da30 [I] Finished GET /features with 200 (6.67 ms)
2022-07-08T10:32:22  [I] start terminating delayed_executor...
2022-07-08T10:32:22  [I] start terminating throttle_limiter...
2022-07-08T10:32:22  [I] start terminating executor...
2022-07-08T10:32:22  [I] shutting down Core ...
2022-07-08T10:32:22  [I] ... Dynflow core terminated.
2022-07-08T10:32:22  [I] start terminating executor dispatcher...
2022-07-08T10:32:22  [I] start terminating client dispatcher...
2022-07-08T10:32:22  [I] stop listening for new events...
2022-07-08T10:32:22  [I] start terminating clock...
2022-07-08T10:32:22  [I] going to shutdown ...
2022-07-08T10:32:22  [I] WEBrick::HTTPServer#start done.
2022-07-08T10:42:26  [I] Successfully initialized 'pulpcore'
2022-07-08T10:42:26  [W] Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.
2022-07-08T10:42:26  [I] Execution plan cleaner removing 0 execution plans.
2022-07-08T10:42:26  [I] Successfully initialized 'dynflow'
2022-07-08T10:42:27  [I] Successfully initialized 'script'
2022-07-08T10:42:27  [I] Successfully initialized 'foreman_proxy'
2022-07-08T10:42:27  [I] Successfully initialized 'templates'
2022-07-08T10:42:27  [I] Successfully initialized 'tftp'
2022-07-08T10:42:27  [I] Successfully initialized 'puppetca_http_api'
2022-07-08T10:42:27  [I] Successfully initialized 'puppetca_hostname_whitelisting'
2022-07-08T10:42:27  [I] Successfully initialized 'puppetca'
2022-07-08T10:42:27  [I] Successfully initialized 'puppet_proxy_puppet_api'
2022-07-08T10:42:27  [I] Successfully initialized 'puppet'
2022-07-08T10:42:27  [I] Successfully initialized 'logs'
2022-07-08T10:42:27  [I] Successfully initialized 'registration'
2022-07-08T10:42:27  [I] WEBrick 1.6.1
2022-07-08T10:42:27  [I] ruby 2.7.4 (2021-07-07) [x86_64-linux]
2022-07-08T10:42:27  [I]
Certificate:
    Data:
        Version: 3 (0x2)
      ...
        Validity
            Not Before: Dec  7 22:13:18 2020 GMT
            Not After : Dec  5 22:13:18 2030 GMT
        ...
2022-07-08T10:42:27  [I] WEBrick::HTTPServer#start: pid=2446 port=9090
2022-07-08T10:42:27  [I] Smart proxy has launched on 2 socket(s), waiting for requests
2022-07-08T10:42:28 892b451a [I] Started GET /v2/features
2022-07-08T10:42:29 892b451a [I] Finished GET /v2/features with 200 (718.94 ms)
2022-07-08T10:42:29 892b451a [I] Started GET /v2/features
2022-07-08T10:42:29 892b451a [I] Finished GET /v2/features with 200 (58.72 ms)
2022-07-08T10:42:29 c9d7f9bd [I] Started GET /v2/features
2022-07-08T10:42:29 c9d7f9bd [I] Finished GET /v2/features with 200 (76.34 ms)
2022-07-08T10:42:29 c9d7f9bd [I] Started GET /v2/features
2022-07-08T10:42:30 c9d7f9bd [I] Finished GET /v2/features with 200 (614.15 ms)


 $ grep Script /var/log/foreman-installer/katello.log
2022-07-08 10:35:32 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Plugin[remote_execution_ssh]/before: before to Foreman_proxy::Module[remote_execution_ssh]
2022-07-08 10:35:32 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Plugin[remote_execution_ssh]/notify: notify to Class[Foreman_proxy::Service]
2022-07-08 10:35:32 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/notify: notify to Class[Foreman_proxy::Service]
2022-07-08 10:35:33 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]: Adding autorequire relationship with File[/etc/foreman-proxy]
2022-07-08 10:35:33 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]: Adding autorequire relationship with Group[foreman-proxy]
2022-07-08 10:35:33 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Remote_execution::Script]: Starting to evaluate the resource (93 of 1907)
2022-07-08 10:35:33 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Remote_execution::Script]: Evaluated in 0.00 seconds
2022-07-08 10:37:59 [DEBUG ] [configure] Foreman_proxy::Feature[Script]: Starting to evaluate the resource (1754 of 1993)
2022-07-08 10:37:59 [DEBUG ] [configure] Foreman_proxy::Feature[Script]: Evaluated in 0.00 seconds
2022-07-08 10:37:59 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Feature[Script]/Datacat_fragment[foreman_proxy::enabled_features::Script]: Starting to evaluate the resource (1763 of 1993)
2022-07-08 10:37:59 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Feature[Script]/Datacat_fragment[foreman_proxy::enabled_features::Script]: Evaluated in 0.00 seconds
2022-07-08 10:37:59 [DEBUG ] [configure] Foreman_proxy::Feature[Script]: Starting to evaluate the resource (1764 of 1993)
2022-07-08 10:37:59 [DEBUG ] [configure] Foreman_proxy::Feature[Script]: Evaluated in 0.00 seconds
2022-07-08 10:37:59 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]: Starting to evaluate the resource (1765 of 1993)
2022-07-08 10:37:59 [INFO  ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]/content:
2022-07-08 10:37:59 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]: Filebucketed /etc/foreman-proxy/settings.d/remote_execution_ssh.yml to puppet with sum d40b55494af299da564bebf53a7728a5
2022-07-08 10:37:59 [INFO  ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]/content: content changed '{md5}d40b55494af299da564bebf53a7728a5' to '{md5}60e8d5007bdd260849eda36d773c6439'
2022-07-08 10:37:59 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]: The container Foreman_proxy::Settings_file[remote_execution_ssh] will propagate my refresh event
2022-07-08 10:37:59 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Plugin::Remote_execution::Script/Foreman_proxy::Plugin::Module[remote_execution_ssh]/Foreman_proxy::Module[remote_execution_ssh]/Foreman_proxy::Settings_file[remote_execution_ssh]/File[/etc/foreman-proxy/settings.d/remote_execution_ssh.yml]: Evaluated in 0.03 seconds
2022-07-08 10:37:59 [DEBUG ] [configure] Foreman_proxy::Plugin::Module[remote_execution_ssh]: The container Class[Foreman_proxy::Plugin::Remote_execution::Script] will propagate my refresh event
2022-07-08 10:37:59 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Remote_execution::Script]: Starting to evaluate the resource (1769 of 1993)
2022-07-08 10:37:59 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Remote_execution::Script]: The container Stage[main] will propagate my refresh event
2022-07-08 10:37:59 [DEBUG ] [configure] Class[Foreman_proxy::Plugin::Remote_execution::Script]: Evaluated in 0.00 seconds
2022-07-08 10:42:27 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Collected {"features"=>["Puppet", "Puppet CA", "TFTP", "Templates", "Logs", "Registration", "Pulpcore", "Dynflow", "Script"]}
2022-07-08 10:42:29 [ERROR ] [configure] Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-08 10:42:29 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[cpt-infra05l-p.acceo.com]/features: change from ["Dynflow", "Logs", "Pulpcore", "Puppet", "Puppet CA", "Registration", "SSH", "TFTP", "Templates"] to ["Dynflow", "Logs", "Pulpcore", "Puppet", "Puppet CA", "Registration", "Script", "TFTP", "Templates"] failed: Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-08 10:42:30 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[cpt-infra05l-p.acceo.com]: Failed to call refresh: Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-08 10:42:30 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[cpt-infra05l-p.acceo.com]: Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors

Thank you.

More details, with DEBUG activated in /etc/foreman-proxy/settings.yml:

$ foreman-installer --no-enable-foreman-plugin-chef --enable-foreman-cli-remote-execution --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-script --foreman-proxy-plugin-remote-execution-script-mode="ssh" --foreman-proxy-content-pulpcore-cache-enabled=true
...
2022-07-08 11:21:13 [NOTICE] [configure] Starting system configuration.
2022-07-08 11:21:25 [NOTICE] [configure] 250 configuration steps out of 1907 steps complete.
2022-07-08 11:21:28 [NOTICE] [configure] 500 configuration steps out of 1909 steps complete.
2022-07-08 11:21:32 [NOTICE] [configure] 750 configuration steps out of 1914 steps complete.
2022-07-08 11:21:32 [NOTICE] [configure] 1000 configuration steps out of 1919 steps complete.
2022-07-08 11:22:53 [NOTICE] [configure] 1250 configuration steps out of 1919 steps complete.
2022-07-08 11:22:53 [NOTICE] [configure] 1500 configuration steps out of 1955 steps complete.
2022-07-08 11:22:54 [NOTICE] [configure] 1750 configuration steps out of 1955 steps complete.
2022-07-08 11:23:06 [ERROR ] [configure] Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-08 11:23:06 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[cpt-infra05l-p.acceo.com]/features: change from ["Dynflow", "Logs", "Pulpcore", "Puppet", "Puppet CA", "Registration", "TFTP", "Templates"] to ["Dynflow", "Logs", "Pulpcore", "Puppet", "Puppet CA", "Registration", "Script", "TFTP", "Templates"] failed: Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-08 11:23:06 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[cpt-infra05l-p.acceo.com]: Failed to call refresh: Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-08 11:23:06 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[cpt-infra05l-p.acceo.com]: Proxy cpt-infra05l-p.acceo.com has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-07-08 11:23:10 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

$ cat /var/log/foreman-proxy/proxy.log
...
2022-07-08T11:22:59  [D] Executor heartbeat
2022-07-08T11:23:03  [I] start terminating delayed_executor...
2022-07-08T11:23:03  [I] start terminating throttle_limiter...
2022-07-08T11:23:03  [I] start terminating executor...
2022-07-08T11:23:03  [I] shutting down Core ...
2022-07-08T11:23:03  [I] ... Dynflow core terminated.
2022-07-08T11:23:03  [I] start terminating executor dispatcher...
2022-07-08T11:23:03  [I] start terminating client dispatcher...
2022-07-08T11:23:03  [I] stop listening for new events...
2022-07-08T11:23:03  [I] start terminating clock...
2022-07-08T11:23:03  [D] close TCPSocket(0.0.0.0, 9090)
2022-07-08T11:23:03  [D] close TCPSocket(::, 9090)
2022-07-08T11:23:03  [I] going to shutdown ...
2022-07-08T11:23:03  [I] WEBrick::HTTPServer#start done.
2022-07-08T11:23:05  [I] Successfully initialized 'pulpcore'
2022-07-08T11:23:05  [W] Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.
2022-07-08T11:23:05  [I] Execution plan cleaner removing 0 execution plans.
2022-07-08T11:23:05  [I] Successfully initialized 'dynflow'
2022-07-08T11:23:05  [I] Successfully initialized 'script'
2022-07-08T11:23:05  [I] Successfully initialized 'foreman_proxy'
2022-07-08T11:23:05  [I] Successfully initialized 'templates'
2022-07-08T11:23:05  [I] Successfully initialized 'tftp'
2022-07-08T11:23:05  [I] Successfully initialized 'puppetca_http_api'
2022-07-08T11:23:05  [I] Successfully initialized 'puppetca_hostname_whitelisting'
2022-07-08T11:23:05  [I] Successfully initialized 'puppetca'
2022-07-08T11:23:05  [I] Successfully initialized 'puppet_proxy_puppet_api'
2022-07-08T11:23:05  [I] Successfully initialized 'puppet'
2022-07-08T11:23:05  [I] Successfully initialized 'logs'
2022-07-08T11:23:05  [I] Successfully initialized 'registration'
2022-07-08T11:23:05  [I] WEBrick 1.6.1
2022-07-08T11:23:05  [I] ruby 2.7.4 (2021-07-07) [x86_64-linux]
2022-07-08T11:23:05  [I]
Certificate:
   ...

2022-07-08T11:23:05  [I] WEBrick::HTTPServer#start: pid=15104 port=9090
2022-07-08T11:23:05  [I] Smart proxy has launched on 2 socket(s), waiting for requests
2022-07-08T11:23:05 5add91e4 [I] Started GET /v2/features
2022-07-08T11:23:05 5add91e4 [I] Finished GET /v2/features with 200 (156.95 ms)
2022-07-08T11:23:06 5add91e4 [I] Started GET /v2/features
2022-07-08T11:23:06 5add91e4 [I] Finished GET /v2/features with 200 (40.4 ms)
2022-07-08T11:23:06 701eb84e [I] Started GET /v2/features
2022-07-08T11:23:06 701eb84e [I] Finished GET /v2/features with 200 (40.09 ms)
2022-07-08T11:23:06 701eb84e [I] Started GET /v2/features
2022-07-08T11:23:06 701eb84e [I] Finished GET /v2/features with 200 (270.3 ms)

More errors:

  • Tasks > Remote action: Run hostname on …

Errors:

Could not use any proxy for the [“SSH”, “Script”] job. Consider configuring remote_execution_global_proxy, remote_execution_fallback_proxy in settings

But:

That’s not “more errors”, that’s the same thing the installer tells you about manifesting itself differently.

It most likely all stems from seeds not being run for some reasons. What happens if you try running them manually with forman-rake db:seed?

$ foreman-rake db:seed --trace
** Invoke db:seed (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Execute db:seed
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke db:load_config
** Execute db:abort_if_pending_migrations
** Invoke dynflow:abort_if_pending_migrations (first_time)
** Invoke environment
** Execute dynflow:abort_if_pending_migrations
User with login admin already exists, not seeding as admin.
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Remote execution features is invalid
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/validations.rb:80:in raise_validation_error' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/validations.rb:53:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:318:in block in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:366:in with_transaction_returning_status' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:318:in save!’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/suppressor.rb:48:in save!' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-7.1.0/app/models/job_template.rb:72:in import_raw!’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/db/seeds.d/75-job_templates.rb:10:in block (3 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/db/seeds.d/75-job_templates.rb:6:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/db/seeds.d/75-job_templates.rb:6:in block (2 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in without_auditing’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/db/seeds.d/75-job_templates.rb:3:in block in <top (required)>' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in as’
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in as_anonymous_admin' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/db/seeds.d/75-job_templates.rb:2:in <top (required)>’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:318:in load' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:318:in block in load’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:291:in load_dependency' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:318:in load’
/usr/share/foreman/app/services/foreman_seeder.rb:51:in block (3 levels) in execute' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in as’
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in as_anonymous_admin' /usr/share/foreman/app/services/foreman_seeder.rb:50:in block (2 levels) in execute’
/usr/share/foreman/app/services/foreman_seeder.rb:44:in each' /usr/share/foreman/app/services/foreman_seeder.rb:44:in block in execute’
/usr/share/foreman/app/services/foreman/advisory_lock_manager.rb:21:in block in with_transaction_lock' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in transaction' /usr/share/foreman/app/services/foreman/advisory_lock_manager.rb:17:in with_transaction_lock’
/usr/share/foreman/app/services/foreman_seeder.rb:38:in execute' /usr/share/foreman/db/seeds.rb:15:in <top (required)>’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:318:in load' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:318:in block in load’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:291:in load_dependency' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/dependencies.rb:318:in load’
/usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in block in load' /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in benchmark’
/usr/share/foreman/config/initializers/0_print_time_spent.rb:38:in load' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.4.7/lib/rails/engine.rb:559:in load_seed’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/tasks/database_tasks.rb:440:in load_seed' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/railties/databases.rake:331:in block (2 levels) in <top (required)>’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in block in execute' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in each’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in execute' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in block in invoke_with_call_chain’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in synchronize' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in invoke_with_call_chain’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in invoke' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in invoke_task’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in block (2 levels) in top_level' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in each’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in block in top_level' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in run_with_threads’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in top_level' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in block in run’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in standard_exception_handling' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in run’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in <top (required)>' /opt/rh/rh-ruby27/root/usr/bin/rake:23:in load’
/opt/rh/rh-ruby27/root/usr/bin/rake:23:in `’
Tasks: TOP => db:seed

Thank you.

I see this:
image

when should I see this other:
image

The solution:

Repete the upgrade F3.2/K4.4, with this modification in file:
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-6.0.0/app/helpers/remote_execution_helper.rb

def job_report_template
    template = ReportTemplate.where(name: Setting['remote_execution_job_invocation_report_template']).first
 -   template if template.template_inputs.where(name: 'job_id').exists?
 +   template if template && template.template_inputs.where(name: 'job_id').exists?
  end
  def job_report_template_parameters(job_invocation, template)

AND:

$ su - postgres -c “echo “select id,label,name,job_template_id from remote_execution_features order by id;” | psql foreman”

id | label | name | job_template_id
----±----------------------------------±------------------------------------------±----------------
1 | katello_package_install | Katello: Install Package | 111
2 | katello_package_update | Katello: Update Package | 121
3 | katello_package_remove | Katello: Remove Package | 115
4 | katello_group_install | Katello: Install Package Group | 109
5 | katello_group_update | Katello: Update Package Group | 119
6 | katello_group_remove | Katello: Remove Package Group | 113
7 | katello_errata_install | Katello: Install Errata | 107
8 | katello_service_restart | Katello: Service Restart | 117
9 | ansible_run_host | Run Ansible roles | 123
10 | ansible_run_insights_plan | Ansible: Run Insights maintenance plan | 126
11 | foreman_chef_run_chef_client | Run chef-client Once |
12 | katello_module_stream_action | Katello: Module Stream Actions | 145
13 | puppet_run_host | Run Puppet Once | 104
14 | ansible_run_playbook | Run playbook | 205
15 | ansible_enable_web_console | Enable web console | 250
16 | ansible_run_capsule_upgrade | Upgrade Capsules on given hosts | 245
17 | ansible_configure_cloud_connector | Configure Cloud Connector on given hosts | 246
18 | katello_host_tracer_resolve | Katello: Resolve Traces |
19 | katello_errata_install_by_search | Katello: Install errata by search query |
20 | katello_errata_install_by_search | Katello: Install errata by search query |
21 | katello_package_install_by_search | Katello: Install packages by search query |
22 | katello_packages_update_by_search | Katello: Update Packages by search query |
23 | katello_packages_remove_by_search | Katello: Remove Packages by search query |
(23 rows)

$ echo “delete from remote_execution_features where id in (11,19, 20)” | su - postgres -c ‘psql -d foreman’
DELETE 3
$ foreman-rake db:seed --trace --verbose
** Invoke db:seed (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Execute db:seed
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke db:load_config
** Execute db:abort_if_pending_migrations
** Invoke dynflow:abort_if_pending_migrations (first_time)
** Invoke environment
** Execute dynflow:abort_if_pending_migrations
User with login admin already exists, not seeding as admin.

Continue now with upgrade F3.3/K4.5 without problemes.

Thank you.