Foreman 1.20 upgrade issues - wrong number of arguments (given 0, expected 1)

Hi folks,

I upgraded our Foreman instance to 1.20 (katello 3.9) on Friday and assumed all was ok as i never encountered any issues along the way and the gui looked fine post upgrade too.

Anyway I’ve since noticed that none of the daily syncs have completed and I’m seeing this error when i browse any of my defined products - wrong number of arguments (given 0, expected 1).

I kicked off a manual sync but none of the tasks ever completed, they all sat at 0%.

I decided to refresh the manifest based on some posts I’ve read, that task got to 16% and didn’t go any further.

I’m running CentOS 7.6 patched to the latest level.

This is what I can see in the production log in regards to the wrong number of arguments error.

2019-06-03T09:40:46 [E|app|96288] ActionView::Template::Error: wrong number of arguments (given 0, expected 1)
/opt/rh/rh-ruby25/root/usr/local/share/gems/gems/foreman_datacenter-1.20.3/app/helpers/foreman_datacenter/application_helper.rb:79:in `object_name'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:19:in `try!'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/object/try.rb:8:in `try'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.1/app/views/katello/api/v2/layouts/resource.json.erb:1:in `_3bfc97bc82160529dbfb6a6437c6eb13'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
/opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.3.2/lib/deface/action_view_extensions.rb:41:in `render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:66:in `render_with_layout'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/template_renderer.rb:16:in `render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:44:in `render_template'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/renderer/renderer.rb:25:in `render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:103:in `_render_template'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:84:in `render_to_body'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/rendering.rb:25:in `render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:36:in `render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
/opt/rh/rh-ruby25/root/usr/share/ruby/benchmark.rb:308:in `realtime'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:31:in `cleanup_view_runtime'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:45:in `render'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.1/app/lib/katello/api/v2/rendering.rb:44:in `respond_with_template'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.1/app/lib/katello/api/v2/rendering.rb:54:in `respond_with_template_resource'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.1/app/lib/katello/api/v2/rendering.rb:6:in `respond_for_show'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.1/app/controllers/katello/api/v2/products_controller.rb:90:in `show'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.9.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Active jobs

foreman=# select id, label, started_at, state, result from foreman_tasks_tasks where state != 'stopped';
                  id                  |                      label                      |     started_at      |   state   | result
--------------------------------------+-------------------------------------------------+---------------------+-----------+---------
 a0a7be11-93db-4110-a631-c2049ff0b906 | CreatePulpDiskSpaceNotifications                |                     | scheduled | pending
 9f60d644-5116-40ab-9b3c-cf4b66383569 | Actions::Katello::Repository::Sync              | 2019-06-03 06:34:45 | running   | pending
 caf484cd-95a9-40a2-b41e-a8c5884a7624 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:03 | running   | pending
 7a898089-bfcd-4704-aafb-ca9050fc63ff | Actions::Katello::Repository::Sync              | 2019-06-03 06:34:53 | running   | pending
 5a30543c-f383-49a2-974a-b5476c624b20 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:08 | running   | pending
 5f399c59-ced7-431c-b0bb-ee20223a742e | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:09 | running   | pending
 e3ba6db7-1a02-43dd-bff2-c8b674257d9d | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:20 | running   | pending
 63049575-ce8f-4222-87be-8ec5f7612210 | Actions::Katello::Repository::Sync              | 2019-06-03 06:34:31 | running   | pending
 9be6ff0c-f804-4427-b79b-6f2ca85f71de | Actions::Katello::Repository::Sync              | 2019-06-03 06:34:32 | running   | pending
 3347196c-fae5-4338-ad7e-26077491509b | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:13 | running   | pending
 a8dc4070-5033-4307-ba2e-46b4cc0a3b5b | Actions::Katello::Repository::Sync              | 2019-06-03 06:34:25 | running   | pending
 1c60eefb-b4ea-4ee2-bd34-b8d756c84efe | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:19 | running   | pending
 68fede1d-1253-4681-88cc-3d16d099c2c4 | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:17 | running   | pending
 e17dfbf8-dce2-4b9e-8120-c1cb3a5c72f8 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:59 | running   | pending
 8c73e4a2-14b8-45dc-9e34-5ab1e8b07a5c | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:06 | running   | pending
 170c3d65-9b50-451f-b748-ef7e4aec976a | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:34 | running   | pending
 69ffadf7-3996-4e7e-8ae8-eb10e6f2d9a1 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:19 | running   | pending
 eafaa3e1-e56d-4439-9e30-2f5e8f26453f | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:38 | running   | pending
 11e10442-5fa8-4d7f-b2d2-ce8722dc6e7d | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:48 | running   | pending
 84f01317-14d0-4563-b3b5-694e934d9a9f | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:25 | running   | pending
 fbfe69e8-3747-4167-8129-685f409a9c3a | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:34 | running   | pending
 4452497b-db5a-4dad-b9e9-dd08262e65b5 | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:11 | running   | pending
 c6113c0c-8e41-4ba1-a33c-304cc88ff79d | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:21 | running   | pending
 01f62e49-4d86-4bce-b574-e62942e3cd4f | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:11 | running   | pending
 c1fb4c2e-702a-4d32-897d-464167c765ac | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:28 | running   | pending
 23995ea9-93df-4e31-970b-6b23ec11a720 | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:02 | running   | pending
 9c6c1270-3bf2-4136-89ca-1815ccfdc524 | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:29 | running   | pending
 61bc8100-e150-4190-93d2-f60f56c54f8f | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:33 | running   | pending
 a09e38ac-822f-4d24-abf8-05994234cf9f | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:55 | running   | pending
 7d5040a5-6301-44a9-ba21-d07caddb75ca | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:08 | running   | pending
 6875fc47-efc6-477f-b07e-ecffc81ba869 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:36 | running   | pending
 482f4bf7-4825-40a2-b8a1-6a35a5f5c704 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:31 | running   | pending
 07bed047-921d-4165-a159-9790b8d2a764 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:41 | running   | pending
 e861090f-e564-44df-8edc-138ff9a66e56 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:44 | running   | pending
 d6a7fda8-ec19-4ce9-865d-23b2d407762f | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:54 | running   | pending
 5bd7e5ca-4b78-42b1-8138-0bcc0ebfe00d | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:25 | running   | pending
 2d53a3de-b31d-438b-ade2-7506748cc069 | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:39 | running   | pending
 b3b41e7d-c1d2-4ba9-abbf-31c6656bfb74 | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:43 | running   | pending
 cb9de7e0-e911-4168-8e77-7a001e705664 | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:38 | running   | pending
 4f0d5e3f-25ce-469d-b189-b7f31314693b | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:49 | running   | pending
 c1536e5d-9164-4a0d-b734-37d5fd3e6936 | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:58 | running   | pending
 674a518d-548a-4c46-9ade-db843f1e47e7 | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:16 | running   | pending
 8390be98-2c64-4b90-b7a7-2947dce7daf2 | Actions::Katello::Repository::Sync              | 2019-06-03 06:37:28 | running   | pending
 a1bdaceb-2cbf-4d24-95b3-09bc329f7c0d | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:50 | running   | pending
 e8d29dc5-63a9-481e-9273-2b7535c6f2f1 | Actions::Candlepin::ListenOnCandlepinEvents     | 2019-05-31 14:09:43 | running   | pending
 df6a92da-f038-47e2-a558-828900f973bf | Actions::Katello::EventQueue::Monitor           | 2019-05-31 14:09:44 | running   | pending
 3815d45e-712b-4701-84b8-9daa6b15834e | Actions::Katello::Repository::Sync              | 2019-06-03 06:34:59 | running   | pending
 c9055874-cce1-492b-b037-59f0fdf50070 | Actions::Katello::Repository::Sync              | 2019-06-03 06:34:47 | running   | pending
 0a5ca21e-3fb3-4faf-a042-e0956138d323 | Actions::Katello::Repository::Sync              | 2019-06-03 06:35:57 | running   | pending
 aa4ebc48-bc69-43a0-8c85-b9dc1b79b422 | Actions::Katello::Repository::Sync              | 2019-06-03 06:36:03 | running   | pending
 9d7906ab-565e-4782-be23-bf83cbe57219 | Actions::Katello::Organization::ManifestRefresh | 2019-06-03 06:56:42 | running   | warning
(51 rows)

I’ve also just spotted these errors in the logs too.

2019-06-03T07:33:38 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/products/sync"):
2019-06-03T07:36:00 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:36:16 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:37:47 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:39:29 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:41:35 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:44:07 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:44:16 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:45:11 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:45:17 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:46:28 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:50:19 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:51:21 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:52:08 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:53:15 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:54:28 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:55:50 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T07:55:55 [F|app|] ActionController::RoutingError (No route matches [GET] "/assets/spinner.gif"):
2019-06-03T08:00:23 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:19:29 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:25:25 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:27:19 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:32:01 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:35:47 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:39:37 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:41:16 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:42:16 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T08:59:07 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:04:57 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:06:04 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:07:15 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:23:51 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:24:55 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:26:07 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:30:58 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:34:28 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:35:53 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:45:17 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:46:17 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:47:13 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:50:21 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:51:18 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:54:39 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:57:55 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):
2019-06-03T09:59:26 [F|app|] ActionController::RoutingError (No route matches [POST] "/katello/api/v2/repositories/sync_complete"):

the error seems to coming from the foreman datacenter plugin, I’m not sure if it has been updated recently?

the line in question is:

I suggest you open an issue at https://github.com/theforeman/foreman_datacenter/issues

Thanks for the response.

Yeah I upgraded to the latest version of the DC plugin after upgrading Foreman to 1.19 (started out at 1.17).

Would that be related to the issues/error I’m seeing in regards to the repository syncs?

the rest of the issues seems to be katello specific, perhaps open a new thread with those…?