Problem:
So I have a few sync-plans that recently stopped updating the products they are attached to. When I went to try and modify them (i.e. turn them off and back on) I get an error “undefined method frozen=' for nil:NilClass" and when I try to delete the sync plan using hammer I get an error "Could not destroy the sync plan: undefined method cancel’ for nil:NilClass”. Not all of my sync-plans have this error (there are a handful of them still running normally) and I can create new ones without issues.
Foreman and Proxy versions:
Foreman 3.5.1 and Katello 4.7.1
Other relevant data:
Here’s the output from /var/log/foreman/production.log when I try to delete one of the affected sync-plans:
2023-03-15T12:15:27 [I|app|d81d2838] Started GET "/apidoc/apipie_checksum" for 10.9.3.40 at 2023-03-15 12:15:27 -0400
2023-03-15T12:15:27 [I|app|d81d2838] Processing by Apipie::ApipiesController#apipie_checksum as JSON
2023-03-15T12:15:27 [I|app|d81d2838] Parameters: {"apipy"=>{}}
2023-03-15T12:15:30 [I|app|d81d2838] Rendered /usr/share/gems/gems/apipie-rails-0.8.2/app/views/apipie/apipies/apipie_checksum.json.erb (Duration: 2401.0ms | Allocations: 2689697)
2023-03-15T12:15:30 [I|app|d81d2838] Completed 200 OK in 2410ms (Views: 2407.0ms | ActiveRecord: 0.0ms | Allocations: 2695574)
2023-03-15T12:15:30 [I|app|c2317282] Started DELETE "/katello/api/organizations/1/sync_plans/7" for 10.9.3.40 at 2023-03-15 12:15:30 -0400
2023-03-15T12:15:30 [I|app|c2317282] Processing by Katello::Api::V2::SyncPlansController#destroy as JSON
2023-03-15T12:15:30 [I|app|c2317282] Parameters: {"api_version"=>"v2", "organization_id"=>"1", "id"=>"7", "sync_plan"=>{}}
2023-03-15T12:15:30 [I|app|c2317282] Authorized user admin(Administrative User)
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on name Oracle Linux OpSys Products (BROKEN)
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on description Weekly Repository Sync Plan for Oracle Linux Operating System Products
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on sync_date 2022-10-24 03:00:00 UTC
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on interval weekly
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on organization_id 1
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on foreman_tasks_recurring_logic_id 7
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on cron_expression
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on task_group_id 13
2023-03-15T12:15:30 [I|aud|c2317282] Katello::SyncPlan (7) destroy event on product_ids []
2023-03-15T12:15:30 [E|app|c2317282] NoMethodError: undefined method `cancel' for nil:NilClass
c2317282 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan.rb:311:in `block in cancel'
c2317282 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan.rb:311:in `tap'
c2317282 | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan.rb:311:in `cancel'
c2317282 | /usr/share/gems/gems/foreman-tasks-7.1.1/app/models/foreman_tasks/task/dynflow_task.rb:32:in `cancel'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/delegation.rb:88:in `each'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/relation/delegation.rb:88:in `each'
c2317282 | /usr/share/gems/gems/foreman-tasks-7.1.1/app/models/foreman_tasks/recurring_logic.rb:87:in `cancel'
c2317282 | /usr/share/gems/gems/katello-4.7.1/app/models/katello/sync_plan.rb:124:in `cancel_recurring_logic'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:427:in `block in make_lambda'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:603:in `catch'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:603:in `block in default_terminator'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:199:in `block in halting'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:512:in `block in invoke_before'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:512:in `each'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:512:in `invoke_before'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:105:in `run_callbacks'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:824:in `_run_destroy_callbacks'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/callbacks.rb:439:in `destroy'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:294:in `block in destroy'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/transactions.rb:294:in `destroy'
c2317282 | /usr/share/gems/gems/katello-4.7.1/app/models/katello/model.rb:23:in `destroy!'
c2317282 | /usr/share/gems/gems/katello-4.7.1/app/controllers/katello/api/v2/sync_plans_controller.rb:87:in `destroy'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/abstract_controller/base.rb:228:in `process_action'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
c2317282 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
c2317282 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
c2317282 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
c2317282 | /usr/share/gems/gems/audited-5.0.2/lib/audited/sweeper.rb:16:in `around'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
c2317282 | /usr/share/gems/gems/audited-5.0.2/lib/audited/sweeper.rb:16:in `around'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:137:in `run_callbacks'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/notifications.rb:203:in `block in instrument'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/notifications.rb:203:in `instrument'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
c2317282 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
c2317282 | /usr/share/gems/gems/katello-4.7.1/app/controllers/katello/concerns/api/api_controller.rb:50:in `process_action'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/abstract_controller/base.rb:165:in `process'
c2317282 | /usr/share/gems/gems/actionview-6.1.7/lib/action_view/rendering.rb:39:in `process'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal.rb:190:in `dispatch'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_controller/metal.rb:254:in `dispatch'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/mapper.rb:49:in `serve'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/journey/router.rb:50:in `block in serve'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/journey/router.rb:32:in `each'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/journey/router.rb:32:in `serve'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/route_set.rb:842:in `call'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/engine.rb:539:in `call'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/railtie.rb:207:in `public_send'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/railtie.rb:207:in `method_missing'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/mapper.rb:20:in `block in <class:Constraints>'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/mapper.rb:49:in `serve'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/journey/router.rb:50:in `block in serve'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/journey/router.rb:32:in `each'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/journey/router.rb:32:in `serve'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/routing/route_set.rb:842:in `call'
c2317282 | /usr/share/gems/gems/katello-4.7.1/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
c2317282 | /usr/share/gems/gems/katello-4.7.1/lib/katello/middleware/event_daemon.rb:10:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
c2317282 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/static_dispatcher.rb:68:in `call'
c2317282 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/extractor/recorder.rb:137:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/static.rb:24:in `call'
c2317282 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
c2317282 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
c2317282 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/conditional_get.rb:40:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
c2317282 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/cookies.rb:689:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/callbacks.rb:98:in `run_callbacks'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/rack/logger.rb:37:in `call_app'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/rack/logger.rb:28:in `call'
c2317282 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
c2317282 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/request_id.rb:26:in `call'
c2317282 | /usr/share/gems/gems/katello-4.7.1/lib/katello/prevent_json_parsing.rb:12:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
c2317282 | /usr/share/gems/gems/activesupport-6.1.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/ssl.rb:77:in `call'
c2317282 | /usr/share/gems/gems/actionpack-6.1.7/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
c2317282 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/engine.rb:539:in `call'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/railtie.rb:207:in `public_send'
c2317282 | /usr/share/gems/gems/railties-6.1.7/lib/rails/railtie.rb:207:in `method_missing'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
c2317282 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
c2317282 | /usr/share/gems/gems/puma-5.6.5/lib/puma/configuration.rb:252:in `call'
c2317282 | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:77:in `block in handle_request'
c2317282 | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
c2317282 | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:76:in `handle_request'
c2317282 | /usr/share/gems/gems/puma-5.6.5/lib/puma/server.rb:443:in `process_client'
c2317282 | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
c2317282 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-03-15T12:15:30 [I|app|c2317282] Completed 500 Internal Server Error in 225ms (Views: 0.5ms | ActiveRecord: 37.0ms | Allocations: 54841)