Problem:
All repos fail to sync, exception error: Katello::Errors::PulpError: PLP0000: can’t compare datetime.datetime to NoneType
Expected outcome:
Successful Repo sync
Foreman and Proxy versions:
Foreman 1.24.3
Foreman-tasks 0.17.5
Pulp Version 1.5.0
Pulp server version 2.21.5
Katello 3.14.1
Distribution and version:
CentOS 7.9.2009
Other relevant data:
Recent system change was a yum update all resulting in the pulp services failing to start with an error: “Pulp Fail with error: pulp.server.db.migrate.models:ERROR: pulp.server.db.migrations hasn’t been updated to the latest available migration.”
Resolved pulp not starting by reverting the pulp.server.db.migrations version back one version which has now resulted in none of the repos syncing with the PLP0000: can’t compare datetime.datetime to NoneType error.
Action:
Actions::Pulp::Repository::Sync
Input:
{“smart_proxy_id”=>1,
“repo_id”=>25,
“source_url”=>nil,
“options”=>{},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>nil,
“current_timezone”=>“Europe/London”,
“current_user_id”=>6,
“current_organization_id”=>1,
“current_location_id”=>nil}
Output:
{“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/682c317f-eba2-451e-a1b6-304be254994c/",
“task_id”=>“682c317f-eba2-451e-a1b6-304be254994c”,
“tags”=>
[“pulp:repository:2f8aba55-25d1-4556-b3c3-b3854be6c7ff”,
“pulp:action:sync”],
“finish_time”=>“2021-06-02T08:49:35Z”,
“_ns”=>“task_status”,
“start_time”=>“2021-06-02T08:49:29Z”,
“traceback”=>nil,
“spawned_tasks”=>
[{"_href"=>"/pulp/api/v2/tasks/a0645879-db09-47f6-9ed3-1100f446380f/",
“task_id”=>“a0645879-db09-47f6-9ed3-1100f446380f”}],
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“items_total”=>0,
“state”=>“FINISHED”,
“error_details”=>,
“details”=>
{“rpm_total”=>0, “rpm_done”=>0, “drpm_total”=>0, “drpm_done”=>0},
“size_total”=>0,
“size_left”=>0,
“items_left”=>0},
“comps”=>{“state”=>“FINISHED”},
“purge_duplicates”=>{“state”=>“FINISHED”},
“distribution”=>
{“items_total”=>0,
“state”=>“FINISHED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“FINISHED”},
“errata”=>{“state”=>“FINISHED”},
“metadata”=>{“state”=>“FINISHED”}}},
“queue”=>“reserved_resource_worker-2@HOSTNAME.domain.dq2”,
“state”=>“finished”,
“worker_name”=>“reserved_resource_worker-2@HOSTNAME.domain”,
“result”=>
{“result”=>“success”,
“importer_id”=>“yum_importer”,
“exception”=>nil,
“repo_id”=>“2f8aba55-25d1-4556-b3c3-b3854be6c7ff”,
“traceback”=>nil,
“started”=>“2021-06-02T08:49:29Z”,
“_ns”=>“repo_sync_results”,
“completed”=>“2021-06-02T08:49:35Z”,
“importer_type_id”=>“yum_importer”,
“error_message”=>nil,
“summary”=>
{“modules”=>{“state”=>“FINISHED”},
“content”=>{“state”=>“FINISHED”},
“comps”=>{“state”=>“FINISHED”},
“purge_duplicates”=>{“state”=>“FINISHED”},
“distribution”=>{“state”=>“FINISHED”},
“errata”=>{“state”=>“FINISHED”},
“metadata”=>{“state”=>“FINISHED”}},
“added_count”=>0,
“removed_count”=>0,
“updated_count”=>0,
“id”=>“60b7461f2685ab403b28ba70”,
“details”=>
{“modules”=>{“state”=>“FINISHED”},
“content”=>
{“size_total”=>0,
“items_left”=>0,
“items_total”=>0,
“state”=>“FINISHED”,
“size_left”=>0,
“details”=>
{“rpm_total”=>0, “rpm_done”=>0, “drpm_total”=>0, “drpm_done”=>0},
“error_details”=>},
“comps”=>{“state”=>“FINISHED”},
“purge_duplicates”=>{“state”=>“FINISHED”},
“distribution”=>
{“items_total”=>0,
“state”=>“FINISHED”,
“error_details”=>,
“items_left”=>0},
“errata”=>{“state”=>“FINISHED”},
“metadata”=>{“state”=>“FINISHED”}}},
“error”=>nil,
“_id”=>{"$oid"=>“60b74619bc3ca80407d949f8”},
“id”=>“60b74619bc3ca80407d949f8”},
{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.publish.publish”,
“_href”=>"/pulp/api/v2/tasks/a0645879-db09-47f6-9ed3-1100f446380f/",
“task_id”=>“a0645879-db09-47f6-9ed3-1100f446380f”,
“tags”=>
[“pulp:repository:2f8aba55-25d1-4556-b3c3-b3854be6c7ff”,
“pulp:action:publish”],
“finish_time”=>“2021-06-02T08:49:35Z”,
“_ns”=>“task_status”,
“start_time”=>“2021-06-02T08:49:35Z”,
“traceback”=>
“Traceback (most recent call last):\n” +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 688, in call\n" +
" return super(Task, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 110, in call\n" +
" return super(PulpTask, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call\n" +
" return self.run(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 1142, in publish\n" +
" result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 1184, in check_publish\n" +
" dist_updated = dist.last_updated > last_published\n" +
“TypeError: can’t compare datetime.datetime to NoneType\n”,
“spawned_tasks”=>,
“progress_report”=>{},
“queue”=>“reserved_resource_worker-2@HOSTNAME.domain.dq2”,
“state”=>“error”,
“worker_name”=>“reserved_resource_worker-2@HOSTNAME.domain”,
“result”=>nil,
“error”=>
{“code”=>“PLP0000”,
“data”=>{},
“description”=>“can’t compare datetime.datetime to NoneType”,
“sub_errors”=>},
“_id”=>{"$oid"=>“60b7461fbc3ca80407d94a79”},
“id”=>“60b7461fbc3ca80407d94a79”}],
“contents_changed”=>false,
“poll_attempts”=>{“total”=>9, “failed”=>1}}
Exception:
Katello::Errors::PulpError: PLP0000: can’t compare datetime.datetime to NoneType
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:122:in block in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:120:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:120:in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/repository/sync.rb:28:in external_task=’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:22:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/cancellable.rb:14:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:45:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:563:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:16:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:17:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:11:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:22:in with_locale' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:11:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:562:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:561:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:561:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:556:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:285:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director.rb:93:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:15:in block in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors.rb:12:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:14:in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/context.rb:46:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/executes_context.rb:7:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:106:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/awaits.rb:15:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:47:in block in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:59:in with_backtrace' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:47:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:38:in process_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:20:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/termination.rb:55:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/removes_child.rb:10:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:162:in process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:96:in block in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:119:in block (2 levels) in schedule_execution' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in block in synchronize’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in synchronize' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:116:in block in schedule_execution' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:18:in call’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:96:in work' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:77:in block in call_job’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in run_task' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in block (3 levels) in create_worker’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in loop' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in block (2 levels) in create_worker’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in block in create_worker’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’