Problem: Smart Proxies do not finish syncing content succesfully (and do not work as proxy)
Hi Guys,
After the better part of this week trying to make it work, I’m lost. And after replying to a few older threads with some scattered info, I decided to group everything I found out so far in a single post
I have re-installed or upgraded my lab’s Foreman/Katello installation a dozen times by now. But there is just no scenario which I can find that does content syncing to proxies properly. I have ran the following scenario’s:
- Composite content views, does not work, see the stacktrace below
- Just a single non-composite content view, same error.
- Re-installed instead of upgrading
- Check with pulp-admin to see if I can get any more information, but it only throws PLP0000 with the python stacktrace as shown below in the journal.
The only situtation that does work (albeit unsurprisingly) is without any LCE assigned to the proxy. So the communication is there and works.
Things I noticed:
- The ‘Content’ Tab of the proxy does show the published content views that are in the assigned LCE and the amount of packages that they contain
- When the download_policy is set to ‘immediate’ the disk of the proxy is filled up with data
- The errors seem (but I don’t know for sure!) to have “something” to do with yum groups?
- It happens with both EL7 and EL8 repos (so not limited to AppStream or anything else)
Expected outcome: Content to be synced and proxies able to do host discovery, deployment etc.
Foreman and Proxy versions:
Foreman 2.0, 2.1, 2.2, 2.3. Either upgraded or freshly installed
Katello 3.15, 3.16, 3.17, 3.18. Either upgraded or freshly installed
Foreman and Proxy plugin versions:
Distribution and version:
CentOS 7
Other relevant data:
Python stacktrace
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) 'NoneType' object has no attribute 'findall'
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) Traceback (most recent call last):
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 294, in run
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) self.update_content(metadata_files, url)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 627, in update_content
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) purge.purge_unwanted_units(metadata_files, self.conduit, self.config, catalog)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 46, in purge_unwanted_units
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) remove_missing_groups(metadata_files, conduit)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 165, in remove_missing_groups
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) remote_named_tuples = get_remote_units(file_function, group.GROUP_TAG, process_func)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 277, in get_remote_units
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) for unit in package_info_generator:
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) package_info = process_func(element)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) packagelist.findall('packagereq'))
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) AttributeError: 'NoneType' object has no attribute 'findall'
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp.server.async.tasks:INFO: [8a40d3d9] Task failed : [8a40d3d9-a7f2-4694-8e5a-05bb317e6da9]
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) Task pulp.server.managers.repo.sync.sync[8a40d3d9-a7f2-4694-8e5a-05bb317e6da9] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) Traceback (most recent call last):
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) R = retval = fun(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) return super(Task, self).__call__(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) return super(PulpTask, self).__call__(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) return self.run(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 854, in sync
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) PulpExecutionException: Importer indicated a failed response
Output from Foreman Task
Actions::Pulp::Consumer::SyncCapsule
Input:
{"capsule_id"=>2,
"repo_pulp_id"=>"1-CV_test-Library-1771db1b-36b8-4500-817e-234e7b9f8a4c",
"repository_id"=>3,
"sync_options"=>{"remove_missing"=>true},
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"current_request_id"=>nil,
"current_timezone"=>"Europe/Amsterdam",
"current_user_id"=>4,
"current_organization_id"=>nil,
"current_location_id"=>nil}
Output:
{"smart_proxy_history_id"=>3,
"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.sync.sync",
"_href"=>"/pulp/api/v2/tasks/fc365a17-4ac8-4706-b5ac-da83522e8345/",
"task_id"=>"fc365a17-4ac8-4706-b5ac-da83522e8345",
"tags"=>
["pulp:repository:1-CV_test-Library-1771db1b-36b8-4500-817e-234e7b9f8a4c",
"pulp:action:sync"],
"finish_time"=>"2020-12-11T21:32:27Z",
"_ns"=>"task_status",
"start_time"=>"2020-12-11T21:29:29Z",
"traceback"=>
"Traceback (most recent call last):\n" +
" File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" +
" return super(Task, self).__call__(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" +
" return super(PulpTask, self).__call__(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
" return self.run(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 854, in sync\n" +
" raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" +
"PulpExecutionException: Importer indicated a failed response\n",
"spawned_tasks"=>[],
"progress_report"=>
{"yum_importer"=>
{"content"=>
{"size_total"=>0,
"items_left"=>0,
"items_total"=>0,
"state"=>"FAILED",
"size_left"=>0,
"details"=>
{"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
"error"=>"'NoneType' object has no attribute 'findall'",
"error_details"=>[]},
"comps"=>{"state"=>"NOT_STARTED"},
"purge_duplicates"=>{"state"=>"NOT_STARTED"},
"distribution"=>
{"items_total"=>0,
"state"=>"NOT_STARTED",
"error_details"=>[],
"items_left"=>0},
"modules"=>{"state"=>"NOT_STARTED"},
"errata"=>{"state"=>"NOT_STARTED"},
"metadata"=>{"state"=>"FINISHED"}}},
"queue"=>"reserved_resource_worker-0@smartproxy.lont.htm.lan.dq2",
"state"=>"error",
"worker_name"=>"reserved_resource_worker-0@smartproxy.lont.htm.lan",
"result"=>nil,
"error"=>
{"code"=>"PLP0000",
"data"=>{},
"description"=>"Importer indicated a failed response",
"sub_errors"=>[]},
"_id"=>{"$oid"=>"5fd3e4b9ae22663ace405dd5"},
"id"=>"5fd3e4b9ae22663ace405dd5"}],
"poll_attempts"=>{"total"=>32, "failed"=>1}}
Exception:
Katello::Errors::PulpError: PLP0000: Importer indicated a failed response
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:11:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'