Problem:
*# foreman-maintain content switchover*
*Running Switch support for certain content from Pulp 2 to Pulp 3*
*================================================================================*
*Switch support for certain content from Pulp 2 to Pulp 3: *
*Performing final content migration before switching content [FAIL]*
*Failed executing foreman-rake katello:pulp3_migration, exit status 1:*
* enabled*
*Migration failed, You will want to investigate: https://katello/foreman_tasks/tasks/469c87ff-1433-468c-acf9-8565d2223f17*
*rake aborted!*
*ForemanTasks::TaskError: Task 469c87ff-1433-468c-acf9-8565d2223f17: Katello::Errors::Pulp3Error: null value in column "stream" violates not-null constraint*
*DETAIL: Failing row contains (9c908b9d-d383-47af-af5a-60a995144405, 2021-01-07 13:15:21.860409+00, 2021-01-07 13:15:21.860415+00, libuv, null, {"epel8-buildroot": ["devel"]}, 35915a072c14b42259f6a283cb3848d0a8d251c0f9b9e8777fea0c7e12fdd2ef, 3-CV_CentOS_8-v16_0-9c3c3f91-0eaa-4fdc-8250-3ecddff4689c, a8cb0ecc-8480-4cc2-b9bc-bff1cac57891).*
*/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/lib/katello/tasks/pulp3_migration.rake:31:in `block (2 levels) in <top (required)>'*
*/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'*
*Tasks: TOP => katello:pulp3_migration*
*(See full trace by running task with --trace)*
*Starting task.*
*2021-01-07 08:15:33 -0500: Pre-migrating Pulp 2 ERRATUM content (detail info) 34615/35159*
--------------------------------------------------------------------------------
Scenario [Switch support for certain content from Pulp 2 to Pulp 3] failed.
The following steps ended up in failing state:
[content-switchover]
**NOTE**
This execution took less than 10 minutes to come up with this error.
I am in the process of re-running the migration
* gathered information for this post then re-ran the below. No changes to the system or content. I did not restart the services nor reboot the server.
* foreman-maintain content migration-stats
* foreman-maintain content prepare
* foreman-maintain content switchover
It has been running for ~30 minutes at this time. I will update this with the results.
I am currently re-running
Expected outcome:
Successful switch to pulp3
Foreman and Proxy versions:
- foreman-2.3.1-1.el7.noarch
- foreman-cli-2.3.1-1.el7.noarch
- foreman-debug-2.3.1-1.el7.noarch
- foreman-dynflow-sidekiq-2.3.1-1.el7.noarch
- foreman-installer-2.3.1-1.el7.noarch
- foreman-installer-katello-2.3.1-1.el7.noarch
- foreman-postgresql-2.3.1-1.el7.noarch
- foreman-proxy-2.3.1-1.el7.noarch
- foreman-release-2.3.1-1.el7.noarch
- foreman-release-scl-7-3.el7.noarch
- foreman-selinux-2.3.1-1.el7.noarch
- foreman-service-2.3.1-1.el7.noarch
- katello-3.18.0-1.el7.noarch
- katello-3.18.0-1.el7.noarch
Distribution and version:
CentOS Linux release 7.9.2009 (Core)
Other relevant data:
From TASK:
null value in column "stream" violates not-null constraint
DETAIL: Failing row contains (9c908b9d-d383-47af-af5a-60a995144405, 2021-01-07 13:15:21.860409+00, 2021-01-07 13:15:21.860415+00, libuv, null, {"epel8-buildroot": ["devel"]}, 35915a072c14b42259f6a283cb3848d0a8d251c0f9b9e8777fea0c7e12fdd2ef, 3-CV_CentOS_8-v16_0-9c3c3f91-0eaa-4fdc-8250-3ecddff4689c, a8cb0ecc-8480-4cc2-b9bc-bff1cac57891).
From DYNFLOW:
2: Actions::Pulp3::ContentMigration (skipped) [ 244.19s / 5.54s ]
Queue: default
Started at: 2021-01-07 13:11:32 UTC
Ended at: 2021-01-07 13:15:36 UTC
Real time: 244.19s
Execution time (excluding suspended state): 5.54s
Input:
---
services_checked:
- pulp3
smart_proxy_id: 1
remote_user: admin
remote_cp_user: admin
current_request_id:
current_timezone: UTC
current_user_id: 1
current_organization_id:
current_location_id:
Output:
---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/46506b1c-9019-4e62-913d-edcf0ae71ff5/"
pulp_created: '2021-01-07T13:11:34.626+00:00'
state: failed
name: pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2
started_at: '2021-01-07T13:11:34.947+00:00'
finished_at: '2021-01-07T13:15:21.997+00:00'
error:
traceback: |2
File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
rv = job.perform()
File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
self._result = self._execute()
File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
return self.func(*self.args, **self.kwargs)
File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 137, in migrate_from_pulp2
pre_migrate_all_content(plan)
File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 68, in pre_migrate_all_content
pre_migrate_content_type(content_model, mutable_type, lazy_type, premigrate_hook)
File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 212, in pre_migrate_content_type
content_model.pulp_2to3_detail.pre_migrate_content_detail(pulp2content_batch)
File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py", line 672, in pre_migrate_content_detail
batch_size=DEFAULT_BATCH_SIZE)
File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 468, in bulk_create
self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1211, in _batched_insert
self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)
File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
description: |
null value in column "stream" violates not-null constraint
DETAIL: Failing row contains (9c908b9d-d383-47af-af5a-60a995144405, 2021-01-07 13:15:21.860409+00, 2021-01-07 13:15:21.860415+00, libuv, null, {"epel8-buildroot": ["devel"]}, 35915a072c14b42259f6a283cb3848d0a8d251c0f9b9e8777fea0c7e12fdd2ef, 3-CV_CentOS_8-v16_0-9c3c3f91-0eaa-4fdc-8250-3ecddff4689c, a8cb0ecc-8480-4cc2-b9bc-bff1cac57891).
worker: "/pulp/api/v3/workers/c63718df-291f-44fd-bc0f-b28a93c3acb8/"
child_tasks: []
task_group: "/pulp/api/v3/task-groups/52421fdd-2e82-4908-8235-9cebc50a4ed4/"
progress_reports:
- message: Pre-migrating Pulp 2 RPM content (general info)
code: premigrating.content.general
state: completed
total: 71624
done: 71624
- message: Pre-migrating Pulp 2 RPM content (detail info)
code: premigrating.content.detail
state: completed
total: 71624
done: 71624
- message: Pre-migrating Pulp 2 ERRATUM content (general info)
code: premigrating.content.general
state: completed
total: 35303
done: 35303
- message: Pre-migrating Pulp 2 SRPM content (general info)
code: premigrating.content.general
state: completed
total: 0
done: 0
- message: Pre-migrating Pulp 2 SRPM content (detail info)
code: premigrating.content.detail
state: completed
total: 0
done: 0
- message: Pre-migrating Pulp 2 ERRATUM content (detail info)
code: premigrating.content.detail
state: completed
total: 35303
done: 35303
- message: Processing Pulp 2 repositories, importers, distributors
code: processing.repositories
state: completed
total: 533
done: 533
- message: Pre-migrating Pulp 2 DISTRIBUTION content (general info)
code: premigrating.content.general
state: completed
total: 5
done: 5
- message: Pre-migrating Pulp 2 DISTRIBUTION content (detail info)
code: premigrating.content.detail
state: completed
total: 5
done: 5
- message: Pre-migrating Pulp 2 MODULEMD content (general info)
code: premigrating.content.general
state: completed
total: 189
done: 189
- message: Pre-migrating Pulp 2 MODULEMD content (detail info)
code: premigrating.content.detail
state: completed
total: 189
done: 189
- message: Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)
code: premigrating.content.detail
state: running
total: 360
done: 0
- message: Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)
code: premigrating.content.general
state: running
total: 360
done: 360
created_resources:
- "/pulp/api/v3/task-groups/52421fdd-2e82-4908-8235-9cebc50a4ed4/"
reserved_resources_record:
- pulp_2to3_migration
task_groups:
- pulp_href: "/pulp/api/v3/task-groups/52421fdd-2e82-4908-8235-9cebc50a4ed4/"
description: Migration Sub-tasks
all_tasks_dispatched: false
waiting: 0
skipped: 0
running: 0
completed: 0
canceled: 0
failed: 1
group_progress_reports:
- message: Repo version creation
code: create.repo_version
total: 0
done: 0
- message: Distribution creation
code: create.distribution
total: 0
done: 0
poll_attempts:
total: 35
failed: 1
Error:
Katello::Errors::Pulp3Error
null value in column "stream" violates not-null constraint DETAIL: Failing row contains (9c908b9d-d383-47af-af5a-60a995144405, 2021-01-07 13:15:21.860409+00, 2021-01-07 13:15:21.860415+00, libuv, null, {"epel8-buildroot": ["devel"]}, 35915a072c14b42259f6a283cb3848d0a8d251c0f9b9e8777fea0c7e12fdd2ef, 3-CV_CentOS_8-v16_0-9c3c3f91-0eaa-4fdc-8250-3ecddff4689c, a8cb0ecc-8480-4cc2-b9bc-bff1cac57891).
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:102:in
`block in check_for_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in
`each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in
`check_for_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:133:in
`poll_external_task'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in
`poll_external_task_with_rescue'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in
`block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:40:in
`block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/models/katello/concerns/user_extensions.rb:21:in
`cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:27:in
`as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:39:in
`as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in
`wrap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in
`with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in
`restore_current_request_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in
`restore_curent_timezone'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in
`restore_curent_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
`restore_current_taxonomies'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in
`block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in
`catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in
`block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in
`block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in
`catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in
`with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in
`execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
`block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in
`with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
`block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
`open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
`block (2 levels) in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in
`run_user_code'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
`block in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
`with_telemetry'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
`perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in
`perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in
`execute_job'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in
`block (2 levels) in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in
`block in invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in
`invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in
`block in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in
`block (6 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in
`local'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in
`block (5 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in
`block in <module:Sidekiq>'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in
`block (4 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in
`stats'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in
`block (3 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in
`block (2 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in
`global'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in
`block in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in
`with_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in
`with_job_hash_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in
`dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in
`process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in
`process_one'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in
`watchdog'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in
`block in safe_thread'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in
`block in create_with_logging_context'"