Unable to upload/download from file repo

Support
1

Problem:
After I upgraded from Foreman 2.2/Katello 3.17 to Foreman 2.3.1/Katello 3.18, I am unable to upload or download artifacts from existing file repos.

Expected outcome:
To be able to download/upload files.

Foreman and Proxy versions:
Version 2.3.1

Foreman and Proxy plugin versions:

Pulp

Version

2.1.0

Pulp server version

2.21.4

Pulpcore

Version

2.1.0

Supported Content Types

  • docker
  • file
  • yum (supported, but overridden to use Pulp 2)

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:

When I try to upload an artifact to a file repo and logged in as ADMIN:

 Error during upload: Task 2375014f-0ced-4939-aa4d-9f02cfe05e7a: PulpcoreClient::ApiError: Error message: the server returns an error HTTP status code: 403 Response headers: {"date"=>"Tue, 05 Jan 2021 15:12:56 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"} Response body: {"detail":"You do not have permission to perform this action."}; PulpcoreClient::ApiError: Error message: the server returns an error HTTP status code: 403 Response headers: {"date"=>"Tue, 05 Jan 2021 15:12:56 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"} Response body: {"detail":"You do not have permission to perform this action."}; PulpFileClient::ApiError: Error message: the server returns an error HTTP status code: 400 Response headers: {"date"=>"Tue, 05 Jan 2021 15:12:56 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"44", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"} Response body: {"base_path":["This field must be unique."]}; Errno::ENOENT: No such file or directory @ apply2files - /usr/share/foreman/tmp/uploads/RackMultipart20210105-3378-rk81zj.tgz

And in foreman.log:

2021-01-05T16:53:36 [I|app|c3e3ca52] Started POST "/katello/api/v2/repositories/77/upload_content" for 127.0.0.1 at 2021-01-05 16:53:36 +0200
2021-01-05T16:53:36 [I|app|c3e3ca52] Processing by Katello::Api::V2::RepositoriesController#upload_content as HTML
2021-01-05T16:53:36 [I|app|c3e3ca52]   Parameters: {"content"=>[#<ActionDispatch::Http::UploadedFile:0x0000000008214500 @tempfile=#<Tempfile:/tmp/RackMultipart20210105-3378-1uiw0hi.tgz>, @original_filename="cni-plugins-linux-amd64-v0.8.7.tgz", @content_type="application/x-compressed", @headers="Content-Disposition: form-data; name=\"content[]\"; filename=\"cni-plugins-linux-amd64-v0.8.7.tgz\"\r\nContent-Type: application/x-compressed\r\n">], "authenticity_token"=>"vlSVALNnv40ZVY0LKicCOcAwV3uqhM+Vl0eQq0MBZ+7RKpZZbrP4RIx7IuX4CmJrn8LQqfuPs8YtHfH4cxruww==", "api_version"=>"v2", "id"=>"77"}
2021-01-05T16:53:36 [I|bac|c3e3ca52] Task {label: Actions::Katello::Repository::UploadFiles, id: cb4120f8-2309-47b7-804a-58e201d59b31, execution_plan_id: d0932cd7-1c8f-4e5d-8b15-8d88d6c9cd05} state changed: planning
2021-01-05T16:53:38 [I|bac|] Task {label: Actions::Katello::Repository::UploadFiles, id: cb4120f8-2309-47b7-804a-58e201d59b31, execution_plan_id: d0932cd7-1c8f-4e5d-8b15-8d88d6c9cd05} state changed: planned
2021-01-05T16:53:38 [I|bac|] Task {label: Actions::Katello::Repository::UploadFiles, id: cb4120f8-2309-47b7-804a-58e201d59b31, execution_plan_id: d0932cd7-1c8f-4e5d-8b15-8d88d6c9cd05} state changed: running
2021-01-05T16:53:38 [E|bac|] Error message: the server returns an error
 | HTTP status code: 403
 | Response headers: {"date"=>"Tue, 05 Jan 2021 14:53:38 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"}
 | Response body: {"detail":"You do not have permission to perform this action."} (PulpcoreClient::ApiError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.7.1/lib/pulpcore_client/api_client.rb:81:in `call_api'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.7.1/lib/pulpcore_client/api/tasks_api.rb:283:in `read_with_http_info'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.7.1/lib/pulpcore_client/api/tasks_api.rb:230:in `read'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/services/katello/pulp3/task.rb:56:in `task_data'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/services/katello/pulp3/task.rb:64:in `task_group_href'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:119:in `block in add_task_groups'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:118:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:118:in `add_task_groups'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:112:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-01-05T16:53:38 [E|bac|] Error message: the server returns an error
 | HTTP status code: 403
 | Response headers: {"date"=>"Tue, 05 Jan 2021 14:53:38 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"}
 | Response body: {"detail":"You do not have permission to perform this action."} (PulpcoreClient::ApiError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.7.1/lib/pulpcore_client/api_client.rb:81:in `call_api'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.7.1/lib/pulpcore_client/api/tasks_api.rb:283:in `read_with_http_info'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulpcore_client-3.7.1/lib/pulpcore_client/api/tasks_api.rb:230:in `read'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/services/katello/pulp3/task.rb:56:in `task_data'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/services/katello/pulp3/task.rb:64:in `task_group_href'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:119:in `block in add_task_groups'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:118:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:118:in `add_task_groups'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:112:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/execute_if_contents_changed.rb:5:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-01-05T16:53:38 [E|bac|] No such file or directory @ apply2files - /usr/share/foreman/tmp/uploads/RackMultipart20210105-3378-1uiw0hi.tgz (Errno::ENOENT)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/katello/repository/upload_files.rb:74:in `delete'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/katello/repository/upload_files.rb:74:in `block in delete_tmp_files'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/katello/repository/upload_files.rb:74:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/katello/repository/upload_files.rb:74:in `delete_tmp_files'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/katello/repository/upload_files.rb:50:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:593:in `block (2 levels) in execute_finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:20:in `block in finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:19:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:23:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:19:in `block in finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:19:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:19:in `block in finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:19:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:19:in `block in finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:19:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `block in finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:40:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:592:in `block in execute_finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:591:in `execute_finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:78:in `run_step'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:63:in `dispatch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:70:in `block in run_in_sequence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:70:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:70:in `all?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:70:in `run_in_sequence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:59:in `dispatch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:28:in `block in finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:48:in `finalize_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:48:in `finalize_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:48:in `finalize_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:48:in `finalize_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:48:in `finalize_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:48:in `finalize_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/common/transaction.rb:10:in `finalize_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director/sequential_manager.rb:27:in `finalize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:122:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-01-05T16:53:38 [E|bac|] Error message: the server returns an error
 | HTTP status code: 400
 | Response headers: {"date"=>"Tue, 05 Jan 2021 14:53:38 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"44", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"}
 | Response body: {"base_path":["This field must be unique."]} (PulpFileClient::ApiError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_file_client-1.3.0/lib/pulp_file_client/api_client.rb:81:in `call_api'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_file_client-1.3.0/lib/pulp_file_client/api/distributions_file_api.rb:79:in `create_with_http_info'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_file_client-1.3.0/lib/pulp_file_client/api/distributions_file_api.rb:28:in `create'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/services/katello/pulp3/repository.rb:233:in `create_distribution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/services/katello/pulp3/repository.rb:227:in `refresh_distributions'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/repository/refresh_distribution.rb:27:in `invoke_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/execute_if_contents_changed.rb:5:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2021-01-05T16:53:39 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: d7099064-e928-43e3-a00a-12fea73972a3, execution_plan_id: 70e0e3ca-6d86-4368-b716-f0b0e9ea940f} state changed: planning
2021-01-05T16:53:39 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: d7099064-e928-43e3-a00a-12fea73972a3, execution_plan_id: 70e0e3ca-6d86-4368-b716-f0b0e9ea940f} state changed: planned
2021-01-05T16:53:39 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: d7099064-e928-43e3-a00a-12fea73972a3, execution_plan_id: 70e0e3ca-6d86-4368-b716-f0b0e9ea940f} state changed: running
2021-01-05T16:53:39 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: d7099064-e928-43e3-a00a-12fea73972a3, execution_plan_id: 70e0e3ca-6d86-4368-b716-f0b0e9ea940f} state changed: stopped  result: success
2021-01-05T16:53:39 [I|bac|] Task {label: Actions::Katello::Repository::CapsuleSync, id: d7099064-e928-43e3-a00a-12fea73972a3, execution_plan_id: 70e0e3ca-6d86-4368-b716-f0b0e9ea940f} state changed: stopped  result: success
2021-01-05T16:53:39 [I|bac|] Task {label: Actions::Katello::Repository::UploadFiles, id: cb4120f8-2309-47b7-804a-58e201d59b31, execution_plan_id: d0932cd7-1c8f-4e5d-8b15-8d88d6c9cd05} state changed: stopped  result: warning
2021-01-05T16:53:39 [I|bac|] Task {label: Actions::Katello::Repository::UploadFiles, id: cb4120f8-2309-47b7-804a-58e201d59b31, execution_plan_id: d0932cd7-1c8f-4e5d-8b15-8d88d6c9cd05} state changed: stopped  result: warning
2021-01-05T16:53:39 [E|app|] ForemanTasks::TaskError: Task cb4120f8-2309-47b7-804a-58e201d59b31: PulpcoreClient::ApiError: Error message: the server returns an error
 | HTTP status code: 403
 | Response headers: {"date"=>"Tue, 05 Jan 2021 14:53:38 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"}
 | Response body: {"detail":"You do not have permission to perform this action."}; PulpcoreClient::ApiError: Error message: the server returns an error
 | HTTP status code: 403
 | Response headers: {"date"=>"Tue, 05 Jan 2021 14:53:38 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"}
 | Response body: {"detail":"You do not have permission to perform this action."}; PulpFileClient::ApiError: Error message: the server returns an error
 | HTTP status code: 400
 | Response headers: {"date"=>"Tue, 05 Jan 2021 14:53:38 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"44", "via"=>"1.1 satellite.do.cbn.ro", "connection"=>"close"}
 | Response body: {"base_path":["This field must be unique."]}; Errno::ENOENT: No such file or directory @ apply2files - /usr/share/foreman/tmp/uploads/RackMultipart20210105-3378-1uiw0hi.tgz
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/lib/foreman_tasks.rb:59:in `block in sync_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/lib/foreman_tasks.rb:58:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/lib/foreman_tasks.rb:58:in `sync_task'

I didn’t figure out if it’s a filesystem permission error, account error or something else…
Any ideas where to look next?

2

I am experiencing the same problem. I get the same error after upgrading from Foreman 2.2/Katello 3.17 to Foreman 2.3.1/Katello 3.18.
Is there any update on this issue?

3

To be more precise, the error that shows up in the messages file, is the following.Hopefully this will point in the right direction.

Jun 9 20:38:15 hs-fm-gen-001 pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 4ef16730-b809-4b4a-bfbc-337d29201715
Jun 9 20:38:15 hs-fm-gen-001 pulpcore-api: - - [09/Jun/2021:20:38:15 +0000] “POST /pulp/api/v3/uploads/8db260b2-a96b-4d22-9249-8ed9bf10d596/commit/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 9 20:38:15 hs-fm-gen-001 pulpcore-api: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/daa00ca3-5ad5-4512-a52b-80ec8655dd06/
Jun 9 20:38:15 hs-fm-gen-001 pulpcore-api: - - [09/Jun/2021:20:38:15 +0000] “GET /pulp/api/v3/tasks/daa00ca3-5ad5-4512-a52b-80ec8655dd06/ HTTP/1.1” 403 63 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 9 20:38:15 hs-fm-gen-001 pulpcore-worker-2: pulp: rq.worker:INFO: 1376@hs-fm-gen-001.funix.nl: daa00ca3-5ad5-4512-a52b-80ec8655dd06
Jun 9 20:38:15 hs-fm-gen-001 pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (4ef16730-b809-4b4a-bfbc-337d29201715)
Jun 9 20:38:15 hs-fm-gen-001 pulpcore-api: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/daa00ca3-5ad5-4512-a52b-80ec8655dd06/
Jun 9 20:38:15 hs-fm-gen-001 pulpcore-api: - - [09/Jun/2021:20:38:15 +0000] “GET /pulp/api/v3/tasks/daa00ca3-5ad5-4512-a52b-80ec8655dd06/ HTTP/1.1” 403 63 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 9 20:38:15 hs-fm-gen-001 dynflow-sidekiq@worker: 2021-06-09T20:38:15.254Z 1396 TID-2begf8 WARN: {“context”:“Job raised exception”,“job”:{“queue”:“default”,“args”:[{“class”:“Dynflow::Director::StepWorkItem”,“execution_plan_id”:“34864438-25ae-4336-bdba-fd72f791211a”,“queue”:“default”,“sender_orchestrator_id”:“67034a09-83d4-438a-967c-e31bd18f8501”,“step”:{“execution_plan_uuid”:“34864438-25ae-4336-bdba-fd72f791211a”,“id”:4,“state”:“pending”,“class”:“Dynflow::ExecutionPlan::Steps::RunStep”,“action_class”:“Actions::Pulp3::Repository::CommitUpload”,“action_id”:3,“error”:null,“started_at”:null,“ended_at”:null,“execution_time”:0.0,“real_time”:0.0,“progress_done”:0,“progress_weight”:1.0,“queue”:“default”}}],“class”:“Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork”,“retry”:false,“backtrace”:true,“jid”:“f7289291c8964121bd96013b”,“created_at”:1623271095.0910335,“enqueued_at”:1623271095.0910726},“jobstr”:"{“queue”:“default”,“args”:[{“class”:“Dynflow::Director::StepWorkItem”,“execution_plan_id”:“34864438-25ae-4336-bdba-fd72f791211a”,“queue”:“default”,“sender_orchestrator_id”:“67034a09-83d4-438a-967c-e31bd18f8501”,“step”:{“execution_plan_uuid”:“34864438-25ae-4336-bdba-fd72f791211a”,“id”:4,“state”:“pending”,“class”:“Dynflow::ExecutionPlan::Steps::RunStep”,“action_class”:“Actions::Pulp3::Repository::CommitUpload”,“action_id”:3,“error”:null,“started_at”:null,“ended_at”:null,“execution_time”:0.0,“real_time”:0.0,“progress_done”:0,“progress_weight”:1.0,“queue”:“default”}}],“class”:“Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork”,“retry”:false,“backtrace”:true,“jid”:“f7289291c8964121bd96013b”,“created_at”:1623271095.0910335,“enqueued_at”:1623271095.0910726}"}
Jun 9 20:38:15 hs-fm-gen-001 dynflow-sidekiq@worker: 2021-06-09T20:38:15.255Z 1396 TID-2begf8 WARN: PulpcoreClient::ApiError: Error message: the server returns an error
Jun 9 20:38:15 hs-fm-gen-001 dynflow-sidekiq@worker: HTTP status code: 403
Jun 9 20:38:15 hs-fm-gen-001 dynflow-sidekiq@worker: Response headers: {“date”=>“Wed, 09 Jun 2021 20:38:15 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, PATCH, DELETE, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“63”, “via”=>“1.1 hs-fm-gen-001.funix.nl”, “connection”=>“close”}
Jun 9 20:38:15 hs-fm-gen-001 dynflow-sidekiq@worker: Response body: {“detail”:“You do not have permission to perform this action.”}

4

Could I have a little help here please? It would be much appreciated.
It’s a real pity that we can not add new files to our files repo anymore.
I also tried to create a new files repo, but that gives the same result.

To be clear, I have not tried the migration to Pulp 3 yet.

5

Can we first find out which version of pulp you are using?
/pulp/api/v3 in your log messages means that is pulp 3.

1 Like