Problem:
Ubuntu repository sync fails
Error messages:
PLP0000: [Errno 2] No such file or directory: u’/var/cache/pulp/reserved_resource_worker-0@katelloserver.domain/874cae80-5372-4924-a300-924fde786980/packages/02/c9/dsdp_5.8-9.4_amd64.deb’
it’s always a different .deb, at first I had the whole repository in one without specifying components, then I splitted it up in different repositories, but the error persists in the universe repo, since it contains over 50k packages
it sometimes stuck at 10k packages, sometimes 20k, completely random. I have to use a proxy, and it seems if any package fails the whole sync stops und cache is cleared so it has to start all over again
Expected outcome:
sync completes
Foreman and Proxy versions:
Foreman and Proxy plugin versions:
foreman-2.3.2-1.el7.noarch
foreman-cli-2.3.2-1.el7.noarch
foreman-debug-2.3.2-1.el7.noarch
foreman-dynflow-sidekiq-2.3.2-1.el7.noarch
foreman-installer-2.3.2-1.el7.noarch
foreman-installer-katello-2.3.2-1.el7.noarch
foreman-postgresql-2.3.2-1.el7.noarch
foreman-proxy-2.3.2-1.el7.noarch
foreman-release-2.3.2-1.el7.noarch
foreman-selinux-2.3.2-1.el7.noarch
foreman-service-2.3.2-1.el7.noarch
katello-3.18.1-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.18.1-1.el7.noarch
katello-debug-3.18.1-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-3.18.1-1.el7.noarch
katello-selinux-3.5.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
pulp-katello-1.0.3-1.el7.noarch
rubygem-foreman_maintain-0.7.1-1.el7.noarch
tfm-rubygem-foreman_remote_execution-4.2.1-1.fm2_3.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-foreman-tasks-3.0.2-1.fm2_3.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-0.24.2-1.el7.noarch
tfm-rubygem-katello-3.18.1-1.el7.noarch
Distribution and version:
Red Hat Enterprise Linux Server release 7.9 (Maipo)
Other relevant data:
2021-01-21T16:13:25 [I|app|69befe41] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/views/foreman_tasks/api/tasks/details.json.rabl
2021-01-21T16:13:25 [I|app|69befe41] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/views/foreman_tasks/api/tasks/details.json.rabl (Duration: 33.7ms | Allocations: 31905)
2021-01-21T16:13:25 [I|app|69befe41] Completed 200 OK in 41ms (Views: 31.8ms | ActiveRecord: 3.5ms | Allocations: 34665)
2021-01-21T16:13:30 [I|app|e69f1894] Started GET "/foreman_tasks/api/tasks/e93368df-842c-4bc0-b7f4-2930fceed997/details?include_permissions" for 127.0.0.1 at 2021-01-21 16:13:30 +0100
2021-01-21T16:13:30 [I|app|e69f1894] Processing by ForemanTasks::Api::TasksController#details as JSON
2021-01-21T16:13:30 [I|app|e69f1894] Parameters: {"include_permissions"=>nil, "id"=>"e93368df-842c-4bc0-b7f4-2930fceed997"}
2021-01-21T16:13:30 [I|app|e69f1894] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/views/foreman_tasks/api/tasks/details.json.rabl
2021-01-21T16:13:30 [I|app|e69f1894] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/views/foreman_tasks/api/tasks/details.json.rabl (Duration: 31.7ms | Allocations: 31905)
2021-01-21T16:13:30 [I|app|e69f1894] Completed 200 OK in 38ms (Views: 29.9ms | ActiveRecord: 3.1ms | Allocations: 34665)
2021-01-21T16:17:41 [E|bac|] PLP0000: [Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-0@uslinrepo01.wstw.energy-it.net/874cae80-5372-4924-a300-924fde786980/packages/02/c9/dsdp_5.8-9.4_amd64.deb' (Katello::Errors::PulpError)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp/repository/sync.rb:29:in `external_task='
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
| /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
| [ sidekiq ]
| [ concurrent-ruby ]
2021-01-21T16:17:41 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: e93368df-842c-4bc0-b7f4-2930fceed997, execution_plan_id: 46daac40-b5e9-4880-afaf-556fc07305bf} state changed: stopped result: warning
2021-01-21T16:17:41 [I|bac|] Task {label: Actions::Katello::Repository::Sync, id: e93368df-842c-4bc0-b7f4-2930fceed997, execution_plan_id: 46daac40-b5e9-4880-afaf-556fc07305bf} state changed: stopped result: warning
2021-01-21T16:19:15 [I|app|ea38cd4c] Started GET "/notification_recipients" for 127.0.0.1 at 2021-01-21 16:19:15 +0100
2021-01-21T16:19:15 [I|app|ea38cd4c] Processing by NotificationRecipientsController#index as JSON
2021-01-21T16:19:15 [I|app|ea38cd4c] Completed 200 OK in 8ms (Views: 0.1ms | ActiveRecord: 1.5ms | Allocations: 2243)
2021-01-21T16:19:15 [I|app|62a472ee] Started GET "/foreman_tasks/api/tasks/e93368df-842c-4bc0-b7f4-2930fceed997/details?include_permissions" for 127.0.0.1 at 2021-01-21 16:19:15 +0100
2021-01-21T16:19:15 [I|app|62a472ee] Processing by ForemanTasks::Api::TasksController#details as JSON
2021-01-21T16:19:15 [I|app|62a472ee] Parameters: {"include_permissions"=>nil, "id"=>"e93368df-842c-4bc0-b7f4-2930fceed997"}
2021-01-21T16:19:15 [I|app|62a472ee] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/views/foreman_tasks/api/tasks/details.json.rabl
2021-01-21T16:19:15 [I|app|62a472ee] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/views/foreman_tasks/api/tasks/details.json.rabl (Duration: 45.9ms | Allocations: 42666)
2021-01-21T16:19:15 [I|app|62a472ee] Completed 200 OK in 56ms (Views: 42.4ms | ActiveRecord: 6.4ms | Allocations: 45471)
