Problem:
Some AlmaLinux repositories are not syncing
Expected outcome:
Synced repos
Foreman and Proxy versions:
- Foreman 2.5.2
- Katello 4.1.2
Foreman and Proxy plugin versions:
Installed Packages
- candlepin-4.0.4-1.el8.noarch
- candlepin-selinux-4.0.4-1.el8.noarch
- foreman-2.5.2-1.el8.noarch
- foreman-cli-2.5.2-1.el8.noarch
- foreman-debug-2.5.2-1.el8.noarch
- foreman-dynflow-sidekiq-2.5.2-1.el8.noarch
- foreman-installer-2.5.2-3.el8.noarch
- foreman-installer-katello-2.5.2-3.el8.noarch
- foreman-postgresql-2.5.2-1.el8.noarch
- foreman-proxy-2.5.2-1.el8.noarch
- foreman-release-2.5.2-1.el8.noarch
- foreman-service-2.5.2-1.el8.noarch
- katello-4.1.2-1.el8.noarch
- katello-certs-tools-2.7.3-1.el8.noarch
- katello-client-bootstrap-1.7.6-1.el8.noarch
- katello-common-4.1.2-1.el8.noarch
- katello-debug-4.1.2-1.el8.noarch
- katello-default-ca-1.0-1.noarch
- katello-repos-4.1.2-1.el8.noarch
- katello-server-ca-1.0-1.noarch
- katello.comtec.lan-apache-1.0-1.noarch
- katello.comtec.lan-foreman-client-1.0-1.noarch
- katello.comtec.lan-foreman-proxy-1.0-1.noarch
- katello.comtec.lan-foreman-proxy-client-1.0-1.noarch
- katello.comtec.lan-puppet-client-1.0-1.noarch
- pulp-client-1.0-1.noarch
- python3-pulp-ansible-0.8.0-1.el8.noarch
- python3-pulp-certguard-1.4.0-1.el8.noarch
- python3-pulp-container-2.7.0-1.el8.noarch
- python3-pulp-deb-2.13.0-1.el8.noarch
- python3-pulp-file-1.8.1-1.el8.noarch
- python3-pulp-rpm-3.13.3-1.el8.noarch
- python3-pulpcore-3.14.1-1.el8.noarch
- qpid-proton-c-0.32.0-3.el8.x86_64
- rubygem-foreman-tasks-4.1.2-1.fm2_5.el8.noarch
- rubygem-foreman-tasks-core-0.3.5-1.fm2_5.el8.noarch
- rubygem-foreman_maintain-0.8.6-1.el8.noarch
- rubygem-foreman_remote_execution-4.5.1-1.fm2_5.el8.noarch
- rubygem-foreman_remote_execution_core-1.4.4-1.el8.noarch
- rubygem-hammer_cli-2.5.1-1.el8.noarch
- rubygem-hammer_cli_foreman-2.5.1-1.el8.noarch
- rubygem-hammer_cli_foreman_remote_execution-0.2.1-2.fm2_5.el8.noarch
- rubygem-hammer_cli_foreman_tasks-0.0.15-2.fm2_5.el8.noarch
- rubygem-hammer_cli_katello-1.1.2-1.el8.noarch
- rubygem-katello-4.1.2-1.el8.noarch
- rubygem-pulp_ansible_client-0.8.0-1.el8.noarch
- rubygem-pulp_certguard_client-1.4.0-1.el8.noarch
- rubygem-pulp_container_client-2.7.0-1.el8.noarch
- rubygem-pulp_deb_client-2.13.0-1.el8.noarch
- rubygem-pulp_file_client-1.8.1-1.el8.noarch
- rubygem-pulp_rpm_client-3.13.3-1.el8.noarch
- rubygem-pulpcore_client-3.14.1-1.el8.noarch
- rubygem-qpid_proton-0.32.0-3.el8.x86_64
- rubygem-smart_proxy_pulp-3.0.0-1.fm2_5.el8.noarch
Distribution and version:
- CentOS 8.4
Other relevant data:
Actions::Pulp3::Repository::Sync
Input:
{"repo_id"=>56,
"smart_proxy_id"=>1,
"options"=>{"source_url"=>nil},
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"current_request_id"=>"13c39407-b1c7-46fa-a0f6-3ef7040c381e",
"current_timezone"=>"Europe/Bucharest",
"current_organization_id"=>1,
"current_location_id"=>3,
"current_user_id"=>1}
Output:
{"pulp_tasks"=>
[{"pulp_href"=>"/pulp/api/v3/tasks/26b29291-d86d-4c26-b1bc-fb3c4940c00b/",
"pulp_created"=>"2021-08-01T22:03:06.608+00:00",
"state"=>"failed",
"name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
"logging_cid"=>"13c39407-b1c7-46fa-a0f6-3ef7040c381e",
"started_at"=>"2021-08-01T22:05:42.855+00:00",
"finished_at"=>"2021-08-01T22:12:43.471+00:00",
"error"=>
{"traceback"=>
" File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 266, in _perform_task\n" +
" result = func(*args, **kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 422, in synchronize\n" +
" version = dv.create()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
" return future.result()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
" await self.run()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 181, in run\n" +
" pb.done += task.result() # download_count\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 207, in _handle_content_unit\n" +
" await asyncio.gather(*downloaders_for_content)\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 89, in download\n" +
" download_result = await downloader.run(extra_data=self.extra_data)\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 258, in run\n" +
" return await download_wrapper()\n" +
" File \"/usr/lib/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n" +
" ret = await target(*args, **kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 256, in download_wrapper\n" +
" return await self._run(extra_data=extra_data)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py\", line 93, in _run\n" +
" to_return = await self._handle_response(response)\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/download/http.py\", line 210, in _handle_response\n" +
" chunk = await response.content.read(1048576) # 1 megabyte\n" +
" File \"/usr/lib64/python3.6/site-packages/aiohttp/streams.py\", line 380, in read\n" +
" await self._wait(\"read\")\n" +
" File \"/usr/lib64/python3.6/site-packages/aiohttp/streams.py\", line 306, in _wait\n" +
" await waiter\n" +
" File \"/usr/lib64/python3.6/site-packages/aiohttp/helpers.py\", line 656, in __exit__\n" +
" raise asyncio.TimeoutError from None\n",
"description"=>""},
"worker"=>"/pulp/api/v3/workers/4d517a8d-12d9-4fc8-b67c-350c4d00c37f/",
"child_tasks"=>[],
"progress_reports"=>
[{"message"=>"Downloading Metadata Files",
"code"=>"sync.downloading.metadata",
"state"=>"completed",
"done"=>12},
{"message"=>"Downloading Artifacts",
"code"=>"sync.downloading.artifacts",
"state"=>"failed",
"done"=>627},
{"message"=>"Associating Content",
"code"=>"associating.content",
"state"=>"canceled",
"done"=>500},
{"message"=>"Parsed Modulemd",
"code"=>"sync.parsing.modulemds",
"state"=>"completed",
"total"=>6,
"done"=>6},
{"message"=>"Parsed Modulemd-defaults",
"code"=>"sync.parsing.modulemd_defaults",
"state"=>"completed",
"total"=>0,
"done"=>0},
{"message"=>"Parsed Packages",
"code"=>"sync.parsing.packages",
"state"=>"completed",
"total"=>2220,
"done"=>2220},
{"message"=>"Parsed Comps",
"code"=>"sync.parsing.comps",
"state"=>"completed",
"total"=>1,
"done"=>1},
{"message"=>"Parsed Advisories",
"code"=>"sync.parsing.advisories",
"state"=>"completed",
"total"=>279,
"done"=>279}],
"created_resources"=>[],
"reserved_resources_record"=>
["/pulp/api/v3/remotes/rpm/rpm/3e2e2182-c961-4c65-9617-21cc7c11048e/",
"/pulp/api/v3/repositories/rpm/rpm/55cbe81c-dafa-4243-8b8f-3ef01bbbcdc4/"]}],
"create_version"=>true,
"task_groups"=>[],
"poll_attempts"=>{"total"=>56, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: Pulp task error
Backtrace:
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.1.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.1.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
)