Unable to sync some repositories (AlmaLinux in this case)

Support
13

It’s in the pulp issue regarding the bug in pulp-rpm 3.13: Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp

14

Small verification, are these errors also related to the issues with pulp-rpm 3.13.3

2021-08-07T15:41:03 [E|bac|c7fc8ebd] Pulp task error (Katello::Errors::Pulp3Error)
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 c7fc8ebd | [ sidekiq ]
 c7fc8ebd | [ concurrent-ruby ]

This is Katello 4.1.2 on CentOS8, which I installed to test if it had anything to do with Rocky8 (I was having these errors there as well)

NOTE: This error happened when syncing Rocky8 BaseOS. I’m running a Complete Sync now (as suggested earlier). Haven’t looked into the repair script yet

EDIT2: Nope, still broke even with Complete Sync

15

I think that’s the trace back for a failed task. You have to check the task or subtask to see the pulp error which caused the task to fail…

16

I had a look on (another fresh install, man I love Ansible :wink: )

Environment details (all freshly installed):

Hardware: KVM with 3 cores and 12GB RAM (lab VM to test Ansible role for the configuration of Foreman) and 100GB NVMe storage
OS: Rocky Linux release 8.4 (Green Obsidian)
Foreman version: 2.5.2
Katello version: 4.1.2.1

I know, it’s a bit scarce w.r.t. minimum requirements, but I have had working installation with earlier versions Foreman/Katello :slight_smile:

Package list:

Installed Packages

    candlepin-4.0.6-1.el8.noarch
    candlepin-selinux-4.0.6-1.el8.noarch
    foreman-2.5.2-1.el8.noarch
    foreman-cli-2.5.2-1.el8.noarch
    foreman-debug-2.5.2-1.el8.noarch
    foreman-dynflow-sidekiq-2.5.2-1.el8.noarch
    foreman-installer-2.5.2-3.el8.noarch
    foreman-installer-katello-2.5.2-3.el8.noarch
    foreman-postgresql-2.5.2-1.el8.noarch
    foreman-proxy-2.5.2-1.el8.noarch
    foreman-release-2.5.2-1.el8.noarch
    foreman-selinux-2.5.2-1.el8.noarch
    foreman-service-2.5.2-1.el8.noarch
    foreman.lbhr.htm.lan-apache-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-client-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-proxy-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-proxy-client-1.0-1.noarch
    foreman.lbhr.htm.lan-puppet-client-1.0-1.noarch
    katello-4.1.2.1-1.el8.noarch
    katello-certs-tools-2.7.3-1.el8.noarch
    katello-client-bootstrap-1.7.6-1.el8.noarch
    katello-common-4.1.2.1-1.el8.noarch
    katello-debug-4.1.2.1-1.el8.noarch
    katello-default-ca-1.0-1.noarch
    katello-repos-4.1.2.1-1.el8.noarch
    katello-selinux-4.0.0-1.el8.noarch
    katello-server-ca-1.0-1.noarch
    pulp-client-1.0-1.noarch
    pulpcore-selinux-1.2.4-1.el8.x86_64
    python3-pulp-ansible-0.8.0-1.el8.noarch
    python3-pulp-certguard-1.4.0-1.el8.noarch
    python3-pulp-container-2.7.0-1.el8.noarch
    python3-pulp-deb-2.13.0-1.el8.noarch
    python3-pulp-file-1.8.1-1.el8.noarch
    python3-pulp-rpm-3.13.3-1.el8.noarch
    python3-pulpcore-3.14.1-1.el8.noarch
    qpid-proton-c-0.32.0-3.el8.x86_64
    rubygem-foreman-tasks-4.1.2-1.fm2_5.el8.noarch
    rubygem-foreman-tasks-core-0.3.6-1.fm2_5.el8.noarch
    rubygem-foreman_maintain-0.8.6-1.el8.noarch
    rubygem-foreman_remote_execution-4.5.1-1.fm2_5.el8.noarch
    rubygem-foreman_remote_execution_core-1.4.4-1.el8.noarch
    rubygem-hammer_cli-2.5.1-1.el8.noarch
    rubygem-hammer_cli_foreman-2.5.1-1.el8.noarch
    rubygem-hammer_cli_foreman_remote_execution-0.2.1-2.fm2_5.el8.noarch
    rubygem-hammer_cli_foreman_tasks-0.0.15-2.fm2_5.el8.noarch
    rubygem-hammer_cli_katello-1.1.2-1.el8.noarch
    rubygem-katello-4.1.2.1-1.el8.noarch
    rubygem-pulp_ansible_client-0.8.0-1.el8.noarch
    rubygem-pulp_certguard_client-1.4.0-1.el8.noarch
    rubygem-pulp_container_client-2.7.0-1.el8.noarch
    rubygem-pulp_deb_client-2.13.0-1.el8.noarch
    rubygem-pulp_file_client-1.8.1-1.el8.noarch
    rubygem-pulp_rpm_client-3.13.3-1.el8.noarch
    rubygem-pulpcore_client-3.14.1-1.el8.noarch
    rubygem-qpid_proton-0.32.0-3.el8.x86_64
    rubygem-smart_proxy_pulp-3.0.0-1.fm2_5.el8.noarch

The task in question: sync LiteServer Mirror and LiteServer Mirror

It fails again, same stacktrace as my previous post. But as you mentioned, I dug into Dynflow on the task to get some more detail:

Name: 6: Actions::Pulp3::Repository::Sync (skipped) [ 670.55s / 15.46s ]
Input:

---
repo_id: 2
smart_proxy_id: 1
options:
  source_url: 
remote_user: admin
remote_cp_user: admin
current_request_id: 60cff264-6222-4408-a6b3-2b721aeee8c3
current_timezone: UTC
current_organization_id: 
current_location_id: 
current_user_id: 4

Output:

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/e74bac14-a1c8-4f51-bda7-e244d48e533f/"
  pulp_created: '2021-08-07T21:34:43.632+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: 60cff264-6222-4408-a6b3-2b721aeee8c3
  started_at: '2021-08-07T21:34:43.842+00:00'
  finished_at: '2021-08-07T21:45:51.835+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 266, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 422, in synchronize
          version = dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 181, in run
          pb.done += task.result()  # download_count
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 207, in _handle_content_unit
          await asyncio.gather(*downloaders_for_content)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 89, in download
          download_result = await downloader.run(extra_data=self.extra_data)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run
          return await download_wrapper()
        File "/usr/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
          ret = await target(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
          return await self._run(extra_data=extra_data)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 93, in _run
          to_return = await self._handle_response(response)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 210, in _handle_response
          chunk = await response.content.read(1048576)  # 1 megabyte
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 380, in read
          await self._wait("read")
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
          await waiter
        File "/usr/lib64/python3.6/site-packages/aiohttp/helpers.py", line 656, in __exit__
          raise asyncio.TimeoutError from None
    description: ''
  worker: "/pulp/api/v3/workers/16e9b699-1817-41c0-a2e7-320d02539db5/"
  child_tasks: []
  progress_reports:
  - message: Parsed Modulemd-defaults
    code: sync.parsing.modulemd_defaults
    state: completed
    total: 45
    done: 45
  - message: Downloading Metadata Files
    code: sync.downloading.metadata
    state: completed
    done: 17
  - message: Parsed Modulemd
    code: sync.parsing.modulemds
    state: completed
    total: 103
    done: 103
  - message: Downloading Artifacts
    code: sync.downloading.artifacts
    state: failed
    done: 1085
  - message: Parsed Packages
    code: sync.parsing.packages
    state: canceled
    total: 6350
    done: 3959
  - message: Associating Content
    code: associating.content
    state: canceled
    done: 1000
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/b0ba2bc5-0bb8-4e2f-af4f-4f819af666b9/"
  - "/pulp/api/v3/remotes/rpm/rpm/2d3e5144-033e-4fbe-a02f-d4c58622beea/"
create_version: true
task_groups: []
poll_attempts:
  total: 61
  failed: 1

Error:

Katello::Errors::Pulp3Error
Pulp task error 
---
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in
  `block in check_for_errors'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in
  `each'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in
  `check_for_errors'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in
  `poll_external_task'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in
  `wrap'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels)
  in process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block
  in invoke'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'"
- "/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block
  in create_with_logging_context'"
17

It seems that you hit the same error I got with AlmaLinux.
I really hope that the pulp-rpm package update will hit the repo soon…

18

Just poking in to say that packaging folks are taking a look into why pulp-rpm didn’t make it into the EL8 repo.

2 Likes
19

Man, I hope they’ll manage to fix it soon, it became so frustrating, lots of repos are out of sync already.

20

im seeing these “raise asyncio.timeouterror from none” for both rpm and deb content.
i have some rpm repos which have completed successfully, however ubuntu 18 & 20 always fail as above, as do some rpm repos-namely centos7 updates & epel7
my setup
CentOS Linux release 8.4.2105
foreman.noarch 2.5.2-1.el8
katello.noarch 4.1.2.1-1.el8

python3-pulp-deb.noarch 2.13.0-1.el8
python3-pulp-file.noarch 1.8.1-1.el8
python3-pulp-rpm.noarch 3.13.3-1.el8
python3-pulpcore.noarch 3.14.1-1.el8

21

Here’s the pipeline to watch: pulpcore-3.14-rpm-pipeline [Jenkins]

3 Likes
22

“passed”: it’s there now: https://yum.theforeman.org/pulpcore/3.14/el8/x86_64/python3-pulp-rpm-3.14.0-1.el8.noarch.rpm

To whom it may concern: after the update you should probably restart foreman to make pulp use the new version and also don’t forget to run the repair script from Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp to fix all that broken metadata in the database…

3 Likes
23

Just installed the new pulp modules on el8, sync, content view creation and applicability issue is fixed for me.

1 Like
24

Thank you (and to the team)! Huge relief to have this fixed.

1 Like
25

Finally, it’s working, all repos are in sync.

26

I think I am experiencing a different issue then. I still experience the issues mentioned in Unable to sync some repositories (AlmaLinux in this case) - #16 by Thulium-Drake

What can be causes of this?

27

Try to delete content, re-sync repo and republish metadata.

28

Well, I have reinstalled the whole Foreman server from scratch (I’m working in a Lab environment and I’m configuring Foreman using Ansible).

So I’m pretty sure that there was no prior repo content :sweat_smile:

29

You can also try to restrict the number of concurrent download threads, maybe the repo provider have some restrictions.

  • hammer repository update --id=repo_id_number --download-concurrency=1
1 Like
30

Good tip, I’ll give it a shot! I’ll post when I have updates :slight_smile:

31

Thank you! That seemed to do the trick! :slight_smile:

Maybe it’s also resource-related on my poor laptop :wink:

32

Glad it helped. Usually the restriction in this case is from repo provider, not from your resources.