Unable to sync some repositories (AlmaLinux in this case)

Support
11

Any news on the missing pulp-rpm 3.14 for el8?

The longer people running with the broken version the more broken packages they will have in their system. And so far, the repair script hasnā€™t been advertised much, so chance is people will be running with broken packages for a longer time, even after the updateā€¦

12

Is there a repair script? Can you share it, please?

13

Itā€™s in the pulp issue regarding the bug in pulp-rpm 3.13: Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp

14

Small verification, are these errors also related to the issues with pulp-rpm 3.13.3

2021-08-07T15:41:03 [E|bac|c7fc8ebd] Pulp task error (Katello::Errors::Pulp3Error)
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 c7fc8ebd | /usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 c7fc8ebd | /usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 c7fc8ebd | /usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 c7fc8ebd | [ sidekiq ]
 c7fc8ebd | [ concurrent-ruby ]

This is Katello 4.1.2 on CentOS8, which I installed to test if it had anything to do with Rocky8 (I was having these errors there as well)

NOTE: This error happened when syncing Rocky8 BaseOS. Iā€™m running a Complete Sync now (as suggested earlier). Havenā€™t looked into the repair script yet

EDIT2: Nope, still broke even with Complete Sync

15

I think thatā€™s the trace back for a failed task. You have to check the task or subtask to see the pulp error which caused the task to failā€¦

16

I had a look on (another fresh install, man I love Ansible :wink: )

Environment details (all freshly installed):

Hardware: KVM with 3 cores and 12GB RAM (lab VM to test Ansible role for the configuration of Foreman) and 100GB NVMe storage
OS: Rocky Linux release 8.4 (Green Obsidian)
Foreman version: 2.5.2
Katello version: 4.1.2.1

I know, itā€™s a bit scarce w.r.t. minimum requirements, but I have had working installation with earlier versions Foreman/Katello :slight_smile:

Package list:

Installed Packages

    candlepin-4.0.6-1.el8.noarch
    candlepin-selinux-4.0.6-1.el8.noarch
    foreman-2.5.2-1.el8.noarch
    foreman-cli-2.5.2-1.el8.noarch
    foreman-debug-2.5.2-1.el8.noarch
    foreman-dynflow-sidekiq-2.5.2-1.el8.noarch
    foreman-installer-2.5.2-3.el8.noarch
    foreman-installer-katello-2.5.2-3.el8.noarch
    foreman-postgresql-2.5.2-1.el8.noarch
    foreman-proxy-2.5.2-1.el8.noarch
    foreman-release-2.5.2-1.el8.noarch
    foreman-selinux-2.5.2-1.el8.noarch
    foreman-service-2.5.2-1.el8.noarch
    foreman.lbhr.htm.lan-apache-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-client-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-proxy-1.0-1.noarch
    foreman.lbhr.htm.lan-foreman-proxy-client-1.0-1.noarch
    foreman.lbhr.htm.lan-puppet-client-1.0-1.noarch
    katello-4.1.2.1-1.el8.noarch
    katello-certs-tools-2.7.3-1.el8.noarch
    katello-client-bootstrap-1.7.6-1.el8.noarch
    katello-common-4.1.2.1-1.el8.noarch
    katello-debug-4.1.2.1-1.el8.noarch
    katello-default-ca-1.0-1.noarch
    katello-repos-4.1.2.1-1.el8.noarch
    katello-selinux-4.0.0-1.el8.noarch
    katello-server-ca-1.0-1.noarch
    pulp-client-1.0-1.noarch
    pulpcore-selinux-1.2.4-1.el8.x86_64
    python3-pulp-ansible-0.8.0-1.el8.noarch
    python3-pulp-certguard-1.4.0-1.el8.noarch
    python3-pulp-container-2.7.0-1.el8.noarch
    python3-pulp-deb-2.13.0-1.el8.noarch
    python3-pulp-file-1.8.1-1.el8.noarch
    python3-pulp-rpm-3.13.3-1.el8.noarch
    python3-pulpcore-3.14.1-1.el8.noarch
    qpid-proton-c-0.32.0-3.el8.x86_64
    rubygem-foreman-tasks-4.1.2-1.fm2_5.el8.noarch
    rubygem-foreman-tasks-core-0.3.6-1.fm2_5.el8.noarch
    rubygem-foreman_maintain-0.8.6-1.el8.noarch
    rubygem-foreman_remote_execution-4.5.1-1.fm2_5.el8.noarch
    rubygem-foreman_remote_execution_core-1.4.4-1.el8.noarch
    rubygem-hammer_cli-2.5.1-1.el8.noarch
    rubygem-hammer_cli_foreman-2.5.1-1.el8.noarch
    rubygem-hammer_cli_foreman_remote_execution-0.2.1-2.fm2_5.el8.noarch
    rubygem-hammer_cli_foreman_tasks-0.0.15-2.fm2_5.el8.noarch
    rubygem-hammer_cli_katello-1.1.2-1.el8.noarch
    rubygem-katello-4.1.2.1-1.el8.noarch
    rubygem-pulp_ansible_client-0.8.0-1.el8.noarch
    rubygem-pulp_certguard_client-1.4.0-1.el8.noarch
    rubygem-pulp_container_client-2.7.0-1.el8.noarch
    rubygem-pulp_deb_client-2.13.0-1.el8.noarch
    rubygem-pulp_file_client-1.8.1-1.el8.noarch
    rubygem-pulp_rpm_client-3.13.3-1.el8.noarch
    rubygem-pulpcore_client-3.14.1-1.el8.noarch
    rubygem-qpid_proton-0.32.0-3.el8.x86_64
    rubygem-smart_proxy_pulp-3.0.0-1.fm2_5.el8.noarch

The task in question: sync LiteServer Mirror and LiteServer Mirror

It fails again, same stacktrace as my previous post. But as you mentioned, I dug into Dynflow on the task to get some more detail:

Name: 6: Actions::Pulp3::Repository::Sync (skipped) [ 670.55s / 15.46s ]
Input:

---
repo_id: 2
smart_proxy_id: 1
options:
  source_url: 
remote_user: admin
remote_cp_user: admin
current_request_id: 60cff264-6222-4408-a6b3-2b721aeee8c3
current_timezone: UTC
current_organization_id: 
current_location_id: 
current_user_id: 4

Output:

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/e74bac14-a1c8-4f51-bda7-e244d48e533f/"
  pulp_created: '2021-08-07T21:34:43.632+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: 60cff264-6222-4408-a6b3-2b721aeee8c3
  started_at: '2021-08-07T21:34:43.842+00:00'
  finished_at: '2021-08-07T21:45:51.835+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 266, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 422, in synchronize
          version = dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 181, in run
          pb.done += task.result()  # download_count
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 207, in _handle_content_unit
          await asyncio.gather(*downloaders_for_content)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 89, in download
          download_result = await downloader.run(extra_data=self.extra_data)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run
          return await download_wrapper()
        File "/usr/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
          ret = await target(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
          return await self._run(extra_data=extra_data)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 93, in _run
          to_return = await self._handle_response(response)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 210, in _handle_response
          chunk = await response.content.read(1048576)  # 1 megabyte
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 380, in read
          await self._wait("read")
        File "/usr/lib64/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
          await waiter
        File "/usr/lib64/python3.6/site-packages/aiohttp/helpers.py", line 656, in __exit__
          raise asyncio.TimeoutError from None
    description: ''
  worker: "/pulp/api/v3/workers/16e9b699-1817-41c0-a2e7-320d02539db5/"
  child_tasks: []
  progress_reports:
  - message: Parsed Modulemd-defaults
    code: sync.parsing.modulemd_defaults
    state: completed
    total: 45
    done: 45
  - message: Downloading Metadata Files
    code: sync.downloading.metadata
    state: completed
    done: 17
  - message: Parsed Modulemd
    code: sync.parsing.modulemds
    state: completed
    total: 103
    done: 103
  - message: Downloading Artifacts
    code: sync.downloading.artifacts
    state: failed
    done: 1085
  - message: Parsed Packages
    code: sync.parsing.packages
    state: canceled
    total: 6350
    done: 3959
  - message: Associating Content
    code: associating.content
    state: canceled
    done: 1000
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/b0ba2bc5-0bb8-4e2f-af4f-4f819af666b9/"
  - "/pulp/api/v3/remotes/rpm/rpm/2d3e5144-033e-4fbe-a02f-d4c58622beea/"
create_version: true
task_groups: []
poll_attempts:
  total: 61
  failed: 1

Error:

Katello::Errors::Pulp3Error
Pulp task error 
---
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in
  `block in check_for_errors'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in
  `each'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in
  `check_for_errors'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in
  `poll_external_task'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in
  `wrap'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels)
  in process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block
  in invoke'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'"
- "/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block
  in create_with_logging_context'"
17

It seems that you hit the same error I got with AlmaLinux.
I really hope that the pulp-rpm package update will hit the repo soonā€¦

18

Just poking in to say that packaging folks are taking a look into why pulp-rpm didnā€™t make it into the EL8 repo.

2 Likes
19

Man, I hope theyā€™ll manage to fix it soon, it became so frustrating, lots of repos are out of sync already.

20

im seeing these ā€œraise asyncio.timeouterror from noneā€ for both rpm and deb content.
i have some rpm repos which have completed successfully, however ubuntu 18 & 20 always fail as above, as do some rpm repos-namely centos7 updates & epel7
my setup
CentOS Linux release 8.4.2105
foreman.noarch 2.5.2-1.el8
katello.noarch 4.1.2.1-1.el8

python3-pulp-deb.noarch 2.13.0-1.el8
python3-pulp-file.noarch 1.8.1-1.el8
python3-pulp-rpm.noarch 3.13.3-1.el8
python3-pulpcore.noarch 3.14.1-1.el8

21

Hereā€™s the pipeline to watch: pulpcore-3.14-rpm-pipeline [Jenkins]

3 Likes
22

ā€œpassedā€: itā€™s there now: https://yum.theforeman.org/pulpcore/3.14/el8/x86_64/python3-pulp-rpm-3.14.0-1.el8.noarch.rpm

To whom it may concern: after the update you should probably restart foreman to make pulp use the new version and also donā€™t forget to run the repair script from Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp to fix all that broken metadata in the databaseā€¦

3 Likes
23

Just installed the new pulp modules on el8, sync, content view creation and applicability issue is fixed for me.

1 Like
24

Thank you (and to the team)! Huge relief to have this fixed.

1 Like
25

Finally, itā€™s working, all repos are in sync.

26

I think I am experiencing a different issue then. I still experience the issues mentioned in Unable to sync some repositories (AlmaLinux in this case) - #16 by Thulium-Drake

What can be causes of this?

27

Try to delete content, re-sync repo and republish metadata.

28

Well, I have reinstalled the whole Foreman server from scratch (Iā€™m working in a Lab environment and Iā€™m configuring Foreman using Ansible).

So Iā€™m pretty sure that there was no prior repo content :sweat_smile:

29

You can also try to restrict the number of concurrent download threads, maybe the repo provider have some restrictions.

  • hammer repository update --id=repo_id_number --download-concurrency=1
1 Like
30

Good tip, Iā€™ll give it a shot! Iā€™ll post when I have updates :slight_smile: