Foreman 3.2 / Katello 4.4 unable to sync repos

Problem:
This is a new installation of Foreman/Katello, running the latest 3.2 and 4.x katello respectively. I tried synching the content from a repo and is not synching, failing with the following:

Mar 28 10:43:29 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:43:29 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:43:45 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:43:45 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:44:01 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:44:01 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:44:17 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:44:17 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:44:33 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:44:33 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:44:50 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:44:50 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:45:06 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:45:06 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:45:22 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:45:22 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:45:38 sat pulpcore-api: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]:  - - [28/Mar/2022:17:45:38 +0000] "GET /pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.16.0/ruby"
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-37' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3bdc160>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-39' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3bdc1c0>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-41' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3bdc220>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-43' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3bdc280>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-62' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3376be0>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-30' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3bd4fd0>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-32' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3bdc070>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: asyncio:ERROR: Task was destroyed but it is pending!
Mar 28 10:45:39 sat pulpcore-worker-4: task: <Task pending name='Task-35' coro=<Event.wait() running at /opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f00b3bdc100>()]>>
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: pulpcore.tasking.pulpcore_worker:INFO: Task c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc failed (Fatal parser error)
Mar 28 10:45:39 sat pulpcore-worker-4: pulp [581a0920-0780-4e9e-8c30-e87d026ed11f]: pulpcore.tasking.pulpcore_worker:INFO:   File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 377, in _perform_task
Mar 28 10:45:39 sat pulpcore-worker-4: result = func(*args, **kwargs)
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 544, in synchronize
Mar 28 10:45:39 sat pulpcore-worker-4: repo_version = dv.create() or repo.latest_version()
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Mar 28 10:45:39 sat pulpcore-worker-4: loop.run_until_complete(pipeline)
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
Mar 28 10:45:39 sat pulpcore-worker-4: return future.result()
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Mar 28 10:45:39 sat pulpcore-worker-4: await asyncio.gather(*futures)
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Mar 28 10:45:39 sat pulpcore-worker-4: await self.run()
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 824, in run
Mar 28 10:45:39 sat pulpcore-worker-4: await self.parse_repository_metadata(repomd, repomd_files, file_extension)
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 885, in parse_repository_metadata
Mar 28 10:45:39 sat pulpcore-worker-4: groups_list = await self.parse_packages_components(comps_result)
Mar 28 10:45:39 sat pulpcore-worker-4: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 1021, in parse_packages_components
Mar 28 10:45:39 sat pulpcore-worker-4: comps.fromxml_f(comps_result.path)

Expected outcome:
The expected outcome is for the reposync to complete without errors.

Foreman and Proxy versions:
foreman-installer-katello-3.2.0-1.el7.noarch
foreman-service-3.2.0-1.el7.noarch
foreman-release-3.2.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-3.2.0-1.20220214175114git323f240.el7.noarch
foreman-installer-3.2.0-1.el7.noarch
foreman-postgresql-3.2.0-1.el7.noarch
tfm-rubygem-foreman-tasks-6.0.0-1.fm3_2.el7.noarch
foreman-debug-3.2.0-1.el7.noarch
tfm-rubygem-foreman_remote_execution-6.0.0-1.fm3_2.el7.noarch
foreman-proxy-3.2.0-1.el7.noarch
foreman-dynflow-sidekiq-3.2.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el7.noarch
foreman-cli-3.2.0-1.el7.noarch
foreman-3.2.0-1.el7.noarch
rubygem-foreman_maintain-1.0.3-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el7.noarch
foreman-installer-katello-3.2.0-1.el7.noarch
tfm-rubygem-hammer_cli_katello-1.4.1-1.el7.noarch
tfm-rubygem-katello-4.4.0-1.el7.noarch
katello-client-bootstrap-1.7.8-1.el7.noarch
katello-debug-4.4.0-1.el7.noarch
katello-repos-4.4.0-1.el7.noarch
katello-certs-tools-2.8.2-1.el7.noarch
katello-4.4.0-1.el7.noarch
katello-common-4.4.0-1.el7.noarch
Foreman and Proxy plugin versions:

Distribution and version:
RHEL 7.9

Other relevant data:
Job Errors below:

Action:

Actions::Pulp3::Repository::Sync

Input:

{"repo_id"=>2,
 "smart_proxy_id"=>1,
 "options"=>{"optimize"=>false},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/",
    "pulp_created"=>"2022-03-28T17:38:58.102+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
    "started_at"=>"2022-03-28T17:38:58.138+00:00",
    "finished_at"=>"2022-03-28T17:45:39.907+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x33cfc bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x33d10 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x33d24 bytes=137 start="  File \"/o" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x33d38 bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0x33d4c bytes=142 start="  File \"/o" end="in create\n"> +
       #<Sequel::SQL::Blob:0x33d60 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x33d74 bytes=110 start="  File \"/o" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x33d88 bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x33d9c bytes=135 start="  File \"/o" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x33db0 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x33dc4 bytes=127 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x33dd8 bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x33dec bytes=129 start="  File \"/o" end="4, in run\n"> +
       #<Sequel::SQL::Blob:0x33e00 bytes=79 start="    await " end="xtension)\n"> +
       #<Sequel::SQL::Blob:0x33e14 bytes=151 start="  File \"/o" end="_metadata\n"> +
       #<Sequel::SQL::Blob:0x33e28 bytes=69 start="    groups" end="s_result)\n"> +
       #<Sequel::SQL::Blob:0x33e3c bytes=152 start="  File \"/o" end="omponents\n"> +
       #<Sequel::SQL::Blob:0x33e50 bytes=39 start="    comps." end="ult.path)\n">,
      "description"=>"Fatal parser error"},
    "worker"=>"/pulp/api/v3/workers/d83116b7-6d57-42b0-9f42-1a1b41d80c56/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>6},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>195},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>9000},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"completed",
       "done"=>9904}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/9829b79c-0c3d-41e6-98c2-f03d9de85651/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/adf90f61-ff52-467a-981e-32eb548763df/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>46, "failed"=>1}}

Exception:

Katello::Errors::Pulp3Error: Fatal parser error

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:94:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Action:

Actions::Pulp3::Repository::Sync

Input:

{"repo_id"=>2,
 "smart_proxy_id"=>1,
 "options"=>{"optimize"=>false},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/c90f4c3f-19da-414c-9dfd-ac2bfbca2cfc/",
    "pulp_created"=>"2022-03-28T17:38:58.102+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
    "started_at"=>"2022-03-28T17:38:58.138+00:00",
    "finished_at"=>"2022-03-28T17:45:39.907+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x33e64 bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x33e78 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x33e8c bytes=137 start="  File \"/o" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x33ea0 bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0x33eb4 bytes=142 start="  File \"/o" end="in create\n"> +
       #<Sequel::SQL::Blob:0x33ec8 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x33edc bytes=110 start="  File \"/o" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x33ef0 bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x33f04 bytes=135 start="  File \"/o" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x33f18 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x33f2c bytes=127 start="  File \"/o" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x33f40 bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x33f54 bytes=129 start="  File \"/o" end="4, in run\n"> +
       #<Sequel::SQL::Blob:0x33f68 bytes=79 start="    await " end="xtension)\n"> +
       #<Sequel::SQL::Blob:0x33f7c bytes=151 start="  File \"/o" end="_metadata\n"> +
       #<Sequel::SQL::Blob:0x33f90 bytes=69 start="    groups" end="s_result)\n"> +
       #<Sequel::SQL::Blob:0x33fa4 bytes=152 start="  File \"/o" end="omponents\n"> +
       #<Sequel::SQL::Blob:0x33fb8 bytes=39 start="    comps." end="ult.path)\n">,
      "description"=>"Fatal parser error"},
    "worker"=>"/pulp/api/v3/workers/d83116b7-6d57-42b0-9f42-1a1b41d80c56/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>6},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>195},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>9000},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"completed",
       "done"=>9904}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/9829b79c-0c3d-41e6-98c2-f03d9de85651/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/adf90f61-ff52-467a-981e-32eb548763df/"]}],
 "create_version"=>true,
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>46, "failed"=>1}}

Action:

Actions::Pulp3::Repository::SaveVersion

Input:

{"repository_id"=>2,
 "tasks"=>Step(6).output[:pulp_tasks],
 "repository_details"=>nil,
 "force_fetch_version"=>true,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Action:

Actions::Pulp3::Repository::CreatePublication

Input:

{"repository_id"=>2,
 "smart_proxy_id"=>1,
 "contents_changed"=>true,
 "skip_publication_creation"=>Step(9).output[:publication_provided],
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Action:

Actions::Pulp3::Repository::SavePublication

Input:

{"repository_id"=>2,
 "tasks"=>Step(12).output,
 "contents_changed"=>true,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Action:

Actions::Pulp3::Orchestration::Repository::Sync

Input:

{"subaction_output"=>Step(9).output,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Action:

Actions::Katello::Repository::ErrataMail

Input:

{"repo"=>2,
 "contents_changed"=>Step(19).output[:contents_changed],
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Action:

Actions::Katello::Repository::ErrataMail

Input:

{"repo"=>2,
 "contents_changed"=>Step(19).output[:contents_changed],
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Action:

Actions::Katello::Repository::Sync

Input:

{"repository"=>
  {"id"=>2,
   "name"=>"rhel-8-for-x86_64-baseos-rpms",
   "label"=>"rhel-8-for-x86_64-baseos-rpms"},
 "product"=>
  {"id"=>1, "name"=>"RHEL8", "label"=>"RHEL8", "cp_id"=>"323929453190"},
 "provider"=>{"id"=>1, "name"=>"Anonymous"},
 "organization"=>
  {"id"=>1, "name"=>"GS_Organization", "label"=>"GS_Organization"},
 "services_checked"=>["pulp3"],
 "id"=>2,
 "sync_result"=>Step(19).output,
 "skip_metadata_check"=>true,
 "validate_contents"=>false,
 "contents_changed"=>Step(19).output[:contents_changed],
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Action:

Actions::Katello::Repository::Sync

Input:

{"repository"=>
  {"id"=>2,
   "name"=>"rhel-8-for-x86_64-baseos-rpms",
   "label"=>"rhel-8-for-x86_64-baseos-rpms"},
 "product"=>
  {"id"=>1, "name"=>"RHEL8", "label"=>"RHEL8", "cp_id"=>"323929453190"},
 "provider"=>{"id"=>1, "name"=>"Anonymous"},
 "organization"=>
  {"id"=>1, "name"=>"GS_Organization", "label"=>"GS_Organization"},
 "services_checked"=>["pulp3"],
 "id"=>2,
 "sync_result"=>Step(19).output,
 "skip_metadata_check"=>true,
 "validate_contents"=>false,
 "contents_changed"=>Step(19).output[:contents_changed],
 "current_request_id"=>"581a0920-0780-4e9e-8c30-e87d026ed11f",
 "current_timezone"=>"America/Los_Angeles",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>4}

Output:

{}

Hi @cucu007,

Is this repository public? If so, having the URL so we could try syncing ourselves would be helpful.

Hi Iballou,
Unfortunately is a private repo.

Let me try synching a public repo like EPEL to see if the outcome is different.

Sounds good. Reproducing the issue on our side makes it much more likely that we can fix something quickly.

I was able to sync the content from EPEL without issues using a public mirror, I am not sure why my internal repo would not sync properly…any ways to find clues?

This is what I see on the details of the job when it runs:

No content added.
Total steps: 18910/18910
--------------------------------
Associating Content: 9000/9000
Downloading Artifacts: 0/0
Downloading Metadata Files: 6/6
Parsed Packages: 9904/9904

The job ends with the following error:

Errors:

Fatal parser error

@cucu007 It seems like the failure is happening when parsing the “comps” XML file. Would it be possible to share that metadata file? Is this a fully custom repo (from in-house RPMs) or a clone of a private third-party repo, or a combination of public and private RPMs?

I finally got it to sync, it looks like my repo path was wrong initially. Thanks for the help.