Repo sync failing

Problem:
Starting recently, any attempt to sync Graylog’s repo results in the error “Katello::Errors::PulpError: PLP0000: mismatched tag: Line 20, column 2” and sync fails.
Expected outcome:
Repos syncs successfully
Foreman and Proxy versions:
Foreman 1.24.2
Katello 3.14.1
Foreman and Proxy plugin versions:

Distribution and version:
Oracle Linux 7.7
Other relevant data:

Preformatted text
Action:

Actions::Pulp::Repository::Sync

Input:

{“smart_proxy_id”=>1,
“repo_id”=>20332,
“source_url”=>nil,
“options”=>{},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/63c2add0-1e77-43c0-985a-61cc0ce2bf32/",
“task_id”=>“63c2add0-1e77-43c0-985a-61cc0ce2bf32”,
“tags”=>
[“pulp:repository:f579203a-e597-4eaa-8dbf-68a1946b5ea7”,
“pulp:action:sync”],
“finish_time”=>“2020-03-24T16:36:10Z”,
“_ns”=>“task_status”,
“start_time”=>“2020-03-24T16:36:09Z”,
“traceback”=>
“Traceback (most recent call last):\n” +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 686, in call\n" +
" return super(Task, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 108, in call\n" +
" return super(PulpTask, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call\n" +
" return self.run(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 793, in sync\n" +
" sync_report = sync_repo(transfer_repo, conduit, call_config)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 894, in wrap_f\n" +
" return f(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py”, line 84, in sync_repo\n" +
" report = self._current_sync.run()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 269, in run\n" +
" if not self.sync_feed:\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 139, in sync_feed\n" +
" if self.check_metadata(repo_url):\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 440, in check_metadata\n" +
" metadata_files.parse_repomd()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py”, line 185, in parse_repomd\n" +
" for event, element in xml_iterator:\n" +
" File “”, line 91, in next\n" +
“ParseError: mismatched tag: line 20, column 2\n”,
“spawned_tasks”=>,
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“details”=>
{“rpm_total”=>0, “rpm_done”=>0, “drpm_total”=>0, “drpm_done”=>0},
“size_total”=>0,
“size_left”=>0,
“items_left”=>0},
“comps”=>{“state”=>“NOT_STARTED”},
“purge_duplicates”=>{“state”=>“NOT_STARTED”},
“distribution”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“NOT_STARTED”},
“errata”=>{“state”=>“NOT_STARTED”},
“metadata”=>{“state”=>“NOT_STARTED”}}},
“queue”=>“reserved_resource_worker-0@s2-foreman.s2.internal.dq2”,
“state”=>“error”,
“worker_name”=>“reserved_resource_worker-0@s2-foreman.s2.internal”,
“result”=>nil,
“error”=>
{“code”=>“PLP0000”,
“data”=>{},
“description”=>“mismatched tag: line 20, column 2”,
“sub_errors”=>},
“_id”=>{"$oid"=>“5e7a36f99459ad0021d61401”},
“id”=>“5e7a36f99459ad0021d61401”}],
“contents_changed”=>true,
“poll_attempts”=>{“total”=>2, “failed”=>1}}

Exception:

Katello::Errors::PulpError: PLP0000: mismatched tag: line 20, column 2

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:122:in block in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:120:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:120:in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/repository/sync.rb:28:in external_task=’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/polling.rb:22:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/cancellable.rb:14:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/pulp/abstract_async_task.rb:45:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:563:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:16:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action/progress.rb:17:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:11:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:22:in with_locale' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/actions/middleware/keep_locale.rb:11:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:562:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:561:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:561:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:475:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:556:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/action.rb:285:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/director.rb:93:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:15:in block in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors.rb:12:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/executors/parallel/worker.rb:14:in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/context.rb:46:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/executes_context.rb:7:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:106:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/awaits.rb:15:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:47:in block in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:59:in with_backtrace' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/actor.rb:47:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:38:in process_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/buffer.rb:20:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/termination.rb:55:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/removes_child.rb:10:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:162:in process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:96:in block in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:119:in block (2 levels) in schedule_execution' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in block in synchronize’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in synchronize' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/synchronization/mutex_lockable_object.rb:41:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:116:in block in schedule_execution' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:18:in call’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:96:in work' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/serialized_execution.rb:77:in block in call_job’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in run_task' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in block (3 levels) in create_worker’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in loop' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in block (2 levels) in create_worker’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in block in create_worker’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

Action:

Actions::Pulp::Repository::Sync

Input:

{“smart_proxy_id”=>1,
“repo_id”=>20332,
“source_url”=>nil,
“options”=>{},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/63c2add0-1e77-43c0-985a-61cc0ce2bf32/",
“task_id”=>“63c2add0-1e77-43c0-985a-61cc0ce2bf32”,
“tags”=>
[“pulp:repository:f579203a-e597-4eaa-8dbf-68a1946b5ea7”,
“pulp:action:sync”],
“finish_time”=>“2020-03-24T16:36:10Z”,
“_ns”=>“task_status”,
“start_time”=>“2020-03-24T16:36:09Z”,
“traceback”=>
“Traceback (most recent call last):\n” +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 686, in call\n" +
" return super(Task, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 108, in call\n" +
" return super(PulpTask, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call\n" +
" return self.run(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 793, in sync\n" +
" sync_report = sync_repo(transfer_repo, conduit, call_config)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 894, in wrap_f\n" +
" return f(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py”, line 84, in sync_repo\n" +
" report = self._current_sync.run()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 269, in run\n" +
" if not self.sync_feed:\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 139, in sync_feed\n" +
" if self.check_metadata(repo_url):\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 440, in check_metadata\n" +
" metadata_files.parse_repomd()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py”, line 185, in parse_repomd\n" +
" for event, element in xml_iterator:\n" +
" File “”, line 91, in next\n" +
“ParseError: mismatched tag: line 20, column 2\n”,
“spawned_tasks”=>,
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“details”=>
{“rpm_total”=>0, “rpm_done”=>0, “drpm_total”=>0, “drpm_done”=>0},
“size_total”=>0,
“size_left”=>0,
“items_left”=>0},
“comps”=>{“state”=>“NOT_STARTED”},
“purge_duplicates”=>{“state”=>“NOT_STARTED”},
“distribution”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“NOT_STARTED”},
“errata”=>{“state”=>“NOT_STARTED”},
“metadata”=>{“state”=>“NOT_STARTED”}}},
“queue”=>“reserved_resource_worker-0@s2-foreman.s2.internal.dq2”,
“state”=>“error”,
“worker_name”=>“reserved_resource_worker-0@s2-foreman.s2.internal”,
“result”=>nil,
“error”=>
{“code”=>“PLP0000”,
“data”=>{},
“description”=>“mismatched tag: line 20, column 2”,
“sub_errors”=>},
“_id”=>{"$oid"=>“5e7a36f99459ad0021d61401”},
“id”=>“5e7a36f99459ad0021d61401”}],
“contents_changed”=>true,
“poll_attempts”=>{“total”=>2, “failed”=>1}}

Action:

Actions::Pulp::Orchestration::Repository::Sync

Input:

{“subaction_output”=>Step(4).output,
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{}

Action:

Actions::Katello::Repository::IndexContent

Input:

{“id”=>20332,
“contents_changed”=>Step(6).output[:contents_changed],
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{}

Action:

Actions::Katello::Repository::ErrataMail

Input:

{“repo”=>20332,
“last_updated”=>“2020-03-24 09:36:09 -0700”,
“contents_changed”=>Step(6).output[:contents_changed],
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{}

Action:

Actions::Katello::Repository::ErrataMail

Input:

{“repo”=>20332,
“last_updated”=>“2020-03-24 09:36:09 -0700”,
“contents_changed”=>Step(6).output[:contents_changed],
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{}

Action:

Actions::Pulp::Repository::RegenerateApplicability

Input:

{“repository_id”=>20332,
“contents_changed”=>Step(6).output[:contents_changed],
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{}

Action:

Actions::Katello::Repository::Sync

Input:

{“repository”=>
{“id”=>20332, “name”=>“EL7 Graylog 3.2”, “label”=>“EL7_Graylog_3_2”},
“product”=>
{“id”=>162,
“name”=>“EL7 Graylog 3.2”,
“label”=>“EL7_Graylog_3_2”,
“cp_id”=>“36438101356”},
“provider”=>{“id”=>3, “name”=>“Anonymous”},
“organization”=>{“id”=>3, “name”=>“SDSC-HCI”, “label”=>“sdsc-hci”},
“services_checked”=>[“pulp”, “pulp_auth”],
“id”=>20332,
“sync_result”=>Step(6).output,
“skip_metadata_check”=>false,
“validate_contents”=>false,
“contents_changed”=>Step(6).output[:contents_changed],
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{}

Action:

Actions::Katello::Repository::ImportApplicability

Input:

{“repo_id”=>20332,
“contents_changed”=>Step(6).output[:contents_changed],
“locale”=>“en”,
“current_request_id”=>nil,
“current_timezone”=>“America/Los_Angeles”,
“current_user_id”=>6,
“current_organization_id”=>3,
“current_location_id”=>6}

Output:

{}

Looks like the text formatting didn’t quite work :wink:

What’s the URL of the repo so we can try to reproduce the problem and help you out?

The initial URL, as entered in Foreman, is:
https://packages.graylog2.org/el/stable/3.2/x86_64/
This redirects to:
graylog2-package-repository.s3.eu-west-1.amazonaws.com

Both are whitelisted through our proxy.

We resolved this issue which was unrelated to Katello. The error messages were less than helpful, however.