Katello 3.16.1-1 repo sync file validation error

Support
,
1

Problem:
Unable to sync repositories using immediate download policy.
Foreman/Katello is sat behind a proxy server which may be complicating the issue. Using OnDemand allows the repository to sync, but immediate fails, usually at the same point but I can’t figure out what rpm is causing the problem. I’ve tried the following:
Remove repository, recreate and re-sync
Discover repository from web UI, sync
Use alternative mirror
Even tried syncing the repository on serving locally instead
Trying multiple repositories such as extras/baseos etc
Similar issues with Ubuntu

Small repositories seemed to work ok such as centosplus but that seems unreliable i.e. not always.

Expected outcome:
Repository sync successful

Foreman and Proxy versions:
foreman.noarch 2.1.3-1.el7 @foreman-local
foreman-cli.noarch 2.1.3-1.el7 @foreman-local
foreman-debug.noarch 2.1.3-1.el7 @foreman-local
foreman-dynflow-sidekiq.noarch 2.1.3-1.el7 @foreman-local
foreman-installer.noarch 1:2.1.3-1.el7 @foreman-local
foreman-installer-katello.noarch 1:2.1.3-1.el7 @foreman-local
foreman-postgresql.noarch 2.1.3-1.el7 @foreman-local
foreman-proxy.noarch 2.1.3-1.el7 @foreman-local
foreman-selinux.noarch 2.1.3-1.el7 @foreman-local
foreman-service.noarch 2.1.3-1.el7 @foreman-local

Foreman and Proxy plugin versions:
katello.noarch 3.16.1-1.el7 @katello-local
katello-certs-tools.noarch 2.7.0-1.el7 @foreman-local
katello-client-bootstrap.noarch 1.7.5-1.el7 @katello-local
katello-common.noarch 3.16.1-1.el7 @katello-local
katello-debug.noarch 3.16.1-1.el7 @katello-local
katello-default-ca.noarch 1.0-1 installed
katello-selinux.noarch 3.4.0-1.el7 @foreman-local
katello-server-ca.noarch 1.0-1 installed

Distribution and version:
CentOS 7.8.2003

Other relevant data:
Error message:
Katello::Errors::Pulp3Error: A file failed validation due to checksum.

Error Output

{“pulp_tasks”=>
[{“pulp_href”=>"/pulp/api/v3/tasks/a41f8cb7-ba46-4717-9a91-49e42161e255/",
“pulp_created”=>“2020-10-21T23:18:15.381+00:00”,
“state”=>“failed”,
“name”=>“pulp_rpm.app.tasks.synchronizing.synchronize”,
“started_at”=>“2020-10-21T23:19:14.113+00:00”,
“finished_at”=>“2020-10-21T23:20:10.965+00:00”,
“error”=>
{“traceback”=>
" File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 883, in perform_job\n" +
" rv = job.perform()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 645, in perform\n" +
" self._result = self._execute()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 651, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 264, in synchronize\n" +
" dv.create()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete\n" +
" return future.result()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call\n" +
" await self.run()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 133, in run\n" +
" pb.done += task.result() # download_count\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 159, in _handle_content_unit\n" +
" await asyncio.gather(*downloaders_for_content)\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py”, line 88, in download\n" +
" download_result = await downloader.run(extra_data=self.extra_data)\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 227, in run\n" +
" return await self._run(extra_data=extra_data)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py”, line 46, in _run\n" +
" to_return = await self._handle_response(response)\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/download/http.py”, line 170, in _handle_response\n" +
" await self.finalize()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 146, in finalize\n" +
" self.validate_digests()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/download/base.py”, line 196, in validate_digests\n" +
" raise DigestValidationError()\n",
“description”=>“A file failed validation due to checksum.”},
“worker”=>"/pulp/api/v3/workers/239583c3-41a9-4f09-aa36-8bd45720c530/",
“child_tasks”=>,
“progress_reports”=>
[{“message”=>“Downloading Metadata Files”,
“code”=>“downloading.metadata”,
“state”=>“canceled”,
“done”=>4},
{“message”=>“Downloading Artifacts”,
“code”=>“downloading.artifacts”,
“state”=>“failed”,
“done”=>71},
{“message”=>“Associating Content”,
“code”=>“associating.content”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Un-Associating Content”,
“code”=>“unassociating.content”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Parsed Comps”,
“code”=>“parsing.comps”,
“state”=>“completed”,
“total”=>109,
“done”=>109},
{“message”=>“Parsed Packages”,
“code”=>“parsing.packages”,
“state”=>“canceled”,
“total”=>10070,
“done”=>6007}],
“created_resources”=>,
“reserved_resources_record”=>
["/pulp/api/v3/remotes/rpm/rpm/a340b70d-0f4b-4acf-a4b0-2bf0062c4f97/",
“/pulp/api/v3/repositories/rpm/rpm/f499b1c1-d64f-4ff8-b9f0-50937de974db/”]}],
“create_version”=>true,
“task_groups”=>,
“poll_attempts”=>{“total”=>28, “failed”=>1}}

Backtrace

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:101:in block in check_for_errors' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:99:in check_for_errors' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:132:in poll_external_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in <module:Sidekiq>' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

2

Hi @andyfry,

I’m guessing this is a Pulp 3 bug we haven’t seen yet. Can you confirm for me that your pulp-rpm version matches python3-pulp-rpm-3.5.1-1.el7.noarch.rpm and that this is a brand new Katello install (not upgrade)?

3

Hi there,

Yep, it was a fresh install.

# yum list installed python3-pulp-rpm
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Installed Packages
python3-pulp-rpm.noarch 3.5.1-1.el7 @katello-pulpcore-local

In fairness though it’s a “disconnected” install, so I pulled all the rpm’s from the repository and build my own katello and foreman repos locally. Not sure if that has a bearing on this issue. I also ran into this issue: Issue 7321 and manually applied the patch sync through proxy fix 1837

Hope that’s useful.

Andy

4

Just ran into another issue:
Katello::Errors::Pulp3Error: Cannot create repository version. Path is duplicated: rubygem-rspec-doc-3.5.0-1.el7.noarch.rpm.

Which seems to relate to Issue 6303. Is it possible that is a fix which has somehow been reverted in 3.5.1-1?

5

Hi @andyfry,

Thanks for the debugging info. I will show this info to the Pulp team on Monday. I’ll get back to you if we need any more information and let you know if we open any Pulp bugs for it.

In the meantime, If you try creating a new repo and syncing it, do you consistently get one error over the other?

6

@andyfry,

Was the proxy still involved when you tried syncing the repository locally?

7

Hi @iballou,

Not entirely sure I understand you what you’re after?

If it does help it seems the smaller the repository is the more likely it is to sync.

Once a repository has synchronised I get the duplicates error.

And no, the proxy wasn’t involved in the local sync, I don’t doubt that the proxy may be part of the issue.

I did manage to download the repo to disc and then import the packages individually to pulp, but then when I pointed the pulp repo through the proxy at the repo I got the file validation error again.

Andy

8

Since you got the error when you served the repository locally, and the proxy wasn’t involved, I’m thinking something else might be at play. Which repository did you clone locally and try syncing from? It would be helpful if we can reproduce the same issue.

9

Hi @iballou,

It was multiple repositories, but I guess the primary one was Centos 7 base os form the aarnet mirror(or Optus for that matter).

http://mirror.aarnet.edu.au/pub/centos/7/os/x86_64/

But now I’m getting this from an Ubuntu sync:

Katello::Errors::PulpError: PLP0000: [Errno 2] No such file or directory: u’/var/cache/pulp/reserved_resource_worker-0@ednpas3181/3f8cd76f-0110-4d5f-995a-0bc656751e27/packages/21/40/language-pack-gnome-da_20.04+20200416_all.deb’

Maybe I should just go for a fresh install?

10

Hi @iballou,

I’m forcing immediate because I’m thinking I need it in order to export a content view to another foreman/katello server on an air gapped network. Is that the case, or can I use on Demand?

Andy

11

Hi @andyfry,

If you’re looking to use content view import/export on Katello 3.16, I’m going to recommend switching to using Pulp 2. The import/export feature is not fully finished for Katello 3.16 with Pulp 3. I haven’t seen documentation about that, so I’ll see about adding it. Apologies about any confusion there.

To switch to Pulp 2, you must:

  1. Either do a full Katello reset or delete all of your yum, file, and docker repositories.
  2. Run the foreman-installer with
foreman-installer --katello-use-pulp-2-for-docker 1 --katello-use-pulp-2-for-file 1  --katello-use-pulp-2-for-yum 1

You’re right about using the Immediate download policy for repositories that you want to export.

12

Hi @iballou,

I found I had to delete the existing repositories before converting to pulp-2 but that was ok. Sadly I’m still getting these file validation errors so I’m thinking the proxy is “interfering” in some way.

Is it worth taking a step back to katello 3.15?

13

Hi @andyfry,

You’re still getting:
Katello::Errors::Pulp3Error: A file failed validation due to checksum. ?

If it is still the Pulp 3 error then you’re still on Pulp 3.

14

Hi @iballou,

No it’s not exactly the same error message and certainly no refernce to Pulp 3.

RuntimeError: An error occurred during the sync
{“expected_size”=>85788, “error_code”=>“size_mismatch”, “name”=>“libfdisk-devel”, “unit_key”=>{“epoch”=>“0”, “version”=>“2.32.1”, “name”=>“libfdisk-devel”, “release”=>“22.el8”, “checksum”=>“18ea773537aa6b6776a33e7cac766c1cc776306ead17b4806c5f810ef3f08205”, “arch”=>“i686”, “checksumtype”=>“sha256”}, “actual_size”=>84010}

As an example

15

Ah, ok. I’m going to do some digging on that error, maybe something has popped up relating to proxies. I don’t think it would be worth going back to Katello 3.15, unless you’ve tried it before with the same proxy configuration.

16

Cool, will wait to see what you turn up. In the meantime I’ve kinda got around it… I think.

I created local repos so I could avoid the proxy all together for the initial sync, then switched to the aarnet mirror to sync in future. So far so good, but I’m not convinced that the proxy isn’t going to cause me an issue down the track. Interesting though that small repos and indiviual new packages in a repo don’t seem to be as much of an issue.

17

Interesting that larger volumes of RPM content seems to cause the issue, I’ll keep that in mind.

18

After talking with the Pulp team, it sounds likely that the issue has more to do with your proxy configuration than with a Pulp 2/3 issue. Issues arise now and then with proxies and Pulp since there are so many different proxies out there.

I might not be super helpful at helping with the proxy configuration, but perhaps if you give us some of your proxy information something worthy of notice might come up.

Also, it’s a long shot, but maybe something in here is helpful? Foreman :: Plugin Manuals

19

There is also another user who is posting on the pulp mailing list about proxy issues with syncing. Maybe we’ll be able to learn something from them https://www.redhat.com/archives/pulp-list/2020-November/msg00003.html

20

Hi @iballou,

I did some testing of local packages and tried to down load them through the proxy using wget. Pretty odd results but it seems the content in the proxy was corrupt. I had to use a --nocache to get the valid file to download. Once I’d done that the repositories were able to sync.

I need to find the proxy owners and get them to fix it up!

Thanks for you help.

Andy

1 Like