Failed validation due to checksum - repo sync Foreman 3.1.1 / Katello 4.3

Problem:
After we;ve migrated fo Foreman 3.1.1 and Katello 4.3 we have an issue with a single repository

A file located at the url https://repo.mongodb.org/yum/redhat/8Server/mongodb-org/4.4/x86_64/RPMS/mongosh-0.5.0-x86_64.rpm failed validation due to checksum.

I’ve deleted it many times and recreate and resync, no luck.

Expected outcome:

Clean sync

Foreman and Proxy versions:

  • Foreman 3.1.1
  • Katello 4.3

Distribution and version:

  • AlmaLInux 8.5

Other relevant data:

2022-02-01T16:54:43 [E|bac|9e3ffd31] A file located at the url https://repo.mongodb.org/yum/redhat/8Server/mongodb-org/4.4/x86_64/RPMS/mongosh-0.5.0-x86_64.rpm failed validation due to checksum. (Katello::Errors::Pulp3Error)
9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:108:in block in check_for_errors' 9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in each’
9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:106:in check_for_errors' 9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:160:in poll_external_task’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:22:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in run' 9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in block (3 levels) in execute_run' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in block in run’
9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' 9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' 9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
9e3ffd31 | /usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in run' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass’
9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' 9e3ffd31 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in wrap’
9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in with_progress_calculation' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in pass’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in pass' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user' 9e3ffd31 | /usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in run’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in call' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in execute’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in block (2 levels) in execute_run' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in catch’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in block in execute_run 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in execute’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in block (2 levels) in execute_run' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in catch’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in block in execute_run' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in block in with_error_handling’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in catch' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in with_error_handling’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in execute_run' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in execute’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:94:in execute’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in run_user_code’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' 9e3ffd31 | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
9e3ffd31 | [ sidekiq ]
9e3ffd31 | [ concurrent-ruby ]

Seems to work on my production install. Can you try downloading the rpm (https://repo.mongodb.org/yum/redhat/8Server/mongodb-org/4.4/x86_64/RPMS/mongosh-0.5.0-x86_64.rpm) and the primary.xml.gz (https://repo.mongodb.org/yum/redhat/8Server/mongodb-org/4.4/x86_64/repodata/8df408e8cbd1413e6039c04068037e2aefe631b9-primary.xml.gz)

can you then run the sha1sum mongosh-0.5.0-x86_64.rpm and
then compare the checksum to the value in primary xml ?

Yes, the check-sums are the same and now it’s working!

I haven’t made any changes, that’s very weird as I tried all day long to figure it out by deleting and recreating the repo.

Is it possible that the primary.xml was outdated?

Very, very strange.