While syncing proxy an error shows: Katello::Errors::Pulp3Error: 404, message='Not Found'

Support
#1

Problem:
While syncing a proxy and error is shown:

Katello::Errors::Pulp3Error: 404, message='Not Found', url=URL('https://uscdc01plsat001.globalservices.bcdtravel.local/pulp/content/bcdAmericas/Production/cv-Centos7/custom/CentOS_Linux_Server/CentOS_7_x86_64_Updates/Packages/j/java-11-openjdk-jmods-11.0.9.11-0.el7_9.x86_64.rpm')

Also seeing this error in the task error log:

‘RuntimeError: Could not lookup a publication_href for repo 4441’

Expected outcome:
No error and proxy is synced

Foreman and Proxy versions:
F 3.5.3 K 4.76

Foreman and Proxy plugin versions:
F 3.5.3

Distribution and version:
Rocky 8

Other relevant data:
Output of task

{"smart_proxy_history_id"=>8812,
 "pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/0fc7b252-a084-47fe-bd2d-600ecce9de5f/",
    "pulp_created"=>"2023-08-14T15:48:51.985+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"c2632b7e-5eb8-4698-9282-cae2a5a76354",
    "started_at"=>"2023-08-14T15:49:27.685+00:00",
    "finished_at"=>"2023-08-14T16:38:13.165+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x4ae020 bytes=106 start="  File \"/u" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x4ae034 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x4ae048 bytes=104 start="  File \"/u" end="nchronize\n"> +
       #<Sequel::SQL::Blob:0x4ae05c bytes=56 start="    repo_v" end="version()\n"> +
       #<Sequel::SQL::Blob:0x4ae070 bytes=109 start="  File \"/u" end="in create\n"> +
       #<Sequel::SQL::Blob:0x4ae084 bytes=38 start="    loop.r" end="pipeline)\n"> +
       #<Sequel::SQL::Blob:0x4ae098 bytes=86 start="  File \"/u" end="_complete\n"> +
       #<Sequel::SQL::Blob:0x4ae0ac bytes=27 start="    return" end=".result()\n"> +
       #<Sequel::SQL::Blob:0x4ae0c0 bytes=102 start="  File \"/u" end="_pipeline\n"> +
       #<Sequel::SQL::Blob:0x4ae0d4 bytes=35 start="    await " end="*futures)\n"> +
       #<Sequel::SQL::Blob:0x4ae0e8 bytes=94 start="  File \"/u" end=" __call__\n"> +
       #<Sequel::SQL::Blob:0x4ae0fc bytes=21 start="    await " end="elf.run()\n"> +
       #<Sequel::SQL::Blob:0x4ae110 bytes=102 start="  File \"/u" end="3, in run\n"> +
       #<Sequel::SQL::Blob:0x4ae124 bytes=47 start="    pb.don" end="oad_count\n"> +
       #<Sequel::SQL::Blob:0x4ae138 bytes=119 start="  File \"/u" end="tent_unit\n"> +
       #<Sequel::SQL::Blob:0x4ae14c bytes=51 start="    await " end="_content)\n"> +
       #<Sequel::SQL::Blob:0x4ae160 bytes=98 start="  File \"/u" end=" download\n"> +
       #<Sequel::SQL::Blob:0x4ae174 bytes=12 content="    raise e\n"> +
       #<Sequel::SQL::Blob:0x4ae188 bytes=98 start="  File \"/u" end=" download\n"> +
       #<Sequel::SQL::Blob:0x4ae19c bytes=71 start="    downlo" end="tra_data)\n"> +
       #<Sequel::SQL::Blob:0x4ae1b0 bytes=86 start="  File \"/u" end="3, in run\n"> +
       #<Sequel::SQL::Blob:0x4ae1c4 bytes=36 start="    return" end="wrapper()\n"> +
       #<Sequel::SQL::Blob:0x4ae1d8 bytes=80 start="  File \"/u" end=" in retry\n"> +
       #<Sequel::SQL::Blob:0x4ae1ec bytes=40 start="    ret = " end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x4ae200 bytes=99 start="  File \"/u" end="d_wrapper\n"> +
       #<Sequel::SQL::Blob:0x4ae214 bytes=50 start="    return" end="tra_data)\n"> +
       #<Sequel::SQL::Blob:0x4ae228 bytes=89 start="  File \"/u" end=", in _run\n"> +
       #<Sequel::SQL::Blob:0x4ae23c bytes=36 start="    self.r" end="response)\n"> +
       #<Sequel::SQL::Blob:0x4ae250 bytes=101 start="  File \"/u" end="or_status\n"> +
       #<Sequel::SQL::Blob:0x4ae264 bytes=32 start="    respon" end="_status()\n"> +
       #<Sequel::SQL::Blob:0x4ae278 bytes=101 start="  File \"/u" end="or_status\n"> +
       #<Sequel::SQL::Blob:0x4ae28c bytes=31 start="    raise " end="nseError(\n">,
      "description"=>
       "404, message='Not Found', url=URL('https://uscdc01plsat001.globalservices.bcdtravel.local/pulp/content/bcdAmericas/Production/cv-Centos7/custom/CentOS_Linux_Server/CentOS_7_x86_64_Updates/Packages/j/java-11-openjdk-jmods-11.0.9.11-0.el7_9.x86_64.rpm')"},
    "worker"=>"/pulp/api/v3/workers/fb596cde-cc87-44ab-bd5a-27034dfdfb9c/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>5},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"failed",
       "done"=>1627},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"canceled",
       "done"=>1500},
      {"message"=>"Skipping Packages",
       "code"=>"sync.skipped.packages",
       "state"=>"completed",
       "total"=>0,
       "done"=>0},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "total"=>3693,
       "done"=>2505}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/de5a73ce-0ea5-413e-a648-54d0085d4b84/",
      "shared:/pulp/api/v3/remotes/rpm/rpm/d3ff6fe3-05ab-4960-acf5-654e25ab681c/"]}],
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>186, "failed"=>1}}

Exception:

Katello::Errors::Pulp3Error: 404, message='Not Found', url=URL('https://uscdc01plsat001.globalservices.bcdtravel.local/pulp/content/bcdAmericas/Production/cv-Centos7/custom/CentOS_Linux_Server/CentOS_7_x86_64_Updates/Packages/j/java-11-openjdk-jmods-11.0.9.11-0.el7_9.x86_64.rpm')

Backtrace:

/usr/share/gems/gems/katello-4.7.6/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:22:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/cancellable.rb:14:in `run'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:26:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.7.6/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:94:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Exception:

RuntimeError: Could not lookup a publication_href for repo 4441

Backtrace:

/usr/share/gems/gems/katello-4.7.6/app/services/katello/pulp3/repository_mirror.rb:197:in `refresh_distributions'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/pulp3/capsule_content/refresh_distribution.rb:17:in `invoke_external_task'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:19:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/cancellable.rb:14:in `run'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/execute_if_contents_changed.rb:5:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.7.6/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

When I go to the rpm link it does show 404, but when I go back a directory on the webpage it does show the package as being listed, but if I click on it, I get the 404 error

Capture

We had this issue last month and figured after we publish our monthly CV it would be resolved but we still have this issue. Thank your help in advance!

1 Like
#2

Hi @heyitsgary ,

We can try republishing the metadata for your broken repository.

First question, an important one, which mirroring policy is your CentOS_7_x86_64_Updates repo using? Additive, Content Only, or Mirror Complete?

If it’s using mirror complete, we cannot regenerate the metadata exactly – it’ll need to be generated by Pulp, which will mean any signatures that may be on the repo will no longer exist. If you’re not using signatures then it should be all fine.

To regenerate the metadata:

hammer content-view version republish-repositories --content-view "content view name" --organization "org name" --version <broken content view version number>

You may also need to add the --force flag, it differs a bit between versions.

Once the regeneration is done you should be good to re-run your smart proxy sync.

#4

Hi @iballou
Thank you for your response. For the Centos 7 Updates repo we are using Additive for the mirroring policy. However, the other Centos 7 repos such as sclo, sclo-rh, base, and extras use Content Only for the mirrioring policy. Will that be an issue and will regenerating the metadata add any packages from the Library to the CV?
I ask because we do not want to add anything from the library to the current CV as it is locked for the month and we are now patching production systems.

Thank you.

1 Like
#5

Hi @heyitsgary ,

Regenerating the metadata should be perfectly safe since you use additive and content only mirroring.

It won’t change the contents of your content view version repositories. The repository under the hood in Pulp will remain exactly the same. That action should only fix the content distribution bits.

2 Likes
#6

Excellent, I will give it a try and let you know. Thank you!

1 Like
#7

Hi @iballou,
This morning I ran the above hammer commands on both versions in the Centos CV and I am still getting the same error message as the original.

hammer content-view version republish-repositories --content-view "cv-Centos7" --organization-id 3 --version 105 --force yes
hammer content-view version republish-repositories --content-view "cv-Centos7" --organization-id 3 --version 104 --force yes

Both completed successfully without any errors. I then did a complete sync on the proxy and unfortunately the same message shows.

#8

Hi @heyitsgary ,

It could be that the file is actually corrupt in Pulp then. Have you tried running a “verify content checksum” sync on the repositories? It’s among the advanced sync options on the repository’s details page.

#9

I have not tried that as of yet but I did not find the package in our file system on the Foreman server. I was able to download the package from the external Centos7 repo and then I created out custom repo from it and synced, which it did look like it found the package. However the proxy sync is still failing, I think I probably have to republish it which is not possible no and will try again next week. I am now able to download the package from the web address that I was getting the 404 error from.

#10

So the proxy syncing is failing with a 404 on https://uscdc01plsat001.globalservices.bcdtravel.local/pulp/content/bcdAmericas/Production/cv-Centos7/custom/CentOS_Linux_Server/CentOS_7_x86_64_Updates/Packages/j/java-11-openjdk-jmods-11.0.9.11-0.el7_9.x86_64.rpm

But you’re able to download that file manually from your Foreman? Or is that same link giving a 404 with cURL as well?

If you can download that file from your Library repository but not from the content view one, then yeah it seems your content view repository was corrupt. I agree that republishing should be the easiest way to solve the problem.

#11

Yes, that is the failure message that is shown when I go to Infrastructure > Smart Proxies > Select the proxy that I want.

I’ve tried syncing again since I have manually added that Java package to Foreman and recreated/synced the repo. I am now able to curl that link directly on the proxy but syncing it still fails with the same error, which I believe now is because the CV doesn’t have the package. Previously that link would just give me a 404 error and I was not able to download the package, now I can. Yes, republishing it will be the next step here and hopefully that will solve the issue. Thank you again for your help, I will let you know once I am able to publish and try the sync again. :slight_smile:

1 Like
#12

Hi iballou. I was able to publish a new CV this afternoon, removed any previous version and was successful in the sync of our proxy. Thank you for your help!

1 Like