Problem:
I still see occasional sync errors when syncing content to my content proxy after promoting a content view. I think it only happens if multiple syncs are running at the same time although I cannot tell for sure.
I had similar issues in the past but it got better, i.e. it’s not happening so often. If the sync error occurs some new packages in some repositories are missing.
Previous topics Katello 4.1 content proxy sync errors and Some CVs on content proxies are missing files
There is a redmine/github issue, too.
Foreman and Proxy versions:
Foreman 3.1.2
Foreman and Proxy plugin versions:
Katello 4.3.1
Distribution and version:
CentOS 7.9
Other relevant data:
Error log in task view:
{"smart_proxy_history_id"=>22603,
"pulp_tasks"=>
[{"pulp_href"=>"/pulp/api/v3/tasks/844d42dc-da61-4442-b5d9-8c74a9a79064/",
"pulp_created"=>"2022-04-18T01:54:51.726+00:00",
"state"=>"failed",
"name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
"logging_cid"=>"53b4a24f-d66c-46a1-b974-d0a616de2d70",
"started_at"=>"2022-04-18T01:54:51.902+00:00",
"finished_at"=>"2022-04-18T02:02:42.830+00:00",
"error"=>
{"traceback"=>
#<Sequel::SQL::Blob:0x12a28d4 bytes=139 start=" File \"/o" end="form_task\n"> +
#<Sequel::SQL::Blob:0x12a28e8 bytes=35 start=" result" end="**kwargs)\n"> +
#<Sequel::SQL::Blob:0x12a28fc bytes=137 start=" File \"/o" end="nchronize\n"> +
#<Sequel::SQL::Blob:0x12a2910 bytes=56 start=" repo_v" end="version()\n"> +
#<Sequel::SQL::Blob:0x12a2924 bytes=142 start=" File \"/o" end="in create\n"> +
#<Sequel::SQL::Blob:0x12a2938 bytes=38 start=" loop.r" end="pipeline)\n"> +
#<Sequel::SQL::Blob:0x12a294c bytes=110 start=" File \"/o" end="_complete\n"> +
#<Sequel::SQL::Blob:0x12a2960 bytes=27 start=" return" end=".result()\n"> +
#<Sequel::SQL::Blob:0x12a2974 bytes=135 start=" File \"/o" end="_pipeline\n"> +
#<Sequel::SQL::Blob:0x12a2988 bytes=35 start=" await " end="*futures)\n"> +
#<Sequel::SQL::Blob:0x12a299c bytes=127 start=" File \"/o" end=" __call__\n"> +
#<Sequel::SQL::Blob:0x12a29b0 bytes=21 start=" await " end="elf.run()\n"> +
#<Sequel::SQL::Blob:0x12a29c4 bytes=134 start=" File \"/o" end="6, in run\n"> +
#<Sequel::SQL::Blob:0x12a29d8 bytes=55 start=" await " end="content)(\n"> +
#<Sequel::SQL::Blob:0x12a29ec bytes=114 start=" File \"/o" end=" __call__\n"> +
#<Sequel::SQL::Blob:0x12a2a00 bytes=55 start=" ret = " end="out=None)\n"> +
#<Sequel::SQL::Blob:0x12a2a14 bytes=94 start=" File \"/o" end=" wait_for\n"> +
#<Sequel::SQL::Blob:0x12a2a28 bytes=21 start=" return" end="await fut\n"> +
#<Sequel::SQL::Blob:0x12a2a3c bytes=100 start=" File \"/o" end="7, in run\n"> +
#<Sequel::SQL::Blob:0x12a2a50 bytes=48 start=" result" end="f.kwargs)\n"> +
#<Sequel::SQL::Blob:0x12a2a64 bytes=120 start=" File \"/o" end="d_handler\n"> +
#<Sequel::SQL::Blob:0x12a2a78 bytes=33 start=" return" end="**kwargs)\n"> +
#<Sequel::SQL::Blob:0x12a2a8c bytes=135 start=" File \"/o" end="d_content\n"> +
#<Sequel::SQL::Blob:0x12a2aa0 bytes=56 start=" Reposi" end="_content)\n"> +
#<Sequel::SQL::Blob:0x12a2ab4 bytes=131 start=" File \"/o" end="er_method\n"> +
#<Sequel::SQL::Blob:0x12a2ac8 bytes=63 start=" return" end="**kwargs)\n"> +
#<Sequel::SQL::Blob:0x12a2adc bytes=127 start=" File \"/o" end="lk_create\n"> +
#<Sequel::SQL::Blob:0x12a2af0 bytes=39 start=" obj_wi" end="= self.db\n"> +
#<Sequel::SQL::Blob:0x12a2b04 bytes=123 start=" File \"/o" end=" __exit__\n"> +
#<Sequel::SQL::Blob:0x12a2b18 bytes=24 start=" connec" end=".commit()\n"> +
#<Sequel::SQL::Blob:0x12a2b2c bytes=118 start=" File \"/o" end=" in inner\n"> +
#<Sequel::SQL::Blob:0x12a2b40 bytes=33 start=" return" end="**kwargs)\n"> +
#<Sequel::SQL::Blob:0x12a2b54 bytes=128 start=" File \"/o" end="in commit\n"> +
#<Sequel::SQL::Blob:0x12a2b68 bytes=19 content=" self._commit()\n"> +
#<Sequel::SQL::Blob:0x12a2b7c bytes=129 start=" File \"/o" end="n _commit\n"> +
#<Sequel::SQL::Blob:0x12a2b90 bytes=36 start=" return" end=".commit()\n"> +
#<Sequel::SQL::Blob:0x12a2ba4 bytes=116 start=" File \"/o" end=" __exit__\n"> +
#<Sequel::SQL::Blob:0x12a2bb8 bytes=64 start=" raise " end="exc_value\n"> +
#<Sequel::SQL::Blob:0x12a2bcc bytes=129 start=" File \"/o" end="n _commit\n"> +
#<Sequel::SQL::Blob:0x12a2be0 bytes=36 start=" return" end=".commit()\n">,
"description"=>
#<Sequel::SQL::Blob:0x12a2bf4 bytes=145 start="insert or " end="ore_repo\"\n"> +
#<Sequel::SQL::Blob:0x12a2c08 bytes=121 start="DETAIL: K" end="version\".\n">},
"worker"=>"/pulp/api/v3/workers/7a93fe04-23d5-4a3e-8185-a9823dda92eb/",
"child_tasks"=>[],
"progress_reports"=>
[{"message"=>"Downloading Metadata Files",
"code"=>"sync.downloading.metadata",
"state"=>"completed",
"done"=>6},
{"message"=>"Downloading Artifacts",
"code"=>"sync.downloading.artifacts",
"state"=>"completed",
"done"=>1},
{"message"=>"Associating Content",
"code"=>"associating.content",
"state"=>"completed",
"done"=>13},
{"message"=>"Parsed Packages",
"code"=>"sync.parsing.packages",
"state"=>"completed",
"done"=>8717},
{"message"=>"Un-Associating Content",
"code"=>"unassociating.content",
"state"=>"completed",
"done"=>1},
{"message"=>"Downloading Metadata Files",
"code"=>"sync.downloading.metadata",
"state"=>"completed",
"done"=>7},
{"message"=>"Downloading Artifacts",
"code"=>"sync.downloading.artifacts",
"state"=>"completed",
"done"=>0},
{"message"=>"Associating Content",
"code"=>"associating.content",
"state"=>"failed",
"done"=>17500},
{"message"=>"Parsed Modulemd",
"code"=>"sync.parsing.modulemds",
"state"=>"completed",
"total"=>294,
"done"=>294},
{"message"=>"Parsed Modulemd-defaults",
"code"=>"sync.parsing.modulemd_defaults",
"state"=>"completed",
"total"=>45,
"done"=>45},
{"message"=>"Parsed Packages",
"code"=>"sync.parsing.packages",
"state"=>"completed",
"done"=>17534},
{"message"=>"Parsed Comps",
"code"=>"sync.parsing.comps",
"state"=>"completed",
"total"=>68,
"done"=>68},
{"message"=>"Parsed Comps",
"code"=>"sync.parsing.comps",
"state"=>"completed",
"total"=>42,
"done"=>42},
{"message"=>"Parsed Advisories",
"code"=>"sync.parsing.advisories",
"state"=>"completed",
"total"=>0,
"done"=>0},
{"message"=>"Parsed Advisories",
"code"=>"sync.parsing.advisories",
"state"=>"completed",
"total"=>0,
"done"=>0}],
"created_resources"=>
["/pulp/api/v3/repositories/rpm/rpm/beec8c74-156f-418b-aaa7-0d0bb1557d6a/versions/34/"],
"reserved_resources_record"=>
["/pulp/api/v3/repositories/rpm/rpm/beec8c74-156f-418b-aaa7-0d0bb1557d6a/",
"shared:/pulp/api/v3/remotes/rpm/rpm/e2631782-486f-4ce2-904e-7719fee96152/"]}],
"task_groups"=>[],
"poll_attempts"=>{"total"=>43, "failed"=>1}}
Exception:
Katello::Errors::Pulp3Error: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL: Key (version_added_id)=(96b7cfb4-8497-46cd-a4cc-92ed352d3a28) is not present in table "core_repositoryversion".
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:26:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:94:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
pulpcore worker journal on content proxy:
-- Logs begin at Sun 2022-04-10 12:09:34 CEST, end at Mon 2022-04-18 09:57:45 CEST. --
Apr 18 03:54:44 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 3a9ac246-5ee0-40ca-ba84-a8b205739dc6
Apr 18 03:54:44 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 3a9ac246-5ee0-40ca-ba84-a8b205739dc6
Apr 18 03:54:48 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 04f84fb3-7db7-4080-86d0-e2ff656aee5a
Apr 18 03:54:48 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 04f84fb3-7db7-4080-86d0-e2ff656aee5a
Apr 18 03:54:51 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 844d42dc-da61-4442-b5d9-8c74a9a79064
Apr 18 03:54:51 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=1-el8-Testing-820129b7-1278-4be4-b0d0-bd9d9d950926 remote=1-el8-Testing-820129b7-1278-4be4-b0d0-bd9d9d950926
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulpcore.tasking.pulpcore_worker:INFO: Task 844d42dc-da61-4442-b5d9-8c74a9a79064 failed (insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: DETAIL: Key (version_added_id)=(96b7cfb4-8497-46cd-a4cc-92ed352d3a28) is not present in table "core_repositoryversion".
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: )
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: pulp [53b4a24f-d66c-46a1-b974-d0a616de2d70]: pulpcore.tasking.pulpcore_worker:INFO: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 377, in _perform_task
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: result = func(*args, **kwargs)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 544, in synchronize
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: repo_version = dv.create() or repo.latest_version()
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: loop.run_until_complete(pipeline)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: return future.result()
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: await asyncio.gather(*futures)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: await self.run()
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/plugin/stages/content_stages.py", line 296, in run
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: await sync_to_async(self.new_version.add_content)(
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py", line 444, in __call__
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: ret = await asyncio.wait_for(future, timeout=None)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/tasks.py", line 455, in wait_for
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: return await fut
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: result = self.fn(*self.args, **self.kwargs)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: return func(*args, **kwargs)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/repository.py", line 782, in add_content
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: RepositoryContent.objects.bulk_create(repo_content)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 523, in bulk_create
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: obj_without_pk._state.db = self.db
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/transaction.py", line 246, in __exit__
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: connection.commit()
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/utils/asyncio.py", line 33, in inner
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: return func(*args, **kwargs)
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/base/base.py", line 266, in commit
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: self._commit()
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/base/base.py", line 242, in _commit
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: return self.connection.commit()
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: raise dj_exc_value.with_traceback(traceback) from exc_value
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/base/base.py", line 242, in _commit
Apr 18 04:02:42 foreman-content.example.com pulpcore-worker-7[21258]: return self.connection.commit()
Apr 18 04:03:47 foreman-content.example.com pulpcore-worker-7[21258]: pulp [788606d6-5978-4313-bc88-ecb54d1d8700]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 95a39aa0-13b1-45da-b2cf-2a6dc3e6041b
Apr 18 04:03:47 foreman-content.example.com pulpcore-worker-7[21258]: pulp [788606d6-5978-4313-bc88-ecb54d1d8700]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 95a39aa0-13b1-45da-b2cf-2a6dc3e6041b