At wits end - Please help - can't get proxies to sync continuous pulp errors

sinewave · October 29, 2021, 8:08am

Problem:
Hi, please can someone help. Ive been having problems syncing to proxy servers with particular repos. I have found that the repos sometimes dont error with every lifecycle only some and just seems to complain about a metadata repo that. Ive created a test lifecycle to promote those content view.
I have regenerated the metadata and prmotoed new versions but this doesnt seem to make any difference. I can see the offensive item from browser and can download using wget.

I’ve done a browse to the location and file is there.
Ive done a wget and the file downloads fine.Ive unzipped and looked in the file and looks ok to me.
Why does pulp not like this file ? This isnt the only one that fails.

So far I have:
resync’d repo/
publish new version
promoted version
republished metadata
perfomed full sync
restarted all services , cleared all queues and old jobs

Expected outcome:
ability to sync to proxy servers
Foreman and Proxy versions:

Foreman and Proxy plugin versions:

foreman-2.3.5-1.el7.noarch
foreman-proxy-2.3.5-1.el7.noarch
katello-3.18.3-1.el7.noarch
candlepin-3.1.22-1.el7.noarch
pulp-admin-client-2.21.5-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.9-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
pulp-rpm-admin-extensions-2.21.5-1.el7.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
pulp-selinux-2.21.5-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch

Distribution and version:
Version 2.3.5
Other relevant data:

iballou · October 29, 2021, 12:41pm

Hi @sinewave,

Since you’re on Katello 3.18, we’d like to know if you’re running Pulp 2 or Pulp 3. Have you run the Pulp 3 migration and switchover at any point? What text do you see under “Pulpcore” under the “Services” tab on your primary smary proxy? You can find it with ::SmartProxy.pulp_primary in the foreman console.

sinewave · October 29, 2021, 1:40pm

Hi I dont see a services tab ? do you mean in the foreman web gui ? I’ve been using the about in the administrators tab.
using foreman-rake I was able to get the following but no mention of pulp. could you expand further please.

foreman-rake console
Loading production environment (Rails 6.0.3.4)
irb(main):001:0> ::SmartProxy.pulp_primary
=> #<SmartProxy id: 1, name: “blah-foreman.inf”, url: “https://blah-foreman.inf:9090”, created_at: “2021-04-23 11:38:44”, updated_at: “2021-04-23 11:38:45”, pubkey: “ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6ZdN7JdEDuP…”, expired_logs: “0”, puppet_path: “/etc/puppetlabs/code/environments”, download_policy: “on_demand”>
irb(main):002:0>

iballou · October 29, 2021, 2:07pm

I did mean in the Web UI. Go to the Smart Proxies page (under “Infrastructure” on the side bar) and click on your smart proxy that popped up in the console command. Then go over to the “Services” tab and show me what’s under “Pulpcore”. I’ll be looking for things like “yum (supported, but overridden to use Pulp 2)” or just “yum” along with other content types.

sinewave · October 29, 2021, 2:22pm

Thanks
everything I’ve looke at point to pulp2 that I’ve seen

Pulp

Version 2.1.0
Pulp server version 2.21.5
Database connection - green tick
Messaging connection - green tick
Workers 10
Supported Content Types* puppet

Pulpcore

Version 2.1.0
Supported Content Types

deb
docker
file
yum

from RPM’s installed - dont see a pulp3

pulp-docker-plugins-3.2.9-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-rpm-plugins-2.21.5-1.el7.noarch
pulp-selinux-2.21.5-1.el7.noarch
pulp-admin-client-2.21.5-1.el7.noarch
pulp-puppet-tools-2.21.5-1.el7.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulpcore-selinux-1.2.3-2.el7.x86_64
pulp-katello-1.0.3-1.el7.noarch
pulp-rpm-admin-extensions-2.21.5-1.el7.noarch
pulp-puppet-plugins-2.21.5-1.el7.noarch

iballou · October 29, 2021, 2:47pm

Pulpcore is the same thing as Pulp 3, so you are on Pulp 3 now.

You mentioned it “seems to complain about a metadata repo”. Are there errors perhaps on the sync tasks that you’re seeing? I’m guessing there will be both a backtrace from Pulp and Katello in the errored out task. If there are, can you paste them there?

sinewave · October 29, 2021, 3:15pm

Thanks, does that mean my smart proxys are on pulp2 as they dont have that pulpcore attribute in the services,

Pulp node

Version 2.1.0
Pulp server version2.21.5

Actually I’ve foud an error on a repo saying there are too many pulp files open syncing CentOS6_updates_x86_64
[Errno 24] Too many open files: ‘/var/lib/pulp/media/artifact/7f/aa5a0d50ae191297b5198525e957601d2b5d1e2453b229a242ce5224b04dc3’

The sync errors look similar. here is one action with the backtrace. The repo issue above is not in this content.

Action
Actions::Pulp::Consumer::SyncCapsule

###############################################

Input:
{“capsule_id”=>5,
“repo_pulp_id”=>
“3-Oracle7_Foreman23_katello_client_UEK3-Spacewalk-6fa6697f-aa8a-45ee-8f91-5ae5c91328d9”,
“repository_id”=>677,
“sync_options”=>{“remove_missing”=>true, “force_full”=>true},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>nil,
“current_timezone”=>“UTC”,
“current_user_id”=>4,
“current_organization_id”=>nil,
“current_location_id”=>nil}

##################################################

Output:
{“smart_proxy_history_id”=>20147,
“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/3c897482-eb5e-43d8-8138-bb71a5f5354e/",
“task_id”=>“3c897482-eb5e-43d8-8138-bb71a5f5354e”,
“tags”=>
[“pulp:repository:3-Oracle7_Foreman23_katello_client_UEK3-Spacewalk-6fa6697f-aa8a-45ee-8f91-5ae5c91328d9”,
“pulp:action:sync”],
“finish_time”=>“2021-10-18T18:46:30Z”,
“_ns”=>“task_status”,
“start_time”=>“2021-10-18T18:44:47Z”,
“traceback”=>
“Traceback (most recent call last):\n” +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 688, in call\n" +
" return super(Task, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 110, in call\n" +
" return super(PulpTask, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call\n" +
" return self.run(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 802, in sync\n" +
" sync_report = sync_repo(transfer_repo, conduit, call_config)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 905, in wrap_f\n" +
" return f(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py”, line 84, in sync_repo\n" +
" report = self._current_sync.run()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 287, in run\n" +
" metadata_files = self.get_metadata(metadata_files)\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 490, in get_metadata\n" +
" metadata_files.download_metadata_files()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py”, line 236, in download_metadata_files\n" +
" raise PulpCodedException(error_code=error_codes.RPM1004, reason=reason)\n" +
“PulpCodedException: Error retrieving metadata: Error ‘Proxy Error’ for https://gbh-foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Spacewalk/Oracle7_Foreman23_katello_client_UEK3/custom/CentOS7_Base/Centos7_epel_x86_64/repodata/f4816e53454a8750d718a44a8558857192b1cf14b3ac9a06a920d28e8f84d1ec-primary.xml.gz.\n”,
“spawned_tasks”=>,
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“details”=>
{“rpm_total”=>0, “rpm_done”=>0, “drpm_total”=>0, “drpm_done”=>0},
“size_total”=>0,
“size_left”=>0,
“items_left”=>0},
“comps”=>{“state”=>“NOT_STARTED”},
“purge_duplicates”=>{“state”=>“NOT_STARTED”},
“distribution”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“NOT_STARTED”},
“errata”=>{“state”=>“NOT_STARTED”},
“metadata”=>
{“state”=>“FAILED”,
“error”=>
"Error retrieving metadata: Error ‘Proxy Error’ for https://gbh-foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Spacewalk/Oracle7_Foreman23_katello_client_UEK3/custom/CentOS7_Base/Centos7_epel_x86_64/repodata/f4816e53454a8750d718a44a8558857192b1cf14b3ac9a06a920d28e8f84d1ec-primary.xml.gz."}}},
“queue”=>
“reserved_resource_worker-4@zajb-foreman-proxy.inf.itsystems.global.dq2”,
“state”=>“error”,
“worker_name”=>
“reserved_resource_worker-4@zajb-foreman-proxy.inf.itsystems.global”,
“result”=>nil,
“error”=>
{“code”=>“RPM1004”,
“data”=>
{“reason”=>
“Error ‘Proxy Error’ for https://gbh-foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Spacewalk/Oracle7_Foreman23_katello_client_UEK3/custom/CentOS7_Base/Centos7_epel_x86_64/repodata/f4816e53454a8750d718a44a8558857192b1cf14b3ac9a06a920d28e8f84d1ec-primary.xml.gz."},
“description”=>
“Error retrieving metadata: Error ‘Proxy Error’ for https://gbh-foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Spacewalk/Oracle7_Foreman23_katello_client_UEK3/custom/CentOS7_Base/Centos7_epel_x86_64/repodata/f4816e53454a8750d718a44a8558857192b1cf14b3ac9a06a920d28e8f84d1ec-primary.xml.gz.",
“sub_errors”=>},
“_id”=>{”$oid”=>“616dc0978360d5043a7e55ce”},
“id”=>“616dc0978360d5043a7e55ce”}],
“poll_attempts”=>{“total”=>23, “failed”=>1}}

##########################################

Exception:

Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Error ‘Proxy Error’ for https://gbh-foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Spacewalk/Oracle7_Foreman23_katello_client_UEK3/custom/CentOS7_Base/Centos7_epel_x86_64/repodata/f4816e53454a8750d718a44a8558857192b1cf14b3ac9a06a920d28e8f84d1ec-primary.xml.gz.

################################################

Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:121:in block in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:119:in each’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:119:in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in run’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:45:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:31:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:16:in run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’ /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in with_progress_calculation’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in block in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in catch’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

iballou · November 1, 2021, 12:43pm

Yeah, your Smart Proxies weren’t part of the Pulp 3 migration. That’s fine though, you should still be able to sync to Pulp 2 Smart Proxies.

Did this error occur during a verify content sync, or a normal one? The Pulp version in 3.18 is decently old, so I wouldn’t be surprised if it was fixed in the Pulp 3 versions that Katello 4.1 and 4.2 have.

As for the Proxy Errors, you’re able to wget/curl/etc those failing metadata files from your Katello server?

sinewave · November 1, 2021, 1:34pm

Hi, I’m not sure I remember. Ive tred so many different ways. and none of them have worked. they all fail on a similar theme.
Ive done optimised sync, and verification as well as the
hammer capsule content synchronize --id --skip-metadata-check true
I can rerun a sync a particular way of it helps.
Yes I can curl and wget the repomd file it complains about. as well as browse to it so the file exists I dont know why it doesnt like it.

iballou · November 1, 2021, 1:53pm

It looks like there are a couple issues going on. Seeing the error that pops up for a normal repository sync would be helpful if it fails.

As for the proxy error, I’ll see if I can some information from folks who know Pulp 2 better.

iballou · November 1, 2021, 2:22pm

What happens if you wget those metadata files from the Smart Proxy that is syncing? Does it still work?

I’d also be curious if you see any errors from the apache logs on your Katello server, for example:

/var/log/httpd/foreman-ssl_error_ssl.log
/var/log/httpd/foreman-ssl_access_ssl.log

iballou · November 1, 2021, 3:39pm

Here’s something to try:

Edit /etc/systemd/system/pulpcore-content.service and increase the number of workers by bumping up the -w 2 flag in ExecStart. With the version of Pulpcore running on Katello 3.18, it would start throwing errors like the proxy one you saw if it was getting overloaded. If your current worker amount is currently at 2, perhaps try 4. Then, restart the Pulpcore services with systemctl restart pulpcore* --all.

I’d recommend that you upgrade to Katello 4.1 if these issues keep causing troubles. There have been some big updates to Pulp 3 between Katello 3.18 and Katello 4.1.

sinewave · November 1, 2021, 3:58pm

Hi the foreman-sel_error_ssl.log is full of this error
Mon Nov 01 15:45:25.311226 2021] [ssl:warn] [pid 9599] [client ..**.94:43536] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’
no errors in the access.log

sinewave · November 1, 2021, 4:21pm

Thanks, I’ve made the change and trying to sync one of the lifecycle environments

sinewave · November 1, 2021, 6:59pm

I’ve rerun the sync with only one of my lifecycle environments. The sync page ha updated to 100%
howeve closed with a warning and the same errors as before.

I should add that I currently have this fix in place - Issue #8713: Pulp 3 to Pulp 2 sync fails if comps.xml has a group with an empty packagelist - RPM Support - Pulp
/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py
xml_options={“default_explicit”: True, “empty_groups”: False, “uservisible_explicit”: True},

#######################################################
Warning Message

RPM1004: Error retrieving metadata: Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Centos6_foreman23_Katello_client/custom/Katello_client_EL6_dependencies/Non-foreman_katello_client_EL6_dependencies/repodata/c0cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898-updateinfo.xml.gz.RPM1004: Error retrieving metadata: Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Oracle7_Foreman23_katello_client/custom/Oracle_Linux_7_Base/Oracle_Linux_7_Update_8/repodata/c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml.gz.RPM1004: Error retrieving metadata: Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Centos6_foreman23_Katello_client/custom/CentOS6_Base/CentOS6_extras_x86_64/repodata/73c072697867587f168eccf620d08fb6a851272c5629e10ef9cc5868b4800e8f-other.xml.gz.
#######################################################

From the proxy server I’m syncing to.

wget https://server.blah.blah/pulp/repos/org_name/Spacewalk/Oracle7_Foreman23_katello_client/custom/Oracle_Linux_7_Base/Oracle_Linux_7_Update_8/repodata/c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml.gz

Unzipped the content
2236951 Oct 17 17:34 c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml

Example of content

<?xml version="1.0" encoding="UTF-8"?> - Initial RPMification, based on Jan "Yenya" Kasprzak's & lt;kas@fi.muni.cz> spec. - Use autoconf - New binary names - Back to old binary names - Renamed init script - Added links for slogin

#################################################################
RAW Error from task

ACTION:

Actions::Pulp::Consumer::SyncCapsule

#################################
Input:

{“capsule_id”=>2,
“repo_pulp_id”=>
“3-Oracle7_Foreman23_katello_client-Spacewalk-8204c320-0e5f-46fe-9c26-4bbcf1e7f298”,
“repository_id”=>623,
“sync_options”=>{“remove_missing”=>true},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>nil,
“current_timezone”=>“Europe/London”,
“current_user_id”=>4,
“current_organization_id”=>3,
“current_location_id”=>2}

################################################
Output:

{“smart_proxy_history_id”=>22516,
“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/32744cf3-8be6-48e2-893f-ef8d5725a3fb/",
“task_id”=>“32744cf3-8be6-48e2-893f-ef8d5725a3fb”,
“tags”=>
[“pulp:repository:3-Oracle7_Foreman23_katello_client-Spacewalk-8204c320-0e5f-46fe-9c26-4bbcf1e7f298”,
“pulp:action:sync”],
“finish_time”=>“2021-11-01T16:16:49Z”,
“_ns”=>“task_status”,
“start_time”=>“2021-11-01T16:16:44Z”,
“traceback”=>
“Traceback (most recent call last):\n” +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 688, in call\n" +
" return super(Task, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 110, in call\n" +
" return super(PulpTask, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call\n" +
" return self.run(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 802, in sync\n" +
" sync_report = sync_repo(transfer_repo, conduit, call_config)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 905, in wrap_f\n" +
" return f(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py”, line 84, in sync_repo\n" +
" report = self._current_sync.run()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 287, in run\n" +
" metadata_files = self.get_metadata(metadata_files)\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 490, in get_metadata\n" +
" metadata_files.download_metadata_files()\n" +
" File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py”, line 236, in download_metadata_files\n" +
" raise PulpCodedException(error_code=error_codes.RPM1004, reason=reason)\n" +
“PulpCodedException: Error retrieving metadata: Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Oracle7_Foreman23_katello_client/custom/Oracle_Linux_7_Base/Oracle_Linux_7_Update_8/repodata/c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml.gz.\n”,
“spawned_tasks”=>,
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“details”=>
{“rpm_total”=>0, “rpm_done”=>0, “drpm_total”=>0, “drpm_done”=>0},
“size_total”=>0,
“size_left”=>0,
“items_left”=>0},
“comps”=>{“state”=>“NOT_STARTED”},
“purge_duplicates”=>{“state”=>“NOT_STARTED”},
“distribution”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“NOT_STARTED”},
“errata”=>{“state”=>“NOT_STARTED”},
“metadata”=>
{“state”=>“FAILED”,
“error”=>
"Error retrieving metadata: Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Oracle7_Foreman23_katello_client/custom/Oracle_Linux_7_Base/Oracle_Linux_7_Update_8/repodata/c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml.gz."}}},
“queue”=>
“reserved_resource_worker-3@gbb-foreman-proxy.server.blah.blah.dq2”,
“state”=>“error”,
“worker_name”=>
“reserved_resource_worker-3@gbb-foreman-proxy.server.blah.blah”,
“result”=>nil,
“error”=>
{“code”=>“RPM1004”,
“data”=>
{“reason”=>
“Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Oracle7_Foreman23_katello_client/custom/Oracle_Linux_7_Base/Oracle_Linux_7_Update_8/repodata/c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml.gz."},
“description”=>
“Error retrieving metadata: Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Oracle7_Foreman23_katello_client/custom/Oracle_Linux_7_Base/Oracle_Linux_7_Update_8/repodata/c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml.gz.",
“sub_errors”=>},
“_id”=>{”$oid”=>“618012a4842207964d5e5dc8”},
“id”=>“618012a4842207964d5e5dc8”}],
“poll_attempts”=>{“total”=>23, “failed”=>1}}

######################################################

Exception:

Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Error ‘Proxy Error’ for https://server.blah.blah/pulp/repos/org_name/Spacewalk/Oracle7_Foreman23_katello_client/custom/Oracle_Linux_7_Base/Oracle_Linux_7_Update_8/repodata/c3147f49ba1e9e4248f70bc61a07b4760cf807f1c80cde01a43f289078a8e5eb-other.xml.gz.

######################################################################
Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:121:in block in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:119:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:119:in external_task=' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in poll_external_task_with_rescue’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/pulp/abstract_async_task.rb:45:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:31:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/app/lib/actions/middleware/remote_action.rb:16:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in block in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

iballou · November 2, 2021, 2:35pm

Hi @sinewave,

Do you see any errors in /var/log/messages during the sync? I’m still thinking it has to do with the Pulpcore content service getting overloaded.

How many content app workers are you running now?

iballou · November 2, 2021, 2:50pm

I looked at my own /etc/systemd/system/pulpcore-content.service, and I have -w set to (2 * CPU cores + 1). I also have the --timeout 90 flag set. You may want to add that flag and bump up your workers accordingly if you haven’t tried already.

sinewave · November 2, 2021, 3:51pm

Hi,
dont seem to have any pulp errors in messages but I do get this error between the foreman server and the proxy server.
Nov 2 15:27:49 server-foreman qdrouterd: 2021-11-02 15:27:49.143309 +0000 SERVER (info) [C47073470] Accepted connection to :5647 from -foreman-proxy:38582
Nov 2 15:27:49 server-foreman qdrouterd: 2021-11-02 15:27:49.266896 +0000 ROUTER_CORE (info) [C47073470] Connection Opened: dir=in host=foreman-proxy:38582 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=anonymous container_id=59a05cc8-25ed-4560-a8f8-456f79376bff props=
Nov 2 15:37:24 server-foreman qdrouterd: 2021-11-02 15:37:24.807373 +0000 SERVER (info) [C47089434] Connection from foreman-proxy:38756 (to :5647) failed: amqp:connection:framing-error SSL Failure: Unknown error

I’ll try increase the resource as well.

sinewave · November 2, 2021, 6:54pm

I increased the workers to cores+1 and I reduced the number of lifecycles to one and the sync completed without error. Which is great.
However when adding additional lifecycle the same errors returned. I have multple proxy’s so I’m going to sync them all to at least that lifecycle. if they all work that at least is progress.

iballou · November 2, 2021, 7:24pm

I’m glad you were able to sync some environments. Besides further tuning the worker amount and that timeout, I don’t have many more suggestions right now.

I’d personally start preparing to upgrade up to Katello 4.1 or 4.2.1 when it’s out. It should help with this issue any potential others too.

Also, it won’t be necessary yet, but by 4.3 you’ll need your smart proxies to be running Pulp 3 as well. There’s no migration process for smart proxies other than the one on your Katello server, so you’ll need to install them fresh and re-sync.