At wits end - Please help - can't get proxies to sync continuous pulp errors

Support
, ,
22

Thanks for your help its much appreciated.

2 Likes
23

All has been syncing fine, however for nor real reason. I’m now getting error and failed sync

" PulpCodedException: Checksum type “sha256” is not available for all units in the repository. Make sure those units have been downloaded "

I dont have an srpm’s which can cause this seems to widespread through all content views.
Things that have changed
installed ansible plugin
applied updates of 7.9 centos to Foreman and proxy’s
I edited /etc/systemd/system/pulpcore-content.service increased the workers to 2*+1CPU, I never could get that timeout syntax to work. Pulp would never srtart
ExecStart=/usr/libexec/pulpcore/gunicorn pulpcore.content:server
–worker-class ‘aiohttp.GunicornWebWorker’
-w 25
–access-logfile -
ExecReload=/bin/kill -s HUP $MAINPID
SyslogIdentifier=pulpcore-content

I have removed product packages of repo, resyc’d, published, republished contentviews.
full sync with removed lifecycles - which works but didnt really seem to clear out the DB the way I expected.
Now just trying to sync using just 1 content view - fails
removed pulp worker cache as flesystem went to 100%
restart pulp workers, reboot proxy and foreman
removed orphans
Ive been round this in a loop a few times now

Any sugggestions ?

24

@iballou any thought on this ?

25

Hi @sinewave,

Are you still on Katello 3.18? Do you have the latest pulp-rpm 3.11 version?

I found a bugzilla talking about what I think is the same issue: 1958415 – On_demand capsule sync (pulp3 to pulp2) for non-sha256 repos fails with a checksum type issue

The problem seems to have been partially fixed in pulp-rpm 3.11 but it seems some of it had to be fixed in later versions: https://bugzilla.redhat.com/show_bug.cgi?id=1958415#c12

A workaround is mentioned there if necessary.

There won’t be more releases of pulp-rpm 3.11, so if the issue isn’t solved by the patch mentioned in that bugzilla, I’d highly recommend upgrading.

1 Like
26

Hi, thanks for replyin. I’ve out the proxy on immediatesync.
Yes I’m still on same version until I an get a test system set up to run tthough the upgrade. Hopefully can get that upgrade planned out soon.
IIt is taking a very long time to sync. its not happy with something

Actions::Pulp::Consumer::SyncCapsule
State:suspended

Input:
{“capsule_id”=>6,
“repo_pulp_id”=>
“3-Centos7_content-test1-0a63d369-6561-4dce-8f2f-4b9a621af369”,
“repository_id”=>1528,
“sync_options”=>{“remove_missing”=>true, “force_full”=>true},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>nil,
“current_timezone”=>“Europe/London”,
“current_user_id”=>4,
“current_organization_id”=>3,
“current_location_id”=>2}
Output:
{“smart_proxy_history_id”=>30442,
“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/adbacfc4-276c-4691-9d88-df3dbd7ae8e8/",
“task_id”=>“adbacfc4-276c-4691-9d88-df3dbd7ae8e8”,
“tags”=>
[“pulp:repository:3-Centos7_content-test1-0a63d369-6561-4dce-8f2f-4b9a621af369”,
“pulp:action:sync”],
“finish_time”=>nil,
“_ns”=>“task_status”,
“start_time”=>“2022-04-06T08:47:50Z”,
“traceback”=>nil,
“spawned_tasks”=>,
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“items_total”=>10573,
“state”=>“IN_PROGRESS”,
“error_details”=>
[{“url”=>
"https://foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/test1/Centos7_content/custom/CentOS7_Base/Centos7_extras_x86_64/Packages/m/mvapich2-2.2-2.2-4.el7.i686.rpm",
** “errors”=>[“Proxy Error”]}],**
“details”=>
{“rpm_total”=>10573,
“rpm_done”=>9838,
“drpm_total”=>0,
“drpm_done”=>0},
“size_total”=>10542941600,
“size_left”=>550327256,
“items_left”=>735},
“comps”=>{“state”=>“NOT_STARTED”},
“purge_duplicates”=>{“state”=>“NOT_STARTED”},
“distribution”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“NOT_STARTED”},
“errata”=>{“state”=>“NOT_STARTED”},
“metadata”=>{“state”=>“FINISHED”}}},
“queue”=>
“reserved_resource_worker-9@foreman-proxy.inf.itsystems.global.dq2”,
“state”=>“running”,
“worker_name”=>
“reserved_resource_worker-9@foreman-proxy.inf.itsystems.global”,
“result”=>nil,
“error”=>nil,
“_id”=>{"$oid"=>“624d53b61a288934535d83c9”},
“id”=>“624d53b61a288934535d83c9”}],
“poll_attempts”=>{“total”=>1227, “failed”=>0}}

27

Only one proxy server seems to have a problem now. I removed all content and did a sync which worked. Then added small amount of test content and that worked.
However sync job just runs for days looks like it just times out then moves on. Any idea @iballou ?

{“capsule_id”=>6,
“repo_pulp_id”=>
“3-Oracle7_Foreman23_katello_client_UEK3-Spacewalk-cca1332e-f426-40ee-b917-581a287f0740”,
“repository_id”=>1059,
“sync_options”=>{“remove_missing”=>true},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>nil,
“current_timezone”=>“Europe/London”,
“current_user_id”=>4,
“current_organization_id”=>3,
“current_location_id”=>2}
Output:
{“smart_proxy_history_id”=>31451,
“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/2a528de9-0c25-476c-b7d6-12e0fe3d0858/",
“task_id”=>“2a528de9-0c25-476c-b7d6-12e0fe3d0858”,
“tags”=>
[“pulp:repository:3-Oracle7_Foreman23_katello_client_UEK3-Spacewalk-cca1332e-f426-40ee-b917-581a287f0740”,
“pulp:action:sync”],
“finish_time”=>nil,
“_ns”=>“task_status”,
“start_time”=>“2022-04-19T15:40:56Z”,
“traceback”=>nil,
“spawned_tasks”=>,
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“items_total”=>821,
“state”=>“IN_PROGRESS”,
"error_details"=>
** [{“url”=>**
** “https://gbh-foremanl/pulp/repos/IT_Systems_Global/Spacewalk/Oracle7_Foreman23_katello_client_UEK3/custom/Oracle_Linux_7_UEK_3/Oracle_Linux_7_UEK_3/Packages/k/kernel-uek-firmware-3.8.13-118.15.3.el7uek.noarch.rpm”,**
** “errors”=>[“Proxy Error”]}],**
“details”=>
{“rpm_total”=>821,
“rpm_done”=>803,
“drpm_total”=>0,
“drpm_done”=>0},
“size_total”=>11656402624,
“size_left”=>276469776,
“items_left”=>18},
“comps”=>{“state”=>“NOT_STARTED”},
“purge_duplicates”=>{“state”=>“NOT_STARTED”},
“distribution”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“NOT_STARTED”},
“errata”=>{“state”=>“NOT_STARTED”},
“metadata”=>{“state”=>“FINISHED”}}},
“queue”=>
“reserved_resource_worker-9@foreman-proxy.dq2”,
“state”=>“running”,
“worker_name”=>
“reserved_resource_worker-9@foreman-proxy”,
“result”=>nil,
“error”=>nil,
“_id”=>{"$oid"=>“625ea7ec5f4417acf07a6f58”},
“id”=>“625ea7ec5f4417acf07a6f58”}],
“poll_attempts”=>{“total”=>4536, “failed”=>0}}

28

Hi @sinewave,

Are there any proxies between your smart proxy mirror (id 6) and your foreman instance? Just trying to download that kernel-uek-firmware-3.8.13-118.15.3.el7uek.noarch.rpm package would be a good first test.

I’ll ask the Pulp team if they remember any issues relating to Pulp 2 → Pulp 3 syncs and Proxy Error.

1 Like
29

Another place to look for info would be the apache logs. The Proxy Error is likely indicating that there’s a timeout going on somewhere

1 Like
30

my wget from proxy to the foreman server worked fine with no problems

31

I removed all content and did a sync which worked. Then added small amount of test content and that worked.

“rpm_total”=>821,
“rpm_done”=>803,

This seems to me that pulp is succeeding with some of the packages and then failing for a few. Do you see any relevant logs at /var/log/messages on the proxy when syncing stops?

32

there are some wrrors in the apache log on the proxy.

The sync has bee running for days and still running probably due to these continuous timeouts.
foreman-proxy goferd: [WARNING][pulp.agent.47cf9c70-116e-4009-bc88-87cbe18157b0] gofer.messaging.adapter.proton.reliability:54 - Connection amqps://foreman.global:5647 disconnected: Condition(‘amqp:resource-limit-exceeded’, ‘local-idle-timeout expired’)

33

One other funny is that even with 0 content and job completes successfully. On the smart proxy page in foreman where you start synchronisation it never says 100% on the green bar. all other proxy’s are fine.

34

What are the foreman server and proxy version you are on?

Is the dynflow task for the capsule sync task complete or does it say pending/paused? Some screenshots of the task and expanded sub-task if it’s in paused/suspended states will be helpful.

Also a complete sync on the proxy and monitoring the /var/log/messages in case it’s stuck for any errors.

35

Hi, thanks for your help on this.
Here are version list , apache error logs, message log and dynflow console for actions of success, pulp checking and pending.

I’m running Foreman 2.3.5
• foreman-2.3.5-1.el7.noarch
• foreman-cli-2.3.5-1.el7.noarch
• foreman-debug-2.3.5-1.el7.noarch
• foreman-dynflow-sidekiq-2.3.5-1.el7.noarch
• foreman-installer-2.3.5-1.el7.noarch
• foreman-installer-katello-2.3.5-1.el7.noarch
• foreman-postgresql-2.3.5-1.el7.noarch
• foreman-proxy-2.3.5-1.el7.noarch
• foreman-release-2.3.5-1.el7.noarch
• foreman-selinux-2.3.5-1.el7.noarch
• foreman-service-2.3.5-1.el7.noarch

Katello 3.14
Pulp 3 on foreman
Pulp2 on proxy

Proxy /var/log/messages a few examples:

Apr 22 11:23:29 foreman-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.inf.itsystems.global
Apr 22 11:23:29 foreman-proxy goferd: [INFO][pulp.agent.47cf9c70-116e-4009-bc88-87cbe18157b0] proton:1054 - Connected to foreman.inf.itsystems.global
Apr 22 11:23:29 foreman-proxy goferd: [INFO][pulp.agent.47cf9c70-116e-4009-bc88-87cbe18157b0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://foreman.inf.itsystems.global:5647
Apr 22 11:23:29 foreman-proxy goferd: [INFO][pulp.agent.47cf9c70-116e-4009-bc88-87cbe18157b0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://foreman.inf.itsystems.global:5647
Apr 22 11:23:32 foreman-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Production/Centos6_foreman23_Katello_client/custom/CentOS6_Base/CentOS6_epel_x86_64/Packages/u/unison227-2.27.57-13.el6.x86_64.rpm.
Apr 22 11:23:32 foreman-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.inf.itsystems.global
Apr 22 11:23:32 foreman-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Non-Production/Oracle6_foreman23_Katello_client/custom/CentOS6_Base/CentOS6_epel_x86_64/Packages/p/perl-Devel-Hexdump-0.02-1.el6.x86_64.rpm.
Apr 22 11:23:32 foreman-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.inf.itsystems.global
Apr 22 11:23:33 foreman-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Production/Centos6_foreman23_Katello_client/custom/CentOS6_Base/CentOS6_epel_x86_64/Packages/s/snmpcheck-1.8-4.el6.noarch.rpm.
Apr 22 11:23:33 foreman-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.inf.itsystems.global
Apr 22 11:23:33 foreman-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Non-Production/Oracle6_Content/custom/Oracle_Linux_6_Base/Oracle_Linux_6_base_x86_64/Packages/j/java-1.8.0-openjdk-headless-debug-1.8.0.262.b10-0.el6_10.x86_64.rpm.
Apr 22 11:23:33 foreman-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.inf.itsystems.global
Apr 22 11:23:37 foreman-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Non-Production/Oracle6_Content/custom/Oracle_Linux_6_Base/Oracle_Linux_6_base_x86_64/Packages/k/kdemultimedia-devel-4.3.4-3.el6.i686.rpm.
Apr 22 11:23:37 foreman-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.inf.itsystems.global
Apr 22 11:23:37 foreman-proxy qdrouterd: SERVER (info) [C25825] Accepted connection to :5647 from 100.8.127:60840
Apr 22 11:23:37 foreman-proxy qdrouterd: SERVER (info) [C25825] Connection from “ a ClientIP”:60840 (to :5647) failed: amqp:connection:framing-error No protocol header found (connection aborted)
Apr 22 11:23:37 foreman-proxy qdrouterd: SERVER (info) [C25826] Accepted connection to :5647 from 100.:34340
Apr 22 11:23:37 foreman-proxy qdrouterd: SERVER (info) [C25826] Connection from “a Client”:34340 (to :5647) failed: amqp:connection:framing-error No protocol header found (connection aborted)
Apr 22 11:23:37 foreman-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.inf.itsystems.global/pulp/repos/IT_Systems_Global/Non-Production/Oracle6_foreman23_Katello_client/custom/CentOS6_Base/CentOS6_epel_x86_64/Packages/p/python-remoteobjects-1.2.1-1.el6.noarch.rpm.

Apache error logs

pulp-https_error_ssl.log
[Wed Apr 20 12:13:34.569342 2022] [ssl:warn] [pid 585] [client “Foreman”:53468] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’

error_log
[Tue Apr 19 11:36:12.053596 2022] [so:warn] [pid 1290] AH01574: module proxy_module is already loaded, skipping
[Tue Apr 19 11:36:12.054299 2022] [so:warn] [pid 1290] AH01574: module proxy_http_module is already loaded, skipping
[Tue Apr 19 11:36:12.054476 2022] [so:warn] [pid 1290] AH01574: module ssl_module is already loaded, skipping
[Tue Apr 19 11:36:12.054501 2022] [so:warn] [pid 1290] AH01574: module systemd_module is already loaded, skipping
[Tue Apr 19 11:36:12.054541 2022] [so:warn] [pid 1290] AH01574: module cgi_module is already loaded, skipping
[Tue Apr 19 11:36:12.058129 2022] [alias:warn] [pid 1290] AH00671: The Alias directive in /etc/httpd/conf.d/autoindex.conf at line 21 will probably never match because it overlaps an earlier Alias.
[Tue Apr 19 11:36:12.058523 2022] [mime_magic:error] [pid 1290] (2)No such file or directory: AH01515: mod_mime_magic: can’t read magic file /etc/httpd/NEVER_EVER_USE
[Tue Apr 19 11:36:12.064838 2022] [lbmethod_heartbeat:notice] [pid 1290] AH02282: No slotmem from mod_heartmonitor
[Tue Apr 19 11:36:12.069656 2022] [mpm_prefork:notice] [pid 1290] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 configured – resuming normal operations
[Tue Apr 19 11:36:12.069682 2022] [core:notice] [pid 1290] AH00094: Command line: ‘/usr/sbin/httpd -D FOREGROUND’

From Dynflow console :
This is a success action example: took neary 24 hours to complete and 4 hours running for just this one.

Queue: default

Started at: 2022-04-19 12:15:40 UTC

Ended at: 2022-04-20 23:22:44 UTC

Real time: 126424.90s

Execution time (excluding suspended state): 13893.61s

Input:

capsule_id: 6
repo_pulp_id: 3-foreman2_3_Proxy-Non-Production-c297a282-c900-491b-87ed-dba70af43020
repository_id: 1403
sync_options:
remove_missing: true
remote_user: admin
remote_cp_user: admin
current_request_id:
current_timezone: Europe/London
current_user_id: 4
current_organization_id: 3
current_location_id: 2
Output:

smart_proxy_history_id: 31460
pulp_tasks:

  • exception:
    task_type: pulp.server.managers.repo.sync.sync
    _href: “/pulp/api/v2/tasks/64489192-6776-4388-9071-9ae33f4f72f0/”
    task_id: 64489192-6776-4388-9071-9ae33f4f72f0
    tags:
    • pulp:repository:3-foreman2_3_Proxy-Non-Production-c297a282-c900-491b-87ed-dba70af43020
    • pulp:action:sync
      finish_time: ‘2022-04-20T22:41:37Z’
      _ns: task_status
      start_time: ‘2022-04-20T11:22:47Z’
      traceback:
      spawned_tasks:
    • _href: “/pulp/api/v2/tasks/505ebf0f-2d92-4020-8916-cc7f8f5a0a8a/”
      task_id: 505ebf0f-2d92-4020-8916-cc7f8f5a0a8a
      progress_report:
      yum_importer:
      content:
      items_total: 829
      state: FINISHED
      error_details:
      details:
      rpm_total: 829
      rpm_done: 829
      drpm_total: 0
      drpm_done: 0
      size_total: 6831914204
      size_left: 0
      items_left: 0
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      items_total: 0
      state: FINISHED
      error_details:
      items_left: 0
      modules:
      state: FINISHED
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      queue: reserved_resource_worker-7@foreman-proxy.inf.itsystems.global.dq2
      state: finished
      worker_name: reserved_resource_worker-7@ -foreman-proxy.inf.itsystems.global
      result:
      result: success
      importer_id: yum_importer
      exception:
      repo_id: 3-foreman2_3_Proxy-Non-Production-c297a282-c900-491b-87ed-dba70af43020
      traceback:
      started: ‘2022-04-20T11:22:47Z’
      _ns: repo_sync_results
      completed: ‘2022-04-20T22:41:37Z’
      importer_type_id: yum_importer
      error_message:
      summary:
      modules:
      state: FINISHED
      content:
      state: FINISHED
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      state: FINISHED
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      added_count: 3569
      removed_count: 0
      updated_count: 0
      id: 62608c21d1c8df53996884a0
      details:
      modules:
      state: FINISHED
      content:
      size_total: 6831914204
      items_left: 0
      items_total: 829
      state: FINISHED
      size_left: 0
      details:
      rpm_total: 829
      rpm_done: 829
      drpm_total: 0
      drpm_done: 0
      error_details:
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      items_total: 0
      state: FINISHED
      error_details:
      items_left: 0
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      error:
      _id:
      “$oid”: 625ea7ee5f4417acf07a7025
      id: 625ea7ee5f4417acf07a7025
  • exception:
    task_type: pulp.server.managers.repo.publish.publish
    _href: “/pulp/api/v2/tasks/505ebf0f-2d92-4020-8916-cc7f8f5a0a8a/”
    task_id: 505ebf0f-2d92-4020-8916-cc7f8f5a0a8a
    tags:
    • pulp:repository:3-foreman2_3_Proxy-Non-Production-c297a282-c900-491b-87ed-dba70af43020
    • pulp:action:publish
      finish_time: ‘2022-04-20T23:22:41Z’
      _ns: task_status
      start_time: ‘2022-04-20T23:21:04Z’
      traceback:
      spawned_tasks:
      progress_report:
      3-foreman2_3_Proxy-Non-Production-c297a282-c900-491b-87ed-dba70af43020:
      • num_success: 1
        description: Initializing repo metadata
        step_type: initialize_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ed79f850-7632-48cc-be39-d404a67a764e
        num_processed: 1
      • num_success: 0
        description: Publishing Distribution files
        step_type: distribution
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 631e9447-60fe-4810-8ba9-99d69b3b98eb
        num_processed: 0
      • num_success: 3569
        description: Publishing RPMs
        step_type: rpms
        items_total: 3569
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 6644ccdc-76ff-49fb-b8dc-a9ae908883a6
        num_processed: 3569
      • num_success: 0
        description: Publishing Delta RPMs
        step_type: drpms
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: bcb70b3d-3071-4c98-8ac4-b23d7f040e6b
        num_processed: 0
      • num_success: 0
        description: Publishing Errata
        step_type: errata
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 4d53da87-ee5b-41d4-af17-a8342941be45
        num_processed: 0
      • num_success: 0
        description: Publishing Modules
        step_type: modules
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ‘04925283-b1ea-4fd6-bbf9-04bfe901b1ca’
        num_processed: 0
      • num_success: 0
        description: Publishing Comps file
        step_type: comps
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: b72cfdaf-a3fe-42d5-8702-fa2382fada6b
        num_processed: 0
      • num_success: 0
        description: Publishing Metadata.
        step_type: metadata
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 8e504874-9c56-4ac4-9aa1-359cc906f11b
        num_processed: 0
      • num_success: 1
        description: Closing repo metadata
        step_type: close_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: d5396a3f-445c-4a6e-b937-faaec70dd1e0
        num_processed: 1
      • num_success: 0
        description: Generating sqlite files
        step_type: generate sqlite
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ‘0594f177-6d17-40b7-bc66-970968fdb49b’
        num_processed: 0
      • num_success: 1
        description: Removing old repodata
        step_type: remove_old_repodata
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ac5575b1-05f9-496a-a50c-20e8b0b42a7f
        num_processed: 1
      • num_success: 0
        description: Generating HTML files
        step_type: repoview
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 70c7e274-4a72-42a7-9251-e8d90d987e50
        num_processed: 0
      • num_success: 1
        description: Publishing files to web
        step_type: publish_directory
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: bbffeeca-459b-43f2-8046-176b2dc2f5b7
        num_processed: 1
      • num_success: 1
        description: Writing Listings File
        step_type: initialize_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: f8c57e0d-c34f-482d-a625-2f47e48b05ca
        num_processed: 1
      • num_success: 1
        description: Writing Listings File
        step_type: initialize_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 79a7e563-5da4-4805-86b3-9fadaa4e9896
        num_processed: 1
        queue: reserved_resource_worker-7@ -foreman-proxy.inf.itsystems.global.dq2
        state: finished
        worker_name: reserved_resource_worker-7@ -foreman-proxy.inf.itsystems.global
        result:
        result: success
        exception:
        repo_id: 3-foreman2_3_Proxy-Non-Production-c297a282-c900-491b-87ed-dba70af43020
        started: ‘2022-04-20T23:21:04Z’
        _ns: repo_publish_results
        completed: ‘2022-04-20T23:22:41Z’
        traceback:
        distributor_type_id: yum_distributor
        summary:
        generate sqlite: SKIPPED
        rpms: FINISHED
        initialize_repo_metadata: FINISHED
        remove_old_repodata: FINISHED
        modules: SKIPPED
        close_repo_metadata: FINISHED
        drpms: SKIPPED
        comps: FINISHED
        distribution: FINISHED
        repoview: SKIPPED
        publish_directory: FINISHED
        errata: FINISHED
        metadata: FINISHED
        error_message:
        distributor_id: 3-foreman2_3_Proxy-Non-Production-c297a282-c900-491b-87ed-dba70af43020
        id: 626095c1d1c8df5399689105
        details:
      • num_success: 1
        description: Initializing repo metadata
        step_type: initialize_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ed79f850-7632-48cc-be39-d404a67a764e
        num_processed: 1
      • num_success: 0
        description: Publishing Distribution files
        step_type: distribution
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 631e9447-60fe-4810-8ba9-99d69b3b98eb
        num_processed: 0
      • num_success: 3569
        description: Publishing RPMs
        step_type: rpms
        items_total: 3569
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 6644ccdc-76ff-49fb-b8dc-a9ae908883a6
        num_processed: 3569
      • num_success: 0
        description: Publishing Delta RPMs
        step_type: drpms
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: bcb70b3d-3071-4c98-8ac4-b23d7f040e6b
        num_processed: 0
      • num_success: 0
        description: Publishing Errata
        step_type: errata
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 4d53da87-ee5b-41d4-af17-a8342941be45
        num_processed: 0
      • num_success: 0
        description: Publishing Modules
        step_type: modules
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ‘04925283-b1ea-4fd6-bbf9-04bfe901b1ca’
        num_processed: 0
      • num_success: 0
        description: Publishing Comps file
        step_type: comps
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: b72cfdaf-a3fe-42d5-8702-fa2382fada6b
        num_processed: 0
      • num_success: 0
        description: Publishing Metadata.
        step_type: metadata
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 8e504874-9c56-4ac4-9aa1-359cc906f11b
        num_processed: 0
      • num_success: 1
        description: Closing repo metadata
        step_type: close_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: d5396a3f-445c-4a6e-b937-faaec70dd1e0
        num_processed: 1
      • num_success: 0
        description: Generating sqlite files
        step_type: generate sqlite
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ‘0594f177-6d17-40b7-bc66-970968fdb49b’
        num_processed: 0
      • num_success: 1
        description: Removing old repodata
        step_type: remove_old_repodata
        items_total: 0
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: ac5575b1-05f9-496a-a50c-20e8b0b42a7f
        num_processed: 1
      • num_success: 0
        description: Generating HTML files
        step_type: repoview
        items_total: 1
        state: SKIPPED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 70c7e274-4a72-42a7-9251-e8d90d987e50
        num_processed: 0
      • num_success: 1
        description: Publishing files to web
        step_type: publish_directory
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: bbffeeca-459b-43f2-8046-176b2dc2f5b7
        num_processed: 1
      • num_success: 1
        description: Writing Listings File
        step_type: initialize_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: f8c57e0d-c34f-482d-a625-2f47e48b05ca
        num_processed: 1
      • num_success: 1
        description: Writing Listings File
        step_type: initialize_repo_metadata
        items_total: 1
        state: FINISHED
        error_details:
        details: ‘’
        num_failures: 0
        step_id: 79a7e563-5da4-4805-86b3-9fadaa4e9896
        num_processed: 1
        error:
        _id:
        “$oid”: 62608c215f4417acf01264cf
        id: 62608c215f4417acf01264cf
        poll_attempts:
        total: 7009
        failed: 0

**This is an example of Pulpchecking action: **

256: Actions::Pulp::Consumer::SyncCapsule (checking Pulp task status) [ 247913.26s / 34415.13s ] Cancel
Queue: default

Started at: 2022-04-19 12:15:40 UTC

Ended at: 2022-04-22 09:07:33 UTC

Real time: 247913.26s

Execution time (excluding suspended state): 34415.13s

Input:

capsule_id: 6
repo_pulp_id: 3-foreman2_3_Proxy-Spacewalk-c297a282-c900-491b-87ed-dba70af43020
repository_id: 1415
sync_options:
remove_missing: true
remote_user: admin
remote_cp_user: admin
current_request_id:
current_timezone: Europe/London
current_user_id: 4
current_organization_id: 3
current_location_id: 2
Output:

smart_proxy_history_id: 31461
pulp_tasks:

  • exception:
    task_type: pulp.server.managers.repo.sync.sync
    _href: “/pulp/api/v2/tasks/d4ccbe8b-e43d-4b59-896d-ce6d19d97ea2/”
    task_id: d4ccbe8b-e43d-4b59-896d-ce6d19d97ea2
    tags:
    • pulp:repository:3-foreman2_3_Proxy-Spacewalk-c297a282-c900-491b-87ed-dba70af43020
    • pulp:action:sync
      finish_time: ‘2022-04-21T03:45:25Z’
      _ns: task_status
      start_time: ‘2022-04-19T15:58:35Z’
      traceback:
      spawned_tasks:
    • _href: “/pulp/api/v2/tasks/bf8edaa7-565b-4001-92c8-7698661ee895/”
      task_id: bf8edaa7-565b-4001-92c8-7698661ee895
      progress_report:
      yum_importer:
      content:
      items_total: 3569
      state: FINISHED
      error_details:
      details:
      rpm_total: 3569
      rpm_done: 3569
      drpm_total: 0
      drpm_done: 0
      size_total: 22939796152
      size_left: 0
      items_left: 0
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      items_total: 0
      state: FINISHED
      error_details:
      items_left: 0
      modules:
      state: FINISHED
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      queue: reserved_resource_worker-11@ -foreman-proxy.inf.itsystems.global.dq2
      state: finished
      worker_name: reserved_resource_worker-11@ -foreman-proxy.inf.itsystems.global
      result:
      result: success
      importer_id: yum_importer
      exception:
      repo_id: 3-foreman2_3_Proxy-Spacewalk-c297a282-c900-491b-87ed-dba70af43020
      traceback:
      started: ‘2022-04-19T15:58:35Z’
      _ns: repo_sync_results
      completed: ‘2022-04-21T03:45:25Z’
      importer_type_id: yum_importer
      error_message:
      summary:
      modules:
      state: FINISHED
      content:
      state: FINISHED
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      state: FINISHED
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      added_count: 3569
      removed_count: 0
      updated_count: 0
      id: 6260d355d1c8df4cc2c37cc5
      details:
      modules:
      state: FINISHED
      content:
      size_total: 22939796152
      items_left: 0
      items_total: 3569
      state: FINISHED
      size_left: 0
      details:
      rpm_total: 3569
      rpm_done: 3569
      drpm_total: 0
      drpm_done: 0
      error_details:
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      items_total: 0
      state: FINISHED
      error_details:
      items_left: 0
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      error:
      _id:
      “$oid”: 625ea7ed5f4417acf07a6fb4
      id: 625ea7ed5f4417acf07a6fb4
  • exception:
    task_type: pulp.server.managers.repo.publish.publish
    _href: “/pulp/api/v2/tasks/bf8edaa7-565b-4001-92c8-7698661ee895/”
    task_id: bf8edaa7-565b-4001-92c8-7698661ee895
    tags:
    • pulp:repository:3-foreman2_3_Proxy-Spacewalk-c297a282-c900-491b-87ed-dba70af43020
    • pulp:action:publish
      finish_time:
      _ns: task_status
      start_time:
      traceback:
      spawned_tasks:
      progress_report: {}
      queue: ‘’
      state: waiting
      worker_name:
      result:
      error:
      _id:
      “$oid”: 6260d3555f4417acf0331ba4
      id: 6260d3555f4417acf0331ba4
      poll_attempts:
      total: 13317
      failed: 0

** This is Example of Pulp waiting to finish task**
Queue: default

Started at: 2022-04-19 12:15:42 UTC

Ended at: 2022-04-22 09:07:48 UTC

Real time: 247926.75s

Execution time (excluding suspended state): 32441.67s

Input:

capsule_id: 6
repo_pulp_id: 3-Centos6_content-Spacewalk-2a50be50-3052-4790-aa33-d70daff3574a
repository_id: 583
sync_options:
remove_missing: true
remote_user: admin
remote_cp_user: admin
current_request_id:
current_timezone: Europe/London
current_user_id: 4
current_organization_id: 3
current_location_id: 2
Output:

smart_proxy_history_id: 31474
pulp_tasks:

  • exception:
    task_type: pulp.server.managers.repo.sync.sync
    _href: “/pulp/api/v2/tasks/b29fb0ba-be46-40b3-a750-80cc7eaabd21/”
    task_id: b29fb0ba-be46-40b3-a750-80cc7eaabd21
    tags:
    • pulp:repository:3-Centos6_content-Spacewalk-2a50be50-3052-4790-aa33-d70daff3574a
    • pulp:action:sync
      finish_time: ‘2022-04-21T10:43:45Z’
      _ns: task_status
      start_time: ‘2022-04-20T21:06:48Z’
      traceback:
      spawned_tasks:
    • _href: “/pulp/api/v2/tasks/85590d90-a8a7-4a80-90e0-f6495128d512/”
      task_id: 85590d90-a8a7-4a80-90e0-f6495128d512
      progress_report:
      yum_importer:
      content:
      items_total: 7428
      state: FINISHED
      error_details:
      details:
      rpm_total: 7428
      rpm_done: 7428
      drpm_total: 0
      drpm_done: 0
      size_total: 7434395420
      size_left: 0
      items_left: 0
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      items_total: 4
      state: FINISHED
      error_details:
      items_left: 0
      modules:
      state: FINISHED
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      queue: reserved_resource_worker-2@ -foreman-proxy.inf.itsystems.global.dq2
      state: finished
      worker_name: reserved_resource_worker-2@ -foreman-proxy.inf.itsystems.global
      result:
      result: success
      importer_id: yum_importer
      exception:
      repo_id: 3-Centos6_content-Spacewalk-2a50be50-3052-4790-aa33-d70daff3574a
      traceback:
      started: ‘2022-04-20T21:06:48Z’
      _ns: repo_sync_results
      completed: ‘2022-04-21T10:43:45Z’
      importer_type_id: yum_importer
      error_message:
      summary:
      modules:
      state: FINISHED
      content:
      state: FINISHED
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      state: FINISHED
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      added_count: 7768
      removed_count: 0
      updated_count: 0
      id: 62613561d1c8df557360ffd4
      details:
      modules:
      state: FINISHED
      content:
      size_total: 7434395420
      items_left: 0
      items_total: 7428
      state: FINISHED
      size_left: 0
      details:
      rpm_total: 7428
      rpm_done: 7428
      drpm_total: 0
      drpm_done: 0
      error_details:
      comps:
      state: FINISHED
      purge_duplicates:
      state: FINISHED
      distribution:
      items_total: 4
      state: FINISHED
      error_details:
      items_left: 0
      errata:
      state: FINISHED
      metadata:
      state: FINISHED
      error:
      _id:
      “$oid”: 625ea7f05f4417acf07a7153
      id: 625ea7f05f4417acf07a7153
  • exception:
    task_type: pulp.server.managers.repo.publish.publish
    _href: “/pulp/api/v2/tasks/85590d90-a8a7-4a80-90e0-f6495128d512/”
    task_id: 85590d90-a8a7-4a80-90e0-f6495128d512
    tags:
    • pulp:repository:3-Centos6_content-Spacewalk-2a50be50-3052-4790-aa33-d70daff3574a
    • pulp:action:publish
      finish_time:
      _ns: task_status
      start_time:
      traceback:
      spawned_tasks:
      progress_report: {}
      queue: ‘’
      state: waiting
      worker_name:
      result:
      error:
      _id:
      “$oid”: 626135615f4417acf051c7d5
      id: 626135615f4417acf051c7d5
      poll_attempts:
      total: 13440
      failed: 0
36

We did not release pulp3 with katello untill Katello 3.18. I am not sure this is right.

37

Hi, I was surprised im running pulp 3 on foreman as It wasn’t intended.
however I’ve been told that I am. these are the detail from the about page.

image
image1318×307 19.4 KB

Installed Packages

  • candlepin-3.1.22-1.el7.noarch
  • candlepin-selinux-3.1.22-1.el7.noarch
  • foreman-2.3.5-1.el7.noarch
  • foreman-cli-2.3.5-1.el7.noarch
  • foreman-debug-2.3.5-1.el7.noarch
  • foreman-dynflow-sidekiq-2.3.5-1.el7.noarch
  • foreman-installer-2.3.5-1.el7.noarch
  • foreman-installer-katello-2.3.5-1.el7.noarch
  • foreman-postgresql-2.3.5-1.el7.noarch
  • foreman-proxy-2.3.5-1.el7.noarch
  • foreman-release-2.3.5-1.el7.noarch
  • foreman-selinux-2.3.5-1.el7.noarch
  • foreman-service-2.3.5-1.el7.noarch
  • -foreman.inf.itsystems.global-apache-1.0-1.noarch
  • -foreman.inf.itsystems.global-foreman-client-1.0-1.noarch
  • -foreman.inf.itsystems.global-foreman-proxy-1.0-1.noarch
  • -foreman.inf.itsystems.global-foreman-proxy-client-1.0-1.noarch
  • -foreman.inf.itsystems.global-puppet-client-1.0-1.noarch
  • -foreman.inf.itsystems.global-qpid-broker-1.0-1.noarch
  • -foreman.inf.itsystems.global-qpid-client-cert-1.0-1.noarch
  • -foreman.inf.itsystems.global-qpid-router-client-1.0-1.noarch
  • -foreman.inf.itsystems.global-qpid-router-server-1.0-1.noarch
  • katello-3.18.3-1.el7.noarch
  • katello-agent-3.5.4-1.el7.noarch
  • katello-ca-consumer-gbh-foreman.inf.itsystems.global-1.0-1.noarch
  • katello-certs-tools-2.7.3-1.el7.noarch
  • katello-client-bootstrap-1.7.5-1.el7.noarch
  • katello-common-3.18.3-1.el7.noarch
  • katello-debug-3.18.3-1.el7.noarch
  • katello-default-ca-1.0-1.noarch
  • katello-host-tools-3.5.4-1.el7.noarch
  • katello-host-tools-fact-plugin-3.5.4-1.el7.noarch
  • katello-repos-3.18.3-1.el7.noarch
  • katello-selinux-3.5.0-1.el7.noarch
  • katello-server-ca-1.0-1.noarch
  • pulp-admin-client-2.21.5-1.el7.noarch
  • pulp-client-1.0-1.noarch
  • pulp-deb-plugins-1.10.2-1.el7.noarch
  • pulp-docker-plugins-3.2.9-1.el7.noarch
  • pulp-katello-1.0.3-1.el7.noarch
  • pulp-puppet-plugins-2.21.5-1.el7.noarch
  • pulp-puppet-tools-2.21.5-1.el7.noarch
  • pulp-rpm-admin-extensions-2.21.5-1.el7.noarch
  • pulp-rpm-plugins-2.21.5-1.el7.noarch
  • pulp-selinux-2.21.5-1.el7.noarch
  • pulp-server-2.21.5-1.el7.noarch
  • pulpcore-selinux-1.2.3-2.el7.x86_6
38
  • katello-3.18.3-1.el7.noarch

I see this so it should be pulp3 which is right…The 3.14 threw me off a little…

I don’t know why the capsule syncs would take a day to finish…I can imagine some memory constraints. The access log errors don’t seem related to capsule syncs. I’d strongly suggest upgrading to later versions on both the server and the capsule.

39

Unfortunately I’m unable to upgrade at this time. My 4 other proxies are fine it just this one.
I didnt know if there were some proxy side tuning settings that could be used or a pulp database clear out of proxy. Could the pulp database be corrupt ?

40

There don’t seem to be any errors on the capule sync itself…I’d check the memory on the capsule. A full capsule sync should fix any corrupt data although the syncs taking a day to finish make that really cumbersome in this case.

I’d maybe empty the capsule by disassociating all environments from the capsule and running delete orphaned content task against the capsule to empty it out. It will need a full sync after reassociating environments to the capsule once that is done.

1 Like
41

@sinewave How much internet bandwidth do you have? If we’re talking about syncing RHEL 7, that repo is ~60gb, it wouldn’t be too surprising for it to take a really long time over a slow proxy.