Katello 2.2RC Repository sync gets stuck at 50% after downloading packages

Nathan_Peters · April 22, 2015, 4:01pm

I'm attempting to sync a repository for the first time (Fedora Base - 30GB).

It downloaded all of the packages (35656) except 85. After downloading
them all, it just got stuck at 50% and has been running for over 16 hours.

To try to debug this I had to go into the dynflow console.

Step 3 went into suspended state, but no option to resume is given. My
only option was to cancel. After hitting cancel, and then leaving and
re-entering the console, step 3 is listed as success.

What logs should I post to troubleshoot why this is getting stuck? I'm
still having this types of problems intermittently in this RC so this
problem does not appear to be completely fixed since 2.1.

I have shown the before I hit cancel and after I hit cancel as comparison

------ before ------

3: Actions::Pulp::Repository::Sync (suspended) [ 77681.71s / 574.43s ]
Cancel

Started at: 2015-04-21 18:07:37 UTC

Ended at: 2015-04-22 15:42:18 UTC

Real time: 77681.71s

Execution time (excluding suspended state): 574.43s

Input:

···

--- pulp_id: My_Company-Fedora_20-Fedora_20_-_x86_64_-_Base task_id: remote_user: admin-375a9869 remote_cp_user: admin locale: en

Output:

pulp_tasks:

exception:
task_type: pulp.server.managers.repo.sync.sync
_href: /pulp/api/v2/tasks/ba20b1a3-6ae8-4e88-b9b5-c9327bc4168b/
task_id: ba20b1a3-6ae8-4e88-b9b5-c9327bc4168b
tags:
- pulp:repository:My_Company-Fedora_20-Fedora_20_-x86_64-_Base
- pulp:action:sync
  finish_time:
  _ns: task_status
  start_time: ‘2015-04-21T18:07:37Z’
  traceback:
  spawned_tasks: []
  progress_report:
  yum_importer:
  content:
  size_total: 34708806576
  items_left: 0
  items_total: 34656
  state: FINISHED
  size_left: 0
  details:
  rpm_total: 34656
  rpm_done: 34656
  drpm_total: 0
  drpm_done: 0
  error_details:
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/e/elfutils-devel-static-0.157-1.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-ajaxian-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-albumselect-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-archiveupload-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-captcha-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-carbon-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-cart-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-classic-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-colorpack-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-comment-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-customfield-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-dcraw-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-debug-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-digibug-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-dynamicalbum-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-ecard-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-exif-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-ffmpeg-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-flashvideo-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-floatrix-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-fotokasten-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-gd-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-getid3-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-hidden-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-httpauth-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-hybrid-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-icons-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-imageblock-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-imageframe-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-imagemagick-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gallery2-itemadd-2.3.2-10.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gadget-0.0.3-10.fc19.noarch.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/ghc-ghc-paths-0.1.0.9-2.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/ghc-mwc-random-0.12.0.1-5.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gloox-devel-1.0.3-1.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/glpk-doc-4.52.1-2.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/glpk-4.52.1-2.fc20.i686.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/glpk-static-4.52.1-2.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/glpk-utils-4.52.1-2.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/glue-validator-1.0.2-7.fc20.noarch.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/glue-schema-2.0.10-2.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gnet2-2.0.8-9.fc20.i686.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/g/gtatool-pcd-1.5.2-8.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/j/java-1.8.0-openjdk-javadoc-1.8.0.0-0.19.b106.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/j/java-1.8.0-openjdk-devel-1.8.0.0-0.19.b106.fc20.x86_64.rpm
  errors:
  - Service Temporarily Unavailable
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/j/javacc-manual-5.0-9.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/j/java-1.8.0-openjdk-src-1.8.0.0-0.19.b106.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/k/knm-new-fixed-fonts-1.1-19.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/l/laughlin-backgrounds-single-14.1.0-9.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/l/libcutl-devel-1.7.1-2.fc20.i686.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/l/libdwarves1-devel-1.10-4.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/l/libmatroska-devel-1.4.1-1.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/m/maven-local-3.4.1-1.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/m/mercator-0.3.2-3.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/m/moodbar-0.1.2-11.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/m/mqtt-client-1.5-3.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/o/opensaml-java-openws-javadoc-1.4.4-6.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/o/openscap-perl-0.9.12-2.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/palapeli-devel-4.11.3-1.fc20.i686.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/perl-Test-NoWarnings-1.04-4.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/perl-XML-Parser-2.41-11.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/pharosc-synopsys-8.3-8.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/plplot-qt-5.9.9-19.svn12479.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/poppler-utils-0.24.3-2.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/popt-static-1.13-15.fc20.i686.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/pony-0.4-8.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/postgresql-jdbc-javadoc-9.2.1002-4.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, error(104, ‘Connection reset by peer’))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/python-fedora-turbogears2-0.3.32.3-3.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/python-peewee-2.1.4-2.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/p/python-rosdep-0.10.21-1.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/r/rubygem-aruba-doc-0.5.2-2.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/s/skyviewer-1.0.0-13.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/s/speedcrunch-0.11-0.7.alpha.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/s/sxiv-1.1.1-2.fc20.x86_64.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Packages/t/tesseract-langpack-mal-3.02-7.fc20.noarch.rpm
  errors:
  - (‘Connection aborted.’, BadStatusLine("’’",))
  - url:
  http://download.fedoraproject.org/pub/fedora/linux/releases/20/Everything/x86_64/os/Package

Nathan_Peters · April 22, 2015, 4:22pm

/var/log/foreman/production.log at the time the repo sync finished. It
appears to finish fine other than the errors about the packages it could
not download. After that, the last couple lines I posted just repeat until
I log in this morning.

There are no obvious errors stating why it went into suspended mode
immediately after downloading the packages.

Nathan_Peters · April 22, 2015, 4:33pm

Relevant log entries from /var/log/messages :

Nathan_Peters · April 22, 2015, 4:41pm

More issues with this job: After resuming the job by cancelling it (which
doesn't make much sense but anyway…) It went through the next couple of
steps successfully.

5: Actions::ElasticSearch::Repository::IndexContent (success) [ 854.52s /
854.52s ]
15: Actions::Katello::Repository::ErrataMail (success) [ 0.12s / 0.12s ]
17: Actions::Katello::Repository::Sync (success) [ 0.58s / 0.58s ]

Then it hit the step after that and after 6 seconds of processing went into
suspend mode :

20: Actions::Pulp::Repository::RegenerateApplicability (suspended) [
2250.60s / 6.82s ] Cancel

Started at: 2015-04-22 15:56:53 UTC

Ended at: 2015-04-22 16:34:24 UTC

Real time: 2250.60s

Execution time (excluding suspended state): 6.82s

Input:

···

--- pulp_id: My_Company-Fedora_20-Fedora_20_-_x86_64_-_Base remote_user: admin-375a9869 remote_cp_user: admin locale: en

Output:

pulp_tasks:

exception:
task_type:
pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos
_href: /pulp/api/v2/tasks/988267f8-e6c3-42b0-b050-7647bf3249c7/
task_id: 988267f8-e6c3-42b0-b050-7647bf3249c7
tags:
- pulp:action:content_applicability_regeneration
  finish_time:
  start_time:
  traceback:
  spawned_tasks: []
  progress_report: {}
  queue: reserved_resource_worker-0@katello.dq
  state: waiting
  worker_name: reserved_resource_worker-0@katello
  result:
  error:
  _id:
  $oid: 5537c4c5816ecb0590a05328
  id: 5537c4c8229dcf0c7208fcf0
  poll_attempts:
  total: 160
  failed: 0
  If I open the dynflow console and look there is another task created at the
  exact time time as this one started that is also suspended and only has the
  option to cancel, and not resume :The one above that froze up is called :

b3551189-719b-4590-8fda-31bc7ce12724 Actions::Katello::Repository::Sync
running pending 2015-04-21 18:07:37 UTC

The new one below that also froze at the exact same time is called:

f65582d8-2b32-4163-bd00-21757319ce16
Actions::Katello::Repository::NodeMetadataGenerate running pending
2015-04-22 15:56:53 UTC Show

3: Actions::Pulp::Repository::DistributorPublish (suspended) [ 2444.53s /
7.39s ] Cancel

Started at: 2015-04-22 15:56:54 UTC

Ended at: 2015-04-22 16:37:38 UTC

Real time: 2444.53s

Execution time (excluding suspended state): 7.39s

Input:
— pulp_id: My_Company-Fedora_20-Fedora_20_-x86_64-_Base dependency:
distributor_type_id: nodes_http_distributor remote_user: admin-375a9869
remote_cp_user: admin locale: en

Output:
— pulp_tasks: - exception: task_type:
pulp.server.managers.repo.publish.publish href:
/pulp/api/v2/tasks/38d41dce-a026-49df-9a2c-ff2aae216211/ task_id:
38d41dce-a026-49df-9a2c-ff2aae216211 tags: -
pulp:repository:My_Company-Fedora_20-Fedora_20-x86_64-_Base -
pulp:action:publish finish_time: start_time: traceback: spawned_tasks: []
progress_report: {} queue: reserved_resource_worker-1@katello.dq state:
waiting worker_name: reserved_resource_worker-1@katello result: error: _id:
$oid: 5537c4c6816ecb0590a05329 id: 5537c4c8229dcf0c7208fcf7 poll_attempts:
total: 172 failed: 0

Nathan_Peters · April 22, 2015, 4:44pm

/var/log/messages at the time of the second freeze :

Apr 22 15:56:54 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[dbf55cea-70bb-4588-bbf6-cd5e119ae778]
Apr 22 15:56:54 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[dc407153-cdd3-4cfd-8bdc-2667620732d5]
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos[988267f8-e6c3-42b0-b050-7647bf3249c7]
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._release_resource[6b753300-a3d3-4575-ad70-4e43d699bdb6]
Apr 22 15:56:56 katello pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[dbf55cea-70bb-4588-bbf6-cd5e119ae778]
succeeded in 1.40132945299s: None
Apr 22 15:56:56 katello pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[dc407153-cdd3-4cfd-8bdc-2667620732d5]
succeeded in 0.0270361080184s: None
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.managers.repo.publish.publish[38d41dce-a026-49df-9a2c-ff2aae216211]
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._release_resource[93438e5b-1f6b-4325-979b-f4f7106a652b]

/var/log/foreman/production.log at the time of the 2nd freeze:

2015-04-22 15:56:03 [I] Connected to server.
2015-04-22 15:56:04 [I] Client connected.
2015-04-22 15:56:53 [W] Failed to send email notification
katello_sync_errata: No recipients found for Fedora 20 - x86_64 - Base sync
report

ehelms · April 22, 2015, 5:56pm

TLDR; This is a result of not being able to communicate with the Capsule.
We are aware that when this occurs this halts the ability to sync (and has
an impact on repository creation and updates) and are working on testing
some fixes for it. To unstick it, the connection between the Pulp node and
server need to be established, then katello-services restarted so that the
task can be resumed.

Eric

···

On Wed, Apr 22, 2015 at 12:44 PM, Nathan Peters wrote:

/var/log/messages at the time of the second freeze :

Apr 22 15:56:54 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[dbf55cea-70bb-4588-bbf6-cd5e119ae778]
Apr 22 15:56:54 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._queue_reserved_task[dc407153-cdd3-4cfd-8bdc-2667620732d5]
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos[988267f8-e6c3-42b0-b050-7647bf3249c7]
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._release_resource[6b753300-a3d3-4575-ad70-4e43d699bdb6]
Apr 22 15:56:56 katello pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[dbf55cea-70bb-4588-bbf6-cd5e119ae778]
succeeded in 1.40132945299s: None
Apr 22 15:56:56 katello pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[dc407153-cdd3-4cfd-8bdc-2667620732d5]
succeeded in 0.0270361080184s: None
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.managers.repo.publish.publish[38d41dce-a026-49df-9a2c-ff2aae216211]
Apr 22 15:56:56 katello pulp: celery.worker.strategy:INFO: Received task:
pulp.server.async.tasks._release_resource[93438e5b-1f6b-4325-979b-f4f7106a652b]

/var/log/foreman/production.log at the time of the 2nd freeze:

2015-04-22 15:56:03 [I] Connected to server.
2015-04-22 15:56:04 [I] Client connected.
2015-04-22 15:56:53 [W] Failed to send email notification
katello_sync_errata: No recipients found for Fedora 20 - x86_64 - Base sync
report

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Nathan_Peters · April 22, 2015, 6:28pm

This applies to single server setups too? I don't have any external
capsules, it's all on one system.

···

On Wednesday, April 22, 2015 at 10:56:11 AM UTC-7, Eric Helms wrote: > > TLDR; This is a result of not being able to communicate with the Capsule. > We are aware that when this occurs this halts the ability to sync (and has > an impact on repository creation and updates) and are working on testing > some fixes for it. To unstick it, the connection between the Pulp node and > server need to be established, then katello-services restarted so that the > task can be resumed. > > >

ehelms · April 22, 2015, 8:13pm

Ahh, then this looks like another issue. Looking at the errors and the task
output closer I am guessing that it entered a suspended state due to the
how long the sync was taking (due to the issues syncing). Did you see
errors with the syncing after you cancelled and tried again?

···

On Wed, Apr 22, 2015 at 2:28 PM, Nathan Peters wrote:

This applies to single server setups too? I don’t have any external
capsules, it’s all on one system.

On Wednesday, April 22, 2015 at 10:56:11 AM UTC-7, Eric Helms wrote:

TLDR; This is a result of not being able to communicate with the Capsule.
We are aware that when this occurs this halts the ability to sync (and has
an impact on repository creation and updates) and are working on testing
some fixes for it. To unstick it, the connection between the Pulp node and
server need to be established, then katello-services restarted so that the
task can be resumed.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Nathan_Peters · April 22, 2015, 9:10pm

Yes, there are definitely still issues happening. After this I tried to
sync a smaller repo (4.3GB DVD image) and that failed also. I did a
"katello-service restart" and then tried to publish a view. That failed
also. It would fail after like 6 seconds of processing so I restarted the
service and tried again.

After restarting katello-service 2 or 3 times I was finally able to sync a
repo and publish a view, but a single restart did not fix it.

It seems to be extremely flaky and fails about 50% of the time on any
operation I do, no matter how small or fast. I can see no correlation
between the run time and the chance to fail or timeout.

···

On Wednesday, April 22, 2015 at 1:13:49 PM UTC-7, Eric Helms wrote: > > Ahh, then this looks like another issue. Looking at the errors and the > task output closer I am guessing that it entered a suspended state due to > the how long the sync was taking (due to the issues syncing). Did you see > errors with the syncing after you cancelled and tried again? > > On Wed, Apr 22, 2015 at 2:28 PM, Nathan Peters > wrote: > >> This applies to single server setups too? I don't have any external >> capsules, it's all on one system. >> >> On Wednesday, April 22, 2015 at 10:56:11 AM UTC-7, Eric Helms wrote: >>> >>> TLDR; This is a result of not being able to communicate with the >>> Capsule. We are aware that when this occurs this halts the ability to sync >>> (and has an impact on repository creation and updates) and are working on >>> testing some fixes for it. To unstick it, the connection between the Pulp >>> node and server need to be established, then katello-services restarted so >>> that the task can be resumed. >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-user...@googlegroups.com . >> To post to this group, send email to forema...@googlegroups.com >> . >> Visit this group at http://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > >