State: Waiting for pulp to start the task

Support
1

Problem:
Repo sync or content-view publishing are taking 2 days to complete the task (Refer the attached screen shot.

Expected outcome:
Task should complete within 30 mins.

Foreman and Proxy versions:
foreman-proxy-1.14.3-1.el7.noarch

Foreman and Proxy plugin versions:
candlepin-0.9.54.10-1.el7.noarch
candlepin-common-1.0.23-1.el7.noarch
candlepin-selinux-0.9.54.10-1.el7.noarch
foreman-1.14.3-1.el7.noarch
foreman-compute-1.14.3-1.el7.noarch
foreman-debug-1.14.3-1.el7.noarch
foreman-gce-1.14.3-1.el7.noarch
foreman-installer-1.14.3-1.el7.noarch
foreman-installer-katello-3.3.1.4-1.el7.noarch
foreman-libvirt-1.14.3-1.el7.noarch
foreman-ovirt-1.14.3-1.el7.noarch
foreman-postgresql-1.14.3-1.el7.noarch
foreman-proxy-1.14.3-1.el7.noarch
foreman-release-1.14.3-1.el7.noarch
foreman-release-scl-3-1.el7.noarch
foreman-selinux-1.14.3-1.el7.noarch
foreman-vmware-1.14.3-1.el7.noarch
katello-3.3.2-1.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-client-bootstrap-1.2.2-1.el7.noarch
katello-common-3.3.2-1.el7.noarch
katello-debug-3.3.2-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-installer-base-3.3.1.4-1.el7.noarch
katello-repos-3.3.1-1.el7.noarch
katello-selinux-3.0.2-1.el7.noarch
katello-server-ca-1.0-11.noarch
katello-service-3.3.2-1.el7.noarch
pulp-admin-client-2.10.3-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-docker-plugins-2.1.0-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
pulp-puppet-plugins-2.10.3-1.el7.noarch
pulp-puppet-tools-2.10.3-1.el7.noarch
pulp-rpm-admin-extensions-2.10.3-1.el7.noarch
pulp-rpm-handlers-2.10.3-1.el7.noarch
pulp-rpm-plugins-2.10.3-1.el7.noarch
pulp-selinux-2.10.3-1.el7.noarch
pulp-server-2.10.3-1.el7.noarch
python-gofer-qpid-2.7.6-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-agent-lib-2.10.3-1.el7.noarch
python-pulp-bindings-2.10.3-1.el7.noarch
python-pulp-client-lib-2.10.3-1.el7.noarch
python-pulp-common-2.10.3-1.el7.noarch
python-pulp-docker-common-2.1.0-1.el7.noarch
python-pulp-oid_validation-2.10.3-1.el7.noarch
python-pulp-puppet-common-2.10.3-1.el7.noarch
python-pulp-repoauth-2.10.3-1.el7.noarch
python-pulp-rpm-common-2.10.3-1.el7.noarch
python-pulp-streamer-2.10.3-1.el7.noarch
python-qpid-1.36.0-1.el7.noarch
python-qpid-proton-0.17.0-3.el7.x86_64
python-qpid-qmf-1.36.0-1.el7.x86_64
python-urllib3-1.10.2-2.katello.el7.noarch
qpid-cpp-client-1.36.0-1.el7.x86_64
qpid-cpp-client-devel-1.36.0-1.el7.x86_64
qpid-cpp-server-1.36.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.36.0-1.el7.x86_64
qpid-dispatch-router-0.7.0-1.el7.x86_64
qpid-proton-c-0.17.0-3.el7.x86_64
qpid-qmf-1.36.0-1.el7.x86_64
qpid-tools-1.36.0-1.el7.noarch
rhss-2.eng.velocix.com-foreman-client-1.0-6.noarch
rhss-2.eng.velocix.com-foreman-proxy-1.0-7.noarch
rhss-2.eng.velocix.com-foreman-proxy-client-1.0-6.noarch
rhss-2.eng.velocix.com-qpid-broker-1.0-7.noarch
rhss-2.eng.velocix.com-qpid-client-cert-1.0-6.noarch
rhss-2.eng.velocix.com-qpid-router-client-1.0-6.noarch
rhss-2.eng.velocix.com-qpid-router-server-1.0-6.noarch
rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch
tfm-rubygem-activerecord-session_store-0.1.2-2.el7.noarch
tfm-rubygem-addressable-2.3.6-3.el7.noarch
tfm-rubygem-algebrick-0.7.3-4.el7.noarch
tfm-rubygem-ancestry-2.2.1-1.el7.noarch
tfm-rubygem-anemone-0.7.2-14.el7.noarch
tfm-rubygem-angular-rails-templates-0.1.2-5.el7.noarch
tfm-rubygem-apipie-bindings-0.0.18-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-1.el7.noarch
tfm-rubygem-apipie-rails-0.3.7-1.el7.noarch
tfm-rubygem-audited-4.2.0-1.el7.noarch
tfm-rubygem-audited-activerecord-4.2.0-2.el7.noarch
tfm-rubygem-autoparse-0.3.3-5.el7.noarch
tfm-rubygem-awesome_print-1.7.0-1.el7.noarch
tfm-rubygem-bastion-4.2.2-1.fm1_14.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-1.el7.noarch
tfm-rubygem-clamp-1.0.0-5.el7.noarch
tfm-rubygem-colorize-0.7.7-4.el7.noarch
tfm-rubygem-commonjs-0.2.7-4.el7.noarch
tfm-rubygem-concurrent-ruby-1.0.0-1.fm1_10.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.2.0-4.fm1_12.el7.noarch
tfm-rubygem-css_parser-1.3.7-1.el7.noarch
tfm-rubygem-daemons-1.2.3-3.el7.noarch
tfm-rubygem-deacon-1.0.0-1.el7.noarch
tfm-rubygem-deep_cloneable-2.2.2-1.el7.noarch
tfm-rubygem-deface-1.0.2-3.el7.noarch
tfm-rubygem-docker-api-1.28.0-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-1.el7.noarch
tfm-rubygem-dynflow-0.8.17-1.fm1_14.el7.noarch
tfm-rubygem-excon-0.51.0-1.el7.noarch
tfm-rubygem-extlib-0.9.16-4.el7.noarch
tfm-rubygem-facter-2.4.0-3.el7.x86_64
tfm-rubygem-faraday-0.9.1-3.el7.noarch
tfm-rubygem-fast_gettext-1.1.0-1.el7.noarch
tfm-rubygem-fog-1.38.0-1.el7.noarch
tfm-rubygem-fog-aws-0.13.0-1.el7.noarch
tfm-rubygem-fog-core-1.42.0-1.el7.noarch
tfm-rubygem-fog-google-0.1.0-2.el7.noarch
tfm-rubygem-fog-json-1.0.2-4.el7.noarch
tfm-rubygem-fog-libvirt-0.2.0-1.el7.noarch
tfm-rubygem-fog-openstack-0.1.12-1.el7.noarch
tfm-rubygem-fog-rackspace-0.1.1-1.el7.noarch
tfm-rubygem-fog-vsphere-1.7.0-1.el7.noarch
tfm-rubygem-fog-xenserver-0.2.3-1.el7.noarch
tfm-rubygem-fog-xml-0.1.2-4.el7.noarch
tfm-rubygem-foreman-tasks-0.8.6-1.fm1_14.el7.noarch
tfm-rubygem-foreman-tasks-core-0.1.1-1.fm1_14.el7.noarch
tfm-rubygem-foreman_bootdisk-9.0.0-1.fm1_14.el7.noarch
tfm-rubygem-foreman_discovery-8.0.1-1.fm1_14.el7.noarch
tfm-rubygem-foreman_docker-3.0.0-1.fm1_13.el7.noarch
tfm-rubygem-foreman_hooks-0.3.14-1.fm1_14.el7.noarch
tfm-rubygem-formatador-0.2.1-9.el7.noarch
tfm-rubygem-friendly_id-5.1.0-3.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.2.1-3.el7.noarch
tfm-rubygem-google-api-client-0.8.2-5.el7.noarch
tfm-rubygem-hammer_cli-0.8.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.8.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-5.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.4-2.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.10-2.fm1_12.el7.noarch
tfm-rubygem-hammer_cli_import-0.11.3-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.3.0-1.el7.noarch
tfm-rubygem-hashie-2.0.5-4.el7.noarch
tfm-rubygem-hashr-0.0.22-7.el7.noarch
tfm-rubygem-highline-1.6.21-5.el7.noarch
tfm-rubygem-hooks-0.2.2-7.el7.noarch
tfm-rubygem-http-cookie-1.0.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-9.el7.noarch
tfm-rubygem-jquery-ui-rails-4.1.2-5.el7.noarch
tfm-rubygem-jwt-1.2.0-4.el7.noarch
tfm-rubygem-katello-3.3.2-1.el7.noarch
tfm-rubygem-launchy-2.4.3-4.el7.noarch
tfm-rubygem-ldap_fluff-0.4.4-1.el7.noarch
tfm-rubygem-little-plugger-1.1.3-21.el7.noarch
tfm-rubygem-locale-2.0.9-11.el7.noarch
tfm-rubygem-logging-1.8.2-4.el7.noarch
tfm-rubygem-multipart-post-1.2.0-4.el7.noarch
tfm-rubygem-net-ldap-0.15.0-1.el7.noarch
tfm-rubygem-net-scp-1.1.0-6.el7.noarch
tfm-rubygem-net-ssh-3.0.2-2.el7.noarch
tfm-rubygem-netrc-0.7.7-7.el7.noarch
tfm-rubygem-oauth-0.4.7-8.el7.noarch
tfm-rubygem-parse-cron-0.1.4-2.fm1_11.el7.noarch
tfm-rubygem-passenger-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64
tfm-rubygem-pg-0.15.1-3.el7.x86_64
tfm-rubygem-powerbar-1.0.17-1.el7.noarch
tfm-rubygem-qpid_messaging-0.34.1-1.el7.x86_64
tfm-rubygem-rabl-0.12.0-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-5.el7.noarch
tfm-rubygem-rails-i18n-4.0.9-1.el7.noarch
tfm-rubygem-rails-observers-0.1.2-7.el7.noarch
tfm-rubygem-rbovirt-0.1.2-1.el7.noarch
tfm-rubygem-rbvmomi-1.9.4-1.el7.noarch
tfm-rubygem-responders-2.3.0-1.el7.noarch
tfm-rubygem-rest-client-1.8.0-1.el7.noarch
tfm-rubygem-retriable-1.4.1-4.el7.noarch
tfm-rubygem-roadie-3.1.1-1.el7.noarch
tfm-rubygem-roadie-rails-1.1.1-1.el7.noarch
tfm-rubygem-robotex-1.0.0-19.el7.noarch
tfm-rubygem-ruby-libvirt-0.5.2-4.el7.x86_64
tfm-rubygem-ruby2ruby-2.1.3-4.el7.noarch
tfm-rubygem-ruby_parser-3.6.3-4.el7.noarch
tfm-rubygem-runcible-1.9.1-1.el7.noarch
tfm-rubygem-safemode-1.2.4-2.el7.noarch
tfm-rubygem-scoped_search-3.3.0-1.el7.noarch
tfm-rubygem-secure_headers-3.4.1-1.el7.noarch
tfm-rubygem-sequel-4.20.0-5.el7.noarch
tfm-rubygem-sexp_processor-4.4.4-3.el7.noarch
tfm-rubygem-signet-0.6.0-5.el7.noarch
tfm-rubygem-sshkey-1.6.0-4.el7.noarch
tfm-rubygem-strong_parameters-0.2.1-13.el7.noarch
tfm-rubygem-table_print-1.5.1-6.el7.noarch
tfm-rubygem-trollop-2.1.2-1.el7.noarch
tfm-rubygem-unf-0.1.3-5.el7.noarch
tfm-rubygem-unf_ext-0.0.6-6.el7.x86_64
tfm-rubygem-useragent-0.16.8-1.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-3.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-1.el7.noarch
tfm-rubygem-wicked-1.1.0-4.el7.noarch
tfm-rubygem-will_paginate-3.1.5-1.el7.noarch
tfm-rubygem-x-editable-rails-1.5.5-1.el7.noarch
tfm-runtime-3.2-5.el7.x86_64

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)

logs

I tried the following troubleshooting steps, but still the issue is not resolved.

ForemanTasks::Task.where(:state => :running).where(:label => "Actions::Katello::Repository::CapsuleGenerateAndSync").destroy_all

foreman-rake katello:reimport
Successfully encrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully encrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Importing Katello::Erratum
Importing Katello::PackageGroup
Importing Katello::PuppetModule
Importing Katello::Rpm
Importing Katello::FileUnit
Importing Katello::Subscription
Importing Katello::Pool
Importing Katello::DockerManifest
Importing Katello::DockerTag
Importing Katello::ContentViewPuppetEnvironment
Importing Activation Key Subscriptions

katello-service restart --exclude httpd
Redirecting to /bin/systemctl stop  foreman-tasks.service

Redirecting to /bin/systemctl stop  pulp_celerybeat.service

Redirecting to /bin/systemctl stop  foreman-proxy.service

Redirecting to /bin/systemctl stop  pulp_streamer.service

Redirecting to /bin/systemctl stop  pulp_resource_manager.service

Redirecting to /bin/systemctl stop  pulp_workers.service

Redirecting to /bin/systemctl stop  tomcat.service

Redirecting to /bin/systemctl stop  squid.service

Redirecting to /bin/systemctl stop  qdrouterd.service

Redirecting to /bin/systemctl stop  qpidd.service

Redirecting to /bin/systemctl stop  postgresql.service

Redirecting to /bin/systemctl stop  mongod.service

Success!
Redirecting to /bin/systemctl start  mongod.service

Redirecting to /bin/systemctl start  postgresql.service

Redirecting to /bin/systemctl start  qpidd.service

Redirecting to /bin/systemctl start  qdrouterd.service

Redirecting to /bin/systemctl start  squid.service

Redirecting to /bin/systemctl start  tomcat.service

Redirecting to /bin/systemctl start  pulp_workers.service

Redirecting to /bin/systemctl start  pulp_resource_manager.service

Redirecting to /bin/systemctl start  pulp_streamer.service

Redirecting to /bin/systemctl start  foreman-proxy.service

Redirecting to /bin/systemctl start  pulp_celerybeat.service

Redirecting to /bin/systemctl start  foreman-tasks.service

Success!
```![error|690x455](upload://3EbNvUGt8aY53RlFvYtEgbjdnXx.png)
2

Increased PULP_CONCURRENCY from 10 to 15 and katello service restarted. But still the issue is not resolved.

3

Could you grep /var/log/messages on the capsule for any possible errors around the task.

4

No errors found in message log.

I could see the below entry in Dynflow console with delayed_executor"=true.

01ee9b70-efdd-40b1-ba24-87f00f18c861 {"hostname"=>"satellite-server.local", "pid"=>48923, "delayed_executor"=>true}

5

Waiting for pulp issue not only for capsule sync, facing this issue even on repo sync and CV publish.

Errors from capsule server /var/log/message.

Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072) Task pulp.server.managers.content.orphan.delete_all_orphans[54878059-2ffd-4300-90b8-1efdbb4ec58c] raised unexpected: CursorNotFound('Cursor not found, cursor id: 170074440498',)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072) Traceback (most recent call last):
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     R = retval = fun(*args, **kwargs)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     return super(Task, self).__call__(*args, **kwargs)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     return super(PulpTask, self).__call__(*args, **kwargs)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     return self.run(*args, **kwargs)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 170, in delete_all_orphans
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     OrphanManager.delete_orphans_by_type(content_type_id)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 219, in delete_orphans_by_type
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     fields=['_id', '_storage_path']):
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 106, in generate_orphans_by_type
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     for content_unit in content_units_collection.find({}, projection=fields):
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     if len(self.__data) or self._refresh():
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     self.__max_await_time_ms))
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     codec_options=self.__codec_options)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072)     raise CursorNotFound(msg, 43, errobj)
Jan 26 03:58:25 capsule pulp: celery.worker.job:ERROR: (2821-55072) CursorNotFound: Cursor not found, cursor id: 170074440498
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008) Task pulp.server.managers.content.orphan.delete_all_orphans[0e46d093-ccde-45f7-8a60-c7d6726d3c84] raised unexpected: CursorNotFound('Cursor not found, cursor id: 168392406785',)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008) Traceback (most recent call last):
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     R = retval = fun(*args, **kwargs)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     return super(Task, self).__call__(*args, **kwargs)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     return super(PulpTask, self).__call__(*args, **kwargs)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     return self.run(*args, **kwargs)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 170, in delete_all_orphans
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     OrphanManager.delete_orphans_by_type(content_type_id)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 219, in delete_orphans_by_type
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     fields=['_id', '_storage_path']):
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 106, in generate_orphans_by_type
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     for content_unit in content_units_collection.find({}, projection=fields):
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     if len(self.__data) or self._refresh():
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     self.__max_await_time_ms))
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     codec_options=self.__codec_options)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008)     raise CursorNotFound(msg, 43, errobj)
Jan 26 03:58:27 capsule pulp: celery.worker.job:ERROR: (2831-19008) CursorNotFound: Cursor not found, cursor id: 168392406785
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760) Task pulp.server.managers.content.orphan.delete_all_orphans[ef6113e5-643f-4446-8277-197653049c48] raised unexpected: CursorNotFound('Cursor not found, cursor id: 169675950902',)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760) Traceback (most recent call last):
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     R = retval = fun(*args, **kwargs)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     return super(Task, self).__call__(*args, **kwargs)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     return super(PulpTask, self).__call__(*args, **kwargs)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     return self.run(*args, **kwargs)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 170, in delete_all_orphans
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     OrphanManager.delete_orphans_by_type(content_type_id)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 219, in delete_orphans_by_type
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     fields=['_id', '_storage_path']):
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 106, in generate_orphans_by_type
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     for content_unit in content_units_collection.find({}, projection=fields):
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     if len(self.__data) or self._refresh():
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     self.__max_await_time_ms))
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     codec_options=self.__codec_options)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760)     raise CursorNotFound(msg, 43, errobj)
Jan 26 03:59:12 capsule pulp: celery.worker.job:ERROR: (2833-69760) CursorNotFound: Cursor not found, cursor id: 169675950902
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584) Task pulp.server.managers.content.orphan.delete_all_orphans[46577d27-7c42-4811-bfd2-deab17012c45] raised unexpected: CursorNotFound('Cursor not found, cursor id: 169610477887',)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584) Traceback (most recent call last):
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     R = retval = fun(*args, **kwargs)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     return super(Task, self).__call__(*args, **kwargs)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     return super(PulpTask, self).__call__(*args, **kwargs)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     return self.run(*args, **kwargs)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 170, in delete_all_orphans
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     OrphanManager.delete_orphans_by_type(content_type_id)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 219, in delete_orphans_by_type
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     fields=['_id', '_storage_path']):
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 106, in generate_orphans_by_type
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     for content_unit in content_units_collection.find({}, projection=fields):
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     if len(self.__data) or self._refresh():
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     self.__max_await_time_ms))
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     codec_options=self.__codec_options)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584)     raise CursorNotFound(msg, 43, errobj)
Jan 26 03:59:13 capsule pulp: celery.worker.job:ERROR: (2835-35584) CursorNotFound: Cursor not found, cursor id: 169610477887
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528) Task pulp.server.managers.content.orphan.delete_all_orphans[7227e05a-549f-4de9-a52e-22384cb55090] raised unexpected: CursorNotFound('Cursor not found, cursor id: 170968759404',)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528) Traceback (most recent call last):
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     R = retval = fun(*args, **kwargs)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     return super(Task, self).__call__(*args, **kwargs)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     return super(PulpTask, self).__call__(*args, **kwargs)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     return self.run(*args, **kwargs)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 170, in delete_all_orphans
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     OrphanManager.delete_orphans_by_type(content_type_id)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 219, in delete_orphans_by_type
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     fields=['_id', '_storage_path']):
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 106, in generate_orphans_by_type
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     for content_unit in content_units_collection.find({}, projection=fields):
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     if len(self.__data) or self._refresh():
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     self.__max_await_time_ms))
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     codec_options=self.__codec_options)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528)     raise CursorNotFound(msg, 43, errobj)
Jan 26 03:59:22 capsule pulp: celery.worker.job:ERROR: (2825-02528) CursorNotFound: Cursor not found, cursor id: 170968759404
6

@rengan : Looking at the logs, I see you are hitting the Cursor timeout in mongo which is fixed in https://pulp.plan.io/issues/2584 . Applying the patch should resolve this.