Problem:
In my system logs, I am seeing messages like “pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running”. I want to find out why that is happening and how to solve this.
Expected outcome:
Pulp should run fine.
Foreman and Proxy versions:
foreman-1.15
Foreman and Proxy plugin versions:
katello-3.14
Other relevant data:
[root@hostname ~]# journalctl -u pulp_workers.service -u pulp_celerybeat.service --since=today --no-pager
-- Logs begin at Tue 2018-03-20 16:26:21 EDT, end at Fri 2018-03-23 05:35:06 EDT. --
Mar 23 00:00:10 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 00:10:10 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 00:20:10 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 00:30:10 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 00:40:08 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.queue_reap_expired_documents)
Mar 23 00:40:10 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 00:50:10 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 00:59:36 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 00:59:36 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:00:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 01:00:25 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 01:00:30 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 01:10:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 01:13:41 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 01:13:41 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:14:26 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 01:14:26 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 01:14:28 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 01:14:31 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 01:16:18 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 01:16:18 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 01:17:33 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 01:17:38 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 01:18:03 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 01:18:05 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 01:20:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 01:30:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 01:40:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 01:48:54 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 01:49:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 01:49:26 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 01:50:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 02:00:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 02:02:53 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 02:03:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 02:03:26 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 02:03:51 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 02:03:53 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 02:10:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 02:20:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 02:30:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 02:40:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 02:50:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 03:00:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 03:06:19 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 03:06:19 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:06:55 hostname pulp[2849]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.tasks:INFO: Task canceled: ee10a551-ded9-45d1-bf30-5427e81a1124.
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 03:08:06 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 03:08:11 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 03:08:36 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 03:08:36 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 03:09:02 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 03:09:07 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 03:10:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 03:20:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 03:30:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 03:35:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 03:35:21 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:35:21 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 03:40:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 03:50:16 hostname pulp[2849]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Mar 23 03:52:27 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 03:52:27 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.tasks:INFO: Task canceled: f222314a-c69d-4e52-bd8a-c0f0db62a449.
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_manager process running.
Mar 23 03:52:29 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.
Mar 23 03:52:34 hostname pulp[2849]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler@hostname' discovered
Mar 23 03:54:38 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'scheduler@hostname' has gone missing, removing from list of workers
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@hostname' has gone missing, removing from list of workers
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@hostname' has gone missing, removing from list of workers
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@hostname' has gone missing, removing from list of workers
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@hostname' has gone missing, removing from list of workers
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@hostname is missing. Canceling the tasks in its queue.
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@hostname' has gone missing, removing from list of workers
Mar 23 03:55:05 hostname pulp[2849]: pulp.server.async.tasks:ERROR: The worker named resource_manager@hostname is missing. Canceling the tasks in its queue.
I am also noticing that at times, on demand syncs turn into deferred downoad policy as shown in the above logs. Is it due to pulp workers not running?