Pulp issue on capsule/smart proxy

NeilHanlon · October 24, 2016, 6:25pm

For some reason pulp/celery keeps failing/stopping, resulting in the proxy
being unable to pull down content (operating systems), so it can't get the
vmlinuz and initrd for kickstarting/pxe.

gist.github.com

https://gist.github.com/NeilHanlon/80a88623fc9ca442e4b04a6a4d88213f

pulp_celerybeat

Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944) Message Error: Couldn't apply scheduled task download_deferred_content: Exception from Agent: {u'error_code': 7, u'error_text': 'Queue celery: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/ee7a4c7e-b14e-4745-8523-a713b1ac73dc.jrnl" symlink="/var/lib/qpidd/.qpidd/qls/jrnl2/celery/ee7a4c7e-b14e-4745-8523-a713b1ac73dc.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:425)'}
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944) ['  File "/usr/bin/celery", line 9, in <module>\n    load_entry_point(\'celery==3.1.11\', \'console_scripts\', \'celery\')()\n', '  File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 30, in main\n    main()\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main\n    cmd.execute_from_commandline(argv)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline\n    super(CeleryCommand, self).execute_from_commandline(argv)))\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 306, in execute_from_commandline\n    return self.handle_argv(self.prog_name, argv[1:])\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv\n    return self.execute(command, argv)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute\n    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 310, in run_from_argv\n    sys.argv if argv is None else argv, command)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 372, in handle_argv\n    return self(*args, **options)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 269, in __call__\n    ret = self.run(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/beat.py", line 79, in run\n    return beat().run()\n', '  File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 83, in run\n    self.start_scheduler()\n', '  File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in start_scheduler\n    beat.start()\n', '  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start\n    interval = self.scheduler.tick()\n', '  File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 264, in tick\n    ret = self.call_tick(self, celerybeat_name)\n', '  File "/usr/lib/python2.7/site-p
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944) ackages/pulp/server/async/scheduler.py", line 229, in call_tick\n    ret = super(Scheduler, self).tick()\n', '  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick\n    next_time_to_run = self.maybe_due(entry, self.publisher)\n', '  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 206, in maybe_due\n    exc, traceback.format_stack(), exc_info=True)\n']
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944) Traceback (most recent call last):
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 203, in maybe_due
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944)     result = self.apply_async(entry, publisher=publisher)
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 259, in apply_async
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944)     entry, exc=exc)), sys.exc_info()[2])
Oct 24 14:17:07 m4katello01.m4. pulp[3834]: celery.beat:ERROR: (3834-82944)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 255, in apply_async

This file has been truncated. show original

pulp_resource_manager

Oct 21 10:14:11 m4katello01.m4 pulp[3111]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456) Unrecoverable error: Exception('Exception from Agent: {u\'error_code\': 7, u\'error_text\': \'Queue resource_manager: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/5b67e2fb-f7ea-437b-acc5-b3b9fa4f19fe.jrnl" symlink="/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/5b67e2fb-f7ea-437b-acc5-b3b9fa4f19fe.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:425)\'}',)
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456) Traceback (most recent call last):
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456)   File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456)     self.blueprint.start(self)
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456)     step.start(parent)
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 373, in start
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456)     return self.obj.start()
Oct 21 10:14:11 m4katello01.m4 pulp[3111]: celery.worker:ERROR: (3111-75456)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start

This file has been truncated. show original

Also seeing this in my /var/log/messages

NotFound: no such queue: pulp.task

Just started happening, as I've just installed this capsule. I've tried
restarting pulp/etc, but to no avail.