Trouble with pulpcore workers after trying to sync deb repo

Support
1

Problem:
I have been trying to sync a deb repo (ubuntu 20.04) but i haden’t installed python3-pulp-deb yet. the sync job returns me an error telling there is no /var/lib/pulp/content/deb. So I created one…

Pulp do the job (instead of pulpcore as i understand now: Pulp_deb for pulp3 in Katello ) but now pulpcore-workers can’t start.

In first place I found this log with journalctl -f :

Feb 02 09:50:03 myforemanserver pulpcore-worker-2[3774]: pulp: pulp_2to3_migration.app.plugin:INFO: Plugin pulp_deb is not installed in pulp3 therefore it will not be migrated from pulp2
Feb 02 09:50:03 myforemanserver pulpcore-worker-2[3774]: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker ‘3774@myforemanserver’.
Feb 02 09:50:03 myforemanserver pulpcore-worker-2[3774]: pulp: rq.worker:INFO: Worker rq:worker:3774@myforemanserver: started, version 1.5.2
Feb 02 09:50:03 myforemanserver pulpcore-worker-2[3774]: pulp: rq.worker:INFO: *** Listening on 3774@myforemanserver…
Feb 02 09:50:03 myforemanserver pulpcore-worker-2[3774]: pulp: pulpcore.tasking.services.worker_watcher:INFO: New worker ‘3774@myforemanserver’ discovered
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: pulp: pulp_2to3_migration.app.plugin:INFO: Plugin pulp_deb is not installed in pulp3 therefore it will not be migrated from pulp2
Feb 02 09:50:04 myforemanserver pulpcore-worker-1[3773]: pulp: pulp_2to3_migration.app.plugin:INFO: Plugin pulp_deb is not installed in pulp3 therefore it will not be migrated from pulp2
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker ‘resource-manager’.
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: Traceback (most recent call last):
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return self.cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: psycopg2.errors.UndefinedColumn: column core_task._resource_job_id does not exist
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: LINE 1: …k".“parent_task_id”, “core_task”.“task_group_id”, “core_task…
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: ^
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: The above exception was the direct cause of the following exception:
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: Traceback (most recent call last):
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/bin/rq”, line 11, in
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: load_entry_point(‘rq==1.5.2’, ‘console_scripts’, ‘rq’)()
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 829, in call
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return self.main(*args, **kwargs)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 782, in main
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: rv = self.invoke(ctx)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1259, in invoke
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return _process_result(sub_ctx.command.invoke(sub_ctx))
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 1066, in invoke
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return ctx.invoke(self.callback, **ctx.params)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 610, in invoke
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return callback(*args, **kwargs)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/rq/cli/cli.py”, line 78, in wrapper
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return ctx.invoke(func, cli_config, *args[1:], **kwargs)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/click/core.py”, line 610, in invoke
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return callback(*args, **kwargs)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/rq/cli/cli.py”, line 268, in worker
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: max_jobs=max_jobs, with_scheduler=with_scheduler)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 508, in work
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: self.register_birth()
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py”, line 167, in register_birth
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: mark_worker_offline(self.name, normal_shutdown=True)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/services/worker_watcher.py”, line 163, in mark_worker_offline
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: for task_status in worker.tasks.filter(state__in=TASK_INCOMPLETE_STATES):
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: self._fetch_all()
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: self._result_cache = list(self._iterable_class(self))
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1142, in execute_sql
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return executor(sql, params, many, context)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return self.cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: raise dj_exc_value.with_traceback(traceback) from exc_value
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: return self.cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: django.db.utils.ProgrammingError: column core_task._resource_job_id does not exist
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: LINE 1: …k”.“parent_task_id”, “core_task”.“task_group_id”, “core_task…
Feb 02 09:50:04 myforemanserver pulpcore-resource-manager[3781]: ^
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘10175@myforemanserver’ has gone missing, removing from list of workers
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 10175@myforemanserver is missing. Canceling the tasks in its queue.
Feb 02 09:50:04 myforemanserver pulpcore-worker-1[3773]: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker ‘3773@myforemanserver’.
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: pulp: rq.worker:ERROR: Worker rq:worker:3774@myforemanserver: found an unhandled exception, quitting…
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: Traceback (most recent call last):
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: return self.cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: psycopg2.errors.UndefinedColumn: column core_task._resource_job_id does not exist
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: LINE 1: …k”.“parent_task_id”, “core_task”.“task_group_id”, “core_task…
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: ^
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: The above exception was the direct cause of the following exception:
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: Traceback (most recent call last):
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 531, in work
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: self.check_for_suspension(burst)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 465, in check_for_suspension
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: while not self._stop_requested and is_suspended(self.connection, self):
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/rq/suspension.py”, line 7, in is_suspended
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: worker.heartbeat(pipeline=pipeline)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py”, line 184, in heartbeat
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: check_worker_processes()
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/services/worker_watcher.py”, line 79, in check_worker_processes
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: mark_worker_offline(worker.name)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/services/worker_watcher.py”, line 163, in mark_worker_offline
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: for task_status in worker.tasks.filter(state__in=TASK_INCOMPLETE_STATES):
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: self._fetch_all()
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: self._result_cache = list(self._iterable_class(self))
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1142, in execute_sql
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: return executor(sql, params, many, context)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: return self.cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: raise dj_exc_value.with_traceback(traceback) from exc_value
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: return self.cursor.execute(sql, params)
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: django.db.utils.ProgrammingError: column core_task._resource_job_id does not exist
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: LINE 1: …k”.“parent_task_id”, “core_task”.“task_group_id”, “core_task…
Feb 02 09:50:04 myforemanserver pulpcore-worker-2[3774]: ^
Feb 02 09:50:04 myforemanserver pulpcore-worker-1[3773]: pulp: rq.worker:INFO: Worker rq:worker:3773@myforemanserver: started, version 1.5.2
Feb 02 09:50:04 myforemanserver pulpcore-worker-1[3773]: pulp: rq.worker:INFO: *** Listening on 3773@myforemanserver…
Feb 02 09:50:04 myforemanserver pulpcore-worker-1[3773]: pulp: pulpcore.tasking.services.worker_watcher:INFO: New worker ‘3773@myforemanserver’ discovered
Feb 02 09:50:04 myforemanserver systemd[1]: pulpcore-resource-manager.service: main process exited, code=exited, status=1/FAILURE
Feb 02 09:50:04 myforemanserver systemd[1]: Unit pulpcore-resource-manager.service entered failed state.
Feb 02 09:50:04 myforemanserver systemd[1]: pulpcore-resource-manager.service failed.
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker ‘10175@myforemanserver’ has gone missing, removing from list of workers
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 10175@myforemanserver is missing. Canceling the tasks in its queue.
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: pulp: rq.worker:ERROR: Worker rq:worker:3773@myforemanserver: found an unhandled exception, quitting…
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: Traceback (most recent call last):
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: return self.cursor.execute(sql, params)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: psycopg2.errors.UndefinedColumn: column core_task._resource_job_id does not exist
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: LINE 1: …k”.“parent_task_id”, “core_task”.“task_group_id”, “core_task…
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: ^
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: The above exception was the direct cause of the following exception:
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: Traceback (most recent call last):
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 531, in work
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: self.check_for_suspension(burst)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 465, in check_for_suspension
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: while not self._stop_requested and is_suspended(self.connection, self):
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/rq/suspension.py”, line 7, in is_suspended
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: worker.heartbeat(pipeline=pipeline)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py”, line 184, in heartbeat
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: check_worker_processes()
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/services/worker_watcher.py”, line 79, in check_worker_processes
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: mark_worker_offline(worker.name)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/pulpcore/tasking/services/worker_watcher.py”, line 163, in mark_worker_offline
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: for task_status in worker.tasks.filter(state__in=TASK_INCOMPLETE_STATES):
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: self._fetch_all()
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: self._result_cache = list(self._iterable_class(self))
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1142, in execute_sql
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: cursor.execute(sql, params)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: return executor(sql, params, many, context)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: return self.cursor.execute(sql, params)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: raise dj_exc_value.with_traceback(traceback) from exc_value
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: return self.cursor.execute(sql, params)
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: django.db.utils.ProgrammingError: column core_task._resource_job_id does not exist
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: LINE 1: …k”.“parent_task_id”, “core_task”.“task_group_id”, "core_task…
Feb 02 09:50:05 myforemanserver pulpcore-worker-1[3773]: ^
Feb 02 09:50:08 myforemanserver systemd[1]: pulpcore-resource-manager.service holdoff time over, scheduling restart.
Feb 02 09:50:08 myforemanserver systemd[1]: pulpcore-worker@2.service holdoff time over, scheduling restart.
Feb 02 09:50:08 myforemanserver systemd[1]: Stopped Pulp RQ Worker.

So i installed python3-pulp-deb but the psycopg2 error is till there

Feb 02 09:53:17 srvtlpfk01.dit.cb pulpcore-worker-2[4653]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Feb 02 09:53:17 srvtlpfk01.dit.cb pulpcore-worker-2[4653]: return self.cursor.execute(sql, params)
Feb 02 09:53:17 srvtlpfk01.dit.cb pulpcore-worker-2[4653]: psycopg2.errors.UndefinedColumn: column core_task._resource_job_id does not exist
Feb 02 09:53:17 srvtlpfk01.dit.cb pulpcore-worker-2[4653]: LINE 1: …k".“parent_task_id”, “core_task”.“task_group_id”, "core_task…
Feb 02 09:53:17 srvtlpfk01.dit.cb pulpcore-worker-2[4653]: ^

Pulpcore-workers are trying to start in a loop

Foreman-maintain check health can’t start the services neither

Starting the following service(s):
pulpcore-worker@*
\ All services started
| Try 1/5: checking status of hammer ping
Couldn’t connect to the server: undefined method `to_sym’ for nil:NilClass

Expected outcome:
Pulp/pulpcore starting

Foreman and Proxy versions:
Foreman 2.2.2
Pulp/Pulpcore 2.1.0

Foreman and Proxy plugin versions:
Katello 3.17.1-1

Distribution and version:
CentOS Linux release 7.9.2009

Other relevant data:
I can provide log if needed.

2

This situation should not be allowed to exist (without at least some kind of graceful error handling). Without python3-pulp-deb installed, the API endpoints for synchronizing deb repos don’t exist on the Pulp side. At that point everything else is just a follow on error.

The error you now get sounds a bit like some non existent tasks where referenced somewhere in the Pulp DB (but I am speculating here). I don’t have any good ideas how to fix that (or indeed if I have really diagnosed the problem correctly). Perhaps we can get a core pulp developer to have a look at this? Or perhaps a veteran Katello developer has some relevant experience with something like this (perhaps in relation to other plugins)?

3

I can’t agree more !

I hope to find someone with this skills

4

I have been told, that the issue you are now having sounds like a mismatch between the Pulp code installed on your system, and the DB migrations that were applied.

This begs the question: How exactly did you install your system? Is it possible you missed some steps that would have been performed using “official” installation methods? Any information in that direction might help further diagnose your problem.

5

It was a fresh installation of Foreman 3.17 with the katello scenario proceeded as the official doc explain it:

foreman-installer --scenario katello -i --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh

So it come with pulp3 and we didn’t migrate from pulp2
In first place we just have yum repo, but the recents annoncements about Centos8 make us give a try to Ubuntu LTS… And here we are.

6

Official debian support with pulp3 wasn’t added until katello 3.18. I’d be expecting 3.17 to be using pulp2. Did you manually install the pulp3 debian plugin via yum/dnf?

Would you be able to re-try this on Katello 3.18?

7

Yes i installed python3-pulp-deb manually after I found a message in log telling it was missing…

I am trying to restore my server to last week, with luck before I tried to sync the deb repo.
If reverted to normal operations I can try a upgrade to 3.18, but in the actual state I don’t think it will works…

PS: why it didn’t work with pulp2 ? Whasn’t it the excepted outcome?

8

Response to myself:
Ok, I found that I should have run the foreman-installer with the --katello-enable-deb option to enable it. Can it be done after the initial setup?

9

I don’t understand why is it possible to choose the deb type when creating a repo if the option hasn’t been enabled. There is a missing check IHMO

10

I am not 100% sure, but I think on Katello 3.17 it should have defaulted to using the Pulp 2 version of pulp_deb.
@Justin_Sherrill can you confirm or correct?

11

Yesterday I restored my server but I can’t get a backup older enought to go back before I setup a deb repo. At least now, the python3-pulp-deb package is not installed anymore.

When checking in /about page I cant see that the following package are installed:

  • pulp-deb-plugins-1.10.2-1.el7.noarch
  • python-pulp-deb-common-1.10.2-1.el7.noarch

And when looking at the foreman-installer --help

= Module katello:
    --katello-enable-deb          Enable debian content plugin (current: true)

So I guess the way I installed Foreman-Katello ( foreman-installer --scenario katello) automatically provide the option.

The point is pulpcore-resource-manager and pulpcore-worker-1 / pulpcore-worker-2 can’t start because of this errors:

pulpcore-resource-manager[5257]: django.db.utils.ProgrammingError: column core_task._resource_job_id does not exist
pulpcore-worker-1[5249]: django.db.utils.ProgrammingError: column core_task._resource_job_id does not exist
pulpcore-worker-2[5250]: django.db.utils.ProgrammingError: column core_task._resource_job_id does not exist

I try to acces to “foreman” pgsql db to investigate but i can’t connect with the credentials i found in
/etc/foreman/database.yml

Anyway thanks fo helping! :+1:

12

Maybe this error log is more significant:

pulpcore-resource-manager[5257]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-resource-manager[5257]: return self.cursor.execute(sql, params)
pulpcore-resource-manager[5257]: psycopg2.errors.UndefinedColumn: column core_task._resource_job_id does not exist
pulpcore-resource-manager[5257]: LINE 1: ...k"."parent_task_id", "core_task"."task_group_id", "core_task...
pulpcore-resource-manager[5257]: ^
13

I’m in, but i don’t know in which table look for…

14

If I am not mistaken “Enable debian content plugin” means “use the Pulp 3 version of pulp_deb”.
It does not mean: “Enable deb type repos in Katello”

If this is disabled you can still use deb type repos, but they would default back to using the Pulp 2 version of pulp_deb under the hood.

If I am right then it is a bug for this to be enabled on a Katello version that does not yet have pulp_deb for Pulp 3 support. It should not even allow you to enable it (much less enable it by default).

@Justin_Sherrill or @m-bucher can hopefully confirm if I got that right.

15

actually that setting just enables it in katello in general. If its set to false it will hide all the UI bits.

@GC29 that looks like the pulp3 db isn’t fully migrated for some reason. The installer should handle it, so you could try re-running that with no options ‘foreman-installer’.

alternatively you can do it manually:

sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' python3-django-admin migrate --no-input
1 Like
16

New update:
With nothing left to lose I try to update to 3.18!

the upgrade_check was ok except it reports me:
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache

So I does it without troubles and launched the katello upgrade.

At first there were configuration error with pulpcore-api.socket and pulpcore-content.socket. I re-execute foreman-installer a second time without doing anything, no more errors about the sockets but foreman-proxy couldn’t load “Registration” features (which seems to be a new one).
Once foreman-proxy restarted the next foreman-installer execution went well, ending by a green “Success!”

foreman-maintain health check is all good
Everything looks good in the web UI too

Now deb is mentioned under Pulpcore supported content in my proxy services tab
pulpcore

Is the overridden to use Pulp2 normal?

I don’t know what I have done but certainly something that shouldn’t be allowed to do in 3.17 and the update probably fix it…

17

Yes, that ‘overridden to use pulp2’ is normal. You’ll need to go through a migration process to move that content to pulp3 (assuming you’ve got existing content). Its not available yet for debian content, but will be made available in 3.18 in the future. You can hold tight and continue using pulp2 for now.

If you don’t have any debian repos (i mean zero), then it is possible to ‘switch’ over to using pulp3 for debian content. Its not really a documented process as typically people will want to go through the migration workflow, but i can provide instructions if you really have no debian content already in pulp2.

18

We have no deb content for now, I think we will use pulp2 until pulp3 availability.
Thanks again for your help :+1:

19

So somes days later I’m back trying to sync a deb repo. (Ubuntu focal-updates)

My sync task ends with this error

PLP0000: [Errno 13] Permission denied: ‘/var/lib/pulp/content/units’

My server has no ‘units’ repository in /var/lib/pulp/content/
What am I supposed to do ? (I don’t want to break my server again)

20

Could you try this:

ls -ltr /var/lib/pulp/content/units

The owner should be apache:apache… Sometimes this causes issues with syncing.

1 Like