Pulp v3 sync error

Problem:
trying to syncronise a repo after upgrading from katello 3.16rc5 to 3.16
failed on some repos with a Actions::Pulp3::Repository::Sync error: “rpm_packagegroup_related_packages” does not exist

Expected outcome:
expected it to successfully syncronise

Foreman and Proxy versions:
Foreman v2.1.4
this is on the central foreman instance

Foreman and Proxy plugin versions:

Distribution and version:
centos v7

Other relevant data:
{“pulp_tasks”=>
[{“pulp_href”=>"/pulp/api/v3/tasks/a2ff7798-9763-423f-9b9d-20810481a12f/",
“pulp_created”=>“2020-10-23T15:18:44.054+00:00”,
“state”=>“failed”,
“name”=>“pulp_rpm.app.tasks.synchronizing.synchronize”,
“started_at”=>“2020-10-23T15:18:44.141+00:00”,
“finished_at”=>“2020-10-23T15:18:45.226+00:00”,
“error”=>
{“traceback”=>
" File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 883, in perform_job\n" +
" rv = job.perform()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 645, in perform\n" +
" self._result = self._execute()\n" +
" File “/usr/lib/python3.6/site-packages/rq/job.py”, line 651, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 167, in synchronize\n" +
" is_optimized = (\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete\n" +
" return future.result()\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call\n" +
" await self.run()\n" +
" File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py”, line 861, in run\n" +
" self.dc_environments = \n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/manager.py”, line 82, in manager_method\n" +
" return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 474, in bulk_create\n" +
" ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 1211, in _batched_insert\n" +
" self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 1186, in _insert\n" +
" return query.get_compiler(using=using).execute_sql(return_id)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1375, in execute_sql\n" +
" cursor.execute(sql, params)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute\n" +
" return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers\n" +
" return executor(sql, params, many, context)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute\n" +
" return self.cursor.execute(sql, params)\n" +
" File “/usr/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit\n" +
" raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
" File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute\n" +
" return self.cursor.execute(sql, params)\n",
“description”=>
“relation “rpm_packagegroup_related_packages” does not exist\n” +
“LINE 1: INSERT INTO “rpm_packagegroup_related_packages” (“packagegro…\n” +
" ^\n”},
“worker”=>"/pulp/api/v3/workers/f54c1f82-8dad-4ca5-b0f1-986a33c251ca/",
“child_tasks”=>,
“progress_reports”=>
[{“message”=>“Downloading Metadata Files”,
“code”=>“downloading.metadata”,
“state”=>“completed”,
“done”=>4},
{“message”=>“Downloading Artifacts”,
“code”=>“downloading.artifacts”,
“state”=>“completed”,
“done”=>0},
{“message”=>“Associating Content”,
“code”=>“associating.content”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Un-Associating Content”,
“code”=>“unassociating.content”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Parsed Comps”,
“code”=>“parsing.comps”,
“state”=>“completed”,
“total”=>1,
“done”=>1},
{“message”=>“Parsed Packages”,
“code”=>“parsing.packages”,
“state”=>“completed”,
“total”=>14,
“done”=>14}],
“created_resources”=>,
“reserved_resources_record”=>
["/pulp/api/v3/repositories/rpm/rpm/62335352-3416-4619-b124-07c95217c447/",
“/pulp/api/v3/remotes/rpm/rpm/158dd13d-c7ae-4566-9b5d-eff8f3b0aa0b/”]}],
“create_version”=>true,
“task_groups”=>,
“poll_attempts”=>{“total”=>3, “failed”=>1}}

@simonmcq thank you for bringing this to our attention. This appears to be a pulpcore relation that doesn’t exist in the pulpcore database.

@goosemania can you comment on this error?

If I delete the repository from foreman, and then create it again from scratch, would it populate the tables with the correct entries and there by fix it?
or would I just end up corrupting the repositories completely??
can you use the same “label” when recreating the same repository a second time round?
and what would happen to the proxies that are synced with it?? would they erase their current version and sync the new one?

or would it just be easier to add the missing entries into the database :wink:
thanks in advance

@simonmcq If the relation is missing, then it’s likely to be a problem with the migrations pulpcore and pulp_rpm runs during install.

Recreating the repository would not fix this issue. It’s probably an installer issue, but I am hoping someone from the pulp_rpm team can comment first.

@simonmcq can you issue a restart for pulpcore* services?

If after the restart you still see this error, can you send the output of the pulpcore status?

curl https://`hostname`/pulp/api/v3/status/   --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key

Good morning

{“versions”:[{“component”:“pulpcore”,“version”:“3.4.1”},{“component”:“pulp_2to3_migration”,“version”:“0.2.0b7”},{“component”:“pulp_rpm”,“version”:“3.5.1”},{“component”:“pulp_file”,“version”:“1.0.1”},{“component”:“pulp_container”,“version”:“1.4.2”},{“component”:“pulp_certguard”,“version”:“0.1.0rc5”}],“online_workers”:[{“pulp_href”:"/pulp/api/v3/workers/ee7fb151-4cc8-4b1a-bb05-9f5992903130/",“pulp_created”:“2020-10-29T07:33:08.460142Z”,“name”:“89695@foreman.###.net”,“last_heartbeat”:“2020-10-29T07:35:24.966658Z”},{“pulp_href”:"/pulp/api/v3/workers/2f0ce0df-cf9f-4681-9d6d-2c357f03fb47/",“pulp_created”:“2020-10-29T07:33:08.514484Z”,“name”:“89694@foreman.###.net”,“last_heartbeat”:“2020-10-29T07:35:25.167193Z”},{“pulp_href”:"/pulp/api/v3/workers/5b0f85a6-ca1e-4b1d-8d3a-870ab15dcfbf/",“pulp_created”:“2020-07-31T17:43:06.866439Z”,“name”:“resource-manager”,“last_heartbeat”:“2020-10-29T07:35:29.885599Z”}],“online_content_apps”:[{“name”:“89570@foreman.###.net”,“last_heartbeat”:“2020-10-29T07:35:27.536488Z”},{“name”:“89571@foreman.###.net”,“last_heartbeat”:“2020-10-29T07:35:27.536822Z”}],“database_connection”:{“connected”:true},“redis_connection”:{“connected”:true},“storage”:{“total”:76056969216,“used”:64154787840,“free”:11902181376}}

BTW: could this be related to the problem
when the foreman-installer runs, it looks like it fails to complete the upgrade

[DEBUG 2020-10-29T10:42:52 main] Upgrade Step 1/3: katello:correct_repositories. This may take a long while.
[DEBUG 2020-10-29T10:42:52 main] Processing Repository 1/189: BaseOS (1)
[DEBUG 2020-10-29T10:42:52 main] Repository 1 Missing
[DEBUG 2020-10-29T10:42:52 main] Recreating 1
[DEBUG 2020-10-29T10:42:52 main] Failed upgrade task: katello:correct_repositories, see logs for more information

after restarting of all foreman services, the INSERT INTO “rpm_packagegroup_related_packages” has gone.
Had an issue that it wasn’t syncing the repos via the proxy, but followed the instructions on Katello is bypassing the configured HTTP proxy

Now all my repos have synced.
I assume the step that was missing was the “restart”?

Should I log a separate call for the “Failed upgrade task: katello:correct_repositories”??

Thanks for you help

@simonmcq I am glad to see we have resolved at least the first issue for you.

Yes, let’s open another discussion re the installer failure. Thanks!