I have updated my main server and content proxy to pulp-rpm 3.14.2 and pulpcore 3.14.5 yesterday morning. I have published and promoted a new version of my CVs this morning.
No change. So I regenerated metadata on the centos8 CV and after that did a complete sync on the content proxy.
The complete sync seems to have fixed it for the Production/centos8. The images directory and everything is there now, too.
However, the Testing/centos8 publication is still missing it.
The complete sync finished with an error:
{"smart_proxy_history_id"=>10441,
"pulp_tasks"=>
[{"pulp_href"=>"/pulp/api/v3/tasks/adbfecf7-1a3c-466a-acad-aee8db1dd8ff/",
"pulp_created"=>"2021-08-27T08:14:20.731+00:00",
"state"=>"failed",
"name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
"logging_cid"=>"4c5d35d9-5d04-4f89-8291-a522fd6aecca",
"started_at"=>"2021-08-27T08:16:30.581+00:00",
"finished_at"=>"2021-08-27T08:16:42.552+00:00",
"error"=>
{"traceback"=>
" File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n" +
" result = func(*args, **kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 458, in synchronize\n" +
" subrepo_version = dv.create()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
" return future.result()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
" await self.run()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 275, in run\n" +
" self.new_version.add_content(Content.objects.filter(pk__in=to_add))\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line 764, in add_content\n" +
" RepositoryContent.objects.bulk_create(repo_content)\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n" +
" return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 480, in bulk_create\n" +
" obj_without_pk._state.db = self.db\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/transaction.py\", line 240, in __exit__\n" +
" connection.commit()\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 262, in commit\n" +
" self._commit()\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 240, in _commit\n" +
" return self.connection.commit()\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n" +
" raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 240, in _commit\n" +
" return self.connection.commit()\n",
"description"=>
"insert or update on table \"core_repositorycontent\" violates foreign key constraint \"core_repositoryconte_version_added_id_d5113f18_fk_core_repo\"\n" +
"DETAIL: Key (version_added_id)=(3c6fc91f-c147-44df-936b-41ab5c30b7d8) is not present in table \"core_repositoryversion\".\n"},
"worker"=>"/pulp/api/v3/workers/920a4446-1259-468d-bfc8-6305902efa47/",
"child_tasks"=>[],
"progress_reports"=>
[{"message"=>"Downloading Metadata Files",
"code"=>"sync.downloading.metadata",
"state"=>"completed",
"done"=>8},
{"message"=>"Downloading Artifacts",
"code"=>"sync.downloading.artifacts",
"state"=>"canceled",
"done"=>0},
{"message"=>"Associating Content",
"code"=>"associating.content",
"state"=>"failed",
"done"=>0},
{"message"=>"Parsed Modulemd",
"code"=>"sync.parsing.modulemds",
"state"=>"completed",
"total"=>119,
"done"=>119},
{"message"=>"Parsed Modulemd-defaults",
"code"=>"sync.parsing.modulemd_defaults",
"state"=>"completed",
"total"=>45,
"done"=>45},
{"message"=>"Parsed Packages",
"code"=>"sync.parsing.packages",
"state"=>"canceled",
"total"=>6865,
"done"=>2959}],
"created_resources"=>[],
"reserved_resources_record"=>
["/pulp/api/v3/remotes/rpm/rpm/dab1ea41-48f0-489e-84fb-a2ad1c44ecad/",
"/pulp/api/v3/repositories/rpm/rpm/5f8e4e3e-647e-428f-bbd4-e3cca8e28276/"]}],
"task_groups"=>[],
"poll_attempts"=>{"total"=>26, "failed"=>1}}
I have checked the remote dab1ea41-48f0-489e-84fb-a2ad1c44ecad in the pulpcore database on the content proxy:
pulpcore=# select pulp_id,url from core_remote where pulp_id = 'dab1ea41-48f0-489e-84fb-a2ad1c44ecad';
pulp_id | url
--------------------------------------+-----------------------------------------------------------------------------------------
dab1ea41-48f0-489e-84fb-a2ad1c44ecad | https://foreman.example.com/pulp/content/ORG/Testing/centos8/custom/centos8/BaseOS_x86_64/
(1 row)
So that’s exactly affecting the broken publication.
The core_distribution table also says it hasn’t been updated:
pulpcore=# select pulp_id,pulp_last_updated, pulp_type, name, base_path, publication_id from core_distribution where base_path like '%BaseOS%';
pulp_id | pulp_last_updated | pulp_type | name | base_path | publication_id
--------------------------------------+-------------------------------+-----------+-----------------------------------------------------------------+------------------------------------------------------------+--------------------------------------
03b550be-a94d-41be-8cfa-198f781a5985 | 2021-08-27 10:21:33.927403+02 | rpm.rpm | 1-centos8-Production-936cee46-e27f-4efa-a60d-c51970827e8f | ORG/Production/centos8/custom/centos8/BaseOS_x86_64 | 843cacef-8846-457f-a7f9-6a62a63576cb
b9fb3fe3-0281-42a3-9caf-93486d3827fe | 2021-08-27 10:22:47.742452+02 | rpm.rpm | 1-centos8-epel8-Testing-936cee46-e27f-4efa-a60d-c51970827e8f | ORG/Testing/centos8-epel8/custom/centos8/BaseOS_x86_64 | 9f268e77-d6d2-4f97-bdd9-93da5a20088a
21e97523-cdcf-46ac-9b1b-19b6a3dee3a3 | 2021-08-24 06:25:24.091505+02 | rpm.rpm | 1-centos8-Testing-936cee46-e27f-4efa-a60d-c51970827e8f | ORG/Testing/centos8/custom/centos8/BaseOS_x86_64 | b79232e3-7acf-4d4a-847c-0d86aa80c8f1
3c3ccbf7-529d-4b80-98d1-d83049776b21 | 2021-08-27 10:22:47.441628+02 | rpm.rpm | 1-centos8-epel8-Production-936cee46-e27f-4efa-a60d-c51970827e8f | ORG/Production/centos8-epel8/custom/centos8/BaseOS_x86_64 | af0663df-59af-4bbd-931a-08513c4d19db
(4 rows)
The mentioned repositoryversion from the error messages does not exist in the database:
pulpcore=# select * from core_repositoryversion where pulp_id = '3c6fc91f-c147-44df-936b-41ab5c30b7d8';
pulp_id | pulp_created | pulp_last_updated | number | complete | base_version_id | repository_id
---------+--------------+-------------------+--------+----------+-----------------+---------------
(0 rows)
After a brief search I haven’t found anything in the logs for “3c6fc91f” beyond that exception. So I can’t tell why the repository version was not created…
