Some CVs on content proxies are missing files

Running Foreman 2.5.2 with Katello 4.1.2.1 on CentOS 7.9 and a separate content proxy with the same versions.

I have two CVs for CentOS 8: “centos8” and “centos8-epel8”. The former is “without EPEL8” the latter with EPEL8.

“centos8-epel8” contains all EL8 related repositories with no filters.

“centos8” contains all repositories CentOS 8 repositories as well as all product repos which don’t depend on EPEL8. “centos8” contains EPEL8 with a filter which only includes RPMs from EPEL8 which are required for our environment (nagios checks, puppet dependencies…). Only EPEL8 is filtered into centos8. All other repositories are unfiltered.

I have noticed that the CentOS 8 BaseOS directory doesn’t look the same on the content proxy. For the centos8-epel8 CV (https://foreman-content.example.com/pulp/content/ORG/Production/centos8-epel8/custom/centos8/BaseOS_x86_64/) it contains:

    .treeinfo
    AppStream/
    Packages/
    config.repo
    images/
    repodata/

For the centos8 CV it looks like this:

    .treeinfo
    Packages/
    config.repo
    repodata/

I have checked on the main server and there it always contains everything, etc., regardless of LE or CV including the Library versions.

Seems it’s an issue reported here: Issue #9230: Syncing kickstart trees with mirror=true does not properly include images in publication - RPM Support - Pulp.

Are you sure that’s the same issue? The main server contains the images directory. The content proxy has it in one CV but not in the other. I have “Mirror on Sync” set on all my repositories…

Still under investigation. Will update with progress.

@gvde did you confirm that requesting those files (the vmlinuz for example), actually throws a 404? @lfu is attempting to reproduce as we speak.

I just tried it for my Production LE. Result is consistent with what is listed for the directory urls.

CV centos8-epel8 against content proxy: OK

> GET /pulp/content/ORG/Production/centos8-epel8/custom/centos8/BaseOS_x86_64/images/pxeboot/vmlinuz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foreman-content.example.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 26 Aug 2021 14:49:32 GMT
< Server: Python/3.6 aiohttp/3.7.4
< Content-Type: application/octet-stream
< Last-Modified: Thu, 26 Aug 2021 14:49:08 GMT
< Content-Length: 10026120

CV centos8 against content proxy: not OK.

> GET /pulp/content/ORG/Production/centos8/custom/centos8/BaseOS_x86_64/images/pxeboot/vmlinuz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foreman-content.example.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Thu, 26 Aug 2021 14:51:37 GMT
< Server: Python/3.6 aiohttp/3.7.4
< Content-Type: text/plain; charset=utf-8
< Content-Length: 14

To compare CV centos8-epel8 against main server: OK

> GET /pulp/content/ORG/Production/centos8-epel8/custom/centos8/BaseOS_x86_64/images/pxeboot/vmlinuz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foreman.example.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 26 Aug 2021 14:52:09 GMT
< Server: Python/3.6 aiohttp/3.7.4
< Content-Type: application/octet-stream
< Last-Modified: Sun, 06 Jun 2021 11:31:37 GMT
< Content-Length: 10026120

CV centos8 against main server: OK

> GET /pulp/content/ORG/Production/centos8/custom/centos8/BaseOS_x86_64/images/pxeboot/vmlinuz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foreman.example.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 26 Aug 2021 14:52:31 GMT
< Server: Python/3.6 aiohttp/3.7.4
< Content-Type: application/octet-stream
< Last-Modified: Sun, 06 Jun 2021 11:31:37 GMT
< Content-Length: 10026120

I have updated my main server and content proxy to pulp-rpm 3.14.2 and pulpcore 3.14.5 yesterday morning. I have published and promoted a new version of my CVs this morning.

No change. So I regenerated metadata on the centos8 CV and after that did a complete sync on the content proxy.

The complete sync seems to have fixed it for the Production/centos8. The images directory and everything is there now, too.

However, the Testing/centos8 publication is still missing it.

The complete sync finished with an error:

{"smart_proxy_history_id"=>10441,
 "pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/adbfecf7-1a3c-466a-acad-aee8db1dd8ff/",
    "pulp_created"=>"2021-08-27T08:14:20.731+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid"=>"4c5d35d9-5d04-4f89-8291-a522fd6aecca",
    "started_at"=>"2021-08-27T08:16:30.581+00:00",
    "finished_at"=>"2021-08-27T08:16:42.552+00:00",
    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 272, in _perform_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 458, in synchronize\n" +
       "    subrepo_version = dv.create()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
       "    loop.run_until_complete(pipeline)\n" +
       "  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
       "    return future.result()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
       "    await asyncio.gather(*futures)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
       "    await self.run()\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 275, in run\n" +
       "    self.new_version.add_content(Content.objects.filter(pk__in=to_add))\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py\", line 764, in add_content\n" +
       "    RepositoryContent.objects.bulk_create(repo_content)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n" +
       "    return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 480, in bulk_create\n" +
       "    obj_without_pk._state.db = self.db\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/transaction.py\", line 240, in __exit__\n" +
       "    connection.commit()\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 262, in commit\n" +
       "    self._commit()\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 240, in _commit\n" +
       "    return self.connection.commit()\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n" +
       "    raise dj_exc_value.with_traceback(traceback) from exc_value\n" +
       "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 240, in _commit\n" +
       "    return self.connection.commit()\n",
      "description"=>
       "insert or update on table \"core_repositorycontent\" violates foreign key constraint \"core_repositoryconte_version_added_id_d5113f18_fk_core_repo\"\n" +
       "DETAIL:  Key (version_added_id)=(3c6fc91f-c147-44df-936b-41ab5c30b7d8) is not present in table \"core_repositoryversion\".\n"},
    "worker"=>"/pulp/api/v3/workers/920a4446-1259-468d-bfc8-6305902efa47/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Downloading Metadata Files",
       "code"=>"sync.downloading.metadata",
       "state"=>"completed",
       "done"=>8},
      {"message"=>"Downloading Artifacts",
       "code"=>"sync.downloading.artifacts",
       "state"=>"canceled",
       "done"=>0},
      {"message"=>"Associating Content",
       "code"=>"associating.content",
       "state"=>"failed",
       "done"=>0},
      {"message"=>"Parsed Modulemd",
       "code"=>"sync.parsing.modulemds",
       "state"=>"completed",
       "total"=>119,
       "done"=>119},
      {"message"=>"Parsed Modulemd-defaults",
       "code"=>"sync.parsing.modulemd_defaults",
       "state"=>"completed",
       "total"=>45,
       "done"=>45},
      {"message"=>"Parsed Packages",
       "code"=>"sync.parsing.packages",
       "state"=>"canceled",
       "total"=>6865,
       "done"=>2959}],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/remotes/rpm/rpm/dab1ea41-48f0-489e-84fb-a2ad1c44ecad/",
      "/pulp/api/v3/repositories/rpm/rpm/5f8e4e3e-647e-428f-bbd4-e3cca8e28276/"]}],
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>26, "failed"=>1}}

I have checked the remote dab1ea41-48f0-489e-84fb-a2ad1c44ecad in the pulpcore database on the content proxy:

pulpcore=# select pulp_id,url from core_remote where pulp_id = 'dab1ea41-48f0-489e-84fb-a2ad1c44ecad';
               pulp_id                |                                           url                                           
--------------------------------------+-----------------------------------------------------------------------------------------
 dab1ea41-48f0-489e-84fb-a2ad1c44ecad | https://foreman.example.com/pulp/content/ORG/Testing/centos8/custom/centos8/BaseOS_x86_64/
(1 row)

So that’s exactly affecting the broken publication.

The core_distribution table also says it hasn’t been updated:

pulpcore=# select pulp_id,pulp_last_updated, pulp_type, name, base_path, publication_id from core_distribution where base_path like '%BaseOS%';
               pulp_id                |       pulp_last_updated       | pulp_type |                              name                               |                         base_path                          |            publication_id            
--------------------------------------+-------------------------------+-----------+-----------------------------------------------------------------+------------------------------------------------------------+--------------------------------------
 03b550be-a94d-41be-8cfa-198f781a5985 | 2021-08-27 10:21:33.927403+02 | rpm.rpm   | 1-centos8-Production-936cee46-e27f-4efa-a60d-c51970827e8f       | ORG/Production/centos8/custom/centos8/BaseOS_x86_64       | 843cacef-8846-457f-a7f9-6a62a63576cb
 b9fb3fe3-0281-42a3-9caf-93486d3827fe | 2021-08-27 10:22:47.742452+02 | rpm.rpm   | 1-centos8-epel8-Testing-936cee46-e27f-4efa-a60d-c51970827e8f    | ORG/Testing/centos8-epel8/custom/centos8/BaseOS_x86_64    | 9f268e77-d6d2-4f97-bdd9-93da5a20088a
 21e97523-cdcf-46ac-9b1b-19b6a3dee3a3 | 2021-08-24 06:25:24.091505+02 | rpm.rpm   | 1-centos8-Testing-936cee46-e27f-4efa-a60d-c51970827e8f          | ORG/Testing/centos8/custom/centos8/BaseOS_x86_64          | b79232e3-7acf-4d4a-847c-0d86aa80c8f1
 3c3ccbf7-529d-4b80-98d1-d83049776b21 | 2021-08-27 10:22:47.441628+02 | rpm.rpm   | 1-centos8-epel8-Production-936cee46-e27f-4efa-a60d-c51970827e8f | ORG/Production/centos8-epel8/custom/centos8/BaseOS_x86_64 | af0663df-59af-4bbd-931a-08513c4d19db
(4 rows)

The mentioned repositoryversion from the error messages does not exist in the database:

pulpcore=# select * from core_repositoryversion where pulp_id = '3c6fc91f-c147-44df-936b-41ab5c30b7d8';
 pulp_id | pulp_created | pulp_last_updated | number | complete | base_version_id | repository_id 
---------+--------------+-------------------+--------+----------+-----------------+---------------
(0 rows)

After a brief search I haven’t found anything in the logs for “3c6fc91f” beyond that exception. So I can’t tell why the repository version was not created…

I have published a new version of CV centos8 and promoted it to Testing. According to the package counts nothing has changed. Now the automated, optimized sync to the content proxy fixed the missing files for Test/centos8. So now both CVs in both LEs show the images directory.

To be sure, I ran another complete sync of the content proxy which finished successful now. Good.

Then I promoted the the CV centos8 to Production to have it in line again as I had before. The optimized sync ran successfully.

Taking a closer look, however, I now have the AppStream directory missing for the Production/centos8 publication on the content proxy. It’s there for all three others on the content proxy. It’s there on the main server.

So, it’s still not perfectly working in all cases. Unfortunately, I don’t remember if the AppStream directory was there before. I only looked for the images directory for my tests, so it might have been there and now got lost again or it might have still been missing when the images directory popped up…

Another complete sync of the content proxy straightened it out. Now all four contain all directories and I think also all files (though I haven’t checked it). I hope it won’t happen again…

So now, I am still fighting with Katello 4.1.2.1 - 404 error through content proxy due to incorrect location_href and those broken links when accessed through the content proxy.

I tried to reproduce on pulp-rpm 3.14.2 and couldn’t. Let me know if it pops up again

I am on 4.1.3 with pulp-rpm 3.14.3 by now.

Tonight, some CentOS 8 updates arrived in the mirrors which were successfully synced to my main server.

Then I have published and promoted my CVs. The sync to the proxy ended with warning status and the same error again:

Katello::Errors::Pulp3Error: insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
DETAIL:  Key (version_added_id)=(13596b1b-148b-4b8f-bb8a-685bd40eb43f) is not present in table "core_repositoryversion".

This time it was pretty easy to find which repository is affected, because some servers don’t have all the updates available. It is the CentOS 8 BaseOS repository in my Testing environment in my “centos8” CV.

I have checked with the browser and everything seems fine on the main server at

https://foreman.example.com/pulp/content/ORG/Testing/centos8/custom/centos8/BaseOS_x86_64/Packages/

but not on the content proxy at the same place:

https://foreman-content.example.com/pulp/content/ORG/Testing/centos8/custom/centos8/BaseOS_x86_64/Packages/

The content proxy is missing the latest updates to the CentOS 8 BaseOS repo, e.g. ca-certificates-2021.2.50-80.0.el8_4.noarch.rpm.

There seems to be a pulp Issue #8967: "duplicate key value violates unique constraint" when syncing two repositories with identical content in parallel - Pulp which mentions the error message. The problem seems to happen during simultaneous syncs which would explain why it is only happen occasionally…