CentOS 8.4 BaseOS Sync error

Foreman 2.5.3, Katello 4.1.1 on CentOS 7

The CentOS 8.4 BaseOS sync got an error tonight:

Actions::Pulp3::Repository::CreatePublication
...
PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Jul 2021 00:36:22 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"7df0e419-ebfb-4a1a-a0a5-3bc0c271493b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.dkrz.de", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/publications_rpm_api.rb:79:in `create_with_http_info'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/publications_rpm_api.rb:28:in `create'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/repository.rb:214:in `create_publication'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/lib/actions/pulp3/repository/create_publication.rb:15:in `invoke_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:19:in `run'
...

and

Actions::Katello::Repository::IndexContent
...
PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Jul 2021 00:36:23 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"7df0e419-ebfb-4a1a-a0a5-3bc0c271493b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.dkrz.de", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/ not found for repositoryversion."]

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:236:in `list_with_http_info'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:130:in `list'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:93:in `content_unit_list'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:106:in `fetch_content_list'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:75:in `block (2 levels) in pulp_units_batch_for_repo'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:69:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:69:in `block in pulp_units_batch_for_repo'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/concerns/pulp_database_unit.rb:162:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/concerns/pulp_database_unit.rb:162:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/concerns/pulp_database_unit.rb:162:in `import_for_repository'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/repository.rb:904:in `block (2 levels) in index_content'
...

As it’s involving the BaseOS repo and I now have issues updating systemd from baseos to the latest version, I guess it could be related:

# dnf update --nobest
...
 Problem: The operation would result in removing the following protected packages: systemd, systemd-udev
...
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 systemd         x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  3.6 M
 systemd         x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  3.6 M
 systemd-libs    x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  1.1 M
 systemd-libs    x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  1.1 M
 systemd-pam     x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  468 k
 systemd-pam     x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  469 k
 systemd-udev    x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  1.4 M
 systemd-udev    x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  1.4 M

See also Katello 4.1 and Rocky Linux 8 - not ready for prime time yet it seems

For anyone stumbling on this post in the future: Seems like this is due to a Pulp issue, Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp

see also the post linked above.

The update problem with the message

Problem: The operation would result in removing the following protected packages: systemd, systemd-udev

is related to the pulp issue.

I am not sure if the sync error has anything to do with that. It’s probably more a coincidence that it happened on the baseos repo on the same day there are some updates in the baseos where the missing filelists become relevant. I have updated to 4.1.1/pulpcore 3.14 on July 16th so technically the pulp issue was there since then but it’s only noticeable when new updates are synced which break dependencies due to missing filelists…

1 Like

I am trying to find out more about those sync errors, but I don’t find much. All I have found so far is in the pulpcore-api journal:

pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:21 +0000] "GET /pulp/api/v3/tasks/77e069d2-4a4c-4bd8-bd9f-b75727c713cb/ HTTP/1.1" 200 2516 "-" "OpenAPI-Generator/3.14.1/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]: django.request:WARNING: Bad Request: /pulp/api/v3/publications/rpm/rpm/
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "POST /pulp/api/v3/publications/rpm/rpm/ HTTP/1.1" 400 67 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/?base_path=ORG%2FLibrary%2Fcustom%2Fcentos8%2FBaseOS_x86_64 HTTP/1.1" 200 549 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/84710568-2b25-4e5a-a79a-fcf150ff7eee/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "GET /pulp/api/v3/tasks/efa9721f-a670-41c2-b542-aaab741656ae/ HTTP/1.1" 200 510 "-" "OpenAPI-Generator/3.14.1/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "GET /pulp/api/v3/tasks/efa9721f-a670-41c2-b542-aaab741656ae/ HTTP/1.1" 200 562 "-" "OpenAPI-Generator/3.14.1/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/84710568-2b25-4e5a-a79a-fcf150ff7eee/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]: django.request:WARNING: Bad Request: /pulp/api/v3/content/rpm/packages/
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:23 +0000] "GET /pulp/api/v3/content/rpm/packages/?arch__ne=src&fields=pulp_href%2Cname%2Cversion%2Crelease%2Carch%2Cepoch%2Csummary%2Cis_modular%2Crpm_sourcerpm%2Clocation_href%2CpkgId&limit=2000&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Fpublications%2Frpm%2Frpm%2Fd44e483c-977c-48c1-93b7-9e26e64e7874%2F HTTP/1.1" 400 112 "-" "OpenAPI-Generator/3.13.3/ruby"

O.K. There seems to be something broken. Looking at the error of IndexContent above:

Response body: ["URI /pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/ not found for repositoryversion."]

The publication is there:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.dkrz.de/pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/' | python -m json.tool 
{
    "gpgcheck": 0,
    "metadata_checksum_type": "unknown",
    "package_checksum_type": "unknown",
    "pulp_created": "2021-07-21T00:34:59.582368Z",
    "pulp_href": "/pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/",
    "repo_gpgcheck": 1,
    "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
    "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/3/",
    "sqlite_metadata": true
}

But the repository listed there is missing:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.dkrz.de/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/' | python -m json.tool 
{
    "detail": "Not found."
}

This looks to me as if there is something broken.

If I look into the pulpcore database I can find the repository with uuid 3f9dc526-a51c-4a25-9547-95f82bedb3ee:

pulpcore=# select * from core_publication where pulp_id = 'd44e483c-977c-48c1-93b7-9e26e64e7874';
               pulp_id                |         pulp_created          |       pulp_last_updated       | pulp_type | complete | pass_through |        repository_version_id         
--------------------------------------+-------------------------------+-------------------------------+-----------+----------+--------------+--------------------------------------
 d44e483c-977c-48c1-93b7-9e26e64e7874 | 2021-07-21 02:34:59.582368+02 | 2021-07-21 02:35:03.083383+02 | rpm.rpm   | t        | f            | 67cca843-6639-479b-805a-f912cd4cd9f4
(1 row)

pulpcore=# select * from core_repositoryversion where pulp_id = '67cca843-6639-479b-805a-f912cd4cd9f4';
               pulp_id                |         pulp_created          |       pulp_last_updated       | number | complete | base_version_id |            repository_id             
--------------------------------------+-------------------------------+-------------------------------+--------+----------+-----------------+--------------------------------------
 67cca843-6639-479b-805a-f912cd4cd9f4 | 2021-07-21 02:33:18.980391+02 | 2021-07-21 02:34:59.342965+02 |      3 | t        |                 | 3f9dc526-a51c-4a25-9547-95f82bedb3ee
(1 row)

pulpcore=# select * from core_repository where pulp_id = '3f9dc526-a51c-4a25-9547-95f82bedb3ee';
               pulp_id                |         pulp_created          |       pulp_last_updated       |                                              name                                              | description | next_version | pulp_type | remote_id | retained_versions | user_hidden 
--------------------------------------+-------------------------------+-------------------------------+------------------------------------------------------------------------------------------------+-------------+--------------+-----------+-----------+-------------------+-------------
 3f9dc526-a51c-4a25-9547-95f82bedb3ee | 2021-07-07 14:33:17.628538+02 | 2021-07-22 22:36:04.957548+02 | ../../../AppStream/x86_64/os/-31410651d4c07a347a6dbba79cfb4df7c6418e39b0a19c70a84eff2c8b3b8953 |             |            5 | rpm.rpm   |           |                   | f
(1 row)

There is also a rpm publication and repository:

pulpcore=# select * from rpm_rpmpublication where publication_ptr_id = 'd44e483c-977c-48c1-93b7-9e26e64e7874';
          publication_ptr_id          | metadata_checksum_type | package_checksum_type | gpgcheck | repo_gpgcheck | sqlite_metadata 
--------------------------------------+------------------------+-----------------------+----------+---------------+-----------------
 d44e483c-977c-48c1-93b7-9e26e64e7874 | unknown                | unknown               |        0 |             1 | t
(1 row)

pulpcore=# select * from rpm_rpmrepository where repository_ptr_id = '3f9dc526-a51c-4a25-9547-95f82bedb3ee';
          repository_ptr_id           | sub_repo | metadata_signing_service_id |         last_sync_remote_id          | last_sync_repo_version | last_sync_revision_number |                                                                                        original_checksum_types                       
                                                                 | retain_package_versions |                    last_sync_repomd_checksum                     | autopublish | gpgcheck | metadata_checksum_type | package_checksum_type | repo_gpgcheck | sqlite_metadata 
--------------------------------------+----------+-----------------------------+--------------------------------------+------------------------+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------+-------------------------+------------------------------------------------------------------+-------------+----------+------------------------+-----------------------+---------------+-----------------
 3f9dc526-a51c-4a25-9547-95f82bedb3ee | t        |                             | 69abed5c-b1e1-452a-b58b-f0cc23548924 |                      4 | 8.4.2105                  | {"group": "sha256", "other": "sha256", "modules": "sha256", "primary": "sha256", "group_xz": "sha256", "other_db": "sha256", "filelis
ts": "sha256", "primary_db": "sha256", "filelists_db": "sha256"} |                       0 | c21c5d2410544fccf2dcc78ce0f472fd8ed9d8c3fc0de1a8f8061a72629a5c7e | f           |        0 | sha256                 | sha256                |             0 | f
(1 row)

I don’t know what exactly is expected in the database, but superficially it looks as if everything is there. So it is unclear to me why the api request for rpm repository cannot find it.

Another observation: if I check the number of rpm publications and repositories via api I get these counts:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.example.com/pulp/api/v3/publications/rpm/rpm/?limit=10000' | python -m json.tool
{
    "count": 350,
...
# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.example.com/pulp/api/v3/repositories/rpm/rpm/?limit=10000' | python -m json.tool
{
    "count": 253,
...

But in the database the repository count is different:

pulpcore=# select count(*) from rpm_rpmpublication ;
 count 
-------
   350
(1 row)

pulpcore=# select count(*) from rpm_rpmrepository ;
 count 
-------
   258
(1 row)

Another insight: Using the pulp api I have checked all repository URI returned in the publications list and check the repository URI for ‘Not found’ and I found only two publications which reference the “missing” repository 3f9dc526-a51c-4a25-9547-95f82bedb3ee:

        {
            "gpgcheck": 0,
            "metadata_checksum_type": "unknown",
            "package_checksum_type": "unknown",
            "pulp_created": "2021-07-22T20:36:04.962028Z",
            "pulp_href": "/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/",
            "repo_gpgcheck": 1,
            "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
            "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/4/",
            "sqlite_metadata": true
        },
...
        {
            "gpgcheck": 0,
            "metadata_checksum_type": "unknown",
            "package_checksum_type": "unknown",
            "pulp_created": "2021-07-21T00:34:59.582368Z",
            "pulp_href": "/pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/",
            "repo_gpgcheck": 1,
            "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
            "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/3/",
            "sqlite_metadata": true
        },

Thus, for all other publications it is correct. It only this repository which is messed up.

The “missing” repository seems to be the CentOS BaseOS repo:

pulpcore=# select core_remote.* from rpm_rpmrepository left join core_remote on ( last_sync_remote_id = core_remote.pulp_id ) where repository_ptr_id = '3f9dc526-a51c-4a25-9547-95f82bedb3ee';
               pulp_id                |         pulp_created          |       pulp_last_updated       | pulp_type |         name         |                         url                         | ca_cert | client_cert | client_key | tls_validation
 | proxy_url | username | password | download_concurrency |  policy   | connect_timeout | sock_connect_timeout | sock_read_timeout | total_timeout | headers | rate_limit | proxy_password | proxy_username | max_retries 
--------------------------------------+-------------------------------+-------------------------------+-----------+----------------------+-----------------------------------------------------+---------+-------------+------------+---------------
-+-----------+----------+----------+----------------------+-----------+-----------------+----------------------+-------------------+---------------+---------+------------+----------------+----------------+-------------
 69abed5c-b1e1-452a-b58b-f0cc23548924 | 2021-07-07 07:10:41.527911+02 | 2021-07-07 14:33:16.042855+02 | rpm.rpm   | BaseOS_x86_64-172119 | http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ |         |             |            | t             
 |           |          |          |                   20 | immediate |                 |                      |                   |           300 |         |            |                |                |            
(1 row)

I guess it all comes down to why the pulp api doesn’t find the repository even though it’s in the database?

I have just noticed that last night another baseos sync failed with this error:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 22 Jul 2021 20:36:46 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"ded10b33-c063-4471-9182-3b62facbd36b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 22 Jul 2021 20:36:47 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"ded10b33-c063-4471-9182-3b62facbd36b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/ not found for repositoryversion."]

It goes into the same repository:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.example.com/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/' | python -m json.tool
{
    "gpgcheck": 0,
    "metadata_checksum_type": "unknown",
    "package_checksum_type": "unknown",
    "pulp_created": "2021-07-22T20:36:04.962028Z",
    "pulp_href": "/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/",
    "repo_gpgcheck": 1,
    "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
    "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/4/",
    "sqlite_metadata": true
}

So it’s nothing new. However, I run this sync task every four hours. So this error only occurs occasionally…

As so far it seems to involve only pulp I have created an pulp issue: Issue #9133: RPM repository sync error - Pulp

There seems to be something worse: I have just noticed that 5 days ago the sync for the baseos of CentOS 8 Stream (not 8.4 as above) ended with a warning, and again this morning. I have the 8 Stream repos only for testing, i.e. currently I only have one client and it’s using the Library environment.

5 days ago:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 19 Jul 2021 02:00:55 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"b05492c2-dc3a-4cfb-9099-22674dcc652d", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 19 Jul 2021 02:00:57 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"b05492c2-dc3a-4cfb-9099-22674dcc652d", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/8c78f8f7-4278-44d7-bef9-30ce30843c4c/ not found for repositoryversion."]

Now that publication 8c78f8f7-4278-44d7-bef9-30ce30843c4c really doesn’t exist. It’s not found via API nor it is in the core_publication or rpm_rpmpublication tables.

This morning:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:40 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:41 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:41 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/ad1067e2-4255-4898-b55e-d2341e59cb99/ not found for repositoryversion."]

Now that publication ad1067e2-4255-4898-b55e-d2341e59 has the same issue as the baseos for 8.4 above: the publication references a repository which exists in the database but the API says “Not found”.

What’s worse: the client cannot access the repository anymore:

# dnf check-update
...
Errors during downloading metadata for repository 'ORG_centos8stream_BaseOS_x86_64_os':
  - Status code: 404 for https://foreman.example.com/pulp/repos/ORG/Library/custom/centos8stream/BaseOS_x86_64_os/repodata/repomd.xml (IP: xxxx:xxxx::1)
Error: Failed to download metadata for repo 'ORG_centos8stream_BaseOS_x86_64_os': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried