CentOS 8.4 BaseOS Sync error

gvde · July 21, 2021, 2:23pm

Foreman 2.5.3, Katello 4.1.1 on CentOS 7

The CentOS 8.4 BaseOS sync got an error tonight:

Actions::Pulp3::Repository::CreatePublication
...
PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Jul 2021 00:36:22 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"7df0e419-ebfb-4a1a-a0a5-3bc0c271493b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.dkrz.de", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/publications_rpm_api.rb:79:in `create_with_http_info'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/publications_rpm_api.rb:28:in `create'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/repository.rb:214:in `create_publication'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/lib/actions/pulp3/repository/create_publication.rb:15:in `invoke_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:19:in `run'
...

and

Actions::Katello::Repository::IndexContent
...
PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Jul 2021 00:36:23 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"7df0e419-ebfb-4a1a-a0a5-3bc0c271493b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.dkrz.de", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/ not found for repositoryversion."]

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:236:in `list_with_http_info'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:130:in `list'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:93:in `content_unit_list'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:106:in `fetch_content_list'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:75:in `block (2 levels) in pulp_units_batch_for_repo'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:69:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/services/katello/pulp3/pulp_content_unit.rb:69:in `block in pulp_units_batch_for_repo'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/concerns/pulp_database_unit.rb:162:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/concerns/pulp_database_unit.rb:162:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/concerns/pulp_database_unit.rb:162:in `import_for_repository'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/repository.rb:904:in `block (2 levels) in index_content'
...

As it’s involving the BaseOS repo and I now have issues updating systemd from baseos to the latest version, I guess it could be related:

# dnf update --nobest
...
 Problem: The operation would result in removing the following protected packages: systemd, systemd-udev
...
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 systemd         x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  3.6 M
 systemd         x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  3.6 M
 systemd-libs    x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  1.1 M
 systemd-libs    x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  1.1 M
 systemd-pam     x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  468 k
 systemd-pam     x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  469 k
 systemd-udev    x86_64 239-45.el8            ORG_centos8_BaseOS_x86_64  1.4 M
 systemd-udev    x86_64 239-45.el8_4.2        ORG_centos8_BaseOS_x86_64  1.4 M

See also Katello 4.1 and Rocky Linux 8 - not ready for prime time yet it seems

jeremylenz · July 22, 2021, 7:49pm

For anyone stumbling on this post in the future: Seems like this is due to a Pulp issue, Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp

see also the post linked above.

gvde · July 23, 2021, 4:51am

The update problem with the message

Problem: The operation would result in removing the following protected packages: systemd, systemd-udev

is related to the pulp issue.

I am not sure if the sync error has anything to do with that. It’s probably more a coincidence that it happened on the baseos repo on the same day there are some updates in the baseos where the missing filelists become relevant. I have updated to 4.1.1/pulpcore 3.14 on July 16th so technically the pulp issue was there since then but it’s only noticeable when new updates are synced which break dependencies due to missing filelists…

gvde · July 23, 2021, 5:11am

I am trying to find out more about those sync errors, but I don’t find much. All I have found so far is in the pulpcore-api journal:

pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:21 +0000] "GET /pulp/api/v3/tasks/77e069d2-4a4c-4bd8-bd9f-b75727c713cb/ HTTP/1.1" 200 2516 "-" "OpenAPI-Generator/3.14.1/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]: django.request:WARNING: Bad Request: /pulp/api/v3/publications/rpm/rpm/
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "POST /pulp/api/v3/publications/rpm/rpm/ HTTP/1.1" 400 67 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "GET /pulp/api/v3/distributions/rpm/rpm/?base_path=ORG%2FLibrary%2Fcustom%2Fcentos8%2FBaseOS_x86_64 HTTP/1.1" 200 549 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/84710568-2b25-4e5a-a79a-fcf150ff7eee/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "GET /pulp/api/v3/tasks/efa9721f-a670-41c2-b542-aaab741656ae/ HTTP/1.1" 200 510 "-" "OpenAPI-Generator/3.14.1/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "GET /pulp/api/v3/tasks/efa9721f-a670-41c2-b542-aaab741656ae/ HTTP/1.1" 200 562 "-" "OpenAPI-Generator/3.14.1/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:22 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/84710568-2b25-4e5a-a79a-fcf150ff7eee/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.13.3/ruby"
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]: django.request:WARNING: Bad Request: /pulp/api/v3/content/rpm/packages/
pulpcore-api[10010]: pulp [7df0e419-ebfb-4a1a-a0a5-3bc0c271493b]:  - - [21/Jul/2021:00:36:23 +0000] "GET /pulp/api/v3/content/rpm/packages/?arch__ne=src&fields=pulp_href%2Cname%2Cversion%2Crelease%2Carch%2Cepoch%2Csummary%2Cis_modular%2Crpm_sourcerpm%2Clocation_href%2CpkgId&limit=2000&offset=0&repository_version=%2Fpulp%2Fapi%2Fv3%2Fpublications%2Frpm%2Frpm%2Fd44e483c-977c-48c1-93b7-9e26e64e7874%2F HTTP/1.1" 400 112 "-" "OpenAPI-Generator/3.13.3/ruby"

gvde · July 23, 2021, 5:33am

O.K. There seems to be something broken. Looking at the error of IndexContent above:

Response body: ["URI /pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/ not found for repositoryversion."]

The publication is there:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.dkrz.de/pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/' | python -m json.tool 
{
    "gpgcheck": 0,
    "metadata_checksum_type": "unknown",
    "package_checksum_type": "unknown",
    "pulp_created": "2021-07-21T00:34:59.582368Z",
    "pulp_href": "/pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/",
    "repo_gpgcheck": 1,
    "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
    "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/3/",
    "sqlite_metadata": true
}

But the repository listed there is missing:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.dkrz.de/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/' | python -m json.tool 
{
    "detail": "Not found."
}

This looks to me as if there is something broken.

If I look into the pulpcore database I can find the repository with uuid 3f9dc526-a51c-4a25-9547-95f82bedb3ee:

pulpcore=# select * from core_publication where pulp_id = 'd44e483c-977c-48c1-93b7-9e26e64e7874';
               pulp_id                |         pulp_created          |       pulp_last_updated       | pulp_type | complete | pass_through |        repository_version_id         
--------------------------------------+-------------------------------+-------------------------------+-----------+----------+--------------+--------------------------------------
 d44e483c-977c-48c1-93b7-9e26e64e7874 | 2021-07-21 02:34:59.582368+02 | 2021-07-21 02:35:03.083383+02 | rpm.rpm   | t        | f            | 67cca843-6639-479b-805a-f912cd4cd9f4
(1 row)

pulpcore=# select * from core_repositoryversion where pulp_id = '67cca843-6639-479b-805a-f912cd4cd9f4';
               pulp_id                |         pulp_created          |       pulp_last_updated       | number | complete | base_version_id |            repository_id             
--------------------------------------+-------------------------------+-------------------------------+--------+----------+-----------------+--------------------------------------
 67cca843-6639-479b-805a-f912cd4cd9f4 | 2021-07-21 02:33:18.980391+02 | 2021-07-21 02:34:59.342965+02 |      3 | t        |                 | 3f9dc526-a51c-4a25-9547-95f82bedb3ee
(1 row)

pulpcore=# select * from core_repository where pulp_id = '3f9dc526-a51c-4a25-9547-95f82bedb3ee';
               pulp_id                |         pulp_created          |       pulp_last_updated       |                                              name                                              | description | next_version | pulp_type | remote_id | retained_versions | user_hidden 
--------------------------------------+-------------------------------+-------------------------------+------------------------------------------------------------------------------------------------+-------------+--------------+-----------+-----------+-------------------+-------------
 3f9dc526-a51c-4a25-9547-95f82bedb3ee | 2021-07-07 14:33:17.628538+02 | 2021-07-22 22:36:04.957548+02 | ../../../AppStream/x86_64/os/-31410651d4c07a347a6dbba79cfb4df7c6418e39b0a19c70a84eff2c8b3b8953 |             |            5 | rpm.rpm   |           |                   | f
(1 row)

There is also a rpm publication and repository:

pulpcore=# select * from rpm_rpmpublication where publication_ptr_id = 'd44e483c-977c-48c1-93b7-9e26e64e7874';
          publication_ptr_id          | metadata_checksum_type | package_checksum_type | gpgcheck | repo_gpgcheck | sqlite_metadata 
--------------------------------------+------------------------+-----------------------+----------+---------------+-----------------
 d44e483c-977c-48c1-93b7-9e26e64e7874 | unknown                | unknown               |        0 |             1 | t
(1 row)

pulpcore=# select * from rpm_rpmrepository where repository_ptr_id = '3f9dc526-a51c-4a25-9547-95f82bedb3ee';
          repository_ptr_id           | sub_repo | metadata_signing_service_id |         last_sync_remote_id          | last_sync_repo_version | last_sync_revision_number |                                                                                        original_checksum_types                       
                                                                 | retain_package_versions |                    last_sync_repomd_checksum                     | autopublish | gpgcheck | metadata_checksum_type | package_checksum_type | repo_gpgcheck | sqlite_metadata 
--------------------------------------+----------+-----------------------------+--------------------------------------+------------------------+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------+-------------------------+------------------------------------------------------------------+-------------+----------+------------------------+-----------------------+---------------+-----------------
 3f9dc526-a51c-4a25-9547-95f82bedb3ee | t        |                             | 69abed5c-b1e1-452a-b58b-f0cc23548924 |                      4 | 8.4.2105                  | {"group": "sha256", "other": "sha256", "modules": "sha256", "primary": "sha256", "group_xz": "sha256", "other_db": "sha256", "filelis
ts": "sha256", "primary_db": "sha256", "filelists_db": "sha256"} |                       0 | c21c5d2410544fccf2dcc78ce0f472fd8ed9d8c3fc0de1a8f8061a72629a5c7e | f           |        0 | sha256                 | sha256                |             0 | f
(1 row)

I don’t know what exactly is expected in the database, but superficially it looks as if everything is there. So it is unclear to me why the api request for rpm repository cannot find it.

gvde · July 23, 2021, 5:43am

Another observation: if I check the number of rpm publications and repositories via api I get these counts:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.example.com/pulp/api/v3/publications/rpm/rpm/?limit=10000' | python -m json.tool
{
    "count": 350,
...
# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.example.com/pulp/api/v3/repositories/rpm/rpm/?limit=10000' | python -m json.tool
{
    "count": 253,
...

But in the database the repository count is different:

pulpcore=# select count(*) from rpm_rpmpublication ;
 count 
-------
   350
(1 row)

pulpcore=# select count(*) from rpm_rpmrepository ;
 count 
-------
   258
(1 row)

gvde · July 23, 2021, 5:56am

Another insight: Using the pulp api I have checked all repository URI returned in the publications list and check the repository URI for ‘Not found’ and I found only two publications which reference the “missing” repository 3f9dc526-a51c-4a25-9547-95f82bedb3ee:

        {
            "gpgcheck": 0,
            "metadata_checksum_type": "unknown",
            "package_checksum_type": "unknown",
            "pulp_created": "2021-07-22T20:36:04.962028Z",
            "pulp_href": "/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/",
            "repo_gpgcheck": 1,
            "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
            "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/4/",
            "sqlite_metadata": true
        },
...
        {
            "gpgcheck": 0,
            "metadata_checksum_type": "unknown",
            "package_checksum_type": "unknown",
            "pulp_created": "2021-07-21T00:34:59.582368Z",
            "pulp_href": "/pulp/api/v3/publications/rpm/rpm/d44e483c-977c-48c1-93b7-9e26e64e7874/",
            "repo_gpgcheck": 1,
            "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
            "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/3/",
            "sqlite_metadata": true
        },

Thus, for all other publications it is correct. It only this repository which is messed up.

gvde · July 23, 2021, 6:09am

The “missing” repository seems to be the CentOS BaseOS repo:

pulpcore=# select core_remote.* from rpm_rpmrepository left join core_remote on ( last_sync_remote_id = core_remote.pulp_id ) where repository_ptr_id = '3f9dc526-a51c-4a25-9547-95f82bedb3ee';
               pulp_id                |         pulp_created          |       pulp_last_updated       | pulp_type |         name         |                         url                         | ca_cert | client_cert | client_key | tls_validation
 | proxy_url | username | password | download_concurrency |  policy   | connect_timeout | sock_connect_timeout | sock_read_timeout | total_timeout | headers | rate_limit | proxy_password | proxy_username | max_retries 
--------------------------------------+-------------------------------+-------------------------------+-----------+----------------------+-----------------------------------------------------+---------+-------------+------------+---------------
-+-----------+----------+----------+----------------------+-----------+-----------------+----------------------+-------------------+---------------+---------+------------+----------------+----------------+-------------
 69abed5c-b1e1-452a-b58b-f0cc23548924 | 2021-07-07 07:10:41.527911+02 | 2021-07-07 14:33:16.042855+02 | rpm.rpm   | BaseOS_x86_64-172119 | http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ |         |             |            | t             
 |           |          |          |                   20 | immediate |                 |                      |                   |           300 |         |            |                |                |            
(1 row)

I guess it all comes down to why the pulp api doesn’t find the repository even though it’s in the database?

gvde · July 23, 2021, 7:46am

I have just noticed that last night another baseos sync failed with this error:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 22 Jul 2021 20:36:46 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"ded10b33-c063-4471-9182-3b62facbd36b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Thu, 22 Jul 2021 20:36:47 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"ded10b33-c063-4471-9182-3b62facbd36b", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/ not found for repositoryversion."]

It goes into the same repository:

# curl -s --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman.example.com/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/' | python -m json.tool
{
    "gpgcheck": 0,
    "metadata_checksum_type": "unknown",
    "package_checksum_type": "unknown",
    "pulp_created": "2021-07-22T20:36:04.962028Z",
    "pulp_href": "/pulp/api/v3/publications/rpm/rpm/1ad1ad0e-c9c1-42ef-94f6-0b188c29f72d/",
    "repo_gpgcheck": 1,
    "repository": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/",
    "repository_version": "/pulp/api/v3/repositories/rpm/rpm/3f9dc526-a51c-4a25-9547-95f82bedb3ee/versions/4/",
    "sqlite_metadata": true
}

So it’s nothing new. However, I run this sync task every four hours. So this error only occurs occasionally…

gvde · July 23, 2021, 7:58am

As so far it seems to involve only pulp I have created an pulp issue: Issue #9133: RPM repository sync error - Pulp

gvde · July 24, 2021, 4:31pm

There seems to be something worse: I have just noticed that 5 days ago the sync for the baseos of CentOS 8 Stream (not 8.4 as above) ended with a warning, and again this morning. I have the 8 Stream repos only for testing, i.e. currently I only have one client and it’s using the Library environment.

5 days ago:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 19 Jul 2021 02:00:55 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"b05492c2-dc3a-4cfb-9099-22674dcc652d", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 19 Jul 2021 02:00:57 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"b05492c2-dc3a-4cfb-9099-22674dcc652d", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/8c78f8f7-4278-44d7-bef9-30ce30843c4c/ not found for repositoryversion."]

Now that publication 8c78f8f7-4278-44d7-bef9-30ce30843c4c really doesn’t exist. It’s not found via API nor it is in the core_publication or rpm_rpmpublication tables.

This morning:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:40 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"67", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"repository_version":["Invalid hyperlink - Incorrect URL match."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:41 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Sat, 24 Jul 2021 02:03:41 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"ed16fdfd-cd0a-48c5-8bb9-5f40df4c4d46", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/ad1067e2-4255-4898-b55e-d2341e59cb99/ not found for repositoryversion."]

Now that publication ad1067e2-4255-4898-b55e-d2341e59 has the same issue as the baseos for 8.4 above: the publication references a repository which exists in the database but the API says “Not found”.

What’s worse: the client cannot access the repository anymore:

# dnf check-update
...
Errors during downloading metadata for repository 'ORG_centos8stream_BaseOS_x86_64_os':
  - Status code: 404 for https://foreman.example.com/pulp/repos/ORG/Library/custom/centos8stream/BaseOS_x86_64_os/repodata/repomd.xml (IP: xxxx:xxxx::1)
Error: Failed to download metadata for repo 'ORG_centos8stream_BaseOS_x86_64_os': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

sjansen · July 26, 2021, 7:18am

I have the same issues on my Foreman/Katello installation 2.5.2/ Katello 4.1. This is with AlmaLinux and not CentOS.

Sync sometimes fails with the error above.

Setup a repo, sync repo and delete repo also breaks content view creation with

Error message: the server returns an error
HTTP status code: 400
Response headers: {“Date”=>“Fri, 23 Jul 2021 14:54:37 GMT”, “Server”=>“gunicorn”, “Content-Type”=>“application/json”, “Vary”=>“Accept,Cookie”, “Allow”=>“GET, POST, HEAD, OPTIONS”, “X-Frame-Options”=>“SAMEORIGIN”, “Content-Length”=>“112”, “Correlation-ID”=>“1de9b280-f297-41fd-8ce2-3cb330660fa4”, “Access-Control-Expose-Headers”=>“Correlation-ID”, “Via”=>“1.1 XXX”, “Connection”=>“close”}
Response body: [“URI /pulp/api/v3/publications/rpm/rpm/700e7182-674c-41d2-ab94-fc3465a834a5/ not found for repositoryversion.”]

gvde · July 26, 2021, 7:44am

@jeremylenz Should I create am issue in the foreman/katello redmine as well for this?

gvde · July 26, 2021, 7:50am

Can you please check which repository exactly? For me, currently only the BaseOS of CentOS 8 and 8 Stream are affected. It would be interesting so see, if it’s the BaseOS for you, too, or a different one.

gvde · July 26, 2021, 7:55am

Create redmine issue Bug #33136: CentOS 8 repository sync errors - Katello - Foreman as it obviously affects katello.

sjansen · July 26, 2021, 7:55am

Sorry cant remember, this issue broke my installation with a stuck publish content view task, i cancled it with foreman-rake but the half baked content view never disappeard i was unable to create a new one, so i rolled that machine back.

jruk · July 26, 2021, 5:31pm

I had this issue as well on foreman 2.5.2 / katello 4.1.1. when syncing CentOS 8 OS. To get around it, I went to Products, selected CentOS 8 then from the “Select Action” drop-down, I chose “Advanced Sync”. From here, I then selected “Complete Sync” and once finished this fixed the issue for me. I was also able to subsequently publish a new version of the associated content view.

HTH

gvde · July 26, 2021, 7:31pm

It seems to me as if the sync error only occurs when there are actually some new packages available. If nothing changes the sync goes through. I sync the BaseOS repo every 4 hours and not every sync ends in an error.

jruk · July 27, 2021, 6:51pm

I found this to be the case as well. However, even after several “successful” syncs of BaseOS, when I subsequently attempted to publish a new version of the content view then the same http 400 error would be displayed as during the sync previously. It’s very strange but fortunately for me “Complete Sync” appeared to fix the issue and I have been able to apply updates to my hosts.

pjayKHV · July 28, 2021, 7:24am

Good day all. Same problem here. Fresh install Foreman 2.5.2 + Katello 4.1 over CentOS 8 Stream. Unable to sync CentOS 8 Stream repos. Some go through though.