Katello 4.2.2 - Not syncing all repositories to content proxy

gvde · February 8, 2022, 12:31pm

Problem:

I have upgraded to 4.2.2/3.0.1 last week. This morning I have noticed that some updates for our perfsonar server are available, however, it cannot download the rpms from the content proxy because of a 404. According to the metadata rpms are supposed to be in the RPMS directory, however, the synced content has everything in the packages directory.

I have did a complete sync and also tried republish metadata, then published and promoted the content view which I use, but it seems the repository doesn’t change on the content proxy, even if I start a complete sync of the content proxy.

For instance in the repomd.xml on my main katello server the repository in the content view has

<revision>1644310078</revision>

from today. On the content proxy it remains at

<revision>1644248069</revision>

Nothing I have tried got the repository updated on the content proxy.

Expected outcome:
Repository updated on the content proxy.

Foreman and Proxy versions:
3.0.1

Foreman and Proxy plugin versions:
Katello 4.2.2

Distribution and version:
CentOS 7.9

jeremylenz · February 8, 2022, 4:22pm

Hi @gvde

May we have some more info about the 404 you’re getting? Is there a backtrace?

jeremylenz · February 8, 2022, 4:29pm

You should be able to see a better traceback by checking journalctl -u pulpcore-worker@*

gvde · February 8, 2022, 5:14pm

That’s the thing: everything seems to work normally. There is no traceback.

An example: one pending update is perfsonar-core-4.4.3-1.el7.noarch. However a update fails:

# yum -y update perfsonar-core.noarch
...
perfsonar-core-4.4.3-1.el7.noa FAILED                                          
https://foreman-content.example.com/pulp/content/ORG/Production/centos7-epel7/custom/perfsonar/perfsonar-el7/RPMS/perfsonar-core-4.4.3-1.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.

RPMS/perfsonar-core-4.4.3-1.el7.noarch.rpm is the path mentioned in repodata/f9d176ea6fa1dc923dcaea11834416e9be6eb866296b60dfc50252e9dac1a099-primary.xml.gz on the content proxy:

  <location href="RPMS/perfsonar-core-4.4.3-1.el7.noarch.rpm"/>

The repository directory contains three directories/files on the content proxy:

    config.repo
    packages/
    repodata/

The correct path to perfsonar-core would be packages/perfsonar-core-4.4.3-1.el7.noarch.rpm

Now the same directory on the main server contains:

    Packages/
    config.repo
    repodata/

Here repodata/0703588fcb6583294605ebc1b2c1b6bd5b26c6f77a5ccc391a453d0483f8571a-primary.xml.gz points to

<location href="Packages/p/perfsonar-core-4.4.3-1.el7.noarch.rpm"/>

which is where the rpm actually is found. On the main server it all looks good.

But I don’t get it synced to the content proxy, not even with a complete sync. It doesn’t change. Metadata and directories remain the same on the content proxy. And content of metadata doesn’t match real paths.

The upstream repo URL is Index of /rpms/el7/x86_64/main That’s the origin of the RPMS directory. That’s also where the metadata is from: it contains the exact copy of f9d176ea6fa1dc923dcaea11834416e9be6eb866296b60dfc50252e9dac1a099-primary.xml.gz which is on the content proxy.

So basically: the metadata was mirrored, but the rpms were organized differently on the katello server and the content proxy (the main server had the same issue originally). Now on the katello server I have managed to get correct metadata doing a complete sync of the repository and republish metadata, then published my content view and promoted.

The content proxy, however, remains out of sync, even after a complete sync.

jeremylenz · February 8, 2022, 6:02pm

If you look at the Dynflow task details for this complete sync, do you see sync steps for each repository?

gvde · February 8, 2022, 7:27pm

Yes. I can find this in the pulpcore database on the content proxy:

pulpcore=# select pulp_id, name, url from core_remote where url like '%perfsonar%';
               pulp_id                |                              name                               |                                                url                                                 
--------------------------------------+-----------------------------------------------------------------+----------------------------------------------------------------------------------------------------
 09f8d5be-af3c-4eb7-8124-7628c8f294a3 | 1-centos7-epel7-Production-a74843f9-144f-4651-9ef7-23d0c759e259 | https://foreman.example.com/pulp/content/ORG/Production/centos7-epel7/custom/perfsonar/perfsonar-el7/
 cb4436b9-5ae4-4189-914e-55526bcb0ca5 | 1-centos7-epel7-Testing-a74843f9-144f-4651-9ef7-23d0c759e259    | https://foreman.example.com/pulp/content/ORG/Testing/centos7-epel7/custom/perfsonar/perfsonar-el7/
(2 rows)

Searching for 09f8d5be-af3c-4eb7-8124-7628c8f294a3 in the dynflow task output, I can find the sync. The repository contains 229 rpms. So that’s correct, too.

 837: Actions::Pulp3::CapsuleContent::Sync (success) [ 11.57s / 0.70s ]

Queue: default
Started at: 2022-02-08 10:17:24 UTC
Ended at: 2022-02-08 10:17:35 UTC
Real time: 11.57s
Execution time (excluding suspended state): 0.70s

Input:

---
repository_id: 12021
smart_proxy_id: 7
options:
  skip_metadata_check: true
remote_user: admin
remote_cp_user: admin
current_request_id: 9211fcc4-3fcb-48f1-bcdc-d3d799baa905
current_timezone: Europe/Berlin
current_organization_id: 1
current_location_id: 2
current_user_id: 5

Output:

---
smart_proxy_history_id: 20141
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/7fe3bc8c-bc3e-4dd4-996a-a3e6fd8ec42f/"
  pulp_created: '2022-02-08T10:17:24.951+00:00'
  state: completed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: 9211fcc4-3fcb-48f1-bcdc-d3d799baa905
  started_at: '2022-02-08T10:17:30.982+00:00'
  finished_at: '2022-02-08T10:17:32.910+00:00'
  worker: "/pulp/api/v3/workers/a2569c83-b909-4b73-aa71-68fd6150bd29/"
  child_tasks: []
  progress_reports:
  - message: Downloading Metadata Files
    code: sync.downloading.metadata
    state: completed
    done: 5
  - message: Downloading Artifacts
    code: sync.downloading.artifacts
    state: completed
    done: 0
  - message: Associating Content
    code: associating.content
    state: completed
    done: 0
  - message: Parsed Packages
    code: sync.parsing.packages
    state: completed
    done: 229
  - message: Parsed Advisories
    code: sync.parsing.advisories
    state: completed
    total: 0
    done: 0
  - message: Un-Associating Content
    code: unassociating.content
    state: completed
    done: 0
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/remotes/rpm/rpm/09f8d5be-af3c-4eb7-8124-7628c8f294a3/"
  - "/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/"
task_groups: []
poll_attempts:
  total: 1
  failed: 0

839: Actions::Pulp3::CapsuleContent::GenerateMetadata (success) [ 0.27s / 0.27s ]

Queue: default
Started at: 2022-02-08 10:17:41 UTC
Ended at: 2022-02-08 10:17:41 UTC
Real time: 0.27s
Execution time (excluding suspended state): 0.27s
Input:

---
repository_id: 12021
smart_proxy_id: 7
options:
  skip_metadata_check: true
  sync_task_output:
  - pulp_href: "/pulp/api/v3/tasks/7fe3bc8c-bc3e-4dd4-996a-a3e6fd8ec42f/"
    pulp_created: '2022-02-08T10:17:24.951+00:00'
    state: completed
    name: pulp_rpm.app.tasks.synchronizing.synchronize
    logging_cid: 9211fcc4-3fcb-48f1-bcdc-d3d799baa905
    started_at: '2022-02-08T10:17:30.982+00:00'
    finished_at: '2022-02-08T10:17:32.910+00:00'
    worker: "/pulp/api/v3/workers/a2569c83-b909-4b73-aa71-68fd6150bd29/"
    child_tasks: []
    progress_reports:
    - message: Downloading Metadata Files
      code: sync.downloading.metadata
      state: completed
      done: 5
    - message: Downloading Artifacts
      code: sync.downloading.artifacts
      state: completed
      done: 0
    - message: Associating Content
      code: associating.content
      state: completed
      done: 0
    - message: Parsed Packages
      code: sync.parsing.packages
      state: completed
      done: 229
    - message: Parsed Advisories
      code: sync.parsing.advisories
      state: completed
      total: 0
      done: 0
    - message: Un-Associating Content
      code: unassociating.content
      state: completed
      done: 0
    created_resources: []
    reserved_resources_record:
    - "/pulp/api/v3/remotes/rpm/rpm/09f8d5be-af3c-4eb7-8124-7628c8f294a3/"
    - "/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/"
  contents_changed: true
remote_user: admin
remote_cp_user: admin
current_request_id: 9211fcc4-3fcb-48f1-bcdc-d3d799baa905
current_timezone: Europe/Berlin
current_organization_id: 1
current_location_id: 2
current_user_id: 5

Output:

---
pulp_tasks: []
task_groups: []

Justin_Sherrill · February 8, 2022, 9:09pm

@gvde it sounds like the distribution in pulp3 on the smart proxy is referring to an older repository version. Or else a new repository version was never created.

On your katello server, it’d be interesting to run:

curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://$SMART_PROXY_HOSTNAME/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/

Where your smart proxy’s hostname replaces $SMART_PROXY_HOSTNAME.

That should include the latest version of the repository. You should see something like:

"latest_version_href":"/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/versions/1/"

Then lets see if there’s a publication that was created from it:

curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://$SMART_PROXY_HOSTNAME/pulp/api/v3/publications/rpm/rpm/?repository_version=$REPOSITORY_VERSION_HREF

replacing $REPOSITORY_VERSION_HREF with the latest version href from above.

That will give something like:

{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "pulp_href": "/pulp/api/v3/publications/rpm/rpm/776af70c-f09f-4eee-9d24-fa005bc73020/",
      "pulp_created": "2022-02-08T20:44:08.296472Z",
      "repository_version": "/pulp/api/v3/repositories/rpm/rpm/09b8d64d-2cba-4233-a534-5fa297467b83/versions/1/",
      "repository": "/pulp/api/v3/repositories/rpm/rpm/09b8d64d-2cba-4233-a534-5fa297467b83/",
      "metadata_checksum_type": "sha256",
      "package_checksum_type": "sha256",
      "gpgcheck": 0,
      "repo_gpgcheck": 0,
      "sqlite_metadata": false
    }
  ]
}

So we can get our publication_href of /pulp/api/v3/publications/rpm/rpm/776af70c-f09f-4eee-9d24-fa005bc73020/

Lets switch over to the pulp database On the smart proxy:

sudo -u postgres psql pulpcore
pulpcore=# select base_path, publication_id from core_distribution;

                  base_path                   |            publication_id            
----------------------------------------------+--------------------------------------
 Default_Organization/Library/custom/test/zoo | 776af70c-f09f-4eee-9d24-fa005bc73020

In theory you should see the base_path for that repository, along side the publication_id from the previous step. I’m curious what all yours shows.

gvde · February 9, 2022, 5:57am

O.K. RPM repository:

[root@foreman ~]# curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://foreman-content.example.com/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/  | python -m json.tool
{
    "autopublish": false,
    "description": null,
    "gpgcheck": 0,
    "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/versions/6/",
    "metadata_checksum_type": "sha256",
    "metadata_signing_service": null,
    "name": "1-centos7-epel7-Production-a74843f9-144f-4651-9ef7-23d0c759e259",
    "package_checksum_type": "sha256",
    "pulp_created": "2021-07-07T14:26:57.094511Z",
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/",
    "pulp_labels": {},
    "remote": null,
    "repo_gpgcheck": 0,
    "retain_package_versions": 0,
    "retained_versions": null,
    "sqlite_metadata": false,
    "versions_href": "/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/versions/"
}

Publication of the latest_version_href:

[root@foreman ~]# curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key 'https://foreman-content.example.com/pulp/api/v3/publications/rpm/rpm/?repository_version=/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/versions/6/' | python -m json.tool
{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "gpgcheck": 0,
            "metadata_checksum_type": "unknown",
            "package_checksum_type": "unknown",
            "pulp_created": "2022-02-08T05:00:16.110366Z",
            "pulp_href": "/pulp/api/v3/publications/rpm/rpm/f28363b1-dc93-4809-92e2-04a2e1fb8d57/",
            "repo_gpgcheck": 0,
            "repository": "/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/",
            "repository_version": "/pulp/api/v3/repositories/rpm/rpm/60d89e2e-a443-4d56-9c67-e191fd0739d4/versions/6/",
            "sqlite_metadata": true
        }
    ]
}

Searching in the database on the content proxy:

pulpcore=# select base_path, publication_id from core_distribution  where publication_id = 'f28363b1-dc93-4809-92e2-04a2e1fb8d57';
                          base_path                           |            publication_id            
--------------------------------------------------------------+--------------------------------------
 ORG/Production/centos7-epel7/custom/perfsonar/perfsonar-el7 | f28363b1-dc93-4809-92e2-04a2e1fb8d57
(1 row)

Justin_Sherrill · February 9, 2022, 2:46pm

that actually all looks fine The latest repository version has a publication created and its associated with a distribution.

So to be clear, when you navigate to 'https://HOSTNAME/ ORG/Production/centos7-epel7/custom/perfsonar/perfsonar-el7 /repodata/repomd.xml on the smart proxy you see a file that doesn’t match the same path on the katello server?

With mirrored metadata, they should match exactly.

I’m curious what your apache logs looks like, and if you see any POST requests to the
/pulp/api/v3/publications/rpm/rpm/ endpoint. Something like this might reveal it:

grep  POST /var/log/httpd/*  | grep 'publications/rpm/rpm'

gvde · February 9, 2022, 3:05pm

Yes. repomd.xml files on both servers are different:
https://foreman.example.com/pulp/content/ORG/Production/centos7-epel7/custom/perfsonar/perfsonar-el7/repodata/repomd.xml

<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1644310078</revision>
  <data type="primary">
    <checksum type="sha256">0703588fcb6583294605ebc1b2c1b6bd5b26c6f77a5ccc391a453d0483f8571a</checksum>
    <open-checksum type="sha256">c354922f4a19b221e7b8cd1409f53f1aec46a9748017f558c32279a330b87f2d</open-checksum>
    <location href="repodata/0703588fcb6583294605ebc1b2c1b6bd5b26c6f77a5ccc391a453d0483f8571a-primary.xml.gz"/>
    <timestamp>1644310078</timestamp>
    <size>70169</size>
    <open-size>513178</open-size>
  </data>
...

https://foreman-content.dkrz.de/pulp/content/DKRZ/Production/centos7-epel7/custom/perfsonar/perfsonar-el7/repodata/repomd.xml

<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1644248069</revision>
  <data type="primary">
    <checksum type="sha256">f9d176ea6fa1dc923dcaea11834416e9be6eb866296b60dfc50252e9dac1a099</checksum>
    <open-checksum type="sha256">62a3e2d0b6644619d1a6992c2b863b605a006a7357df13d3b8bacfc3a13dcbb1</open-checksum>
    <location href="repodata/f9d176ea6fa1dc923dcaea11834416e9be6eb866296b60dfc50252e9dac1a099-primary.xml.gz"/>
    <timestamp>1644248068</timestamp>
    <size>69657</size>
    <open-size>1024355</open-size>
  </data>
...

As are the others files, directory structure and the linked rpms which are broken on the content proxy.

There are none on the content proxy since the upgrade from Katello 4.1.4 to 4.2.2. (There are some on the main server, if that matters)

dodo · February 9, 2022, 4:44pm

same problem

ulp task error[Errno 2] No such file or directory: ‘./tmpahbdykvv’[Errno 2] No such file or directory: './tmpw4r8z_9n’Could not lookup a publication_href for repo 6593Pulp task error[Errno 2] No such file or directory: './tmpcw3qd5x6’Could not lookup a publication_href for repo 4590

how can i troubleshoot? thank you

gvde · February 9, 2022, 4:53pm

That’s nowhere the same problem. Please open a new thread.

dralley · February 10, 2022, 4:55am

This is potentially relevant:

It looks like the upstream repo http://software.internet2.edu/rpms/el7/x86_64/main/ has the same packages listed multiple times in multiple places (see: RPMS/ and packages/ have the same files listed, and the metadata includes both)

I had to guess, something like this is happening: Pulp is seeing both packages and only publishing one of them, and either A) the metadata includes both and DNF / Yum are picking “the other one” or B) the metadata includes only one of them, and publishes the link to the wrong one.

At the end of the day, this upstream repo metadata is terrible and asking for issues, but if either A or B is true then we need to be handling that better on the Pulp side, because this we come across a repo like this every month or two.

I don’t have time to confirm that theory at the moment but I’ll investigate further tomorrow.

gvde · February 10, 2022, 5:02am

Seriously: don’t hijack threads about other issues. Open a new thread!

dralley · February 10, 2022, 5:03am

@gvde, I am talking about your issue, please read my comment.

dralley · February 10, 2022, 5:11am

My point is, the metadata itself has a few problems which may contribute to Pulp getting confused. You say that the content proxy only has

    config.repo
    packages/
    repodata/

wheras Yum is giving you a 404 on an RPM with a path prefix of RPMS/

https://foreman-content.example.com/pulp/content/ORG/Production/centos7-epel7/custom/perfsonar/perfsonar-el7/RPMS/perfsonar-core-4.4.3-1.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

If the upstream repo has the exact same packages present twice in the metadata, with one copy in packages/ and one copy in RPMS/, and Pulp is only writing links to one copy of those, but Yum / DNF is picking the other one, then this feels like a potential explanation for your issue. Or maybe not, it could be something else. I’ll investigate further tomorrow morning.

Although even if this isn’t the root cause or the only cause of your issue, we should still try to reach out to get them to fix their metadata.

gvde · February 10, 2022, 5:18am

Your screenshot is a screenshot. It’s almost unreadable.
Please read my issue: My problem is that the repository doesn’t sync metadata between the main Katello server and the content proxy. The repository is looking good on the main server.

Yes, still another topic and not even for this board.

dralley · February 10, 2022, 2:24pm

@gvde I have read your issue. You are describing multiple problems and in any event it seems very likely to me that there could be multiple simultaneous causes for the behavior you are seeing. In any event, I am trying to help so maybe chill a bit?

You say the metadata on the content proxy has a “revision” of “1644248069”, which matches the current state of the upstream repo, and likewise the checksum of primary.xml is the same. But the layout is different. The upstream repo and therefore the metadata has an RPMS/ directory and the Pulp mirror does not. Yum hits a 404 on that file. The 404 could either be caused by

A) Yum picks a package but the location_href does not exist in the repository
B) This repo is on_demand, the content proxy is storing the wrong upstream URL for some reason, Pulp tries to download that file (on_demand) but hits a 404 and then proxies that forward to Yum.

Please help me determine whether it is A or B. I promise, we will get to the sync issue, but this is relevant to how that ought to be investigated.

Also could you provide the pulp component versions of the currently running processes on the content proxy pulp status | jq '.versions'?

gvde · February 10, 2022, 3:38pm

Well, maybe we have a basic misunderstanding, but I am describing only one problem: the repository isn’t synced from the main server to the content proxy. Any upstream issue doesn’t really bother me at the moment, because I ran a republish metadata for the repository on the main server, thus on the main server the repository is fully consistent with the metadata.

Yes, because the content proxy doesn’t sync from the main proxy. The content proxy still has the state of the repository sync from upstream because it doesn’t pick up the repository from the main server.

The server is connected to the content proxy which still has the upstream (broken) primary with the duplicates. The content proxy only has the packages path not the RPMs path. It seem pulp only puts the second path into its database (i.e. packages/, or processes the file in order and the second identical rpm overwrites the first entry) and yum prefers to use the first path (RPMS/). That’s why it breaks.

But that’s not my problem. While annoying and should be fixed upstream and/or properly handled in pulp, I have “fixed” the issue for the meantime by republishing meta data on the main server. Now I have a good presentation of the repository on the main server (even though it’s no mirror view anymore), but the content proxy doesn’t pick it up. That’s currently my problem.

The repository is not on demand. The content proxy is on demand. But it’s my understanding that this shouldn’t matter if I do a complete sync on the content proxy.
Pulp mirrors the repodata metadata but only mirrors the content in packages/ and not the duplicates in RPMS/. yum picks up the first entry from the metadata to find the rpm which points to RPMS while pulp only mirrors the second entry in the metadata. The packages/ path is also in the location_href in the rpm_package table on the content proxy.

It’s kind of neither nor. It’s mirrored upstream metadata containing duplicate entries for all rpms, one in RPMS/ and one in packages/ and pulp only picking up/presenting one of those, i.e. packages/
yum picks the first path from metadata which isn’t shown on pulp.
As I wrote before: I suspect pulp reads the metadata entry by entry. First it see the RPMs location_href and puts that into the database. Later it finds the same RPM with the exact same name and exact same checksum/hash but with the packages/ location_href and now changes the previous entry to the latest location_href, probably because it thinks the repository has been reorganized…

I get why this happens. There is even an index on the rpm_package table in the database enforcing this:

    "rpm_package_name_epoch_version_relea_c9003ffa_uniq" UNIQUE CONSTRAINT, btree (name, epoch, version, release, arch, checksum_type, "pkgId")

There is no “pulp” command… I can offer you this:

# rpm -qa *pulp* | sort
pulpcore-selinux-1.2.7-1.el7.x86_64
python3-pulp-ansible-0.9.0-2.el7.noarch
python3-pulp-certguard-1.4.0-3.el7.noarch
python3-pulp-container-2.8.1-0.2.el7.noarch
python3-pulpcore-3.14.9-1.el7.noarch
python3-pulp-deb-2.14.1-2.el7.noarch
python3-pulp-file-1.8.2-2.el7.noarch
python3-pulp-rpm-3.14.8-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-3.1.0-1.fm2_6.el7.noarch

It’s the latest available version for 4.2/3.0…

But all this doesn’t explain why the content proxy doesn’t pick up the perfectly correct, republished repository from the main server.

dralley · February 10, 2022, 10:53pm

It’s fair to focus on the second issue (the sync issue), but it doesn’t make discussion of the first issue irrelevant. Piecing together an step-by-step picture of what happened is important to actually resolving your problems if the two issues are interrelated somehow. I’m just trying to be methodical, not ignore your concerns. And it sounds like the 404s came first, and the sync issues were noticed when trying to work around that.

Anyway - I just remembered a discussion from a few months ago and it seems to describe what is happening here pretty well.

github.com/pulp/pulp_rpm

Comment by dralley to Address issues with sync optimization

pulp:master ← dralley:new-issues

@jlsherrill Heads up. Back during the meeting about mirrored metadata, we had d…iscussed whether it should be possible for mirrored-metadata publications to be re-created even if the repositories themselves haven't changed, and no new repository version is created. And I think we agreed to do that for 7.0 / 7.1? (according to the document). We can actually do that for 3.16.1. Let me know if this would break Katello because we could back off on that change, but if it's beneficial and wouldn't break anything we can keep it. What happens if mirror=True in 3.14 and 3.15 is: * If optimize=False OR optimize=True and the sync can't be optimized because some option changed - a new publication will be created ONLY IF a new repository version was created for the main repo. If it was not, no new publication. * If optimize=True and the main repo (the one where the .treeinfo is located) is the same as the last sync, nothing happens. No new sync happens, no new publication. What happens in 3.16.1 if mirror=True / sync_policy=mirror_complete will be: * If optimize=False, or if ANY of the repos (main repo + any .treeinfo repos) have changed since the last sync, all of the syncs will happen, and a new publication will be created regardless of whether a new repository version was created for the main repo. * If optimize=True and ALL of the repos (main repo + any .treeinfo subrepos) are the same as the last sync, nothing happens. No new sync happens, no new publication.

TL;DR, the repository (probably) synced perfectly fine, it just didn’t replace the metadata because it thought everything was the same. And this is addressed in a newer version, but wasn’t backported because it’s tied in with a bunch of of significant changes of behavior that didn’t fit the risk profile for a pre-emptive backport.

That explains why only this repo was affected, and only after regenerating the metadata and trying to sync again.

So the workaround would be: just regenerate the metadata on the content proxy, too.

I will see if the relevant portion of that PR can be split off and backported to 3.14.