Rewriting the bullet points for clarity, it won’t let me edit anymore.
The content proxy is being synced and then afterwards the layout of the repos on the main system changes, invalidating the stored URLs, and the content proxy isn’t being re-synced after that.
The content proxy is being synced against the correct layout (or both layouts), but isn’t keeping all of the potential URLs as it is supposed to (multiple URLs are stored in order for the same artifacts, they get tried in-order at download time. This is because the same package could be available from different repos.)
The content proxy is getting updated with the new URLs, but something is going wrong when it tries to go through them to download the package. It might be trying the first, probably getting a 404, and giving up before trying the others, which it should be doing.
I have read something about “location_href” in the change log so I though it might have to do with that.
As far as I can tell, all my repositories are set to “mirror on sync” in the gui. My perfsonar repository, for instance, is set so. If I look into /pulp/content/ORG/Library/custom/perfsonar/perfsonar-el7/ on the main server after it ran another sync this morning, I still see the Packages/letter/ directories. That doesn’t match. (O.K. maybe the optimized sync found nothing needs to be synced and thus left it as it is for the moment).
So basically, a regenerate metadata would temporarily change the layout of all repositories on the main server, unless the external repositories already uses Packages/${letter}/${full_name}?
I am not sure about the layout change on the external server. As I wrote before, I also have problems with rpm from the foreman 2.5 release repository and I guess you guys know best whether or not you have changed the layout recently. I usually don’t keep track of the external repo organization.
I did, however, run a regenerate metadata on most/all on my repositories on the main server because I was hoping it would fix my problems (after and advanced or repair sync didn’t help either). So possibly, that caused more problems…
Let me know if I can pull any data from my systems for troubleshooting…
Where do I find/How do I build the URL of the upstream RPM in the pulp database? That’s unclear to me.
I have read something about “location_href” in the change log so I though it might have to do with that.
I had wondered that initially but the change post-dates your initial report, and after reviewing the patch again I didn’t see a way for that to have created the issue.
The “mirror” functionality was introduced in pulp_rpm 3.13.0, and that’s when the “layout” change would have happened - but I believe Katello may still have been doing manual publishes after mirror syncs for some period of time, and in that event it would have looked the same as it would after a manual metadata regeneration. I do not know when they changed/fixed that.
I still see the Packages/letter/ directories. That doesn’t match. (O.K. maybe the optimized sync found nothing needs to be synced and thus left it as it is for the moment).
That is very possible.
So basically, a regenerate metadata would temporarily change the layout of all repositories on the main server, unless the external repositories already uses Packages/${letter}/${full_name}?
Yes. Doing a manual publish will always use a standard layout, but mirroring the external repo will give you whatever layout the external layout used.
I am not sure about the layout change on the external server. As I wrote before, I also have problems with rpm from the foreman 2.5 release repository and I guess you guys know best whether or not you have changed the layout recently. I usually don’t keep track of the external repo organization.
All the problems are with the content proxy, yes? If so, then I don’t expect the upstream repo layout would make a difference. It’s much more likely to be a change w/ on the main Pulp / Katello server.
Honestly we should probably reconsider the implications of “regenerate metadata” for mirrored repos, maybe it should not be available at all. Since we’re just cloning the original metadata, it is unlikely to solve any problems.
Where do I find/How do I build the URL of the upstream RPM in the pulp database? That’s unclear to me.
select * from core_remoteartifact;
The table has columns for “sha1”, “sha256” and so forth. You can filter by those to look at the records for individual RPMs, since the RPM metadata contains this for each package.
I just did skip metadata sync and a repair sync on the perfsonar repository. But that didn’t change a thing with the layout on the main server.
O.K. Interesting. So basically, if I do a regenerate it’ll go from external layout, to standard layout and later at some point back to external layout?
Well, I guess not. As I have mentioned above (post 14), after my update to pulp_rpm 3.14.3 there were some new updates to the perfsonar repository. After that sync the main server didn’t work either. But for the main server the regenerate metadata fixed the problem.
But that I have only noticed because I have checked manually. Currently, all my clients connect to the content proxy and only my three foreman servers connect to the main server. So if there were more problems on the main server I wouldn’t necessarily notice. I did notice these issues with the foreman release rpms, though, so that was the main server. But for the main server, I can run the regenerate metadata and that helps.
Absolutely. If it changes the directory layout (and in particular back and forth) it shouldn’t be a general option. But still, something like that is necessary, as it did - at least for me - fix some of my problems, just not for the content on the proxy. The complete sync (aka metadata skip sync) doesn’t seem to fix everything at this point.
Great. I have just checked the database on the content proxy:
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%perfsonar-el7%Packages%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-----------------------------+-------
2021-07-09 07:01:24.207283+02 | 2021-07-09 07:01:29.003191+02 | 2021-07-09 07:01:24.207315+02 | 2021-07-09 07:01:29.0032+02 | 2142
(1 row)
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%perfsonar-el7%packages%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-09-08 06:44:05.836841+02 | 2021-09-08 06:45:05.048783+02 | 2021-09-08 06:44:05.836889+02 | 2021-09-08 06:45:05.048792+02 | 454
(1 row)
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%perfsonar-el7%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-07-09 07:01:24.207283+02 | 2021-09-08 06:45:05.048783+02 | 2021-07-09 07:01:24.207315+02 | 2021-09-08 06:45:05.048792+02 | 2596
(1 row)
I sync two LEs “Testing” and “Production” to the content proxy and for both it’s half of the above. So that’s consistent.
I have randomly checked some with the “packages” in the url and those are broken. The current layout on the main server is:
Packages/
config.repo
repodata/
As you can see from the timestamps, the 454 broken ones are those synced two days ago. All others were two months old (which well can be as the perfsonar repo doesn’t get updates too often).
So it seems to me, I have had “Packages/{letter}/” on the main server at some point before the updates two days ago. The new updates caused a “mirror on sync” reorganizing everything to the external layout.
However, it seems they have changed the directory layout on the perfsonar repository as well during that time. They still have a RPMS and a packages directory with the identical content. As I have seen RPMS urls in the logs I guess that was the layout two months ago and now they have adjusted their metadata to point to the packages directory?
Via the standard, automated content proxy sync after I have promoted a new CV version, it updated the content proxy with the external layout URLs. But as I had those broken URLs on the main server and I did a regenerate metadata at some point, it messed it up.
So a “Complete Sync” neither on the repository on the main server nor for the complete content proxy does mitigate issues with the directory layout for RPMs which are already in the database. Can that be? For the content proxy that seems right as you can see in the database above…
For the foreman release on the content proxy pulpcore database:
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%/foreman/2_5_el7_x86_64/%' and url not like '%/foreman/2_5_el7_x86_64/Packages/_/%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-08-24 13:24:16.562816+02 | 2021-08-31 06:13:19.377518+02 | 2021-08-24 13:24:16.562847+02 | 2021-08-31 06:13:19.377525+02 | 54
(1 row)
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%/foreman/2_5_el7_x86_64/%' and url like '%/foreman/2_5_el7_x86_64/Packages/_/%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-07-07 16:31:38.788943+02 | 2021-07-20 08:17:03.716721+02 | 2021-07-07 16:31:38.788972+02 | 2021-07-20 08:17:03.716732+02 | 1370
(1 row)
O.K. Here is one way on how to get a 404 after changing paths.
Create a test (external) repository on a web server with rpms in a Packages subdirectory: Package/abc.rpm
Important: make sure those RPMs do not already exists in any form on the server. I think pulp can associate RPMs by hash and thus would use a local copy if hash fits regardless of where it moved in the external repository.
Now create the repository in katello with “mirror on sync” and “on demand” download policy and sync.
This creates entries in core_remoteartifact with Packages/ urls.
Now “restructure” your external repository, e.g. move all RPMS from the Packages subdirectory to the root and rerun “createrepo” to update metadata.
Sync again: nothing changes in the database. core_remoteartifact remains as it is. If I now try to access the rpm through pulp I get a 404 because it tries to download using the Packages subdirectory.
I guess, something similar happens between the main server and the content proxy (with on demand) if you regenerate metadata on the main server…
And if this is confirmed then this would be a Pulp bug, because it should be capturing the new locations as well. I’ll focus some attention on that aspect.
So it seems it backported to pulpcore 3.14.7. I hope it doesn’t take too long to get it into katello.
The usual question: is there some cleanup or resync necessary after the updated version is installed? How do those broken URLs get repaired or cleaned up? In particular for the content proxy where I can’t really do much except an optimized or complete sync.
So I guess, this translate to a “Complete Sync” of the products (or repositories) on the main Katello server and a “Complete Sync” of the content proxy on the smart proxy page.
But we completely agree that it’s painful to run into these errors on the default settings. Mirroring the metadata has a lot of benefits (repository signatures stay intact) but also a lot of limitations, such as this one, unfortunately, which weren’t anticipated.
We’re working on some sync modes that let you mirror the content without mirroring the metadata.
