@gvde The only thing that matters is the “location_href” written in the metadata, and the actual location of the files as they are published by Pulp. The “location_href” information via the content APIs is misleading because Pulp doesn’t use it when syncing and it doesn’t use it when publishing - really this field should be called “filename” because that is the only portion of it that is actually being used. Eventually we’ll rename the field, because the extra information is useless to us, and misleads users.
Just to give you an update, I’ve been trying to reproduce this and having difficulty doing so. Please let me know - is your “main” server and content proxy both using mirror-sync, or is one doing so and one is not? And if - on the server where you believe the incorrect metadata is originating from, you flip the value of “mirror” to the opposite of whatever it currently is, do you still encounter issues?
And if - on the server where you believe the incorrect metadata is originating from, you flip the value of “mirror” to the opposite of whatever it currently is, do you still encounter issues?
After a subsequent sync, because obviously just flipping the value won’t do much.
I don’t know what you mean exactly. My main server is doing mirror on sync on all repositories. My content proxy is doing “on demand”, if that’s what you mean. But I have never really touched that.
The metadata seems correct in either place (after I have generated the metadata for my perfsonar repository in this last case). primary.xml points to the urls listed in Packages on the main server as well as the content server. The problem is just, that the content server is using different URLs to access the main server.
After the metadata regeneration and complete content sync, both main server and content proxy present the packages in the Packages/?/ directories with ? begin the first letter of the rpm.
However, the content proxy uses URIs like “packages/pscheduler-tool-ethr-4.4.1-1.el7.noarch.rpm” to access RPMs from the main server. So, for whatever reason it is using “packages” instead of “Packages” and doesn’t use the first letter subdirectory.
It’s unclear to me where the content proxy picks up those “prefixes” to the rpms.
I can’t really tell when it’s broken and when not. I suspect, one reason could be if at some point the external repository changes it’s directory layout for the rpms. Pulp tries to “mimic” this layout in its own content presentation and if it changes externally, it results in confusion. It’s also unclear to me, under which circumstances pulp_rpm mimics the layout of the external repository and when not. I have some repositories, where pulp presents the content simply in “Packages/?/” even though the external repositories uses a different layout. For other repositories, like the kibana it uses (or used) the external layout.
I think pulp_rpm 3.14.3 also changed something in the layout. From my observations, it seems to prefer the “Packages/?/” layout now… If that’s true, it would further complicate things because now after updating to pulp_rpm 3.14.3 the layout changes through the system (after a metadata regeneration at least, I guess).
There was no change, so I’m not sure what’s going on there. But I do now have some ideas about what could be causing the issue. Let me give some background.
Katello has an option called “mirror on sync”. This provides a flag “mirror=True” to Pulp. When syncing in this mode, Pulp adopts the behavior of pulling down an exact copy of the metadata, and making the packages available in the exact same locations they were in at the upstream. It doesn’t do a real “publish”, it basically just clones the repo.
If you regenerate the metadata, this is a manual publish, which will use a standardized layout - packages get put into Packages/${letter}/${full_name}.
When you do an ‘on_demand’ sync, Pulp stores the URLs of the artifacts so that they can be downloaded in the future.
It seems like at any given time the repo metadata on both systems is actually correct, but one or perhaps several of these things is happening:
The content proxy is being synced and then afterwards the layout of the repos on the main system changes, invalidating the stored URLs, and the content proxy isn’t being re-synced after that
The content proxy is being synced against the correct layout / both layouts, but isn’t updating the URLs on sync as it is supposed to (multiple URLs are stored in order for the same artifacts, they get tried in-order at download time)
The content proxy is getting updated with the new URLs, but something is going wrong when it tries the various ones.
I guess another scenario might be if upstream changes the layout at some point and reorganizes the whole repository…
So that basically is what is happening, but the “upstream” that is being reorganized is the main instance, when you alternate between syncing and regenerating the metadata.
Rewriting the bullet points for clarity, it won’t let me edit anymore.
The content proxy is being synced and then afterwards the layout of the repos on the main system changes, invalidating the stored URLs, and the content proxy isn’t being re-synced after that.
The content proxy is being synced against the correct layout (or both layouts), but isn’t keeping all of the potential URLs as it is supposed to (multiple URLs are stored in order for the same artifacts, they get tried in-order at download time. This is because the same package could be available from different repos.)
The content proxy is getting updated with the new URLs, but something is going wrong when it tries to go through them to download the package. It might be trying the first, probably getting a 404, and giving up before trying the others, which it should be doing.
I have read something about “location_href” in the change log so I though it might have to do with that.
As far as I can tell, all my repositories are set to “mirror on sync” in the gui. My perfsonar repository, for instance, is set so. If I look into /pulp/content/ORG/Library/custom/perfsonar/perfsonar-el7/ on the main server after it ran another sync this morning, I still see the Packages/letter/ directories. That doesn’t match. (O.K. maybe the optimized sync found nothing needs to be synced and thus left it as it is for the moment).
So basically, a regenerate metadata would temporarily change the layout of all repositories on the main server, unless the external repositories already uses Packages/${letter}/${full_name}?
I am not sure about the layout change on the external server. As I wrote before, I also have problems with rpm from the foreman 2.5 release repository and I guess you guys know best whether or not you have changed the layout recently. I usually don’t keep track of the external repo organization.
I did, however, run a regenerate metadata on most/all on my repositories on the main server because I was hoping it would fix my problems (after and advanced or repair sync didn’t help either). So possibly, that caused more problems…
Let me know if I can pull any data from my systems for troubleshooting…
Where do I find/How do I build the URL of the upstream RPM in the pulp database? That’s unclear to me.
I have read something about “location_href” in the change log so I though it might have to do with that.
I had wondered that initially but the change post-dates your initial report, and after reviewing the patch again I didn’t see a way for that to have created the issue.
The “mirror” functionality was introduced in pulp_rpm 3.13.0, and that’s when the “layout” change would have happened - but I believe Katello may still have been doing manual publishes after mirror syncs for some period of time, and in that event it would have looked the same as it would after a manual metadata regeneration. I do not know when they changed/fixed that.
I still see the Packages/letter/ directories. That doesn’t match. (O.K. maybe the optimized sync found nothing needs to be synced and thus left it as it is for the moment).
That is very possible.
So basically, a regenerate metadata would temporarily change the layout of all repositories on the main server, unless the external repositories already uses Packages/${letter}/${full_name}?
Yes. Doing a manual publish will always use a standard layout, but mirroring the external repo will give you whatever layout the external layout used.
I am not sure about the layout change on the external server. As I wrote before, I also have problems with rpm from the foreman 2.5 release repository and I guess you guys know best whether or not you have changed the layout recently. I usually don’t keep track of the external repo organization.
All the problems are with the content proxy, yes? If so, then I don’t expect the upstream repo layout would make a difference. It’s much more likely to be a change w/ on the main Pulp / Katello server.
Honestly we should probably reconsider the implications of “regenerate metadata” for mirrored repos, maybe it should not be available at all. Since we’re just cloning the original metadata, it is unlikely to solve any problems.
Where do I find/How do I build the URL of the upstream RPM in the pulp database? That’s unclear to me.
select * from core_remoteartifact;
The table has columns for “sha1”, “sha256” and so forth. You can filter by those to look at the records for individual RPMs, since the RPM metadata contains this for each package.
I just did skip metadata sync and a repair sync on the perfsonar repository. But that didn’t change a thing with the layout on the main server.
O.K. Interesting. So basically, if I do a regenerate it’ll go from external layout, to standard layout and later at some point back to external layout?
Well, I guess not. As I have mentioned above (post 14), after my update to pulp_rpm 3.14.3 there were some new updates to the perfsonar repository. After that sync the main server didn’t work either. But for the main server the regenerate metadata fixed the problem.
But that I have only noticed because I have checked manually. Currently, all my clients connect to the content proxy and only my three foreman servers connect to the main server. So if there were more problems on the main server I wouldn’t necessarily notice. I did notice these issues with the foreman release rpms, though, so that was the main server. But for the main server, I can run the regenerate metadata and that helps.
Absolutely. If it changes the directory layout (and in particular back and forth) it shouldn’t be a general option. But still, something like that is necessary, as it did - at least for me - fix some of my problems, just not for the content on the proxy. The complete sync (aka metadata skip sync) doesn’t seem to fix everything at this point.
Great. I have just checked the database on the content proxy:
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%perfsonar-el7%Packages%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-----------------------------+-------
2021-07-09 07:01:24.207283+02 | 2021-07-09 07:01:29.003191+02 | 2021-07-09 07:01:24.207315+02 | 2021-07-09 07:01:29.0032+02 | 2142
(1 row)
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%perfsonar-el7%packages%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-09-08 06:44:05.836841+02 | 2021-09-08 06:45:05.048783+02 | 2021-09-08 06:44:05.836889+02 | 2021-09-08 06:45:05.048792+02 | 454
(1 row)
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%perfsonar-el7%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-07-09 07:01:24.207283+02 | 2021-09-08 06:45:05.048783+02 | 2021-07-09 07:01:24.207315+02 | 2021-09-08 06:45:05.048792+02 | 2596
(1 row)
I sync two LEs “Testing” and “Production” to the content proxy and for both it’s half of the above. So that’s consistent.
I have randomly checked some with the “packages” in the url and those are broken. The current layout on the main server is:
Packages/
config.repo
repodata/
As you can see from the timestamps, the 454 broken ones are those synced two days ago. All others were two months old (which well can be as the perfsonar repo doesn’t get updates too often).
So it seems to me, I have had “Packages/{letter}/” on the main server at some point before the updates two days ago. The new updates caused a “mirror on sync” reorganizing everything to the external layout.
However, it seems they have changed the directory layout on the perfsonar repository as well during that time. They still have a RPMS and a packages directory with the identical content. As I have seen RPMS urls in the logs I guess that was the layout two months ago and now they have adjusted their metadata to point to the packages directory?
Via the standard, automated content proxy sync after I have promoted a new CV version, it updated the content proxy with the external layout URLs. But as I had those broken URLs on the main server and I did a regenerate metadata at some point, it messed it up.
So a “Complete Sync” neither on the repository on the main server nor for the complete content proxy does mitigate issues with the directory layout for RPMs which are already in the database. Can that be? For the content proxy that seems right as you can see in the database above…
For the foreman release on the content proxy pulpcore database:
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%/foreman/2_5_el7_x86_64/%' and url not like '%/foreman/2_5_el7_x86_64/Packages/_/%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-08-24 13:24:16.562816+02 | 2021-08-31 06:13:19.377518+02 | 2021-08-24 13:24:16.562847+02 | 2021-08-31 06:13:19.377525+02 | 54
(1 row)
pulpcore=# select min(pulp_created),max(pulp_created), min(pulp_last_updated),max(pulp_last_updated), count(*) from core_remoteartifact where url like '%/foreman/2_5_el7_x86_64/%' and url like '%/foreman/2_5_el7_x86_64/Packages/_/%';
min | max | min | max | count
-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------
2021-07-07 16:31:38.788943+02 | 2021-07-20 08:17:03.716721+02 | 2021-07-07 16:31:38.788972+02 | 2021-07-20 08:17:03.716732+02 | 1370
(1 row)
O.K. Here is one way on how to get a 404 after changing paths.
Create a test (external) repository on a web server with rpms in a Packages subdirectory: Package/abc.rpm
Important: make sure those RPMs do not already exists in any form on the server. I think pulp can associate RPMs by hash and thus would use a local copy if hash fits regardless of where it moved in the external repository.
Now create the repository in katello with “mirror on sync” and “on demand” download policy and sync.
This creates entries in core_remoteartifact with Packages/ urls.
Now “restructure” your external repository, e.g. move all RPMS from the Packages subdirectory to the root and rerun “createrepo” to update metadata.
Sync again: nothing changes in the database. core_remoteartifact remains as it is. If I now try to access the rpm through pulp I get a 404 because it tries to download using the Packages subdirectory.
I guess, something similar happens between the main server and the content proxy (with on demand) if you regenerate metadata on the main server…
And if this is confirmed then this would be a Pulp bug, because it should be capturing the new locations as well. I’ll focus some attention on that aspect.
So it seems it backported to pulpcore 3.14.7. I hope it doesn’t take too long to get it into katello.
The usual question: is there some cleanup or resync necessary after the updated version is installed? How do those broken URLs get repaired or cleaned up? In particular for the content proxy where I can’t really do much except an optimized or complete sync.
So I guess, this translate to a “Complete Sync” of the products (or repositories) on the main Katello server and a “Complete Sync” of the content proxy on the smart proxy page.
But we completely agree that it’s painful to run into these errors on the default settings. Mirroring the metadata has a lot of benefits (repository signatures stay intact) but also a lot of limitations, such as this one, unfortunately, which weren’t anticipated.
We’re working on some sync modes that let you mirror the content without mirroring the metadata.
