I have four CentOS 7 repositories setup in Foreman which have been working as expected for a couple of years. Recently I notice the repo for CentOS 7 Updates which is synced to CentOS Mirror is having an odd issue. When I go to Products → “Name” → Repositories → “Name” → CentOS 7 Updates → Packages I can tell that it is missing the latest versions of a bunch of packages. For example for open-vm-tools the latest version it has is 9.4 it is missing 9.6 and 9.7. From what I can tell it is missing the versions from the last 3-4 months. When I check the actual published repo on the foreman server I can see these missing later versions are here. So it appears the Foreman server has correctly synced the repo to itself its just the web portal that is not correctly reflecting this.
In the Foreman web portal for the CentOS 7 Updates it has a package count of 5053 when I checked the package count of the actual repo for CentOS 7 Updates on the Foreman server it has a package count of
5,367. The other 3 repos that I have seem to be fine and the counts all match between the Foreman web portal and the Foreman repo.
I have tried Republishing Repository Metadata and verify the content checksums but neither seems to help. Foreman is on version 2.4.1 and Katello is on 4.0.1.
The effect you are describing sounds like there was a Sync task which completed all of its actions except Actions::Pulp3::Repository::SaveVersion successfully. If that is the case a re-publish repository metadata would not help, since that repeats the Actions::Pulp3::Repository::CreatePublication action.
My expectation would be that this situation is “self healing” the next time a sync of the repo in question actually adds some new content to the repo, but I am not certain about that.
If you navigate to “Monitor > Tasks” in the UI, you could try searching/filtering for something like “Synchronize repository <your_centos_7_repository_name_here>” to list all sync tasks for the repo in question. Do any of them not show success in the Result column?
If you open an individual task, you can click the “Dynflow console” button to open a different task view that lists each action that the task performed. If I am right, one of those should show at least the Actions::Pulp3::Repository::SaveVersion action in yellow (or at least not green). While most other actions are green.
I am not sure there is a way to force a repeat of just that one action, the way there is for “republish repository metadata”. Does a complete sync help? (Its possible that a sync that does not add any new packages because the upstream has not changed will simply skip the action we want).
Thanks for pointing me to the dynflows. Looking at them and most of them complete in a few seconds and all parts of the run show as a success. I did notice that every couple of weeks there will be one that takes like 5-10 minutes where it looks like it trying to get updated packages. These ones all have a warning. If I look at the step you mentioned “Actions::Pulp3::Repository::SaveVersion” that will show as a success and looks to show the correct number of packages in the repo 5,367. Where it looks to be having issues is this step “Actions::Katello::Repository::IndexContent (skipped) [ 35.99s / 34.04s ]”
That step is erroring out.
Queue: default
Started at: 2023-10-21 11:07:06 UTC
Ended at: 2023-10-21 11:07:41 UTC
Real time: 35.99s
Execution time (excluding suspended state): 34.04s
Error message: the server returns an error HTTP status code: 502 Response headers: {“date”=>“Sat, 21 Oct 2023 11:07:07 GMT”, “server”=>“Apache”, “content-length”=>“445”, “connection”=>“close”, “content-type”=>“text/html; charset=iso-8859-1”} Response body: 502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /pulp/api/v3/content/rpm/packages/.
“/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.2/app/services/katello/pulp3/pulp_content_unit.rb:69:in
`block in pulp_units_batch_for_repo’”
Googling my error message I was able to come across someone with a similar issue and a solution. I needed to set the bulk_load_size to 500 in my /etc/foreman/plugins/katello.yaml for the pulp setting. Unlike the examples I found my katello.yaml did not actually have a pulp section at all so I added that in with the settings like this
:pulp:
:bulk_load_size: 500
Restarted the server and then did a complete sync of the repo and it was successful and now correctly shows the 5,367 packages.
