On Foreman 3.2/Katello 4.4 and have a google chrome repo with the following settings:
Upstream URL: https://dl.google.com/linux/chrome/rpm/stable/x86_64
Download Policy = Immediate
Mirroring Policy = Additive
Retain package versions = 5
Restrict to architecture = x86_64
With this setting, I assumed I would only see max 5 of the latest versions of each package in the repo. The problem is that I see more then 5 versions of each package.
Did I misunderstood this setting or is it broken?
I guess you just created that repo and thus only one version of each package is there.
With “Mirroring Policy = Additive” overtime the amount of versions of packages will grow. I created this repo a few months ago and there are already more then 5 versions of each package:
Have another repo with the same settings taking data from https://packages.gitlab.com/runner/gitlab-runner/el/7/x86_64 for a few months and it has 9 versions of each package. Did I misunderstand the usage of the “Retain package versions” setting on a repo with Mirroring Policy = Additive?
I was hoping to use it to keep the size of the products under control but at the same time have more local versions then 1 that I would get with Mirroring Policy = Current Only.
It’s possible something is broken, I will investigate.
I don’t think you’ve misunderstood anything.
This seems to be working fine for me with the currently-supported upstream release of pulp_rpm, I haven’t been able to reproduce. @partha did reproduce using an older build of Katello, but I couldn’t tell you what specifically might have changed between the two.
Hmm yeah it seems to be working correctly even on 4.3 for me. I tested with various repositories and configurations. Can you do a “Complete Sync” and confirm you still see this issue? Basically Go to Advanced Sync -> Complete Sync on the repo details page and see if that fixes it.
Before sync:
# hammer package list --repository-id 51
--------|-----------------------------------------|-------------------------------------
ID | FILENAME | SOURCE RPM
--------|-----------------------------------------|-------------------------------------
143250 | gitlab-runner-14.7.1-1.x86_64.rpm | gitlab-runner-14.7.1-1.src.rpm
143249 | gitlab-runner-14.8.1-1.x86_64.rpm | gitlab-runner-14.8.1-1.src.rpm
143248 | gitlab-runner-14.8.2-1.x86_64.rpm | gitlab-runner-14.8.2-1.src.rpm
143247 | gitlab-runner-14.9.0-1.x86_64.rpm | gitlab-runner-14.9.0-1.src.rpm
143246 | gitlab-runner-14.9.1-1.x86_64.rpm | gitlab-runner-14.9.1-1.src.rpm
624143 | gitlab-runner-14.9.2-1.x86_64.rpm | gitlab-runner-14.9.2-1.src.rpm
310258 | gitlab-runner-14.10.0-1.x86_64.rpm | gitlab-runner-14.10.0-1.src.rpm
624144 | gitlab-runner-14.10.1-1.x86_64.rpm | gitlab-runner-14.10.1-1.src.rpm
1062059 | gitlab-runner-15.0.0-1.x86_64.rpm | gitlab-runner-15.0.0-1.src.rpm
143255 | gitlab-runner-fips-14.7.1-1.x86_64.rpm | gitlab-runner-fips-14.7.1-1.src.rpm
143254 | gitlab-runner-fips-14.8.1-1.x86_64.rpm | gitlab-runner-fips-14.8.1-1.src.rpm
143253 | gitlab-runner-fips-14.8.2-1.x86_64.rpm | gitlab-runner-fips-14.8.2-1.src.rpm
143252 | gitlab-runner-fips-14.9.0-1.x86_64.rpm | gitlab-runner-fips-14.9.0-1.src.rpm
143251 | gitlab-runner-fips-14.9.1-1.x86_64.rpm | gitlab-runner-fips-14.9.1-1.src.rpm
624145 | gitlab-runner-fips-14.9.2-1.x86_64.rpm | gitlab-runner-fips-14.9.2-1.src.rpm
310259 | gitlab-runner-fips-14.10.0-1.x86_64.rpm | gitlab-runner-fips-14.10.0-1.src.rpm
624146 | gitlab-runner-fips-14.10.1-1.x86_64.rpm | gitlab-runner-fips-14.10.1-1.src.rpm
1062060 | gitlab-runner-fips-15.0.0-1.x86_64.rpm | gitlab-runner-fips-15.0.0-1.src.rpm
--------|-----------------------------------------|-------------------------------------
During “Advanced Sync → Complete Sync”
Total steps: 240/240
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 106/106
Downloading Metadata Files: 4/4
Parsed Packages: 130/130
Production log:
2022-05-27T19:57:50 [I|app|0c488cf5] Started POST "/katello/api/v2/repositories/51/sync?organization_id=1" for 10.107.17.18 at 2022-05-27 19:57:50 +0000
2022-05-27T19:57:50 [I|app|0c488cf5] Processing by Katello::Api::V2::RepositoriesController#sync as HTML
2022-05-27T19:57:50 [I|app|0c488cf5] Parameters: {"id"=>"51", "skip_metadata_check"=>true, "organization_id"=>"1", "api_version"=>"v2", "repository"=>{"id"=>"51"}}
2022-05-27T19:57:50 [I|bac|0c488cf5] Task {label: , execution_plan_id: a11fe3ca-483f-4d98-8977-9602c83f7b34} state changed: pending
2022-05-27T19:57:50 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::Sync, id: 9e7fd6d4-cf43-41c2-9794-e6fe655d2c38, execution_plan_id: a11fe3ca-483f-4d98-8977-9602c83f7b34} state changed: planning
2022-05-27T19:57:50 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::Sync, id: 9e7fd6d4-cf43-41c2-9794-e6fe655d2c38, execution_plan_id: a11fe3ca-483f-4d98-8977-9602c83f7b34} state changed: planned
2022-05-27T19:57:50 [I|app|0c488cf5] Rendering /usr/share/gems/gems/katello-4.4.0.2/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource
2022-05-27T19:57:50 [I|app|0c488cf5] Rendered /usr/share/gems/gems/katello-4.4.0.2/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 24.8ms | Allocations: 24040)
2022-05-27T19:57:50 [I|app|0c488cf5] Completed 202 Accepted in 353ms (Views: 26.7ms | ActiveRecord: 19.4ms | Allocations: 129944)
2022-05-27T19:57:50 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::Sync, id: 9e7fd6d4-cf43-41c2-9794-e6fe655d2c38, execution_plan_id: a11fe3ca-483f-4d98-8977-9602c83f7b34} state changed: running
2022-05-27T20:01:02 [I|aud|0c488cf5] Katello::Repository (51) update event on last_contents_changed 2022-05-20 04:09:42 +0200, 2022-05-27 22:01:02 +0200
2022-05-27T20:01:04 [I|aud|0c488cf5] Katello::Repository (51) update event on publication_href /pulp/api/v3/publications/rpm/rpm/df88e9ae-ca32-4133-a4f6-d362c8243f05/, /pulp/api/v3/publications/rpm/rpm/aa962c33-057c-4362-a126-200c54a3c8db/
2022-05-27T20:01:05 [I|app|0c488cf5] CONTENT_INDEX type=Katello::Rpm success=true duration=87.12
2022-05-27T20:01:05 [I|app|0c488cf5] CONTENT_INDEX type=Katello::ModuleStream success=true duration=62.15
2022-05-27T20:01:05 [I|app|0c488cf5] CONTENT_INDEX type=Katello::Erratum success=true duration=60.48
2022-05-27T20:01:05 [I|app|0c488cf5] CONTENT_INDEX type=Katello::PackageGroup success=true duration=59.11
2022-05-27T20:01:06 [I|app|0c488cf5] CONTENT_INDEX type=Katello::Srpm success=true duration=65.06
2022-05-27T20:01:06 [I|aud|0c488cf5] Katello::Repository (51) update event on last_indexed 2022-05-20 04:10:40 +0200, 2022-05-27 22:01:06 +0200
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: , execution_plan_id: 553f6307-da6b-42e1-a3a7-a351d67462d8} state changed: pending
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::CapsuleSync, id: f0fe223d-c8fe-4422-97d8-66181f2f5664, execution_plan_id: 553f6307-da6b-42e1-a3a7-a351d67462d8} state changed: planning
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::CapsuleSync, id: f0fe223d-c8fe-4422-97d8-66181f2f5664, execution_plan_id: 553f6307-da6b-42e1-a3a7-a351d67462d8} state changed: planned
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::CapsuleSync, id: f0fe223d-c8fe-4422-97d8-66181f2f5664, execution_plan_id: 553f6307-da6b-42e1-a3a7-a351d67462d8} state changed: running
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: , execution_plan_id: e1b656a9-9ec0-4d6a-8bb4-a0ea23af0275} state changed: pending
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7cf326d2-5830-4526-823e-4ba603a45ddd, execution_plan_id: e1b656a9-9ec0-4d6a-8bb4-a0ea23af0275} state changed: planning
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::Sync, id: 9e7fd6d4-cf43-41c2-9794-e6fe655d2c38, execution_plan_id: a11fe3ca-483f-4d98-8977-9602c83f7b34} state changed: stopped result: success
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7cf326d2-5830-4526-823e-4ba603a45ddd, execution_plan_id: e1b656a9-9ec0-4d6a-8bb4-a0ea23af0275} state changed: planned
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: , execution_plan_id: 8d8313f6-4ae5-4c04-b55a-fb3a580975ab} state changed: pending
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7cf326d2-5830-4526-823e-4ba603a45ddd, execution_plan_id: e1b656a9-9ec0-4d6a-8bb4-a0ea23af0275} state changed: running
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7e71ef02-488a-4b81-9ae0-4996b27fde57, execution_plan_id: 8d8313f6-4ae5-4c04-b55a-fb3a580975ab} state changed: planning
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7e71ef02-488a-4b81-9ae0-4996b27fde57, execution_plan_id: 8d8313f6-4ae5-4c04-b55a-fb3a580975ab} state changed: planned
2022-05-27T20:01:06 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7e71ef02-488a-4b81-9ae0-4996b27fde57, execution_plan_id: 8d8313f6-4ae5-4c04-b55a-fb3a580975ab} state changed: running
2022-05-27T20:01:11 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7e71ef02-488a-4b81-9ae0-4996b27fde57, execution_plan_id: 8d8313f6-4ae5-4c04-b55a-fb3a580975ab} state changed: stopped result: success
2022-05-27T20:01:12 [I|bac|0c488cf5] Task {label: Actions::Katello::CapsuleContent::Sync, id: 7cf326d2-5830-4526-823e-4ba603a45ddd, execution_plan_id: e1b656a9-9ec0-4d6a-8bb4-a0ea23af0275} state changed: stopped result: success
2022-05-27T20:01:12 [I|bac|0c488cf5] Task {label: Actions::Katello::Repository::CapsuleSync, id: f0fe223d-c8fe-4422-97d8-66181f2f5664, execution_plan_id: 553f6307-da6b-42e1-a3a7-a351d67462d8} state changed: stopped result: success
Packages after:
# hammer package list --repository-id 51
--------|-----------------------------------------|-------------------------------------
ID | FILENAME | SOURCE RPM
--------|-----------------------------------------|-------------------------------------
143250 | gitlab-runner-14.7.1-1.x86_64.rpm | gitlab-runner-14.7.1-1.src.rpm
143249 | gitlab-runner-14.8.1-1.x86_64.rpm | gitlab-runner-14.8.1-1.src.rpm
143248 | gitlab-runner-14.8.2-1.x86_64.rpm | gitlab-runner-14.8.2-1.src.rpm
143247 | gitlab-runner-14.9.0-1.x86_64.rpm | gitlab-runner-14.9.0-1.src.rpm
143246 | gitlab-runner-14.9.1-1.x86_64.rpm | gitlab-runner-14.9.1-1.src.rpm
624143 | gitlab-runner-14.9.2-1.x86_64.rpm | gitlab-runner-14.9.2-1.src.rpm
310258 | gitlab-runner-14.10.0-1.x86_64.rpm | gitlab-runner-14.10.0-1.src.rpm
624144 | gitlab-runner-14.10.1-1.x86_64.rpm | gitlab-runner-14.10.1-1.src.rpm
1062059 | gitlab-runner-15.0.0-1.x86_64.rpm | gitlab-runner-15.0.0-1.src.rpm
143255 | gitlab-runner-fips-14.7.1-1.x86_64.rpm | gitlab-runner-fips-14.7.1-1.src.rpm
143254 | gitlab-runner-fips-14.8.1-1.x86_64.rpm | gitlab-runner-fips-14.8.1-1.src.rpm
143253 | gitlab-runner-fips-14.8.2-1.x86_64.rpm | gitlab-runner-fips-14.8.2-1.src.rpm
143252 | gitlab-runner-fips-14.9.0-1.x86_64.rpm | gitlab-runner-fips-14.9.0-1.src.rpm
143251 | gitlab-runner-fips-14.9.1-1.x86_64.rpm | gitlab-runner-fips-14.9.1-1.src.rpm
624145 | gitlab-runner-fips-14.9.2-1.x86_64.rpm | gitlab-runner-fips-14.9.2-1.src.rpm
310259 | gitlab-runner-fips-14.10.0-1.x86_64.rpm | gitlab-runner-fips-14.10.0-1.src.rpm
624146 | gitlab-runner-fips-14.10.1-1.x86_64.rpm | gitlab-runner-fips-14.10.1-1.src.rpm
1062060 | gitlab-runner-fips-15.0.0-1.x86_64.rpm | gitlab-runner-fips-15.0.0-1.src.rpm
--------|-----------------------------------------|-------------------------------------
Sync details after:
No content added.
Total steps: 374/374
--------------------------------
Associating Content: 120/120
Downloading Artifacts: 120/120
Downloading Metadata Files: 4/4
Parsed Packages: 130/130
Well, that part isn’t surprising, it’s a known limitation that I’m working on Make retain_package_versions more useful at sync time · Issue #2479 · pulp/pulp_rpm · GitHub
The main concern would be if it isn’t working right at the repository level - all the packages are currently being downloaded and stuff (because of that limitation), but it should still only retain the latest ones in the repo at the end of the sync.
Is this potentially the issue? Bug #34469: Retain packages on Repository does not synchronize the specified number of packages - Katello - Foreman
Read the bug report and that is not at all what I see.
What I did notice yesterday when messing around with a repo is that the setting seems to act more like a filter towards the hosts. I have a GitLab repo and since those packages are so big (1 GB) each I only keep the 5 latest by scripting adding/removing rpm from the repo. Since I needed an older version of GitLab I manually downloaded a old version, added to the repo so it now had 5 of the latest versions and one old one. With the “retain package versions” set to 5, it was impossible to show the 6th oldest package from the hosts even if the package was in the repo. After removing the “retain package versions”, I still could not see the 6th package so had to remove it from the repo, add it again, this time the old 6th rpm was visible.
Meaning the setting will not try and only keep the 5 latest versions (removing old versions from the repo), but actually only show the 5 latest versions. I was hoping it would actually remove old versions from the repo to keep the size down, but since it does not do that, this setting is pretty useless for me.
Meaning the setting will not try and only keep the 5 latest versions (removing old versions from the repo), but actually only show the 5 latest versions. I was hoping it would actually remove old versions from the repo to keep the size down, but since it does not do that, this setting is pretty useless for me.
I mean, that’s exactly what it should be doing, and what it does, for me. When new versions of packages are added, old packages are removed entirely.
Since I needed an older version of GitLab I manually downloaded a old version, added to the repo so it now had 5 of the latest versions and one old one. With the “retain package versions” set to 5, it was impossible to show the 6th oldest package from the hosts even if the package was in the repo.
I think what you may be seeing is that if you add an old package and the repository is already at the limit, it might remove it immediately after attempting to add the package. Hence the 6th package would never have been present in the repo.
After removing the “retain package versions”, I still could not see the 6th package so had to remove it from the repo, add it again, this time the old 6th rpm was visible.
You might not have been able to see it because it wasn’t there? Or shouldn’t have been there.
There was an issue with Katello’s recordkeeping related to this that may have been showing some things that weren’t present or vice versa. Bug #35120: Retain packages on Repository removes RPMs from Pulp but not from Katello - Katello - Foreman
Right, could make sense if that is the bug, I see the old packages in the repo but it does not show the old packages to the host so maybe they are just “ghost” packages and not really there.