Problem: On Monday I upgraded from katello 4.9.2 up to 4.11.1, since then the Sync Repository on Smart Proxy(ies)
takes very long time. I noticed there are quite a lot of Update Content Counts tasks which seem to me blocking the other tasks. It is very similar to Proxy sync 10x slower with Foreman 3.8/Katello 4.10 but that was fixed by updating of several packages but my packages are up to date.
Expected outcome: Capsule sync takes same time as on katello 4.9.2
Foreman and Proxy versions: foreman 3.9.1, katello 4.11.1
Would you be able to provide more information about your setup/workflow and the timings on the Sync Repository on Smart Proxy(ies) task? Do you see any long running step in the sync task details on dynflow console? Also the overall time Update content counts task is taking on your system?
For more context, the Update content counts action is new in 4.11 and runs asynchronously after every smart proxy sync to populate the data for the content counts of repositories synced to smart proxy which is shown on the Smart Proxy > Content tab. It is run a separate task to not block any other smart proxy tasks or functions.
thank you for explaining, what Update content count task does. The problem with slow sync is during daily sync. We have 7 capsules and about 150 repos. The sync to all capsules of the first repo in the list seems to take about the same time as before (minutes), but then several Update content tasks are launched and at the same time another repo sync which takes longer (10 minutes) and then again several Update conent tasks and then the last repos takes hours to sync. May be the Update content task is not blocking it but somehow slowing it down.
I started complete sync of one repo and as soon as Update Content Tasks appeared I started complete sync of another repo. This second sync remained in waiting state until Sync Repository on Smart Proxy(ies) task from the previous one finished. This wasn’t in katello 4.9.2. I think. So there seems to be some blocking or slowness between all these tasks.
It can be easily simulated in other than my environment in my opinion.
Capsule sync task does take a lock on the capsule. If individual repo syncs are causing capsule syncs and a lot of these repo syncs are happening at once, I can see the capsule syncs waiting on each other in that case. However, it shouldn’t block repo sync itself on katello. I’ll see if something introduced that behavior.
I did a test with foreman 3.9.1, katello 4.11 with my production setup with 10 world wide proxies a while back and also noticed the slow behavior so rolled back right away to avoid the impact. Even though I would like it solved, I can not “test” new releases and debug the problems with my production setup so I will be stuck at 3.8/4.10 a while I think.
Will test 3.10/4.12 soon but I do not have too high hopes for a smooth rollout so will prepare for rollback again to snapshots.
I have filed Bug #37356: Slow smart proxy sync in 4.11 - Katello - Foreman to get some work going on this one. We call the update content counts tasks upon each capsule sync nd it might be useful to have some optimizations around it to not add waits between that task and the proxy sync.
@tedevil , what was the test that you performed? Did it involve just syncing smart proxies, or perhaps repo syncing / CV publishing that triggered the smart proxy syncs?
All I did was to publish one new rpm to our internal repo and sync the repo in Foreman and then watch the sync start towards all proxies after the upgrade. I do not use content views yet.
I will as I said soon dedicate a few hours and snapshot foreman + 10 proxies and perform some upgrades and see what happens when using the newest version.
3.8/4.10 sync very fast as this point, under 1 minute worldwide so that is why I hesitate to upgrade this time considering the hassle I had last time
We haven’t been able to reproduce the slow sync behavior yet, so it likely has to do with a specific set of content synced to Katello. We’ve also had performance tests run for smart proxy syncs on Katello 4.11 and no issues cropped up, which is why I’m surprised by the slowness reported here.
Anyway, we’ll keep trying to reproduce it on our end.
