Very slow proxy sync after upgrade to Foreman 3.9 and katello 4.11.1

Problem: On Monday I upgraded from katello 4.9.2 up to 4.11.1, since then the Sync Repository on Smart Proxy(ies)
takes very long time. I noticed there are quite a lot of Update Content Counts tasks which seem to me blocking the other tasks. It is very similar to Proxy sync 10x slower with Foreman 3.8/Katello 4.10 but that was fixed by updating of several packages but my packages are up to date.

Expected outcome: Capsule sync takes same time as on katello 4.9.2

Foreman and Proxy versions: foreman 3.9.1, katello 4.11.1

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

Hi @JendaVodka,

Would you be able to provide more information about your setup/workflow and the timings on the Sync Repository on Smart Proxy(ies) task? Do you see any long running step in the sync task details on dynflow console? Also the overall time Update content counts task is taking on your system?

For more context, the Update content counts action is new in 4.11 and runs asynchronously after every smart proxy sync to populate the data for the content counts of repositories synced to smart proxy which is shown on the Smart Proxy > Content tab. It is run a separate task to not block any other smart proxy tasks or functions.

Hi @sajha ,

thank you for explaining, what Update content count task does. The problem with slow sync is during daily sync. We have 7 capsules and about 150 repos. The sync to all capsules of the first repo in the list seems to take about the same time as before (minutes), but then several Update content tasks are launched and at the same time another repo sync which takes longer (10 minutes) and then again several Update conent tasks and then the last repos takes hours to sync. May be the Update content task is not blocking it but somehow slowing it down.

I ran a following test:

I started complete sync of one repo and as soon as Update Content Tasks appeared I started complete sync of another repo. This second sync remained in waiting state until Sync Repository on Smart Proxy(ies) task from the previous one finished. This wasn’t in katello 4.9.2. I think. So there seems to be some blocking or slowness between all these tasks.
It can be easily simulated in other than my environment in my opinion.

Capsule sync task does take a lock on the capsule. If individual repo syncs are causing capsule syncs and a lot of these repo syncs are happening at once, I can see the capsule syncs waiting on each other in that case. However, it shouldn’t block repo sync itself on katello. I’ll see if something introduced that behavior.

For information only: I checked the Daily sync job running times and in katello 4.9 it took max 55 minutes, in 4.11.1 it takes 3 hours

I did a test with foreman 3.9.1, katello 4.11 with my production setup with 10 world wide proxies a while back and also noticed the slow behavior so rolled back right away to avoid the impact. Even though I would like it solved, I can not “test” new releases and debug the problems with my production setup so I will be stuck at 3.8/4.10 a while I think.
Will test 3.10/4.12 soon but I do not have too high hopes for a smooth rollout so will prepare for rollback again to snapshots.