Very slow proxy sync after upgrade to Foreman 3.9 and katello 4.11.1

Support
1

Problem: On Monday I upgraded from katello 4.9.2 up to 4.11.1, since then the Sync Repository on Smart Proxy(ies)
takes very long time. I noticed there are quite a lot of Update Content Counts tasks which seem to me blocking the other tasks. It is very similar to Proxy sync 10x slower with Foreman 3.8/Katello 4.10 but that was fixed by updating of several packages but my packages are up to date.

Expected outcome: Capsule sync takes same time as on katello 4.9.2

Foreman and Proxy versions: foreman 3.9.1, katello 4.11.1

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

2

Hi @JendaVodka,

Would you be able to provide more information about your setup/workflow and the timings on the Sync Repository on Smart Proxy(ies) task? Do you see any long running step in the sync task details on dynflow console? Also the overall time Update content counts task is taking on your system?

For more context, the Update content counts action is new in 4.11 and runs asynchronously after every smart proxy sync to populate the data for the content counts of repositories synced to smart proxy which is shown on the Smart Proxy > Content tab. It is run a separate task to not block any other smart proxy tasks or functions.

3

Hi @sajha ,

thank you for explaining, what Update content count task does. The problem with slow sync is during daily sync. We have 7 capsules and about 150 repos. The sync to all capsules of the first repo in the list seems to take about the same time as before (minutes), but then several Update content tasks are launched and at the same time another repo sync which takes longer (10 minutes) and then again several Update conent tasks and then the last repos takes hours to sync. May be the Update content task is not blocking it but somehow slowing it down.

4

I ran a following test:

I started complete sync of one repo and as soon as Update Content Tasks appeared I started complete sync of another repo. This second sync remained in waiting state until Sync Repository on Smart Proxy(ies) task from the previous one finished. This wasn’t in katello 4.9.2. I think. So there seems to be some blocking or slowness between all these tasks.
It can be easily simulated in other than my environment in my opinion.

1 Like
5

Capsule sync task does take a lock on the capsule. If individual repo syncs are causing capsule syncs and a lot of these repo syncs are happening at once, I can see the capsule syncs waiting on each other in that case. However, it shouldn’t block repo sync itself on katello. I’ll see if something introduced that behavior.

6

For information only: I checked the Daily sync job running times and in katello 4.9 it took max 55 minutes, in 4.11.1 it takes 3 hours

7

I did a test with foreman 3.9.1, katello 4.11 with my production setup with 10 world wide proxies a while back and also noticed the slow behavior so rolled back right away to avoid the impact. Even though I would like it solved, I can not “test” new releases and debug the problems with my production setup so I will be stuck at 3.8/4.10 a while I think.
Will test 3.10/4.12 soon but I do not have too high hopes for a smooth rollout so will prepare for rollback again to snapshots.

8

I have filed Bug #37356: Slow smart proxy sync in 4.11 - Katello - Foreman to get some work going on this one. We call the update content counts tasks upon each capsule sync nd it might be useful to have some optimizations around it to not add waits between that task and the proxy sync.

9

@tedevil , what was the test that you performed? Did it involve just syncing smart proxies, or perhaps repo syncing / CV publishing that triggered the smart proxy syncs?

10

All I did was to publish one new rpm to our internal repo and sync the repo in Foreman and then watch the sync start towards all proxies after the upgrade. I do not use content views yet.
I will as I said soon dedicate a few hours and snapshot foreman + 10 proxies and perform some upgrades and see what happens when using the newest version.
3.8/4.10 sync very fast as this point, under 1 minute worldwide so that is why I hesitate to upgrade this time considering the hassle I had last time :slight_smile:

1 Like
11

I do not know the testing schema for each new release of Katello/Foreman, but I would suggest always to test at least:

  1. Repository sync time from internet and to smart proxies
  2. Orphaned content purge time
  3. Foreman tasks

Almost after every upgrade of Katello/Foreman, there is a problem in some of these areas.

12

We haven’t been able to reproduce the slow sync behavior yet, so it likely has to do with a specific set of content synced to Katello. We’ve also had performance tests run for smart proxy syncs on Katello 4.11 and no issues cropped up, which is why I’m surprised by the slowness reported here.

Anyway, we’ll keep trying to reproduce it on our end.

If anyone is curious, the full integration tests that run on every build of Katello are here: forklift/bats at master ¡ theforeman/forklift ¡ GitHub

We have more tests too in here: GitHub - SatelliteQE/robottelo: Robottelo is a test suite that exercises The Foreman.

And then of course we have our unit tests, but that’s more for exercising the code then performance.

I’m not sure where the performance specific tests live – they may be done by hand.

13

So during vacation times, our Foreman server is used less so thought I would have another go at this. Will push my production servers to latest version from 3.8/4.10 and measure sync times. Let me know if I can provide you with any additional logs. Also note that since this is production, if the sync times are so slow it will start blocking nominal flow, I will need to revert back to a “working” version.

Short background of the setup. One Foreman server in Sweden + 10 proxies all over the world with different latency.

Latency between Foreman server and proxies.
auys-fmproxy 420 ms (Australia)
cais-fmproxy 170 ms (Canada)
clpa-fmproxy 262 ms (Southern Chile)
clpe-fmproxy 235 ms (Northen Chile)
sebr-fmproxy 2 ms (Southern Sweden, same cluster as Foreman server)
sees-fmproxy 19 ms (Northen Sweden)
thsr-fmproxy 357 ms (Thailand)
usak-fmproxy 183 ms (USA, Alaska)
ushi-fmproxy 214 ms (USA, Hawaii)
usvf-fmproxy 103 ms (USA, Pennsylvania)

3.8/4.10, this version is working as expected with sync times under a minute worldwide.
Add one package to a repo and sync it:
Synchronize smart proxy ‘auys-fmproxy’ stopped success July 16, 2024 at 12:08:06 PM 52 seconds
Synchronize smart proxy ‘usak-fmproxy’ stopped success July 16, 2024 at 12:08:05 PM 36 seconds
Synchronize smart proxy ‘thsr-fmproxy’ stopped success July 16, 2024 at 12:08:04 PM 46 seconds
Synchronize smart proxy ‘sees-fmproxy’ stopped success July 16, 2024 at 12:08:04 PM 28 seconds
Synchronize smart proxy ‘usvf-fmproxy’ stopped success July 16, 2024 at 12:08:03 PM 30 seconds
Synchronize smart proxy ‘clpe-fmproxy’ stopped success July 16, 2024 at 12:08:02 PM 35 seconds
Synchronize smart proxy ‘clpa-fmproxy’ stopped success July 16, 2024 at 12:08:01 PM 37 seconds
Synchronize smart proxy ‘ushi-fmproxy’ stopped success July 16, 2024 at 12:08:00 PM 34 seconds
Synchronize smart proxy ‘sebr-fmproxy’ stopped success July 16, 2024 at 12:07:59 PM 26 seconds
Synchronize smart proxy ‘cais-fmproxy’ stopped success July 16, 2024 at 12:07:59 PM 34 seconds
Sync Repository on Smart Proxy(ies)

3.9.3/4.11.1, latency hits these tasks pretty bad.
remove one package from a repo and sync it.
Update Content Counts stopped success July 17, 2024 at 11:22:29 AM 11 minutes
Update Content Counts stopped success July 17, 2024 at 11:22:29 AM 8 minutes
Update Content Counts stopped success July 17, 2024 at 11:18:08 AM 6 minutes
Update Content Counts stopped success July 17, 2024 at 11:17:57 AM 7 minutes
Update Content Counts stopped success July 17, 2024 at 11:17:55 AM 3 minutes
Update Content Counts stopped success July 17, 2024 at 11:17:51 AM 6 minutes
Update Content Counts stopped success July 17, 2024 at 11:17:46 AM 1 minute
Update Content Counts stopped success July 17, 2024 at 11:17:45 AM 5 minutes
Update Content Counts stopped success July 17, 2024 at 11:17:23 AM 43 seconds
Synchronize smart proxy ‘auys-fmproxy’ stopped success July 17, 2024 at 11:17:14 AM 5 minutes
Synchronize smart proxy ‘usak-fmproxy’ stopped success July 17, 2024 at 11:17:13 AM 55 seconds
Synchronize smart proxy ‘thsr-fmproxy’ stopped success July 17, 2024 at 11:17:11 AM 5 minutes
Synchronize smart proxy ‘sees-fmproxy’ stopped success July 17, 2024 at 11:17:11 AM 35 seconds
Synchronize smart proxy ‘usvf-fmproxy’ stopped success July 17, 2024 at 11:17:10 AM 45 seconds
Synchronize smart proxy ‘clpe-fmproxy’ stopped success July 17, 2024 at 11:17:08 AM 2 minutes
Synchronize smart proxy ‘clpa-fmproxy’ stopped success July 17, 2024 at 11:17:07 AM 50 seconds
Synchronize smart proxy ‘ushi-fmproxy’ stopped success July 17, 2024 at 11:17:06 AM 45 seconds
Synchronize smart proxy ‘sebr-fmproxy’ stopped success July 17, 2024 at 11:17:05 AM 18 seconds
Synchronize smart proxy ‘cais-fmproxy’ stopped success July 17, 2024 at 11:17:04 AM 41 seconds
Sync Repository on Smart Proxy(ies)

Add one package to a repo and sync it.:
Update Content Counts stopped success July 17, 2024 at 11:41:47 AM 10 minutes
Update Content Counts stopped success July 17, 2024 at 11:36:44 AM 13 minutes
Update Content Counts stopped success July 17, 2024 at 11:36:41 AM 7 minutes
Update Content Counts stopped success July 17, 2024 at 11:36:41 AM 6 minutes
Update Content Counts stopped success July 17, 2024 at 11:36:41 AM 7 minutes
Update Content Counts stopped success July 17, 2024 at 11:36:26 AM 5 minutes
Update Content Counts stopped success July 17, 2024 at 11:36:25 AM 3 minutes
Update Content Counts stopped success July 17, 2024 at 11:36:25 AM 5 minutes
Update Content Counts stopped success July 17, 2024 at 11:35:59 AM 1 minute
Synchronize smart proxy ‘auys-fmproxy’ stopped success July 17, 2024 at 11:35:42 AM 6 minutes
Synchronize smart proxy ‘usak-fmproxy’ stopped success July 17, 2024 at 11:35:41 AM 1 minute
Synchronize smart proxy ‘usvf-fmproxy’ stopped success July 17, 2024 at 11:35:40 AM 45 seconds
Synchronize smart proxy ‘cais-fmproxy’ stopped success July 17, 2024 at 11:35:39 AM 47 seconds
Synchronize smart proxy ‘thsr-fmproxy’ stopped success July 17, 2024 at 11:35:37 AM 1 minute
Update Content Counts stopped success July 17, 2024 at 11:35:37 AM 36 seconds
Synchronize smart proxy ‘clpe-fmproxy’ stopped success July 17, 2024 at 11:35:36 AM 1 minute
Synchronize smart proxy ‘clpa-fmproxy’ stopped success July 17, 2024 at 11:35:34 AM 1 minute
Synchronize smart proxy ‘sees-fmproxy’ stopped success July 17, 2024 at 11:35:34 AM 25 seconds
Synchronize smart proxy ‘ushi-fmproxy’ stopped success July 17, 2024 at 11:35:33 AM 51 seconds
Synchronize smart proxy ‘sebr-fmproxy’ stopped success July 17, 2024 at 11:35:32 AM 5 seconds
Sync Repository on Smart Proxy(ies) stopped success July 17, 2024 at 11:35:32 AM 8 minutes

14

3.10.0/4.12.1, simlar behaviour as 3.9.3/4.11.1. Total time for all tasks to finish ~18 minutes.
remove one package from a repo and sync it.
Update Content Counts stopped success July 17, 2024 at 02:39:00 PM 12 minutes
Update Content Counts stopped success July 17, 2024 at 02:37:21 PM 10 minutes
Update Content Counts stopped success July 17, 2024 at 02:34:45 PM 7 minutes
Update Content Counts stopped success July 17, 2024 at 02:34:34 PM 6 minutes
Update Content Counts stopped success July 17, 2024 at 02:34:21 PM 7 minutes
Update Content Counts stopped success July 17, 2024 at 02:34:15 PM 5 minutes
Update Content Counts stopped success July 17, 2024 at 02:34:10 PM 7 minutes
Update Content Counts stopped success July 17, 2024 at 02:34:01 PM 3 minutes
Update Content Counts stopped success July 17, 2024 at 02:33:49 PM 1 minute
Update Content Counts stopped success July 17, 2024 at 02:33:44 PM 50 seconds
Synchronize smart proxy ‘auys-fmproxy’ stopped success July 17, 2024 at 02:33:30 PM 6 minutes
Synchronize smart proxy ‘usak-fmproxy’ stopped success July 17, 2024 at 02:33:29 PM 1 minute
Synchronize smart proxy ‘thsr-fmproxy’ stopped success July 17, 2024 at 02:33:27 PM 4 minutes
Synchronize smart proxy ‘ushi-fmproxy’ stopped success July 17, 2024 at 02:33:26 PM 1 minute
Synchronize smart proxy ‘clpe-fmproxy’ stopped success July 17, 2024 at 02:33:25 PM 56 seconds
Synchronize smart proxy ‘cais-fmproxy’ stopped success July 17, 2024 at 02:33:23 PM 52 seconds
Synchronize smart proxy ‘usvf-fmproxy’ stopped success July 17, 2024 at 02:33:22 PM 39 seconds
Synchronize smart proxy ‘sebr-fmproxy’ stopped success July 17, 2024 at 02:33:21 PM 22 seconds
Synchronize smart proxy ‘sees-fmproxy’ stopped success July 17, 2024 at 02:33:21 PM 28 seconds
Synchronize smart proxy ‘clpa-fmproxy’ stopped success July 17, 2024 at 02:33:19 PM 51 seconds
Sync Repository on Smart Proxy(ies) stopped success July 17, 2024 at 02:33:19 PM 8 minutes

Add one package to a repo and sync it:
Update Content Counts stopped success July 17, 2024 at 03:07:44 PM 10 minutes
Update Content Counts stopped success July 17, 2024 at 03:02:40 PM 13 minutes
Update Content Counts stopped success July 17, 2024 at 03:02:40 PM 8 minutes
Update Content Counts stopped success July 17, 2024 at 03:02:40 PM 7 minutes
Update Content Counts stopped success July 17, 2024 at 03:02:31 PM 5 minutes
Update Content Counts stopped success July 17, 2024 at 03:02:13 PM 6 minutes
Update Content Counts stopped success July 17, 2024 at 03:02:04 PM 5 minutes
Update Content Counts stopped success July 17, 2024 at 03:01:56 PM 3 minutes
Update Content Counts stopped success July 17, 2024 at 03:01:53 PM 46 seconds
Update Content Counts stopped success July 17, 2024 at 03:01:47 PM 1 minute
Synchronize smart proxy ‘auys-fmproxy’ stopped success July 17, 2024 at 03:01:29 PM 6 minutes
Synchronize smart proxy ‘usak-fmproxy’ stopped success July 17, 2024 at 03:01:28 PM 1 minute
Synchronize smart proxy ‘clpe-fmproxy’ stopped success July 17, 2024 at 03:01:27 PM 1 minute
Synchronize smart proxy ‘clpa-fmproxy’ stopped success July 17, 2024 at 03:01:26 PM 1 minute
Synchronize smart proxy ‘ushi-fmproxy’ stopped success July 17, 2024 at 03:01:25 PM 48 seconds
Synchronize smart proxy ‘sebr-fmproxy’ stopped success July 17, 2024 at 03:01:24 PM 29 seconds
Synchronize smart proxy ‘cais-fmproxy’ stopped success July 17, 2024 at 03:01:23 PM 40 seconds
Synchronize smart proxy ‘sees-fmproxy’ stopped success July 17, 2024 at 03:01:23 PM 23 seconds
Synchronize smart proxy ‘usvf-fmproxy’ stopped success July 17, 2024 at 03:01:22 PM 34 seconds
Synchronize smart proxy ‘thsr-fmproxy’ stopped success July 17, 2024 at 03:01:21 PM 1 minute
Sync Repository on Smart Proxy(ies) stopped success July 17, 2024 at 03:01:21 PM 7 minutes

15

3.11.0/4.13.0, simlar behaviour as 3.10.0/4.12.1. Total time for all tasks to finish ~16-18 minutes.
remove one package from a repo and sync it.

Update Content Counts	stopped	success	July 17, 2024 at 06:44:51 PM 10 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:41:29 PM 12 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:39:10 PM 7 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:39:02 PM 6 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:38:36 PM 8 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:38:27 PM 6 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:38:22 PM 1 minute
Update Content Counts	stopped	success	July 17, 2024 at 06:38:22 PM 6 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:38:13 PM 3 minutes
Update Content Counts	stopped	success	July 17, 2024 at 06:38:13 PM 47 seconds
Synchronize smart proxy 'cais-fmproxy'	stopped	success	July 17, 2024 at 06:37:50 PM 1 minute
Synchronize smart proxy 'auys-fmproxy'	stopped	success	July 17, 2024 at 06:37:48 PM 7 minutes
Synchronize smart proxy 'sees-fmproxy'	stopped	success	July 17, 2024 at 06:37:47 PM 35 seconds
Synchronize smart proxy 'thsr-fmproxy'	stopped	success	July 17, 2024 at 06:37:45 PM 4 minutes
Synchronize smart proxy 'sebr-fmproxy'	stopped	success	July 17, 2024 at 06:37:45 PM 28 seconds
Synchronize smart proxy 'usak-fmproxy'	stopped	success	July 17, 2024 at 06:37:43 PM 1 minute
Synchronize smart proxy 'ushi-fmproxy'	stopped	success	July 17, 2024 at 06:37:42 PM 45 seconds
Synchronize smart proxy 'clpa-fmproxy'	stopped	success	July 17, 2024 at 06:37:40 PM 56 seconds
Synchronize smart proxy 'usvf-fmproxy'	stopped	success	July 17, 2024 at 06:37:39 PM 34 seconds
Synchronize smart proxy 'clpe-fmproxy'	stopped	success	July 17, 2024 at 06:37:38 PM 44 seconds
Sync Repository on Smart Proxy(ies)	stopped	success	July 17, 2024 at 06:37:38 PM 7 minutes

Add one package to a repo and sync it:
Update Content Counts	stopped	success	July 17, 2024 at 10:00:11 PM 12 minutes
Update Content Counts	stopped	success	July 17, 2024 at 10:00:11 PM 9 minutes
Update Content Counts	stopped	success	July 17, 2024 at 09:55:52 PM 7 minutes
Update Content Counts	stopped	success	July 17, 2024 at 09:55:52 PM 6 minutes
Update Content Counts	stopped	success	July 17, 2024 at 09:55:52 PM 7 minutes
Update Content Counts	stopped	success	July 17, 2024 at 09:54:50 PM 6 minutes
Update Content Counts	stopped	success	July 17, 2024 at 09:54:50 PM 5 minutes
Update Content Counts	stopped	success	July 17, 2024 at 09:54:48 PM 1 minute
Update Content Counts	stopped	success	July 17, 2024 at 09:54:42 PM 3 minutes
Update Content Counts	stopped	success	July 17, 2024 at 09:54:42 PM 48 seconds
Synchronize smart proxy 'auys-fmproxy'	stopped	success	July 17, 2024 at 09:54:18 PM 6 minutes
Synchronize smart proxy 'cais-fmproxy'	stopped	success	July 17, 2024 at 09:54:17 PM 2 minutes
Synchronize smart proxy 'thsr-fmproxy'	stopped	success	July 17, 2024 at 09:54:16 PM 6 minutes
Synchronize smart proxy 'sees-fmproxy'	stopped	success	July 17, 2024 at 09:54:16 PM 32 seconds
Synchronize smart proxy 'usak-fmproxy'	stopped	success	July 17, 2024 at 09:54:15 PM 35 seconds
Synchronize smart proxy 'ushi-fmproxy'	stopped	success	July 17, 2024 at 09:54:14 PM 2 minutes
Synchronize smart proxy 'sebr-fmproxy'	stopped	success	July 17, 2024 at 09:54:14 PM 28 seconds
Synchronize smart proxy 'clpa-fmproxy'	stopped	success	July 17, 2024 at 09:54:12 PM 2 minutes
Synchronize smart proxy 'usvf-fmproxy'	stopped	success	July 17, 2024 at 09:54:12 PM 30 seconds
Synchronize smart proxy 'clpe-fmproxy'	stopped	success	July 17, 2024 at 09:54:11 PM 39 seconds
Sync Repository on Smart Proxy(ies)	stopped	success	July 17, 2024 at 09:54:11 PM 8 minutes
16

So all versions 3.9.3/4.11.1 and newer are impacted.
With this behavior the Foreman server will take around 18 minutes for one repo update.
I have today 62 repos that are sync:ed each night so if say 30 of those gets a new package/image, we are talking about a 540 minutes or 9 hours to get all the metadata out to all proxies. Then during the day our developers push out another 30-40 new packages/images during a day and Foreman will not be doing anything else then running sync and " Update Content Counts" tasks all day.

Will wait for the sync during the night to run to see how it will behave (maybe some miracle job will run during the night that will speed things up), if nothing gets better then this I guess I need to go back to 3.8/4.10 again.

17

The issue we had for turning this automatic task off is in 4.13. Bug #37356: Slow smart proxy sync in 4.11 - Katello - Foreman

In prior versions the setting “Sync Smart Proxies after content view promotion” can be used to turn off automatic syncs (+ update content count actions) on repo syncs, cv promotions etc.

In 4.13, you’ll have the new setting “Calculate content counts on smart proxies automatically” to turn off the update content count task specifically.

The data you provided is really helpful and does point to update content counts running very long despite syncs running fairly quickly. We are working on a fix to optimize the content count task.

2 Likes
18

Thanks for the heads up about the “Calculate content counts on smart proxies automatically” setting, totally missed it. I have now set it to “No” and performed another test:

Synchronize smart proxy 'auys-fmproxy'	stopped	success	July 18, 2024 at 09:09:02 AM 55 seconds
Synchronize smart proxy 'usak-fmproxy'	stopped	success	July 18, 2024 at 09:09:01 AM 36 seconds
Synchronize smart proxy 'clpe-fmproxy'	stopped	success	July 18, 2024 at 09:09:00 AM 37 seconds
Synchronize smart proxy 'usvf-fmproxy'	stopped	success	July 18, 2024 at 09:09:00 AM 31 seconds
Synchronize smart proxy 'sebr-fmproxy'	stopped	success	July 18, 2024 at 09:08:59 AM 28 seconds
Synchronize smart proxy 'thsr-fmproxy'	stopped	success	July 18, 2024 at 09:08:58 AM 44 seconds
Synchronize smart proxy 'clpa-fmproxy'	stopped	success	July 18, 2024 at 09:08:57 AM 41 seconds
Synchronize smart proxy 'ushi-fmproxy'	stopped	success	July 18, 2024 at 09:08:56 AM 36 seconds
Synchronize smart proxy 'sees-fmproxy'	stopped	success	July 18, 2024 at 09:08:56 AM 28 seconds
Synchronize smart proxy 'cais-fmproxy'	stopped	success	July 18, 2024 at 09:08:55 AM 35 seconds
Sync Repository on Smart Proxy(ies)	stopped	success	July 18, 2024 at 09:08:55 AM 1 minute

So this totally fixed my issue.

Can you explain the “Calculate content counts” a little more? What do I miss with this turned off?
I read above it can be seen somewhere in the Foreman GUI?

1 Like