So, after having spent a number of hours yesterday chatting with the knowledgeable people over in the Pulp users chat channel, we’ve learnt more about this issue, and come up with a combination of changes which affect a workaround.
The timeout issue is caused by a number of things:
- yum.oracle.com/Akamai download throttling - this one can be really obvious: at times I saw really high download speeds for a while, which would later drop down to much slower speeds
- The default download timeout settings in Katello (“Sync Connection Timeout” in Foreman settings, under “Content”) of 300s or 5 minutes. When the downloads are really being throttled this can easily be hit.
- The OL8 appstream repo has quite a few large SRPMs which vary in size from 1.3GiB to 2.5GiB. When “mirror on sync” is enabled (even if you’ve enabled “Ignore SRPMs”) SRPMs get downloaded, as Pulp is providing a complete mirror of the repo.
- This is an arguable one…Pulp is meant to retry a download a number of times (defaults to 3), but doesn’t do this when a download hits the timeout period.
- If you are performing one of these large syncs, and get part way through before the sync fails with the timeout, it seems everything which was downloaded into the tmp directory is thrown away, and not kept. So, the next time the sync is tried, it has to download everything again.
So, the combination of changes to manage a workaround:
- I set the timeout to 3000 seconds, but without the other changes this still wasn’t enough, because even that timeout (50 minutes!) was getting hit.
- Change concurrent downloads from the default of 10 to 5. Ditto the last sentence of the above.
- Change the repository to turn off “Mirror on sync” and turn on “Ignore SRPMs”
- Point 3. above then managed to hit another Pulp issue for these repositories, which would cause the sync to fail for another reason, albeit a lot sooner in the sync…
- In /etc/pulp/settings.py set “RPM_ITERATIVE_PARSING = False” - this changes the RPM metadata parsing to an older routine, the routine which led to very high RAM usage in pulpcore-worker instances during sync of repositories with large metadata sets, like OL8/RHEL repositories.
All of the above leads to syncs which work, but don’t download SRPMs, so you have to accept that restriction. Also, RAM usage is high during the sync runs of these repositories.