General question about how Katello manages identical content views being in multiple LEs.
I have a setup with several LEs (Eng>RC>UAT>Dev>Test>PAT>Prod) and multiple smart-proxies.
We find that when a new release is created and promoted to Eng, Katello does a bunch of work and then the content is synched out to the smart proxies, which also do a bunch of work.
e.g. Create CCV-stable version X
Then when we promote that CCV version into RC - Katello appears to do a bunch more work. And then the smart-proxies also do a bunch more work. Pretty much the same volume of work as for the Eng release.
Same pattern emerges when we promote these identical content views (CCVs in this case) all the way through to Prod. It’s as if the CCV and it’s associated repos is being re-created in each of the LEs.
Does Katello do any form of Copy on Write to link CCV promotions back to instances which have already been released? I’m sure the actual content within the CCVs is referenced via pointers as we don’t see storage shooting through the roof. But it certainly looks like the metadata around each CCV is rebuilt from scratch each time the CCV is promoted to a new LE.
If CCV-stable version X already exists in Eng, surely when I promote it through to Prod each LE could just point back to the version in Eng? It’s not as if changes are made to a CCV version after it’s promoted. If we do make changes, that comes in the form of another CCV version being published.
We have a lot of de-duplication in play when the same “version” is published to multiple environments. There is also some data that needs to be created for these repos in the new environment to make it available to hosts and to let katello (and pulp and subscriptions) know about the existence of this new repo in the new environment. The actual content copying which is the long-running task is not triggered on simple promotes for yum repos (but it is with non-yum repos.)
In general, if CVs don’t have any filters, we do not copy content individually and simply point the new version repos to the same back-end version even in case of publishing new versions which makes the whole thing much faster.
Smart proxy syncs are slightly different because it lives outside the purview of katello on the main server and we do create and sync new repositories for every promote which creates all the pulp artifacts necessary to serve content at the distribution paths. There has been some discussion around reducing the amount of data we create on proxies to support our workflows and hopefully we will de-duplicate more than we do today on the proxies.
So this kind of matches what we see in the main Katello server. Although we do see Publish & Promote taking longer over time. So after 6 monthly releases, the Publish/Promote of the next release will be measurably longer than the first. But that’s not my immediate concern.
We are dealing with nearly exclusively yum repos. We have a single file repo in our CCV.
It’s definitely the effect on our smart-proxies which is causing the problem. If there is no de-duping or back-linking of identical content to a primary source, then I would say this exactly matches our experience.
In order to get our historic content views pushed into new infrastructure, we had to run the publish/promote without any sync to the smart-proxies. Then trigger individual LE syncs on each of the smart-proxies.
So a round of syncs was triggered via hammer with “–lifecycle-environment-id 2”. Once that was complete another round of syncs to LE 3, and so on through to LE 8. Performance data collected from the smart-capsules shows that each of these syncs has nearly an identical profile on CPU, network IO, disk IO, etc.
This is what lead to my conclusion that the smart-proxies aren’t doing any de-duping. Thanks for confirming this.
