This topic seems to pop up, provide some excellent discussion, possibly a few points of progress, before slipping back beneath the surface again.
Is there any work being done in this direction at the moment? It would be great to be able to build a Katello instance which is HA and capable of scaling.
Hi,
there are several people who took a crack at it and got very close, but it never got into production ready state. Last blocker IIRC was Pulp2, which is now resolved as Pulp3 should be container ready. If youâd like more info, contact @Ondrej_Prazak whoâs done most of the work on the last try.
Today I think we are just lacking resources (mostly hands) to take another look at it and come up with good enough production ready solution, test it, document it 
What are your interests in it? Would you be able to help by any means?
I have worked as a Linux Engineer for a range of banks over the years, all Satellite customers. Not one of them is impressed with the lack of HA or scalability from the downstream product.
For years it has seemed to me to be obvious that the best way forward for HA and scalability of a Katello/Satellite environment is through splitting the services into smaller, properly self-contained modules. These could either then be split onto multiple servers, or containerized and run on a container platform. either route would be good.
Yes - I have made sure these clients have reported this in support tickets. Sadly to no avail. But itâs a significant change, not an overnight fix.
But with MongoDB being removed from the equation, perhaps one of the bigger blockers has gone?
My coding isnât up to project standards, but Iâm a good troubleshooter and problem solver.
Thatâs an awesome initiative!
Personally I took a stab at an easier task - creating a containerized test environment, which went pretty well.
A couple of improvements are needed there, but all in all the container is workable.
I would suggest trying to tackle smaller tasks, like creating a containerized development environment before we try and tackle the âproductionâ setup.
I see a couple of issues with a production container - for example the fact that there will be no way to customize the plugin installation process. Besides that, I really donât know how easy it would be to work with certificates in a prebuilt katello image.
@ehelms , as one that already played with the idea of contanerizing Katello, can you share your thoughts on it?
I feel somewhat qualified to weigh in here. Itâs not just the MongoDB removal but rather the whole architecture of Pulp 3 thatâs much better suited to scaling out. And itâs not just Pulp 3 but what Iâve also tried to focus on is how you configure things. For example, Katello is slowly learning to properly direct clients to the correct URLs. This helps with splitting things up.
Concrete example: Katello used to assume that the hostname in the Smart Proxy URL was also the hostname in the content URL. It now learns the URL from the Smart Proxy configuration. This allows multi homing and/or splitting: you can run the Smart Proxy on sp-nnn.mgmt.example.com but direct clients to pulp-content.example.com. As I always said during the design phase: it should be possible to run Pulp in a Kubernetes(/OpenShift) cluster and have a Smart Proxy just to point at it.
And on a similar note, weâre laying the ground work for doing the same thing with the subscription service.
So while not directly targeted at containerizing, I strongly believe we must build for HA support to really support a containerized deployment. Thatâs why Iâm trying to decouple services and prepare designs for such a world.
Another thing that I worry about are properly dealing with failure modes. What if a service is unavailable? Right now a lot is built with the assumption that a backend service is available. This is also why upgrades often work best with the blunt âshut down all services, upgrade, start them up againâ. In a containerized world that is probably a lot harder.
Yes - I was over-brief in my first post, but I do know that there are loads of places where hostname assumptions are made. Or even âlocalhostâ being used (although that may be somewhat apocryphal).
There must be a multitude of issues which need resolved on the march towards containerization. The vast majority of those will be resolved as part of other work I would guess. Is there anywhere tracking the blockers to containerization? Would be great to see the entire scale of the work ahead.
With you 100% on this. Even if not containerizing the product, it would provide the only credible HA solution by splitting various processes onto different load balanced servers.
Indeed. Upgrading a containerised system would be tricky. Would be great to be able to replace containers running the old version with containers running the new version of each microservice. Then upgrade the databases/schema etc. I am oversimplifying there, but it would be a neat trick.
And if it was âjustâ HA microservices running on multiple servers, it would be outstanding to be able to run a yum upgrade on all the app servers, then finally run the database/schema upgrade once all services upgraded.
D