there are several people who took a crack at it and got very close, but it never got into production ready state. Last blocker IIRC was Pulp2, which is now resolved as Pulp3 should be container ready. If you’d like more info, contact @Ondrej_Prazak who’s done most of the work on the last try.
Today I think we are just lacking resources (mostly hands) to take another look at it and come up with good enough production ready solution, test it, document it
What are your interests in it? Would you be able to help by any means?
I have worked as a Linux Engineer for a range of banks over the years, all Satellite customers. Not one of them is impressed with the lack of HA or scalability from the downstream product.
For years it has seemed to me to be obvious that the best way forward for HA and scalability of a Katello/Satellite environment is through splitting the services into smaller, properly self-contained modules. These could either then be split onto multiple servers, or containerized and run on a container platform. either route would be good.
Yes - I have made sure these clients have reported this in support tickets. Sadly to no avail. But it’s a significant change, not an overnight fix.
But with MongoDB being removed from the equation, perhaps one of the bigger blockers has gone?
My coding isn’t up to project standards, but I’m a good troubleshooter and problem solver.
That’s an awesome initiative!
Personally I took a stab at an easier task - creating a containerized test environment, which went pretty well.
A couple of improvements are needed there, but all in all the container is workable.
I would suggest trying to tackle smaller tasks, like creating a containerized development environment before we try and tackle the “production” setup.
I see a couple of issues with a production container - for example the fact that there will be no way to customize the plugin installation process. Besides that, I really don’t know how easy it would be to work with certificates in a prebuilt katello image.
@ehelms , as one that already played with the idea of contanerizing Katello, can you share your thoughts on it?
I feel somewhat qualified to weigh in here. It’s not just the MongoDB removal but rather the whole architecture of Pulp 3 that’s much better suited to scaling out. And it’s not just Pulp 3 but what I’ve also tried to focus on is how you configure things. For example, Katello is slowly learning to properly direct clients to the correct URLs. This helps with splitting things up.
Concrete example: Katello used to assume that the hostname in the Smart Proxy URL was also the hostname in the content URL. It now learns the URL from the Smart Proxy configuration. This allows multi homing and/or splitting: you can run the Smart Proxy on sp-nnn.mgmt.example.com but direct clients to pulp-content.example.com. As I always said during the design phase: it should be possible to run Pulp in a Kubernetes(/OpenShift) cluster and have a Smart Proxy just to point at it.
And on a similar note, we’re laying the ground work for doing the same thing with the subscription service.
So while not directly targeted at containerizing, I strongly believe we must build for HA support to really support a containerized deployment. That’s why I’m trying to decouple services and prepare designs for such a world.
Another thing that I worry about are properly dealing with failure modes. What if a service is unavailable? Right now a lot is built with the assumption that a backend service is available. This is also why upgrades often work best with the blunt “shut down all services, upgrade, start them up again”. In a containerized world that is probably a lot harder.
Yes - I was over-brief in my first post, but I do know that there are loads of places where hostname assumptions are made. Or even ‘localhost’ being used (although that may be somewhat apocryphal).
There must be a multitude of issues which need resolved on the march towards containerization. The vast majority of those will be resolved as part of other work I would guess. Is there anywhere tracking the blockers to containerization? Would be great to see the entire scale of the work ahead.
With you 100% on this. Even if not containerizing the product, it would provide the only credible HA solution by splitting various processes onto different load balanced servers.
Indeed. Upgrading a containerised system would be tricky. Would be great to be able to replace containers running the old version with containers running the new version of each microservice. Then upgrade the databases/schema etc. I am oversimplifying there, but it would be a neat trick.
And if it was ‘just’ HA microservices running on multiple servers, it would be outstanding to be able to run a yum upgrade on all the app servers, then finally run the database/schema upgrade once all services upgraded.