@Ondrej_Prazak and @ohadlevy, who have both spent time looking into containerizing Foreman, will give an overview of their findings.
This event will be livestreamed on YouTube and web chat will be open for questions.
However, the meeting is open and if you would like a link to the meeting, please send @mcorr a message or reply here.
Please
Hello @canob
Welcome to the community.
I will DM you a link the week of the meeting 
Thanks!
Link to the livestream for later today:
Hi all,
Thanks to @Ondrej_Prazak and @ohadlevy for a great discussion.
A lot of questions were raised as part of this session.
I would very much like to hear everyone’s thoughts.
When you get a chance to watch back, if you’ve any questions, let’s continue the discussion here.
Thanks indeed.
First of all, I think I agree the biggest thing is more of an organizational choice than a technical choice: do we want to.
I’ll also agree that a lot of the problems have been solved. Not by us, but by others. It does mean we need to adopt current best practices. https://12factor.net/ is now many years old and Foreman is far from compliant. Let alone even more modern best practices that have evolved in a containerized world.
That makes me think that other than wanting to, we also need to educate.
This is a large effort and various open source projects have taught us that you can do it in two ways.
The first is in separate branches. These branches have historically been very hard to maintain and I think it’ll be obvious to most that this is undesirable.
The second is incremental improvements. That’s what I’ve been trying to apply. One example is that in Foreman 2.1 we’re switching from Passenger to Puma. This slowly decouples us from Apache. It turns out that this breaks external authentication we currently use (#30535). In Foreman 2.0 we introduced Keycloak as a feature, but that may also be broken under Puma. Monitoring of it also hasn’t been solved yet. Consolidating The Console is another that’s been proposed about 2 years ago and has seen very little progress, but the current model is just broken in many setups.
That leads me to conclude while the best practices in the industry have been widely known for years, but there has been minimal effort on the Foreman project to actually follow those. The only solution I see is that people who actually care about this put the effort to make this happen. Sadly, I don’t see that happening so I’m willing to bet some beers that in 3 years we’ll be in largely the same position.
I couldn’t join this live, but watched the recording as soon as I could. Thanks for an enlightening discussion.
I hope someone (well, more than one to be honest) cares enough to move this forward. From a user/customer perspective, HA and horizontal scaling are two of the weak points for the Foreman/Katello stack.
Perhaps these things can be solved without moving to containerized microservices, but they go in such similar directions that moving the solutions forwards together would be beneficial. Does anyone care enough about horizontal scaling or HA? I wouldn’t say that nobody cares, but it seems to be a goal which always has a lower priority than other tasks.
A Foreman/Katello deployment of fully containerized microservices should still be viable on a single host. And with slightly more effort, on a large cluster of K8s container hosts. The level of scalability and HA is then up to the user.
It would be a shame to be buying beers for @ekohl in 3 years time.
D
I do agree. These things should have been solved years ago. Most of the problems are not hard, but require effort from multiple people. I have been pushing for a more isolated design with Pulp 3 that I believe prepares us much better for these kind of things. Rather than guessing or hard coded configs, there is a lot more service discovery. Pulp’s hostname is no longer tied to the Smart Proxy’s hostname. My theory is that allows deploying a HA scaled out Pulp 3 setup and just point a single Smart Proxy to it for service discovery. With things falling more into place, it’s on my agenda to deploy such a setup to test this theory.
If that works, the biggest limitation is solved. With removal of Pulp 2, we can also drop qpid so that’s another thing we no longer have to worry about. Then the RHSM endpoint is another. This is currently tied to the content proxy, but IMHO should be its own Smart Proxy feature to allow service discovery. This allows running the Smart Proxy on some private IP and the RHSM API endpoint on a public IP. Again, all about decoupling services.
Authentication between services is another thing that deserves a good look. I think certificates may not be the best fit in a containerized world.
This is exactly what I’d advocate. We can solve a lot of the application issues in the current system which open up the path to a properly containerized platform. That a path is open doesn’t mean we need to take it, but at least we’ll have a more stable and well behaved application.
I’d say people care about it, but you need 3 things:
At most people have 2 out of 3.
It may help if Red Hat Satellite customers tell their account manager that they care about this. when I ask about it I usually hear that customers don’t care (enough) about it and will just vertically scale up a machine or deploy multiple instances. However, I have talked to multiple customers who have said that if it was an option, they would appreciate it.
For community members it really comes down to contributing code.
I have a near identical experience. It just doesn’t seem to translate (yet) into customers directly requesting HA & scaling. Which is annoying as no noise means no action. To be fair to customers, they are still prioritising functionality RFEs which are perceived as gaps from the old Sat5 product.
I would like to think I have all 3 (although Skills is debatable), they just rarely manifest themselves at the same time. Specifically Dedication and Time seem to be on an inverse relationship with each other. 
I’ve always though DogTag would be a good fit here, but I don’t have much experience down at the engineering level. It does a good job in FreeIPA though. Is it too cumbersome here? As a private infrastructure for Foreman+Plugins ?
Yes, I should have mentioned that they have to be present simultaneous.
I’ll be honest and say I’ve only glanced at it. If we look at Katello, then it requires a custom certificate with a specific name for Candlepin (Feature #30368: Reuse the Foreman client SSL settings for Candlepin - Katello - Foreman) and the same for Pulp 3 (no issue yet). If a Katello install didn’t need 3 separate sets of key/cert to connect to external services, then it would already be much easier to use today. I think both can done while keeping the current security level.
I have questions on two items which were not covered in the talk
I guess it depends on how we progress, e.g. are we doing containers inside rpms for a while, this would not change at all (just different kind of wrapping), the main difference would be when you start assuming there is something like Kubernetes available, in that case, I would imagine it to be something like:
Minishift was (its no longer available since openshift 4) a tool for developers, in upstream kuberenets developers use minikube, or for openshift the alternative is CRC (Code ready containers).
There are multiple ways to get working on kube, including every cloud provider that has its own set of kuberenets offering.
for upstream, I would consider a light container runtime, maybe something like fedora / rhcos (no kubernetes) - e.g a vm image / iso or using something like GitHub - home-assistant/operating-system: đź”° Home Assistant Operating System.
at the end of the day, it will reduce the various combination of software stack that we need to test and certify.
there is always the docker-compose / systemd path as well to run on any regular linux machine…
I have been thinking about the next steps and I came up with the following:
Create a repo under theforeman namespace on Github where the container-related efforts will be concentrated.
Create a corresponding project in Redmine for tracking issues.
I believe both will increase the overall visibility and provide a clear way for community members to contribute.
The initial scope for the new repository would be:
Quick demo/preview of Foreman using containers - I see 2 areas where this would have a value:
Ease up the burden for packaging by running images under systemd. The intention is to reduce the amount of time and work spent on packaging.
Thoughts? Any suggestion on the repo name?
I was wondering if we want to experiment with creating a new way to deploy foreman, something like how GitHub - home-assistant/supervised-installer: Installer for a generic Linux system is deploying on an existing operating system, where later a different application (GitHub - home-assistant/supervisor: 🏡 Home Assistant Supervisor) owns which containers are installed, how they are configured and updated and expose an api for the ui to interact with and control the whole application lifecycle.
We might. It reminds me of something that @Marek_Hulan mentioned - configuring Foreman from within Foreman, like installing additional smart proxies for example.
I like attempting to track it as a main citizen of the Foreman ecosystem as a way to invest and build growth in the area. From my view, for containers to be a deployment vector all of the existing elements of release need to exist to do automated build and verification.
The only way I see this reducing the burden on packaging potentially is building the containers with RPMs and running that same container on Debian. That at least drops native Debian packaging. RPM packaging has a much much longer road to be removed from my view point.
I hear grapple is a pretty good name
I missed the three year mark of this post, but I think you’re owed beers right?
I suppose so, but recently there’s been some progress. Back in December @ehelms, @evgeni, @akumari and I met in person to discuss a long term strategy here.
This came from a request from more and more Satellite customers to support HA. I strongly feel that the classical HA deployment methods are hard to support generically and relying OpenShift is a logical choice to make it manageable.
However, we want to continue supporting single-machine deployments without heavy tooling requirements like OpenShift.
Since this original talk, new tooling has popped up. In particular, we see Podman’s quadlets as a potential solution to this. See Make systemd better for Podman with Quadlet for more.
Those who have paid attention to the repositories may have seen GitHub - theforeman/foreman-quadlet and it’s where we’re now experimenting with building up such as stack and where the issues are (like with certs: configure ssl_ca (if set) when using Net::HTTP by evgeni · Pull Request #40 · theforeman/smart_proxy_pulp · GitHub & Fixes #30368 - Make Candlepin CA file optional by ekohl · Pull Request #8832 · Katello/katello · GitHub).
I have good hope that soon we can consider the proof of concept done and can reflect on it. With that knowledge, we can look at the path ahead.
Also note that while the request came from downstream customers, we will continue to work in an upstream-first way (as we’ve always done). Hence why the experimental repo lives here in upstream.