This RFC proposes to switch the production application server for Foreman to use Puma and Apache to reverse proxy to the Puma server by default. This has the following benefits:
Aligns production and development environments
Aligns proposed container deployment methods with standard service deployment
Provides easier path to running Foreman as a systemd container
Separates SSL handling and load balancing from application server
As part of the design, changes will need to be made to the way puppet-foreman handles deployment. Some of the open questions that will heavily influence this design:
Do we keep Passenger as an option or keep opinionated and stick to Puma + Apache?
Do we allow switching between Apache + Passenger and Puma/Systemd + Apache for deployments?
Do we make the choice of application server configurable (e.g. use of gunicorn, etc.) ?
Initially I’d propose to add an option to configure as a reverse proxy instead of using passenger. We already have an option to use a (systemd) daemon. AFAIK we need to export some extra variables to use puma.
Another application server could be done by making the service name a parameter but I’d not focus on it initially.
As an observer (“someone not fully understanding every technology in the Foreman stack”) can I ask what it’s replacing? I presume it’s tomcat - but is there any more than that?
As I see it: before it’s Apache + mod_passenger to serve the Ruby on Rails application. After it’s Apache (with mod_proxy) + puma to serve the RoR app. It could easily be any other webserver to proxy (like nginx) but I’d prefer to stick with Apache in our default installs for a while.
Clear +1 from me, I think if this unlocks websocket support it’s worth the effort. That’s also why I would be opinionated about this and drop mod_passenger support as we won’t be able to get websockets with mod_passenger + apache.
I’m curious if there is any performance impact to this. That’s why we could still offer a fallback to mod_passenger. We could tell users that if they’re having issues they can fall back to passenger but should let us know to investigate as we’ll eventually drop passenger for good.
How do you want to handle Client Certificate Handling? I currently know three places where we use this (Foreman, Katello and Foreman DLM Plugin). Do you want to configure apache to pass ssl headers to puma?
Would we gain anything from that? I would try to keep the stack streamlined and just offer puma unless there is a valid reason to provide anything else.
Only offering and supporting one stack to me seems like a much better approach, otherwise I’m afraid we’ll be stuck in the middle supporting both for a long time.
If I understand correctly, the main concerns are around performance and certificate handling. If we can show that puma is capable of handling these issues just as well as passenger, I’d be all for replacing one with the other even for just for enabling websockets support, not to mention the other motivations Eric mentioned in the OP.
I had a quick stab at refactoring the puppet-foreman module to make the Apache config support passenger optionally. Unit tests pass but the current setup probably doesn’t work yet:
One goal I already had is to make https://github.com/theforeman/puppet-katello_devel/blob/master/manifests/apache.pp obsolete. We can probably reactor it in such a way that it uses the foreman::config::apache class which makes using the puppet-katello class a lot easier thus greatly reducing the code needed in puppet-katello_devel. This goes a long way in that direction.
Another benefit is that the SCL situation is a lot easier when you don’t have passenger in the mix. Building RPMs for it is complex.
Update: the above PR does mostly work, it configures Apache as a reverse proxy and launches the standalone foreman service. The acceptance test can retrieve the login page but is failing because it can’t properly detect whether the service is running and enabled for some unknown reason.
What’s left to do is modify the standalone service to use puma. Not sure what’s the cleanest place to do so: packaging or puppet-foreman.
Big +1 from me, maintaining SELinux policy for passenger was a HUGE pain. Also scaling was more difficult since we could not easily configure Passenger FOSS to recycle processes after reaching memory limits.
Also initial work towards Puma on proxy is pending initial review, this is a good timing - having everything on a single stack is ideal.
I vote for being opinionated about Puma here and taking advantage of good timing (post 1.20 branching). If possible, in cooperating with Red Hat Satellite Performance team who could help us scale up 1.21 a bit and provide feedback to the community so we can decide or fix bugs.
It looks like Puma either support app preloading or rolling restarts (they call this phased restart). Both are quite useful features, which one we will configure by default? To me it looks the former is more useful.
Not to me, I offered it as a question to get the response you gave I think as a community we’ve leaned towards choice rather than being prescriptive and thus left this question here to gauge thoughts. Generally from this thread I get the sense of pick one way and go that path rather than providing a matrix of options to users which I generally I like as it reduces testing, support, etc.
I know, @ekohl can speak for himself, but I’m so excited, that I want to share some progress.
@ekohl has a working prototype running. The only real obstacle we faced was how the client certificate is passed to the rails app. We have to use headers that are set in the apache reverse proxy but that replaced newlines with spaces. When we then try to load the certificate in ruby, it fails.
So basically we have to teach our rails app to handle this. I guess it would be best to introduce some middleware that handles this throughout the app.
did anyone look into memory usage over time with Puma? the main benefit
with passenger was the ability to scale worker, and recycle them over xxx
requests, this would ensure no memory abuse happen over time.
Puma does have a cluster mode, but afair, we are not planning to use it…
in the old days i used to use a tool like monit or god to ensure memory
thresholds for web processes, imho we would probably need to introduce such
a tool along with the change…?
lastly, regarding websockets, IMHO we should not default to ruby stack for
websockets, its known as a non ideal, imho we should consider exposing ws
though apache to another process (we can use the same port and route WS/WSS
traffic to another local daemon).
Puma does exactly the same, you need a configuration option to enable this. On top of that, it can also do rolling restarts or app preloading (these two features are exclusive - you need to pick one or another).
Oh we do plan to use it, except for proxy where we likely do this later (per module). Ohad, please keep in mind there are currently two concurrent threads about Puma:
bringing Puma onboard for Foreman Rails app (this one)
migrating foreman-proxy from webrick to Puma (POC PR already exists)
So to recap - this is the plan at least how I understand it for now:
foreman rails - puma in cluster mode, single-threaded
foreman proxy - puma in one-process mode, multi-threaded
Puma has a nice plugin architecture (very simplified tho) and this kind of tool already exists, for example https://github.com/schneems/puma_worker_killer and I believe we should be doing this from the day one.
I think as a community we’ve leaned towards choice rather than being prescriptive and thus left this question here to gauge thoughts. Generally from this thread I get the sense of pick one way and go that path rather than providing a matrix of options to users which I generally I like as it reduces testing, support, etc.