RFC: Add puma as the default smart-proxy server


#16

Even though I don’t see connection stacking issue in my environment, slow-header tests definitely show service impact for SmartProxy that may explain some instability of the latter in our production (all of the sudden I see that no responses coming back from foreman-proxy yet the process is running - I think I posted a question about that in a support forum at some point back).

Also, increasing performance 3x is a definite and a big “yes” factor, so I’d love to run Puma under our production load and see how things are. Are there any (even preliminary) docs describing a process of SmartProxy migration to Puma?

Thanks!


#17

You are reading the document right now :slight_smile:


#18

Well, yes, but I meant a doc that may have actual steps like those that you had to do to perform testing of smart-proxy under Puma vs Webrick. So unfamiliar with Puma like myself could try it before the deployment procedure is fully automated (under installer or whatever). Again, if there is such a doc already.

Thanks!


#19

The code located here: https://github.com/ruby/webrick/blob/master/lib/webrick/httpresponse.rb#L108

I’ll try to figure out how to change it, I have no experience effecting webrick directly.


#20

Then turn it off and try with telnet, I am pretty sure this does not affect slow DoS attacks at all.


#21

I have mixed feeling about the subject following some tests I’ved made, and took also the weekend to think of it a bit before answering.

A. In order to turn off the response, we need to entangle ourself more with Webrick.
I was unable to access the response settings at all with the current way we implement things.

B. Moving to Puma, means a lot of other work such as supporting scl, packaging will change, and I haven’t researched how to do it, but following a lot of reading about both Rack and Webrick, I think it’s more or less the same as Webrick in that case.

Being more server agnostic means that we need to inject code on runtime based on the server type.
But it can render new issues that I don’t really think we should have.

That’s why I have mixed feelings here.
I dislike the idea that just because of settings we are failing here, but making SP more entangled with a single server I’m not sure is the proper way to go as well.

BTW I didn’t test a standalone Webrick app that is fully configured, because of the above issue.


#22

It’s a single gem without other dependencies [1]. It requires Ruby 2.2, which is why I guess you mention SCL. We could use 3.11.4 (just two months older version) that is compatible with 1.9.3+. According to [2], we wouldn’t lose much. If puma opens a potential to at least test proxy in multithreaded setup, I think it’s a big win already.

Is there some other work that I’m missing?

[1] https://rubygems.org/gems/puma
[2] https://github.com/puma/puma/blob/master/History.md


#23

And then things become more interesting IMO.

We are starting to use SCL right?
So now we can also add higher version of Ruby as well, because we are controlling now the installation.
And now we are at a dependency game that each decision can be taken based on packaging abilities, rather then what arrives from the system itself.

So it’s like a Pandora’s box here, if you start with even a single dependency management using SCL, it’s an opening for everything else.

Should we take that path?

Sorry for being the Devil’s advocate on this discussion :stuck_out_tongue:


#24

Sorry I don’t follow.

Why he have to move to SCL? If we take 3.11.4, it will run on EL7 without SCL as its Ruby is 2.0.

Let’s say we decide to SCL-ize the proxy, the majority of our packaging already uses SCL, proxy is just a small subset with limited dependencies. How would that be different from core? What problems this brought to core so far?

What’s the difference between packaging our dependencies in SCL and non-SCL? Most of gems don’t have RPMs and we need to build them anyway. See how many things we package today [1]. Most if not all specs has an SCL variant and it’s just a matter of switching configuration how we want to build them AFAIK. I think it would be best to ask @packaging team for their potential concerns.

[1] https://github.com/theforeman/foreman-packaging/tree/rpm/develop/packages/foreman


#25

You always want to be able to update things if there is any security issue, or there are known bugs.
I dislike the idea that the distro handles the Ruby support.
The reason for it is that for example ruby v2.0 is not supported anymore by Matz, so if there is any bug that requires update, we can’t count on the distro to solve it, because they will not have anywhere to take an update, and sometimes there are new implementations at the Ruby level (e.g. rewrite of the GC that happens many times).

That’s actually my point, or at least what I have tried to raise :slight_smile:

We either control everything or nothing, we can’t just control Puma for example.
That’s the point I was trying to make.

But my question is about the energy and effort on the matter. What is better here?

M original question is, should we control webrick, because it is less of an effort then starting SCL, even though it will render us single server solution.
But controlling Puma open new issues, for example requests to control unicorn for example or other rack related server.

And that was another point of mine in the matter.


#26

Why don’t we just open a PR that moves smart-proxy to puma. I think, there are a lot of advantages and this is completely independent from SCLing the proxy. That way we see how much effort this actually is and can test some modules. We can then return to the discussion once we have learned more and have actual experience.


#27

From a packaging perspective I wouldn’t mind SCLing the proxy. We could solve the dynflow situation with _core packages could be solved much cleaner. On the packaging side we’d also simplify a lot of things when we only rely on an SCL for Ruby. For that we’d also need to SCL the installer but nowadays that could be done (why we couldn’t before is a bit off topic here). On the CI side it’d also simplify our test matrices when we can move to Ruby 2.3+ for everything. IMHO SCLing shouldn’t be a blocker for moving to puma.


#28

That’s what I was trying to say. If we use puma 3.11.4, the SCL decision does not have to be made now.


#29

It’s on me, I’m going to support Puma after the shutdown we are having.

My discussion is that prior of going and doing that, let’s talk about the cost of it, and if it is really needed, something that I still fill we are not talking about.


#30

Following discussion on the IRC, the move for Puma will happen, and we will open a new discussion regarding SCL


#31

Can you explain this? I am not following.

I feel there is a lot of noise about this RFC but the reality is not that complex. Eric wrote that SCLing is out of table now, but Ewoud is now telling us on the IRC that SCLing “should not be that of a big deal”.

Of course, if we are going to SCL proxy anytime soon we can easily bump to more recent version of webrick which is little bit better.

This RFC would be still valid tho, it’s worth exploring what Puma could give us. The good approach will be evolution and not revolution, therefore some kind of a “tech preview” would be appropriate.

Honestly, I currently don’t have time to work on Puma support, anyone is welcome to take a look on this. I thought it’s good idea.


#32

That’s not how I interpreted this reply:

It’s still a decently sized effort and I never denied that, but as I said: it’s not a blocker. It does need some planning and effort. With the 1.20 merge window coming up in less than a month I’d say it’s more 1.21+ thing.


#33

After the shutdown I’ll change the code to support Puma. unless you did/do it :slight_smile:


#34

If you plan working on this, please keep webrick as opt-in so users can fallback. We don’t know all the consequences e.g. on Windows of this.


#35

Ack