I apologize for being harsh, I am little surprised and scared. Let me explain my thinking. Our Rails codebase and dependencies were never considered to be thread safe, there was no discussion, no extensive testing. Now, I believe that Dynflow at some point loads Rails environment with all our code and deps, at least in one thread. We had just one thread, now we have two concurrent threads at least (maybe more) in our Rails app.
You said that Dynflow works multi-threaded now for over 5 years with nothing dramatic, sure but on the other hand we have Dynflow execution workflow and Rails execution workflow in one process and that creates vastly different environment. It used to be separate process with different execution behaviour. We don’t know what to expect, that’s my main point here.
My concern is all about complexity - we already started using multi-threaded web server in development starting with Rails 4.x, this already created some pains, for example I was debugging concurrency problem around logging for a week (https://github.com/theforeman/foreman/pull/5197). If we don’t have any plans on how to move on to multi-threading in production yet, so it makes little sense to me to spend our cycles on debugging concurrency issues when we don’t know if we are actually going to use this in the end. The fact Rails + Dynflow now works for few months without major issues in development setup does not tell anything about behaviour in production - that’s totally different workload and story.
We need a plan, discussion and proper production-setup testing to identify possible concurrent issues. Now, I think things will go bad, but maybe I am wrong and our tests will show that our application executes just fine. But we need to be sure before we start playing around with multiple threads in our development setups, for this reason I used the “no-go” term because I was always told (and I experienced myself) that debugging concurrency issues is much more difficult. Whatever the outcome of the testing is, I’d rather be debugging concurrency issues in two separate processes - Rails app (web requests) and Dynflow (background jobs) both in production or development - just to make things easier and more consistent.
Here is my proposal to workaround my concerns:
- move Dynflow to Procfile
- perform production multi-threaded tests of various scenarios
- analyze the results
Then we can decide to leave this status quo properly:
- Start moving towards multi-threading in production (move away from Passenger I guess).
- Keep using single thread and also disable multi-threading in development to avoid further pains.
This is the reason why I think putting Dynflow into Rails was a bad idea in the first place - it brings nothing but possible issues, unless we decide to go towards multi-threading. I’d appreciate stepping back a bit in this case, waiting is also an option, maybe the Ruby 3x3 will solve concurrency for us, maybe single-threaded containers are the answer, I don’t know. I’d like to hear opinion from folks about moving Dynflow to Procfile and future of deployment in terms of threading and deployment.
