In the early days of remote execution, we wanted to get dynflow to the smart proxy to be able to reuse some code we had on Foreman’s side and to have the same engine on both sides. Sadly at the time, smart proxy wasn’t SCL’d and Dynflow required newer ruby than what was available outside of SCL.
To address this situation, we created
smart_proxy_dynflow_core is a sidecar service to the smart proxy, essentially just Dynflow runtime with an API.
smart_proxy_dynflow is a smart proxy plugin which rewrites requests coming to
/dynflow/* and proxies them to
From this point onwards, I’ll refer to
To make things more interesting,
SPDC was written in a way that it could still act as a regular smart proxy plugin, meaning that if all dependencies could be satisfied, it could be loaded into smart proxy and not require the sidecar service. On debian based platforms, there are no SCLs and therefore we had no issues with things being or not being available in a SCL, and so we deployed
SPDC as a smart proxy plugin, as opposed to EL* based platforms, where it was a standalone service.
Having this solved the immediate issue, but created a couple of new ones:
- different deployments on different platforms
- higher network traffic when
SPDCruns as a standalone service due to the request proxying between smart proxy and
- code duplication (
SPDCreimplements selected parts of smart proxy)
- weird gems existing due to us wanting to be able to load the same code into Foreman, smart proxy and
- getting SSL certs right is rather hard, installer does that for us but people are struggling when using custom certs
- certain things became hard (client-cert based auth on
SPDC's side, since we don’t get the “real” client’s cert)
Fast forward to today, smart proxy is SCL’d, the original issue is gone and we are left with just the downsides. To address this, I suggest getting rid of
SPDC (the service, we would keep the gem for a bit more) for good.
1) Patch RPM packaging to deploy bundler.d files to both smart proxy and
Currently when installing smart proxy plugins, they drop bundler.d files into locations where
SPDC is able to pick them up. However, those are not files for the gems which are providing them, but for their
*_core counterparts, so for example
smart_proxy_ansible drops a file to have
foreman_ansible_core loaded in
smart_proxy_remote_execution_ssh sets up
foreman_remote_execution_core and so on.
The first step would change this to have the smart proxy plugins set up the bundler.d files to have them loaded into both
SPDC and smart proxy. Until we change
external_core setting for
smart_proxy_dynflow_core, the gems would be loaded by both services, but only used by
This step doesn’t really add value on its own, apart from making the following step safer by not having to coordinate gem, puppet module and installer releases precisely.
2) Toggle the setting
Once we have the gems loaded in both services, we can start phasing out
SPDC. This step would be relatively straightforward, it would boil down just to flipping a single parameter in puppet-foreman_proxy, deprecating the standalone mode and optionally stopping
SPDC service either by installer or from some packaging hook.
This should follow shortly after the first step is done. I’d be glad if we could get this done soon, ideally targeting Foreman 2.5.
3) Code cleanup
As far as
SPDC (the gem) is concerned, we could get rid of most of the code we copied from smart proxy to be able to make it run on its own. We could also drop all the request rewriting and proxying code from
smart_proxy_dynflow. Eventually, we should be able to get rid of
SPDC completely by merging it into
There are many gems which are currently arguable at best and wouldn’t make any sense at all once we get rid of
SPDC. This would be the stage where we move most of the existing code from
*_core plugins to their
smart_proxy_* counterparts. After this is done, whatever remains of the
*_core gems can be dropped completely.
This is probably the biggest step of the three, but since the gems in question can be used as-is in both deployments, it is not urgent. In any case, I’d probably leave one release for the deployment change to settle in and then get to refactoring the plugins. If we manage to get the previous steps into 2.5, then it would suggest the refactored plugins landing in 2.6.
While the split deployment model has some downsides, it is not all bad. The upside of it is that even if the things running on
SPDC get CPU intensive, it won’t affect the smart proxy process.
Another thing worth considering is that in the split deployment, if you push
SPDC hard enough,
SPDC may not be able too keep up with incoming requests. If we run dynflow inside regular smart proxy as per this proposal, it might start dropping other, non-dynflow, requests when under load. On the other hand, smart-proxy uses puma (as opposed to webrick on
SPDC) which should perform better. Sadly I didn’t try this at a large enough scale to have any meaningful data, so I cannot really confirm nor refute this. What keeps me optimistic is not recalling any debian users complaining about it happening.
We have a chance to simplify things quite a bit. This proposal should allow us to address all the catches described above and I believe it is a step in the right direction.