There’s Foreman Hooks plugin which allows users to hook arbitrary scripts on various ActiveRecord and Foreman orchestration actions. The problem with this approach is that this is deeply integrated with the two technologies, users need to understand them and check scripts after major Rails upgrades because behavior of ActiveRecord changes over time.
More importantly, the scripts are being called during ActiveRecord handling when records or their associacions are not yet saved, this can lead to fields not being set. Particularly with Foreman Orchestration which is by design during validation. Users cannot find out if model was created, updated or deleted when they hook to some common callbacks as well.
Current hooks are actually part of transactions which can lead to a rollback, this is a feature which creates more problems when upgrading or dealing with incompatibility. Last but not least, the plugin is currently unmaintained. Anyone is more than welcome to take it and improve.
The proposal:
Provide an alternative and simple callback mechanism based on Rails controller actions rather than model by creating an application generic around filter. For each controller/action the filter publishes an event. No data from our database is sent along the event, subscribers are expected to use our standard API to find relevant records. Key features:
- Before callbacks send all request parameters along with an event.
- After callbacks send database record primary ID along with an event (but not more).
- Callback API available through the application for ad-hoc callback points when necessary.
- Subscribers are running outside of Rails application on purpose.
- Core feature with tests, not a plugin.
Now, publishing those events. I was thinking ActionCable, it’s in Rails 5, easy to use, documented and there are 3rd party libraries for subscribing and consuming the content like https://github.com/tobiasfeistmantl/python-actioncable-zwei. The problem is that ActionCable is not supported on Passenger running on Apache.
Alternatively, this could be as simple as publishing into Redis via low-level Ruby client and picking that up using script. We were discussing possibility of adding Redis into the core stack, that could be nice use of it (no ActionCable or websockets involved).
Another way is reusing our statsd telemetry stack and publishing events via UDP datagrams. Looks like statds supports an extension for that, unfortunately Prometheus does not.
Finally, another option is to call HTTP endpoint, that’s more straightforward but this will scale worse. Generally we should avoid making HTTP calls when processing web requests.
Wrap up
This does not look like lot of work, if we agree on the best publishing mechanism I’d like to proceed and create a prototype to see how it works in real use. Then we can decide what to do next. It’s important to know that this does not aim to replace Foreman Hooks, but to complement it. The whole designed learned from my experience with debugging and supporting various problems with foreman_hooks, it aims to be as simple as possible.
I am open to ideas and comments.