I am in a train from OSAD 2019 Munich event. It was fun, thanks to ATIX for having me. I gave a talk about discovery improvements and the feedback was positive. Then we had a good chat with several Foreman / Orachino users. One thing during various topics was in common - all users integrate Foreman with other external systems. They either use Foreman Hooks with some degree of success, or write own plugins. And we all agreed that improving Foreman Hooks would be very much appreciated.
Then we discussed how to do integration and at that point I realized we unlikely come to an agreenment that it should be webservice hook call, redis message or shell script what does the integration. I presented the idea of creating a simple plugin API that would expose needed methods and minimalistic contract.
The contract proposal is to maintain a list of triggers. Each having simply a name (e.g. :host), an action (e.g. :created) and either a block returning Ruby hash or a template returning JSON string to generate the payload. All triggers are one-way only, fire and forget, no way to block Foreman request. The API should allow adding those triggers to the registry and some DSL to call those hooks at various places in the codebase.
Foreman core would only publish those simple events with payload, probably via Rails Instrumentation API or something similar internally within the Rails process. Then we would create separate implementations for various use. Timo looks interested in writing a webook plugin to publish those events via HTTPS, I would provide a plugin for simple shell execution as a replacement of Foreman Hooks.
The API is deliberately designed to be explicit, one-way and with user-defined payload for easier integration and debugging. I think leveraging template renderer to generate the payload data gives more flexibility to users - by default we will only export ID and NAME fields, but users could be free to choose anything they want.
Could you clarify what you mean by blocking? Do you mean rejecting the call (i.e., a rollback) or that it runs fully async and the main Foreman thread continues after firing the event?
I really like the Github webhook implementation. Especially the logging and ability to replay failed hooks is powerful.
Could foreman_hooks be rewritten to use this formal API? Hooks is an already established plugin and it would be nice if we could continue supporting it.
Yes, I donāt want this to be possible. This created so much problems in the past and also since the proposal is deliberately not based on ActiveRecord, it would not work in all contexts.
This is exactly what Timo has on his mind yesterday.
While this is technically possible, I can hardly imagine how to proceed. Foreman hooks d not have any tests, there are no guarantees, users can hook to any ActiveRecord callback and our goal is actually to provide minimum set of explicitly defined triggers for basic objects like host, hostgroup, subnet etc. Payloads will be different and I would really love to see the default payloads to be very simple, something like a JSON with just ID and NAME by default. We can explain in ERB comments what the trigger is all about and the context and we can slowly start adding more and more flags when needed.
Iām slightly leaning to doing a major version that replaces the current implementation. That might break some users setups and we can place upgrade warnings, but given our bad history with removing plugins and the very good name of the current plugin, Iād seriously consider it.
I donāt know what name Timo gives it. I started using a different term ātriggersā instead āhooksā because I wanted to stress out that these will not be compatible. But if the feature is called hooks in Foreman core, I am not against keeping foreman_hooks however we will likely see plugins named like foreman_webhooks and I am not sure if new naming convention would be more clear:
foreman-triggers-shell
foreman-triggers-web
Now that I wrote it it looks a bit werird, webhook is defacto standard name Letās see.
Thatās why I wanted to go for foreman_webhooks. We can definitely call the core parts triggers, so basically weād implement āthe triggers APIā in core. There will be several plugins that can use this API.
Iād really like to have separate plugins for the actual implementations.
foreman_webhooks ā Github like webhooks
foreman_hooks NG ā can run shell scripts
Btw: WIP code is here:
It currently just defines a new model, the WebhookTarget. Iād like to implement a MVP first, we can than add re-triggering of events at a later point.
The design of the new generation hooks consist of two parts: core and plugins. We need to implement the integration part as plugins anyway because we could not agree what was the best thing - webhook HTTPS requests or Redis message or calling local shell script? Everybody likes something else, thus the āopen coreā model.
Itās shaping up. Iāve raised a concern about ActiveRecord model payload - the format should be probably JSON, but what to put inside. I am not so keen serializing whole instances - changes cause pain:
My prototype which subscribes to WIP core event API and sends them over Redis pub/sub to an external script which launch event scripts works well. However the API and the payload is (by design) very different from what foreman_hooks provide, therefore I suggest not to try to rewrite foreman_hooks and confuse users as I think foreman_hooks will be working for couple of more releases to give our users time to migrate to new hook platform.
If there are no objections, I hereby request creation of new theforeman.org github repository called foreman_redishooks which will provide the new capabilities and could be installed alongside legacy hooks plugin. @tbrisker
While weāre at this, can we move https://github.com/timogoebel/foreman_webhooks to the foreman org? As I probably wonāt have a lot of time to continue working on it, the plugin will have more visibility in core. The code might be a little immature for core, but Iām generally open to merging this into core if thatās preferred.
Something I would love to see folks opinion on. What would be betterā¦
a foreman_webhooks where the payload can be configured some how on the out
a larger set of plugins which each target a specific application
The example I am thinking of is a host created hook. I could see folks wanting to get tihs into ServiceNow, BMC, AWX, or any number of other inventory systems. In eac hcase, i would expect the API to be different. Would folks prefer that to be a
I would prefer to have one webhook plugin instead of multiple plugins. (BTW, what about moving foreman_webhook to the theforeman github org). I guess, foreman_webhook is a great start and should replace foreman_hook.
That was the idea of the webhooks plugin. We currently develop a specific plugin for a specific use case. This is great, but relatively expensive as we have to maintain the plugins so they continue to work with newer foreman release.
The idea of the webhooks plugins was to allow a user to integrate Foreman with other tools by simply creating a small webservice. If we can keep the API stable, it should not be a great hassle for users to maintain their small webservice. And the webservice can be developed in any language the user feels most comfortable with or that has the best api bindings. I could see a similar repo to community-templates where we share integrations with popular services.
What I donāt recommend and see serious problems is if users can change the payload that is sent when the hook is fired. This is completely unnecessary as users can just ignore the data they donāt need. We should give them as much data as possible. I believe this is the industry standard as well.
Iād love to do that but donāt have the permissions to do it by myself. I also wonāt have the time to work on the plugin in the near future. Any help is appreciated.
I agree with Timo, minimum possible payload and itās up to the user to create his/her own webservice implementation which will fetch more data from Foreman and perform the integration. Of course, itās better to share, so users are free to open up their implementations of webservices and put it to github, but as separate (micro) services.
The root cause of issues with the original foreman_hooks was that users were doing crazy things within the foreman (passenger) process and supporting them was a huge pain. We want the exact opposite of this, let them know that something happened and thatās end of the story for Foreman.
Thatās my takeaway from our discussions in Brno last week too, I will probably push my Redis prototype to my own github place but foreman_webhooks plugin done by Time should be the only supported one by the project of course encouraging users to write their own plugin if they must to.
Timo, would you mind expanding the plugin README file with more content? The problem, the solution, how it works and more importantly an example of a service that does something useful. We donāt want users to start their webservices from scratch. Then, letās promote the plugin to the official GH organization.
With a generic webhook plugin that requires the user to build some other external web service, I worry about 2 things:
The barrier to entry is high, so as a user, I have to now create a web service to be able to integrate, whereas before I dumped an executable.
Other systems like Jenkins/ectā¦ have to build something in order to integrate. That may not be a terrible thing, but itās going to take time (if they choose to do it at all)
Iām not suggesting we stick with foreman_hooks, but I think we should make webhooks easier to consume. Maybe it should be targeted to only work with automation platforms like AWX/Rundeck/ectā¦ ? That way someone can continue to write crazy automation so they can continue to do whatever they want in arguably the right place.
Githubās implementation of webhooks works well because itās targeted at developers who have the expertise and inclination to build integrations, I donāt feel like the same is true for the majority of our user base, most people want to āPlug and playā IMO.
Letās see.