The future of Foreman Hooks plugin

Hello,

I am in a train from OSAD 2019 Munich event. It was fun, thanks to ATIX for having me. I gave a talk about discovery improvements and the feedback was positive. Then we had a good chat with several Foreman / Orachino users. One thing during various topics was in common - all users integrate Foreman with other external systems. They either use Foreman Hooks with some degree of success, or write own plugins. And we all agreed that improving Foreman Hooks would be very much appreciated.

Then we discussed how to do integration and at that point I realized we unlikely come to an agreenment that it should be webservice hook call, redis message or shell script what does the integration. I presented the idea of creating a simple plugin API that would expose needed methods and minimalistic contract.

The contract proposal is to maintain a list of triggers. Each having simply a name (e.g. :host), an action (e.g. :created) and either a block returning Ruby hash or a template returning JSON string to generate the payload. All triggers are one-way only, fire and forget, no way to block Foreman request. The API should allow adding those triggers to the registry and some DSL to call those hooks at various places in the codebase.

Foreman core would only publish those simple events with payload, probably via Rails Instrumentation API or something similar internally within the Rails process. Then we would create separate implementations for various use. Timo looks interested in writing a webook plugin to publish those events via HTTPS, I would provide a plugin for simple shell execution as a replacement of Foreman Hooks.

The API is deliberately designed to be explicit, one-way and with user-defined payload for easier integration and debugging. I think leveraging template renderer to generate the payload data gives more flexibility to users - by default we will only export ID and NAME fields, but users could be free to choose anything they want.

Tell us what you think.

4 Likes

Could you clarify what you mean by blocking? Do you mean rejecting the call (i.e., a rollback) or that it runs fully async and the main Foreman thread continues after firing the event?

I really like the Github webhook implementation. Especially the logging and ability to replay failed hooks is powerful.

Could foreman_hooks be rewritten to use this formal API? Hooks is an already established plugin and it would be nice if we could continue supporting it.

Overall I like the idea of a formal Ruby API.

1 Like

Yes, I don’t want this to be possible. This created so much problems in the past and also since the proposal is deliberately not based on ActiveRecord, it would not work in all contexts.

This is exactly what Timo has on his mind yesterday.

While this is technically possible, I can hardly imagine how to proceed. Foreman hooks d not have any tests, there are no guarantees, users can hook to any ActiveRecord callback and our goal is actually to provide minimum set of explicitly defined triggers for basic objects like host, hostgroup, subnet etc. Payloads will be different and I would really love to see the default payloads to be very simple, something like a JSON with just ID and NAME by default. We can explain in ERB comments what the trigger is all about and the context and we can slowly start adding more and more flags when needed.

I’m slightly leaning to doing a major version that replaces the current implementation. That might break some users setups and we can place upgrade warnings, but given our bad history with removing plugins and the very good name of the current plugin, I’d seriously consider it.

1 Like

I don’t know what name Timo gives it. I started using a different term “triggers” instead “hooks” because I wanted to stress out that these will not be compatible. But if the feature is called hooks in Foreman core, I am not against keeping foreman_hooks however we will likely see plugins named like foreman_webhooks and I am not sure if new naming convention would be more clear:

  • foreman-triggers-shell
  • foreman-triggers-web

Now that I wrote it it looks a bit werird, webhook is defacto standard name :slight_smile: Let’s see.

1 Like

That’s why I wanted to go for foreman_webhooks. We can definitely call the core parts triggers, so basically we’d implement “the triggers API” in core. There will be several plugins that can use this API.

I’d really like to have separate plugins for the actual implementations.

  • foreman_webhooks -> Github like webhooks
  • foreman_hooks NG -> can run shell scripts

Btw: WIP code is here:

It currently just defines a new model, the WebhookTarget. I’d like to implement a MVP first, we can than add re-triggering of events at a later point.

1 Like

A quick update:

I have a working PoC. It’s far from perfect, but I’d like to get some feedback before I continue.

Core PR:

Reference Implementation (Webhooks):

@aruzicka would your recent design work in this scenario? Can we reuse some parts?

Then it’s probably better to stick with the same name in core as well? Hook is fine.

Could the plugin be developed in a way, it’s easy to be merged to core when it’s done? This feels as something, that should live in core.

The design of the new generation hooks consist of two parts: core and plugins. We need to implement the integration part as plugins anyway because we could not agree what was the best thing - webhook HTTPS requests or Redis message or calling local shell script? Everybody likes something else, thus the “open core” model.

It’s shaping up. I’ve raised a concern about ActiveRecord model payload - the format should be probably JSON, but what to put inside. I am not so keen serializing whole instances - changes cause pain:

My prototype which subscribes to WIP core event API and sends them over Redis pub/sub to an external script which launch event scripts works well. However the API and the payload is (by design) very different from what foreman_hooks provide, therefore I suggest not to try to rewrite foreman_hooks and confuse users as I think foreman_hooks will be working for couple of more releases to give our users time to migrate to new hook platform.

If there are no objections, I hereby request creation of new theforeman.org github repository called foreman_redishooks which will provide the new capabilities and could be installed alongside legacy hooks plugin. @tbrisker

While we’re at this, can we move https://github.com/timogoebel/foreman_webhooks to the foreman org? As I probably won’t have a lot of time to continue working on it, the plugin will have more visibility in core. The code might be a little immature for core, but I’m generally open to merging this into core if that’s preferred.

Something I would love to see folks opinion on. What would be better…

  1. a foreman_webhooks where the payload can be configured some how on the out
  2. a larger set of plugins which each target a specific application

The example I am thinking of is a host created hook. I could see folks wanting to get tihs into ServiceNow, BMC, AWX, or any number of other inventory systems. In eac hcase, i would expect the API to be different. Would folks prefer that to be a

Sorry, website issues. Would folks perfer that Each endpoint be a unique plugin, or have some way of fortmatting the body of hte hook on the way out?

I would prefer to have one webhook plugin instead of multiple plugins. (BTW, what about moving foreman_webhook to the theforeman github org). I guess, foreman_webhook is a great start and should replace foreman_hook.

2 Likes

That was the idea of the webhooks plugin. We currently develop a specific plugin for a specific use case. This is great, but relatively expensive as we have to maintain the plugins so they continue to work with newer foreman release.
The idea of the webhooks plugins was to allow a user to integrate Foreman with other tools by simply creating a small webservice. If we can keep the API stable, it should not be a great hassle for users to maintain their small webservice. And the webservice can be developed in any language the user feels most comfortable with or that has the best api bindings. I could see a similar repo to community-templates where we share integrations with popular services.
What I don’t recommend and see serious problems is if users can change the payload that is sent when the hook is fired. This is completely unnecessary as users can just ignore the data they don’t need. We should give them as much data as possible. I believe this is the industry standard as well.

I’d love to do that but don’t have the permissions to do it by myself. I also won’t have the time to work on the plugin in the near future. Any help is appreciated.

I agree with Timo, minimum possible payload and it’s up to the user to create his/her own webservice implementation which will fetch more data from Foreman and perform the integration. Of course, it’s better to share, so users are free to open up their implementations of webservices and put it to github, but as separate (micro) services.

The root cause of issues with the original foreman_hooks was that users were doing crazy things within the foreman (passenger) process and supporting them was a huge pain. We want the exact opposite of this, let them know that something happened and that’s end of the story for Foreman.

That’s my takeaway from our discussions in Brno last week too, I will probably push my Redis prototype to my own github place but foreman_webhooks plugin done by Time should be the only supported one by the project of course encouraging users to write their own plugin if they must to.

Timo, would you mind expanding the plugin README file with more content? The problem, the solution, how it works and more importantly an example of a service that does something useful. We don’t want users to start their webservices from scratch. Then, let’s promote the plugin to the official GH organization.

With a generic webhook plugin that requires the user to build some other external web service, I worry about 2 things:

  • The barrier to entry is high, so as a user, I have to now create a web service to be able to integrate, whereas before I dumped an executable.

  • Other systems like Jenkins/ect… have to build something in order to integrate. That may not be a terrible thing, but it’s going to take time (if they choose to do it at all)

I’m not suggesting we stick with foreman_hooks, but I think we should make webhooks easier to consume. Maybe it should be targeted to only work with automation platforms like AWX/Rundeck/ect… ? That way someone can continue to write crazy automation so they can continue to do whatever they want in arguably the right place.

Github’s implementation of webhooks works well because it’s targeted at developers who have the expertise and inclination to build integrations, I don’t feel like the same is true for the majority of our user base, most people want to “Plug and play” IMO.