RFC: Foreman webhooks

Hello,

during the next sprint, I would like to extend the initial idea and implementation of Foreman core subscription API and @TimoGoebel 's foreman_webhooks plugin. We think it is the best and most flexible way of preaparing Foreman for the future in regard to integration.

The initial idea and motivation remains the same - we will maintain a simple subscription API in Foreman core and allow plugins to hook on various events similarly to the existing foreman_hooks plugin. Although I planned developing some other plugins (I mentioned Redis pub/sub), for the best user experience we would like to go forward with foreman_webhooks as the defacto implementation. For more details about the process how we got here, read:

Timo and our friends at iRonin layed the basics: we have subscription API in Foreman core and there’s the WIP prototype published at:

It provides a simple interface to create webhooks, define URL and actions on which they should trigger. It integrates with the subscription API to hook on these events and it processes them by creating ActiveJobs to deliver them to external services.

We would like to expand on that:

  • Add new template kind “webhook” defining payloads in a flexible way.
  • Set of new macros which allows loading basic records via a database ID (couple of these already exist).
  • Integrate template payload to the hook definitions.
  • Rewrite UI in React.
  • Deliver API and CLI portions of the same.
  • Prepare some example payload templates for selected external services.
  • Create a smart proxy module webhook “shell” endpoint and payload template for easy testing and upgrade path for foreman_hooks users (executes a shell script).

We will not be aiming for 100% compatibility with current foreman_hooks, the plugin is not going anywhere and can be used alongside with foreman_webhooks going forward. We expect to ship it “until it breaks”, at least for few major releases so there is enough transition time for all users.

I’d like to ask @TimoGoebel if he is willing to contribute his WIP as a starting point for this work. It looks like 90% of the WIP code can be directly used for what we are planning to build.

Edit: I’ve described the minimum viable product, we would like to deliver other features like:

  • Audit support and extensive logging.
  • Request-session tracking via HTTP headers.
  • Add configurable timeout
  • Add re-triggering support and perhaps configurable amount of retries
  • Flag which can enable or disable hooks per hook item
  • Configurable HTTP headers (e.g. for secret tokens)

Take this as a TODO list which I will keep up to date as we dig deeper.

2 Likes

Sure, by all means. What do you need? Do you want to transfer the plugin to the Foreman org? Do you want me open a PR so we can move the code to core? Do you want me to just say, “yes”.

Note, that the plugin has a small todo list. Most of the stuff are low-hanging fruit that can easily be delivered and bring high user value: https://github.com/timogoebel/foreman_webhooks/issues

That’s all for now, thank you. If you can help with reviews of the code changes, we’d appreciate your valuable input at any point. Feel free.

@tbrisker I would like to request creating of new repository theforeman/foreman_webhooks where I would push the original Timo’s WIP prototype as the starting point for the next development.

Oh this is already nice, I am going to add this to my OP into the TODO section for now. These are all good points we should definitely implement.

Can you elaborate on this one please:

Isn’t secret token part of the URL which should be configurable? I expect both payload (body) and URL to be fully configurable on a per hook basis using ERB, so any variable or static text (a token) could be entered.

why not just move the repo to the foreman org? (assuming @TimoGoebel is fine with that)
that way the issues will also be migrated.

If that’s possible then I am all in. Just do not fork it, that does not look good and also time would not be able to fork it back anymore.

@tbrisker: I’ve invited you to the repo and will give you permissions to move it.

That depends. Gitlab has such a feature, see here: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5664

1 Like

Okay so you mean “implement flexible way of setting http headers”?

@tbrisker: If you temporarily grant me permission, I’ll move the repo. Looks like it’s not possible to grant you admin permissions on a personal repo.

granted, let me know once the transfer is complete.

It’s done. :slight_smile:
I’ll leave setting up permissions etc. to you.

1 Like

Thanks! I’ve made both you and @lzap admins so you can manage whatever is needed.

2 Likes

I see some similarities between web_hooks and web_sockets, wondering we can/should implement it in a way that would benefit web_socket later.

I sketched this in a few minutes to explain what I mean:

This makes a lot of sense technically, but I need to ask: what is the business purpose?

I mean, webhooks is the defacto standard, every company or most individual know how to write and deploy a web service, that’s why webhooks are popular. In fact, microservices is a big thing and it’s literally based on web services.

The current hooks plugin is based on spawning a process for each individual event, that’s quite an operation even for UNIX. In the current proposal, webhooks should be handled by ActiveJob which is much lighter resource than a full UNIX process. At least that’s my impression today.

I am not big fan or making a code “ready to be extended later”. I always push for simplest code as possible, less code means less bugs. Then, the time that was saved not doing any crazy design patterns can be invested in refactoring later, if needed. Often times, there is no refactoring at all :slight_smile:

Oh, I didn’t mean that it needs to be an actual “service”. I wanted to write “service/job/whatever” but used service for some reason, ActiveJob is reasonable imo.
Same for the “WEB_SOCKETS_SERVICE” that can just be a route in the current API :slight_smile:

I agree, I just wanted to explain how web_sockets can benefit from this work, or web_hooks can benefit from the web_sockets.

Because web_sockets is already a work in progress, I wonder if it make more sense to complete each other instead of competing each other.

1 Like

Web sockets are really nice, it’s actually a messaging platform ideal for this. I suggest we would write a new plugin that would expose all internal events over websockets if UI needs this. Or it can be implemented in core as well.

I just don’t see any use case yet other than user would be able to subscribe to arbitrary events as notifications (?). That’s indeed interesting.

For the record, we are about to merge a change in core subscription API. Instead of having stripped-down payload (just ID, name in case od delete) we are going to pass whole ActiveRecord object. The reason for that is plugin authors would do Model.find(id) anyway so there is no added value.

Another big reason for this is we would like to refactor foreman_webhooks plugin and enhance it with templates, so users would be able to define their own payloads. We will ship a default template that will have basic details for models like host, hostgroups and other important ones and this will be our API, but more flexible and testable (we will have a test which will render all templates to ensure they work).

To ensure the current design fits real-world scenarios, I am going to do a research of publicly avaialble API documentation for services our users would like to integrate with. Feel free to add your own services or do research for other technologies. This entry is a wiki.

Party API/Payload Authentication
Ansible Tower (AWX) REST/JSON HTTP Basic or OAuth2 with configurable expiration
Netbox REST/JSON Permanent (pregenerated) token in HTTP header
ServiceNow REST/JSON HTTP Basic or OAuth2 with configurable expiration
NetApp ONTAP9 REST HTTP Basic
Veaam REST/XML Custom login

Typically OAuth2 is a two-step process when client id + secret is used to acquire a temporary access token (and refresh token). However many services have configurable time for both access and refresh token expiration. It can be set to longer timeframe (month, year) so the token can be then hardcoded into HTTP header. This is the only way webhooks plugin can work, otherwise webhook must be implemented as a 3rd party service.

It looks like Netbox has quite similar design we target for - their webhooks are highly configurable (URL, actions, headers, body) with jinja2 templating engine:

  • Name - A unique name for the webhook. The name is not included with outbound messages.
  • Object type(s) - The type or types of NetBox object that will trigger the webhook.
  • Enabled - If unchecked, the webhook will be inactive.
  • Events - A webhook may trigger on any combination of create, update, and delete events. At least one event type must be selected.
  • HTTP method - The type of HTTP request to send. Options include GET, POST, PUT, PATCH, and DELETE.
  • URL - The fuly-qualified URL of the request to be sent. This may specify a destination port number if needed.
  • HTTP content type - The value of the request’s Content-Type header. (Defaults to application/json )
  • Additional headers - Any additional headers to include with the request (optional). Add one header per line in the format Name: Value . Jinja2 templating is supported for this field (see below).
  • Body template - The content of the request being sent (optional). Jinja2 templating is supported for this field (see below). If blank, NetBox will populate the request body with a raw dump of the webhook context. (If the HTTP cotent type is set to application/json , this will be formatted as a JSON object.)
  • Secret - A secret string used to prove authenticity of the request (optional). This will append a X-Hook-Signature header to the request, consisting of a HMAC (SHA-512) hex digest of the request body using the secret as the key.
  • SSL verification - Uncheck this option to disable validation of the receiver’s SSL certificate. (Disable with caution!)
  • CA file path - The file path to a particular certificate authority (CA) file to use when validating the receiver’s SSL certificate (optional).

From what I can see, it looks like the most common approach is ability to generate an oauth2 token/secret with configurable timeout which is then used to get session token. Some parties allows temporary tokens and some plain username/password authentication. With templating, we should be able to cover some integration, definitely not all.

When an app supports plain HTTP or allows generation of permanent token or session token with infinite/long timeout a single call can be used. How feasible this is gonna be in practice is hard to tell, I’d have to integrate every single one of them to find out what session API endpoint returns.

Overall I am impressed how much security by obscurity is this, since all calls should be HTTPS with CA check by default, it should not matter what’s transferred over the secure line. Even plain HTTP auth should be okay as long as TLS/SSL is not flawed. You either leak username/password, or oauth2 initial token/secret or predefined permanent token. Results are the same. I do not buy this explanation.

Anyway, there always be a chance to integrate against man-in-the-middle service which can provide the access via oauth2 short-lived tokens. So I suggest to go with our proposal, unless there is something I don’ t see. What you think? @TimoGoebel @Marek_Hulan @ofedoren and others?