Webhooks orchestration steps/ordering

Support
1

I’m looking at replacing our foreman_hooks scripts with webhooks. In foreman_hooks we can number the hook scripts to control not only the order in which they run but also where in the orchestration process they run. For example, a hook script prefixed with 05_ will run before Foreman creates dns/dhcp records. Explained in GitHub - theforeman/foreman_hooks: Run custom hook scripts on Foreman events

We use this on the update event - if subnet or ip is changed for a host we want to save any comments and aliases (cnames) for the dns host object (in infoblox) before Foreman deletes it and creates a new host object. We then call a hook script again after Foreman has created the new dns object to add comment/aliases to the new object.

In webhooks I see no obvious way of ordering things like this - is it possible?

2

Hello Adam!

Webhooks were designed from the day one to be fire and forget, there is no concept of performing a “before” hook that is guaranteed to delay order of execution of any in Foreman.

I think that CNAME support should be added as a feature into Foreman, this would be definitely useful to other users as well. However I understand you would need to have this functionality now, not in one year from now.

In this case, I suggest you that you disable DNS orchestration (unset DNS proxy) and write a host webhook that will perform the DNS update completely. There is one more thing - previously if you knew what you are doing, both old and new record was available so you could compare them and see the difference for edits. This is no longer possible in webhooks, I think if this is what you need then we could also send latest audit record with that update where you could dig this information from. Because after a host rename operation, you will probably need to know the old hostname to find the record.

Another alternative would be writing a new DNS implementation based on “shell” scripts. A DNS module is trivial to implement:

I’ve done this for BMC already, the idea was to allow running arbitrary shell commands on various BMC calls (power on, off, reboot). These are configurable and although this is mostly used for testing, I think for DNS updates this should be good enough for you - all you need to do is to run Infoblox cmd utility, if there’s any:

3

Also if you need to create the order of webhooks, you can use shellhooks smart proxy plugin. That is closed to the original hooks, one Foreman webhooks triggers one shell script on the proxy. That script would then contain several sub-steps that would always be executed in the right order. But it won’t delay the rest of the operation in Foreman as @lzap pointed out, so I’m not sure whether that helps in your case.

4

Thank you very much for the replies!

I never even considered disabling the dns proxy, but since we already do a lot of dns stuff in our hook scripts (adding cnames, adding comments, adding ipv6 address) it might actually not be that big a deal to also manage creation/deletion of host objects (infoblox has a special ‘host object’ which is sort of a wrapper containing host record (ipv4 and ipv6), dhcp reservation, cnames and other stuff).

I will look into this further. I guess such a dns webhook (for dns record creation) would subscribe to host_created, only question is if the webhook can get the host record in place in time for the PXE boot of the host?

Also, I would have to manage ip address changes in my webhook as well - if I have a webhook subscribed to host_updated, can I send it what was changed? I.e. which parameters that are changed and the old and new value?

Thanks also for the tip about the shellhooks plugin, I just found that actually and even if it wont help me in this particular case I think it will absolutely make the transition from foreman_hooks to webhooks easier.

1 Like
5

There is no guarantee, webhook delivery happens via our background service and it can be delayed a little or by a lot too. I would really consider writing a new DNS provider based on shell (or even better ansible). But try the webhooks way and get back to us with the feedback which will be really useful.

I think this is a gap in the current implementation we need to solve. Webhooks are after_commit hooks for various design resons, problem is, there is no access to “old” or “changed?” Rails attribute in this context. @Marek_Hulan

Possible solution is either exposing some fields, explicitly selected, via a slight hack.

Alternatively, we could create hooks for selected audit records, they contain nicely formatted changes:

> Audit.last
=> #<Audited::Audit:0x0000000014b12010
 id: 424,
 auditable_id: 1,
 auditable_type: "Domain",
 user_id: 4,
 user_type: nil,
 username: "Admin User",
 action: "update",
 audited_changes: {"fullname"=>["test2", "test"]},
 version: 8,
 comment: nil,
 associated_id: nil,
 associated_type: nil,
 request_uuid: "9d3af0fd-ab35-4c8f-955e-02d202d3d492",
 created_at: Thu, 24 Jun 2021 07:48:44 UTC +00:00,
 remote_address: "::ffff:192.168.1.5",
 auditable_name: "dummy.lan",
 associated_name: nil>

Or a hybrid solution, since audit record is created in after_create/update and webhooks trigger via after_commit, the associated audit record could be actually added to most webhooks as an extra information. This would be especially useful for update events. What you think?

That was exactly the idea.

6

It looks like it could be as easy as allowing audits for all ApplicationRecords. I am assuming that we filter out sensitive data like passwords from there. Still some unwanted fields could be exposed via this, but since safe mode’s purpose is to prevent calling some Ruby methods like “Kernel.exit” or “Kernel.process” that should be good enough:

> Host.first.audits.last
=> #<Audited::Audit:0x0000000014a21d18
 id: 336,
 auditable_id: 16,
 auditable_type: "Host::Base",
 user_id: 4,
 user_type: nil,
 username: "Admin User",
 action: "update",
 audited_changes: {"build"=>[false, true], "initiated_at"=>[Fri, 14 May 2021 12:20:45 CEST +02:00, Fri, 14 May 2021 12:29:56 CEST +02:00]},
 version: 6,
 comment: nil,
 associated_id: nil,
 associated_type: nil,
 request_uuid: "cbc8b17b-763b-4ca2-9741-b639dd8ed5f4",
 created_at: Fri, 14 May 2021 10:29:56 UTC +00:00,
 remote_address: "::ffff:192.168.1.5",
 auditable_name: "allan-ocampo.dummy.lan",
 associated_name: nil>
7

I think we could allow audits or revision to the safe mode. That way one could access the previous info like this in the webhook template directly

current hostname: <%= @object.hostname %>
previous hostname: <%= @object.revision(:previous).hostname %>

The beauty of this is, the previous version is still instance of the Host object so the same Jail applies.

8

that would be awesome, would make it really easy to track changes

9

So I have a patch ready, bad news is it more complex I was hoping to. Marek’s idea works great, but there were some issues with Safemode inheritance which will need a plugin-breaking change. But I think it is worth the effort:

https://projects.theforeman.org/issues/32881

1 Like