Hello, folks!
Is there a way for me to create hooks for discovered_hosts events?
I don't see anything under ~foreman/app/models but maybe missing
something…
Thanks!
Since it's a plugin, it's probably installed somewhere else.
You can probably add a hook under "host/discovered/" but the JSON you
get passed in may not be complete - Phirince is fixing that under
https://github.com/theforeman/foreman_hooks/issues/24.
···
On 16/09/15 00:06, 'Konstantin Orekhov' via Foreman users wrote: > Hello, folks! > > Is there a way for me to create hooks for discovered_hosts events? > I don't see anything under ~foreman/app/models but maybe missing > something...–
Dominic Cleal
dominic@cleal.org
We'd also like to see more info from discovery events. In particular we'd
like to extract serials and model numbers to add to a CMDB. Idea being the
discovery would occur on a build network immediately upon unpacking a
server, the details would automatically be added to the CMDB and linked
later to the built OS upon deployment.
Thanks for a suggestion, Dominic! Unfortunately that didn't work.
I dropped your simple log.sh into host/discovered/[create|destroy] along
with hook_functions. That did not make any difference for a new hosts
sending their discovery payloads, but it broke the deletion process:
<snip>
> Started DELETE "/discovered_hosts/mac<mac>" for <IP> at 2015-09-17
18:39:35 -0700
2015-09-17 18:39:35 [app] [I] Processing by
DiscoveredHostsController#destroy as HTML
2015-09-17 18:39:35 [app] [I] Parameters:
{"authenticity_token"=>"<token>", "id"=>"mac<mac>"}
2015-09-17 18:39:35 [sql] [W] Host::Discovered doesn't support
orchestration, can't run orchestration hooks: use Rails events instead
2015-09-17 18:39:35 [app] [W] Action failed
> NameError: undefined local variable or method queue' for #<Host::Discovered:0x0000000defded8> > /opt/rh/ruby193/root/usr/share/gems/gems/activemodel-3.2.8/lib/active_model/attribute_methods.rb:407:inmethod_missing'
>
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods.rb:149:in
method_missing' > /opt/rh/ruby193/root/usr/share/gems/gems/foreman_hooks-0.3.8/lib/foreman_hooks/orchestration_hook.rb:36:inblock in queue_hooks'
>
/opt/rh/ruby193/root/usr/share/gems/gems/foreman_hooks-0.3.8/lib/foreman_hooks/orchestration_hook.rb:32:in
`each'
<snip>
I can provide a complete message if needed. I'm on 1.9.1 with 4.0.0
discovery plug-in.
Which Rails event(s) would be applicable here?
Thanks!
Try using after_create or after_commit for creation, and after_destroy
for deletion.
The create/destroy events are part of host orchestration, which is only
on a couple of objects - fully managed hosts (not discovered) and
network interfaces. The Rails events apply to everything, so you can
rely on those being there.
https://github.com/theforeman/foreman_hooks#rails-events links to a full
list.
···
On 18/09/15 02:49, 'Konstantin Orekhov' via Foreman users wrote: > Thanks for a suggestion, Dominic! Unfortunately that didn't work. > > I dropped your simple log.sh into host/discovered/[create|destroy] along > with hook_functions. That did not make any difference for a new hosts > sending their discovery payloads, but it broke the deletion process: > > > > Started DELETE "/discovered_hosts/mac" for at 2015-09-17 > 18:39:35 -0700 > 2015-09-17 18:39:35 [app] [I] Processing by > DiscoveredHostsController#destroy as HTML > 2015-09-17 18:39:35 [app] [I] Parameters: > {"authenticity_token"=>"", "id"=>"mac"} > *2015-09-17 18:39:35 [sql] [W] Host::Discovered doesn't support > orchestration, can't run orchestration hooks: use Rails events instead* > > > I can provide a complete message if needed. I'm on 1.9.1 with 4.0.0 > discovery plug-in. > > Which Rails event(s) would be applicable here?–
Dominic Cleal
dominic@cleal.org
Thanks again, Dominic! I'll definitely play around with that.
In general though, are more "standard" orchestration hooks/rules planned to
be implemented in Discovery plug-in? For example, there are
auto-provisioning rules. Wouldn't it be nice to have an ability to run say
burn-in tests on all new systems checking in before making them available
for provisioning? The way I look at that it could be a rule as well.
The other thing that I saw is being planned is very excited about is the FW
upgrades. However, I'm very curious how exactly is going to be implemented.
Is it going to be some kind of "command channel"/exec smart-proxy API on a
client side that allows to execute scripts, commands, etc? If so, the same
functionality could be used for things like RAID reconfiguration,
above-mentioned burn-in HW tests, etc. Also, I'm wondering if there's going
to be some kind of a "model" that defines what the FW version has to be -
FW upgrade has to know when to upgrade things and when not to, right?
What's your thinking around that?
Thanks!
Hmm, even though I do get those events and hooks triggered, neither
after_create nor after_commit give me enough information about discovered
host:
{"architecture_id":null,"build":false,"certname":null,"comment":null,"compute_profile_id":null,"compute_resource_id":null,"created_at":"2015-09-22T21:14:47Z","discovery_rule_id":null,"disk":null,"enabled":true,"environment_id":null,"grub_pass":"","hostgroup_id":null,"id":613,"image_file":null,"image_id":null,"installed_at":null,"last_compile":null,"last_report":"2015-09-22T21:40:54Z","location_id":null,"managed":false,"medium_id":null,"model_id":2,"name":"mac<mac>","operatingsystem_id":null,"organization_id":null,"otp":null,"owner_id":null,"owner_type":null,"provision_method":null,"ptable_id":null,"puppet_ca_proxy_id":null,"puppet_proxy_id":null,"puppet_status":0,"realm_id":null,"root_pass":"<pass>","salt_environment_id":null,"salt_proxy_id":null,"updated_at":"2015-09-22T21:40:54Z","use_image":null,"uuid":null}
Am I correct placing those hooks under host/discovered? If yes, what am I
missing here?
Thanks!
This is the JSON issue I mentioned earlier on - it's only passing in a
basic JSON representation of the object instead of having all of the
useful data that's shown in the API.
It's only just been fixed, and I think the three necessary patches are
in foreman_hooks 0.3.9, Foreman 1.9.1 and Discovery 4.1.0.
That may also pass the serials and models numbers you require, as long
as they show in the regular GET API response from the Discovery plugin.
If not, file an issue against Discovery to ask for them to be added and
then they'll start showing up in the hooks JSON too.
Regarding your other questions about what Discovery will do in the
future, I can't comment as I'm not developing it.
···
On 22/09/15 22:48, 'Konstantin Orekhov' via Foreman users wrote: > Hmm, even though I do get those events and hooks triggered, neither > after_create nor after_commit give me enough information about > discovered host: > > {"architecture_id":null,"build":false,"certname":null,"comment":null,"compute_profile_id":null,"compute_resource_id":null,"created_at":"2015-09-22T21:14:47Z","discovery_rule_id":null,"disk":null,"enabled":true,"environment_id":null,"grub_pass":"","hostgroup_id":null,"id":613,"image_file":null,"image_id":null,"installed_at":null,"last_compile":null,"last_report":"2015-09-22T21:40:54Z","location_id":null,"managed":false,"medium_id":null,"model_id":2,"name":"mac","operatingsystem_id":null,"organization_id":null,"otp":null,"owner_id":null,"owner_type":null,"provision_method":null,"ptable_id":null,"puppet_ca_proxy_id":null,"puppet_proxy_id":null,"puppet_status":0,"realm_id":null,"root_pass":"","salt_environment_id":null,"salt_proxy_id":null,"updated_at":"2015-09-22T21:40:54Z","use_image":null,"uuid":null} > > Am I correct placing those hooks under host/discovered? If yes, what am > I missing here?–
Dominic Cleal
dominic@cleal.org
Do I direct such questions to Lukas?
As for the new versions you mentioned above - I only see hooks 0.3.9 in
"nightly" repo -
http://yum.theforeman.org/plugins/nightly/el6/x86_64/tfm-rubygem-foreman_hooks-0.3.9-1.el6.noarch.rpm
and could not even find discovery-4.1.0…
Yeah, they've only just been released, keep an eye out and it'll
appear soon. I may take hooks 0.3.9 into the 1.9 repo if I get the
chance, and Discovery 4.1.x may be released to 1.9 but I think Lukas is
waiting for it to be tested some more in nightly (once the package is done).
···
On 23/09/15 22:56, 'Konstantin Orekhov' via Foreman users wrote: > As for the new versions you mentioned above - I only see hooks 0.3.9 in > "nightly" repo - > http://yum.theforeman.org/plugins/nightly/el6/x86_64/tfm-rubygem-foreman_hooks-0.3.9-1.el6.noarch.rpm > and could not even find discovery-4.1.0...–
Dominic Cleal
dominic@cleal.org
> Yeah, they've only just been released, keep an eye out and it'll
> appear soon. I may take hooks 0.3.9 into the 1.9 repo if I get the
> chance, and Discovery 4.1.x may be released to 1.9 but I think Lukas is
> waiting for it to be tested some more in nightly (once the package is done).
Correct, it a pack of new features that should be tested first. Join us
in testing, we pin all the bugs to:
http://projects.theforeman.org/issues/10294
On the hooks, I have no idea why it does not work. I see there is one
pending PR in this regard…
···
-- Later, Lukas #lzap ZapletalI'd be glad to do some testing for you, however, as I mentioned, I only see
hooks 0.3.9 under nightly and not discovery 4.1.0…
> I'd be glad to do some testing for you, however, as I mentioned, I only see
> hooks 0.3.9 under nightly and not discovery 4.1.0…
FYI Discovery 4.1.0 hit nightly last week, it was little delayed.
···
-- Later, Lukas #lzap ZapletalSome updates on checking how hooks (for discovered hosts) are behaving in
1.10.-rc2 (tfm-rubygem-foreman_hooks-0.3.9-1.el7.noarch and
tfm-rubygem-foreman_discovery-4.1.2-1.fm1_10.el7.noarch)- after_commit and
after_destroy are indeed the events I was looking for and now confirmed
have all systems facts I needed (basically whatever one sees in API/UI
facts for the host in question). I looked at after_create event and its
data passed to a hook does not have system facts either, so really
after_commit is the only one I could truly use here. So thanks for fixing
this!
There is still a little problem though. "after_commit" is rather a generic
event and happens not only on system creation but sometimes even several
times for the same discovered host for whatever reason. Also after
"after_destroy" event is followed by "after_commit" as well (with much
smaller amount of system facts though). This makes it practically
indistinguishable event, so I'm wondering if there are plans on introducing
proper "create", "update" and "destroy" events in discovered hosts just
like in managed hosts?
Here are sequence of events happening (I'm watching only after_commit and
after_destroy events as per above findings).
(a) Host discovery event (first one is a real first time, after that I'm
not sure - some updates coming in from smart-proxy on a discovered
machine?):
[root@centos7 tmp]# grep received foreman-hooks.log
Sat Nov 7 00:43:21 UTC 2015: received after_commit for mac38eaa792fef8
Sat Nov 7 00:45:34 UTC 2015: received after_commit for mac38eaa792fef8
Sat Nov 7 00:46:22 UTC 2015: received after_commit for mac38eaa792fef8
Sat Nov 7 00:48:03 UTC 2015: received after_commit for mac38eaa792fef8
[root@centos7 tmp]#
So the above obviously fires 4 sets of hooks against the same machine. Some
hooks may run longer that 2 minutes so they would overlap and step on each
affecting the results (for example, if it was a set HW stress-test of new
machines, or setting up ILOM users//updating password for default IPMI
users).
(b) Example of events while deleting a discovered host:
[root@centos7 tmp]# grep received foreman-hooks.log
Fri Nov 6 22:53:30 UTC 2015: received after_destroy for mac38eaa792fef8
Fri Nov 6 22:53:30 UTC 2015: received after_commit for mac38eaa792fef8
[root@centos7 tmp]#
Yet another hook is triggered here, but because the hook data does not have
all the system facts, I was able to build some intelligence in one of my
hooks to fail gracefully in such situations. But I'm afraid this is a
band-aid approach and not a proper event handling, wouldn't you agree?
Also, the biggest problem is (a) as the hook will keep firing on each
after_commit regardless if it is a real host got discovered or an update
came in.
Thoughts/suggestions?
> Some updates on checking how hooks (for discovered hosts) are behaving
> in 1.10.-rc2 (tfm-rubygem-foreman_hooks-0.3.9-1.el7.noarch and
> tfm-rubygem-foreman_discovery-4.1.2-1.fm1_10.el7.noarch)- after_commit
> and after_destroy are indeed the events I was looking for and now
> confirmed have all systems facts I needed (basically whatever one sees
> in API/UI facts for the host in question). I looked at after_create
> event and its data passed to a hook does not have system facts either,
> so really after_commit is the only one I could truly use here. So thanks
> for fixing this!
>
> There is still a little problem though. "after_commit" is rather a
> generic event and happens not only on system creation but sometimes even
> several times for the same discovered host for whatever reason. Also
> after "after_destroy" event is followed by "after_commit" as well (with
> much smaller amount of system facts though). This makes it practically
> indistinguishable event, so I'm wondering if there are plans on
> introducing proper "create", "update" and "destroy" events in discovered
> hosts just like in managed hosts?
These events come from orchestration, which I doubt would be required in
Discovery, so I don't think it's likely to change.
My only other thought is to try a validation event such as
after_validation and see if the facts are available then. I think
they're missing on after_create because the host is initially created
without facts, then they're imported after:
The create event (if it existed) wouldn't run on the save(:validate =>
false) line, but would be roughly equivalent to running in
after_validation or similar.
> Yet another hook is triggered here, but because the hook data does not
> have all the system facts, I was able to build some intelligence in one
> of my hooks to fail gracefully in such situations. But I'm afraid this
> is a band-aid approach and not a proper event handling, wouldn't you
> agree? Also, the biggest problem is (a) as the hook will keep firing on
> each after_commit regardless if it is a real host got discovered or an
> update came in.
The technique that's commonly used is to record a flag on after_create
or something, then to check for its presence and delete it in
after_commit. If the flag's there you know the commit is probably from
the creation, and if not, then it's some other event. It would be useful
to add some more context into foreman_hooks for sure.
···
On 07/11/15 01:00, 'Konstantin Orekhov' via Foreman users wrote:–
Dominic Cleal
dominic@cleal.org
> There is still a little problem though. "after_commit" is rather a generic
> event and happens not only on system creation but sometimes even several
> times for the same discovered host for whatever reason. Also after
> "after_destroy" event is followed by "after_commit" as well (with much
> smaller amount of system facts though). This makes it practically
> indistinguishable event, so I'm wondering if there are plans on introducing
> proper "create", "update" and "destroy" events in discovered hosts just
> like in managed hosts?
This is a known design issue we would like to solve with introduction of
Host Aspects. Once this feature is merged into core, we can turn
discovered host into an aspect. Then you should be easily able to pull
information about hosts from them - e.g. if this is discovered host or
not.
Currently we do this by checking the "type" flag on both actual and old
records. I don't think these are available in hooks tho:
···
-- Later, Lukas #lzap Zapletal>
>
> These events come from orchestration, which I doubt would be required in
> Discovery, so I don't think it's likely to change.
>
>
I can think of several reasons why orchestration at the host discovery
level would be a required feature for properly managing onboarding of new
machines.
For example, setting IPMI user/password on all new machines into
non-default values, HW burn-in, HW validation rules to make sure that
machine-reported values conform to predefined model(s) (expected # of CPUs,
memory, etc) to catch machines that have less RAM due to DOA memory DIMMs,
etc.
The above also could be implemented as discovery rules in Foreman itself,
not as hooks. Just like auto-provisioning rules, but with some sort of
user-defined priorities, for example:
- on a discovery of a new host execute some single task (for example from
a list of tasks I mentioned above). If successful, move on to the next
rule. If something did not go through, list host as discovered but
failed/non-complaint, so they are not available for next rule, but user can
override/clear that on-demand, of course. - repeat for each of the defined rules
- if all of the above is good, host is listed as discovered and available
for either auto-provisioning rule or user for provisioning.
In fact, having such rules on discovery would be a preferred one over
hook-based orchestration, IMHO. Is there something like this in the works
by any chance? If not, would this be worth of consideration and put on a
roadmap for implementation?
>
> The create event (if it existed) wouldn't run on the save(:validate =>
> false) line, but would be roughly equivalent to running in
> after_validation or similar.
>
Thanks for suggestion, however, upon my attempt I found that
after_validation event is triggered as many times as after_commit, so
unless something like you suggested below is implemented, it is still very
generic.
> The technique that's commonly used is to record a flag on after_create
> or something, then to check for its presence and delete it in
> after_commit. If the flag's there you know the commit is probably from
> the creation, and if not, then it's some other event. It would be useful
> to add some more context into foreman_hooks for sure.
>
Yes, this is my next step.
Sorry, just to be clear I meant that as specifically Foreman's internal
"orchestration" mechanism, which is what provides the "create" hook.
I'm sure the examples you list are entirely valid.
···
On 11/11/15 20:10, 'Konstantin Orekhov' via Foreman users wrote: > > These events come from orchestration, which I doubt would be > required in > Discovery, so I don't think it's likely to change. > > > I can think of several reasons why orchestration at the host discovery > level would be a required feature for properly managing onboarding of > new machines.–
Dominic Cleal
dominic@cleal.org
> Sorry, just to be clear I meant that as specifically Foreman's internal
> "orchestration" mechanism, which is what provides the "create" hook.
> I'm sure the examples you list are entirely valid.
>
Right, I was talking about the same thing. IMHO, discovered hosts deserve
to have a proper internal orchestration events as there are a number of
things to be done on such hosts "outside" of Foreman control. Or, as I
mentioned above, that could be implemented as a set discovery rules to give
Foreman more control over what's happening with the hosts. Flagging
non-complaint hosts in case of HW test/burn-in failures, or failing any
other rule for that matter, for further reporting on a dashboard and API,
is a huge step in covering a complete life-cycle.
Is something like that maybe part of Host Aspects that Lukas mentioned
above?