Plugin Proposal: User-defined ENC mutation

Quite a while ago I posted a thread
<https://groups.google.com/forum/#!searchin/foreman-users/"smart$20classes"|sort:relevance/foreman-users/8BMaCeDXsM4/dVv5XSPqR7kJ>
to the Foreman Users group asking about the concept of "smart classes",
i.e. the ability to dynamically assign Puppet classes (similar to smart
class parameters) depending on fact or other data. I didn't get any
responses, and ultimately, our team ended up doing something
straightforward and created hostgroups for each permutation of Puppet
classes that we had in our environment (rather than sink development time
into solving the problem in a different way). Predictably, this has become
somewhat annoying to maintain as our environment has grown.

I started thinking more about this recently, and came up with what I think
might be a workable solution to this type of requirement. Let me know what
you think.

The basic problem is that Foreman's model for categorizing hosts is a
little bit too rigid to handle certain use cases. For my environment, there
are two basic issues: assigning a class based on fact or parameter data,
and removing inherited classes based on the same. I could imagine other
scenarios as well, however, for example retrieving a class parameter value
from an external system (rather than storing it statically within Foreman).

For users who can't accomplish their host classification with the builtin
tools, there would be a "safety valve" of sorts – a Foreman administrator
could define little bits of code that mutate the Foreman-generated ENC data
arbitrarily (sort of like ENC middleware), based on fact or other data
about the host. This would work in the following way:

The admin would define ENC "mutators" in a configuration directory, e.g.
/etc/foreman/mutators.d. There would be a standard API for these mutators
that might look something like this:

Mutator.create(:some_mutator_name) do |enc, facts|
if facts[:ipaddress] =~ /^10./ and !enc[:classes].has_key?('some::class')
enc[:classes]['some::class'] = nil
end
enc
end

(where facts&#39; is a hash of the requesting host&#39;s facts andenc' is the
standard ENC data returned by Foreman's node classifier)

When the Foreman server boots, it would load these mutators out of the
config dir. When a Puppet server requests ENC data for a host, Foreman
would retrieve the ENC data as normal, and then run the mutators against
that data in some configurable order. Each mutator would return the mutated
ENC, and this would be passed on to the next mutator in the chain. At the
end of the chain, the final ENC data would be passed back to the Puppet
server to use for catalog compilation. (Optionally, a system could be
established to allow the web UI user to pick and choose which host,
hostgroup, etc. gets which mutator and what order the mutators should run,
but this would take a little bit more effort to implement). Ideally, the
ENC data would be exposed in the web UI along each part of the chain so
that operators can inspect and validate its correctness – at the very
least, a 'before' and 'after' image could be displayed.

Before I start building something, I'd like to get some thoughts on this
idea. Would other people find this useful? Are there other approaches
people have tried and found successful?

Hello

thanks for the write up. first of all - I find it useful. Not only for users but
for plugin developers too. I encountered this scenario several times already,
I'd like to extend ENC from plugin. For example in foreman_openscap we provide
puppet module to configure scap client and we need to add the manifest to all
hosts on which certain policy should be applied. In other words we need to add
class and parameters for it into the ENC output. We would find the same thing
useful in remote execution to configure ssh keys.

So I wonder if we should add some helpers to Foreman core that would allow
register "mutators" and your plugin could use it and just add the UI that
allows adding rules that would add mutators. Or other plugins might depend on
your plugin to use mutators. I lean towards that we provide such mechanism in
core, so PRs against core would be welcome (from me at least).

It might be worth of suggesting through our RFC repo -
https://github.com/theforeman/rfcs see PRs for examples

Anyway let me know if I can help somehow with your effort.

··· -- Marek

On Monday 01 of August 2016 10:36:11 Jon McKenzie wrote:

Quite a while ago I posted a thread
<https://groups.google.com/forum/#!searchin/foreman-users/"smart$20classes
%22|sort:relevance/foreman-users/8BMaCeDXsM4/dVv5XSPqR7kJ> to the Foreman
Users group asking about the concept of “smart classes”, i.e. the ability
to dynamically assign Puppet classes (similar to smart class parameters)
depending on fact or other data. I didn’t get any responses, and
ultimately, our team ended up doing something
straightforward and created hostgroups for each permutation of Puppet
classes that we had in our environment (rather than sink development time
into solving the problem in a different way). Predictably, this has become
somewhat annoying to maintain as our environment has grown.

I started thinking more about this recently, and came up with what I think
might be a workable solution to this type of requirement. Let me know what
you think.

The basic problem is that Foreman’s model for categorizing hosts is a
little bit too rigid to handle certain use cases. For my environment, there
are two basic issues: assigning a class based on fact or parameter data,
and removing inherited classes based on the same. I could imagine other
scenarios as well, however, for example retrieving a class parameter value
from an external system (rather than storing it statically within Foreman).

For users who can’t accomplish their host classification with the builtin
tools, there would be a “safety valve” of sorts – a Foreman administrator
could define little bits of code that mutate the Foreman-generated ENC data
arbitrarily (sort of like ENC middleware), based on fact or other data
about the host. This would work in the following way:

The admin would define ENC “mutators” in a configuration directory, e.g.
/etc/foreman/mutators.d. There would be a standard API for these mutators
that might look something like this:

Mutator.create(:some_mutator_name) do |enc, facts|
if facts[:ipaddress] =~ /^10./ and !enc[:classes].has_key?(‘some::class’)
enc[:classes][‘some::class’] = nil
end
enc
end

(where facts' is a hash of the requesting host's facts andenc’ is the
standard ENC data returned by Foreman’s node classifier)

When the Foreman server boots, it would load these mutators out of the
config dir. When a Puppet server requests ENC data for a host, Foreman
would retrieve the ENC data as normal, and then run the mutators against
that data in some configurable order. Each mutator would return the mutated
ENC, and this would be passed on to the next mutator in the chain. At the
end of the chain, the final ENC data would be passed back to the Puppet
server to use for catalog compilation. (Optionally, a system could be
established to allow the web UI user to pick and choose which host,
hostgroup, etc. gets which mutator and what order the mutators should run,
but this would take a little bit more effort to implement). Ideally, the
ENC data would be exposed in the web UI along each part of the chain so
that operators can inspect and validate its correctness – at the very
least, a ‘before’ and ‘after’ image could be displayed.

Before I start building something, I’d like to get some thoughts on this
idea. Would other people find this useful? Are there other approaches
people have tried and found successful?

Thanks Marek, this is helpful. I'll work on an RFC to try and hash out some
of the details.

··· On Tuesday, August 2, 2016 at 9:58:59 AM UTC-4, Marek Hulán wrote: > > Hello > > thanks for the write up. first of all - I find it useful. Not only for > users but > for plugin developers too. I encountered this scenario several times > already, > I'd like to extend ENC from plugin. For example in foreman_openscap we > provide > puppet module to configure scap client and we need to add the manifest to > all > hosts on which certain policy should be applied. In other words we need to > add > class and parameters for it into the ENC output. We would find the same > thing > useful in remote execution to configure ssh keys. > > So I wonder if we should add some helpers to Foreman core that would allow > register "mutators" and your plugin could use it and just add the UI that > allows adding rules that would add mutators. Or other plugins might depend > on > your plugin to use mutators. I lean towards that we provide such mechanism > in > core, so PRs against core would be welcome (from me at least). > > It might be worth of suggesting through our RFC repo - > https://github.com/theforeman/rfcs see PRs for examples > > Anyway let me know if I can help somehow with your effort. > > -- > Marek > > On Monday 01 of August 2016 10:36:11 Jon McKenzie wrote: > > Quite a while ago I posted a thread > > < > https://groups.google.com/forum/#!searchin/foreman-users/%22smart$20classes > > %22|sort:relevance/foreman-users/8BMaCeDXsM4/dVv5XSPqR7kJ> to the > Foreman > > Users group asking about the concept of "smart classes", i.e. the > ability > > to dynamically assign Puppet classes (similar to smart class parameters) > > depending on fact or other data. I didn't get any responses, and > > ultimately, our team ended up doing something > > straightforward and created hostgroups for each permutation of Puppet > > classes that we had in our environment (rather than sink development > time > > into solving the problem in a different way). Predictably, this has > become > > somewhat annoying to maintain as our environment has grown. > > > > I started thinking more about this recently, and came up with what I > think > > might be a workable solution to this type of requirement. Let me know > what > > you think. > > > > The basic problem is that Foreman's model for categorizing hosts is a > > little bit too rigid to handle certain use cases. For my environment, > there > > are two basic issues: assigning a class based on fact or parameter data, > > and removing inherited classes based on the same. I could imagine other > > scenarios as well, however, for example retrieving a class parameter > value > > from an external system (rather than storing it statically within > Foreman). > > > > For users who can't accomplish their host classification with the > builtin > > tools, there would be a "safety valve" of sorts -- a Foreman > administrator > > could define little bits of code that mutate the Foreman-generated ENC > data > > arbitrarily (sort of like ENC middleware), based on fact or other data > > about the host. This would work in the following way: > > > > The admin would define ENC "mutators" in a configuration directory, e.g. > > /etc/foreman/mutators.d. There would be a standard API for these > mutators > > that might look something like this: > > > > Mutator.create(:some_mutator_name) do |enc, facts| > > if facts[:ipaddress] =~ /^10\./ and > !enc[:classes].has_key?('some::class') > > enc[:classes]['some::class'] = nil > > end > > enc > > end > > > > (where `facts' is a hash of the requesting host's facts and `enc' is the > > standard ENC data returned by Foreman's node classifier) > > > > When the Foreman server boots, it would load these mutators out of the > > config dir. When a Puppet server requests ENC data for a host, Foreman > > would retrieve the ENC data as normal, and then run the mutators against > > that data in some configurable order. Each mutator would return the > mutated > > ENC, and this would be passed on to the next mutator in the chain. At > the > > end of the chain, the final ENC data would be passed back to the Puppet > > server to use for catalog compilation. (Optionally, a system could be > > established to allow the web UI user to pick and choose which host, > > hostgroup, etc. gets which mutator and what order the mutators should > run, > > but this would take a little bit more effort to implement). Ideally, the > > ENC data would be exposed in the web UI along each part of the chain so > > that operators can inspect and validate its correctness -- at the very > > least, a 'before' and 'after' image could be displayed. > > > > Before I start building something, I'd like to get some thoughts on this > > idea. Would other people find this useful? Are there other approaches > > people have tried and found successful? > >

Actually I have added a PR [1] yesterday that does something quite similar

  • I have opened the Host#info method for plugins.
    Let me know if it was useful, or if you want more information about the pr.

[1] https://github.com/theforeman/foreman/pull/3701

··· On Wednesday, August 3, 2016 at 8:47:36 PM UTC+3, Jon McKenzie wrote: > > Thanks Marek, this is helpful. I'll work on an RFC to try and hash out > some of the details. > > On Tuesday, August 2, 2016 at 9:58:59 AM UTC-4, Marek Hulán wrote: >> >> Hello >> >> thanks for the write up. first of all - I find it useful. Not only for >> users but >> for plugin developers too. I encountered this scenario several times >> already, >> I'd like to extend ENC from plugin. For example in foreman_openscap we >> provide >> puppet module to configure scap client and we need to add the manifest to >> all >> hosts on which certain policy should be applied. In other words we need >> to add >> class and parameters for it into the ENC output. We would find the same >> thing >> useful in remote execution to configure ssh keys. >> >> So I wonder if we should add some helpers to Foreman core that would >> allow >> register "mutators" and your plugin could use it and just add the UI that >> allows adding rules that would add mutators. Or other plugins might >> depend on >> your plugin to use mutators. I lean towards that we provide such >> mechanism in >> core, so PRs against core would be welcome (from me at least). >> >> It might be worth of suggesting through our RFC repo - >> https://github.com/theforeman/rfcs see PRs for examples >> >> Anyway let me know if I can help somehow with your effort. >> >> -- >> Marek >> >> On Monday 01 of August 2016 10:36:11 Jon McKenzie wrote: >> > Quite a while ago I posted a thread >> > < >> https://groups.google.com/forum/#!searchin/foreman-users/%22smart$20classes >> > %22|sort:relevance/foreman-users/8BMaCeDXsM4/dVv5XSPqR7kJ> to the >> Foreman >> > Users group asking about the concept of "smart classes", i.e. the >> ability >> > to dynamically assign Puppet classes (similar to smart class >> parameters) >> > depending on fact or other data. I didn't get any responses, and >> > ultimately, our team ended up doing something >> > straightforward and created hostgroups for each permutation of Puppet >> > classes that we had in our environment (rather than sink development >> time >> > into solving the problem in a different way). Predictably, this has >> become >> > somewhat annoying to maintain as our environment has grown. >> > >> > I started thinking more about this recently, and came up with what I >> think >> > might be a workable solution to this type of requirement. Let me know >> what >> > you think. >> > >> > The basic problem is that Foreman's model for categorizing hosts is a >> > little bit too rigid to handle certain use cases. For my environment, >> there >> > are two basic issues: assigning a class based on fact or parameter >> data, >> > and removing inherited classes based on the same. I could imagine other >> > scenarios as well, however, for example retrieving a class parameter >> value >> > from an external system (rather than storing it statically within >> Foreman). >> > >> > For users who can't accomplish their host classification with the >> builtin >> > tools, there would be a "safety valve" of sorts -- a Foreman >> administrator >> > could define little bits of code that mutate the Foreman-generated ENC >> data >> > arbitrarily (sort of like ENC middleware), based on fact or other data >> > about the host. This would work in the following way: >> > >> > The admin would define ENC "mutators" in a configuration directory, >> e.g. >> > /etc/foreman/mutators.d. There would be a standard API for these >> mutators >> > that might look something like this: >> > >> > Mutator.create(:some_mutator_name) do |enc, facts| >> > if facts[:ipaddress] =~ /^10\./ and >> !enc[:classes].has_key?('some::class') >> > enc[:classes]['some::class'] = nil >> > end >> > enc >> > end >> > >> > (where `facts' is a hash of the requesting host's facts and `enc' is >> the >> > standard ENC data returned by Foreman's node classifier) >> > >> > When the Foreman server boots, it would load these mutators out of the >> > config dir. When a Puppet server requests ENC data for a host, Foreman >> > would retrieve the ENC data as normal, and then run the mutators >> against >> > that data in some configurable order. Each mutator would return the >> mutated >> > ENC, and this would be passed on to the next mutator in the chain. At >> the >> > end of the chain, the final ENC data would be passed back to the Puppet >> > server to use for catalog compilation. (Optionally, a system could be >> > established to allow the web UI user to pick and choose which host, >> > hostgroup, etc. gets which mutator and what order the mutators should >> run, >> > but this would take a little bit more effort to implement). Ideally, >> the >> > ENC data would be exposed in the web UI along each part of the chain so >> > that operators can inspect and validate its correctness -- at the very >> > least, a 'before' and 'after' image could be displayed. >> > >> > Before I start building something, I'd like to get some thoughts on >> this >> > idea. Would other people find this useful? Are there other approaches >> > people have tried and found successful? >> >>

You should also consider if it should within the foreman or in node.rb.
Both have advantages and disadvantages. In a sense there's already a
mutator in place in node.rb1.

··· On Wed, Aug 03, 2016 at 10:47:35AM -0700, Jon McKenzie wrote: > Thanks Marek, this is helpful. I'll work on an RFC to try and hash out some > of the details. > > On Tuesday, August 2, 2016 at 9:58:59 AM UTC-4, Marek Hulán wrote: > > > > Hello > > > > thanks for the write up. first of all - I find it useful. Not only for > > users but > > for plugin developers too. I encountered this scenario several times > > already, > > I'd like to extend ENC from plugin. For example in foreman_openscap we > > provide > > puppet module to configure scap client and we need to add the manifest to > > all > > hosts on which certain policy should be applied. In other words we need to > > add > > class and parameters for it into the ENC output. We would find the same > > thing > > useful in remote execution to configure ssh keys. > > > > So I wonder if we should add some helpers to Foreman core that would allow > > register "mutators" and your plugin could use it and just add the UI that > > allows adding rules that would add mutators. Or other plugins might depend > > on > > your plugin to use mutators. I lean towards that we provide such mechanism > > in > > core, so PRs against core would be welcome (from me at least). > > > > It might be worth of suggesting through our RFC repo - > > https://github.com/theforeman/rfcs see PRs for examples > > > > Anyway let me know if I can help somehow with your effort. > > > > -- > > Marek > > > > On Monday 01 of August 2016 10:36:11 Jon McKenzie wrote: > > > Quite a while ago I posted a thread > > > < > > https://groups.google.com/forum/#!searchin/foreman-users/%22smart$20classes > > > %22|sort:relevance/foreman-users/8BMaCeDXsM4/dVv5XSPqR7kJ> to the > > Foreman > > > Users group asking about the concept of "smart classes", i.e. the > > ability > > > to dynamically assign Puppet classes (similar to smart class parameters) > > > depending on fact or other data. I didn't get any responses, and > > > ultimately, our team ended up doing something > > > straightforward and created hostgroups for each permutation of Puppet > > > classes that we had in our environment (rather than sink development > > time > > > into solving the problem in a different way). Predictably, this has > > become > > > somewhat annoying to maintain as our environment has grown. > > > > > > I started thinking more about this recently, and came up with what I > > think > > > might be a workable solution to this type of requirement. Let me know > > what > > > you think. > > > > > > The basic problem is that Foreman's model for categorizing hosts is a > > > little bit too rigid to handle certain use cases. For my environment, > > there > > > are two basic issues: assigning a class based on fact or parameter data, > > > and removing inherited classes based on the same. I could imagine other > > > scenarios as well, however, for example retrieving a class parameter > > value > > > from an external system (rather than storing it statically within > > Foreman). > > > > > > For users who can't accomplish their host classification with the > > builtin > > > tools, there would be a "safety valve" of sorts -- a Foreman > > administrator > > > could define little bits of code that mutate the Foreman-generated ENC > > data > > > arbitrarily (sort of like ENC middleware), based on fact or other data > > > about the host. This would work in the following way: > > > > > > The admin would define ENC "mutators" in a configuration directory, e.g. > > > /etc/foreman/mutators.d. There would be a standard API for these > > mutators > > > that might look something like this: > > > > > > Mutator.create(:some_mutator_name) do |enc, facts| > > > if facts[:ipaddress] =~ /^10\./ and > > !enc[:classes].has_key?('some::class') > > > enc[:classes]['some::class'] = nil > > > end > > > enc > > > end > > > > > > (where `facts' is a hash of the requesting host's facts and `enc' is the > > > standard ENC data returned by Foreman's node classifier) > > > > > > When the Foreman server boots, it would load these mutators out of the > > > config dir. When a Puppet server requests ENC data for a host, Foreman > > > would retrieve the ENC data as normal, and then run the mutators against > > > that data in some configurable order. Each mutator would return the > > mutated > > > ENC, and this would be passed on to the next mutator in the chain. At > > the > > > end of the chain, the final ENC data would be passed back to the Puppet > > > server to use for catalog compilation. (Optionally, a system could be > > > established to allow the web UI user to pick and choose which host, > > > hostgroup, etc. gets which mutator and what order the mutators should > > run, > > > but this would take a little bit more effort to implement). Ideally, the > > > ENC data would be exposed in the web UI along each part of the chain so > > > that operators can inspect and validate its correctness -- at the very > > > least, a 'before' and 'after' image could be displayed. > > > > > > Before I start building something, I'd like to get some thoughts on this > > > idea. Would other people find this useful? Are there other approaches > > > people have tried and found successful? > > > > > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.