Foreman Remote Execution design and deep-dive

Hi,

Lately, we've started looking more into the possibilities for bringing the
remote execution capabilities into the Foreman. We started collecting
the user stories that we would like to tackle and sketching some
designs that would allow that to happen.

The designs we have so far are collected here:

http://theforeman.github.io/foreman_remote_execution/design/

There is a deep-dive planned for going over the features and design
tomorrow, April 30th at 2 PM GMT (the google event might have wrong timezone
set up, it should really be 2 PM GMT +0000):

https://plus.google.com/events/cmtqsefb03pt6snommjd5g4b05s

The simplest description for this set of features would be: "let the
user run a script on a set of hosts".

There are however many questions hidden behind this simple sentence,
as well as opportunities to do more than a simple "ssh-loop" kind of thing.

Our main objectives are to:

  • support different network topologies: not everyone has the luxury
    of the Foreman server being able to reach directly to the host and
    vice-versa, not every host can see the Foreman server directly

  • support different remote execution tools: there are multiple
    tools already available (and the number is growing)
    providing the remote-management functionality, (ssh, mcollective, ansible, salt…
    you name it) every one having their pros and cons.

    We would like to avoid doing a choice for you on what tool to use
    (as in different contexts the best choice might be different). Rather, we
    want to provide an extendable architecture, that allows using multiple
    tools (even at once, if that's the case). The choice on what tool to
    implement first depends on the user interest.

  • usability: we want to leverage as much from the exising Foreman concepts
    as possible, such as scoped search for selecting the hosts to run against,
    reusing the bookmarks and improving them if needed

  • scalability: we would like to be able to run a job on thousands of
    hosts across the infrastructure and yet give you tools to not
    DDoS you infrastructure servers by doing so

  • security: we want to provide enough granularity on the permissions
    level to let the users to run just the commands they are allowed to
    run. Taxonomies included.

  • reusability: we would like to have a templating system (extending
    the existing config templates features) so that one can use the
    scripts multiple-times and share with others

  • usable by other plugins/core: it should be possible to use the
    functionality we provide in other Foreman features (puppet run, Katello
    package features, even finish scripts) and there is an API for exposing
    this kind of features

  • reliability: no matter if the hosts are online at the moment, we will track
    if the command run or not, and make sure we rerun it once it goes back
    online, or report that the execution failed

  • scheduling of reoccuring actions: sometimes, cron is not enough

  • accountability: all tasks are subject of auditing, can be reviewed and
    inspected later

  • although we haven't tackled the orchestration part in much details
    yet, the design already counts on the fact that the simple jobs
    should be pluggable together to perfrom more complex scenarios,
    covering multiple hosts. It's still quite foggy right now: therefore
    we leave this design opened for now, but keeping an eye on it already.

We look forward for the feedback, questions and suggestions.

– Ivan

The recording of the deep-dive is available:

https://plus.google.com/u/0/b/102496134326414788199/events/cmtqsefb03pt6snommjd5g4b05s

We went over the whole design page and tried to focus on the most important
parts and concepts the should help with understanding our intentions.
(I apologize for maybe too much information, but remote execution is
really large topic).

If there was something unclear from the deep-dive or the design page, please
let us know and we will try to fix that.

··· ----- Original Message ----- > Hi, > > Lately, we've started looking more into the possibilities for bringing the > remote execution capabilities into the Foreman. We started collecting > the user stories that we would like to tackle and sketching some > designs that would allow that to happen. > > The designs we have so far are collected here: > > http://theforeman.github.io/foreman_remote_execution/design/ > > There is a deep-dive planned for going over the features and design > tomorrow, April 30th at 2 PM GMT (the google event might have wrong timezone > set up, it should really be 2 PM GMT +0000): > > https://plus.google.com/events/cmtqsefb03pt6snommjd5g4b05s > > The simplest description for this set of features would be: "let the > user run a script on a set of hosts". > > There are however many questions hidden behind this simple sentence, > as well as opportunities to do more than a simple "ssh-loop" kind of thing. > > Our main objectives are to: > > * support different network topologies: not everyone has the luxury > of the Foreman server being able to reach directly to the host and > vice-versa, not every host can see the Foreman server directly > > * support different remote execution tools: there are multiple > tools already available (and the number is growing) > providing the remote-management functionality, (ssh, mcollective, ansible, > salt… > you name it) every one having their pros and cons. > > We would like to avoid doing a choice for you on what tool to use > (as in different contexts the best choice might be different). Rather, we > want to provide an extendable architecture, that allows using multiple > tools (even at once, if that's the case). The choice on what tool to > implement first depends on the user interest. > > * usability: we want to leverage as much from the exising Foreman concepts > as possible, such as scoped search for selecting the hosts to run against, > reusing the bookmarks and improving them if needed > > * scalability: we would like to be able to run a job on thousands of > hosts across the infrastructure and yet give you tools to not > DDoS you infrastructure servers by doing so > > * security: we want to provide enough granularity on the permissions > level to let the users to run just the commands they are allowed to > run. Taxonomies included. > > * reusability: we would like to have a templating system (extending > the existing config templates features) so that one can use the > scripts multiple-times and share with others > > * usable by other plugins/core: it should be possible to use the > functionality we provide in other Foreman features (puppet run, Katello > package features, even finish scripts) and there is an API for exposing > this kind of features > > * reliability: no matter if the hosts are online at the moment, we will track > if the command run or not, and make sure we rerun it once it goes back > online, or report that the execution failed > > * scheduling of reoccuring actions: sometimes, cron is not enough > > * accountability: all tasks are subject of auditing, can be reviewed and > inspected later > > * although we haven't tackled the orchestration part in much details > yet, the design already counts on the fact that the simple jobs > should be pluggable together to perfrom more complex scenarios, > covering multiple hosts. It's still quite foggy right now: therefore > we leave this design opened for now, but keeping an eye on it already. > > We look forward for the feedback, questions and suggestions. > > -- Ivan > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >

Hi Ivan,
Remote Execution from Foreman definitely sounds like it could be useful.
I'm only going to comment on the Ansible section of the design document, as
that's my area of experience.

Rather than just using the equivalent of ansible -m module -a arguments, it
would be better to be able to define playbooks or at least execute simple
roles. This would allow the encapsulation of a specific set of
configuration templates and tasks into a simple package.

Using Ansible Galaxy style roles as a mechanism for reusability could
definitely have the ability to simplify such playbooks. Recent changes to
ansible-galaxy have allowed roles to be sourced from arbitrary repositories
(rather than only galaxy.ansible.com)
http://willthames.github.io/2014/09/03/techniques-for-versioning-ansible-ii.html

The main problem then becomes the source of configuration information -
what Ansible calls Inventory. How do we specify what properties an
individual host has (perhaps by virtue of being a member of certain
groups). Inventory sources can be configuration files or scripts or a
combination of both, but a source of truth that can be used by Foreman and
perhaps other Ansible control systems is likely required. Could Foreman
perhaps even provide this source of truth (and by extension for things like
Puppet's hiera), even only as an option.

There's also the use case of e.g. rolling deploys, with the additional
problems of wanting to delegate tasks to things that aren't foreman
controlled hosts (tell monitoring to disable checks for this host, tell the
load balancer to mark this host offline etc). This might well be out of
scope for this though!

Will

··· On Thursday, April 30, 2015 at 2:56:08 AM UTC+10, Ivan Necas wrote: > > Hi, > > Lately, we've started looking more into the possibilities for bringing the > remote execution capabilities into the Foreman. We started collecting > the user stories that we would like to tackle and sketching some > designs that would allow that to happen. > > The designs we have so far are collected here: > > http://theforeman.github.io/foreman_remote_execution/design/ > > There is a deep-dive planned for going over the features and design > tomorrow, April 30th at 2 PM GMT (the google event might have wrong > timezone > set up, it should really be 2 PM GMT +0000): > > https://plus.google.com/events/cmtqsefb03pt6snommjd5g4b05s > > The simplest description for this set of features would be: "let the > user run a script on a set of hosts". > > There are however many questions hidden behind this simple sentence, > as well as opportunities to do more than a simple "ssh-loop" kind of > thing. > > Our main objectives are to: > > * support different network topologies: not everyone has the luxury > of the Foreman server being able to reach directly to the host and > vice-versa, not every host can see the Foreman server directly > > * support different remote execution tools: there are multiple > tools already available (and the number is growing) > providing the remote-management functionality, (ssh, mcollective, ansible, > salt… > you name it) every one having their pros and cons. > > We would like to avoid doing a choice for you on what tool to use > (as in different contexts the best choice might be different). Rather, we > want to provide an extendable architecture, that allows using multiple > tools (even at once, if that's the case). The choice on what tool to > implement first depends on the user interest. > > * usability: we want to leverage as much from the exising Foreman concepts > as possible, such as scoped search for selecting the hosts to run > against, > reusing the bookmarks and improving them if needed > > * scalability: we would like to be able to run a job on thousands of > hosts across the infrastructure and yet give you tools to not > DDoS you infrastructure servers by doing so > > * security: we want to provide enough granularity on the permissions > level to let the users to run just the commands they are allowed to > run. Taxonomies included. > > * reusability: we would like to have a templating system (extending > the existing config templates features) so that one can use the > scripts multiple-times and share with others > > * usable by other plugins/core: it should be possible to use the > functionality we provide in other Foreman features (puppet run, Katello > package features, even finish scripts) and there is an API for exposing > this kind of features > > * reliability: no matter if the hosts are online at the moment, we will > track > if the command run or not, and make sure we rerun it once it goes back > online, or report that the execution failed > > * scheduling of reoccuring actions: sometimes, cron is not enough > > * accountability: all tasks are subject of auditing, can be reviewed and > inspected later > > * although we haven't tackled the orchestration part in much details > yet, the design already counts on the fact that the simple jobs > should be pluggable together to perfrom more complex scenarios, > covering multiple hosts. It's still quite foggy right now: therefore > we leave this design opened for now, but keeping an eye on it already. > > We look forward for the feedback, questions and suggestions. > > -- Ivan >

<blockquote>
Hi,

Lately, we've started looking more into the possibilities for bringing the
remote execution capabilities into the Foreman. We started collecting
the user stories that we would like to tackle and sketching some
designs that would allow that to happen.

The designs we have so far are collected here:

There is a deep-dive planned for going over the features and design
tomorrow, April 30th at 2 PM GMT (the google event might have wrong timezone
set up, it should really be 2 PM GMT +0000):

The simplest description for this set of features would be: "let the
user run a script on a set of hosts".

There are however many questions hidden behind this simple sentence,
as well as opportunities to do more than a simple "ssh-loop" kind of thing.

Our main objectives are to:

  • support different network topologies: not everyone has the luxury
    of the Foreman server being able to reach directly to the host and
    vice-versa, not every host can see the Foreman server directly

  • support different remote execution tools: there are multiple
    tools already available (and the number is growing)
    providing the remote-management functionality, (ssh, mcollective, ansible, salt…
    you name it) every one having their pros and cons.

We would like to avoid doing a choice for you on what tool to use
(as in different contexts the best choice might be different). Rather, we
want to provide an extendable architecture, that allows using multiple
tools (even at once, if that's the case). The choice on what tool to
implement first depends on the user interest.

  • usability: we want to leverage as much from the exising Foreman concepts
    as possible, such as scoped search for selecting the hosts to run against,
    reusing the bookmarks and improving them if needed

  • scalability: we would like to be able to run a job on thousands of
    hosts across the infrastructure and yet give you tools to not
    DDoS you infrastructure servers by doing so

  • security: we want to provide enough granularity on the permissions
    level to let the users to run just the commands they are allowed to
    run. Taxonomies included.

  • reusability: we would like to have a templating system (extending
    the existing config templates features) so that one can use the
    scripts multiple-times and share with others

  • usable by other plugins/core: it should be possible to use the
    functionality we provide in other Foreman features (puppet run, Katello
    package features, even finish scripts) and there is an API for exposing
    this kind of features

  • reliability: no matter if the hosts are online at the moment, we will track
    if the command run or not, and make sure we rerun it once it goes back
    online, or report that the execution failed

  • scheduling of reoccuring actions: sometimes, cron is not enough

  • accountability: all tasks are subject of auditing, can be reviewed and
    inspected later

  • although we haven't tackled the orchestration part in much details
    yet, the design already counts on the fact that the simple jobs
    should be pluggable together to perfrom more complex scenarios,
    covering multiple hosts. It's still quite foggy right now: therefore
    we leave this design opened for now, but keeping an eye on it already.

We look forward for the feedback, questions and suggestions.

– Ivan

Hi Ivan,
Remote Execution from Foreman definitely sounds like it could be useful. I'm only going to comment on the Ansible section of the design document, as that's my area of experience.

Rather than just using the equivalent of ansible -m module -a arguments, it would be better to be able to define playbooks or at least execute simple roles. This would allow the encapsulation of a specific set of configuration templates and tasks into a simple package.

</blockquote>

Yes. The roadmap would be:

  1. ansible -m module -a arguments - this would give us the basic logic for parsing the ansible output and handling the execution itself
  2. playbooks with a single group - the reason for this step is that the work flow almost the same as with other remote-execution tools: you select a set of hosts and run the playbook against it
  3. playbooks with multiple groups: the thing that differs the most from the other cases, as there will be multiple targeting objects needed, representing the groups of hosts: since the
    ability to run a playbook against hosts with different roles is quite unique for the ansible (comparing to other remote execution tools), it would probably deserve a dedicated UI
    that would the foreman_ansible plugin introduce, building on top of the things that we will have in the foreman_remote_execution.

<blockquote>

Using Ansible Galaxy style roles as a mechanism for reusability could definitely have the ability to simplify such playbooks. Recent changes to ansible-galaxy have allowed roles to be sourced from arbitrary repositories (rather than only galaxy.ansible.com)
http://willthames.github.io/2014/09/03/techniques-for-versioning-ansible-ii.html

The main problem then becomes the source of configuration information - what Ansible calls Inventory. How do we specify what properties an individual host has (perhaps by virtue of being a member of certain groups). Inventory sources can be configuration files or scripts or a combination of both, but a source of truth that can be used by Foreman and perhaps other Ansible control systems is likely required. Could Foreman perhaps even provide this source of truth (and by extension for things like Puppet's hiera), even only as an option.

</blockquote>

What if we did is a part of the job template:

Let's take the hostname role as example ( https://github.com/ANXS/hostname ):

It can use the 'hostname_avahi' as the host_var parameter.

You could define a job template, that would look like this (assuming you've defined a playbook in ansible to use this role):

{ "playbook":"hostname",
"host_vars": { "hostname_avahi":"<%= host.parameters['secure'] == 'true' %>"} }

Talking about using host parameters in the templates, we would like to introduce the values as input types, so
you the template then would define the input 'hostname_avahi' that would link to host.parameters['secure'] (or a smart variable/some fact value)
and the template would look like this:

{ "playbook":"hostname",
"host_vars": { "hostname_avahi":"<%= @hostname_avahi %>"} }

The advance of this approach is that we would have in foreman the metadata about what parameters
the playbook needs from the host, and also leverage the smart variables there. We could then take
this even further, and generate the host vars automatically by the template inputs, maybe introducing
some reusable input groups, if more playbooks use the same vars.

I wonder if we then couldn't use similar approach with the hiera, using the templates to generate
the hiera yamls…

<blockquote>

There's also the use case of e.g. rolling deploys, with the additional problems of wanting to delegate tasks to things that aren't foreman controlled hosts (tell monitoring to disable checks for this host, tell the load balancer to mark this host offline etc). This might well be out of scope for this though!

</blockquote>

As I mentioned, it's not the immediate goal of ours. However, it's good to keep it in mind while doing the design
and looking at how would that fit in, as that's something we would like to look at once we're done with
the simpler cases.

– Ivan

<blockquote>

Will

··· ----- Original Message -----

On Thursday, April 30, 2015 at 2:56:08 AM UTC+10, Ivan Necas wrote:

You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com .
For more options, visit https://groups.google.com/d/optout .