I spent some time the other day finding out how would we go about
integrating Ansible, since it seems it has the potential to bring a ton
of users to the Foreman ecosystem, and we could help them manage their
systems.
I HATE building castles in the skies, so for starters I'll publish
the foreman_ansible facts parser soon, I hope this week.
Here are my conclusions:
Plan
Finish FactParser and FactImporter
Write a task that imports facts for certain hosts, a host group,
etc…
Write a callback plugin that when put on ansible.cfg posts facts to
Foreman
Currently facts in Ansible are gathered through ansible -m setup $HOSTNAME. I've been writing a parser and it seems fairly trivial,
FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
'work well'. There's a gotcha though, and it's that these facts are
structured, so we should write a parser that creates these structured
facts like in Chef and Salt.
We could simply make a POST call to api/v2/host/facts and
api/v2/reports after runner_* actions.
I have not played with importing reports at all nor I know the format of
these in Ansible, the good news is we can do a good deal of data
massaging if needed on the callback plugin.
Remote execution
I think the obvious choice here is to rely on
I have a few problems with this approach, that I think we can fix on
future releases of remote-execution:
One-off ssh jobs now invocations are very complicated now even for a
simple case.
They require:
(1) setting up a proxy, installing a plugin on the proxy,
(2) adding the proxy in Foreman or refreshing features,
(3) writing a job template (and understanding the job template form).
I understand this complexity for 'enterprise' networks but not for
small deployments. IMO I'd like to avoid this for Ansible, because
it's harder to deploy, which will mean some users will get frustrated,
and secondly because it's just not needed in a lot of cases.
Ideally you should be able to go to any host, click on 'run job', it
could open a modal, write an ansible command, and run it. Through
foreman_ansible itself, no need for job templates, proxies, etc…
I think this would be easier to start with, and would support a workflow
that most users with small deployments have. Not everybody needs a proxy
and in this case it's only use is when there is a segmented network, for
remote execution.
Ansible inventory
This is not strictly 'foreman_ansible', but we'd make it much simpler
for Ansible people if we wrote a proper inventory that uses Foreman.
Basically this means writing a python script that queries Foreman and
allows Ansible to understand Foreman hostgroups as Ansible group names.
Aside from that, we could automate importing /etc/ansible/hosts to
Foreman as host groups. That should be trivial but I don't know whether
people want this or not.
Playbooks and modules importing
Possibly we could import playbooks and modules as Job templates or
something like that. I didn't have a chance to get into this. In this
case we probably want a proxy to import playbooks in the system, I
agree.
I'd rather finish the rest before, but if you have ideas it'd be great
to have them in this thread.
> Hi foreman devs,
>
> I spent some time the other day finding out how would we go about
> integrating Ansible, since it seems it has the potential to bring a ton
> of users to the Foreman ecosystem, and we could help them manage their
> systems.
>
> I HATE building castles in the skies, so for starters I'll publish
> the foreman_ansible facts parser soon, I hope this week.
>
> Here are my conclusions:
>
> # Plan
>
> - Finish FactParser and FactImporter
> - Write a task that imports facts for certain hosts, a host group,
> etc…
> - Write a callback plugin that when put on ansible.cfg posts facts to
> Foreman
>
> - (?) Parsing reports? Remote execution Ansible provider?
>
> # Importing Facts and Reports (feasible)
>
> Currently facts in Ansible are gathered through ansible -m setup > $HOSTNAME. I've been writing a parser and it seems fairly trivial,
> FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
> 'work well'. There's a gotcha though, and it's that these facts are
> structured, so we should write a parser that creates these structured
> facts like in Chef and Salt.
>
> Initially, as an MVP we can either use a cronjob, or manually trigger
> the retrieval of facts in Foreman, just to prove it works. The way they
> should normally be submitted though, is by writing a callback plugin in
> Ansible. This callback plugin would submit reports and facts after
> completion of certain actions - see
> https://gist.github.com/cliffano/9868180 for an example. Callback
> plugins run automatically after configuring it in ansible.cfg.
>
> We could simply make a POST call to api/v2/host/facts and
> api/v2/reports after runner_* actions.
>
> I have not played with importing reports at all nor I know the format of
> these in Ansible, the good news is we can do a good deal of data
> massaging if needed on the callback plugin.
>
> # Remote execution
>
> I think the obvious choice here is to rely on
> https://github.com/theforeman/foreman_remote_execution/
> I have a few problems with this approach, that I think we can fix on
> future releases of remote-execution:
>
> One-off ssh jobs now invocations are very complicated now even for a
> simple case.
>
> They require:
> (1) setting up a proxy, installing a plugin on the proxy,
> (2) adding the proxy in Foreman or refreshing features,
> (3) writing a job template (and understanding the job template form).
>
> I understand this complexity for 'enterprise' networks but not for
> small deployments. IMO I'd like to avoid this for Ansible, because
> it's harder to deploy, which will mean some users will get frustrated,
> and secondly because it's just not needed in a lot of cases.
>
> Ideally you should be able to go to any host, click on 'run job', it
> could open a modal, write an ansible command, and run it. Through
> foreman_ansible itself, no need for job templates, proxies, etc…
>
> I think this would be easier to start with, and would support a workflow
> that most users with small deployments have. Not everybody needs a proxy
> and in this case it's only use is when there is a segmented network, for
> remote execution.
As a user of remote execution (I have not written any of the code), I don't think the simple scenario you described is contradictory to the intention of the plugin. To me there is a lot of potential refinement on the presentation layer (ie. UI) to make things easier for the simpler use cases. Writing something new for ansible doesn't seem to be in the spirit of making what we have better: Pull-requests welcome, right? I also believe that the remote exec authors intended it to be able to back multiple types (mcollective, ansible, etc.).
As an example, I wrote a docker hammer container. I use the remote exec to run hammer commands through docker on the katello server itself. It is as easy as you describe: Pick a host, type in the command, hit run. Again, some refinement in the UI would be nice and I think more hands in that area from using it with ansible would be the right approach.
···
----- Original Message -----
Ansible inventory
This is not strictly ‘foreman_ansible’, but we’d make it much simpler
for Ansible people if we wrote a proper inventory that uses Foreman.
Basically this means writing a python script that queries Foreman and
allows Ansible to understand Foreman hostgroups as Ansible group names.
Aside from that, we could automate importing /etc/ansible/hosts to
Foreman as host groups. That should be trivial but I don’t know whether
people want this or not.
Playbooks and modules importing
Possibly we could import playbooks and modules as Job templates or
something like that. I didn’t have a chance to get into this. In this
case we probably want a proxy to import playbooks in the system, I
agree.
I’d rather finish the rest before, but if you have ideas it’d be great
to have them in this thread.
> Hi foreman devs,
>
> I spent some time the other day finding out how would we go about
> integrating Ansible, since it seems it has the potential to bring a ton
> of users to the Foreman ecosystem, and we could help them manage their
> systems.
>
> I HATE building castles in the skies, so for starters I'll publish
> the foreman_ansible facts parser soon, I hope this week.
>
> Here are my conclusions:
>
> # Plan
>
> - Finish FactParser and FactImporter
> - Write a task that imports facts for certain hosts, a host group,
> etc…
> - Write a callback plugin that when put on ansible.cfg posts facts to
> Foreman
>
> - (?) Parsing reports? Remote execution Ansible provider?
>
> # Importing Facts and Reports (feasible)
>
> Currently facts in Ansible are gathered through ansible -m setup > $HOSTNAME. I've been writing a parser and it seems fairly trivial,
> FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
> 'work well'. There's a gotcha though, and it's that these facts are
> structured, so we should write a parser that creates these structured
> facts like in Chef and Salt.
>
> Initially, as an MVP we can either use a cronjob, or manually trigger
> the retrieval of facts in Foreman, just to prove it works. The way they
> should normally be submitted though, is by writing a callback plugin in
> Ansible. This callback plugin would submit reports and facts after
> completion of certain actions - see
> https://gist.github.com/cliffano/9868180 for an example. Callback
> plugins run automatically after configuring it in ansible.cfg.
>
> We could simply make a POST call to api/v2/host/facts and
> api/v2/reports after runner_* actions.
>
> I have not played with importing reports at all nor I know the format of
> these in Ansible, the good news is we can do a good deal of data
> massaging if needed on the callback plugin.
>
> # Remote execution
>
> I think the obvious choice here is to rely on
> https://github.com/theforeman/foreman_remote_execution/
> I have a few problems with this approach, that I think we can fix on
> future releases of remote-execution:
>
> One-off ssh jobs now invocations are very complicated now even for a
> simple case.
>
>
> They require:
> (1) setting up a proxy, installing a plugin on the proxy,
> (2) adding the proxy in Foreman or refreshing features,
This is the same as any plugin that has a smart proxy component.
I spent a good amount of time making sure rex is plug-and-play, and it
nearly is. Run the installer with the two plugins enabled, and the
default settings make it all Just Work (see below).
> (3) writing a job template (and understanding the job template form).
We supply some useful defaults, so not required step.
> I understand this complexity for 'enterprise' networks but not for
> small deployments. IMO I'd like to avoid this for Ansible, because
> it's harder to deploy, which will mean some users will get frustrated,
> and secondly because it's just not needed in a lot of cases.
>
> Ideally you should be able to go to any host, click on 'run job', it
> could open a modal, write an ansible command, and run it. Through
> foreman_ansible itself, no need for job templates, proxies, etc…
Go to settings and turn on remote_execution_global_proxy to true
Go to a Foreman host, click "Run Job"
Select from one of the default templates, like "Run command", and
type in "ping google.com"
No tempate writing - use the "Run command" one we provide. You can dump
whatever you like in the input. Step (2) deals with proxy but that is a
default in Iteration 3.
So this is exactly your given workflow. The proxy is still there, but
changing the fundamental architecture is a ship that has sailed.
If you have more suggestions for simplification, redmine's are really
welcome. Ohad has submitted a zillion of them and they've been really
valuable.
>
> I think this would be easier to start with, and would support a workflow
> that most users with small deployments have. Not everybody needs a proxy
> and in this case it's only use is when there is a segmented network, for
> remote execution.
The Smart Proxy architecture is how Foreman talks to external services,
and that's what ansible is. Abandoning that makes no sense, because
that architecture works very well. It supports a very simple use case
and very complex use cases from one code base.
Are we supposed to start of with a core plugin and no proxy
functionality and someday if a user wants a smart proxy they're SOL? Why
not start with that from the beginning?
We run into problems with developers not considering the bigger users of
foreman all the time, and we had to make a lot of changes to support the
idea of host isolation - that a host only ever talks to the proxy.
We're in good shape now, although there's areas where the proxy can't do
some things - like compute resources. That hurts us there – because it
may work for you, but it doesn't for many users that have to reach into
other networks and talk to a vsphere behind a firewall, for example.
Starting out with a plugin that rejects anybody but simple users of
Foreman doesn't seem like the right thing to do.
···
On Mon, Oct 26, 2015 at 07:08:32PM +0100, Daniel Lobato Garcia wrote:
Ansible inventory
This is not strictly ‘foreman_ansible’, but we’d make it much simpler
for Ansible people if we wrote a proper inventory that uses Foreman.
Basically this means writing a python script that queries Foreman and
allows Ansible to understand Foreman hostgroups as Ansible group names.
Aside from that, we could automate importing /etc/ansible/hosts to
Foreman as host groups. That should be trivial but I don’t know whether
people want this or not.
Playbooks and modules importing
Possibly we could import playbooks and modules as Job templates or
something like that. I didn’t have a chance to get into this. In this
case we probably want a proxy to import playbooks in the system, I
agree.
I’d rather finish the rest before, but if you have ideas it’d be great
to have them in this thread.
> Hi foreman devs,
>
> I spent some time the other day finding out how would we go about
> integrating Ansible, since it seems it has the potential to bring a ton
> of users to the Foreman ecosystem, and we could help them manage their
> systems.
>
> I HATE building castles in the skies, so for starters I'll publish
> the foreman_ansible facts parser soon, I hope this week.
>
> Here are my conclusions:
>
> # Plan
>
> - Finish FactParser and FactImporter
> - Write a task that imports facts for certain hosts, a host group,
> etc…
> - Write a callback plugin that when put on ansible.cfg posts facts to
> Foreman
>
> - (?) Parsing reports? Remote execution Ansible provider?
>
> # Importing Facts and Reports (feasible)
>
> Currently facts in Ansible are gathered through ansible -m setup > $HOSTNAME. I've been writing a parser and it seems fairly trivial,
> FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
> 'work well'. There's a gotcha though, and it's that these facts are
> structured, so we should write a parser that creates these structured
> facts like in Chef and Salt.
>
> Initially, as an MVP we can either use a cronjob, or manually trigger
> the retrieval of facts in Foreman, just to prove it works. The way they
> should normally be submitted though, is by writing a callback plugin in
> Ansible. This callback plugin would submit reports and facts after
> completion of certain actions - see
> https://gist.github.com/cliffano/9868180 for an example. Callback
> plugins run automatically after configuring it in ansible.cfg.
>
> We could simply make a POST call to api/v2/host/facts and
> api/v2/reports after runner_* actions.
>
> I have not played with importing reports at all nor I know the format of
> these in Ansible, the good news is we can do a good deal of data
> massaging if needed on the callback plugin.
>
> # Remote execution
>
> I think the obvious choice here is to rely on
> https://github.com/theforeman/foreman_remote_execution/
> I have a few problems with this approach, that I think we can fix on
> future releases of remote-execution:
>
> One-off ssh jobs now invocations are very complicated now even for a
> simple case.
>
> They require:
> (1) setting up a proxy, installing a plugin on the proxy,
> (2) adding the proxy in Foreman or refreshing features,
> (3) writing a job template (and understanding the job template form).
>
> I understand this complexity for 'enterprise' networks but not for
> small deployments. IMO I'd like to avoid this for Ansible, because
> it's harder to deploy, which will mean some users will get frustrated,
> and secondly because it's just not needed in a lot of cases.
>
> Ideally you should be able to go to any host, click on 'run job', it
> could open a modal, write an ansible command, and run it. Through
> foreman_ansible itself, no need for job templates, proxies, etc…
>
> I think this would be easier to start with, and would support a workflow
> that most users with small deployments have. Not everybody needs a proxy
> and in this case it's only use is when there is a segmented network, for
> remote execution.
>
> # Ansible inventory
>
> This is not strictly 'foreman_ansible', but we'd make it much simpler
> for Ansible people if we wrote a proper inventory that uses Foreman.
>
> Basically this means writing a python script that queries Foreman and
> allows Ansible to understand Foreman hostgroups as Ansible group names.
>
> See an example (unmaintained I think) here
> https://github.com/romeotheriault/foreman-ansible , it fetches Foreman
> hostnames but it doesn't take hostgroups into account.
>
> It'd be awesome if we could land that in ansible/ansible contrib
> https://github.com/ansible/ansible/tree/devel/contrib/inventory
>
> Aside from that, we could automate importing /etc/ansible/hosts to
> Foreman as host groups. That should be trivial but I don't know whether
> people want this or not.
>
> # Playbooks and modules importing
>
> Possibly we could import playbooks and modules as Job templates or
> something like that. I didn't have a chance to get into this. In this
> case we probably want a proxy to import playbooks in the system, I
> agree.
> I'd rather finish the rest before, but if you have ideas it'd be great
> to have them in this thread.
>
Given you are looking into this – any thoughts on local management of
Ansible content and what that might look like both from a user perspective
and developer effort?
Thanks,
Eric
···
On Mon, Oct 26, 2015 at 2:08 PM, Daniel Lobato Garcia wrote:
Also spent too much time replying to the remote execution part - for the
initial integration point, I think it's good to have the foundations
first.
The first being automatic configuration of the client - which is
basically how Chef and Salt started, somebody submitted PR's to do this
as snippets. Less relevant for Ansible, but you still do have to deploy
SSH keys, so some "foremnay" way to do that in all the templates is
probably a good first step.
Then facts, reports, ENC, etc.
I think REX is probably the last thing to work on.
···
On Mon, Oct 26, 2015 at 03:58:03PM -0400, Stephen Benjamin wrote:
> On Mon, Oct 26, 2015 at 07:08:32PM +0100, Daniel Lobato Garcia wrote:
> > Hi foreman devs,
> >
> > I spent some time the other day finding out how would we go about
> > integrating Ansible, since it seems it has the potential to bring a ton
> > of users to the Foreman ecosystem, and we could help them manage their
> > systems.
> >
> > I *HATE* building castles in the skies, so for starters I'll publish
> > the foreman_ansible facts parser soon, I hope this week.
> >
> > Here are my conclusions:
> >
> > # Plan
> >
> > - Finish FactParser and FactImporter
> > - Write a task that imports facts for certain hosts, a host group,
> > etc...
> > - Write a callback plugin that when put on ansible.cfg posts facts to
> > Foreman
> >
> > - (?) Parsing reports? Remote execution Ansible provider?
> >
> > # Importing Facts and Reports (feasible)
> >
> > Currently facts in Ansible are gathered through `ansible -m setup
> > $HOSTNAME`. I've been writing a parser and it seems fairly trivial,
> > FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
> > 'work well'. There's a gotcha though, and it's that these facts are
> > structured, so we should write a parser that creates these structured
> > facts like in Chef and Salt.
> >
> > Initially, as an MVP we can either use a cronjob, or manually trigger
> > the retrieval of facts in Foreman, just to prove it works. The way they
> > should normally be submitted though, is by writing a callback plugin in
> > Ansible. This callback plugin would submit reports and facts after
> > completion of certain actions - see
> > https://gist.github.com/cliffano/9868180 for an example. Callback
> > plugins run automatically after configuring it in ansible.cfg.
> >
> > We could simply make a POST call to api/v2/host/facts and
> > api/v2/reports after runner_* actions.
> >
> > I have not played with importing reports at all nor I know the format of
> > these in Ansible, the good news is we can do a good deal of data
> > massaging if needed on the callback plugin.
> >
> > # Remote execution
> >
> > I think the obvious choice here is to rely on
> > https://github.com/theforeman/foreman_remote_execution/
> > I have a few problems with this approach, that I think we can fix on
> > future releases of remote-execution:
> >
> > One-off ssh jobs now invocations are very complicated now even for a
> > simple case.
> >
> >
> > They require:
> > (1) setting up a proxy, installing a plugin on the proxy,
> > (2) adding the proxy in Foreman or refreshing features,
>
> This is the same as any plugin that has a smart proxy component.
>
> I spent a good amount of time making sure rex is plug-and-play, and it
> nearly is. Run the installer with the two plugins enabled, and the
> default settings make it all Just Work (see below).
>
> > (3) writing a job template (and understanding the job template form).
>
> We supply some useful defaults, so not required step.
>
> > I understand this complexity for 'enterprise' networks but not for
> > small deployments. IMO I'd like to avoid this for Ansible, because
> > it's harder to deploy, which will mean some users will get frustrated,
> > and secondly because it's just not needed in a lot of cases.
> >
> > Ideally you should be able to go to any host, click on 'run job', it
> > could open a modal, write an ansible command, and run it. Through
> > foreman_ansible itself, no need for job templates, proxies, etc...
>
> REX does exactly what you say.
>
> 1. foreman-installer --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh
> 2. Go to settings and turn on remote_execution_global_proxy to true
> 3. Go to a Foreman host, click "Run Job"
> 4. Select from one of the default templates, like "Run command", and
> type in "ping google.com"
>
> No tempate writing - use the "Run command" one we provide. You can dump
> whatever you like in the input. Step (2) deals with proxy but that is a
> default in Iteration 3.
>
> So this is *exactly* your given workflow. The proxy is still there, but
> changing the fundamental architecture is a ship that has sailed.
>
> If you have more suggestions for simplification, redmine's are really
> welcome. Ohad has submitted a zillion of them and they've been really
> valuable.
>
> >
> > I think this would be easier to start with, and would support a workflow
> > that most users with small deployments have. Not everybody needs a proxy
> > and in this case it's only use is when there is a segmented network, for
> > remote execution.
>
> The Smart Proxy architecture is how Foreman talks to external services,
> and that's what ansible is. Abandoning that makes no sense, because
> that architecture works very well. It supports a very simple use case
> and very complex use cases from one code base.
>
> Are we supposed to start of with a core plugin and no proxy
> functionality and someday if a user wants a smart proxy they're SOL? Why
> not start with that from the beginning?
>
> We run into problems with developers not considering the bigger users of
> foreman all the time, and we had to make a lot of changes to support the
> idea of host isolation - that a host only ever talks to the proxy.
>
> We're in good shape now, although there's areas where the proxy can't do
> some things - like compute resources. That hurts us there -- because it
> may work for you, but it doesn't for many users that have to reach into
> other networks and talk to a vsphere behind a firewall, for example.
>
> Starting out with a plugin that rejects anybody but simple users of
> Foreman doesn't seem like the right thing to do.
>
> > # Ansible inventory
> >
> > This is not strictly 'foreman_ansible', but we'd make it much simpler
> > for Ansible people if we wrote a proper inventory that uses Foreman.
> >
> > Basically this means writing a python script that queries Foreman and
> > allows Ansible to understand Foreman hostgroups as Ansible group names.
> >
> > See an example (unmaintained I think) here
> > https://github.com/romeotheriault/foreman-ansible , it fetches Foreman
> > hostnames but it doesn't take hostgroups into account.
> >
> > It'd be awesome if we could land that in ansible/ansible contrib :)
> > https://github.com/ansible/ansible/tree/devel/contrib/inventory
> >
> > Aside from that, we could automate importing /etc/ansible/hosts to
> > Foreman as host groups. That should be trivial but I don't know whether
> > people want this or not.
> >
> > # Playbooks and modules importing
> >
> > Possibly we could import playbooks and modules as Job templates or
> > something like that. I didn't have a chance to get into this. In this
> > case we probably want a proxy to import playbooks in the system, I
> > agree.
> > I'd rather finish the rest before, but if you have ideas it'd be great
> > to have them in this thread.
> >
> > Best,
> >
> > --
> > Daniel Lobato Garcia
> >
> > @dLobatog
> > blog.daniellobato.me
> > daniellobato.me
> >
> > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> > Keybase: https://keybase.io/elobato
> >
> > --
> > You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Best Regards,
>
> Stephen Benjamin
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> Hi foreman devs,
>
> I spent some time the other day finding out how would we go about
> integrating Ansible, since it seems it has the potential to bring a ton
> of users to the Foreman ecosystem, and we could help them manage their
> systems.
>
> I HATE building castles in the skies, so for starters I'll publish
> the foreman_ansible facts parser soon, I hope this week.
>
> Here are my conclusions:
>
> # Plan
>
> - Finish FactParser and FactImporter
> - Write a task that imports facts for certain hosts, a host group,
> etc…
> - Write a callback plugin that when put on ansible.cfg posts facts to
> Foreman
>
> - (?) Parsing reports? Remote execution Ansible provider?
Looking forward to see this. I've been playing with that some time ago,
but didn't get to write a proper importer/parser for the facts.
What I did however, is using the callback plugin to save the ansible
events into files.
In the gist, you can see what data were are able to get from on ansible run
(report-1.json and report-2.json were produced by the very same playbook,
the first one is just obligatory call of setup, that ansible does automatically,
the second one is an actual task from my playbook).
The idea is to store the data from playbooks into files, and then import them
into the Foreman. The benefit of this is we can send all the data to
the foreman at once once the playbook finishes, while it's possible to
get the real-time data on demand when needed (similar approach to what we do with ssh).
>
> # Importing Facts and Reports (feasible)
>
> Currently facts in Ansible are gathered through ansible -m setup > $HOSTNAME. I've been writing a parser and it seems fairly trivial,
> FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
> 'work well'. There's a gotcha though, and it's that these facts are
> structured, so we should write a parser that creates these structured
> facts like in Chef and Salt.
>
> Initially, as an MVP we can either use a cronjob, or manually trigger
> the retrieval of facts in Foreman, just to prove it works. The way they
> should normally be submitted though, is by writing a callback plugin in
> Ansible. This callback plugin would submit reports and facts after
> completion of certain actions - see
> https://gist.github.com/cliffano/9868180 for an example. Callback
> plugins run automatically after configuring it in ansible.cfg.
+1
>
> We could simply make a POST call to api/v2/host/facts and
> api/v2/reports after runner_* actions.
>
> I have not played with importing reports at all nor I know the format of
> these in Ansible, the good news is we can do a good deal of data
> massaging if needed on the callback plugin.
>
> # Remote execution
>
> I think the obvious choice here is to rely on
> https://github.com/theforeman/foreman_remote_execution/
> I have a few problems with this approach, that I think we can fix on
> future releases of remote-execution:
>
> One-off ssh jobs now invocations are very complicated now even for a
> simple case.
>
> They require:
> (1) setting up a proxy, installing a plugin on the proxy,
> (2) adding the proxy in Foreman or refreshing features,
> (3) writing a job template (and understanding the job template form).
(1) and (2) are just an installation issues, and we already have an installer
patch to make that in one step (and can be enhanced further with the
scenarios). I know for an existing setup (as you already had) the steps
are more complicated, all this things are fixable (and other features
that rely on the proxy can benefit from that).
The (3) can be solved by seeding the templates: we already do so
for couple of actions in ssh (package actions, puppet run, service actions
or running an arbitrary command). There is nothing that prevents us
from seeding a template for running an arbitrary ansible command.
>
> I understand this complexity for 'enterprise' networks but not for
> small deployments. IMO I'd like to avoid this for Ansible, because
> it's harder to deploy, which will mean some users will get frustrated,
> and secondly because it's just not needed in a lot of cases.
Reasons for running the ansible from proxy from the day one, instead
of the effort to bundling the logic into Foreman main server:
unified approach for simple and complex scenarios - switching from
a simple all-in-one case, where the proxy is running on the same host,
as the main server, to more distributed one is easier, if you start
with a single proxy at the beinning. We prefer this unified approach,
instead of having two - one with bundled server, the second with proxies.
restarting the foreman server doesn't kill the ansible scripts that
are currently running
security - you can run the proxy as a different user, and configure
the sudoers/private keys for this user, while not giving too much
permissions to the foreman server itself. Also, the less Selinux policies
we need to allow for the Foreman itself the better:)
would it make sense to write a proxy in Python for ansible
to lower the entry barrier? (my guess is not, but still…
Anyway, I'm not saying it's not possible. I just believe that we're not
getting that much for the burder of having to maintain the logic twice, once
for the main server, on for proxy.
I would definitely not recommend to start tackling the with/without proxy
question as part of the ansible provider POC: I think we agree on need
to have the proxy variant: we can always revisit and make the proxy-less
approach possible. However, I don't think, now it's the right time for doing so.
We can however, make the user experience of starting to us Foreman remote
execution better, even with proxies there. There are too many question even
without trying to solve this.
>
> Ideally you should be able to go to any host, click on 'run job', it
> could open a modal, write an ansible command, and run it. Through
> foreman_ansible itself, no need for job templates, proxies, etc…
>
> I think this would be easier to start with, and would support a workflow
> that most users with small deployments have. Not everybody needs a proxy
> and in this case it's only use is when there is a segmented network, for
> remote execution.
Segmneted network is not the only use-case for this. Apart from the reasons
I've mentioned before, there is also load-balancing and HA.
But again, let's
open the question once the first steps towards ansible integration are made.
Especially let's try to make the installation better for the current state,
and revisit it in case it's still an issue later. To be honest, if I was
an Ansible user (and I am:), I would care much more about the features the Foreman
gives me than if there is one of two processes needed for doing so,
or if I need to do some extra steps as part of the setup.
>
> # Ansible inventory
>
> This is not strictly 'foreman_ansible', but we'd make it much simpler
> for Ansible people if we wrote a proper inventory that uses Foreman.
>
> Basically this means writing a python script that queries Foreman and
> allows Ansible to understand Foreman hostgroups as Ansible group names.
>
> See an example (unmaintained I think) here
> https://github.com/romeotheriault/foreman-ansible , it fetches Foreman
> hostnames but it doesn't take hostgroups into account.
>
> It'd be awesome if we could land that in ansible/ansible contrib
> https://github.com/ansible/ansible/tree/devel/contrib/inventory
>
> Aside from that, we could automate importing /etc/ansible/hosts to
> Foreman as host groups. That should be trivial but I don't know whether
> people want this or not.
Isn't this feasible by just running the ansible setup and importing
the facts into Foreman? And yes, we need it if we want to allow the
users to use their ansible inventory from the Foreman.
>
> # Playbooks and modules importing
>
> Possibly we could import playbooks and modules as Job templates or
> something like that. I didn't have a chance to get into this. In this
> case we probably want a proxy to import playbooks in the system, I
> agree.
> I'd rather finish the rest before, but if you have ideas it'd be great
> to have them in this thread.
I would be much more interested into this part - with ability
to import the playbooks (into job templates?) and invoking them from the Foreman
(via remote execution). We had a discussion about this already here
some time ago:
I would be most interested in the ability to pull inventory from foreman
with hostgroup and parameter information.
I have used foreman-ansible in the past but it was very limited in what it
could do. My use case for ansible is fairly limited though. I usually use
puppet for config management, salt/mco for one off orchestration commands,
and ansible for one off commands that needed more than a couple steps or
additional fact logic.
For example, for shell shock and heartbleed I didn't put a exec { 'yum update': } in puppet because I have hosts that don't have yum and I only
wanted a one time command to run (with logic to know if it needed to run at
all). I also had hosts that were only accessible with specific ssh keys
and/or via a proxy host. Because of some of those limitations it was
easiest to write a one off ansible playbook that contained my logic and
connection information (salt, mco, and puppet were not available).
For those use cases I don't need ansible integration in foreman but need
forman integration in ansible. I don't need playbooks imported into forman
and like to keep it separate because it's used for different purposes.
···
On Monday, October 26, 2015 at 11:08:45 AM UTC-7, Daniel Lobato wrote:
>
> Hi foreman devs,
>
> I spent some time the other day finding out how would we go about
> integrating Ansible, since it seems it has the potential to bring a ton
> of users to the Foreman ecosystem, and we could help them manage their
> systems.
>
> I *HATE* building castles in the skies, so for starters I'll publish
> the foreman_ansible facts parser soon, I hope this week.
>
> Here are my conclusions:
>
> # Plan
>
> - Finish FactParser and FactImporter
> - Write a task that imports facts for certain hosts, a host group,
> etc...
> - Write a callback plugin that when put on ansible.cfg posts facts to
> Foreman
>
> - (?) Parsing reports? Remote execution Ansible provider?
>
> # Importing Facts and Reports (feasible)
>
> Currently facts in Ansible are gathered through `ansible -m setup
> $HOSTNAME`. I've been writing a parser and it seems fairly trivial,
> FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
> 'work well'. There's a gotcha though, and it's that these facts are
> structured, so we should write a parser that creates these structured
> facts like in Chef and Salt.
>
> Initially, as an MVP we can either use a cronjob, or manually trigger
> the retrieval of facts in Foreman, just to prove it works. The way they
> should normally be submitted though, is by writing a callback plugin in
> Ansible. This callback plugin would submit reports and facts after
> completion of certain actions - see
> https://gist.github.com/cliffano/9868180 for an example. Callback
> plugins run automatically after configuring it in ansible.cfg.
>
> We could simply make a POST call to api/v2/host/facts and
> api/v2/reports after runner_* actions.
>
> I have not played with importing reports at all nor I know the format of
> these in Ansible, the good news is we can do a good deal of data
> massaging if needed on the callback plugin.
>
> # Remote execution
>
> I think the obvious choice here is to rely on
> https://github.com/theforeman/foreman_remote_execution/
> I have a few problems with this approach, that I think we can fix on
> future releases of remote-execution:
>
> One-off ssh jobs now invocations are very complicated now even for a
> simple case.
>
> They require:
> (1) setting up a proxy, installing a plugin on the proxy,
> (2) adding the proxy in Foreman or refreshing features,
> (3) writing a job template (and understanding the job template form).
>
> I understand this complexity for 'enterprise' networks but not for
> small deployments. IMO I'd like to avoid this for Ansible, because
> it's harder to deploy, which will mean some users will get frustrated,
> and secondly because it's just not needed in a lot of cases.
>
> Ideally you should be able to go to any host, click on 'run job', it
> could open a modal, write an ansible command, and run it. Through
> foreman_ansible itself, no need for job templates, proxies, etc...
>
> I think this would be easier to start with, and would support a workflow
> that most users with small deployments have. Not everybody needs a proxy
> and in this case it's only use is when there is a segmented network, for
> remote execution.
>
> # Ansible inventory
>
> This is not strictly 'foreman_ansible', but we'd make it much simpler
> for Ansible people if we wrote a proper inventory that uses Foreman.
>
> Basically this means writing a python script that queries Foreman and
> allows Ansible to understand Foreman hostgroups as Ansible group names.
>
> See an example (unmaintained I think) here
> https://github.com/romeotheriault/foreman-ansible , it fetches Foreman
> hostnames but it doesn't take hostgroups into account.
>
> It'd be awesome if we could land that in ansible/ansible contrib :)
> https://github.com/ansible/ansible/tree/devel/contrib/inventory
>
> Aside from that, we could automate importing /etc/ansible/hosts to
> Foreman as host groups. That should be trivial but I don't know whether
> people want this or not.
>
> # Playbooks and modules importing
>
> Possibly we could import playbooks and modules as Job templates or
> something like that. I didn't have a chance to get into this. In this
> case we probably want a proxy to import playbooks in the system, I
> agree.
> I'd rather finish the rest before, but if you have ideas it'd be great
> to have them in this thread.
>
> Best,
>
> --
> Daniel Lobato Garcia
>
> @dLobatog
> blog.daniellobato.me
> daniellobato.me
>
> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> Keybase: https://keybase.io/elobato
>
> The Smart Proxy architecture is how Foreman talks to external services,
> and that's what ansible is. Abandoning that makes no sense, because
> that architecture works very well. It supports a very simple use case
> and very complex use cases from one code base.
I agree, if there is any complication in the setup we should improve
this (installer, documentation, automation) rather that stepping
backwards.
I'll check out reports soon - for the moment, I made a working version
of the fact importer and parser (https://github.com/dlobatog/foreman_ansible)
, it's missing nested facts support and tests but you can play with it
···
On 10/26, Ivan Necas wrote:
>
> > We could simply make a POST call to api/v2/host/facts and
> > api/v2/reports after runner_* actions.
> >
> > I have not played with importing reports at all nor I know the format of
> > these in Ansible, the good news is we can do a good deal of data
> > massaging if needed on the callback plugin.
>
> see https://gist.github.com/iNecas/c502b5a9a6e6a801709f#file-report-1-json and
> https://gist.github.com/iNecas/c502b5a9a6e6a801709f#file-report-2-json for some
> examples.
Some updates - https://github.com/dlobatog/foreman_ansible , I improved
Ivan's callback plugin a bit so that reports are sent to
foreman_ansible, and if it's a report for the setup module, import the
facts.
Right now, we're at a point where writing the Report importer is kind of
trivial, at least Foreman will get the data easily. We just need to
figure out how to parse it
Other than that, we need nested facts support as some Ansible facts are
hashes. Having a good Ansible importer/reporter seems like an easy start
to support it and provide value to Ansible users.
···
On 10/26, Daniel Lobato Garcia wrote:
>
> # Plan
>
> - Finish FactParser and FactImporter
···
----- Original Message -----
> On 10/26, Ivan Necas wrote:
> >
> > > We could simply make a POST call to api/v2/host/facts and
> > > api/v2/reports after runner_* actions.
> > >
> > > I have not played with importing reports at all nor I know the format of
> > > these in Ansible, the good news is we can do a good deal of data
> > > massaging if needed on the callback plugin.
> >
> > see https://gist.github.com/iNecas/c502b5a9a6e6a801709f#file-report-1-json
> > and
> > https://gist.github.com/iNecas/c502b5a9a6e6a801709f#file-report-2-json for
> > some
> > examples.
>
> I'll check out reports soon - for the moment, I made a working version
> of the fact importer and parser (https://github.com/dlobatog/foreman_ansible)
> , it's missing nested facts support and tests but you can play with it
> :)
>
> --
> Daniel Lobato Garcia
>
> @dLobatog
> blog.daniellobato.me
> daniellobato.me
>
> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
> Keybase: https://keybase.io/elobato
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> I would be most interested in the ability to pull inventory from foreman
> with hostgroup and parameter information.
>
> I have used foreman-ansible in the past but it was very limited in what it
> could do. My use case for ansible is fairly limited though. I usually use
> puppet for config management, salt/mco for one off orchestration commands,
> and ansible for one off commands that needed more than a couple steps or
> additional fact logic.
>
> For example, for shell shock and heartbleed I didn't put a exec { 'yum > update': } in puppet because I have hosts that don't have yum and I only
> wanted a one time command to run (with logic to know if it needed to run at
> all). I also had hosts that were only accessible with specific ssh keys
> and/or via a proxy host. Because of some of those limitations it was
> easiest to write a one off ansible playbook that contained my logic and
> connection information (salt, mco, and puppet were not available).
>
> For those use cases I don't need ansible integration in foreman but need
> forman integration in ansible. I don't need playbooks imported into forman
> and like to keep it separate because it's used for different purposes.
>
Hi Justin
I completely agree, in terms of bang for buck this is the most desirable
bit of functionality that we would want too. Foreman is a rich CMDB and to
be able to talk about your infrastructure in the same terms within puppet
and ansible would be very very useful.
Charlie
···
On Wednesday, 28 October 2015 03:57:52 UTC, Justin Garrison wrote:
On Monday, October 26, 2015 at 11:08:45 AM UTC-7, Daniel Lobato wrote:
Hi foreman devs,
I spent some time the other day finding out how would we go about
integrating Ansible, since it seems it has the potential to bring a ton
of users to the Foreman ecosystem, and we could help them manage their
systems.
I HATE building castles in the skies, so for starters I’ll publish
the foreman_ansible facts parser soon, I hope this week.
Here are my conclusions:
Plan
Finish FactParser and FactImporter
Write a task that imports facts for certain hosts, a host group,
etc…
Write a callback plugin that when put on ansible.cfg posts facts to
Foreman
Currently facts in Ansible are gathered through ansible -m setup $HOSTNAME. I’ve been writing a parser and it seems fairly trivial,
FactImporter#add_new_facts, #delete_removed_facts and #update_facts all
’work well’. There’s a gotcha though, and it’s that these facts are
structured, so we should write a parser that creates these structured
facts like in Chef and Salt.
Initially, as an MVP we can either use a cronjob, or manually trigger
the retrieval of facts in Foreman, just to prove it works. The way they
should normally be submitted though, is by writing a callback plugin in
Ansible. This callback plugin would submit reports and facts after
completion of certain actions - see https://gist.github.com/cliffano/9868180 for an example. Callback
plugins run automatically after configuring it in ansible.cfg.
We could simply make a POST call to api/v2/host/facts and
api/v2/reports after runner_* actions.
I have not played with importing reports at all nor I know the format of
these in Ansible, the good news is we can do a good deal of data
massaging if needed on the callback plugin.
One-off ssh jobs now invocations are very complicated now even for a
simple case.
They require:
(1) setting up a proxy, installing a plugin on the proxy,
(2) adding the proxy in Foreman or refreshing features,
(3) writing a job template (and understanding the job template form).
I understand this complexity for ‘enterprise’ networks but not for
small deployments. IMO I’d like to avoid this for Ansible, because
it’s harder to deploy, which will mean some users will get frustrated,
and secondly because it’s just not needed in a lot of cases.
Ideally you should be able to go to any host, click on ‘run job’, it
could open a modal, write an ansible command, and run it. Through
foreman_ansible itself, no need for job templates, proxies, etc…
I think this would be easier to start with, and would support a workflow
that most users with small deployments have. Not everybody needs a proxy
and in this case it’s only use is when there is a segmented network, for
remote execution.
Ansible inventory
This is not strictly ‘foreman_ansible’, but we’d make it much simpler
for Ansible people if we wrote a proper inventory that uses Foreman.
Basically this means writing a python script that queries Foreman and
allows Ansible to understand Foreman hostgroups as Ansible group names.
Aside from that, we could automate importing /etc/ansible/hosts to
Foreman as host groups. That should be trivial but I don’t know whether
people want this or not.
Playbooks and modules importing
Possibly we could import playbooks and modules as Job templates or
something like that. I didn’t have a chance to get into this. In this
case we probably want a proxy to import playbooks in the system, I
agree.
I’d rather finish the rest before, but if you have ideas it’d be great
to have them in this thread.
