Foreman and Chef integration

Support
1

Hello everybody,

Chef integration into Foreman is going forward.
We are currently trying to design how should recipes and roles (the
run_list) should be managed by Foreman.
Obsviously the design of chef-server vs puppet ENC is quite differente and
force us to rethink how Foreman should associate cookbooks to a node.

For what I see, problems are :

  • How to map current foreman objects (env, classes, hostgroups, smart
    variables, global parameters, etc …) to chef objects (env, recipes (easy
    one), roles, data bags)
  • As foreman can't be used directly with chef-client (as puppet does with
    ENC) : how do we sync state between Foreman and Chef-Server.

For the second part, we may have 2 proposal for now and are looking for new
ideas and remarks.

  • We could have a one way sync Foreman => Chef-server. We could configure
    nodes (add recipes, roles, etc) and when saving the host on Foreman, it
    sends node's modifications on chef-server.
    ** Advantages :
    => Foreman remains source of truth
    => easier to implement as it keep some internal logic that is already
    implemented
    => support and node definition complexity can be improved over time
    ** Drawbacks :
    => Users can't change anymore node state from knife or chef-webui

  • We could consider chef-server as source of truth for configuration
    management. Basicaly Foreman doesn't store the node state in DB.
    It would use chef-server like knife does. When he needs to display a node
    information he would do basicaly the same call as "knife node show" and
    when he needs to edit it, a 'knife node edit". We may envisage a local
    cache to minimize perf problems. This cache could be invalidated by a knife
    plugin when node edited directly from knife.

** Advantages :
=> We don't break chef-server management with knife CLI
=> We have are sure to be consistent with chef-server about node state.
** Drawbacks :
=> Poor performances when displaying / editing nodes as we need to wait
for chef-server API
=> Harder to implement (I think ?)

The goal would be note to re-implement the whole chef-server logic on
Foreman.
I hope I didn't forget anything.
Any idea how this should be handled ?

Regards,
Romain / fitzdsl

2

Hello,

I like the second approach more. You can learn from Katello codebase how
fetching of remote data from APIs is implemented there and I don't think
the performance makes big difference. Foreman UI does not need to scale
big and there is always chance to use some caching mechanism later on.

LZ

··· On Wed, Oct 30, 2013 at 09:35:33AM +0100, Romain Vrignaud wrote: > Hello everybody, > > Chef integration into Foreman is going forward. > We are currently trying to design how should recipes and roles (the > run_list) should be managed by Foreman. > Obsviously the design of chef-server vs puppet ENC is quite differente and > force us to rethink how Foreman should associate cookbooks to a node. > > For what I see, problems are : > * How to map current foreman objects (env, classes, hostgroups, smart > variables, global parameters, etc ..) to chef objects (env, recipes (easy > one), roles, data bags) > * As foreman can't be used directly with chef-client (as puppet does with > ENC) : how do we sync state between Foreman and Chef-Server. > > > For the second part, we may have 2 proposal for now and are looking for new > ideas and remarks. > > * We could have a one way sync Foreman => Chef-server. We could configure > nodes (add recipes, roles, etc) and when saving the host on Foreman, it > sends node's modifications on chef-server. > ** Advantages : > => Foreman remains source of truth > => easier to implement as it keep some internal logic that is already > implemented > => support and node definition complexity can be improved over time > ** Drawbacks : > => Users can't change anymore node state from knife or chef-webui > > > * We could consider chef-server as source of truth for configuration > management. Basicaly Foreman doesn't store the node state in DB. > It would use chef-server like knife does. When he needs to display a node > information he would do basicaly the same call as "knife node show" and > when he needs to edit it, a 'knife node edit". We may envisage a local > cache to minimize perf problems. This cache could be invalidated by a knife > plugin when node edited directly from knife. > > ** Advantages : > => We don't break chef-server management with knife CLI > => We have are sure to be consistent with chef-server about node state. > ** Drawbacks : > => Poor performances when displaying / editing nodes as we need to wait > for chef-server API > => Harder to implement (I think ?) > > The goal would be note to re-implement the whole chef-server logic on > Foreman. > I hope I didn't forget anything. > Any idea how this should be handled ? > > Regards, > Romain / fitzdsl > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.

–
Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

3

> Hello everybody,
>
> Chef integration into Foreman is going forward.
> We are currently trying to design how should recipes and roles (the
> run_list) should be managed by Foreman.
> Obsviously the design of chef-server vs puppet ENC is quite differente and
> force us to rethink how Foreman should associate cookbooks to a node.
>
> For what I see, problems are :
> * How to map current foreman objects (env, classes, hostgroups, smart
> variables, global parameters, etc …) to chef objects (env, recipes (easy
> one), roles, data bags)
> * As foreman can't be used directly with chef-client (as puppet does with
> ENC) : how do we sync state between Foreman and Chef-Server.
>
>
> For the second part, we may have 2 proposal for now and are looking for new
> ideas and remarks.
>
> * We could have a one way sync Foreman => Chef-server. We could configure
> nodes (add recipes, roles, etc) and when saving the host on Foreman, it
> sends node's modifications on chef-server.
> ** Advantages :
> => Foreman remains source of truth
> => easier to implement as it keep some internal logic that is already
> implemented
> => support and node definition complexity can be improved over time
> ** Drawbacks :
> => Users can't change anymore node state from knife or chef-webui

improve this later. Having foreman as a source of truth would work (and
actually be useful in some usecases) until foreman can handle more chef
features (roles n:m hosts, configuration versions, etc).

When we are at that point, we can add reading chef-server status (which could
be triggered after every knife command if needed).

Since foreman in many cases is used to provision hosts it's the first place
where host data are stored so it makes sense to me to have it as the only
place (aka source of truth).

> * We could consider chef-server as source of truth for configuration
> management. Basicaly Foreman doesn't store the node state in DB.
> It would use chef-server like knife does. When he needs to display a node
> information he would do basicaly the same call as "knife node show" and
> when he needs to edit it, a 'knife node edit". We may envisage a local
> cache to minimize perf problems. This cache could be invalidated by a knife
> plugin when node edited directly from knife.
>
> ** Advantages :
> => We don't break chef-server management with knife CLI
> => We have are sure to be consistent with chef-server about node state.
> ** Drawbacks :
> => Poor performances when displaying / editing nodes as we need to wait
> for chef-server API
> => Harder to implement (I think ?)

Would it mean in this scenario that foreman environments does not map to chef
environments? Otherwise how would we solve the reassign of an environment of
host in foreman? What would integration mean here? I see it as implementing
new chef webui as a foreman engine which does not make too much sense to me.

Also knife plugin would have to be a bit smarter since not only "knife node
edit" can change node details I think. It depends on what information would we
allow to display/modify in foreman.

>
> The goal would be note to re-implement the whole chef-server logic on
> Foreman.
> I hope I didn't forget anything.
> Any idea how this should be handled ?

I think the most important question is where do we want to get with chef
integration. From my point of view I'd like to use existing features in
foreman and use another underlying tool to accomplish configuration. So for
example if a host can have only one hostgroup I just want chef to respect it
(even if chef can provide more). Hopefully we'll improve hostgroups to match
chef roles functionality at some point or we'd take completely different
approach. If user wants to use other features from chef, he can, he just
shouldn't modify objects created by foreman.

Any comment appreciated.

··· On Wednesday 30 of October 2013 09:35:33 Romain Vrignaud wrote: From my point of view - this approach does not limit us in future so we can

–
Marek

4

I very much support using the chef-server as the source of truth.

Also, I would gladly help make sure direct API access isn't degrade perf.
chef-server-webui does all of its operations through the chef-server API,
so I'm inclined to believe that we can keep things snappy.

··· On Wednesday, October 30, 2013 8:35:33 AM UTC, Romain Vrignaud wrote: > > Hello everybody, > > Chef integration into Foreman is going forward. > We are currently trying to design how should recipes and roles (the > run_list) should be managed by Foreman. > Obsviously the design of chef-server vs puppet ENC is quite differente and > force us to rethink how Foreman should associate cookbooks to a node. > > For what I see, problems are : > * How to map current foreman objects (env, classes, hostgroups, smart > variables, global parameters, etc ..) to chef objects (env, recipes (easy > one), roles, data bags) > * As foreman can't be used directly with chef-client (as puppet does with > ENC) : how do we sync state between Foreman and Chef-Server. > > > For the second part, we may have 2 proposal for now and are looking for > new ideas and remarks. > > * We could have a one way sync Foreman => Chef-server. We could configure > nodes (add recipes, roles, etc) and when saving the host on Foreman, it > sends node's modifications on chef-server. > ** Advantages : > => Foreman remains source of truth > => easier to implement as it keep some internal logic that is already > implemented > => support and node definition complexity can be improved over time > ** Drawbacks : > => Users can't change anymore node state from knife or chef-webui > > > * We could consider chef-server as source of truth for configuration > management. Basicaly Foreman doesn't store the node state in DB. > It would use chef-server like knife does. When he needs to display a node > information he would do basicaly the same call as "knife node show" and > when he needs to edit it, a 'knife node edit". We may envisage a local > cache to minimize perf problems. This cache could be invalidated by a knife > plugin when node edited directly from knife. > > ** Advantages : > => We don't break chef-server management with knife CLI > => We have are sure to be consistent with chef-server about node state. > ** Drawbacks : > => Poor performances when displaying / editing nodes as we need to wait > for chef-server API > => Harder to implement (I think ?) > > The goal would be note to re-implement the whole chef-server logic on > Foreman. > I hope I didn't forget anything. > Any idea how this should be handled ? > > Regards, > Romain / fitzdsl > > > >
5

> Hello,
>
> I like the second approach more. You can learn from Katello codebase how
> fetching of remote data from APIs is implemented there and I don't think
> the performance makes big difference. Foreman UI does not need to scale
> big and there is always chance to use some caching mechanism later on.
We actually did find that fetching data for lists of systems was too
slow and we try to minimize it when possible. We pulled in all the
metadata about repositories into our db as a result of slowness and use
elastic search as our cache for all remote data for our list pages (with
the exception of our system list). You may have better luck than we
did, (and the chef server may have better response times than candlepin
or pulp), but I would do some testing to try to figure out if the apis
are flexible enough and how well they perform.

I would also keep in mind search here as that seems very important in
foreman (and is much harder with option #2).

-Justin

··· On 10/30/2013 07:49 AM, Lukas Zapletal wrote:

LZ

On Wed, Oct 30, 2013 at 09:35:33AM +0100, Romain Vrignaud wrote:

Hello everybody,

Chef integration into Foreman is going forward.
We are currently trying to design how should recipes and roles (the
run_list) should be managed by Foreman.
Obsviously the design of chef-server vs puppet ENC is quite differente and
force us to rethink how Foreman should associate cookbooks to a node.

For what I see, problems are :

  • How to map current foreman objects (env, classes, hostgroups, smart
    variables, global parameters, etc …) to chef objects (env, recipes (easy
    one), roles, data bags)
  • As foreman can’t be used directly with chef-client (as puppet does with
    ENC) : how do we sync state between Foreman and Chef-Server.

For the second part, we may have 2 proposal for now and are looking for new
ideas and remarks.

  • We could have a one way sync Foreman => Chef-server. We could configure
    nodes (add recipes, roles, etc) and when saving the host on Foreman, it
    sends node’s modifications on chef-server.
    ** Advantages :
    => Foreman remains source of truth
    => easier to implement as it keep some internal logic that is already
    implemented
    => support and node definition complexity can be improved over time
    ** Drawbacks :
    => Users can’t change anymore node state from knife or chef-webui

  • We could consider chef-server as source of truth for configuration
    management. Basicaly Foreman doesn’t store the node state in DB.
    It would use chef-server like knife does. When he needs to display a node
    information he would do basicaly the same call as “knife node show” and
    when he needs to edit it, a 'knife node edit". We may envisage a local
    cache to minimize perf problems. This cache could be invalidated by a knife
    plugin when node edited directly from knife.

** Advantages :
=> We don’t break chef-server management with knife CLI
=> We have are sure to be consistent with chef-server about node state.
** Drawbacks :
=> Poor performances when displaying / editing nodes as we need to wait
for chef-server API
=> Harder to implement (I think ?)

The goal would be note to re-implement the whole chef-server logic on
Foreman.
I hope I didn’t forget anything.
Any idea how this should be handled ?

Regards,
Romain / fitzdsl

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

> I would also keep in mind search here as that seems very important in
> foreman (and is much harder with option #2).

+1

··· -- Marek
7

Relooping foreman-users.

I agree with the fact that search is an important Foreman's feature.

> I think the most important question is where do we want to get with chef
> integration. From my point of view I'd like to use existing features in
> foreman and use another underlying tool to accomplish configuration. So for
> example if a host can have only one hostgroup I just want chef to respect
> it
> (even if chef can provide more).

I don't agree that we should align Foreman capabilities to the less
featureful configuration management system or that the CMS should be
limited by foreman.
Foreman community has done the maximum effort to support everything that
could be supported on puppet
(I'm thinking of parameterized class with ENC for instance), I think we
should do the same for every CMS
we are going to support.
IMHO as chef can handle multiple roles (mapped to hostgroup ?) we should
definitely support it.

> Hopefully we'll improve hostgroups to match
> chef roles functionality at some point or we'd take completely different
> approach. If user wants to use other features from chef, he can, he just
> shouldn't modify objects created by foreman.
>

Again I don't agree with that. chef users are used to manage chef-server
with knife. I think that this is a critical
tool for a lot of people and workflow and chef is fully designed to be
managed with knife.
If we prevent users to manage the nodes with knife, I think Foreman will
not be considered as a real and viable solution integrated to chef.

IMHO if we want to open to a wider community we need to accept their best
practices and adapt foreman to them rather than adapt chef
usage to Foreman.

Romain

8

So I've kinda been staying out of this, since as of a couple days ago,
I knew not that much about Chef, other than very high level
differences between Chef and Puppet. (The abstract design/philosophy
differences.)

My take right now is that OpsCode/Chef Inc has spent a great deal of
working refactoring Chef Server with their rewrite of Chef 11. They
basically got rid of CouchDB, took all the data/API handling out of
the Rails app and moved it to a separate set of Erlang daemons that
talk to a mandated PostgresSQL database.

Chef Server now consists of:

  • nginx: (:C) web terminus
  • chef-server-webui: (:Ruby on Rails) Web UI (just frontend, doesn't
    do any heavy lifting. interfaces with erchef)
  • erchef: (:erlang) A complete rewrite of the core API for the server,
    which allows it to be faster and more scalable than previous versions.
    The API itself is still compatible with the original Ruby-based
    server, which means that cookbooks and recipes that were authored for
    the Ruby-based server will continue to work on the Erlang-based
    server. The chef-client is still written in Ruby.
  • bookshelf: (:erlang) Bookshelf is used to store cookbook content
  • chef-expander: (ruby) process that updates solr indexes
  • Postgresql: (:C) database
  • rabbitmq: (:erlang) queuing daemon
  • chef-solr: - (:java) index of all data

I will note that the OSS WebUI, may be on the chopping block [1], and
a Chef Web UI may become a feature that only Enterprise and Hosted
Chef will support. As it is, "Organizations", "groups", "ACLs" and
"Reporting" are the first few features I noticed in Enterprise that
are missing from the OSS version, but there are likely others.

What does all this mean? I think it means that we leverage erchef as
the source of truth, and maybe chef-solr as the search backend. Where
I think we fit in, is basically a better looking and more feature rich
replacement for the standard UI, with fully integrated provisioning,
while still understanding that many people in the chef world have a
knife-only workflow, and never touch the Web UI. For these users we
would want to write a knife plugin that enables foreman based
provisioning.

As an aside, I am still two days into learning chef and may or may not
have more to say on this as I learn more.

I'll add that it is now dirt simple to install a Chef Server as they
have written a tool called Omnibus [2][3], that bundles and installs
all the daemons and dependencies in /opt and /var/opt. (The omni stuff
seems useful, and they have a number of repos available [4].)

e.g. - for Ubuntu:

  1. wget https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.10-1.ubuntu.12.04_amd64.deb
  2. sudo dpkg -i chef-server_11.0.10-1.ubuntu.12.04_amd64.deb
  3. sudo chef-server-ctl reconfigure

-Brian

[1] - https://github.com/opscode/omnibus-chef/pull/47
[2] - https://github.com/opscode/omnibus-ruby
[3] - https://github.com/opscode/omnibus-chef-server
[4] - https://github.com/opscode

··· On Tue, Dec 10, 2013 at 6:54 PM, Joseph Holsten wrote: > I very much support using the chef-server as the source of truth. > > Also, I would gladly help make sure direct API access isn't degrade perf. > chef-server-webui does all of its operations through the chef-server API, so > I'm inclined to believe that we can keep things snappy. > > > On Wednesday, October 30, 2013 8:35:33 AM UTC, Romain Vrignaud wrote: >> >> Hello everybody, >> >> Chef integration into Foreman is going forward. >> We are currently trying to design how should recipes and roles (the >> run_list) should be managed by Foreman. >> Obsviously the design of chef-server vs puppet ENC is quite differente and >> force us to rethink how Foreman should associate cookbooks to a node. >> >> For what I see, problems are : >> * How to map current foreman objects (env, classes, hostgroups, smart >> variables, global parameters, etc ..) to chef objects (env, recipes (easy >> one), roles, data bags) >> * As foreman can't be used directly with chef-client (as puppet does with >> ENC) : how do we sync state between Foreman and Chef-Server. >> >> >> For the second part, we may have 2 proposal for now and are looking for >> new ideas and remarks. >> >> * We could have a one way sync Foreman => Chef-server. We could configure >> nodes (add recipes, roles, etc) and when saving the host on Foreman, it >> sends node's modifications on chef-server. >> ** Advantages : >> => Foreman remains source of truth >> => easier to implement as it keep some internal logic that is already >> implemented >> => support and node definition complexity can be improved over time >> ** Drawbacks : >> => Users can't change anymore node state from knife or chef-webui >> >> >> * We could consider chef-server as source of truth for configuration >> management. Basicaly Foreman doesn't store the node state in DB. >> It would use chef-server like knife does. When he needs to display a node >> information he would do basicaly the same call as "knife node show" and when >> he needs to edit it, a 'knife node edit". We may envisage a local cache to >> minimize perf problems. This cache could be invalidated by a knife plugin >> when node edited directly from knife. >> >> ** Advantages : >> => We don't break chef-server management with knife CLI >> => We have are sure to be consistent with chef-server about node state. >> ** Drawbacks : >> => Poor performances when displaying / editing nodes as we need to wait >> for chef-server API >> => Harder to implement (I think ?) >> >> The goal would be note to re-implement the whole chef-server logic on >> Foreman. >> I hope I didn't forget anything. >> Any idea how this should be handled ? >> >> Regards, >> Romain / fitzdsl >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/groups/opt_out.
9

> Relooping foreman-users.
>
> I agree with the fact that search is an important Foreman's feature.
>
>
>> I think the most important question is where do we want to get with chef
>> integration. From my point of view I'd like to use existing features in
>> foreman and use another underlying tool to accomplish configuration. So
>> for
>> example if a host can have only one hostgroup I just want chef to respect
>> it
>> (even if chef can provide more).
>
>
> I don't agree that we should align Foreman capabilities to the less
> featureful configuration management system or that the CMS should be
> limited by foreman.
> Foreman community has done the maximum effort to support everything that
> could be supported on puppet
> (I'm thinking of parameterized class with ENC for instance), I think we
> should do the same for every CMS
> we are going to support.
> IMHO as chef can handle multiple roles (mapped to hostgroup ?) we should
> definitely support it.
>
>
>> Hopefully we'll improve hostgroups to match
>> chef roles functionality at some point or we'd take completely different
>> approach. If user wants to use other features from chef, he can, he just
>> shouldn't modify objects created by foreman.
>>
>
> Again I don't agree with that. chef users are used to manage chef-server
> with knife. I think that this is a critical
> tool for a lot of people and workflow and chef is fully designed to be
> managed with knife.
> If we prevent users to manage the nodes with knife, I think Foreman will
> not be considered as a real and viable solution integrated to chef.
>
> IMHO if we want to open to a wider community we need to accept their best
> practices and adapt foreman to them rather than adapt chef
> usage to Foreman.
>

How about we start with some real usage stories… e.g.

as A chef user, I would like to be able to assign | update | do something

later on, we can look into each and every usage case, and decide whats the
best way to implement it?

I think what I personally miss is the bigger picture / best practices
around chef, I'm not that interested of what is in foreman today that can
be augmented to work with Chef, rather what would be amazing with regard to
chef if we can support it.

Lets ignore for the sake of discussion the technical limitations.

Romain - thank you for pushing this!

Ohad

··· On Wed, Oct 30, 2013 at 5:15 PM, Romain Vrignaud wrote:

Romain

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/groups/opt_out.