[Host unification] Introducing aspect-based design to Host model

As part of Host Unification feature team, I was working on a design that
will enable the host model being more pluggable and less monolitic than it
is today. I wanted to create a clear separation and ability to mix and
match different aspects of host management. I have identified that today
Host object consists of different aspect categories that are implemented
using different tools, I will try to give some examples for those
categories and their implementations:

  • Inventory
  • Bare metal
    • Virtualization
  • Provisioning
  • PXE
    • Image
  • Configuration
  • Puppet
    • Chef
  • ENC
  • Puppet
    • Salt
  • Content
  • Katello
  • Subscriptions
  • Katello

My vision of the system is a pluggable architecture that will enable the
user to choose which aspects of host management he wants to enable in his
Foreman installation, and how those aspects will be implemented. The user
should be able to say "I want to use puppet in my environment", so Foreman
will give this user the ability to manage his configuration and ENC via
puppet.

As a first step towards this vision I have decided to separate puppet out
of the monolitic host object. The goal was to identify the pain points of
such separation, and create mechanisms in the base framework that will
enable a relatively seemless aggregation of those aspects to te user.
This had led to huge design document:
http://projects.theforeman.org/projects/katello/wiki/HostUnification
And a PR: https://github.com/theforeman/foreman/pull/2555

It would be great to hear your feedback on this PR.
My main concern is usability of the framework from aspect programmer's
perspective. Of course any other feedback is more than welcome.
Feel free to challenge anything in this PR - I want to get as much feedback
as I can.
P.S. If something is not clear, I will gladly explain it.

Shimon

> - Inventory
> - Bare metal
> - Virtualization
> - Provisioning
> - PXE
> - Image
> - Configuration
> - Puppet
> - Chef
> - ENC
> - Puppet
> - Salt
> - Content
> - Katello
> - Subscriptions
> - Katello

  • Discovery (currently Host::Discovered STI)

> My vision of the system is a pluggable architecture that will enable the
> user to choose which aspects of host management he wants to enable in his
> Foreman installation, and how those aspects will be implemented. The user
> should be able to say "I want to use puppet in my environment", so Foreman
> will give this user the ability to manage his configuration and ENC via
> puppet.

Are we getting rid of Host::Base and Host::Managed STI?

Whatever is planned please make sure plugins can easily hook their own
aspects. A prototype for discovery plugin would be great.

··· -- Later, Lukas #lzap Zapletal

How would SCAP fit in here?
– bk

··· On 07/22/2015 04:57 AM, sshtein@redhat.com wrote: > > As part of Host Unification feature team, I was working on a design that > will enable the host model being more pluggable and less monolitic than > it is today. I wanted to create a clear separation and ability to mix > and match different aspects of host management. I have identified that > today Host object consists of different aspect categories that are > implemented using different tools, I will try to give some examples for > those categories and their implementations: > > * Inventory > o Bare metal > o Virtualization > * Provisioning > o PXE > o Image > * Configuration > o Puppet > o Chef > * ENC > o Puppet > o Salt > * Content > o Katello > * Subscriptions > o Katello

After a while, I am reviving this thread.

Things have changed a bit from the original proposal, but the basic idea is
the same.
Just to recap the changes:

  • New name has been chosen: Facets. The reason behind this change is that
    it is the name of design pattern that this change actually implements.
  • Facets list for host has become an in-memory only list that is exposed on
    Host::Managed object
  • Categorization of facets was dropped - no grouping of facets is
    implemented at this stage.

After having a face-to-face review with Daniel, we have decided to split
the monolithic PR into a small and more digestible pieces. Each of those
smaller PRs will have a separate subject.
I have thought about the following partition:

  1. Refactor host cloning procedure to become extensible.
  2. Facets registry mechanism - code that will be responsible for
    declaration of new facets.
  3. Host::Managed ORM changes - things that will enable access to the facet
    model, but without interfering into existing processes
  4. Host::Managed extension points - adding changes to different parts of
    the existing processes, so the facets would be able to add their
    functionality. Examples to such extension points: host views (single and
    list), template selection mechanism and cloning.

One main disadvantage that I can see in this approach, is that you would be
able to actually create facets and play with them only after all pieces of
this puzzle are merged.

Comments? Improvements? Am I missing something?

Shimon.

··· On Wednesday, July 22, 2015 at 11:58:00 AM UTC+3, ssh...@redhat.com wrote: > > > As part of Host Unification feature team, I was working on a design that > will enable the host model being more pluggable and less monolitic than it > is today. I wanted to create a clear separation and ability to mix and > match different aspects of host management. I have identified that today > Host object consists of different aspect categories that are implemented > using different tools, I will try to give some examples for those > categories and their implementations: > > - Inventory > - Bare metal > - Virtualization > - Provisioning > - PXE > - Image > - Configuration > - Puppet > - Chef > - ENC > - Puppet > - Salt > - Content > - Katello > - Subscriptions > - Katello > > My vision of the system is a pluggable architecture that will enable the > user to choose which aspects of host management he wants to enable in his > Foreman installation, and how those aspects will be implemented. The user > should be able to say "I want to use puppet in my environment", so Foreman > will give this user the ability to manage his configuration and ENC via > puppet. > > As a first step towards this vision I have decided to separate puppet out > of the monolitic host object. The goal was to identify the pain points of > such separation, and create mechanisms in the base framework that will > enable a relatively seemless aggregation of those aspects to te user. > This had led to huge design document: > http://projects.theforeman.org/projects/katello/wiki/HostUnification > And a PR: https://github.com/theforeman/foreman/pull/2555 > > It would be great to hear your feedback on this PR. > My main concern is usability of the framework from aspect programmer's > perspective. Of course any other feedback is more than welcome. > Feel free to challenge anything in this PR - I want to get as much > feedback as I can. > P.S. If something is not clear, I will gladly explain it. > > Shimon > >

I'll put a very-very high level design for the discovery plugin right here,
see if it makes sense to you…

Very high level design for discovery plugin from host unification
perspective:

models/discovery_attribute_set.rb will be renamed to discovery_aspect.rb
and will be the main model for interactions with discovery.
models/host/managed_extensions.rb will be the host concern added by the
plugin (renamed to discovery_aspect_extensions.rb)

models/host/discovered.rb will be separated to two: part will go into
discovery_aspect, and part will go into discovery_aspect_extensions

delegate should go - I don't think it will be used, since the aspect will
contain all this info

scoped_search will be moved to discovery_aspect_extensions (ref:
puppet_aspect_extensions.rb
<https://github.com/ShimShtein/foreman/blob/host-concerns/app/models/puppet_aspect.rb#L19>
)

import_host_and_facts will be removed - it was creating a host record, but
now the record will be created (ref: managed.rb
<https://github.com/ShimShtein/foreman/blob/host-concerns/app/models/host/managed.rb#L433>
)
import_facts will be removed - noop

populate_fields_from_facts will be moved to discovery_aspect.rb (ref:
puppet_aspect.rb
<https://github.com/ShimShtein/foreman/blob/host-concerns/app/models/puppet_aspect.rb#L114>
)

refresh_facts will be moved to discovery_aspect.rb
reboot will be moved to discovery_aspect.rb

The controllers will remain intact

views:
discovery_aspect should have a valid template.

··· On Thursday, July 23, 2015 at 2:00:28 PM UTC+3, Lukas Zapletal wrote: > > > - Inventory > > - Bare metal > > - Virtualization > > - Provisioning > > - PXE > > - Image > > - Configuration > > - Puppet > > - Chef > > - ENC > > - Puppet > > - Salt > > - Content > > - Katello > > - Subscriptions > > - Katello > > - Discovery (currently Host::Discovered STI) > > > My vision of the system is a pluggable architecture that will enable the > > user to choose which aspects of host management he wants to enable in > his > > Foreman installation, and how those aspects will be implemented. The > user > > should be able to say "I want to use puppet in my environment", so > Foreman > > will give this user the ability to manage his configuration and ENC via > > puppet. > > Are we getting rid of Host::Base and Host::Managed STI? > > Whatever is planned please make sure plugins can easily hook their own > aspects. A prototype for discovery plugin would be great. > > -- > Later, > Lukas #lzap Zapletal >

>
> How would SCAP fit in here?
> – bk
>

I can see at least two way for development of SCAP feature.

  1. It can become an independent aspect - the parsed SCAP data will be
    stored in an aspect, that will be shown in the host.
  2. I would really like to see SCAP taking advantage of (and extending) our
    existing facts/reports infrastructure. In this phase of the design I didn't
    touched that part of the host, but IMHO it would be better for the project
    to create an abstract concept of facts and reports that will be implemented
    by different providers, such as Puppet or SCAP. This design will require
    SCAP plugin to understand more deeply the format of the SCAP report XML,
    which can introduce more instability to the feature.
    Shlomi, your thoughts?

> Comments? Improvements? Am I missing something?

Overall, I like smaller PRs more.

I would like not to break discovery develop on core develop, therefore
I'd prefer if those smaller PRs do play nice with plugins that extends
host model.

··· -- Later, Lukas #lzap Zapletal

Fresh version of puppet-facet implementation is available at:
https://github.com/ShimShtein/foreman/tree/puppet-facet

··· On Monday, November 30, 2015 at 10:37:19 AM UTC+2, ssh...@redhat.com wrote: > > After a while, I am reviving this thread. > > Things have changed a bit from the original proposal, but the basic idea > is the same. > Just to recap the changes: > * New name has been chosen: Facets. The reason behind this change is that > it is the name of design pattern that this change actually implements. > * Facets list for host has become an in-memory only list that is exposed > on Host::Managed object > * Categorization of facets was dropped - no grouping of facets is > implemented at this stage. > > After having a face-to-face review with Daniel, we have decided to split > the monolithic PR into a small and more digestible pieces. Each of those > smaller PRs will have a separate subject. > I have thought about the following partition: > 1. Refactor host cloning procedure to become extensible. > 2. Facets registry mechanism - code that will be responsible for > declaration of new facets. > 3. Host::Managed ORM changes - things that will enable access to the facet > model, but without interfering into existing processes > 4. Host::Managed extension points - adding changes to different parts of > the existing processes, so the facets would be able to add their > functionality. Examples to such extension points: host views (single and > list), template selection mechanism and cloning. > > One main disadvantage that I can see in this approach, is that you would > be able to actually create facets and play with them only after all pieces > of this puzzle are merged. > > Comments? Improvements? Am I missing something? > > Shimon. > > > > On Wednesday, July 22, 2015 at 11:58:00 AM UTC+3, ssh...@redhat.com wrote: >> >> >> As part of Host Unification feature team, I was working on a design that >> will enable the host model being more pluggable and less monolitic than it >> is today. I wanted to create a clear separation and ability to mix and >> match different aspects of host management. I have identified that today >> Host object consists of different aspect categories that are implemented >> using different tools, I will try to give some examples for those >> categories and their implementations: >> >> - Inventory >> - Bare metal >> - Virtualization >> - Provisioning >> - PXE >> - Image >> - Configuration >> - Puppet >> - Chef >> - ENC >> - Puppet >> - Salt >> - Content >> - Katello >> - Subscriptions >> - Katello >> >> My vision of the system is a pluggable architecture that will enable the >> user to choose which aspects of host management he wants to enable in his >> Foreman installation, and how those aspects will be implemented. The user >> should be able to say "I want to use puppet in my environment", so Foreman >> will give this user the ability to manage his configuration and ENC via >> puppet. >> >> As a first step towards this vision I have decided to separate puppet out >> of the monolitic host object. The goal was to identify the pain points of >> such separation, and create mechanisms in the base framework that will >> enable a relatively seemless aggregation of those aspects to te user. >> This had led to huge design document: >> http://projects.theforeman.org/projects/katello/wiki/HostUnification >> And a PR: https://github.com/theforeman/foreman/pull/2555 >> >> It would be great to hear your feedback on this PR. >> My main concern is usability of the framework from aspect programmer's >> perspective. Of course any other feedback is more than welcome. >> Feel free to challenge anything in this PR - I want to get as much >> feedback as I can. >> P.S. If something is not clear, I will gladly explain it. >> >> Shimon >> >>

> Very high level design for discovery plugin from host unification
> perspective:

LGTM

··· -- Later, Lukas #lzap Zapletal

We are planning that SCAP Arf reports will be part of Foreman's reports
(via sti) and that it will appear in the host's status. the XML file report
will be available regardless.

··· 2015-08-13 11:34 GMT+03:00 :

How would SCAP fit in here?

– bk

I can see at least two way for development of SCAP feature.

  1. It can become an independent aspect - the parsed SCAP data will be
    stored in an aspect, that will be shown in the host.
  2. I would really like to see SCAP taking advantage of (and extending) our
    existing facts/reports infrastructure. In this phase of the design I didn’t
    touched that part of the host, but IMHO it would be better for the project
    to create an abstract concept of facts and reports that will be implemented
    by different providers, such as Puppet or SCAP. This design will require
    SCAP plugin to understand more deeply the format of the SCAP report XML,
    which can introduce more instability to the feature.
    Shlomi, your thoughts?


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.