[OFI] data model reusing Foreman's AR models

Petr_Chalupa · March 6, 2014, 4:54pm

Hi guys,

After yesterdays IRC talk we (Marek and Petr) started to look at OFI
models design. Unfortunately I haven't received any email with link to
that so we jumped in quite late. Anyway we've tried to decode as much
information as possible about staypuft model from etherpad 1. Here's
ER diagram we were able to construct from those information.

Our main concern is that we are recreating models that are already
present in Foreman. We understand Astapor issue that it was very hard
for user understand all Foreman specifics so now we try to do it
Openstack way and use Foreman as provisioning and configuration backend.
However we still think that we should built on top of existing and
tested models and just create a new easy to use and intuitive UI.

inecas: +1 on reusing the Foreman model: the foreman model seems to have
most of the things to support what is required, building something
custom means:

duplication of what is already there
writing throw-away code, that nobody except the OFI itself will
use: the power of Foreman is in the community: building it on the
Foreman model (and enhancing it when needed) means the same work will be
used outside of OFI with all the benefits that come with it
I would like to avoid doing that, especially when it doesn't seem that
much work to use the existing model

So what we propose is following DB schema:

Each deployment is contained under an Organization for separation. Which
also means that same Foreman instance could be normally used when user
leaves the wizard.

Left side is ER model right side is representing relationships between
Hostgroups instances. The one called 'Deploy' is lets say an common
parent Hostgroup for all other hostgroups in the deployment. 'Deploy'
allows sharing of configuration to the other hostgroups. Other
hostgroups are representing Roles (like controller, compute, etc.)

Basically we'd map:
OpenStackDeployment -> Organization
OpenStackLayout -> organization's parameter
OpenStackRole -> Hostgroup ()
OpenStackService -> N/A (at least for April)
OpenStackSeviceParam -> LookupKey (aka Smart Variable)
OpenStackDeployService -> N/A
OpenStackDeployServiceParam -> LookupKey (aka Smart Variable Value)
OpenStackDeployRole -> assignment of Host to Hostgroup ()

At the moment, the only thing that's missing is that Foreman 1.4.1 does
not allow us to set one hostgroup to be used in two parent hostgroups
(one service in two roles). This is going to change in 1.5 when
ConfigRoles will be merged. We suppose to use quickstack modules for
April version so we still have just puppet classes on role level (not on
service level) and therefore we won't allow user to customize services
in roles. Since we need this post April we can stick with current
Foreman model. Later we could either back-port ConfigRoles or base on
newer Foreman.

Benefits we see by reusing foreman models

we save work on need to model everything again
we save work on writing logic that will copy all (Deploy)Service/Role
(Deploy)ServiceParameters values to Hostgroup/LookupKey/LookupValue
ability to turn off the wizard plugin (something like removing the
training wheels) and keeping the Foreman instance still usable

Cons

people may spend some time to better understand existing Foreman code
inecas: that should be under Pros, and by people I don't mean the actual
users of OFI, but its developers

Potential concerns we understood from yesterday

too foreman specific
need of doing some business logic before Deploy is triggered
writing some easy UI around existing models is bigger pain that
creating new objects from scratch
to able to delay host provisioning until whole deployment is configured

Our answers to those

this is just about building new UI that we'll need anyway and
re-labeling (we already have Smart Variable which in fact is LookupKey
model), we could use STI or composition to modify Foreman objects
behavior if they are too restricting
It's easy to extend Dynflow process with a plugin injecting any
additional steps into the deployment process.
if this really becomes a problem we could always wrap existing models
by some other classes
If the hosts are kept in unmanaged state nothing gets applied to
them. Until orchestration takes over and switches them to managed state.

Are we sure that we need to allow user to prepare changes to an
existing-deployment-configuration and then apply them all at the same
time in April? I think we can postpone this for later and keep Foreman's
behavior for now that changes made to a hostgroup are immediately applied.

Looking forward to your feedback Petr and Marek.

Hugh_O_Brock · March 6, 2014, 7:14pm

> Hi guys,
>
> After yesterdays IRC talk we (Marek and Petr) started to look at OFI
> models design. Unfortunately I haven't received any email with link
> to that so we jumped in quite late. Anyway we've tried to decode as
> much information as possible about staypuft model from etherpad 1.
> Here's ER diagram we were able to construct from those information.

Hey Petr and Marek. First of all I really appreciate the effort you have
put into understanding the problem we're trying to solve and trying to
work out the best way to solve it with existing Foreman data
structures.

Unfortunately there are a couple of important bits of the model that you
missed that will make it considerably more difficult to implement using
stock Foreman data structures. (I think Scott has already outlined some
of these bits.) In particular, an OpenStackLayout is actually an entire
reference architecture (think HA vs. non-HA vs – later –
HA-with-separate-neutron-networker vs. HA-with-one-node-per-service)
that will map to a set of roles, which in turn contain 1…n
services. (This is a change we made from the original model he put
together). In this sense an OpenStackLayout is a kind of template that
we will, on deployment, realize by the creation and provisioning of
hostgroups. For this reason, I think there is quite substantial value in
modeling the layout as an entirely separate data structure from the
hostgroups that we will generate in the deploy stage.

For these reasons, I would like to press forward with Scott's original
approach for now. If we arrive post-April and we decide we want to
refactor to use more Foreman-centric data structures, let's revisit it
at that time.

Take care,
–Hugh

···

On Thu, Mar 06, 2014 at 05:54:08PM +0100, Petr Chalupa wrote:

Our main concern is that we are recreating models that are already
present in Foreman. We understand Astapor issue that it was very
hard for user understand all Foreman specifics so now we try to do
it Openstack way and use Foreman as provisioning and configuration
backend. However we still think that we should built on top of
existing and tested models and just create a new easy to use and
intuitive UI.

inecas: +1 on reusing the Foreman model: the foreman model seems to
have most of the things to support what is required, building
something custom means:

duplication of what is already there

writing throw-away code, that nobody except the OFI itself will
use: the power of Foreman is in the community: building it on the
Foreman model (and enhancing it when needed) means the same work
will be used outside of OFI with all the benefits that come with it
I would like to avoid doing that, especially when it doesn’t seem
that much work to use the existing model

So what we propose is following DB schema:

Each deployment is contained under an Organization for separation.
Which also means that same Foreman instance could be normally used
when user leaves the wizard.

Left side is ER model right side is representing relationships
between Hostgroups instances. The one called ‘Deploy’ is lets say an
common parent Hostgroup for all other hostgroups in the deployment.
‘Deploy’ allows sharing of configuration to the other hostgroups.
Other hostgroups are representing Roles (like controller, compute,
etc.)

Basically we’d map:
OpenStackDeployment -> Organization
OpenStackLayout -> organization’s parameter
OpenStackRole -> Hostgroup ()
OpenStackService -> N/A (at least for April)
OpenStackSeviceParam -> LookupKey (aka Smart Variable)
OpenStackDeployService -> N/A
OpenStackDeployServiceParam -> LookupKey (aka Smart Variable Value)
OpenStackDeployRole -> assignment of Host to Hostgroup ()

At the moment, the only thing that’s missing is that Foreman 1.4.1
does not allow us to set one hostgroup to be used in two parent
hostgroups (one service in two roles). This is going to change in
1.5 when ConfigRoles will be merged. We suppose to use quickstack
modules for April version so we still have just puppet classes on
role level (not on service level) and therefore we won’t allow user
to customize services in roles. Since we need this post April we can
stick with current Foreman model. Later we could either back-port
ConfigRoles or base on newer Foreman.

Benefits we see by reusing foreman models

we save work on need to model everything again

we save work on writing logic that will copy all
(Deploy)Service/Role (Deploy)ServiceParameters values to
Hostgroup/LookupKey/LookupValue

ability to turn off the wizard plugin (something like removing the
training wheels) and keeping the Foreman instance still usable

Cons

people may spend some time to better understand existing Foreman code
inecas: that should be under Pros, and by people I don’t mean the
actual users of OFI, but its developers

Potential concerns we understood from yesterday

too foreman specific

need of doing some business logic before Deploy is triggered

writing some easy UI around existing models is bigger pain that
creating new objects from scratch

to able to delay host provisioning until whole deployment is configured

Our answers to those

this is just about building new UI that we’ll need anyway and
re-labeling (we already have Smart Variable which in fact is
LookupKey model), we could use STI or composition to modify Foreman
objects behavior if they are too restricting

It’s easy to extend Dynflow process with a plugin injecting any
additional steps into the deployment process.

if this really becomes a problem we could always wrap existing
models by some other classes

If the hosts are kept in unmanaged state nothing gets applied to
them. Until orchestration takes over and switches them to managed
state.

Are we sure that we need to allow user to prepare changes to an
existing-deployment-configuration and then apply them all at the
same time in April? I think we can postpone this for later and keep
Foreman’s behavior for now that changes made to a hostgroup are
immediately applied.

Looking forward to your feedback Petr and Marek.

Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

–
== Hugh Brock, hbrock@redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==

“I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant.”
–Robert McCloskey

Marek_Hulan · March 6, 2014, 8:19pm

Hello,

we haven't mentioned layout as a specific entity yet because we think we don't
need it for first version. In fact we think we'll use quickstack modules,
so layouts are just simple groups of roles (for first version) and can be
hardcoded. But we would like to introduce a layout entity later, this is something
that foreman does not have yet.

···

-- Marek

----- Original Message -----

From: “Hugh O. Brock” hbrock@redhat.com
To: “Petr Chalupa” pchalupa@redhat.com
Cc: “foreman-dev” foreman-dev@googlegroups.com, rdo-list@redhat.com
Sent: Thursday, March 6, 2014 8:14:38 PM
Subject: [foreman-dev] Re: [Rdo-list] [OFI] data model reusing Foreman’s AR models

On Thu, Mar 06, 2014 at 05:54:08PM +0100, Petr Chalupa wrote:

Hi guys,

After yesterdays IRC talk we (Marek and Petr) started to look at OFI
models design. Unfortunately I haven’t received any email with link
to that so we jumped in quite late. Anyway we’ve tried to decode as
much information as possible about staypuft model from etherpad 1.
Here’s ER diagram we were able to construct from those information.

Hey Petr and Marek. First of all I really appreciate the effort you have
put into understanding the problem we’re trying to solve and trying to
work out the best way to solve it with existing Foreman data
structures.

Unfortunately there are a couple of important bits of the model that you
missed that will make it considerably more difficult to implement using
stock Foreman data structures. (I think Scott has already outlined some
of these bits.) In particular, an OpenStackLayout is actually an entire
reference architecture (think HA vs. non-HA vs – later –
HA-with-separate-neutron-networker vs. HA-with-one-node-per-service)
that will map to a set of roles, which in turn contain 1…n
services. (This is a change we made from the original model he put
together). In this sense an OpenStackLayout is a kind of template that
we will, on deployment, realize by the creation and provisioning of
hostgroups. For this reason, I think there is quite substantial value in
modeling the layout as an entirely separate data structure from the
hostgroups that we will generate in the deploy stage.

For these reasons, I would like to press forward with Scott’s original
approach for now. If we arrive post-April and we decide we want to
refactor to use more Foreman-centric data structures, let’s revisit it
at that time.

Take care,
–Hugh

Our main concern is that we are recreating models that are already
present in Foreman. We understand Astapor issue that it was very
hard for user understand all Foreman specifics so now we try to do
it Openstack way and use Foreman as provisioning and configuration
backend. However we still think that we should built on top of
existing and tested models and just create a new easy to use and
intuitive UI.

inecas: +1 on reusing the Foreman model: the foreman model seems to
have most of the things to support what is required, building
something custom means:

duplication of what is already there

writing throw-away code, that nobody except the OFI itself will
use: the power of Foreman is in the community: building it on the
Foreman model (and enhancing it when needed) means the same work
will be used outside of OFI with all the benefits that come with it
I would like to avoid doing that, especially when it doesn’t seem
that much work to use the existing model

So what we propose is following DB schema:

Each deployment is contained under an Organization for separation.
Which also means that same Foreman instance could be normally used
when user leaves the wizard.

Left side is ER model right side is representing relationships
between Hostgroups instances. The one called ‘Deploy’ is lets say an
common parent Hostgroup for all other hostgroups in the deployment.
‘Deploy’ allows sharing of configuration to the other hostgroups.
Other hostgroups are representing Roles (like controller, compute,
etc.)

Basically we’d map:
OpenStackDeployment -> Organization
OpenStackLayout -> organization’s parameter
OpenStackRole -> Hostgroup ()
OpenStackService -> N/A (at least for April)
OpenStackSeviceParam -> LookupKey (aka Smart Variable)
OpenStackDeployService -> N/A
OpenStackDeployServiceParam -> LookupKey (aka Smart Variable Value)
OpenStackDeployRole -> assignment of Host to Hostgroup ()

At the moment, the only thing that’s missing is that Foreman 1.4.1
does not allow us to set one hostgroup to be used in two parent
hostgroups (one service in two roles). This is going to change in
1.5 when ConfigRoles will be merged. We suppose to use quickstack
modules for April version so we still have just puppet classes on
role level (not on service level) and therefore we won’t allow user
to customize services in roles. Since we need this post April we can
stick with current Foreman model. Later we could either back-port
ConfigRoles or base on newer Foreman.

Benefits we see by reusing foreman models

we save work on need to model everything again

we save work on writing logic that will copy all
(Deploy)Service/Role (Deploy)ServiceParameters values to
Hostgroup/LookupKey/LookupValue

ability to turn off the wizard plugin (something like removing the
training wheels) and keeping the Foreman instance still usable

Cons

people may spend some time to better understand existing Foreman code
inecas: that should be under Pros, and by people I don’t mean the
actual users of OFI, but its developers

Potential concerns we understood from yesterday

too foreman specific

need of doing some business logic before Deploy is triggered

writing some easy UI around existing models is bigger pain that
creating new objects from scratch

to able to delay host provisioning until whole deployment is configured

Our answers to those

this is just about building new UI that we’ll need anyway and
re-labeling (we already have Smart Variable which in fact is
LookupKey model), we could use STI or composition to modify Foreman
objects behavior if they are too restricting

It’s easy to extend Dynflow process with a plugin injecting any
additional steps into the deployment process.

if this really becomes a problem we could always wrap existing
models by some other classes

If the hosts are kept in unmanaged state nothing gets applied to
them. Until orchestration takes over and switches them to managed
state.

Are we sure that we need to allow user to prepare changes to an
existing-deployment-configuration and then apply them all at the
same time in April? I think we can postpone this for later and keep
Foreman’s behavior for now that changes made to a hostgroup are
immediately applied.

Looking forward to your feedback Petr and Marek.

Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

–
== Hugh Brock, hbrock@redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==

“I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant.”
–Robert McCloskey

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hugh_O_Brock · March 6, 2014, 8:46pm

> Hello,
>
> we haven't mentioned layout as a specific entity yet because we think we don't
> need it for first version. In fact we think we'll use quickstack modules,
> so layouts are just simple groups of roles (for first version) and can be
> hardcoded. But we would like to introduce a layout entity later, this is something
> that foreman does not have yet.

Hmm… I have to tell you guys, I'm really skeptical about changing the
direction. I like the idea of a plugin with OpenStack-specific models
that sits on top of the Foreman data structures. While I'm all for doing
something that is reusable later in Foreman, I have a feeling we're
talking about another week of work to get that modeling done… on a
plugin that I want to be largely code complete in two weeks.

I've asked Scott to go ahead and start implementing the design we've
already worked out. If you guys can show me how we can get to the same
point he's aiming for using a different model, in two weeks, then I'm
willing to look at it… but even then I'm not sure it makes
sense. Moreover I can't afford to wait for anything here, right now,
that is in any way tied to a future feature in Foreman. We have to work
with what we have right now.

If you guys want I'm happy to talk further on this tomorrow morning, let
me know.

Take care,
–Hugh

···

On Thu, Mar 06, 2014 at 03:19:30PM -0500, Marek Hulan wrote:

----- Original Message -----

From: “Hugh O. Brock” hbrock@redhat.com
To: “Petr Chalupa” pchalupa@redhat.com
Cc: “foreman-dev” foreman-dev@googlegroups.com, rdo-list@redhat.com
Sent: Thursday, March 6, 2014 8:14:38 PM
Subject: [foreman-dev] Re: [Rdo-list] [OFI] data model reusing Foreman’s AR models

On Thu, Mar 06, 2014 at 05:54:08PM +0100, Petr Chalupa wrote:

Hi guys,

After yesterdays IRC talk we (Marek and Petr) started to look at OFI
models design. Unfortunately I haven’t received any email with link
to that so we jumped in quite late. Anyway we’ve tried to decode as
much information as possible about staypuft model from etherpad 1.
Here’s ER diagram we were able to construct from those information.

Hey Petr and Marek. First of all I really appreciate the effort you have
put into understanding the problem we’re trying to solve and trying to
work out the best way to solve it with existing Foreman data
structures.

Unfortunately there are a couple of important bits of the model that you
missed that will make it considerably more difficult to implement using
stock Foreman data structures. (I think Scott has already outlined some
of these bits.) In particular, an OpenStackLayout is actually an entire
reference architecture (think HA vs. non-HA vs – later –
HA-with-separate-neutron-networker vs. HA-with-one-node-per-service)
that will map to a set of roles, which in turn contain 1…n
services. (This is a change we made from the original model he put
together). In this sense an OpenStackLayout is a kind of template that
we will, on deployment, realize by the creation and provisioning of
hostgroups. For this reason, I think there is quite substantial value in
modeling the layout as an entirely separate data structure from the
hostgroups that we will generate in the deploy stage.

For these reasons, I would like to press forward with Scott’s original
approach for now. If we arrive post-April and we decide we want to
refactor to use more Foreman-centric data structures, let’s revisit it
at that time.

Take care,
–Hugh

Our main concern is that we are recreating models that are already
present in Foreman. We understand Astapor issue that it was very
hard for user understand all Foreman specifics so now we try to do
it Openstack way and use Foreman as provisioning and configuration
backend. However we still think that we should built on top of
existing and tested models and just create a new easy to use and
intuitive UI.

inecas: +1 on reusing the Foreman model: the foreman model seems to
have most of the things to support what is required, building
something custom means:

duplication of what is already there

writing throw-away code, that nobody except the OFI itself will
use: the power of Foreman is in the community: building it on the
Foreman model (and enhancing it when needed) means the same work
will be used outside of OFI with all the benefits that come with it
I would like to avoid doing that, especially when it doesn’t seem
that much work to use the existing model

So what we propose is following DB schema:

Each deployment is contained under an Organization for separation.
Which also means that same Foreman instance could be normally used
when user leaves the wizard.

Left side is ER model right side is representing relationships
between Hostgroups instances. The one called ‘Deploy’ is lets say an
common parent Hostgroup for all other hostgroups in the deployment.
‘Deploy’ allows sharing of configuration to the other hostgroups.
Other hostgroups are representing Roles (like controller, compute,
etc.)

Basically we’d map:
OpenStackDeployment -> Organization
OpenStackLayout -> organization’s parameter
OpenStackRole -> Hostgroup ()
OpenStackService -> N/A (at least for April)
OpenStackSeviceParam -> LookupKey (aka Smart Variable)
OpenStackDeployService -> N/A
OpenStackDeployServiceParam -> LookupKey (aka Smart Variable Value)
OpenStackDeployRole -> assignment of Host to Hostgroup ()

At the moment, the only thing that’s missing is that Foreman 1.4.1
does not allow us to set one hostgroup to be used in two parent
hostgroups (one service in two roles). This is going to change in
1.5 when ConfigRoles will be merged. We suppose to use quickstack
modules for April version so we still have just puppet classes on
role level (not on service level) and therefore we won’t allow user
to customize services in roles. Since we need this post April we can
stick with current Foreman model. Later we could either back-port
ConfigRoles or base on newer Foreman.

Benefits we see by reusing foreman models

we save work on need to model everything again

we save work on writing logic that will copy all
(Deploy)Service/Role (Deploy)ServiceParameters values to
Hostgroup/LookupKey/LookupValue

ability to turn off the wizard plugin (something like removing the
training wheels) and keeping the Foreman instance still usable

Cons

people may spend some time to better understand existing Foreman code
inecas: that should be under Pros, and by people I don’t mean the
actual users of OFI, but its developers

Potential concerns we understood from yesterday

too foreman specific

need of doing some business logic before Deploy is triggered

writing some easy UI around existing models is bigger pain that
creating new objects from scratch

to able to delay host provisioning until whole deployment is configured

Our answers to those

this is just about building new UI that we’ll need anyway and
re-labeling (we already have Smart Variable which in fact is
LookupKey model), we could use STI or composition to modify Foreman
objects behavior if they are too restricting

It’s easy to extend Dynflow process with a plugin injecting any
additional steps into the deployment process.

if this really becomes a problem we could always wrap existing
models by some other classes

If the hosts are kept in unmanaged state nothing gets applied to
them. Until orchestration takes over and switches them to managed
state.

Are we sure that we need to allow user to prepare changes to an
existing-deployment-configuration and then apply them all at the
same time in April? I think we can postpone this for later and keep
Foreman’s behavior for now that changes made to a hostgroup are
immediately applied.

Looking forward to your feedback Petr and Marek.

Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

–
== Hugh Brock, hbrock@redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==

“I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant.”
–Robert McCloskey

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
== Hugh Brock, hbrock@redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==

“I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant.”
–Robert McCloskey

Petr_Chalupa · March 6, 2014, 9:24pm

>> Hello,
>>
>> we haven't mentioned layout as a specific entity yet because we think we don't
>> need it for first version. In fact we think we'll use quickstack modules,
>> so layouts are just simple groups of roles (for first version) and can be
>> hardcoded. But we would like to introduce a layout entity later, this is something
>> that foreman does not have yet.
>
> Hmm… I have to tell you guys, I'm really skeptical about changing the
> direction. I like the idea of a plugin with OpenStack-specific models
> that sits on top of the Foreman data structures. While I'm all for doing
> something that is reusable later in Foreman, I have a feeling we're
> talking about another week of work to get that modeling done… on a
> plugin that I want to be largely code complete in two weeks.
>
> I've asked Scott to go ahead and start implementing the design we've
> already worked out. If you guys can show me how we can get to the same
> point he's aiming for using a different model, in two weeks, then I'm
> willing to look at it… but even then I'm not sure it makes
> sense. Moreover I can't afford to wait for anything here, right now,
> that is in any way tied to a future feature in Foreman. We have to work
> with what we have right now.
>
> If you guys want I'm happy to talk further on this tomorrow morning, let
> me know.
>
> Take care,
> --Hugh

Our main motivation is not to things reusable later by Foreman, but we
think we could actually save some time. Re-usability is just nice
side-effect. If Scott thinks there's no benefit of using existing
Foreman models we don't insist on our design for sure. We've already
finished answers for Scott's comments so we are sending in parallel,
we'll try to make this discussion as short as possible.

Petr

···

On 06.03.14 21:46, Hugh O. Brock wrote: > On Thu, Mar 06, 2014 at 03:19:30PM -0500, Marek Hulan wrote:

----- Original Message -----

From: “Hugh O. Brock” hbrock@redhat.com
To: “Petr Chalupa” pchalupa@redhat.com
Cc: “foreman-dev” foreman-dev@googlegroups.com, rdo-list@redhat.com
Sent: Thursday, March 6, 2014 8:14:38 PM
Subject: [foreman-dev] Re: [Rdo-list] [OFI] data model reusing Foreman’s AR models

On Thu, Mar 06, 2014 at 05:54:08PM +0100, Petr Chalupa wrote:

Hi guys,

After yesterdays IRC talk we (Marek and Petr) started to look at OFI
models design. Unfortunately I haven’t received any email with link
to that so we jumped in quite late. Anyway we’ve tried to decode as
much information as possible about staypuft model from etherpad 1.
Here’s ER diagram we were able to construct from those information.

Hey Petr and Marek. First of all I really appreciate the effort you have
put into understanding the problem we’re trying to solve and trying to
work out the best way to solve it with existing Foreman data
structures.

Unfortunately there are a couple of important bits of the model that you
missed that will make it considerably more difficult to implement using
stock Foreman data structures. (I think Scott has already outlined some
of these bits.) In particular, an OpenStackLayout is actually an entire
reference architecture (think HA vs. non-HA vs – later –
HA-with-separate-neutron-networker vs. HA-with-one-node-per-service)
that will map to a set of roles, which in turn contain 1…n
services. (This is a change we made from the original model he put
together). In this sense an OpenStackLayout is a kind of template that
we will, on deployment, realize by the creation and provisioning of
hostgroups. For this reason, I think there is quite substantial value in
modeling the layout as an entirely separate data structure from the
hostgroups that we will generate in the deploy stage.

For these reasons, I would like to press forward with Scott’s original
approach for now. If we arrive post-April and we decide we want to
refactor to use more Foreman-centric data structures, let’s revisit it
at that time.

Take care,
–Hugh

Our main concern is that we are recreating models that are already
present in Foreman. We understand Astapor issue that it was very
hard for user understand all Foreman specifics so now we try to do
it Openstack way and use Foreman as provisioning and configuration
backend. However we still think that we should built on top of
existing and tested models and just create a new easy to use and
intuitive UI.

inecas: +1 on reusing the Foreman model: the foreman model seems to
have most of the things to support what is required, building
something custom means:

duplication of what is already there

writing throw-away code, that nobody except the OFI itself will
use: the power of Foreman is in the community: building it on the
Foreman model (and enhancing it when needed) means the same work
will be used outside of OFI with all the benefits that come with it
I would like to avoid doing that, especially when it doesn’t seem
that much work to use the existing model

So what we propose is following DB schema:

Each deployment is contained under an Organization for separation.
Which also means that same Foreman instance could be normally used
when user leaves the wizard.

Left side is ER model right side is representing relationships
between Hostgroups instances. The one called ‘Deploy’ is lets say an
common parent Hostgroup for all other hostgroups in the deployment.
‘Deploy’ allows sharing of configuration to the other hostgroups.
Other hostgroups are representing Roles (like controller, compute,
etc.)

Basically we’d map:
OpenStackDeployment -> Organization
OpenStackLayout -> organization’s parameter
OpenStackRole -> Hostgroup ()
OpenStackService -> N/A (at least for April)
OpenStackSeviceParam -> LookupKey (aka Smart Variable)
OpenStackDeployService -> N/A
OpenStackDeployServiceParam -> LookupKey (aka Smart Variable Value)
OpenStackDeployRole -> assignment of Host to Hostgroup ()

At the moment, the only thing that’s missing is that Foreman 1.4.1
does not allow us to set one hostgroup to be used in two parent
hostgroups (one service in two roles). This is going to change in
1.5 when ConfigRoles will be merged. We suppose to use quickstack
modules for April version so we still have just puppet classes on
role level (not on service level) and therefore we won’t allow user
to customize services in roles. Since we need this post April we can
stick with current Foreman model. Later we could either back-port
ConfigRoles or base on newer Foreman.

Benefits we see by reusing foreman models

we save work on need to model everything again

we save work on writing logic that will copy all
(Deploy)Service/Role (Deploy)ServiceParameters values to
Hostgroup/LookupKey/LookupValue

ability to turn off the wizard plugin (something like removing the
training wheels) and keeping the Foreman instance still usable

Cons

people may spend some time to better understand existing Foreman code
inecas: that should be under Pros, and by people I don’t mean the
actual users of OFI, but its developers

Potential concerns we understood from yesterday

too foreman specific

need of doing some business logic before Deploy is triggered

writing some easy UI around existing models is bigger pain that
creating new objects from scratch

to able to delay host provisioning until whole deployment is configured

Our answers to those

this is just about building new UI that we’ll need anyway and
re-labeling (we already have Smart Variable which in fact is
LookupKey model), we could use STI or composition to modify Foreman
objects behavior if they are too restricting

It’s easy to extend Dynflow process with a plugin injecting any
additional steps into the deployment process.

if this really becomes a problem we could always wrap existing
models by some other classes

If the hosts are kept in unmanaged state nothing gets applied to
them. Until orchestration takes over and switches them to managed
state.

Are we sure that we need to allow user to prepare changes to an
existing-deployment-configuration and then apply them all at the
same time in April? I think we can postpone this for later and keep
Foreman’s behavior for now that changes made to a hostgroup are
immediately applied.

Looking forward to your feedback Petr and Marek.

Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

–
== Hugh Brock, hbrock@redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==

“I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant.”
–Robert McCloskey

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

iNecas · March 7, 2014, 8:55am

> > Hi guys,
> >
> > After yesterdays IRC talk we (Marek and Petr) started to look at OFI
> > models design. Unfortunately I haven't received any email with link
> > to that so we jumped in quite late. Anyway we've tried to decode as
> > much information as possible about staypuft model from etherpad 1.
> > Here's ER diagram we were able to construct from those information.
>
> Hey Petr and Marek. First of all I really appreciate the effort you have
> put into understanding the problem we're trying to solve and trying to
> work out the best way to solve it with existing Foreman data
> structures.
>
> Unfortunately there are a couple of important bits of the model that you
> missed that will make it considerably more difficult to implement using
> stock Foreman data structures. (I think Scott has already outlined some
> of these bits.) In particular, an OpenStackLayout is actually an entire
> reference architecture (think HA vs. non-HA vs – later –
> HA-with-separate-neutron-networker vs. HA-with-one-node-per-service)
> that will map to a set of roles, which in turn contain 1…n
> services. (This is a change we made from the original model he put
> together). In this sense an OpenStackLayout is a kind of template that
> we will, on deployment, realize by the creation and provisioning of
> hostgroups. For this reason, I think there is quite substantial value in
> modeling the layout as an entirely separate data structure from the
> hostgroups that we will generate in the deploy stage.
>
> For these reasons, I would like to press forward with Scott's original
> approach for now. If we arrive post-April and we decide we want to
> refactor to use more Foreman-centric data structures, let's revisit it
> at that time.

I'm supporting the approach of reusing the Foreman modeles where it makes
sense (and a lot of things in the original model can be IMO modeled quite
easily with what's already done in Foreman) because, even if it might not
look like that, it should be less work since all of it is already being
used in production environment.

I understand the need for the setup and realization phase being separated,
but introducing new models is just one way of solving that. I have no big
issues with Openstack Roles and Services for the purposes of the wizard and they
could really speed up its creation.

However, having the Deployment Roles and Deployment Services and Deployment Service
Params seems quite odd to me, as I don't see added value in those models
as opposed to hostgroups, global and smart parameters. But it might be just
lack my understanding and I will be happy for somebody bringing more
more lights on the benefits of those.

On the other hand, using the things we already have in Foreman
for the deployment phase allows working on the orchestration without having
the wizard ready to be used (if we agree what the artifacts of the wizard will be).

Anyway, I would encourage looking into the Foreman advances features around
smart parameters, organizations etc. before modeling those things from scratch.

Just my $0.02

– Ivan

···

----- Original Message ----- > On Thu, Mar 06, 2014 at 05:54:08PM +0100, Petr Chalupa wrote:

Take care,
–Hugh

Our main concern is that we are recreating models that are already
present in Foreman. We understand Astapor issue that it was very
hard for user understand all Foreman specifics so now we try to do
it Openstack way and use Foreman as provisioning and configuration
backend. However we still think that we should built on top of
existing and tested models and just create a new easy to use and
intuitive UI.

inecas: +1 on reusing the Foreman model: the foreman model seems to
have most of the things to support what is required, building
something custom means:

duplication of what is already there

writing throw-away code, that nobody except the OFI itself will
use: the power of Foreman is in the community: building it on the
Foreman model (and enhancing it when needed) means the same work
will be used outside of OFI with all the benefits that come with it
I would like to avoid doing that, especially when it doesn’t seem
that much work to use the existing model

So what we propose is following DB schema:

Each deployment is contained under an Organization for separation.
Which also means that same Foreman instance could be normally used
when user leaves the wizard.

Left side is ER model right side is representing relationships
between Hostgroups instances. The one called ‘Deploy’ is lets say an
common parent Hostgroup for all other hostgroups in the deployment.
‘Deploy’ allows sharing of configuration to the other hostgroups.
Other hostgroups are representing Roles (like controller, compute,
etc.)

Basically we’d map:
OpenStackDeployment -> Organization
OpenStackLayout -> organization’s parameter
OpenStackRole -> Hostgroup ()
OpenStackService -> N/A (at least for April)
OpenStackSeviceParam -> LookupKey (aka Smart Variable)
OpenStackDeployService -> N/A
OpenStackDeployServiceParam -> LookupKey (aka Smart Variable Value)
OpenStackDeployRole -> assignment of Host to Hostgroup ()

At the moment, the only thing that’s missing is that Foreman 1.4.1
does not allow us to set one hostgroup to be used in two parent
hostgroups (one service in two roles). This is going to change in
1.5 when ConfigRoles will be merged. We suppose to use quickstack
modules for April version so we still have just puppet classes on
role level (not on service level) and therefore we won’t allow user
to customize services in roles. Since we need this post April we can
stick with current Foreman model. Later we could either back-port
ConfigRoles or base on newer Foreman.

Benefits we see by reusing foreman models

we save work on need to model everything again

we save work on writing logic that will copy all
(Deploy)Service/Role (Deploy)ServiceParameters values to
Hostgroup/LookupKey/LookupValue

ability to turn off the wizard plugin (something like removing the
training wheels) and keeping the Foreman instance still usable

Cons

people may spend some time to better understand existing Foreman code
inecas: that should be under Pros, and by people I don’t mean the
actual users of OFI, but its developers

Potential concerns we understood from yesterday

too foreman specific

need of doing some business logic before Deploy is triggered

writing some easy UI around existing models is bigger pain that
creating new objects from scratch

to able to delay host provisioning until whole deployment is configured

Our answers to those

this is just about building new UI that we’ll need anyway and
re-labeling (we already have Smart Variable which in fact is
LookupKey model), we could use STI or composition to modify Foreman
objects behavior if they are too restricting

It’s easy to extend Dynflow process with a plugin injecting any
additional steps into the deployment process.

if this really becomes a problem we could always wrap existing
models by some other classes

If the hosts are kept in unmanaged state nothing gets applied to
them. Until orchestration takes over and switches them to managed
state.

Are we sure that we need to allow user to prepare changes to an
existing-deployment-configuration and then apply them all at the
same time in April? I think we can postpone this for later and keep
Foreman’s behavior for now that changes made to a hostgroup are
immediately applied.

Looking forward to your feedback Petr and Marek.

Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

–
== Hugh Brock, hbrock@redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==

“I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant.”
–Robert McCloskey

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dLobatog · March 7, 2014, 9:15am

I'd have to vouch for building on Foreman models too. It seems like there
are already many responsibilities that Foreman models cover already and are
easily mapped. Moreover, the work of finding the equivalences with the
Openstack models has already more or less been done from what I read above
too. I think we're basically going to rewrite a large portion of what
Foreman already provides if we start using many new models.
Also we get proven integration with Dynflow for free, if it gets hairy we
can use composition as stated on the first post.

Another important reason is I think development pace should significantly
increase since a portion of the team already knows and has worked with
these models, and we can explain to newcomers how to use them to build
Staypuft.

···

On Fri, Mar 7, 2014 at 9:55 AM, Ivan Necas wrote:

----- Original Message -----

On Thu, Mar 06, 2014 at 05:54:08PM +0100, Petr Chalupa wrote:

Hi guys,

After yesterdays IRC talk we (Marek and Petr) started to look at OFI
models design. Unfortunately I haven’t received any email with link
to that so we jumped in quite late. Anyway we’ve tried to decode as
much information as possible about staypuft model from etherpad 1.
Here’s ER diagram we were able to construct from those information.

Hey Petr and Marek. First of all I really appreciate the effort you have
put into understanding the problem we’re trying to solve and trying to
work out the best way to solve it with existing Foreman data
structures.

Unfortunately there are a couple of important bits of the model that you
missed that will make it considerably more difficult to implement using
stock Foreman data structures. (I think Scott has already outlined some
of these bits.) In particular, an OpenStackLayout is actually an entire
reference architecture (think HA vs. non-HA vs – later –
HA-with-separate-neutron-networker vs. HA-with-one-node-per-service)
that will map to a set of roles, which in turn contain 1…n
services. (This is a change we made from the original model he put
together). In this sense an OpenStackLayout is a kind of template that
we will, on deployment, realize by the creation and provisioning of
hostgroups. For this reason, I think there is quite substantial value in
modeling the layout as an entirely separate data structure from the
hostgroups that we will generate in the deploy stage.

For these reasons, I would like to press forward with Scott’s original
approach for now. If we arrive post-April and we decide we want to
refactor to use more Foreman-centric data structures, let’s revisit it
at that time.

I’m supporting the approach of reusing the Foreman modeles where it makes
sense (and a lot of things in the original model can be IMO modeled quite
easily with what’s already done in Foreman) because, even if it might not
look like that, it should be less work since all of it is already being
used in production environment.

I understand the need for the setup and realization phase being separated,
but introducing new models is just one way of solving that. I have no big
issues with Openstack Roles and Services for the purposes of the wizard
and they
could really speed up its creation.

However, having the Deployment Roles and Deployment Services and
Deployment Service
Params seems quite odd to me, as I don’t see added value in those models
as opposed to hostgroups, global and smart parameters. But it might be just
lack my understanding and I will be happy for somebody bringing more
more lights on the benefits of those.

On the other hand, using the things we already have in Foreman
for the deployment phase allows working on the orchestration without having
the wizard ready to be used (if we agree what the artifacts of the wizard
will be).

Anyway, I would encourage looking into the Foreman advances features around
smart parameters, organizations etc. before modeling those things from
scratch.

Just my $0.02

– Ivan

Take care,
–Hugh

Our main concern is that we are recreating models that are already
present in Foreman. We understand Astapor issue that it was very
hard for user understand all Foreman specifics so now we try to do
it Openstack way and use Foreman as provisioning and configuration
backend. However we still think that we should built on top of
existing and tested models and just create a new easy to use and
intuitive UI.

inecas: +1 on reusing the Foreman model: the foreman model seems to
have most of the things to support what is required, building
something custom means:

duplication of what is already there

writing throw-away code, that nobody except the OFI itself will
use: the power of Foreman is in the community: building it on the
Foreman model (and enhancing it when needed) means the same work
will be used outside of OFI with all the benefits that come with it
I would like to avoid doing that, especially when it doesn’t seem
that much work to use the existing model

So what we propose is following DB schema:

Each deployment is contained under an Organization for separation.
Which also means that same Foreman instance could be normally used
when user leaves the wizard.

Left side is ER model right side is representing relationships
between Hostgroups instances. The one called ‘Deploy’ is lets say an
common parent Hostgroup for all other hostgroups in the deployment.
‘Deploy’ allows sharing of configuration to the other hostgroups.
Other hostgroups are representing Roles (like controller, compute,
etc.)

Basically we’d map:
OpenStackDeployment -> Organization
OpenStackLayout -> organization’s parameter
OpenStackRole -> Hostgroup ()
OpenStackService -> N/A (at least for April)
OpenStackSeviceParam -> LookupKey (aka Smart Variable)
OpenStackDeployService -> N/A
OpenStackDeployServiceParam -> LookupKey (aka Smart Variable Value)
OpenStackDeployRole -> assignment of Host to Hostgroup ()

At the moment, the only thing that’s missing is that Foreman 1.4.1
does not allow us to set one hostgroup to be used in two parent
hostgroups (one service in two roles). This is going to change in
1.5 when ConfigRoles will be merged. We suppose to use quickstack
modules for April version so we still have just puppet classes on
role level (not on service level) and therefore we won’t allow user
to customize services in roles. Since we need this post April we can
stick with current Foreman model. Later we could either back-port
ConfigRoles or base on newer Foreman.

Benefits we see by reusing foreman models

we save work on need to model everything again

we save work on writing logic that will copy all
(Deploy)Service/Role (Deploy)ServiceParameters values to
Hostgroup/LookupKey/LookupValue

ability to turn off the wizard plugin (something like removing the
training wheels) and keeping the Foreman instance still usable

Cons

people may spend some time to better understand existing Foreman code
inecas: that should be under Pros, and by people I don’t mean the
actual users of OFI, but its developers

Potential concerns we understood from yesterday

too foreman specific

need of doing some business logic before Deploy is triggered

writing some easy UI around existing models is bigger pain that
creating new objects from scratch

to able to delay host provisioning until whole deployment is
configured

Our answers to those

this is just about building new UI that we’ll need anyway and
re-labeling (we already have Smart Variable which in fact is
LookupKey model), we could use STI or composition to modify Foreman
objects behavior if they are too restricting

It’s easy to extend Dynflow process with a plugin injecting any
additional steps into the deployment process.

if this really becomes a problem we could always wrap existing
models by some other classes

If the hosts are kept in unmanaged state nothing gets applied to
them. Until orchestration takes over and switches them to managed
state.

Are we sure that we need to allow user to prepare changes to an
existing-deployment-configuration and then apply them all at the
same time in April? I think we can postpone this for later and keep
Foreman’s behavior for now that changes made to a hostgroup are
immediately applied.

Looking forward to your feedback Petr and Marek.

Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

–
== Hugh Brock, hbrock@redhat.com ==
== Senior Engineering Manager, Cloud Engineering ==
== Tuskar: Elastic Scaling for OpenStack ==
== http://github.com/tuskar ==

“I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant.”
–Robert McCloskey

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Daniel Lobato

@elobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30