Sending another update: I've added Object diagram modeling simple web
app with stacks.
deployment.puml (1.85 KB)
Sure, I'll aim to schedule something on the end of next week.
···
On 01.12.14 12:55, Ohad Levy wrote: > Petr, > > Would it be possible to do a deep dive/ review on the model? > > thanks, > Ohad > > On Mon, Dec 1, 2014 at 1:43 PM, Petr Chalupa > wrote: > > Updating ComputeResourceResource to allow image based provisioning. > Adding flow.md describing the deployment > life-cycle, also copied below. > > # Deployment flow > > _If user is not mentioned then the step is done automatically._ > > - Show Deployment form to **User**. > - **User** inputs name and description. > - Form is submitted. > - Deployment is created. > - **User** creates required resources one by one. > - **User** assigns existing foreman `Subnet`s with > `SubnetTypesResource`s. > - **User** Starts by creating all the `Hostgroup`s for each > `HostgroupResource`. > (Custom form allowing to select just items bellow.) > - If parent is not `HostgroupResource`, it needs to provided > by user. > - `Hostgroup` inherits configuration from parent, remaining > options > are provided by user. > - `Hostgroup` is created. > - `ParamResources` adds the parameters to the hostgroup. > - `Puppetclasses` are looked up by `PuppetClassResource`s. > - `ParamOverideResource`s overrides defined puppet class > parameters. > - **User** continues by creating all the `Hosts` for each > `HostResouce`. > (Custom form allowing to select just items bellow.) > - **User** assigns `Subnet`s to subnet types, > - picks `ComputeResource` based on allow-list defined by > `ComputeResourceResource`s. > - **User** selects provisioning (true is default) (Some > managed hosts may be > just reused without provisioning). > - **User** defines number of the `Host`s required within > `min`, `max` > defined on `HostResource`. > - Bare metal hosts are assigned if applicable. > - `Host`s are created or if already existing just assigned to > proper > `Hostgroup`. > - Provisioning may already start if it cannot be delayed. But the > puppetrun has to be always delayed, it's orchestrated later. > - All hosts are just created waiting for provisioning or only > provisioned > without configuration triggered. > - **User** triggers deployment. > - All hosts are provisioned if required. > - Waits until all hosts are provisioned. > - Orchestration of the `Ordered` resources. e.g.: > - Puppetrun an all controller nodes in parallel, > - then puppetrun on all compute nodes in parallel. > > Petr > > > On 28.11.14 9:13, Petr Chalupa wrote: > > Removing any associations to existing Foreman model from stacks > to allow > stack sharing without problems between users. > > On 26.11.14 9:27, Petr Chalupa wrote: > > Hi > > Updating the model for stacks and adding data model of a > deployment. > > Petr > > On 24.11.14 16:15, Petr Chalupa wrote: > > Thanks Greg for the nice writeup. As you've mentioned I > am working on > data model. The first part representing the Stacks > (template of a > Deployment) is attached. > > I'll continue working on Deployment instance > representation tomorrow. > Then I'll send some examples of actual stack instances > describing simple > apps/products. > > I've used PlanUML [1] and Idea plugin [2]. > > ## Legend > > Entities: > (A) = Abstract class > (C) = Class > (M) = Module > > Arrows: > A --|> B (blank closed arrow) = represents inheritance > `B.is_a?(A)` > A --o B (blank diamond) = Aggregation - B is composed of A > > Petr > > [1] http://plantuml.sourceforge.__net/classes.html > > [2] > pluginhttps://plugins.__jetbrains.com/plugin/7017?pr=__phpStorm > > > On 21.11.14 15:39, Greg Sutcliffe wrote: > > Hi all, > > (TL;DR - multi-host orchestration is coming, help us > make it awesome!) > > For a while we've been discussing the concept of > multi-host > orchestration in Foreman, and how to achieve it. > We've formed a > feature team to start looking into implementing it, > and I'm writing > this email to give the community some insight into > our planning so > far, and to invite input at this early to make sure > we haven't missed > anything. (There's also quite a lot of lessons from > Staypuft in this > document as well!) > > Some terms we're using: > - Resource: a Foreman object (Host, Subnet, > Puppetclass etc) > - Stack: the instructions needed to assemble a set > of resources for a > specific purpose (e.g. a database stack) > - Stacks can be nested (e.g. an app stack combines a > db stack with a > webserver stack, etc) > - Deployment: the result of performing the > instructions in a stack, > taking in any necessasry input parameters (MyTwitter > is a deployment > of the app stack) [note: I actually favour the term > Scenario here, but > I got outvoted - anyone else prefer this? :P] > > The current thinking is that we need some new > first-class models in > Foreman for the Stack and Deployment objects, so > that we can store > them. The Stack object is intended to be an abstract > representation of > the thing to be built, the "receipe" if you will. > This contains things > like the number of Hosts to build, the PuppetClasses > to execute on > those hosts, and so on. Stacks can be nested, so > that one can write a > simple stack for, say, a db cluster, and then use > that stack for > defining a more complex object. The Stack also > contains ordering, > where it makes sense, so that you can ensure a db > host is built before > an app host tries to use it. > > A Deployment is an instance of a Stack, where we can > record the exact > hosts used, the parameters passed to the puppet > classes (so we can > correctly share things like DB passwords to the > right hosts, etc), and > where we can at a later time update the deployment. > > For provisioning, the eventual plan is to either > have the user provide > a set of prebuilt hosts (configured with appropriate > authentication so > that Foreman can log in and configure them), or to > have the user > specify one (or more) compute resources (or bare > metal, via discovery > as staypuft does) and allow Foreman to build the > necessary hosts > before proceeding to the provisioning step. Clearly > this is only > needed when creating a Deployment, since a Stack is > an abstract > representation. > > There will of course be a UI eventually, but initial > efforts will be > focussed on creating these stacks/deployments via > the API, and on > defining an appropriate schema for the JSON used to > create them. Any > UI efforts will likely be just for displaying what > has already been > created. > > To get started on this, we've broken it down into > some user stories. > This is by no means a final list, but in *rough* > order, we have > (indents are comments on the story): > > * A data design for stack and deployments > * A JSON schema for creating these objects via the API > * Ability to deploy a single-machine 'stack' > * Basically what Foreman already does, but > allows us to test stack > nesting, etc > * Ensures we can deploy independant services > * Ensures we can deploy intra-dependant services > on a single box > (ordering / puppetclasses) > * Ability to define network configuration for a > stack (single network > only initially) > * Ability to detect / act upon when a host has > completed it's > configuration > * Needed to know when to start the deployment of > the next host in > sequence > * Staypuft uses the incoming Puppet reports, but > we can probably > improve on this > * Ability to deploy a 3 machine stack on the single > network > * 3 is arbitrary > * Introduces inter-depdant services between machines > * Ability to deploy multi-network stack > * Permits concepts such as compute/management > networks, provisioning > networks etc > > * Ability to consume existing hosts within a deployment > * Ability to spin up hosts on a single compute > resource for a > deployment > * Ability to spin up hosts on multiple compute > resources for a > deployment > * Ability to consume discovered hosts via the > Discovery plugin for a > deployment > > The ultimate goal is to make these Stack objects > shareable, so that we > can all contribute to having well designed stacks > running within our > infrastructures. We have a long way to go though. > > Some of you will want to know what research we've > done, and our > thoughts on the tools already out there - I'll let > Daniel reply with > that info. Petr is already working on the data > designs, and will reply > shortly with his work so far. For myself, I'll be > picking up the task > of extending how we know when a host is "finished" > which, since it > involves things like running Puppet after reboot, is > more nuanced than > simply receiving the /unattended/built call from the > installer. > > Thanks for reading, and we want to hear your > thoughts on the design! > > Greg > > > > > > -- > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to foreman-dev+unsubscribe@__googlegroups.com > . > For more options, visit https://groups.google.com/d/__optout > . > > > -- > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-dev+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout.Based on feed-back on deep-dive I am thinking about incorporating
following changes to the data-model / deployment configuration regarding
parameter handling. Please let me know what you think, I'll be working
on updating the model and related notes in meanwhile.
-
Add DeploymentParameter (global parameters on deployment as on
domains, OSs, Hostgroups, Hosts, etc.). -
Allow to define ParameterResource without dependency on
HostgroupResource. Such ParameterResource is then interpreted as
Deployment wide parameter and appropriate DeploymentParameter is created
instead of GroupParameter (as in case when HostgroupResource is
associated to ParameterResource). It can be used by
ParameterOverrideResources as before. -
Let there be a global parameter described in a stack with e.g. with
name 'user'. When deployment instance is being configured and global
parameters are being created, if there is a global parameter 'user'
already present (inherited from parent hostgroup), such parameter is
then used instead of creating new GroupParameter forcing user to supply
the value and overriding the already existing configuration of the
'user' parameter. Previously new GroupParameter was always created. -
ParameterOverrideResources are newly allowed not to have a value. In
that case the value is also considered as an required input for the
stack. Similarly as with global parameter already present, when the
value is already configured by different matcher (e.g. for domain), new
override with matcher based on hostgroup is not defined to force the
user to supply the value again. -
Deployments will be allowed to be matched in Puppet class-parameter
overrides. (Another example of the value being predefined in 4.)
Changes imply that deployed Hosts and Hostgroups will no longer be
independent on Deployment object after deployment, which is not
necessarily a bad thing, but noting.
Petr
···
On 08.12.14 14:04, Petr Chalupa wrote: > Sending another update: I've added Object diagram modeling simple web > app with stacks. > > On 01.12.14 12:43, Petr Chalupa wrote: >> Updating ComputeResourceResource to allow image based provisioning. >> Adding flow.md describing the deployment life-cycle, also copied below. >> >> # Deployment flow >> >> _If user is not mentioned then the step is done automatically._ >> >> - Show Deployment form to **User**. >> - **User** inputs name and description. >> - Form is submitted. >> - Deployment is created. >> - **User** creates required resources one by one. >> - **User** assigns existing foreman `Subnet`s with >> `SubnetTypesResource`s. >> - **User** Starts by creating all the `Hostgroup`s for each >> `HostgroupResource`. >> (Custom form allowing to select just items bellow.) >> - If parent is not `HostgroupResource`, it needs to provided by >> user. >> - `Hostgroup` inherits configuration from parent, remaining >> options >> are provided by user. >> - `Hostgroup` is created. >> - `ParamResources` adds the parameters to the hostgroup. >> - `Puppetclasses` are looked up by `PuppetClassResource`s. >> - `ParamOverideResource`s overrides defined puppet class >> parameters. >> - **User** continues by creating all the `Hosts` for each >> `HostResouce`. >> (Custom form allowing to select just items bellow.) >> - **User** assigns `Subnet`s to subnet types, >> - picks `ComputeResource` based on allow-list defined by >> `ComputeResourceResource`s. >> - **User** selects provisioning (true is default) (Some managed >> hosts may be >> just reused without provisioning). >> - **User** defines number of the `Host`s required within `min`, >> `max` >> defined on `HostResource`. >> - Bare metal hosts are assigned if applicable. >> - `Host`s are created or if already existing just assigned to >> proper >> `Hostgroup`. >> - Provisioning may already start if it cannot be delayed. But the >> puppetrun has to be always delayed, it's orchestrated later. >> - All hosts are just created waiting for provisioning or only >> provisioned >> without configuration triggered. >> - **User** triggers deployment. >> - All hosts are provisioned if required. >> - Waits until all hosts are provisioned. >> - Orchestration of the `Ordered` resources. e.g.: >> - Puppetrun an all controller nodes in parallel, >> - then puppetrun on all compute nodes in parallel. >> >> Petr >> >> On 28.11.14 9:13, Petr Chalupa wrote: >>> Removing any associations to existing Foreman model from stacks to allow >>> stack sharing without problems between users. >>> >>> On 26.11.14 9:27, Petr Chalupa wrote: >>>> Hi >>>> >>>> Updating the model for stacks and adding data model of a deployment. >>>> >>>> Petr >>>> >>>> On 24.11.14 16:15, Petr Chalupa wrote: >>>>> Thanks Greg for the nice writeup. As you've mentioned I am working on >>>>> data model. The first part representing the Stacks (template of a >>>>> Deployment) is attached. >>>>> >>>>> I'll continue working on Deployment instance representation tomorrow. >>>>> Then I'll send some examples of actual stack instances describing >>>>> simple >>>>> apps/products. >>>>> >>>>> I've used PlanUML [1] and Idea plugin [2]. >>>>> >>>>> ## Legend >>>>> >>>>> Entities: >>>>> (A) = Abstract class >>>>> (C) = Class >>>>> (M) = Module >>>>> >>>>> Arrows: >>>>> A --|> B (blank closed arrow) = represents inheritance `B.is_a?(A)` >>>>> A --o B (blank diamond) = Aggregation - B is composed of A >>>>> >>>>> Petr >>>>> >>>>> [1] http://plantuml.sourceforge.net/classes.html >>>>> [2] pluginhttps://plugins.jetbrains.com/plugin/7017?pr=phpStorm >>>>> >>>>> On 21.11.14 15:39, Greg Sutcliffe wrote: >>>>>> Hi all, >>>>>> >>>>>> (TL;DR - multi-host orchestration is coming, help us make it >>>>>> awesome!) >>>>>> >>>>>> For a while we've been discussing the concept of multi-host >>>>>> orchestration in Foreman, and how to achieve it. We've formed a >>>>>> feature team to start looking into implementing it, and I'm writing >>>>>> this email to give the community some insight into our planning so >>>>>> far, and to invite input at this early to make sure we haven't missed >>>>>> anything. (There's also quite a lot of lessons from Staypuft in this >>>>>> document as well!) >>>>>> >>>>>> Some terms we're using: >>>>>> - Resource: a Foreman object (Host, Subnet, Puppetclass etc) >>>>>> - Stack: the instructions needed to assemble a set of resources for a >>>>>> specific purpose (e.g. a database stack) >>>>>> - Stacks can be nested (e.g. an app stack combines a db stack with a >>>>>> webserver stack, etc) >>>>>> - Deployment: the result of performing the instructions in a stack, >>>>>> taking in any necessasry input parameters (MyTwitter is a deployment >>>>>> of the app stack) [note: I actually favour the term Scenario here, >>>>>> but >>>>>> I got outvoted - anyone else prefer this? :P] >>>>>> >>>>>> The current thinking is that we need some new first-class models in >>>>>> Foreman for the Stack and Deployment objects, so that we can store >>>>>> them. The Stack object is intended to be an abstract >>>>>> representation of >>>>>> the thing to be built, the "receipe" if you will. This contains >>>>>> things >>>>>> like the number of Hosts to build, the PuppetClasses to execute on >>>>>> those hosts, and so on. Stacks can be nested, so that one can write a >>>>>> simple stack for, say, a db cluster, and then use that stack for >>>>>> defining a more complex object. The Stack also contains ordering, >>>>>> where it makes sense, so that you can ensure a db host is built >>>>>> before >>>>>> an app host tries to use it. >>>>>> >>>>>> A Deployment is an instance of a Stack, where we can record the exact >>>>>> hosts used, the parameters passed to the puppet classes (so we can >>>>>> correctly share things like DB passwords to the right hosts, etc), >>>>>> and >>>>>> where we can at a later time update the deployment. >>>>>> >>>>>> For provisioning, the eventual plan is to either have the user >>>>>> provide >>>>>> a set of prebuilt hosts (configured with appropriate >>>>>> authentication so >>>>>> that Foreman can log in and configure them), or to have the user >>>>>> specify one (or more) compute resources (or bare metal, via discovery >>>>>> as staypuft does) and allow Foreman to build the necessary hosts >>>>>> before proceeding to the provisioning step. Clearly this is only >>>>>> needed when creating a Deployment, since a Stack is an abstract >>>>>> representation. >>>>>> >>>>>> There will of course be a UI eventually, but initial efforts will be >>>>>> focussed on creating these stacks/deployments via the API, and on >>>>>> defining an appropriate schema for the JSON used to create them. Any >>>>>> UI efforts will likely be just for displaying what has already been >>>>>> created. >>>>>> >>>>>> To get started on this, we've broken it down into some user stories. >>>>>> This is by no means a final list, but in *rough* order, we have >>>>>> (indents are comments on the story): >>>>>> >>>>>> * A data design for stack and deployments >>>>>> * A JSON schema for creating these objects via the API >>>>>> * Ability to deploy a single-machine 'stack' >>>>>> * Basically what Foreman already does, but allows us to test stack >>>>>> nesting, etc >>>>>> * Ensures we can deploy independant services >>>>>> * Ensures we can deploy intra-dependant services on a single box >>>>>> (ordering / puppetclasses) >>>>>> * Ability to define network configuration for a stack (single network >>>>>> only initially) >>>>>> * Ability to detect / act upon when a host has completed it's >>>>>> configuration >>>>>> * Needed to know when to start the deployment of the next host in >>>>>> sequence >>>>>> * Staypuft uses the incoming Puppet reports, but we can probably >>>>>> improve on this >>>>>> * Ability to deploy a 3 machine stack on the single network >>>>>> * 3 is arbitrary >>>>>> * Introduces inter-depdant services between machines >>>>>> * Ability to deploy multi-network stack >>>>>> * Permits concepts such as compute/management networks, >>>>>> provisioning >>>>>> networks etc >>>>>> >>>>>> * Ability to consume existing hosts within a deployment >>>>>> * Ability to spin up hosts on a single compute resource for a >>>>>> deployment >>>>>> * Ability to spin up hosts on multiple compute resources for a >>>>>> deployment >>>>>> * Ability to consume discovered hosts via the Discovery plugin for a >>>>>> deployment >>>>>> >>>>>> The ultimate goal is to make these Stack objects shareable, so >>>>>> that we >>>>>> can all contribute to having well designed stacks running within our >>>>>> infrastructures. We have a long way to go though. >>>>>> >>>>>> Some of you will want to know what research we've done, and our >>>>>> thoughts on the tools already out there - I'll let Daniel reply with >>>>>> that info. Petr is already working on the data designs, and will >>>>>> reply >>>>>> shortly with his work so far. For myself, I'll be picking up the task >>>>>> of extending how we know when a host is "finished" which, since it >>>>>> involves things like running Puppet after reboot, is more nuanced >>>>>> than >>>>>> simply receiving the /unattended/built call from the installer. >>>>>> >>>>>> Thanks for reading, and we want to hear your thoughts on the design! >>>>>> >>>>>> Greg >>>>>> >>>>> >>>> >>> >> >Updates are in Foreman deployment by pitr-ch · Pull Request #1954 · theforeman/foreman · GitHub and attached.
db-stack-example.puml (413 Bytes)
