Hello,
First let me apologies there have been a misunderstanding between me and
Shim, most probably poor choice of words on my part. I meant, we reached
an understanding on the new proposal from Shim not on the way to go with
this feature. We will continue with the discussions until we reach a
consensus how to continue.
Shims ideas are definitely very valuable and it resulted in new
discussions which were very needed to steer this feature in right direction.
It may not seem like it but the too designs are quite similar even
though they use different naming which may be quite confusing.
Extending Original design
Tom spend some quite time documenting the original design adding
configurations to it (which came up form our discussions as missing to
support usage of one-button-click-users (click -> deployed app)). This
design is done in much more depth and I strongly believe that it would
be much better to evolve this one rather than starting new design.
So to that end let me outline incorporation of Shims ideas. I assume
some understanding of the original design [1].
Quick remainder about the original design's flow:
- stack is definition
- stack is imported
- deployment of a stack is created
- existing configuration is chosen or deployment is configured by user
- deployment is planned and executed in Dynflow
Resources
The original design contained two subtypes of resources: configuration
and ordered. Where configuration was creating/modifying foreman objects
and ordered resource were representing actions to be taken in Foreman
(e.g. PuppetRun) (with defined dependencies).
The configuration resources had order of configuration built in which
may had been unnecessarily constraining and hard to grasp. There were
originally included to be able to use Foreman objects as direct storage
of the configuration which was later extracted to be stored separately.
Therefore we can drop the configuration resources to be replaced by
ordered resources removing the constrains and unifying it under one
abstraction.
After that, the resources can be thought about as tasks/actions - steps
which has to be taken to successfully deploy an application or
infrastructure. Examples: TakeHost, TakeHostgroup, TakeSubnet,
PuppetRun, etc. (the 'take' is chosen to be better corresponding with
fact that it can create host or use an existing one).
This brings it much closer to Shim's proposal.
You may ask how are provisioning and other actions, which were before
hard-coded in the deployment process not visible to user, handled. They
will be part of the definition which may in the end help with the
understandability of the whole deployment process. (It will be validated
for deployments that they are present in the definition.)
The Tasks will be directly mapped to Dynflow actions and its ability to
create dependencies based on what outputs action is consuming.
Task (former Resources) Properties
Terms dictionary:
- stack or definition - sharable part of the deployments, independent on
Foreman instance.
- configuration - full or partial configuration values for a given stack
and dependent on particular Foreman instance. They can be merged together.
I very much like Shim's proposal about properties definition and the
yaml format. There will be an DSL which defines all the definable (in
stack) and configurable (in configuration) properties on Foreman objects
or actions (I think we've only talked about this on call it was not in
the email.). Those property definitions are then used to create inputs
for the tasks as shown in Shim's example.
Few examples:
- name property on TakeHost: has type String, can be defined in stack,
or marked as input for configuration
- parent hostgoup property on TakeHost: has type Hostgroup(id), can be
marked as input for configuration or as reference to a result of
different TakeHostgroup task. It cannot have concrete value definition.
The DSL is very important to be able to validate the definitions
thoroughly before trying theirs deployment and to be able to ensure
stack isolation from Foreman instance ensuring smooth ability to share them.
Configuration UI
Stack definition defines exactly it's interface by defining the inputs
in certain properties. The UI is generated automatically based on the
required input by stack definition and the definition of the configured
property (type, enum values, etc.) on tasks.
This can be achieved very quickly and result will be reasonably
understandable to users. Specific UI bit can be added to fix confusing
groups of properties (interface configuration may need something like that).
Growing Low hanging fruits for later
The shift towards being more task oriented and the layer of properties
definitions will prepare ground to able to add more actions for other
Foreman operations (not just those required for deployment). That will
result in ability to create groups of stacks (or in this context let's
talk about processes), processes to support whole management of the
application infrastructure.
Process Examples:
- deployment of an application
- scale up/down the application
- upgrade DB servers without downtime
- update application code on web servers
- backup and destroy the application
- etc.
Summary
By altering the original design to incorporate the main ideas from Shim
(task orientation, yaml definition, property specification):
- We'll avoid hitting same the same problems which we would hit when
doing new design from scratch, without throwing away the original design.
- We will get better understandability for users provided by the
simplification of resources.
Session
We will do a Bluejeans/Hangout session on Thuesday at 1pm(GMT) on the
merge of the two approaches (outlined in this email) to get most of it.
Link will follow.
[1] https://github.com/theforeman/foreman_deployments#design-documentation
···
On 19.04.15 13:12, Shim Shtein wrote:
> Hi all,
>
> After a (very) long discussion with Petr on Thursday, finally we have
> reached an agreement about the design of this feature. Let me introduce
> you the new deployments design, please comment and challenge this
> design, I want to be sure we didn't overlooked something big.
>
> The way that we want to look at the feature is more as a changeset
> mechanism for Foreman objects. That means that each deployment is a
> description of changes that need to take place in the target Foreman
> instance in order to fulfill the purpose of the deployment.
>
> The process itself can be divided into three phases, I will explain each
> of them separately.
>
> 1. Description phase - describe the needed changes to the Foreman objects
> 2. Configuration phase - Additional descriptions (configuration) that
> will be used to fill in the missing information in order to
> create/modify objects in [1].
> 3. Apply phase - planning and executing the changes that were described
> previously.
>
>
> *Description phase*
>
> This is the phase that is responsible of describing what changes need to
> be made in the target system.
> As a language we have chosen a static file format that will give us the
> ability to describe arrays, hashes (key-value store) and simple values.
> I vote for YAML, but it can be anything else, if it is more convenient
> and clear. Each node at the upper hierarchy will describe a change. Here
> is an example of such definition:
>
> >
> ParentHostgroup1:!Hostgroup{
> name:"Base hostgroup"
> }
>
>
> NewHostgroup1:!Hostgroup{
> name:!input {
> type:string,
> regex:".*"
> }
> puppet_classes:
> -"class1"
> -"class2"
> parent:!reference {
> object:"ParentHostgroup1"
> field:"result"
> }
> }
> >
>
>
> Here we can see all basic elements of the description language:
>
> 1. The uppermost element is the change desriptor.
> 2. The type of the foreman object that will be described is defined
> using the custom type notion in YAML (the "!" mark).
> 3. Inside each definition there will be an open list of key - value
> pairs that will be applied as properties to the Foreman object. That
> means that the property "name" of a host_group object with key
> "ParentHostgroup1" will be set to "Base hostgroup".
> 4. Two more custom types will be defined as a value for property:
> "input" and "reference"
> 1. input - means that the system will ask the user to provide more data
> 2. reference - means that the value of the property will be
> generated from the value of the referenced object.
>
>
> *Configuration phase*
>
> It is clear that the description phase will not contain all the
> information needed for creating new objects in Foreman. The
> configuration is the place where the system will store all the data
> needed for actual creation of the objects. I suggest to keep the
> language of the configuration consistent with the description language.
> Basically the configuration phase is about completing needed information
> in order a particular run of deployment to succeed.
> A very good example of proerties that will be in the configuration
> rather then in the description is the set of objects that are local to
> the current run, such as references to image locations, taxonomies,
> puppet environments and so on.
> The system will allow two modes of inputing the configuration:
>
> 1. As a separate file - it will need a whole set of configuration in
> order to move to the next step
> 2. Interactive - it will ask the user to enter all additional
> information, using the regular forms for the mentioned objects with
> the values from the description phase already set in the object.
>
>
> *Apply phase*
>
> This is the actual phase where the objects will be created. This step
> will use dynflow as it scheduling engine, dynflow will also take care of
> the order in which the objects are created. Basically each object
> changeset definition will be translated into a dynflow task, with
> dependencies set as input parameters for it. The output of each task
> will be the object it creates, so it could be passed to the next task
> that needed it.
>
>
> Additional issues that were taken care of in this design:
>
> Ordered actions, such as puppet runs, actual provisioning
>
> Since the design talks only about changing foreman objects there is a
> problem to define a pure action in the same file. This is solved by
> creating "virtual" resource, that will be translated into the desired
> action. For example in order to configure a puppet run before
> provisioning another host, there will be created following definition:
>
> >
> ForcePuppet1:!Puppetrun{
> host:!reference {
> object:"DbHost1"
> field:"result"
> }
> }
>
>
> ProvisionWebHost1:!ProvisionProcess{
> host:!reference {
> object:"WebHost1"
> field:"result"
> }
> dependency:!reference {
> object:"ForcePuppet1"
> field:"result"
> }
> }
> >
>
>
> Main flow of the system:
>
> 1. Parse Definition
> 2. Display all inputs
> 3. Parse config
> 4. Plan Dynflow actions
> 1. If failed => display UI fror all (failed) objects and save as
> new config, repeat step 4.
> 2. If success => Run dynflow actions.
>
>
>
> Thanks,
> Shimon.
>
>
>
> On Thursday, April 16, 2015 at 11:49:46 AM UTC+3, Tomas Strachota wrote:
>
> Hi all,
>
> it's been a while since we first announced the efforts towards
> multi-host orchestration in the Foreman. We spent the last two weeks
> with documenting the design that Greg describes in [1]. We also moved
> the original design a bit further with storing the stack configuration
> separately. The documentation is at [2]. It is now time to discuss and
> refine the concepts. Please share your ideas and opinions.
>
> Very recently, Shim came with a bit different idea of modeling the
> deployments. I'll let him to describe the differences in a response, so
> that we can find the pros/cons and possibly iterate to a final
> solution.
>
> Regards
> Tomas
>
>
> [1]
> https://groups.google.com/forum/#!searchin/foreman-dev/deployment$20objects/foreman-dev/bCzCjTDY0no/MilXayae7K8J
>
>
> [2]
> https://github.com/theforeman/foreman_deployments#design-documentation
>
>
>
