Hi all,
(TL;DR - multi-host orchestration is coming, help us make it awesome!)
For a while we've been discussing the concept of multi-host
orchestration in Foreman, and how to achieve it. We've formed a
feature team to start looking into implementing it, and I'm writing
this email to give the community some insight into our planning so
far, and to invite input at this early to make sure we haven't missed
anything. (There's also quite a lot of lessons from Staypuft in this
document as well!)
Some terms we're using:
- Resource: a Foreman object (Host, Subnet, Puppetclass etc)
- Stack: the instructions needed to assemble a set of resources for a
specific purpose (e.g. a database stack) - Stacks can be nested (e.g. an app stack combines a db stack with a
webserver stack, etc) - Deployment: the result of performing the instructions in a stack,
taking in any necessasry input parameters (MyTwitter is a deployment
of the app stack) [note: I actually favour the term Scenario here, but
I got outvoted - anyone else prefer this? :P]
The current thinking is that we need some new first-class models in
Foreman for the Stack and Deployment objects, so that we can store
them. The Stack object is intended to be an abstract representation of
the thing to be built, the "receipe" if you will. This contains things
like the number of Hosts to build, the PuppetClasses to execute on
those hosts, and so on. Stacks can be nested, so that one can write a
simple stack for, say, a db cluster, and then use that stack for
defining a more complex object. The Stack also contains ordering,
where it makes sense, so that you can ensure a db host is built before
an app host tries to use it.
A Deployment is an instance of a Stack, where we can record the exact
hosts used, the parameters passed to the puppet classes (so we can
correctly share things like DB passwords to the right hosts, etc), and
where we can at a later time update the deployment.
For provisioning, the eventual plan is to either have the user provide
a set of prebuilt hosts (configured with appropriate authentication so
that Foreman can log in and configure them), or to have the user
specify one (or more) compute resources (or bare metal, via discovery
as staypuft does) and allow Foreman to build the necessary hosts
before proceeding to the provisioning step. Clearly this is only
needed when creating a Deployment, since a Stack is an abstract
representation.
There will of course be a UI eventually, but initial efforts will be
focussed on creating these stacks/deployments via the API, and on
defining an appropriate schema for the JSON used to create them. Any
UI efforts will likely be just for displaying what has already been
created.
To get started on this, we've broken it down into some user stories.
This is by no means a final list, but in rough order, we have
(indents are comments on the story):
-
A data design for stack and deployments
-
A JSON schema for creating these objects via the API
-
Ability to deploy a single-machine 'stack'
- Basically what Foreman already does, but allows us to test stack
nesting, etc - Ensures we can deploy independant services
- Ensures we can deploy intra-dependant services on a single box
(ordering / puppetclasses)
- Basically what Foreman already does, but allows us to test stack
-
Ability to define network configuration for a stack (single network
only initially) -
Ability to detect / act upon when a host has completed it's configuration
- Needed to know when to start the deployment of the next host in sequence
- Staypuft uses the incoming Puppet reports, but we can probably
improve on this
-
Ability to deploy a 3 machine stack on the single network
- 3 is arbitrary
- Introduces inter-depdant services between machines
-
Ability to deploy multi-network stack
- Permits concepts such as compute/management networks, provisioning
networks etc
- Permits concepts such as compute/management networks, provisioning
-
Ability to consume existing hosts within a deployment
-
Ability to spin up hosts on a single compute resource for a deployment
-
Ability to spin up hosts on multiple compute resources for a deployment
-
Ability to consume discovered hosts via the Discovery plugin for a deployment
The ultimate goal is to make these Stack objects shareable, so that we
can all contribute to having well designed stacks running within our
infrastructures. We have a long way to go though.
Some of you will want to know what research we've done, and our
thoughts on the tools already out there - I'll let Daniel reply with
that info. Petr is already working on the data designs, and will reply
shortly with his work so far. For myself, I'll be picking up the task
of extending how we know when a host is "finished" which, since it
involves things like running Puppet after reboot, is more nuanced than
simply receiving the /unattended/built call from the installer.
Thanks for reading, and we want to hear your thoughts on the design!
Greg