Extending Foreman host creation process to provide options from external data sources

Hello,

The subject of the thread might be obscure, and the scope of my task just
epic. Possibly, I'm looking for the Holy Grail of infrastructure
management. I'll try to explain as best as I can!

So, our infrastructure is based on static, predictable ip addresses. We
don't use DHCP.
We have a big /8 network that we subnet in /16 per each site, then we
subdivide again to have /24 subnets based on function. Example:

10.x.x.x/8 - big network

10.8.x.x/16 - Sydney datacenter
10.8.24.x/24 - Sydney webservers
10.8.24.20 - Sydney webserver 1
10.8.24.21 - Sydney webserver 2

10.9.x.x/16 - San Francisco datacenter
10.9.24.x/24 - San Francisco webservers
10.9.24.20 - San Francisco webserver 1
10.9.24.21 - San Francisco webserver 2

We are in the process of moving this data in a YAML file, eventually to be
promoted to unique source of truth about the infrastructure. A database
would also work, we are undecided yet. It's in a confluence page right now.
Sadness.

Abstracting a bit, I'd expect to tap into the source of truth and, starting
from these information:

  • server function (example "webserver")
  • server location (example "Sydney")
  • server number (example "1")

Obtain this:

  • Full IP address
  • Vlan
  • additional params depending on what we place in the source of truth

Given this, I would love to do all this in a single, friendly GUI. Enters
Foreman.

Here's an example of what I would love to achieve, in essence:

A Foreman user accesses the frontend (or the API) and wants to create a new
host. He is given the option to choose the server function, server location
and server number mentioned above. This, in the form of pre-populated
dropdowns.
In addition, Foreman would provide this options as it usually does:

  • puppet ca
  • puppet master
  • environment

Also, it would fetch this data from the external source, presenting it in
the frontend in the form of text, dropdowns or editable text:

  • Foreman host group (mapped to server function, somehow)
  • full IP address
  • hostname (generated by a call to our dns server, since our hostnames have
    a predictable scheme)

Pushing it further, the user should be given the option to select other
parameters fetched from, for example, Racktables:

  • rack unit
  • rack number

Then, the user deploys the installation and the rest goes on as usual,
carried by Foreman.

I first and foremost thank you If you had the patience to read up to this
line.
Then, I ask you how much of this would be possible and whether it makes
sense to implement this in Foreman.
As an alternative I've been thinking to implement a script that consumes
the source of truth and passes the ball to Foreman to finish the job,
instead of trying to put everything in Foreman itself.

Thanks for your time,
Nicola

> Hello,
>
> The subject of the thread might be obscure, and the scope of my task just
> epic. Possibly, I'm looking for the Holy Grail of infrastructure
> management. I'll try to explain as best as I can!
>
> So, our infrastructure is based on static, predictable ip addresses. We
> don't use DHCP.
>
not that i want to derail the discussion, but why not? :slight_smile:

We have a big /8 network that we subnet in /16 per each site, then we
> subdivide again to have /24 subnets based on function. Example:
>
> 10.x.x.x/8 - big network
>
> 10.8.x.x/16 - Sydney datacenter
> 10.8.24.x/24 - Sydney webservers
> 10.8.24.20 - Sydney webserver 1
> 10.8.24.21 - Sydney webserver 2
>
> 10.9.x.x/16 - San Francisco datacenter
> 10.9.24.x/24 - San Francisco webservers
> 10.9.24.20 - San Francisco webserver 1
> 10.9.24.21 - San Francisco webserver 2
>
> We are in the process of moving this data in a YAML file, eventually to be
> promoted to unique source of truth about the infrastructure. A database
> would also work, we are undecided yet. It's in a confluence page right now.
> Sadness.
>

> Abstracting a bit, I'd expect to tap into the source of truth and,
> starting from these information:
>
> - server function (example "webserver")
> - server location (example "Sydney")
> - server number (example "1")
>
> Obtain this:
>
> - Full IP address
> - Vlan
> - additional params depending on what we place in the source of truth
>
> Given this, I would love to do all this in a single, friendly GUI. Enters
> Foreman.
>
> Here's an example of what I would love to achieve, in essence:
>
> A Foreman user accesses the frontend (or the API) and wants to create a
> new host. He is given the option to choose the server function, server
> location and server number mentioned above. This, in the form of
> pre-populated dropdowns.
> In addition, Foreman would provide this options as it usually does:
> - puppet ca
> - puppet master
> - environment
>
the two options you have imho, are either using foreman-hooks or a plugin

using foreman hooks you can execute scripts and interact with foreman
during various stages of host creation.

> Also, it would fetch this data from the external source, presenting it in
> the frontend in the form of text, dropdowns or editable text:
> - Foreman host group (mapped to server function, somehow)
> - full IP address
> - hostname (generated by a call to our dns server, since our hostnames
> have a predictable scheme)
>

if you want to interact with a UI, you probably want to consider creating a
foreman plugin, this will allow you to customize either javascript or ruby
/ core foreman objects.

>
> Pushing it further, the user should be given the option to select other
> parameters fetched from, for example, Racktables:
> - rack unit
> - rack number
>

Maybe a more generic foreman-racktables plugin to interface the two.

>
> Then, the user deploys the installation and the rest goes on as usual,
> carried by Foreman.
>
> I first and foremost thank you If you had the patience to read up to this
> line.
> Then, I ask you how much of this would be possible and whether it makes
> sense to implement this in Foreman.
> As an alternative I've been thinking to implement a script that consumes
> the source of truth and passes the ball to Foreman to finish the job,
> instead of trying to put everything in Foreman itself.
>

IMHO - whatever which is very generic to your usage case (e.g. build in
source of truth) consider using foreman hooks, api calls or a plugin, for
the more generic stuff, consider starting a plugin, chances are this is not
only your problem…

Ohad

··· On Fri, Jul 3, 2015 at 6:37 PM, Nicola V wrote:

Thanks for your time,
Nicola


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

>
> So, our infrastructure is based on static, predictable ip addresses. We
>> don't use DHCP.
>>
> not that i want to derail the discussion, but why not? :slight_smile:
>

Hello! It's a good question.
All our firewalls rules are designed around /24 or even smaller subnets,
and the rulesets are "portable" to other DCs based on the predictability pf
the 2nd, 3rd and 4th octects.
It is clunky, hence we have been indeed thinking of moving to DHCP. This
discussion on Serverfault put me off for the time being:

I'd be happy to hear some success stories and reconsider our firewall
approach. We had in mind to distribute the firewalling to the hosts
themselves, as an option.

> the two options you have imho, are either using foreman-hooks or a plugin
> using foreman hooks you can execute scripts and interact with foreman
> during various stages of host creation.
> if you want to interact with a UI, you probably want to consider creating
> a foreman plugin, this will allow you to customize either javascript or
> ruby / core foreman objects.
> Maybe a more generic foreman-racktables plugin to interface the two.
>
IMHO - whatever which is very generic to your usage case (e.g. build in
> source of truth) consider using foreman hooks, api calls or a plugin, for
> the more generic stuff, consider starting a plugin, chances are this is not
> only your problem…
>
> Ohad
>

Thanks for all the suggestions, Ohad. That is exactly what I would love to
do.

The racktables part can be developed afterwards. I'd be more interested in
the "get location/role/network info from source of truth > generate
hostname and populate some options in Foreman" part.

I have the ideas and a somewhat clear concept of what I'm aiming to
achieve. Unfortunately, I have no skills to code it myself. I could well
consider to sponsor the development of some open source plugin, in case my
company is interested and willing to financially contribute.
Would anyone be interested in such a plugin and eventually available to
develop it, if I ever got the green light?

Regards,
Nicola

··· On Sunday, July 5, 2015 at 5:16:13 PM UTC+2, ohad wrote: