Organizing Large Amount of Subnets, Host Groups and Discovery Rules

Support
1

Hello.

Let me apologize up front for this huge post…

I am still very new to Foreman. I’m continuing my proof-of-concept for an organization with roughly 8000 nodes which I would like to manage with Foreman.

Before diving into my exact question, let me give a quick summary of what my organization looks like:

  • 1300 branch offices.
  • Each branch office has two subnets:
    • “Workstation-Net” - This includes a single Linux server for the branch and some Windows workstations (I only care about the server in this scenario).
    • “Kiosk-Net” - This subnet includes 3-5 kiosk-type systems running Linux.

I want to be able to use Foreman to provision both the branch server (in “Workstation-Net”) and the kiosk systems (in “Kiosk-Net”) for each branch.

Now I’m trying to understand how to organize this all in Foreman. Let me explain how I have things setup now in my lab environment.

I have two labs, which are modeled just like a branch office, with two subnets:

  • Lab_01
  • Lab_02

Now because each lab has two subnets, I need to create a total or four subnets in Foreman:

  • Lab_01_Workstation-Net_10.59.24.0/24
  • Lab_01_Kiosk-Net_10.59.152.0/27
  • Lab_02_Workstation-Net_10.59.118.0/24
  • Lab_02_Kiosk-Net_10.59.246.0/27

So my understanding is that when provisioning a system via discovery (and using auto-provision), I need to match up the discovered nodes with a host group. I now create four host groups, one for each subnet:

  • Lab_01_Workstation-Net_HG
  • Lab_01_Kiosk-Net_HG
  • Lab_02_Workstation-Net_HG
  • Lab_02_Kiosk-Net_HG

Now, the way a “server” provisions is much different than how a “kiosk” would provision. So I create two separate kickstart provisioning templates, which are associated with their respective host groups:

  • Server_Kickstart
    • Associated with “Lab_01_Workstation-Net_HG” and “Lab_02_Workstation-Net_HG”.
  • Kiosk_Kickstart
    • Associated with “Lab_01_Kiosk-Net_HG” and “Lab_02_Kiosk-Net_HG”.

I have no idea if this is the most effective way to organize this all, but at least I can now execute the following workflow:

  1. System PXE boots into discovery image.
  2. From Foreman, the discovered host can be manually assigned to its respective host group depending if it is a “server” or a “kiosk”. The host group ultimately dictates what the node provisions as.
  3. The system then reboots and provisions successfully.

However, a requirement of mine is to have auto-provisioning, which means setting up discovery rules. The thing that trips me up though is that each rule can only assign to one host group.

So this basically means I need to create a discovery rule for each host group, each having a search pattern which only matches systems in that discovered host group’s subnet, right?

If I’m understanding this all correctly (which I probably am not), that would seem like a lot of subnets, host groups, provisioning template associations, and discovery rules (if factoring in the entire organization, not just my labs)!

I’m sure that I could script something that pulls this data out of a CSV sheet and creates the needed objects via the Foreman API, but… I am wondering if I’m just over-complicating this and completely missing a better way of organizing this all?

Again, sorry for the huge book here. Thank you so much for reading and if you have any tips for me, I would be so appreciative!

1 Like
2

Hello, I wish everybody seeking for help would write such a nice post. Well done, I have all information.

Unfortunately, yes. Discovery rule provisioning is quite “dumb”, Foreman simply matches rules one after another. You are right that the proper way would be to create ton of rules using our CLI, API or Ansible Foreman Module and match the discovered hosts against IP (network) address.

However, if you are willing to experiment, there are two other ways.

First, the most robust but also quite dumb way is to implement a simple service (shell script or whatever) that would query list of discovered hosts every minute. When a host is found, it fetches its details, finds the IP, matches it against your database (excel, CSV) or Foreman database (subnets) and “manually” starts provisioning process. You can do everything over our API or CLI, this could be as easy as a shell script. Or you can write this in Ruby or Python, we have API libraries for both languages.

Second option is to use foreman_hooks, this is a hacky plugin that allows starting executables (scripts) on various events. The rest would be the same however, it’s just a different way of starting the script. We are actually replacing this plugin with foreman_webhooks in the next Foreman release which will be much powerful and robust, foreman_hooks will still work but we will prefer and suggest to use webhooks instead.

If it was me, I would probably go for option (1) in Python or Ruby and run it as a polling service. Once webhooks are finished, I’d add new feature - when new host is discovered, the service would wake up and query discovered hosts so overall the whole process is faster. But you can live without this just fine, polling every one minute won’t hurt.

1 Like
3

Thank you so much Lukas! I will greatly consider the two solutions you mention. I am just happy to know that I’m starting to understand how this all works (still got a long way to go, but making progress).

I just want to say, I will admit I’m feeling overwhelmed with the idea of implementing Foreman, but folks like you and others have been extremely successful at making me feel more relaxed about it.

I’ve been spending a lot of time on this forum and the IRC channel. So far I have had a fantastic experience with this community.

1 Like
4

That’s sad to hear, we know Foreman is a beast and we are working hard on simplifying it in regard to features and deployment and also improving our documentation.

Take a visit at https://docs.theforeman.org/ where we have our new approach to documentation which is slightly more opinionated but has a better user-oriented content. It focuses on Foreman with Katello deployments (content management included), make sure to read through to understand how things work.

Glad to hear that, hopefully you stick with us!

5

You know, I really don’t think my initial feeling of being overwhelmed was really Foreman’s fault. I think it was mostly my own issues to overcome, if that makes sense.

With that, the documentation for me at first was a bit difficult to consume, but I did end up finding the docs.foreman.org site (thanks to Mcorr). Those docs (particularly the one of provisioning) really did make a world of difference with making me feel comfortable.

A very important part for me in regards to documentation is that it not only explains “how” to do something, but also tells “why”. I think the newer, more opinionated documentation hits the nail on the head in that regard.

Great work Foreman team!

2 Likes