Effectively Integrating Hiera, Git, Foreman, and Puppet?

Support
4

Hello again!

After discussion with numerous people, I was able to devise a solution to this that is true to my original intent, doesn't add unnecessary automation, and should be scalable and modifiable if necessary. I want to thank Nick Maludy for filling in a couple of gaps in my Puppet/Hiera knowledge and a coworker that wishes to remain nameless for writing some array munging code in the site.pp file for their help.

TL;DR: Mapped Paths

The Missing Link

What I didn't realize and what Nick helped me understand was that if a '/' is in a variable that is used for Hiera, it is treated as a path specifier, meaning that "alpha/beta/gamma" will look in a folder "alpha" and then in a folder "beta" for gamma.yaml (with the paths mentioned in the parent post). For Foreman, this means that for the hostgroup path, you could always obtain the file at the end of the hostgroup, so "infrastructure_server/openstack" as a hostgroup would resolve to "nodes/host_group/infrastructure_server/openstack.yaml". This solves part of the problem, as we can now give definitions for the leaves of the tree, but doesn't inherit properly, which leads to the other missing piece.

Mapped Paths

Mapped Paths provide the other necessary component. As a refresher if you haven't seen them, they look like this in hiera.yaml: 
- name: "Some Mapped Path Group"
  mapped_paths: [array_of_strings, tmp, "folder/%{tmp}.yaml"]
Puppet takes in three strings, the first which should resolve to an array when treated as a variable. The second string is used for interpolation in a path, which is what the third string is for. In the above case, for every entry in the array, a yaml file in "folder" would be searched for configuration info. So for the array ["alpha", "beta", "gamma"], we would expect these paths: 
folder/alpha.yaml
folder/beta.yaml
folder/gamma.yaml
This sort-of solves the problem with the first possible solution I posted, as you don't have to have the massive filenames. But, you still end up with the other problem as described, that is, you have to have a bunch of files that are all on the same level and aren't sorted in a meaningful way.

Putting it all together...

After some misunderstandings about the right character to use in Hiera (%, not $) and scoping (top versus node level in site.pp), we came to a solution. By setting a variable before the 'node "default"' block of the site.pp file, we're able to set a top-scoped array that is used in the variable interpolation. 
if $hostgroup {
        $tmp_groups = split($hostgroup, '/')
        $tmp_path_array = $tmp_groups.map |Integer $index, String $group| {
                $tmp_groups[0, $index + 1]
        }
        $tmp_final_path_array = $tmp_path_array.map |Array $paths| {
                join($paths, '/')
        }
}
$sorted_path_array = reverse($tmp_final_path_array)

This code generates an array of each level of the hierarchy. So to continue the earlier example of “infrastructure_server/openstack/controller”, we generate an array with the following values:

- "infrastructure_server/openstack/controller"
- "infrastructure_server/openstack"
- "infrastructure_server"

We reverse the array to ensure that it’s most to least specific to correspond with the rest of the Hiera hierarchy.

Final Thoughts

I understand that all of the above might be seen as too complex, and is likely to have some issues that I haven't foreseen (Dirk makes some good points in a response). However, based on the goals that we have, I think this solves the problem neatly and utilizes the best parts of each of the various components (Foreman, Puppet, Hiera, Git). I will get notifications on this issue, so would be happy for any feedback of any step or any other recommendations. Thank you to the Foreman project for their tireless work on an incredibly complex system.
4 Likes