Handling lots of Puppet environments/classes

Support
1

Problem:
Okay, this is a bit more abstract than my usual support questions, so bear with me.

I’m evaluating Foreman in various potential roles at work, including the “all-in-one” kind of role. It would be great to have Foreman managing Puppet for us, and indeed that all actually works great. The major bug-bear I’ve discovered today, however, is that it takes between 12-15 minutes to import my full set of environments and classes (by pressing the “Import environments from hostname.someplace.com” button).

This isn’t an unreasonable amount of time - I would love if it was a lot faster, but considering it’s importing the entire set of branches it’s pretty good.

The problem that arises is that our workflow usually involves creating a new branch in the control-repo tied to the issue/project we’re trying to solve/work on. In order for me to enable this workflow, I would need to have the user create their branch and then go trigger an import. This is because in order for them to work on a test machine, they would need to assign that environment (and/or role) to a host or hostgroup.

Would it be possible to implement some sort of system where I can scan/import a single environment at a time? In that scenario, they can simply import the new branch they created, which should be very quick compared to scanning/importing 80+ environments with a few hundred classes!

Just thoughts for now, please let me know what you think.

One other enhancement I’d love to see - being able to add Puppet classes to a node or host group that Foreman isn’t yet aware of. This way I could create a new role and flesh it out and test it without needing to do a long full environment import. Then again, being able to import/update a single environment would make this unnecessary.
Foreman and Proxy versions:
1.22.0

Foreman and Proxy plugin versions:
1.22.0

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)

Nothing relevant available yet
2

For what it’s worth - I’m using g10k to sync the various environments to my Puppet environments dir, which works really well.

3

you can import classes from a single environment. in the configure->puppet environments page, click on the dropdown in the button for the environment you want to import from and select “import classes frome $env”

1 Like
4

Ah, cool! That solves my issue for existing environments, so for changing roles/profiles/classes in an existing environment that’s perfect :slight_smile:

Unfortunately it doesn’t help in the case where I need to import a new environment Foreman hasn’t seen before.

Ideally we could cache the list of environments and only import ones we think have changed…

5

you can also manually create environments if you know their name.
I think that changing to the puppet rest api on the proxy side should improve the import performance, but perhaps @ekohl will know better.

1 Like
6

I do wonder how long the listing of environments actually takes. This should just be a REST call to the Puppetserver in the current implementation with minimal code behind it.

Can you try getting https://proxy.example.com/puppet/environments and seeing how long that takes? Note you may need to pass in the right certificates for authentication. Introducing fp-curl has some steps there.

I suspect the problem might be in Foreman and retrieving the classes as well and comparing to what’s in the database. Foreman :: Manual has a workaround that allows you to filter classes by name. This allows you to exclude many, like params, install, config and service. Personally I only import my profiles module:

:filters:
  - !ruby/regexp '/^(?!profiles::)/'

This also makes the assignment UI a lot more usable because there’s no irrelevant classes.

Have you considered using the API after g10k is finished?
https://theforeman.org/api/1.22/apidoc/v2/environments/import_puppetclasses.html

1 Like
7

Sure thing. I’ve had to redact the names of the environments unfortunately, but there’s 104 of them and all get outputted by that command. Log below (but sanitized)

[root@fm-master5 fmtools]# time ./fp-curl /puppet/environments
["wings","wings_fmadd"]

real	0m0.279s
user	0m0.116s
sys	0m0.064s

Awesome, that works a treat!

  • User creates their branch
  • g10k runs and sets up that new environment
  • Go into Foreman and create the environment
  • Import classes for just that environment
  • Profit!

I would like to suggest that the wording on the dropdown could be better - rather than “Import classes from fm-master5.riff.cc” it could say something like “Import classes from production on fm-master5.riff.cc” or just “Import classes from this environment”.

8

Is it possible to do that but filter for just profiles and roles? I kinda need both and I’m hopeless at regex :slight_smile:

9

Sounds like implementing caching on the Puppetserver would actually not solve anything because 0.2 seconds is plenty fast.

It’s actually the UI that always retrieves all the data for all environments that makes this slow. A view where you first retrieve all the environments without the changes and only show changes for selected environments would probably solve a lot.

Untested, but something like this probably works:

:filters:
  - !ruby/regexp '/^(?!(profiles|roles)::)/'