Currently on Foreman 1.16.0 (Puppet 4.9.4) Been having issues with class imports for a while, getting slower and slower as we add environments/code, ever since Foreman 1.14 and on. We have 20-some-odd environments with around 200-800 classes per environment. this makes class imports super duper slow. That being said, lately I’ve seen a ton of “503 service unavailable” errors followed by proxy.log entries about foreman cache being “disabled”. If i try it enough, it will work without failing (but still takes like 10 minutes to import all environments).
Playing with it more, I soon realized that I could simply “restart” the foreman-proxy to re-generate all caches. I’ve updated my Puppet Code update process to “restart” the foreman-proxy service if code is changed (after triggering puppet to update on-disk via the Puppet API), and have virtually eliminated all of my 503 failures on import with corresponding “no cache available” errors in proxy.log. So this is definitely progress
My restart log (Debug Enabled - but filtered slightly) shows all caches are properly generated on “startup” it’s only when code changes that the proxy would have issues…
Pastebin due to character limit in post (Logs are big): https://pastebin.com/Ajw21uG5
I don’t think a restart should be required, but foreman-proxy doesn’t seem to support a “reload” and i can’t find a rake task that would exist to “Initlalize Puppet Class Caches” which might be a better solution to “call” whenever code is changed.
I also looked into this deeper and am perplexed by the logs generated WHEN I do an import. Mainly:
- Multiple identical “Get Environment” API calls are done for each environment imported
At minimum - I’d expect one “GET /puppet/environments” puppet environment imported, but realistically it could probably be called “once” and cached/referred to for the remainder of the import? I see no reason why this should generate 3 “GET /puppet/environments” calls per environment imported?
Additionally - can this be done in parallel? I see nothing that would prevent this from, say, looking up how many cores are available, and performing 2*N CPU core imports simultaneously, or at least make it configurable?
Or maybe i’m misunderstanding this?
Pastebin due to character limit in post (logs are big): https://pastebin.com/BfaLjQAh