Puppet Server Classes Cache Disabled - but it isn't?

Support
,
1

Lately I’ve noticed my class imports are going super slow. Talking 20 minutes or so, with constant/frequent timeout failures causing me to have to retry. We have ~800 classes per environment across 20 environments (don’t ask why we have so many environments…) I know there is an issue around 1024 classes but I “should” be below that. Specifically this is Puppet 4.9.4 on Foreman 1.16.0 and RHEL 7.4

My Smart proxy log shows this frequently during a class import:
W, [2018-03-28T15:37:58.095007 bdabd58c] WARN -- : Puppet server classes cache is disabled, classes retrieval can be slow.

I take this to mean my classes cache is disabled - which I don’t believe to be true (see files below)

How can i “verify” whether this is a false positive log message (and why its triggering) or whether there is something I’m missing in terms of configuring all this to ensure the classes cache is used? Is there some API I can hit to verify whether the classes cache is truly enabled or not?

Note: these below files have been in place and all services (and servers) restarted numerous times without resolving it at this point, so i don’t believe it to be a service restart to update new settings type issue…

PuppetServer itself:

[root@fmnpmpro4 foreman-proxy]# cat /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf | grep environment-class-cache
environment-class-cache-enabled: true

Foreman Proxy’s puppet.yml

[root@fmnpmpro4 settings.d]# cat puppet.yml
---
# Puppet management
:enabled: https
# valid providers:
#   puppet_proxy_puppetrun   (for puppetrun/kick, deprecated in Puppet 3)
#   puppet_proxy_mcollective (uses mco puppet)
#   puppet_proxy_ssh         (run puppet over ssh)
#   puppet_proxy_salt        (uses salt puppet.run)
#   puppet_proxy_customrun   (calls a custom command with args)
#:use_provider: puppet_proxy_puppetrun

:puppet_version: 4.9.4

Foreman Proxy’s Puppet Proxy Legacy

[root@fmnpmpro4 settings.d]# cat puppet_proxy_legacy.yml
#
# puppet_proxy_legacy module is used for puppet versions prior to 4.0
#
# puppet_proxy_legacy is configured automatcially based on
# :puppet_version setting in smart-proxy's puppet.yml configuration file.
#
---
:puppet_conf: /etc/puppetlabs/puppet/puppet.conf
#
# Override use of Puppet's API to list environments, by default it will use only if
# environmentpath is given in puppet.conf, else will look for environments in puppet.conf
# (Puppet versions prior to 4.0 only)

#:use_environment_api: true

#
# URL of the puppet master itself for API requests. Required if puppet_use_environment_api is set to true.
:puppet_url: https://fmnpmpro4.<REMOVED>:8140
#
# SSL certificates used to access the environment API. Required if puppet_use_environment_api is set to true.
:puppet_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:puppet_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/fmnpmpro4.<REMOVED>.pem
:puppet_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/fmnpmpro4.<REMOVED>.pem
#
# Enable/disable puppet class cache
# Cache options
#:use_cache: true

Foreman Proxy puppet_proxy_puppet_api.yml

[root@fmnpmpro4 settings.d]# cat puppet_proxy_puppet_api.yml
#
# puppet_proxy_pupppet_api module is used for puppet versions 4.0 and higher
#
# puppet_proxy_pupppet_api is configured automatcially based on
# :puppet_version setting in smart-proxy's puppet.yml configuration file.
#
---
# URL of the puppet master itself for API requests.
:puppet_url: https://fmnpmpro4.<REMOVED>:8140
#
# SSL certificates used to access the puppet API
:puppet_ssl_ca: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:puppet_ssl_cert: /etc/puppetlabs/puppet/ssl/certs/host.pem
:puppet_ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/host.pem
:api_timeout: 180
2

I take this to mean my classes cache is disabled - which I don’t believe to be true (see files below)

It means that responses to environment-classes api coming from pupperserver are coming without etags, which indicates that classes cache is disabled on puppetserver. Please refer to puppet server docs (https://puppet.com/docs/puppetserver/5.1/config_file_puppetserver.html) for details on how to enable classes cache.

Please note that with classes cache enabled puppet server will need a restart after each change in classes.

3

Hey @Dmitri_Dolguikh

Apologies for my formatting above… I’ve been through that doc a few times - its definitely “enabled” to the extent that i have that line in the config file…

[root@fmnpmpro4 foreman-proxy]# cat /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf | grep environment-class-cache
environment-class-cache-enabled: true

Im re-reading all the docs you linked as well in case i missed something. puppetserver Service has been restarted probably 15x since i’ve begun playing with this today…

4

You can try manually accessing pupperserver to see what you are getting back:

curl -v http://server:port/puppet/v3/environment_classes?environment=environment_name

If the etag isn’t present in the response, puppet classes cache is disabled. Otherwise, there’s something weird going on with smart-proxy and we’ll need to look into it.

Please note that etag is a header and not in the main body of the response. Please also note that you may need to adjust access controls on the api end-point to be able to access it.

5

Okay - so took me a bit to figure out how to present the cert, ca, and key but finally got it working.

Connecting back to “myself” i get the following (partial) curl response:

< HTTP/1.1 200 OK
< Date: Wed, 28 Mar 2018 21:26:10 GMT
**< ETag: cbdc4c17df8ad25f0d3ef7179d4e0e03c53c2584**
< Content-Type: application/json
< X-Puppet-Version: 4.9.4
< Transfer-Encoding: chunked
< Server: Jetty(9.2.z-SNAPSHOT)
<
{"files":[{"classes" (SNIPPED CUZ ITS LONG)

Similarly, I was able to do several sequential class “imports” without selecting anything that completed in 3-5 minutes with no errors in the logs regarding cache being disabled.

With this said - i have a new “question” of sorts. How long does it take for the class cache to “regenerate”. I know we are calling the api properly to “clear” the cache anytime we publish new code, and we then (within a few minutes) try to import those changes. This takes forever, and the logs are full of the “cache disabled” messaegs.
If the regeneration of the cache is taking “too long” perhaps that is the root of my issue? is there a way to see cache (re)generation progress/steps? or would it simply be "repeatedly hit that endpoint until it returns an ETAG?

6

How long does it take for the class cache to “regenerate”

Current version of smart-proxy updates the cache when the etag it receives from puppet-server is different from the one it has.

This takes forever,

Each environment needs to be fetched from the puppet-server, which might take some time if the number of environments is large.

and the logs are full of the “cache disabled” messaegs.

I’m not sure what the exact reason for this is, but my guess would be that puppet-server calculates etags after it loads classes into memory. As I mentioned before, “cache disabled” message is logged only when response doesn’t contain etag header.

Please note there’s a PR opened in smart-proxy github repository that changes caching behaviour to limit the number of environments that are kept in the cache. With that change, once the number of environment exceeds the threshold, the one that was accessed least recently will be discarded. If an environment is present in the cache, the associated etag will be checked to see if an update is required.

7

is there a way to see cache (re)generation progress/steps? or would it simply be "repeatedly hit that endpoint until it returns an ETAG?

I don’t know if there’s a way to check on puppet-server internal state. Might be worth checking with puppet developers, they might clarify server’s caching behaviour too.