**Problem: We run puppet with foreman and i see lot of performance issues with agent run. Master goes hung state when agents run starts build up.
Here is our infra setup style: We have a main foreman server then regional foreman-proxy. There is no loadbalancing or proxy setup. Agent connects to regional foreman-proxy running puppet server. Note: few hundred agent connects to foreman server also.
**Expected outcome:
**Foreman and Proxy versions: Foreman and proxy version 2.0.3. Puppet version: 6.7.2, Puppet server version:puppetserver-6.12
Foreman and Proxy plugin versions:
**Distribution and version: CentOS 7.8
**Other relevant data:
cat /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf
Managed by Puppet
configuration for the JRuby interpreters
jruby-puppet: {
# Where the puppet-agent dependency places puppet, facter, etc…
# Puppet server expects to load Puppet from this location
ruby-load-path: [
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby,
]
# This setting determines where JRuby will install gems. It is used for loading gems,
# and also by the `puppetserver gem` command line tool.
gem-home: /opt/puppetlabs/server/data/puppetserver/jruby-gems
# This setting defines the complete "GEM_PATH" for jruby. If set, it should include
# the gem-home directory as well as any other directories that gems can be loaded
# from (including the vendored gems directory for gems that ship with puppetserver)
gem-path: [${jruby-puppet.gem-home}, "/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems", "/opt/puppetlabs/puppet/lib/ruby/vendor_gems"]
# PLEASE NOTE: Use caution when modifying the below settings. Modifying
# these settings will change the value of the corresponding Puppet settings
# for Puppet Server, but not for the Puppet CLI tools. This likely will not
# be a problem with master-var-dir, master-run-dir, or master-log-dir unless
# some critical setting in puppet.conf is interpolating the value of one
# of the corresponding settings, but it is important that any changes made to
# master-conf-dir and master-code-dir are also made to the corresponding Puppet
# settings when running the Puppet CLI tools. See
# https://docs.puppetlabs.com/puppetserver/latest/puppet_conf_setting_diffs.html#overriding-puppet-settings-in-puppet-server
# for more information.
# (optional) path to puppet conf dir; if not specified, will use
# the puppet default
master-conf-dir: /etc/puppetlabs/puppet
# (optional) path to puppet code dir; if not specified, will use
# the puppet default
master-code-dir: /etc/puppetlabs/code
# (optional) path to puppet var dir; if not specified, will use
# the puppet default
master-var-dir: /opt/puppetlabs/server/data/puppetserver
# (optional) path to puppet run dir; if not specified, will use
# the puppet default
master-run-dir: /var/run/puppetlabs/puppetserver
# (optional) path to puppet log dir; if not specified, will use
# the puppet default
master-log-dir: /var/log/puppetlabs/puppetserver
# (optional) maximum number of JRuby instances to allow
max-active-instances: 31
# (optional) the number of HTTP requests a given JRuby instance will handle in its lifetime.
max-requests-per-instance: 100000
# (optional) The maximum number of requests that may be queued waiting to borrow a JRuby from the pool.
max-queued-requests: 500
# (optional) Sets the upper limit for the random sleep set as a Retry-After header on 503 responses returned when max-queued-requests is enabled.
max-retry-delay: 1800
# (optional) Authorize access to Puppet master endpoints via rules
# specified in the legacy Puppet auth.conf file (if true) or via rules
# specified in the Puppet Server HOCON-formatted auth.conf (if false or not
# specified).
use-legacy-auth-conf: false
# (optional) enable or disable environment class cache
environment-class-cache-enabled: false
}
settings related to HTTPS client requests made by Puppet Server
http-client: {
# A list of acceptable protocols for making HTTPS requests
ssl-protocols: [
TLSv1.2,
]
# A list of acceptable cipher suites for making HTTPS requests
cipher-suites: [
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
]
# Whether to enable http-client metrics; defaults to 'true'.
metrics-enabled: true
# The amount of time, in milliseconds, that an outbound HTTP connection
# will wait for data to be available before closing the socket. If not
# defined, defaults to 20 minutes. If 0, the timeout is infinite and if
# negative, the value is undefined by the application and governed by the
# system default behavior.
idle-timeout-milliseconds: 1200000
# The amount of time, in milliseconds, that an outbound HTTP connection will
# wait to connect before giving up. Defaults to 2 minutes if not set. If 0,
# the timeout is infinite and if negative, the value is undefined in the
# application and governed by the system default behavior.
connect-timeout-milliseconds: 120000
}
settings related to profiling the puppet Ruby code
profiler: {
# enable or disable profiling for the Ruby code;
enabled: true
}
cat /etc/sysconfig/puppetserver
###########################################
Init settings for puppetserver
###########################################
Location of your Java binary (version 8)
JAVA_BIN=/usr/bin/java
Modify this if you’d like to change the memory allocation, enable JMX, etc
JAVA_ARGS="-Xms32G -Xmx32G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"
Modify this as you would JAVA_ARGS but for non-service related subcommands
JAVA_ARGS_CLI="${JAVA_ARGS_CLI:-}"
Modify this if you’d like TrapperKeeper specific arguments
TK_ARGS=""
These normally shouldn’t need to be edited if using OS packages
USER=“puppet”
GROUP=“puppet”
INSTALL_DIR="/opt/puppetlabs/server/apps/puppetserver"
CONFIG="/etc/puppetlabs/puppetserver/conf.d"
Bootstrap path
BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/"
SERVICE_STOP_RETRIES can be set here to alter the default stop timeout in
seconds. For systemd, the shorter of this setting or ‘TimeoutStopSec’ in
the systemd.service definition will effectively be the timeout which is used.
SERVICE_STOP_RETRIES=60
START_TIMEOUT can be set here to alter the default startup timeout in
seconds. For systemd, the shorter of this setting or ‘TimeoutStartSec’
in the service’s systemd.service configuration file will effectively be the
timeout which is used.
START_TIMEOUT=300
Maximum number of seconds that can expire for a service reload attempt before
the result of the attempt is interpreted as a failure.
RELOAD_TIMEOUT=120