Connection reset by peer - SSL_connect Wrapped exception:

Problem:

‘Hosts in error state’ intermittently reports the below. We have over 2000 hosts checking in, but see >300 nodes report ‘SSL_connect issues’ on a daily basis. The alert then clears without intervention. We’re can’t pin this down to specific hosts.

Request to https://proxy01:8140/puppet/v3 failed after 107.113 seconds: Connection reset by peer - SSL_connect
Wrapped exception:
Connection reset by peer - SSL_connect
warning	Puppet	Unable to fetch my node definition, but the agent run will continue:
warning	Puppet	No more routes to puppet

Expected outcome:

All our nodes report without connection issues.

Foreman and Proxy versions:

Foreman and Proxy version 1.24.3

Foreman and Proxy plugin versions:

foreman-tasks 0.17.5
foreman_ansible 4.0.5
foreman_bootdisk 16.0.0
foreman_dhcp_browser 0.0.8
foreman_discovery 16.0.1
foreman_docker 5.0.0
foreman_hooks 0.3.16
foreman_remote_execution 2.0.8
foreman_rescue 2.0.1
katello 3.14.1
puppetdb_foreman 5.0.0

Distribution and version:

Foreman Server runs on CentOS 7.8
Smart Proxies are on CentOS 7.7

Other relevant data:

We have 3 x Smart Proxies running Puppet Server 6.16.0, our clients also run the same version.

Proxy01 - 909 Agents Managed
Proxy02 - 966 Agents Managed
Proxy03 - 228 Agents Managed

Proxy01 and Proxy02 show the SSL Connect Issues. Proxy03 shows no issues.

I suspect we have a load issues on our Proxies that’s causing dropped Connections back to Foreman.

We’d really like some help to confirm and remedy the situation. Our config files are as below for httpd, prefork and SSL.

Proxies:
/etc/httpd/conf/httpd.conf
KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 5

Foreman:
/etc/httpd/conf.modules.d/prefork.conf
StartServers 10
MinSpareServers 5
MaxSpareServers 20
ServerLimit 512
/etc/httpd/conf.d/05-foreman-ssl.conf
KeepAlive on
KeepAliveTimeout 5
MaxKeepAliveRequests 10000

Any help would be appreciated. Please let me know if further information is needed.

Thanks!