[LONG] Performance problem with puppet agent runs and foreman-proxy

hi *,

we just upgraded our foreman to 1.9.3 (from 1.7 via 1.8) and noticed that
foreman-proxy is kind of a bottleneck for us.

quick specs:

  • rhel 6
  • foreman 1.9.3
  • puppet 3.7.4
  • puppetserver 1.1.3
  • puppetdb 2.3.8
  • about 1400 nodes
  • one master (vmware 8 cores 24GB ram)
  • foreman as an enc + report and fact upload
  • foreman-proxy modules: dns, dhcp, tftp and puppet
  • foreman via apache and mod_passenger

on every puppet run foreman queries the dhcp settings from foreman-proxy:

D, [2015-12-17T10:37:17.055401 #19355] DEBUG – : Reading config file /etc/dhcp/dhcpd.conf
D, [2015-12-17T10:37:17.056840 #19355] DEBUG – : Loading subnets for 127.0.0.1
D, [2015-12-17T10:37:17.063957 #19355] DEBUG – : Added appngd5.eb.lan.at (10.18.209.35 / 00:50:56:bc:01:9f) to 10.18.209.0/255.255.255.0
D, [2015-12-17T10:37:17.066466 #19355] DEBUG – : Reading config file /var/lib/dhcpd/dhcpd.leases
D, [2015-12-17T10:37:17.065380 #19355] DEBUG – : Added 10.198.71.0/255.255.255.0 to 127.0.0.1
D, [2015-12-17T10:37:17.201748 #19355] DEBUG – : Added 10.251.130.0/255.255.255.240 to 127.0.0.1
.
.
.
I, [2015-12-17T10:37:17.204382 #19355] INFO – : Enumerated hosts on 10.18.209.0

so foreman-proxy reads /etc/dhcp/dhcpd.conf and
/var/lib/dhcpd/dhcpd.leases on every puppet run. as we have only one
foreman-proxy instance it eats one cpu and blocks the puppetserver.

is there something wrong with our setup or is this intended? i know that
interface handling changed between 1.7 and 1.9 maybe there's an issue
with our setup…

any help would be much appreciated

thanks
toni