Problem: When puppetserver connects to foreman using node.rb, node.rb takes over a minute to complete.
Expected outcome: node.rb when run manually takes a fraction of a second.
Foreman and Proxy versions: 3.13.1, but this is evident on 3.12.x as well.
Foreman and Proxy plugin versions: 3.13.1
Distribution and version: Debian 12 Bookworm. This issue was not evident prior to upgrading to Bookworm.
Other relevant data:
I dug into this quite a bit, and was seeing distinctly different behavior when running node.rb by hand and when puppetserver ran it. When puppetserver ran it, I could see the node.rb process in the list using 100% CPU for approximately a minute. Looking at it’s open files, it had many thousands of read filehandle to /etc/ssl/openssl.cnf.
Eventually, I added debugging and figured out that when openssl.so was loaded the process just hung for approximately a minute before continuing on. With a little bit of googling, that seems to possibly indicate some incompatibility in openssl libraries being utilized by ruby.
This lead me to force node.rb to use the same ruby interpreter as puppetserver, i.e. I changed the first line of node.rb to
#!/opt/puppetlabs/puppet/bin/ruby
Once that was done, puppet runs completed in the expected amount of time and there were no longer stuck node.rb processes.
I should note, I am using the puppet7 packages distributed by Perforce/Puppetlabs.