Problem: When running /etc/puppetlabs/puppet/node.rb example.com we are getting constrained by the database every now and again.
During the fact upload the server responded with: 500 Internal Server Error . Error is ignored and the execution continues.
{
"error": {"message":"could not obtain a connection from the pool within 5.000 seconds (waited 5.003 seconds); all pooled connections were in use"}
}
/etc/puppetlabs/puppet/node.rb:383: warning: constant ::TimeoutError is deprecated
I have edited /etc/foreman/database.yml to have a pool of 50 connections and timeout 10 seconds but it is not listening to those settings, not sure how to let foreman handle that amount of connctions to postgresql, cant find it in the docs. Below is the database config: (postgres server is happy)
production:
adapter: postgresql
username: bob
database: foreman
password: foobarbaz
host: puppetdb.example.com
encoding: unicode
pool: 50
timeout: 10
Expected outcome: Database should be able to handle the connections
Foreman and Proxy versions: 1.19 foreman and foreman-proxy
Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)
1844985: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.098 seconds); all pooled connections were in use
1845102: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.230 seconds); all pooled connections were in use
1845224:2018-10-24T18:07:42 [F|app|] ActionView::Template::Error (could not obtain a connection from the pool within 5.000 seconds (waited 5.413 seconds); all pooled connections were in use):
1845245:2018-10-24T18:07:42 [F|app|] ActionView::Template::Error (could not obtain a connection from the pool within 5.000 seconds (waited 5.087 seconds); all pooled connections were in use):
1845271: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.007 seconds); all pooled connections were in use
1845404:2018-10-24T18:07:46 [F|app|] ActionView::Template::Error (could not obtain a connection from the pool within 5.000 seconds (waited 5.004 seconds); all pooled connections were in use):
1845421: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use
1845547: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use
1845663:2018-10-24T18:07:47 [F|app|] ActionView::Template::Error (could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use):
1845668: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use
1845812: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.013 seconds); all pooled connections were in use
1845930: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use
1846077: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use
1846197: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use
1846334: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use
1846537: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use
1846704: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.004 seconds); all pooled connections were in use
1846829: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.004 seconds); all pooled connections were in use
1846967: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.035 seconds); all pooled connections were in use
1847087: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.049 seconds); all pooled connections were in use
1847205: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.020 seconds); all pooled connections were in use
1847326: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use
1847446: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.017 seconds); all pooled connections were in use
1847585: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.003 seconds); all pooled connections were in use
1847726: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use
1847853: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.044 seconds); all pooled connections were in use
1847984: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use
1848120: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.006 seconds); all pooled connections were in use
1848244: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.006 seconds); all pooled connections were in use
1848364: | ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.003 seconds); all pooled connections were in use