Problem: We’re rolling out CentOS using Foreman to a large amount of systems. Last night during a deployment, the Foreman server starting logging the below error:
2021-10-29T04:27:18 [E|kat|5866a427] could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use
2021-10-29T04:27:18 [E|kat|5866a427] Lost database connection. Attempting reconnect.
2021-10-29T04:27:26 [E|kat|5866a427] Trying to reconnect to the database.
2021-10-29T04:27:27 [I|app|e8829374] Started GET "/rhsm/" for 127.0.0.1 at 2021-10-29 04:27:27 -0400
2021-10-29T04:27:27 [I|app|e8829374] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2021-10-29T04:27:27 [I|app|e8829374] Parameters: {"root"=>{}}
2021-10-29T04:27:27 [I|app|e8829374] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.3/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2021-10-29T04:27:27 [I|app|e8829374] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.3/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (Duration: 3.0ms | Allocations: 41
45)
2021-10-29T04:27:27 [I|app|e8829374] Completed 200 OK in 18ms (Views: 4.9ms | ActiveRecord: 2.7ms | Allocations: 15514)
2021-10-29T04:27:27 [I|app|e9cfa9f7] Started GET "/rhsm/status" for 127.0.0.1 at 2021-10-29 04:27:27 -0400
2021-10-29T04:27:27 [I|app|e9cfa9f7] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2021-10-29T04:27:34 [E|kat|5866a427] Trying to reconnect to the database.
2021-10-29T04:27:34 [I|app|54305794] Started GET "/rhsm/consumers/9d67846e-7867-4ee8-8b90-31ceb15fc62d" for 127.0.0.1 at 2021-10-29 04:27:34 -0400
2021-10-29T04:27:34 [I|app|54305794] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_show as JSON
2021-10-29T04:27:34 [I|app|54305794] Parameters: {"id"=>"9d67846e-7867-4ee8-8b90-31ceb15fc62d"}
2021-10-29T04:27:35 [I|app|ad797234] Started GET "/rhsm/" for 127.0.0.1 at 2021-10-29 04:27:35 -0400
2021-10-29T04:27:35 [I|app|ad797234] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2021-10-29T04:27:35 [I|app|ad797234] Parameters: {"root"=>{}}
2021-10-29T04:27:35 [I|app|ad797234] Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.3/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2021-10-29T04:27:35 [I|app|ad797234] Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.3/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (Duration: 3.0ms | Allocations: 41
45)
2021-10-29T04:27:35 [I|app|ad797234] Completed 200 OK in 29ms (Views: 5.4ms | ActiveRecord: 7.5ms | Allocations: 18619)
2021-10-29T04:27:36 [I|app|8bf0783a] Started GET "/rhsm/consumers/df35830d-f874-4e84-aa6a-ff42794352f0/certificates/serials" for 127.0.0.1 at 2021-10-29 04:27:36 -0400
2021-10-29T04:27:36 [I|app|8bf0783a] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2021-10-29T04:27:36 [I|app|8bf0783a] Parameters: {"id"=>"df35830d-f874-4e84-aa6a-ff42794352f0"}
2021-10-29T04:27:38 [I|app|5827ffa0] Started GET "/rhsm/consumers/ded35944-2c2a-4c5c-a3c5-1838b97bf770" for 127.0.0.1 at 2021-10-29 04:27:38 -0400
2021-10-29T04:27:38 [I|app|5827ffa0] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_show as JSON
2021-10-29T04:27:38 [I|app|5827ffa0] Parameters: {"id"=>"ded35944-2c2a-4c5c-a3c5-1838b97bf770"}
2021-10-29T04:27:39 [E|kat|54305794] ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:221:in `block in wait_poll'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:210:in `loop'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:210:in `wait_poll'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:171:in `internal_poll'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:297:in `internal_poll'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:166:in `block in poll'
54305794 | /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:175:in `synchronize'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:166:in `poll'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:875:in `acquire_connection'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:593:in `checkout'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:437:in `connection'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1119:in `retrieve_connection'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:221:in `retrieve_connection'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:189:in `connection'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/delegation.rb:92:in `connection'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/query_methods.rb:1298:in `preprocess_order_args'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/query_methods.rb:364:in `order!'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/query_methods.rb:359:in `order'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/finder_methods.rb:554:in `ordered_relation'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/finder_methods.rb:519:in `find_nth_with_limit'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/finder_methods.rb:512:in `find_nth'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/relation/finder_methods.rb:120:in `first'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-session_store-1.1.1/lib/active_record/session_store/session.rb:58:in `find_by_session_id'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-session_store-1.1.1/lib/action_dispatch/session/active_record_store.rb:124:in `block in get_session_model'
54305794 | /usr/share/foreman/lib/foreman/silenced_logger.rb:37:in `silence'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-session_store-1.1.1/lib/action_dispatch/session/active_record_store.rb:123:in `get_session_model'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-session_store-1.1.1/lib/action_dispatch/session/active_record_store.rb:140:in `find_session'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:314:in `load_session'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/session/abstract_store.rb:45:in `block in load_session'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/session/abstract_store.rb:53:in `stale_session_check!'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/session/abstract_store.rb:45:in `load_session'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/request/session.rb:234:in `load!'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/request/session.rb:230:in `load_for_write!'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/request/session.rb:132:in `[]='
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/gettext_i18n_rails-1.8.0/lib/gettext_i18n_rails/action_controller.rb:5:in `set_gettext_locale'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:201:in `block in halting'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `block in invoke_before'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `each'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:513:in `invoke_before'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:110:in `block in run_callbacks'
54305794 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:181:in `disable_json_root'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
54305794 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
54305794 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
54305794 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `block in instrument'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `instrument'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.3/app/controllers/katello/concerns/api/api_controller.rb:40:in `process_action'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in `process'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in `process'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in `dispatch'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in `dispatch'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.3/lib/katello/middleware/event_daemon.rb:10:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.2.9/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
54305794 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
54305794 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
54305794 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
54305794 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.3/lib/katello/prevent_json_parsing.rb:12:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
54305794 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-29T04:27:39 [I|app|54305794] Completed 500 Internal Server Error in 5003ms (Views: 0.1ms | ActiveRecord: 0.0ms | Allocations: 707)
After this, any system that would reboot would attempt to PXE, load iPXE and then get stuck at a screen stating “Press Any Key to Reboot”. Note I am working with remote locations that I do not have physical access to, so sadly that is all the information I have about what is going on with the client. I am trying to get a screenshot/camera image of it. I don’t know yet if this message is from iPXE, the BIOS/UEFI, or what…
Luckily, after rebooting our Foreman server, it is stable for now. But I am concerned that this could happen again.
My uneducated concern is that I am now at 581 systems in Foreman. Every morning, a large chunk of these systems reboot per a daily scheduled task, starting around 3:00 AM. Is it possible that I have passed some kind of threshold where Foreman isn’t able to handle all of these concurrent connections?
Sorry for the lack of information! Please let me know what ever I can provide to help with troubleshooting. Trying to investigate, but also work production issues at the same time.
Expected outcome: The Foreman server remains functionable.
Foreman and Proxy versions: 2.2.3
Foreman and Proxy plugin versions: Katello 3.17.3
Distribution and version: CentOS 7
Other relevant data: