Tomcat stops responding after several hours. 502 Errors

Support
1

Problem:
Tomcat behave erratic after few hours. GUI login stops working giving out proxy error 502. Candlepin events errors in 502, hammer ping error 502. Restarting just Tomcat fixes this issues for a few hours. Restarting Tomcat (stop phase) takes very long time.

Issue startet after updating from 2.0.0/1.17rc2 to 2.0.0/1.17

Expected outcome:
No 502 errors all over the place.

Foreman and Proxy versions:
Foreman 2.0.0
foreman-proxy 2.0.0

Foreman and Proxy plugin versions:

candlepin-3.1.22-1.el7.noarch
candlepin-selinux-3.1.22-1.el7.noarch
foreman-2.2.0-1.el7.noarch
foreman-cli-2.2.0-1.el7.noarch
foreman-debug-2.2.0-1.el7.noarch
foreman-dynflow-sidekiq-2.2.0-1.el7.noarch
foreman-ec2-2.2.0-1.el7.noarch
foreman-installer-2.2.0-1.el7.noarch
foreman-installer-katello-2.2.0-1.el7.noarch
foreman-ovirt-2.2.0-1.el7.noarch
foreman-postgresql-2.2.0-1.el7.noarch
foreman-proxy-2.2.0-1.el7.noarch
foreman-release-2.2.0-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-2.2.0-1.el7.noarch
foreman-service-2.2.0-1.el7.noarch
foreman-telemetry-2.2.0-1.el7.noarch
foreman-vmware-2.2.0-1.el7.noarch
foreman.x.x.x-apache-1.0-1.noarch
foreman.x.x.x-foreman-client-1.0-1.noarch
foreman.x.x.x-foreman-proxy-1.0-1.noarch
foreman.x.x.x-foreman-proxy-client-1.0-1.noarch
foreman.x.x.x-puppet-client-1.0-1.noarch
foreman.x.x.x-qpid-broker-1.0-2.noarch
foreman.x.x.x-qpid-client-cert-1.0-1.noarch
foreman.x.x.x-qpid-router-client-1.0-1.noarch
foreman.x.x.x-qpid-router-server-1.0-1.noarch
foreman.x.x.x-tomcat-1.0-1.noarch
katello-3.17.0-1.el7.noarch
katello-certs-tools-2.7.1-2.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.17.0-1.el7.noarch
katello-debug-3.17.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-3.17.0-1.el7.noarch
katello-selinux-3.4.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.14.1-1.el7.noarch
pulp-admin-client-2.21.4-2.el7.noarch
pulp-client-1.0-1.noarch
pulp-consumer-client-2.21.4-2.el7.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.8-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.4-1.el7.noarch
pulp-puppet-tools-2.21.4-1.el7.noarch
pulp-rpm-admin-extensions-2.21.4-1.el7.noarch
pulp-rpm-consumer-extensions-2.21.4-1.el7.noarch
pulp-rpm-handlers-2.21.4-1.el7.noarch
pulp-rpm-plugins-2.21.4-1.el7.noarch
pulp-rpm-yumplugins-2.21.4-1.el7.noarch
pulp-selinux-2.21.4-2.el7.noarch
pulp-server-2.21.4-2.el7.noarch
pulpcore-selinux-1.0.0-2.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python-pulp-agent-lib-2.21.4-2.el7.noarch
python-pulp-bindings-2.21.4-2.el7.noarch
python-pulp-client-lib-2.21.4-2.el7.noarch
python-pulp-common-2.21.4-2.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python-pulp-docker-common-3.2.8-1.el7.noarch
python-pulp-oid_validation-2.21.4-2.el7.noarch
python-pulp-puppet-common-2.21.4-1.el7.noarch
python-pulp-repoauth-2.21.4-2.el7.noarch
python-pulp-rpm-common-2.21.4-1.el7.noarch
python-pulp-streamer-2.21.4-2.el7.noarch
python2-qpid-1.37.0-4.el7.noarch
python2-qpid-proton-0.32.0-2.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.5.0-1.el7.noarch
python3-pulp-certguard-1.0.2-1.el7.noarch
python3-pulp-container-2.0.1-1.el7.noarch
python3-pulp-file-1.2.0-1.el7.noarch
python3-pulp-rpm-3.6.2-1.el7.noarch
python3-pulpcore-3.6.3-2.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.14.0-1.el7.x86_64
qpid-proton-c-0.32.0-2.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.6.13-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.1-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.1-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.1-1.el7.noarch
tfm-rubygem-actionview-6.0.3.1-1.el7.noarch
tfm-rubygem-activejob-6.0.3.1-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.1-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.1-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.2.9-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-arel-9.0.0-4.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-awesome_print-1.8.0-5.el7.noarch
tfm-rubygem-azure_mgmt_compute-0.18.7-1.el7.noarch
tfm-rubygem-azure_mgmt_network-0.19.0-1.el7.noarch
tfm-rubygem-azure_mgmt_resources-0.17.6-1.el7.noarch
tfm-rubygem-azure_mgmt_storage-0.17.10-1.el7.noarch
tfm-rubygem-azure_mgmt_subscriptions-0.18.2-1.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-coffee-rails-5.0.0-1.el7.noarch
tfm-rubygem-coffee-script-2.4.1-4.el7.noarch
tfm-rubygem-coffee-script-source-1.12.2-4.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-declarative-0.0.10-2.el7.noarch
tfm-rubygem-declarative-option-0.1.0-2.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-diffy-3.0.1-5.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.7-1.fm2_2.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-ethon-0.12.0-1.el7.noarch
tfm-rubygem-eventmachine-1.0.6-4.el7.x86_64
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-execjs-2.7.0-4.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.15.4-2.el7.noarch
tfm-rubygem-faraday-cookie_jar-0.0.6-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-aws-3.6.5-1.el7.noarch
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-digitalocean-0.4.0-2.el7.noarch
tfm-rubygem-fog-google-1.8.2-2.el7.noarch
tfm-rubygem-fog-json-1.2.0-3.el7.noarch
tfm-rubygem-fog-openstack-1.0.8-3.el7.noarch
tfm-rubygem-fog-ovirt-1.2.5-1.el7.noarch
tfm-rubygem-fog-vsphere-3.4.0-1.el7.noarch
tfm-rubygem-fog-xenserver-1.0.0-2.el7.noarch
tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
tfm-rubygem-foreman-tasks-3.0.1-1.fm2_2.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible-6.0.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_ansible_core-3.0.4-1.fm2_2.el7.noarch
tfm-rubygem-foreman_azure_rm-2.1.2-1.fm2_2.el7.noarch
tfm-rubygem-foreman_bootdisk-17.0.2-2.fm2_2.el7.noarch
tfm-rubygem-foreman_cockpit-2.0.3-4.fm1_23.el7.noarch
tfm-rubygem-foreman_discovery-16.2.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_docker-5.0.0-3.fm2_1.el7.noarch
tfm-rubygem-foreman_monitoring-2.0.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_openscap-4.0.4-1.fm2_2.el7.noarch
tfm-rubygem-foreman_remote_execution-4.1.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.3.1-1.el7.noarch
tfm-rubygem-foreman_templates-9.0.1-1.fm2_2.el7.noarch
tfm-rubygem-foreman_vmwareannotations-0.0.1-4.fm2_1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext-3.1.4-10.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-git-1.5.0-1.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-google-api-client-0.23.9-3.el7.noarch
tfm-rubygem-googleauth-0.6.7-3.el7.noarch
tfm-rubygem-graphite-api-0.1.5-6.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.2.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.2.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-0.3.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_azure_rm-0.2.0-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-0.1.11-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.1.2-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-0.23.2-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-httpclient-2.8.3-3.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-5.0.1-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.17.0-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.2.2-5.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-memoist-0.16.0-2.el7.noarch
tfm-rubygem-method_source-0.9.2-1.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-ms_rest-0.7.4-2.el7.noarch
tfm-rubygem-ms_rest_azure-0.11.1-2.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.2-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-openscap-0.4.9-3.el7.noarch
tfm-rubygem-optimist-3.0.0-2.el7.noarch
tfm-rubygem-os-1.0.0-2.el7.noarch
tfm-rubygem-ovirt-engine-sdk-4.3.0-1.el7.x86_64
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-passenger-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-10.13.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-10.13.el7.x86_64
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-prometheus-client-1.0.0-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.5.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.3.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.2-1.el7.noarch
tfm-rubygem-pulp_container_client-2.0.0-1.el7.noarch
tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch
tfm-rubygem-pulp_file_client-1.2.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.6.2-1.el7.noarch
tfm-rubygem-pulpcore_client-3.6.0-1.el7.noarch
tfm-rubygem-puma-4.3.3-4.el7.x86_64
tfm-rubygem-qpid_messaging-1.38.0-1.el7.x86_64
tfm-rubygem-quantile-0.2.0-4.el7.noarch
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.1-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.1-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rake-compiler-1.0.7-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-rbovirt-0.1.7-4.el7.noarch
tfm-rubygem-rbvmomi-2.2.0-3.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-representable-3.0.4-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-retriable-3.1.2-2.el7.noarch
tfm-rubygem-rkerberos-0.1.5-18.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.5-3.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-signet-0.11.0-3.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_discovery-1.0.5-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.2.4-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.2.6-1.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_openscap-0.7.4-1.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.0-4.fm2_2.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-text-1.3.0-7.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-timeliness-0.3.10-1.el7.noarch
tfm-rubygem-trollop-2.1.2-3.el7.noarch
tfm-rubygem-typhoeus-1.3.1-1.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-uber-0.1.0-2.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-wicked-1.3.3-2.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-x-editable-rails-1.5.5-5.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-rubygem-zest-0.0.4-1.el7.noarch
tfm-rubygem-zscheduler-0.0.6-5.el7.noarch
tfm-runtime-6.1-3.el7.x86_64

Distribution and version:
CentOS 7.8 and CentOS 7.9

Other relevant data:
Catalina.log gets spammed with:

Nov 12, 2020 10:09:40 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [http-bio-127.0.0.1-8443-exec-1] but has failed to stop it. This is very likely to create a memory leak.
Nov 12, 2020 10:09:40 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
2

Hi @sjansen

This can happen when you’re running low on RAM. Can you share the output of top -o %MEM after restarting tomcat?

3

Hi,

I restarted my server today because everything was stuck again.

Here is my output of top before i ran foreman-maintain service restart --only tomcat

top - 15:38:30 up 5:16, 1 user, load average: 0,05, 0,11, 0,13
Tasks: 318 total, 1 running, 317 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1,5 us, 0,8 sy, 0,0 ni, 97,5 id, 0,2 wa, 0,0 hi, 0,1 si, 0,0 st
KiB Mem : 32779660 total, 678924 free, 25915676 used, 6185060 buff/cache
KiB Swap: 2306044 total, 2287868 free, 18176 used. 5867880 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1530 mongodb 20 0 12,2g 9,0g 6384 S 0,7 28,9 7:15.74 mongod
1129 tomcat 20 0 9154452 1,9g 9988 S 0,7 6,0 3:06.79 java
1643 puppet 20 0 7249588 1,8g 12380 S 1,0 5,9 9:01.87 java
2539 apache 20 0 2404156 1,2g 6480 S 0,0 3,9 17:20.42 celery
3202 foreman 20 0 1574352 1,1g 9336 S 0,0 3,5 6:18.99 ruby
2550 apache 20 0 1967164 1,1g 4752 S 0,0 3,4 8:26.75 celery
1189 foreman 20 0 1342772 995,9m 7416 S 0,0 3,1 5:42.05 sidekiq
3196 foreman 20 0 1288372 883068 9376 S 0,0 2,7 5:17.00 ruby
1196 foreman 20 0 1180304 813808 6660 S 0,0 2,5 2:30.07 sidekiq
1188 foreman 20 0 899584 522528 7444 S 0,0 1,6 1:23.23 sidekiq
1197 foreman 20 0 773000 429256 6552 S 0,0 1,3 0:46.52 ruby
1194 squid 20 0 403068 301276 4872 S 0,3 0,9 1:08.99 squid
1521 postgres 20 0 834636 264804 263236 S 0,0 0,8 0:04.64 postmaster
2544 apache 20 0 1106600 259428 4696 S 0,0 0,8 3:18.57 celery
2540 apache 20 0 1337640 215788 3492 S 0,0 0,7 9:07.47 celery
1749 apache 20 0 1562908 150468 10188 S 0,0 0,5 1:12.25 httpd
1748 apache 20 0 1568288 122720 10188 S 0,0 0,4 1:14.32 httpd
1746 apache 20 0 1567260 119436 10160 S 0,0 0,4 1:11.21 httpd
1957 netdata 20 0 405248 118816 3300 S 1,0 0,4 2:33.15 netdata
2174 pulp 20 0 507832 98392 6024 S 0,0 0,3 0:15.44 gunicorn
2175 pulp 20 0 507832 98308 6024 S 0,3 0,3 0:15.35 gunicorn
1096 pulp 20 0 490496 94640 6808 S 0,3 0,3 0:12.15 rq
1105 pulp 20 0 490500 94640 6808 S 0,0 0,3 0:12.30 rq
1100 pulp 20 0 490492 94636 6804 S 0,0 0,3 0:12.05 rq
1107 pulp 20 0 490444 94584 6804 S 0,3 0,3 0:12.18 rq
1106 pulp 20 0 490488 94564 6800 S 0,3 0,3 0:12.13 rq
1094 pulp 20 0 490584 94560 6800 S 0,0 0,3 0:12.27 rq
1095 pulp 20 0 490444 94436 6804 S 0,3 0,3 0:12.16 rq
1948 pulp 20 0 489152 94048 5800 S 0,0 0,3 0:02.97 gunicorn
1099 apache 20 0 779976 86200 10092 S 0,3 0,3 1:09.12 celery
1471 apache 20 0 776644 84056 10092 S 0,3 0,3 1:00.98 celery
1487 apache 20 0 776224 82592 10092 S 0,3 0,3 1:01.30 celery
1515 apache 20 0 776324 81916 10092 S 0,3 0,2 1:02.49 celery
1102 apache 20 0 995736 81212 9924 S 0,3 0,2 1:10.50 pulp_streamer
1497 apache 20 0 775340 80868 10092 S 0,3 0,2 1:09.29 celery
2553 apache 20 0 925220 77172 3888 S 0,0 0,2 0:31.15 celery
1753 apache 20 0 1062732 75084 8928 S 0,0 0,2 0:06.67 httpd
3256 postgres 20 0 843872 73644 64656 S 0,0 0,2 0:02.35 postmaster
1187 foreman+ 20 0 1089864 66360 6580 S 0,0 0,2 0:10.66 ruby
3170 postgres 20 0 844896 60316 49936 S 0,0 0,2 0:02.90 postmaster
3180 postgres 20 0 850896 59224 41784 S 0,0 0,2 0:01.76 postmaster
1134 foreman+ 20 0 890940 58792 5888 S 0,0 0,2 0:04.35 ruby
1104 apache 20 0 697292 58252 7228 S 0,0 0,2 0:06.41 celery
6320 postgres 20 0 844712 57660 49716 S 0,0 0,2 0:02.70 postmaster
3179 postgres 20 0 837044 56876 53264 S 0,0 0,2 0:01.86 postmaster
1132 root 20 0 384540 54984 8144 S 0,0 0,2 0:03.12 puppet
15448 postgres 20 0 843424 53448 45320 S 0,0 0,2 0:07.17 postmaster
1752 apache 20 0 904016 52548 6196 S 0,0 0,2 0:06.35 httpd
1750 apache 20 0 904016 52508 6200 S 0,0 0,2 0:06.26 httpd
1751 apache 20 0 904016 52412 6200 S 0,0 0,2 0:06.38 httpd

And here after restarting tomcat

top - 15:40:36 up 5:18, 1 user, load average: 0,53, 0,25, 0,18
Tasks: 317 total, 1 running, 316 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0,9 us, 0,7 sy, 0,0 ni, 98,4 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
KiB Mem : 32779660 total, 1070232 free, 25318184 used, 6391244 buff/cache
KiB Swap: 2306044 total, 2287868 free, 18176 used. 6465208 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1530 mongodb 20 0 12,2g 9,0g 6532 S 1,0 28,9 7:16.83 mongod
1643 puppet 20 0 7249588 1,8g 12380 S 0,0 5,9 9:05.54 java
22310 tomcat 20 0 9127952 1,3g 21396 S 0,3 4,2 0:53.27 java
2539 apache 20 0 2404156 1,2g 6480 S 0,0 3,9 17:20.49 celery
3202 foreman 20 0 1566672 1,1g 9344 S 0,0 3,5 6:25.58 ruby
2550 apache 20 0 1967164 1,1g 4752 S 0,0 3,4 8:26.82 celery
1189 foreman 20 0 1342772 995,9m 7416 S 0,0 3,1 5:42.23 sidekiq
3196 foreman 20 0 1274548 882900 9376 S 0,0 2,7 5:20.81 ruby
1196 foreman 20 0 1180304 813808 6660 S 0,3 2,5 2:30.23 sidekiq
1188 foreman 20 0 899584 522528 7444 S 0,7 1,6 1:23.41 sidekiq
1197 foreman 20 0 773000 429256 6552 S 0,0 1,3 0:46.53 ruby
1194 squid 20 0 403068 301276 4872 S 0,0 0,9 1:09.06 squid
1521 postgres 20 0 834636 264804 263236 S 0,0 0,8 0:04.64 postmaster
2544 apache 20 0 1106600 259428 4696 S 0,0 0,8 3:18.63 celery
2540 apache 20 0 1337640 215788 3492 S 0,0 0,7 9:07.54 celery
1749 apache 20 0 1562908 150468 10188 S 0,3 0,5 1:12.44 httpd
1748 apache 20 0 1568288 122720 10188 S 0,0 0,4 1:14.51 httpd
1746 apache 20 0 1567260 119436 10160 S 0,3 0,4 1:11.40 httpd
1957 netdata 20 0 405248 118816 3300 S 0,7 0,4 2:34.16 netdata
2174 pulp 20 0 507832 98392 6024 S 0,0 0,3 0:15.53 gunicorn
2175 pulp 20 0 507832 98308 6024 S 0,0 0,3 0:15.43 gunicorn
1096 pulp 20 0 490496 94640 6808 S 0,0 0,3 0:12.21 rq
1105 pulp 20 0 490500 94640 6808 S 0,0 0,3 0:12.37 rq
1100 pulp 20 0 490492 94636 6804 S 0,0 0,3 0:12.11 rq
1107 pulp 20 0 490444 94584 6804 S 0,0 0,3 0:12.24 rq
1106 pulp 20 0 490488 94564 6800 S 0,0 0,3 0:12.19 rq
1094 pulp 20 0 490584 94560 6800 S 0,0 0,3 0:12.33 rq
1095 pulp 20 0 490444 94436 6804 S 0,0 0,3 0:12.22 rq
1948 pulp 20 0 489152 94048 5800 S 0,0 0,3 0:02.97 gunicorn
1099 apache 20 0 779976 86200 10092 S 0,3 0,3 1:09.48 celery
1471 apache 20 0 776644 84056 10092 S 0,3 0,3 1:01.36 celery
1487 apache 20 0 776224 82592 10092 S 0,3 0,3 1:01.69 celery
1515 apache 20 0 776324 81916 10092 S 0,3 0,2 1:02.86 celery
1102 apache 20 0 995736 81212 9924 S 0,0 0,2 1:10.57 pulp_streamer
1497 apache 20 0 775340 80868 10092 S 0,7 0,2 1:09.68 celery
2553 apache 20 0 925220 77172 3888 S 0,0 0,2 0:31.22 celery
1753 apache 20 0 1062732 75084 8928 S 0,3 0,2 0:06.72 httpd
3256 postgres 20 0 843872 74160 65172 S 0,0 0,2 0:02.52 postmaster
1187 foreman+ 20 0 1089864 66360 6580 S 0,0 0,2 0:10.73 ruby
3170 postgres 20 0 844896 60316 49936 S 0,0 0,2 0:02.90 postmaster
3180 postgres 20 0 850896 59224 41784 S 0,0 0,2 0:01.77 postmaster
1134 foreman+ 20 0 890940 58796 5892 S 0,0 0,2 0:04.36 ruby
1104 apache 20 0 697292 58252 7228 S 0,0 0,2 0:06.45 celery
6320 postgres 20 0 844712 57660 49716 S 0,0 0,2 0:02.70 postmaster
3179 postgres 20 0 837044 56876 53264 S 0,3 0,2 0:01.87 postmaster
1132 root 20 0 384540 54984 8144 S 0,0 0,2 0:03.12 puppet
15448 postgres 20 0 843424 54240 46040 S 0,0 0,2 0:07.57 postmaster
1752 apache 20 0 904016 52548 6196 S 0,0 0,2 0:06.36 httpd
1750 apache 20 0 904016 52508 6200 S 0,0 0,2 0:06.27 httpd
1751 apache 20 0 904016 52412 6200 S 0,0 0,2 0:06.39 httpd

4

Another thing to check is are you running low on free disk space? If the disk is more than 90% full there could be issues. (check logs in /var/log/candlepin)