Candlepin returning 404 and Message: Katello::Resources::Candlepin::CandlepinPing: 404 Not Found (GET /candlepin/status)

stingrayza · March 17, 2020, 3:16pm

Candlepin isn’t starting - and I can’t seem to see any errors that may explain why not

production.log does show something - but I don’t know how to isolate that.

While searching around, I found many references to candlepin and matching certs - my candleping & katello certs do seem to match.

Any assistance would be appreciated, thanks in advance!

Problem:

Candlepin returning 404

Expected outcome:

Candlepin OK

Foreman and Proxy versions:

katello-3.14.1-1.el7.noarch
foreman-1.24.2-1.el7.noarch

Foreman and Proxy plugin versions:

Distribution and version:

Centos 7.7

Other relevant data:

# hammer ping katello
candlepin:
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:
    Status:          FAIL
    Server Response: Message: Katello::Resources::Candlepin::CandlepinPing: 404 Not Found  (GET /candlepin/status)
pulp:
    Status:          ok
    Server Response: Duration: 39ms
pulp_auth:
    Status:          ok
    Server Response: Duration: 18ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 2ms

/var/log/candlepin/candlepin.log

2020-03-17 12:11:46,589 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2020-03-17 12:11:46,631 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.pki.impl.JSSProviderLoader - Using JSS version 4.4
2020-03-17 12:11:46,704 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2020-03-17 12:11:46,707 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.common.config.EncryptedConfiguration - No secret file provided.
2020-03-17 12:11:46,744 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Running under postgresql
2020-03-17 12:11:46,762 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, hypervisors_async, org_level_content_access, guest_limit, ram, batch_bind]
2020-03-17 12:11:47,137 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2020-03-17 12:11:52,613 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2020-03-17 12:11:52,772 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[default-update-timestamps-region] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,779 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[default-query-results-region] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,827 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product.attributes] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,834 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.ProductContent] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,840 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Content] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,850 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product.productContent] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,854 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,859 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Content.modifiedProductIds] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:52,863 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product.dependentProductIds] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-17 12:11:54,372 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.audit.QpidConnection - AMQP connection factory created.

/var/log/foreman/production.log (during ping)

2020-03-17T15:04:36 [I|app|a1b11e74] Started GET "/katello/api/ping" for 10.88.14.26 at 2020-03-17 15:04:36 +0000
2020-03-17T15:04:36 [I|app|a1b11e74] Processing by Katello::Api::V2::KatelloPingController#index as JSON
2020-03-17T15:04:36 [I|app|a1b11e74]   Parameters: {"api_version"=>"v2", "katello_ping"=>{}}
2020-03-17T15:04:36 [W|app|a1b11e74] 404 Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:228:in `backend_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:87:in `block in ping_candlepin_without_auth'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:136:in `exception_watch'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:86:in `ping_candlepin_without_auth'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:24:in `ping'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/controllers/katello/api/v2/katello_ping_controller.rb:15:in `index'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/middleware/event_daemon.rb:10:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-03-17T15:04:36 [W|app|a1b11e74] Katello::Resources::Candlepin::CandlepinPing: 404 Not Found  (GET /candlepin/status)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/http_resource.rb:88:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/lib/katello/resources/candlepin/candlepin_ping.rb:7:in `ping'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:104:in `block in ping_candlepin_with_auth'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:136:in `exception_watch'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:103:in `ping_candlepin_with_auth'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/models/katello/ping.rb:27:in `ping'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/controllers/katello/api/v2/katello_ping_controller.rb:15:in `index'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/middleware/event_daemon.rb:10:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-03-17T15:04:36 [I|app|a1b11e74]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/views/katello/api/v2/katello_ping/show.json.rabl within katello/api/v2/layouts/resource
2020-03-17T15:04:36 [I|app|a1b11e74]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/app/views/katello/api/v2/katello_ping/show.json.rabl within katello/api/v2/layouts/resource (3.9ms)
2020-03-17T15:04:36 [I|app|a1b11e74] Completed 200 OK in 97ms (Views: 4.8ms | ActiveRecord: 2.4ms)

sajha · March 17, 2020, 5:40pm

Hello @stingrayza,

Could you try foreman-maintain service restart. Also do you have enough memory on your machine. It is recommended to have at least 8GB. I sometimes run into this with smaller boxes.

stingrayza · March 18, 2020, 10:44am

Hi @sajha

Thanks so much for taking the time to get back to me.

I’ve tried as you said, and the same issue appears to be happening.

[root@for01 ~]# foreman-maintain service restart
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services:
Stopping the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy
- All services stopped
Starting the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy
- All services started                                                [OK]
--------------------------------------------------------------------------------

[root@for01 ~]# hammer ping
database:
    Status:          ok
    Server Response: Duration: 0ms
candlepin:
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:
    Status:          FAIL
    Server Response: Message: Katello::Resources::Candlepin::CandlepinPing: 404 Not Found  (GET /candlepin/status)
pulp:
    Status:          ok
    Server Response: Duration: 115ms
pulp_auth:
    Status:          ok
    Server Response: Duration: 63ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 3ms

The box had 2cpu/8GB (We only have a small handful of clients) - but I’ve upped this to 4cpu/12GB anyway - and it doesn’t seem to have helped…

Thanks

sajha · March 18, 2020, 2:17pm

Is tomcat service running as you run ping? If yes, do you see any errors in /var/log/candlepin/candlepin.log when this happens?

stingrayza · March 18, 2020, 2:24pm

yes, tomcat is up and running.

nothing appears in the latest catalina.log or the candlepin log during the ping - only in production.log in /var/log/foreman

sajha · March 18, 2020, 3:02pm

@stingrayza:
Could you tail the candlepin and catalina log and restart tomcat?

stingrayza · March 18, 2020, 4:10pm

sure:

/var/log/tomcat/catalina.2020-03-18.log

Mar 18, 2020 4:07:29 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8443"]
Mar 18, 2020 4:07:29 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Mar 18, 2020 4:07:29 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8443"]
Mar 18, 2020 4:07:29 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8443"]
Mar 18, 2020 4:07:29 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslProtocols' to 'TLSv1.1,TLSv1.2' did not find a matching property.
Mar 18, 2020 4:07:29 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.
Mar 18, 2020 4:07:29 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property.
Mar 18, 2020 4:07:29 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Mar 18, 2020 4:07:30 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8443"]
Mar 18, 2020 4:07:30 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 758 ms
Mar 18, 2020 4:07:30 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Mar 18, 2020 4:07:30 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.76
Mar 18, 2020 4:07:30 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/candlepin
Mar 18, 2020 4:07:34 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.OwnerCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.ProductCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ProductCurator.merge(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.EntitlementCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.ConsumerCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.ConsumerCurator.create(org.candlepin.model.Persisted,boolean)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.PoolCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.ContentCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.CdnCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.RulesCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:36 PM com.google.inject.internal.ProxyFactory <init>
WARNING: Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@dacd896]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Mar 18, 2020 4:07:40 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Mar 18, 2020 4:07:40 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/candlepin] startup failed due to previous errors
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
SEVERE: The web application [/candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
  context: /candlepin
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
java.net.URLClassLoader@7a81197d
::org.ehcache.jcache.JCacheCachingProvider@223f8838] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [Statistics Thread-file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
  context: /candlepin
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
java.net.URLClassLoader@7a81197d
::org.ehcache.jcache.JCacheCachingProvider@223f8838-1] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-5] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-6] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-7] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-8] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-9] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-11] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-12] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-13] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-14] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_Worker-15] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/candlepin] appears to have started a thread named [QuartzScheduler_QuartzSchedulerThread] but has failed to stop it. This is very likely to create a memory leak.
Mar 18, 2020 4:07:40 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/candlepin has finished in 9,867 ms
Mar 18, 2020 4:07:40 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8443"]
Mar 18, 2020 4:07:40 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 9902 ms

/var/log/candlepin/candlepin.log

2020-03-18 16:07:34,263 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2020-03-18 16:07:34,269 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.pki.impl.JSSProviderLoader - Using JSS version 4.4
2020-03-18 16:07:34,333 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2020-03-18 16:07:34,336 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.common.config.EncryptedConfiguration - No secret file provided.
2020-03-18 16:07:34,363 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Running under postgresql
2020-03-18 16:07:34,374 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, hypervisors_async, org_level_content_access, guest_limit, ram, batch_bind]
2020-03-18 16:07:34,627 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2020-03-18 16:07:38,715 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2020-03-18 16:07:38,870 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[default-update-timestamps-region] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,878 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[default-query-results-region] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,924 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product.attributes] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,932 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Content.modifiedProductIds] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,936 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,943 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product.productContent] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,947 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Product.dependentProductIds] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,953 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.ProductContent] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:38,957 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.hibernate.orm.cache - HHH90001006: Missing cache[org.candlepin.model.Content] was created on-the-fly. The created cache will use a provider-specific default configuration: make sure you defined one. You can disable this warning by setting 'hibernate.cache.ehcache.missing_cache_strategy' to 'create'.
2020-03-18 16:07:40,232 [thread=localhost-startStop-1] [=, org=, csid=] INFO  org.candlepin.audit.QpidConnection - AMQP connection factory created.

stingrayza · March 18, 2020, 4:13pm

I’ve just spotted an additional file that has some stack traces during that tomcat startup:

/var/log/tomcat/localhost.2020-03-18.log

Mar 18, 2020 4:11:49 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.candlepin.guice.CandlepinContextListener
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error in custom provider, java.lang.NullPointerException
  while locating com.google.inject.persist.jpa.JpaPersistService
  while locating javax.persistence.EntityManager
  at org.candlepin.policy.js.JsRunnerProvider.<init>(JsRunnerProvider.java:85)
  at org.candlepin.guice.CandlepinModule.configure(CandlepinModule.java:267)
  while locating org.candlepin.policy.js.JsRunnerProvider
Caused by: java.lang.NullPointerException
        at com.google.inject.persist.jpa.JpaPersistService.begin(JpaPersistService.java:77)
        at com.google.inject.persist.jpa.JpaPersistService.get(JpaPersistService.java:57)
        at com.google.inject.persist.jpa.JpaPersistService.get(JpaPersistService.java:41)
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
        at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
        at org.candlepin.model.AbstractHibernateCurator.currentSession(AbstractHibernateCurator.java:592)
        at org.candlepin.model.RulesCurator.getDbRules(RulesCurator.java:84)
        at org.candlepin.model.RulesCurator.updateDbRules(RulesCurator.java:91)
        at org.candlepin.policy.js.JsRunnerProvider.<init>(JsRunnerProvider.java:90)
        at org.candlepin.policy.js.JsRunnerProvider$$FastClassByGuice$$db9cfd63.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:56)
        at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:140)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

2) Error injecting constructor, javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory
  at org.candlepin.common.guice.JPAInitializer.<init>(JPAInitializer.java:29)
  at org.candlepin.guice.CandlepinModule.configureJPA(CandlepinModule.java:362)
  while locating org.candlepin.common.guice.JPAInitializer
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory
        at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.persistenceException(EntityManagerFactoryBuilderImpl.java:1016)
        at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:942)
        at org.hibernate.jpa.HibernatePersistenceProvider.createEntityManagerFactory(HibernatePersistenceProvider.java:56)
        at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
        at com.google.inject.persist.jpa.JpaPersistService.start(JpaPersistService.java:108)
        at org.candlepin.common.guice.JPAInitializer.<init>(JPAInitializer.java:30)
        at org.candlepin.common.guice.JPAInitializer$$FastClassByGuice$$80e35e6b.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:56)
        at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:140)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing table [cp_act_key_sp_add_on]
        at org.hibernate.tool.schema.internal.AbstractSchemaValidator.validateTable(AbstractSchemaValidator.java:121)
        at org.hibernate.tool.schema.internal.GroupedSchemaValidatorImpl.validateTables(GroupedSchemaValidatorImpl.java:42)
        at org.hibernate.tool.schema.internal.AbstractSchemaValidator.performValidation(AbstractSchemaValidator.java:89)
        at org.hibernate.tool.schema.internal.AbstractSchemaValidator.doValidation(AbstractSchemaValidator.java:68)
        at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:191)
        at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:72)
        at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:310)
        at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:467)
        at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:939)
        ... 36 more

2 errors
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.Guice.createInjector(Guice.java:99)
        at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:56)
        at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:140)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 18, 2020 4:11:49 PM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class org.candlepin.guice.CandlepinContextListener
java.lang.NullPointerException
        at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.triggerAnnotatedMethods(GuiceResteasyBootstrapServletContextListener.java:151)
        at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextDestroyed(GuiceResteasyBootstrapServletContextListener.java:146)
        at org.candlepin.guice.CandlepinContextListener.contextDestroyed(CandlepinContextListener.java:209)
        at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5165)
        at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5829)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Those don’t look good

sajha · March 18, 2020, 6:50pm

Thanks @stingrayza for the logs. I was looking around for similar issues and I did not find anything immediately useful. I found this Cannot login to foreman 1.16.0/Katello 3.5 which looks similar. However, I am not sure what the fix here might be.

@Jonathon_Turel : Have you seen this before?

Jonathon_Turel · March 19, 2020, 7:50pm

Thanks for providing these details!

The root cause isn’t jumping out at me. Can you tell us a little more about your setup?

Has it ever worked?

Is it a fresh install, upgraded? If upgraded, from which version?

Are you using custom certs?

Does this give a result? https://localhost:8443/candlepin/status

It should look something like this:

{"mode":"NORMAL","modeReason":null,"modeChangeTime":null,"result":true,"version":"3.1.7","rulesVersion":"5.39","release":"1","standalone":true,"timeUTC":"2020-03-19T19:47:26+0000","rulesSource":"default","managerCapabilities":["instance_multiplier","derived_product","vcpu","cert_v3","hypervisors_heartbeat","remove_by_pool_id","syspurpose","storage_band","cores","hypervisors_async","org_level_content_access","guest_limit","ram","batch_bind"]}

stingrayza · March 20, 2020, 11:36am

Hi @Jonathon_Turel thanks for taking the time to look at this

I upgraded from
foreman-1.20 & katello-3.9.1

It was working before.

No custom certs - foreman/katello generated the lot.

# curl -kv https://localhost:8443/candlepin/status
* About to connect() to localhost port 8443 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: CN=for01.example.com,OU=SomeOrgUnit,O=Katello,ST=North Carolina,C=US
*       start date: Nov 12 10:50:24 2018 GMT
*       expire date: Jan 18 10:50:25 2038 GMT
*       common name: for01.example.com
*       issuer: CN=for01.example.com,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North Carolina,C=US
> GET /candlepin/status HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8443
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: Apache-Coyote/1.1
< Content-Length: 0
< Date: Fri, 20 Mar 2020 11:33:17 GMT
<
* Connection #0 to host localhost left intact

Here’s yum history from the upgrade (in case it helps) (I added a . in the katello repo name so as not to ping a load of people on discourse)

Packages Altered:
    Updated     candlepin-2.5.7-1.el7.noarch                                               @.katello-candlepin
    Update                2.9.14-1.el7.noarch                                              @.katello-candlepin
    Updated     candlepin-selinux-2.5.7-1.el7.noarch                                       @.katello-candlepin
    Update                        2.9.14-1.el7.noarch                                      @.katello-candlepin
Loading mirror speeds from cached hostfile
 * base: mirror.sov.uk.goscomb.net
 * centos-sclo-rh: mirror.sov.uk.goscomb.net
 * centos-sclo-sclo: mirror.sov.uk.goscomb.net
 * epel: mirrors.ukfast.co.uk
 * extras: mirrors.clouvider.net
 * updates: mirror.sov.uk.goscomb.net
    Updated     foreman-1.20.3-1.el7.noarch                                                ?
    Obsoleting  foreman-1.24.2-1.el7.noarch                                                @foreman
    Updated     foreman-cli-1.20.3-1.el7.noarch                                            ?
    Update                  1.24.2-1.el7.noarch                                            @foreman
    Obsoleted   foreman-compute-1.20.3-1.el7.noarch                                        ?
    Updated     foreman-debug-1.20.3-1.el7.noarch                                          ?
    Update                    1.24.2-1.el7.noarch                                          @foreman
    Updated     foreman-installer-1:1.20.3-1.el7.noarch                                    ?
    Update                        1:1.24.2-1.el7.noarch                                    @foreman
    Updated     foreman-installer-katello-3.9.1-1.el7.noarch                               ?
    Obsoleting  foreman-installer-katello-1:1.24.2-1.el7.noarch                            @foreman
    Updated     foreman-postgresql-1.20.3-1.el7.noarch                                     ?
    Update                         1.24.2-1.el7.noarch                                     @foreman
    Updated     foreman-proxy-1.20.3-1.el7.noarch                                          ?
    Update                    1.24.2-1.el7.noarch                                          @foreman
    Updated     foreman-selinux-1.20.3-1.el7.noarch                                        ?
    Update                      1.24.2-1.el7.noarch                                        @foreman
    Updated     katello-3.9.1-1.el7.noarch                                                 ?
    Update              3.14.1-1.el7.noarch                                                @.katello
    Updated     katello-certs-tools-2.4.0-1.el7.noarch                                     @.katello
    Update                          2.6.0-1.el7.noarch                                     @foreman
    Updated     katello-client-bootstrap-1.6.0-1.el7.noarch                                @.katello
    Update                               1.7.4-1.el7.noarch                                @.katello
    Updated     katello-common-3.9.1-1.el7.noarch                                          ?
    Update                     3.14.1-1.el7.noarch                                         @.katello
    Updated     katello-debug-3.9.1-1.el7.noarch                                           ?
    Update                    3.14.1-1.el7.noarch                                          @.katello
    Obsoleted   katello-installer-base-3.9.1-1.el7.noarch                                  ?
    Updated     katello-selinux-3.0.3-1.el7.noarch                                         @.katello
    Update                      3.1.1-1.el7.noarch                                         @foreman
    Updated     katello-service-3.9.1-1.el7.noarch                                         ?
    Update                      3.14.1-1.el7.noarch                                        @.katello
    Updated     libmodulemd-1.6.3-1.el7.x86_64                                             @pulp
    Update                  1.7.0-1.el7.x86_64                                             @pulp
    Updated     libsolv-0.6.34-4.el7.x86_64                                                @pulp
    Update              0.7.3-4.el7.x86_64                                                 @pulp
    Updated     pulp-admin-client-2.17.1-1.el7.noarch                                      @pulp
    Update                        2.21.1-1.el7.noarch                                      @pulp
    Updated     pulp-consumer-client-2.17.1-1.el7.noarch                                   @pulp
    Update                           2.21.1-1.el7.noarch                                   @pulp
    Updated     pulp-deb-plugins-1.8.0-1.el7.noarch                                        @pulp
    Update                       1.10.1-1.el7.noarch                                       @pulp
    Updated     pulp-docker-plugins-3.2.1-1.el7.noarch                                     @pulp
    Update                          3.2.6-1.el7.noarch                                     @pulp
    Updated     pulp-katello-1.0.2-1.el7.noarch                                            @.katello
    Update                   1.0.3-1.el7.noarch                                            @.katello
    Updated     pulp-puppet-plugins-2.17.1-1.el7.noarch                                    @pulp
    Update                          2.21.1-1.el7.noarch                                    @pulp
    Updated     pulp-puppet-tools-2.17.1-1.el7.noarch                                      @pulp
    Update                        2.21.1-1.el7.noarch                                      @pulp
    Updated     pulp-rpm-admin-extensions-2.17.1-1.el7.noarch                              @pulp
    Update                                2.21.1-1.el7.noarch                              @pulp
    Updated     pulp-rpm-consumer-extensions-2.17.1-1.el7.noarch                           @pulp
    Update                                   2.21.1-1.el7.noarch                           @pulp
    Updated     pulp-rpm-handlers-2.17.1-1.el7.noarch                                      @pulp
    Update                        2.21.1-1.el7.noarch                                      @pulp
    Updated     pulp-rpm-plugins-2.17.1-1.el7.noarch                                       @pulp
    Update                       2.21.1-1.el7.noarch                                       @pulp
    Updated     pulp-rpm-yumplugins-2.17.1-1.el7.noarch                                    @pulp
    Update                          2.21.1-1.el7.noarch                                    @pulp
    Updated     pulp-selinux-2.17.1-1.el7.noarch                                           @pulp
    Update                   2.21.1-1.el7.noarch                                           @pulp
    Updated     pulp-server-2.17.1-1.el7.noarch                                            @pulp
    Update                  2.21.1-1.el7.noarch                                            @pulp
    Updated     puppet-agent-1.10.14-1.el7.x86_64                                          @puppetlabs-pc1
    Update                   6.14.0-1.el7.x86_64                                           @puppet6
    Updated     puppetserver-2.8.1-1.el7.noarch                                            @puppetlabs-pc1
    Update                   6.9.1-1.el7.noarch                                            @puppet6
    Updated     python-crane-3.2.0-1.el7.noarch                                            @pulp
    Update                   3.3.1-2.el7.noarch                                            @pulp
    Updated     python-gofer-2.12.1-1.el7.noarch                                           @.katello
    Update                   2.12.5-3.el7.noarch                                           @pulp
    Updated     python-gofer-qpid-2.12.1-1.el7.noarch                                      @.katello
    Update                        2.12.5-3.el7.noarch                                      @pulp
    Updated     python-nectar-1.5.6-1.el7.noarch                                           @pulp
    Update                    1.6.0-1.el7.noarch                                           @pulp
    Updated     python-pulp-agent-lib-2.17.1-1.el7.noarch                                  @pulp
    Update                            2.21.1-1.el7.noarch                                  @pulp
    Updated     python-pulp-bindings-2.17.1-1.el7.noarch                                   @pulp
    Update                           2.21.1-1.el7.noarch                                   @pulp
    Updated     python-pulp-client-lib-2.17.1-1.el7.noarch                                 @pulp
    Update                             2.21.1-1.el7.noarch                                 @pulp
    Updated     python-pulp-common-2.17.1-1.el7.noarch                                     @pulp
    Update                         2.21.1-1.el7.noarch                                     @pulp
    Updated     python-pulp-deb-common-1.8.0-1.el7.noarch                                  @pulp
    Update                             1.10.1-1.el7.noarch                                 @pulp
    Updated     python-pulp-docker-common-3.2.1-1.el7.noarch                               @pulp
    Update                                3.2.6-1.el7.noarch                               @pulp
    Updated     python-pulp-oid_validation-2.17.1-1.el7.noarch                             @pulp
    Update                                 2.21.1-1.el7.noarch                             @pulp
    Updated     python-pulp-puppet-common-2.17.1-1.el7.noarch                              @pulp
    Update                                2.21.1-1.el7.noarch                              @pulp
    Updated     python-pulp-repoauth-2.17.1-1.el7.noarch                                   @pulp
    Update                           2.21.1-1.el7.noarch                                   @pulp
    Updated     python-pulp-rpm-common-2.17.1-1.el7.noarch                                 @pulp
    Update                             2.21.1-1.el7.noarch                                 @pulp
    Updated     python-pulp-streamer-2.17.1-1.el7.noarch                                   @pulp
    Update                           2.21.1-1.el7.noarch                                   @pulp
    Updated     python2-amqp-2.4.0-1.el7.noarch                                            @?epel
    Update                   10:2.2.2-5.el7.noarch                                         @pulp
    Updated     python2-billiard-1:3.5.0.5-1.el7.x86_64                                    @?epel
    Update                       10:3.5.0.3-4.el7.x86_64                                   @pulp
    Updated     python2-celery-4.2.1-3.el7.noarch                                          @?epel
    Update                     10:4.0.2-7.el7.noarch                                       @pulp
    Updated     python2-kombu-1:4.2.2-1.el7.noarch                                         @?epel
    Update                    10:4.0.2-12.el7.noarch                                       @pulp
    Updated     python2-solv-0.6.34-4.el7.x86_64                                           @pulp
    Update                   0.7.3-4.el7.x86_64                                            @pulp
    Updated     python2-vine-1.2.0-1.el7.noarch                                            @?epel
    Update                   10:1.1.3-5.el7.noarch                                         @pulp
    Updated     rubygem-concurrent-ruby-1:1.0.3-6.el7.noarch                               @foreman
    Update                              1:1.1.4-2.el7.noarch                               @foreman
    Updated     rubygem-foreman_maintain-0.3.0-1.el7.noarch                                @foreman
    Update                               1:0.5.1-1.el7.noarch                              @foreman
    Updated     rubygem-kafo-2.1.0-1.el7.noarch                                            @foreman
    Update                   4.0.0-1.el7.noarch                                            @foreman
    Updated     rubygem-kafo_parsers-0.1.6-1.el7.noarch                                    @foreman
    Update                           1.0.0-1.el7.noarch                                    @foreman
    Updated     rubygem-little-plugger-1.1.3-23.el7.noarch                                 @foreman
    Update                             1.1.4-1.el7.noarch                                  @foreman
    Updated     rubygem-netrc-0.7.7-3.el7.noarch                                           @epel
    Update                    0.11.0-4.el7.noarch                                          @foreman-plugins
    Updated     rubygem-rb-inotify-0.9.7-2.el7.noarch                                      @foreman
    Update                         0.9.7-3.el7.noarch                                      @foreman
    Updated     rubygem-rsec-0.4.2-1.el7.noarch                                            @foreman
    Update                   0.4.3-2.el7.noarch                                            @foreman
    Updated     rubygem-rubyipmi-0.10.0-3.el7.noarch                                       @foreman
    Update                       0.10.0-4.el7.noarch                                       @foreman
    Updated     rubygem-smart_proxy_ansible-2.0.3-2.fm1_20.el7.noarch                      @foreman-plugins
    Update                                  3.0.1-1.fm1_23.el7.noarch                      @foreman-plugins
    Updated     rubygem-smart_proxy_discovery-1.0.4-1.el7.noarch                           @foreman-plugins
    Update                                    1.0.5-1.fm1_23.el7.noarch                    @foreman-plugins
    Updated     rubygem-smart_proxy_dynflow-0.2.1-1.el7.noarch                             @foreman-plugins
    Update                                  0.2.4-1.el7.noarch                             @foreman-plugins
    Updated     rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch                                @foreman-plugins
    Update                               1.5.0-1.el7.noarch                                @foreman-plugins
    Updated     tfm-ror52-rubygem-concurrent-ruby-1.0.5-1.el7.noarch                       @foreman-rails
    Update                                        1.1.4-1.el7.noarch                       @foreman-rails
    Updated     tfm-ror52-rubygem-i18n-1.0.1-1.el7.noarch                                  @foreman-rails
    Update                             1.4.0-1.el7.noarch                                  @foreman-rails
    Updated     tfm-ror52-rubygem-rack-2.0.5-2.el7.noarch                                  @foreman-rails
    Update                             2.0.6-1.el7.noarch                                  @foreman-rails
    Dep-Install tfm-rubygem-activerecord-import-1.0.0-1.el7.noarch                         @.katello
    Updated     tfm-rubygem-addressable-2.3.6-6.el7.noarch                                 @foreman
    Update                              2.6.0-1.el7.noarch                                 @foreman
    Updated     tfm-rubygem-apipie-bindings-0.2.2-2.el7.noarch                             @foreman
    Update                                  0.3.0-1.el7.noarch                             @foreman
    Updated     tfm-rubygem-audited-4.7.1-2.el7.noarch                                     @foreman
    Update                          4.9.0-1.el7.noarch                                     @foreman
    Obsoleted   tfm-rubygem-bastion-6.1.16-1.fm1_20.el7.noarch                             @foreman-plugins
    Dep-Install tfm-rubygem-bcrypt-3.1.12-1.el7.x86_64                                     @foreman
    Updated     tfm-rubygem-concurrent-ruby-edge-1:0.2.4-2.el7.noarch                      @foreman
    Update                                       1:0.4.1-1.fm1_21.el7.noarch               @foreman
    Dep-Install tfm-rubygem-declarative-0.0.10-1.el7.noarch                                @foreman
    Dep-Install tfm-rubygem-declarative-option-0.1.0-1.el7.noarch                          @foreman
    Updated     tfm-rubygem-deep_cloneable-2.3.2-2.el7.noarch                              @foreman
    Update                                 3.0.0-1.el7.noarch                              @foreman
    Updated     tfm-rubygem-deface-1.3.2-1.el7.noarch                                      @foreman-plugins
    Update                         1.5.3-1.el7.noarch                                      @foreman-plugins
    Obsoleted   tfm-rubygem-docker-api-1.28.0-4.el7.noarch                                 @foreman-plugins
    Updated     tfm-rubygem-dynflow-1.1.2-1.fm1_20.el7.noarch                              @foreman
    Update                          1.4.2-1.fm1_24.el7.noarch                              @foreman
    Dep-Install tfm-rubygem-ethon-0.12.0-1.el7.noarch                                      @.katello
    Dep-Install tfm-rubygem-faraday-0.15.4-1.el7.noarch                                    @foreman
    Dep-Install tfm-rubygem-ffi-1.4.0-8.el7.x86_64                                         @foreman-plugins
    Obsoleted   tfm-rubygem-fog-1.42.1-2.el7.noarch                                        @foreman
    Updated     tfm-rubygem-fog-aws-1.3.0-3.el7.noarch                                     @foreman
    Update                          3.5.0-1.el7.noarch                                     @foreman
    Updated     tfm-rubygem-fog-core-1.45.0-3.el7.noarch                                   @foreman
    Update                           2.1.0-1.el7.noarch                                    @foreman
    Updated     tfm-rubygem-fog-digitalocean-0.3.0-3.el7.noarch                            @foreman
    Update                                   0.4.0-1.el7.noarch                            @foreman
    Updated     tfm-rubygem-fog-google-0.1.0-4.el7.noarch                                  @foreman
    Update                             1.8.2-1.el7.noarch                                  @foreman
    Updated     tfm-rubygem-fog-json-1.0.2-6.el7.noarch                                    @foreman
    Update                           1.2.0-1.el7.noarch                                    @foreman
    Updated     tfm-rubygem-fog-openstack-0.1.25-2.el7.noarch                              @foreman
    Update                                1.0.8-2.el7.noarch                               @foreman
    Updated     tfm-rubygem-fog-ovirt-1.1.2-2.el7.noarch                                   @foreman
    Update                            1.2.0-1.el7.noarch                                   @foreman
    Updated     tfm-rubygem-fog-vsphere-2.3.0-2.el7.noarch                                 @foreman
    Update                              3.2.1-1.el7.noarch                                 @foreman
    Updated     tfm-rubygem-fog-xenserver-0.2.3-3.el7.noarch                               @foreman
    Update                                1.0.0-1.el7.noarch                               @foreman
    Updated     tfm-rubygem-foreman-tasks-0.14.3-1.fm1_20.el7.noarch                       @foreman-plugins
    Update                                0.17.5-1.fm1_24.el7.noarch                       @foreman-plugins
    Updated     tfm-rubygem-foreman-tasks-core-0.2.5-2.fm1_20.el7.noarch                   @foreman-plugins
    Update                                     0.3.3-1.fm1_24.el7.noarch                   @foreman-plugins
    Updated     tfm-rubygem-foreman_ansible-2.2.14-1.fm1_20.el7.noarch                     ?
    Update                                  4.0.5-1.fm1_24.el7.noarch                      @foreman-plugins
    Updated     tfm-rubygem-foreman_ansible_core-2.1.2-2.fm1_20.el7.noarch                 @foreman-plugins
    Update                                       3.0.2-1.fm1_24.el7.noarch                 @foreman-plugins
    Updated     tfm-rubygem-foreman_discovery-14.0.1-1.fm1_20.el7.noarch                   ?
    Update                                    16.0.1-1.fm1_24.el7.noarch                   @foreman-plugins
    Updated     tfm-rubygem-foreman_docker-4.1.0-2.fm1_20.el7.noarch                       @foreman-plugins
    Update                                 5.0.0-2.fm1_22.el7.noarch                       @foreman-plugins
    Updated     tfm-rubygem-foreman_remote_execution-1.6.7-1.fm1_20.el7.noarch             ?
    Update                                           2.0.8-1.fm1_24.el7.noarch             @foreman-plugins
    Updated     tfm-rubygem-foreman_remote_execution_core-1.1.4-1.el7.noarch               @foreman-plugins
    Update                                                1.3.0-1.el7.noarch               @foreman-plugins
    Updated     tfm-rubygem-gettext_i18n_rails-1.2.1-5.el7.noarch                          @foreman
    Update                                     1.8.0-1.el7.noarch                          @foreman
    Dep-Install tfm-rubygem-google-api-client-0.23.9-1.el7.noarch                          @foreman
    Dep-Install tfm-rubygem-googleauth-0.6.7-1.el7.noarch                                  @foreman
    Dep-Install tfm-rubygem-graphql-1.8.14-1.el7.noarch                                    @foreman
    Dep-Install tfm-rubygem-graphql-batch-0.3.10-1.el7.noarch                              @foreman
    Updated     tfm-rubygem-hammer_cli-0.15.1-1.el7.noarch                                 @foreman
    Update                             0.19.2-1.el7.noarch                                 @foreman
    Updated     tfm-rubygem-hammer_cli_foreman-0.15.1-1.el7.noarch                         @foreman
    Update                                     0.19.6-1.el7.noarch                         @foreman
    Updated     tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-7.el7.noarch                 @foreman-plugins
    Update                                              0.3.0-1.el7.noarch                 @foreman-plugins
    Updated     tfm-rubygem-hammer_cli_foreman_discovery-1.0.0-3.el7.noarch                @foreman-plugins
    Update                                               1.0.1-1.fm1_24.el7.noarch         @foreman-plugins
    Updated     tfm-rubygem-hammer_cli_katello-0.15.0-1.el7.noarch                         @.katello
    Update                                     0.20.2-1.el7.noarch                         @.katello
    Dep-Install tfm-rubygem-httpclient-2.8.3-2.el7.noarch                                  @foreman
    Dep-Install tfm-rubygem-jwt-2.2.1-1.el7.noarch                                         @foreman
    Updated     tfm-rubygem-katello-3.9.1-2.el7.noarch                                     ?
    Obsoleting  tfm-rubygem-katello-3.14.1-1.el7.noarch                                    @.katello
    Updated     tfm-rubygem-little-plugger-1.1.3-23.el7.noarch                             @foreman
    Update                                 1.1.4-1.el7.noarch                              @foreman
    Dep-Install tfm-rubygem-memoist-0.16.0-1.el7.noarch                                    @foreman
    Dep-Install tfm-rubygem-multipart-post-2.0.0-1.el7.noarch                              @foreman
    Updated     tfm-rubygem-net-ldap-0.15.0-3.el7.noarch                                   @foreman
    Update                           0.16.1-1.el7.noarch                                   @foreman
    Updated     tfm-rubygem-net-ssh-4.0.1-5.el7.noarch                                     @foreman
    Update                          4.2.0-1.el7.noarch                                     @foreman
    Updated     tfm-rubygem-netrc-0.11.0-3.el7.noarch                                      @foreman
    Update                        0.11.0-4.el7.noarch                                      @foreman
    Dep-Install tfm-rubygem-optimist-3.0.0-1.el7.noarch                                    @foreman
    Dep-Install tfm-rubygem-os-1.0.0-1.el7.noarch                                          @foreman
    Updated     tfm-rubygem-parse-cron-0.1.4-4.fm1_20.el7.noarch                           @foreman-plugins
    Update                             0.1.4-4.fm1_22.el7.noarch                           @foreman-plugins
    Dep-Install tfm-rubygem-promise.rb-0.7.4-1.el7.noarch                                  @foreman
    Dep-Install tfm-rubygem-public_suffix-3.0.3-1.el7.noarch                               @foreman
    Dep-Install tfm-rubygem-pulp_2to3_migration_client-0.0.1a1.dev01570560174-1.el7.noarch @.katello
    Dep-Install tfm-rubygem-pulp_ansible_client-0.2.0b1.dev0.1560866833-1.el7.noarch       @.katello
    Dep-Install tfm-rubygem-pulp_docker_client-4.0.0b6.dev01562331743-1.el7.noarch         @.katello
    Dep-Install tfm-rubygem-pulp_file_client-0.0.1b10.dev.1557779852-1.el7.noarch          @.katello
    Dep-Install tfm-rubygem-pulp_rpm_client-3.0.0b7.dev01570381057-1.el7.noarch            @.katello
    Dep-Install tfm-rubygem-pulpcore_client-3.0.0rc2.dev.1557772471-1.el7.noarch           @.katello
    Dep-Install tfm-rubygem-rack-cors-1.0.2-1.el7.noarch                                   @foreman
    Updated     tfm-rubygem-rbvmomi-1.10.0-3.el7.noarch                                    @foreman
    Update                          2.2.0-1.el7.noarch                                     @foreman
    Dep-Install tfm-rubygem-representable-3.0.4-1.el7.noarch                               @foreman
    Updated     tfm-rubygem-responders-2.4.0-2.el7.noarch                                  @foreman
    Update                             3.0.0-1.el7.noarch                                  @foreman
    Updated     tfm-rubygem-rest-client-2.0.1-3.el7.noarch                                 @foreman
    Update                              2.0.2-1.el7.noarch                                 @foreman
    Dep-Install tfm-rubygem-retriable-3.1.2-1.el7.noarch                                   @foreman
    Updated     tfm-rubygem-roadie-3.2.2-2.el7.noarch                                      @foreman
    Update                         3.4.0-1.el7.noarch                                      @foreman
    Updated     tfm-rubygem-roadie-rails-1.3.0-2.el7.noarch                                @foreman
    Update                               2.0.0-1.el7.noarch                                @foreman
    Updated     tfm-rubygem-ruby2ruby-2.4.0-2.el7.noarch                                   @foreman
    Update                            2.4.2-1.el7.noarch                                   @foreman
    Updated     tfm-rubygem-runcible-2.9.0-2.el7.noarch                                    @.katello
    Update                           2.12.1-1.el7.noarch                                   @.katello
    Updated     tfm-rubygem-scoped_search-4.1.6-1.el7.noarch                               ?
    Update                                4.1.7-1.el7.noarch                               @foreman
    Dep-Install tfm-rubygem-signet-0.11.0-1.el7.noarch                                     @foreman
    Updated     tfm-rubygem-smart_proxy_dynflow_core-0.2.1-1.fm1_20.el7.noarch             @foreman-plugins
    Update                                           0.2.2-1.fm1_22.el7.noarch             @foreman-plugins
    Dep-Install tfm-rubygem-typhoeus-1.3.1-1.el7.noarch                                    @.katello
    Dep-Install tfm-rubygem-uber-0.1.0-1.el7.noarch                                        @foreman
    Updated     tfm-rubygem-unf_ext-0.0.6-8.el7.x86_64                                     @foreman
    Update                          0.0.7.2-1.el7.x86_64                                   @foreman
    Updated     tfm-rubygem-unicode-0.4.4.1-5.el7.x86_64                                   @foreman
    Update                          0.4.4.4-1.el7.x86_64                                   @foreman
    Updated     tfm-rubygem-will_paginate-3.1.5-3.el7.noarch                               @foreman
    Update                                3.1.7-1.el7.noarch                               @foreman
    Dep-Install tfm-rubygem-xmlrpc-0.3.0-1.el7.noarch                                      @foreman
    Updated     tfm-runtime-5.0-2.el7.x86_64                                               @foreman
    Obsoleting  tfm-runtime-5.0-9.el7.x86_64                                               @foreman
    Dep-Install tomcatjss-7.2.1-8.el7_6.noarch                                             @base

stingrayza · March 20, 2020, 1:17pm

re-reading the release notes of 3.14, I just spotted:

Katello supports upgrades from the previous two versions only. Upgrades should be performed sequentially without skipping versions in between.

Oops.
Is it worth trying to roll back and then step through the upgrades, or might it be worth simply restarting from a fresh install?

We only have 20(ish) clients, so it may not be the end of the world to redo this.

Or is there potentially a short way to remove/reinstall candlepin?

stingrayza · March 20, 2020, 2:37pm

Sorry for the multiple replies (I can only edit a previous post for 5 minutes)

I’ve restored to my last snapshot which is working. I’m going to step through the upgrades from 3.9 to 3.14 and see where that gets me

stingrayza · March 21, 2020, 8:06am

Hey @Jonathon_Turel @sajha

I’ve gone through the upgrade again, stepping through the versions and all looks good!

Thanks for your help - and please accept my apologies for the false scramble

[root@for01 ~]# hammer ping
database:
    Status:          ok
    Server Response: Duration: 0ms
candlepin:
    Status:          ok
    Server Response: Duration: 31ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 16ms
pulp:
    Status:          ok
    Server Response: Duration: 49ms
pulp_auth:
    Status:          ok
    Server Response: Duration: 25ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 6ms

[root@for01 ~]# rpm -qa foreman katello
foreman-1.24.2-1.el7.noarch
katello-3.14.1-1.el7.noarch
[root@for01 ~]# curl -k https://localhost:8443/candlepin/status
{"mode":"NORMAL","modeReason":"STARTUP","modeChangeTime":"2020-03-21T07:56:07+0000","result":true,"version":"2.9.14","rulesVersion":"5.37","release":"1","standalone":true,"timeUTC":"2020-03-21T08:03:53+0000","rulesSource":"default","managerCapabilities":["instance_multiplier","derived_product","vcpu","cert_v3","hypervisors_heartbeat","remove_by_pool_id","syspurpose","storage_band","cores","hypervisors_async","org_level_content_access","guest_limit","ram","batch_bind"]}

Jonathon_Turel · March 23, 2020, 1:53pm

@stingrayza thanks for sharing your success! Yeah, it seems like upgrade from 3.9 to 3.14 directly could be what got you. Glad it’s working now!