Candlepin does not support JSS versions less than 4.4?

Daisy-NC · June 17, 2024, 10:51am

Problem:
I have created a Host Collection of Centos7 machines and subscribed them to a Product containing a single repository by going “Host → Host Collections → Select Collection → Subscription Management → Select Subscription → Attach”.

Selecting “Attach” attempts the action, however every host fails to subscribe with the error message “Katello::Errors::CandlepinError: Runtime Error Candlepin does not support JSS versions less than 4.4! at org.candlepin.pki.impl.JSSProviderLoader.getInitializationValuesObject:262”

Expected outcome:
All hosts in the Host Collection are successfully subscribed to the selected Product.

Foreman and Proxy versions:
rpm -qa | grep -i ‘katello|candle’

katello-repos-4.7.6-1.el8.noarch
katello-debug-4.7.6-1.el8.noarch
candlepin-4.2.13-1.el8.noarch
katello-certs-tools-2.9.0-1.el8.noarch
katello-selinux-4.0.2-2.el8.noarch
foreman-installer-katello-3.5.3-1.el8.noarch
katello-common-4.7.6-1.el8.noarch
candlepin-selinux-4.2.13-1.el8.noarch
rubygem-hammer_cli_katello-1.7.3-1.el8.noarch
katello-client-bootstrap-1.7.9-1.el8.noarch
rubygem-katello-4.7.6-1.el8.noarch
katello-4.7.6-1.el8.noarch

Foreman and Proxy plugin versions:

Distribution and version:
cat /etc/os-release

NAME=“AlmaLinux”
VERSION=“8.10 (Cerulean Leopard)”

Other relevant data:
I’m not super familiar with Foreman - it’s possible I’m going about this the wrong way, please let me know if there’s a better way to do this.

I can see the Candlepin package is <4.4, however we have similar setups for other Host Collections (Centos 6) so I don’t understand why this one fails but the other ones work fine with the same version of Candlepin.

Thanks in advance!

Actions::Candlepin::Consumer::AttachSubscription

Input:

{"uuid"=>"89a338a4-d620-4efb-b81f-055c20a7d29f",
 "pool_uuid"=>"2c9c69e48fe2d8fa0190172d1f880141",
 "quantity"=>1,
 "session_id"=>"01026f74-eb58-431f-b8fa-b010debdd313",
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"01026f74-eb58-431f-b8fa-b010debdd313",
 "current_timezone"=>"Europe/London",
 "current_organization_id"=>1,
 "current_location_id"=>nil,
 "current_user_id"=>16}

Output:

{}

Exception:

Katello::Errors::CandlepinError: Runtime Error Candlepin does not support JSS versions less than 4.4! at org.candlepin.pki.impl.JSSProviderLoader.getInitializationValuesObject:262

Backtrace:

/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/propagate_candlepin_errors.rb:26:in `rescue in propagate_candlepin_errors'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/propagate_candlepin_errors.rb:18:in `propagate_candlepin_errors'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/propagate_candlepin_errors.rb:9:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.7.6/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.7.6/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.2.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

evgeni · June 17, 2024, 10:56am

The issue is the new tomcat/jss packages in EL8.10 which break older Candlepin releases.

@ehelms do we happen to have a Candlepin build that’s compatible?

ehelms · June 17, 2024, 11:26am

We did create a special purpose Copy with a build of 4.1:

https://copr.fedorainfracloud.org/coprs/g/theforeman/candlepin-4.1-staging/builds/

I’d just grab the RPM from it. And then consider upgrading your Foreman as
well

evgeni · June 17, 2024, 11:27am

Do we also have something for Candlepin 4.2.z? Katello 4.7 uses that.

Daisy-NC · June 17, 2024, 12:27pm

Would there be any particular process to installing Candlepin 4.1?
Can I just do rpm --install --oldpackage <package>, leave my existing Katello 4.7 packages alone and try again?

Daisy-NC · June 20, 2024, 1:09pm

I tried replacing Candlepin 4.2 with the 4.1 you suggested and I’m running into errors.
The web interface gives me the error:
Oops, we're sorry but something went wrong Katello::Errors::CandlepinNotRunning

Looking up the foreman-rake error gives me the following:

[root@foreman ~]# foreman-rake errors:fetch_log request_id=140bd122
2024-06-20T13:51:05 [I|app|140bd122] Started GET "/" for 172.28.66.100 at 2024-06-20 13:51:05 +0100
2024-06-20T13:51:05 [W|app|140bd122] 404 Not Found
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
 140bd122 | /usr/share/ruby/net/http.rb:933:in `start'
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
 140bd122 | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:275:in `backend_status'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:95:in `block in ping_candlepin_without_auth'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:144:in `exception_watch'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:94:in `ping_candlepin_without_auth'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:245:in `ping_services_for_capsule'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:25:in `ping!'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/services/katello/organization_creator.rb:20:in `create_all_organizations!'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/organization_created_enforcer.rb:12:in `call'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/event_daemon.rb:10:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 140bd122 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/static_dispatcher.rb:68:in `call'
 140bd122 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/extractor/recorder.rb:137:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 140bd122 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/conditional_get.rb:27:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:37:in `call_app'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:28:in `call'
 140bd122 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/lib/katello/prevent_json_parsing.rb:12:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 140bd122 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/engine.rb:539:in `call'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `public_send'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `method_missing'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/configuration.rb:252:in `call'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:77:in `block in handle_request'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:76:in `handle_request'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/server.rb:443:in `process_client'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 140bd122 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-06-20T13:51:05 [I|app|140bd122] Processing by DashboardController#index as HTML
2024-06-20T13:51:05 [I|app|140bd122]   Rendered dashboard/index.html.erb within layouts/application (Duration: 22.5ms | Allocations: 10555)
2024-06-20T13:51:05 [I|app|140bd122]   Rendered layout layouts/application.html.erb (Duration: 22.7ms | Allocations: 10610)
2024-06-20T13:51:05 [W|app|140bd122] Katello::Errors::CandlepinNotRunning
2024-06-20T13:51:05 [I|app|140bd122] Backtrace for 'Katello::Errors::CandlepinNotRunning' error (ActionView::Template::Error): Katello::Errors::CandlepinNotRunning
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/resources/candlepin.rb:27:in `raise_rest_client_exception'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/http_resource.rb:94:in `rescue in issue_request'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/http_resource.rb:84:in `issue_request'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/http_resource.rb:47:in `block (2 levels) in singleton class'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/resources/candlepin/owner.rb:38:in `find'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:20:in `find_owner'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:31:in `owner_details'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:45:in `content_access_mode'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:54:in `block in simple_content_access?'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:752:in `block in save_block_result_to_cache'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `block in instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:751:in `save_block_result_to_cache'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:343:in `fetch'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:53:in `simple_content_access?'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/app/helpers/katello/concerns/dashboard_helper_extensions.rb:35:in `removed_widgets'
 140bd122 | /usr/share/foreman/app/helpers/dashboard_helper.rb:32:in `widgets_to_add'
 140bd122 | /usr/share/foreman/app/helpers/dashboard_helper.rb:16:in `block in dashboard_actions'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/capture_helper.rb:45:in `block in capture'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/capture_helper.rb:209:in `with_output_buffer'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/capture_helper.rb:45:in `capture'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/tag_helper.rb:342:in `content_tag'
 140bd122 | /usr/share/foreman/app/helpers/dashboard_helper.rb:15:in `dashboard_actions'
 140bd122 | /usr/share/foreman/app/views/dashboard/index.html.erb:2:in `_881aa52c78bb781aea19624bcb0c3765'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:247:in `public_send'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:247:in `_run'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:154:in `block in render'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:345:in `instrument_render_template'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:152:in `render'
 140bd122 | /usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:61:in `block (2 levels) in render_template'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:56:in `block in render_template'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:71:in `block in render_with_layout'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:70:in `render_with_layout'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:55:in `render_template'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:11:in `render'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:117:in `block in _render_template'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:273:in `in_rendering_context'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:116:in `_render_template'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/streaming.rb:218:in `_render_template'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:103:in `render_to_body'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/rendering.rb:25:in `render'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:36:in `render'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
 140bd122 | /usr/share/ruby/benchmark.rb:308:in `realtime'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:86:in `cleanup_view_runtime'
 140bd122 | /usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:45:in `render'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:228:in `process_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 140bd122 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 140bd122 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 140bd122 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 140bd122 | /usr/share/gems/gems/audited-5.0.2/lib/audited/sweeper.rb:16:in `around'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 140bd122 | /usr/share/gems/gems/audited-5.0.2/lib/audited/sweeper.rb:16:in `around'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:137:in `run_callbacks'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 140bd122 | /usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:165:in `process'
 140bd122 | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:39:in `process'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:190:in `dispatch'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:254:in `dispatch'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `each'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `serve'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:842:in `call'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/event_daemon.rb:10:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 140bd122 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/static_dispatcher.rb:68:in `call'
 140bd122 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/extractor/recorder.rb:137:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 140bd122 | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/conditional_get.rb:27:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:37:in `call_app'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:28:in `call'
 140bd122 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 140bd122 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 140bd122 | /usr/share/gems/gems/katello-4.7.6/lib/katello/prevent_json_parsing.rb:12:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
 140bd122 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 140bd122 | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 140bd122 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/engine.rb:539:in `call'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `public_send'
 140bd122 | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `method_missing'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
 140bd122 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/configuration.rb:252:in `call'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:77:in `block in handle_request'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:76:in `handle_request'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/server.rb:443:in `process_client'
 140bd122 | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 140bd122 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-06-20T13:51:05 [I|app|140bd122]   Rendered common/500.html.erb within layouts/application (Duration: 4.5ms | Allocations: 6256)
2024-06-20T13:51:05 [I|app|140bd122]   Rendered layouts/base.html.erb (Duration: 5.1ms | Allocations: 7434)
2024-06-20T13:51:05 [I|app|140bd122]   Rendered layout layouts/application.html.erb (Duration: 15.2ms | Allocations: 24698)
2024-06-20T13:51:05 [I|app|140bd122] Completed 500 Internal Server Error in 48ms (Views: 15.6ms | ActiveRecord: 2.0ms | Allocations: 38214)

Foreman version: 3.5.3
Plugins:
 - foreman-tasks 7.2.1
 - foreman_azure_rm 2.2.7
 - foreman_puppet 5.1.2
 - foreman_remote_execution 8.2.0
 - katello 4.7.6
[root@foreman ~]#

Checking the tomcat logs, I see these errors:

[root@foreman ~]# grep -i 'err\|crit\|severe' /var/log/tomcat/catalina.2024-06-20.log
20-Jun-2024 13:45:53.575 SEVERE [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1e4d6796]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(1152999829<closed>)]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Jun-2024 13:45:53.576 SEVERE [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@dbe81e1]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4a5c3d00]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Jun-2024 13:45:53.576 SEVERE [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@3fd8820a]) and a value of type [org.apache.activemq.artemis.utils.actors.HandlerBase.Counter] (value [org.apache.activemq.artemis.utils.actors.HandlerBase$Counter@30e2f9ad]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Jun-2024 13:45:53.592 SEVERE [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@49aad7d9]) and a value of type [org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl] (value [OperationContextImpl [1647663057] [minimalStore=9223372036854775807, storeLineUp=1, stored=1, minimalReplicated=9223372036854775807, replicationLineUp=0, replicated=0, paged=0, minimalPage=9223372036854775807, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Jun-2024 13:45:53.592 SEVERE [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@dbe81e1]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@52309fc6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Jun-2024 13:45:53.592 SEVERE [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@3fd8820a]) and a value of type [org.apache.activemq.artemis.utils.actors.HandlerBase.Counter] (value [org.apache.activemq.artemis.utils.actors.HandlerBase$Counter@57a980e2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Jun-2024 13:45:53.593 SEVERE [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [candlepin] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@3fd8820a]) and a value of type [org.apache.activemq.artemis.utils.actors.HandlerBase.Counter] (value [org.apache.activemq.artemis.utils.actors.HandlerBase$Counter@5fd070c3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
20-Jun-2024 13:45:53.798 SEVERE [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployDirectory Error deploying web application directory [/var/lib/tomcat/webapps/candlepin]
        java.lang.IllegalStateException: Error starting child
        Caused by: java.lang.NoClassDefFoundError: jakarta/servlet/ServletContainerInitializer
20-Jun-2024 13:50:45.165 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public java.lang.Iterable org.candlepin.resource.OwnerResource.createBatchContent(java.lang.String,java.util.List)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@2b611be6]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
20-Jun-2024 13:50:50.186 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
20-Jun-2024 13:50:50.232 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
[root@foreman ~]#

Did I install it wrongly? Is Candlepin 4.1 compatible with Katello 4.7 and Foreman 3.5?

evgeni · June 21, 2024, 11:57am

Nah, I don’t think that downgrading to 4.1 is a good idea for you.

I’ve pushed out 4.2.15 to Builds for @theforeman/candlepin-4.2-staging – would you mind trying that?

Install the pkg, re-run the installer, report back if problems are fixed

eledor · June 26, 2024, 5:15pm

Greetings,

I wanted to chime in here as well because we just encountered this as well. I was attempting to upgrade our cold standby and candlepin refused to start no matter what I tried.

After looking in /var/log/tomcat, i saw this:

cat localhost.2024-06-26.log
26-Jun-2024 00:02:20.012 SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.candlepin.guice.CandlepinContextListener]
org.candlepin.pki.impl.JSSLoaderException: Candlepin does not support JSS versions less than 4.4!

and then Google led me here.

Our current set up on Rocky 8.9 is working but it is broken on Rocky 8.10

We are running Foreman 3.2 / Katello 4.4

@evgeni , would the RPM you pushed up work for us do you think? i could try it on our cold standby.

Thanks

eledor · June 26, 2024, 5:31pm

I should note here that the current candlepin we are running is:

rpm -qa | grep -i candlepin

candlepin-4.1.10-1.el8.noarch
candlepin-selinux-4.1.10-1.el8.noarch

@evgeni and @ehelms please advise here… i am not too keen on “just upgrading” as this is what led to our standby system being in an inoperable state. if we can jump to the next version of foreman and katello without breaking things then we would like to continue the upgrade path

evgeni · June 26, 2024, 6:02pm

If rpm -q candlepin gives anything in the 4.1.z range, use Builds for @theforeman/candlepin-4.1-staging
If rpm -q candlepin gives anything in the 4.2.z range, use Builds for @theforeman/candlepin-4.2-staging

I would expect that to solve your issues, but we stopped testing anything Katello 4.4 related end of 2022, so no guarantees and you better have a backup of your 8.9 based system (in either case, right?).

eledor · June 26, 2024, 6:38pm

ok,

and there was much rejoicing

so i installed the RPM for candlepin 4.1 you listed and re-installed with:

foreman-installer --reset-data --scenario katello --certs-reset

and we are up and running on our cold standby.

thank you for your assistance Evgeni.

a couple of comments here and some questions:

first is some positive feedback- katello 4.4 seems to run better than 4.3. i was seeing OOM occasionally in dmesg but not any longer in 4.4

secondly, is a question on the upgrade method of procedure. not sure how feasible this is but is there a way we can tell if there will be breaking changes going forward?

cheers

evgeni · June 27, 2024, 6:00am

@eledor we do publish release notes, which will include such information (but given any bugfix can be somebody else’s breaking change, you gotta interpret them yourself.

the ones for 4.4 are at Foreman 3.2 and Katello 4.4 Release Notes and you can select the other Foreman/Katello versions in the right upper corner.

eledor · June 27, 2024, 9:23pm

Daisy-NC · July 2, 2024, 1:46pm

Hi Evgeni,
Just had time to try the suggested package but I’m still running into errors about CandlePin not running.
This was the process I took, let me know if I did something wrong?
It doesn’t make sense to me that Foreman would fail because it couldn’t get metadata for a package.
Unfortunately our previous engineer left and I’m not very familiar with how Foreman works.

[root@foreman Candlepin_4.2.15] rpm -i --nosignature --force candlepin-4.2.15-1.el8.noarch.rpm
[root@foreman Candlepin_4.2.15] rpm -e candlepin-4.2.13-1.el8.noarch
[root@foreman Candlepin_4.2.15] rpm -i --nosignature --force candlepin-selinux-4.2.15-1.el8.noarch.rpm
[root@foreman Candlepin_4.2.15] rpm -e candlepin-selinux-4.2.13-1.el8.noarch

[root@foreman Candlepin_4.2.15] rpm -qa | grep candlepin
candlepin-selinux-4.2.15-1.el8.noarch
candlepin-4.2.15-1.el8.noarch

[root@foreman Candlepin_4.2.15]# foreman-installer --list-scenarios
Available scenarios
  Katello (INSTALLED)
        Install Foreman with Katello
  Foreman (use: --scenario foreman)
        Default installation of Foreman
  Foreman Proxy Content (use: --scenario foreman-proxy-content)
        Install a stand-alone Foreman Content Proxy.


[root@foreman Candlepin_4.2.15]# foreman-installer --scenario katello
2024-07-02 14:07:35 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-07-02 14:07:44 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-07-02 14:07:44 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-07-02 14:07:58 [NOTICE] [configure] Starting system configuration.
2024-07-02 14:08:18 [ERROR ] [configure] Could not prefetch package provider 'dnfmodule': Execution of '/usr/bin/dnf module list -y -d 0 -e 1' returned 1: Error: Failed to download metadata for repo 'centos-ansible-29': Cannot prepare internal mirrorlist: Status code: 503 for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=configmanagement-ansible-29 (IP: 100.64.1.1)
2024-07-02 14:08:18 [NOTICE] [configure] 250 configuration steps out of 1887 steps complete.
2024-07-02 14:08:21 [NOTICE] [configure] 500 configuration steps out of 1889 steps complete.
2024-07-02 14:08:22 [NOTICE] [configure] 750 configuration steps out of 1895 steps complete.
2024-07-02 14:08:22 [NOTICE] [configure] 1000 configuration steps out of 1919 steps complete.
2024-07-02 14:08:41 [NOTICE] [configure] 1250 configuration steps out of 1919 steps complete.
2024-07-02 14:09:13 [NOTICE] [configure] 1500 configuration steps out of 1919 steps complete.
2024-07-02 14:09:13 [NOTICE] [configure] 1750 configuration steps out of 1919 steps complete.
2024-07-02 14:09:18 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

[root@foreman Candlepin_4.2.15]# reboot

Logging in and selecting the Host Collections tab gives me a “Katello::Errors::CandlepinNotRunning” message. The foreman-rake output is below:

[root@foreman ~]# foreman-rake errors:fetch_log request_id=0a7d7e9f
2024-07-02T14:31:46 [I|app|0a7d7e9f] Started GET "/host_collections" for 172.28.66.100 at 2024-07-02 14:31:46 +0100
2024-07-02T14:31:46 [W|app|0a7d7e9f] 404 Not Found
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
 0a7d7e9f | /usr/share/ruby/net/http.rb:933:in `start'
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
 0a7d7e9f | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:275:in `backend_status'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:95:in `block in ping_candlepin_without_auth'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:144:in `exception_watch'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:94:in `ping_candlepin_without_auth'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:245:in `ping_services_for_capsule'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/ping.rb:25:in `ping!'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/services/katello/organization_creator.rb:20:in `create_all_organizations!'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/organization_created_enforcer.rb:12:in `call'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/event_daemon.rb:10:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/static_dispatcher.rb:68:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/extractor/recorder.rb:137:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/conditional_get.rb:27:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:37:in `call_app'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:28:in `call'
 0a7d7e9f | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/lib/katello/prevent_json_parsing.rb:12:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 0a7d7e9f | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/engine.rb:539:in `call'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `public_send'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `method_missing'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/configuration.rb:252:in `call'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:77:in `block in handle_request'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:76:in `handle_request'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/server.rb:443:in `process_client'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 0a7d7e9f | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-07-02T14:31:46 [I|app|0a7d7e9f] Processing by Bastion::BastionController#index as HTML
2024-07-02T14:31:46 [I|app|0a7d7e9f]   Parameters: {"bastion_page"=>"host_collections"}
2024-07-02T14:31:46 [I|app|0a7d7e9f]   Rendered /usr/share/gems/gems/katello-4.7.6/engines/bastion/app/views/bastion/layouts/assets.html.erb (Duration: 15.0ms | Allocations: 5391)
2024-07-02T14:31:46 [I|app|0a7d7e9f]   Rendered /usr/share/gems/gems/katello-4.7.6/engines/bastion/app/views/bastion/layouts/application.html.erb (Duration: 16.2ms | Allocations: 6591)
2024-07-02T14:31:46 [W|app|0a7d7e9f] Katello::Errors::CandlepinNotRunning
2024-07-02T14:31:46 [I|app|0a7d7e9f] Backtrace for 'Katello::Errors::CandlepinNotRunning' error (ActionView::Template::Error): Katello::Errors::CandlepinNotRunning
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/resources/candlepin.rb:27:in `raise_rest_client_exception'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/http_resource.rb:94:in `rescue in issue_request'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/http_resource.rb:84:in `issue_request'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/http_resource.rb:47:in `block (2 levels) in singleton class'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/lib/katello/resources/candlepin/owner.rb:38:in `find'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:20:in `find_owner'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:31:in `owner_details'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:45:in `content_access_mode'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:54:in `block in simple_content_access?'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:752:in `block in save_block_result_to_cache'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `block in instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:751:in `save_block_result_to_cache'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:343:in `fetch'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/app/models/katello/glue/candlepin/owner.rb:53:in `simple_content_access?'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/engines/bastion/app/views/bastion/layouts/assets.html.erb:21:in `block in _74814b80b38a1ace2b8933164d158e52'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/capture_helper.rb:45:in `block in capture'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/capture_helper.rb:209:in `with_output_buffer'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/capture_helper.rb:45:in `capture'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/capture_helper.rb:159:in `content_for'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/engines/bastion/app/views/bastion/layouts/assets.html.erb:12:in `_74814b80b38a1ace2b8933164d158e52'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:247:in `public_send'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:247:in `_run'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:154:in `block in render'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:345:in `instrument_render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:152:in `render'
 0a7d7e9f | /usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:61:in `block (2 levels) in render_template'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:56:in `block in render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:75:in `render_with_layout'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:55:in `render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:11:in `render'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/renderer.rb:22:in `render'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/rendering_helper.rb:38:in `block in render'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:273:in `in_rendering_context'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/helpers/rendering_helper.rb:34:in `render'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/engines/bastion/app/views/bastion/layouts/application.html.erb:10:in `_4ff23cf565bb285613f99f4e9b8c847f'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:247:in `public_send'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:247:in `_run'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:154:in `block in render'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:345:in `instrument_render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/template.rb:152:in `render'
 0a7d7e9f | /usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:61:in `block (2 levels) in render_template'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:56:in `block in render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:75:in `render_with_layout'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:55:in `render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/template_renderer.rb:11:in `render'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:117:in `block in _render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/base.rb:273:in `in_rendering_context'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:116:in `_render_template'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/streaming.rb:218:in `_render_template'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:103:in `render_to_body'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/rendering.rb:25:in `render'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:36:in `render'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
 0a7d7e9f | /usr/share/ruby/benchmark.rb:308:in `realtime'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:86:in `cleanup_view_runtime'
 0a7d7e9f | /usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:45:in `render'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/engines/bastion/app/controllers/bastion/bastion_controller.rb:10:in `index'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:228:in `process_action'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 0a7d7e9f | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 0a7d7e9f | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 0a7d7e9f | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 0a7d7e9f | /usr/share/gems/gems/audited-5.0.2/lib/audited/sweeper.rb:16:in `around'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 0a7d7e9f | /usr/share/gems/gems/audited-5.0.2/lib/audited/sweeper.rb:16:in `around'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:137:in `run_callbacks'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 0a7d7e9f | /usr/share/gems/gems/activerecord-6.1.7.3/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:165:in `process'
 0a7d7e9f | /usr/share/gems/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:39:in `process'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:190:in `dispatch'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:254:in `dispatch'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/mapper.rb:49:in `serve'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `each'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `serve'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:842:in `call'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/lib/katello/middleware/event_daemon.rb:10:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/static_dispatcher.rb:68:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/extractor/recorder.rb:137:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/static.rb:24:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 0a7d7e9f | /usr/share/gems/gems/apipie-rails-0.8.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/conditional_get.rb:27:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:37:in `call_app'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:28:in `call'
 0a7d7e9f | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 0a7d7e9f | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 0a7d7e9f | /usr/share/gems/gems/katello-4.7.6/lib/katello/prevent_json_parsing.rb:12:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
 0a7d7e9f | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 0a7d7e9f | /usr/share/gems/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 0a7d7e9f | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/engine.rb:539:in `call'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `public_send'
 0a7d7e9f | /usr/share/gems/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `method_missing'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
 0a7d7e9f | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/configuration.rb:252:in `call'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:77:in `block in handle_request'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/request.rb:76:in `handle_request'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/server.rb:443:in `process_client'
 0a7d7e9f | /usr/share/gems/gems/puma-5.6.5/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 0a7d7e9f | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-07-02T14:31:46 [I|app|0a7d7e9f]   Rendered common/500.html.erb within layouts/application (Duration: 1.4ms | Allocations: 1587)
2024-07-02T14:31:46 [I|app|0a7d7e9f]   Rendered layouts/base.html.erb (Duration: 2.9ms | Allocations: 3076)
2024-07-02T14:31:46 [I|app|0a7d7e9f]   Rendered layout layouts/application.html.erb (Duration: 6.0ms | Allocations: 7082)
2024-07-02T14:31:46 [I|app|0a7d7e9f] Completed 500 Internal Server Error in 32ms (Views: 6.3ms | ActiveRecord: 1.8ms | Allocations: 16554)

Foreman version: 3.5.3
Plugins:
 - foreman-tasks 7.2.1
 - foreman_azure_rm 2.2.7
 - foreman_puppet 5.1.2
 - foreman_remote_execution 8.2.0
 - katello 4.7.6
[root@foreman ~]#

Any help would be appreciated!

evgeni · July 9, 2024, 11:58am

You can safely drop that repo, it’s unused since EL8 ships a new enough ansible-core itself now.

But I agree, the follow up errors do not look like related to the ansible repo at all.

Can you show me systemctl status tomcat output?

Daisy-NC · August 5, 2024, 8:54am

I’m in the process of upgrading to Foreman 3.10, can you confirm that once I reach a supported version there should be no need to update candlepin manually?

Daisy-NC · August 5, 2024, 1:18pm

Upgrading to 3.9 is giving me problems, I get the following error with db:migrate

[root@foreman ~]# foreman-installer --scenario katello
2024-08-05 13:49:31 [WARN  ] [boot] <Array> ["Unsetting environment variable 'http_proxy' for the duration of the install."]
2024-08-05 13:49:31 [WARN  ] [boot] <Array> ["Unsetting environment variable 'https_proxy' for the duration of the install."]
2024-08-05 13:49:32 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-08-05 13:49:40 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-08-05 13:49:40 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-08-05 13:49:51 [NOTICE] [configure] Starting system configuration.
2024-08-05 13:50:17 [NOTICE] [configure] 250 configuration steps out of 1921 steps complete.
2024-08-05 13:50:21 [NOTICE] [configure] 500 configuration steps out of 1922 steps complete.
2024-08-05 13:50:30 [NOTICE] [configure] 750 configuration steps out of 1926 steps complete.
2024-08-05 13:50:31 [NOTICE] [configure] 1000 configuration steps out of 1932 steps complete.
2024-08-05 13:50:31 [NOTICE] [configure] 1250 configuration steps out of 1933 steps complete.
2024-08-05 13:50:49 [NOTICE] [configure] 1500 configuration steps out of 1933 steps complete.
2024-08-05 13:51:44 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-08-05 13:51:44 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-08-05 13:52:14 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-08-05 13:52:14 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-08-05 13:52:29 [NOTICE] [configure] 1750 configuration steps out of 1933 steps complete.
2024-08-05 13:54:13 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Exec resource 'foreman-rake-db:migrate' failed. Logs:
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]
    Adding autorequire relationship with User[foreman]
    Starting to evaluate the resource (1667 of 1933)
    Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
    '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
    Evaluated in 60.26 seconds
  Exec[foreman-rake-db:migrate](provider=posix)
    Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
    Executing '/usr/sbin/foreman-rake db:migrate'
    Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
    Executing '/usr/sbin/foreman-rake db:migrate'
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless
    rake aborted!
    Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `new'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `connect_addrinfo'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:193:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each_with_index'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:25:in `with'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:487:in `block in delete'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `block in instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:205:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:486:in `delete'
    /usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
    /usr/share/foreman/config/initializers/foreman.rb:35:in `block in <top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `instance_exec'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `block in make_lambda'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `catch'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `block in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:199:in `block in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `block in invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `each'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:105:in `run_callbacks'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/reloader.rb:88:in `prepare!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:abort_if_pending_migrations => db:load_config => environment
    (See full trace by running task with --trace)
    rake aborted!
    Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `new'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `connect_addrinfo'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:193:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each_with_index'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:25:in `with'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:487:in `block in delete'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `block in instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:205:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:486:in `delete'
    /usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
    /usr/share/foreman/config/initializers/foreman.rb:35:in `block in <top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `instance_exec'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `block in make_lambda'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `catch'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `block in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:199:in `block in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `block in invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `each'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:105:in `run_callbacks'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/reloader.rb:88:in `prepare!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:abort_if_pending_migrations => db:load_config => environment
    (See full trace by running task with --trace)
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns
    rake aborted!
    Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `new'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `connect_addrinfo'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:193:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each_with_index'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:25:in `with'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:487:in `block in delete'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `block in instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:205:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:486:in `delete'
    /usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
    /usr/share/foreman/config/initializers/foreman.rb:35:in `block in <top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `instance_exec'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `block in make_lambda'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `catch'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `block in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:199:in `block in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `block in invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `each'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:105:in `run_callbacks'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/reloader.rb:88:in `prepare!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:migrate => db:load_config => environment
    (See full trace by running task with --trace)
    change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
    rake aborted!
    Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `new'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `connect_addrinfo'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:193:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each_with_index'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:25:in `with'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:487:in `block in delete'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `block in instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:205:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:486:in `delete'
    /usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
    /usr/share/foreman/config/initializers/foreman.rb:35:in `block in <top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `instance_exec'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `block in make_lambda'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `catch'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `block in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:199:in `block in halting'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `block in invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `each'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:105:in `run_callbacks'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/reloader.rb:88:in `prepare!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:migrate => db:load_config => environment
    (See full trace by running task with --trace)

1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/katello.log

Is there a config file where I need to change db:migrate from notrun to 0? Where is this file located?

Thanks!

Daisy-NC · August 8, 2024, 9:38am

I fixed the above install error, can read more about it here

Will continue upgrading to 3.10 and hopefully the updated Candlepin will resolve the initial issue.