Sorry for being late to the party, but finally had some time to test the RCs 
And I’m not sure what exactly happened yet, but it looks like the frontend can’t connect to Candlepin,
there are several pages not working (Subscription, Red Hat repository, Products, … pages showing either a white page or an Katello::Errors::CandlepinNotRunning error, Repository set page on the host detail page showing only a grey bar)
The only page that is still showing content normally is the sync_status page.
The upgrade process looked completely normal as well as the final repo upgrade worked without any issues.
In the production log, I see this error every few seconds:
2024-06-12T17:21:35 [E|app|81d25a51] Error occurred while starting Katello::CandlepinEventListener
2024-06-12T17:21:35 [E|app|81d25a51] Connection refused - connect(2) for "localhost" port 61613
2024-06-12T17:21:35 [E|app|81d25a51] /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in initialize
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in open
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:461:in block in open_ssl_socket
81d25a51 | /usr/share/ruby/timeout.rb:78:in timeout
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:460:in open_ssl_socket
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/netio.rb:520:in open_socket
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:116:in block in socket
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:109:in synchronize
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/connection/utils.rb:109:in socket
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/connection.rb:173:in initialize
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:134:in new
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:134:in create_connection
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:101:in block in initialize
81d25a51 | /usr/share/ruby/timeout.rb:95:in block in timeout
81d25a51 | /usr/share/ruby/timeout.rb:33:in block in catch
81d25a51 | /usr/share/ruby/timeout.rb:33:in catch
81d25a51 | /usr/share/ruby/timeout.rb:33:in catch
81d25a51 | /usr/share/ruby/timeout.rb:110:in timeout
81d25a51 | /usr/share/gems/gems/stomp-1.4.10/lib/stomp/client.rb:99:in initialize
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/messaging/stomp_connection.rb:69:in new
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/messaging/stomp_connection.rb:69:in client
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/messaging/stomp_connection.rb:43:in subscribe
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/services/katello/candlepin_event_listener.rb:34:in run
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/monitor.rb:40:in block in check_services
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/monitor.rb:30:in each
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/monitor.rb:30:in check_services
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/monitor.rb:16:in block (2 levels) in start
81d25a51 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/execution_wrapper.rb:87:in wrap
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/monitor.rb:15:in block in start
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/monitor.rb:14:in loop
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/monitor.rb:14:in start
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/runner.rb:84:in block (2 levels) in start_monitor_thread
81d25a51 | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/execution_wrapper.rb:91:in wrap
81d25a51 | /usr/share/gems/gems/katello-4.13.0.rc1/app/lib/katello/event_daemon/runner.rb:83:in block in start_monitor_thread
81d25a51 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context
Another one I see is a repeating notification error:
2024-06-12T17:24:37 [I|app|12d98b0b] Started GET "/notification_recipients" for 10.27.12.2 at 2024-06-12 17:24:37 +0200
2024-06-12T17:24:37 [W|app|12d98b0b] 404 Not Found
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in exception_with_response
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in return!
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in process_result
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in block in transmit
12d98b0b | /usr/share/ruby/net/http.rb:933:in start
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in transmit
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in execute
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in execute
12d98b0b | /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in get
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/app/models/katello/ping.rb:264:in backend_status
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/app/models/katello/ping.rb:87:in block in ping_candlepin_without_auth
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/app/models/katello/ping.rb:136:in exception_watch
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/app/models/katello/ping.rb:86:in ping_candlepin_without_auth
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/app/models/katello/ping.rb:236:in ping_services_for_capsule
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/app/models/katello/ping.rb:24:in ping!
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/app/services/katello/organization_creator.rb:20:in create_all_organizations!
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/lib/katello/middleware/organization_created_enforcer.rb:12:in call
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/lib/katello/middleware/event_daemon.rb:10:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/apipie-dsl-2.6.2/lib/apipie_dsl/static_dispatcher.rb:67:in call
12d98b0b | /usr/share/gems/gems/apipie-rails-1.3.0/lib/apipie/static_dispatcher.rb:74:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/static.rb:24:in call
12d98b0b | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in call
12d98b0b | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call
12d98b0b | /usr/share/gems/gems/apipie-rails-1.3.0/lib/apipie/middleware/checksum_in_headers.rb:27:in call
12d98b0b | /usr/share/gems/gems/prometheus-client-4.2.2/lib/prometheus/middleware/exporter.rb:33:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/tempfile_reaper.rb:15:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/etag.rb:27:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/conditional_get.rb:27:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/head.rb:12:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/permissions_policy.rb:22:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/content_security_policy.rb:19:in call
12d98b0b | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:266:in context
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/session/abstract/id.rb:260:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/cookies.rb:697:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:27:in block in call
12d98b0b | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:98:in run_callbacks
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:26:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/debug_exceptions.rb:29:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in call
12d98b0b | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:37:in call_app
12d98b0b | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:28:in call
12d98b0b | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in call
12d98b0b | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/remote_ip.rb:81:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/request_id.rb:26:in call
12d98b0b | /usr/share/gems/gems/katello-4.13.0.rc1/lib/katello/prevent_json_parsing.rb:12:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/method_override.rb:24:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/runtime.rb:22:in call
12d98b0b | /usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/executor.rb:14:in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/sendfile.rb:110:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/ssl.rb:77:in call
12d98b0b | /usr/share/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/host_authorization.rb:142:in call
12d98b0b | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in call
12d98b0b | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/engine.rb:539:in call
12d98b0b | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in public_send
12d98b0b | /usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in method_missing
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in block in call
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in each
12d98b0b | /usr/share/gems/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in call
12d98b0b | /usr/share/gems/gems/puma-6.4.2/lib/puma/configuration.rb:272:in call
12d98b0b | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:100:in block in handle_request
12d98b0b | /usr/share/gems/gems/puma-6.4.2/lib/puma/thread_pool.rb:378:in with_force_shutdown
12d98b0b | /usr/share/gems/gems/puma-6.4.2/lib/puma/request.rb:99:in handle_request
12d98b0b | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:464:in process_client
12d98b0b | /usr/share/gems/gems/puma-6.4.2/lib/puma/server.rb:245:in block in run
12d98b0b | /usr/share/gems/gems/puma-6.4.2/lib/puma/thread_pool.rb:155:in block in spawn_thread
12d98b0b | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context
Also checked if tomcat for candlepin is running and it looks like it does (though not without errors)
● tomcat.service - Apache Tomcat Web Application Container
Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2024-06-12 17:09:27 CEST; 17min ago
Main PID: 1016 (java)
Tasks: 38 (limit: 126422)
Memory: 1017.3M
CGroup: /system.slice/tomcat.service
└─1016 /usr/lib/jvm/jre-17/bin/java -Xms1024m -Xmx4096m -Dcom.redhat.fips=false -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: Hint: Cast the output of the non-recursive term to the correct type.
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: Position: 655
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: ERROR: Exception Primary Source: PostgreSQL 13.14
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: 12-Jun-2024 17:09:59.356 INFO [main] liquibase.command.null Command execution complete
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: 12-Jun-2024 17:09:59.360 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: 12-Jun-2024 17:09:59.362 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: 12-Jun-2024 17:09:59.366 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [candlepin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: 12-Jun-2024 17:09:59.430 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/candlepin] has finished in [25,919] ms
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: 12-Jun-2024 17:09:59.460 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
Jun 12 17:09:59 r8-foreman-prod.fritz.box server[1016]: 12-Jun-2024 17:09:59.671 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [26387] milliseconds
Foreman and Katello versions:
Foreman 3.10 → 3.11
Katello 4.12 → 4.13
Foreman and Proxy plugin versions:
foreman_tasks 9.1.1
foreman_ansible 14.0.0
foreman_bootdisk 21.2.3
foreman_puppet 6.3.0
foreman_remote_execution 13.1.0
foreman_snapshot_management 3.0.1
foreman_statistics 2.1.0
foreman_templates 9.4.0
foreman_ovirt 3.11.0
foreman_vmware 3.11.0
Distribution and version:
Rocky Linux 8.10
I will do a rollback to the snapshot before the upgrade now and try another time in the next days, maybe it was a intermittent fluke on my system.
But still wanted to report that back as the GA is coming closer.
Cheers, Lukas
)
