Oops, we’re sorry but something went wrong A backend service [ Candlepin ] is unreachable

Support
1

Problem: Oops, we’re sorry but something went wrong A backend service [ Candlepin ] is unreachable

**Expected outcome:**Foreman web UI should run without any errors.

**Foreman and Proxy versions:**Foreman 3.12 with katello 4.14

**Foreman and Proxy plugin versions:**Foreman 3.12 with katello 4.14

Distribution and version: Rocky 9.5

Other relevant data:

We are getting candlepin is unreachable error on foreman web UI.

foreman-maintain health check
Running preparation steps required to run the next scenarios
================================================================================
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------


Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [FAIL]
Some components are failing: candlepin, candlepin_auth, candlepin_events
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)] y
Restart applicable services:

Restarting the following service(s):
postgresql (candlepin), tomcat
\ restarting postgresql (candlepin)
postgresql (candlepin) is remote and is UP. Remote databases are not managed by foreman-maintain and therefore was not restarted.

| All services restarted
\ Try 1/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events
\ Try 2/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events
- Try 3/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events
| Try 4/5: checking status of hammer ping
Some components are failing: candlepin, candlepin_auth, candlepin_events
\ Try 5/5: checking status of hammer ping                             [FAIL]
Server response check failed!
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check whether all services are running using the ping call:           [FAIL]
Some components are failing: candlepin, candlepin_auth, candlepin_events
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no)] n
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario::FilteredScenario] failed.

The following steps ended up in failing state:

  [server-ping]

Resolve the failed steps and rerun the command.
In case the failures are false positives, use
--whitelist="server-ping


[root@testserver candlepin]# hammer ping
database:
    Status:          ok
    Server Response: Duration: 0ms
cache:
    servers:
     1) Status:          ok
        Server Response: Duration: 0ms
candlepin:
    Status:          FAIL
    Server Response: Message: Failed to open TCP connection to localhost:23443 (Connection refused - connect(2) for "localhost" port 23443)
candlepin_auth:
    Status:          FAIL
    Server Response: Message: A backend service [ Candlepin ] is unreachable
candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:
    Status:          ok
    Server Response: Duration: 71ms
pulp3_content:
    Status:          ok
    Server Response: Duration: 129ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 2ms

One server is for foreman and another server we are using it for postgresql DB.

 cat candlepin.log
2025-01-02 10:02:37,771 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-02 10:02:37,778 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-02 10:02:37,782 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-02 10:02:37,825 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-02 10:02:37,834 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-02 10:02:37,839 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-02 10:02:37,908 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-02 10:02:39,419 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-02 10:02:39,569 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-02 10:02:39,920 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-02 10:02:39,920 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-02 10:02:39,920 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-02 10:02:39,921 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-02 10:02:39,922 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-02 10:02:39,922 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-02 10:02:39,922 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-02 10:02:39,923 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-02 10:02:39,924 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-02 10:02:39,925 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-02 10:02:39,925 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-02 10:02:39,925 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-02 10:02:39,926 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-02 10:02:39,926 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-02 10:02:41,594 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:02:41,712 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:02:41,713 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:02:42,571 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381
2025-01-02 10:07:26,602 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin initializing context.
2025-01-02 10:07:26,603 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin reading configuration.
2025-01-02 10:07:26,607 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Loading candlepin.conf configuration!
2025-01-02 10:07:26,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Validating configurations.
2025-01-02 10:07:26,656 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CandlepinContextListener - Candlepin will show support for the following capabilities: [instance_multiplier, derived_product, vcpu, cert_v3, hypervisors_heartbeat, remove_by_pool_id, syspurpose, storage_band, cores, multi_environment, hypervisors_async, org_level_content_access, typed_environments, guest_limit, ram, batch_bind]
2025-01-02 10:07:26,660 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.DatabaseConnectionManager - Attempt 1 out of 3 to connect to the database.
2025-01-02 10:07:26,727 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Liquibase startup management set to Manage
2025-01-02 10:07:28,161 [thread=main] [=, org=, csid=] INFO  org.candlepin.database.MigrationManager - Candlepin database is up to date!
2025-01-02 10:07:28,310 [thread=main] [=, org=, csid=] INFO  org.candlepin.guice.CustomizableModules - Found custom module module.config.adapter_module
2025-01-02 10:07:28,645 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ActiveEntitlementJob: org.candlepin.async.tasks.ActiveEntitlementJob
2025-01-02 10:07:28,646 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CertificateCleanupJob: org.candlepin.async.tasks.CertificateCleanupJob
2025-01-02 10:07:28,646 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlerJob: org.candlepin.async.tasks.EntitlerJob
2025-01-02 10:07:28,646 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitleByProductsJob: org.candlepin.async.tasks.EntitleByProductsJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExpiredPoolsCleanupJob: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ExportJob: org.candlepin.async.tasks.ExportJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HealEntireOrgJob: org.candlepin.async.tasks.HealEntireOrgJob
2025-01-02 10:07:28,647 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorHeartbeatUpdateJob: org.candlepin.async.tasks.HypervisorHeartbeatUpdateJob
2025-01-02 10:07:28,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: HypervisorUpdateJob: org.candlepin.async.tasks.HypervisorUpdateJob
2025-01-02 10:07:28,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportJob: org.candlepin.async.tasks.ImportJob
2025-01-02 10:07:28,648 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ImportRecordCleanerJob: org.candlepin.async.tasks.ImportRecordCleanerJob
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: JobCleaner: org.candlepin.async.tasks.JobCleaner
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ManifestCleanerJob: org.candlepin.async.tasks.ManifestCleanerJob
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsForProductJob: org.candlepin.async.tasks.RefreshPoolsForProductJob
2025-01-02 10:07:28,649 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RefreshPoolsJob: org.candlepin.async.tasks.RefreshPoolsJob
2025-01-02 10:07:28,650 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenEnvEntitlementCertsJob: org.candlepin.async.tasks.RegenEnvEntitlementCertsJob
2025-01-02 10:07:28,650 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: RegenProductEntitlementCertsJob: org.candlepin.async.tasks.RegenProductEntitlementCertsJob
2025-01-02 10:07:28,650 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UndoImportsJob: org.candlepin.async.tasks.UndoImportsJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: InactiveConsumerCleanerJob: org.candlepin.async.tasks.InactiveConsumerCleanerJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: CloudAccountOrgSetupJob: org.candlepin.async.tasks.CloudAccountOrgSetupJob
2025-01-02 10:07:28,651 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: ConsumerMigrationJob: org.candlepin.async.tasks.ConsumerMigrationJob
2025-01-02 10:07:28,652 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: EntitlementRevokingJob: org.candlepin.async.tasks.RevokeEntitlementsJob
2025-01-02 10:07:30,296 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:31,257 [thread=main] [=, org=, csid=] INFO  org.candlepin.policy.js.JsRunnerProvider - Recompiling rules with timestamp: 2024-12-31 14:53:08.381


cat /var/log/candlepin/error.log
2025-01-02 10:02:41,594 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:02:41,712 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:02:41,713 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:30,296 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:07:30,406 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute
2025-01-02 10:08:43,715 [thread=main] [=, org=, csid=] WARN  org.hibernate.id.UUIDHexGenerator - HHH000409: Using org.hibernate.id.UUIDHexGenerator which does not generate IETF RFC 4122 compliant UUID values; consider using org.hibernate.id.UUIDGenerator instead
2025-01-02 10:08:43,828 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000038: Composite-id class does not override equals(): org.candlepin.model.PoolAttribute
2025-01-02 10:08:43,828 [thread=main] [=, org=, csid=] WARN  org.hibernate.mapping.RootClass - HHH000039: Composite-id class does not override hashCode(): org.candlepin.model.PoolAttribute

Please help us on this issue. Please let me know if logs required from our end.

2

Is anyone can help us on this issue

3

Is this a fresh installation? If no, what did you change before it stopped working (e.g. upgrade)?

In any case, I guess the root cause is right here:

Your candlepin logs look alright, according to the logs candlepin is up and running just fine. So my guess would be, that the hostfirewall or some sort of security software is blocking access to candlepins port.
Check your firewall or any security software you might have installed if they block localhost connections to port 23443.

4

Its a fresh installation and could see the below errors in tomcat logs.

Dec 30 08:15:15  server[3376632]: 30-Dec-2024 08:15:15.456 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
Dec 30 08:15:15  server[3376632]: 30-Dec-2024 08:15:15.460 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/candlepin] startup failed due to previous errors
5

HI Team,

Anyone please help us on this issue

6

See Requirements for foreman katello