Oops, we're sorry but something went wrong A backend service [ Candlepin ] is unreachable

Support
1

Problem: WebUI shows this error while I try to open ,“Oops, we’re sorry but something went wrong** A backend service [ Candlepin ] is unreachable”

Expected outcome: Use WebUI without any Candlepin Unreachable error.

Foreman and Proxy versions:
Foreman 3.3 and Katello 4.5

Distribution and version:
Rocky-8-GenericCloud-8.5

Other relevant data:
Firewall is disabled in the machine.
Status:

candlepin:
    Status:          FAIL
    Server Response: Message: Failed to open TCP connection to localhost:23443 (
Connection refused - connect(2) for "localhost" port 23443)
candlepin_auth:
    Status:          FAIL
    Server Response: Message: A backend service [ Candlepin ] is unreachable
candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:
    Status:          ok
    Server Response: Duration: 216ms
    Server Response: Duration: 0ms
pulp3:
    Status:          ok
    Server Response: Duration: 201ms
pulp3_content:
    Status:          ok
    Server Response: Duration: 156ms
foreman_tasks:
    Status:          FAIL
    Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status
2

Tomcat service is in failed state.

● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Sat 2022-09-03 18:43:42 UTC; 2 days ago
 Main PID: 189718 (code=killed, signal=KILL)

Sep 03 18:24:07 foreman-rocky server[189718]: 03-Sep-2022 18:24:07.794 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.RulesCurator.delete
(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@7760e931]. This could indicate a bug.  The method may be interce
pted twice, or may not be intercepted at all.
Sep 03 18:24:07 foreman-rocky server[189718]: 03-Sep-2022 18:24:07.795 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public org.candlepin.model.Persisted org.candlepin.
model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@7760e931]. This could indicate a bug.
The method may be intercepted twice, or may not be intercepted at all.
Sep 03 18:24:08 foreman-rocky server[189718]: 03-Sep-2022 18:24:08.045 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.ContentCurator.dele
te(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@7760e931]. This could indicate a bug.  The method may be inter
cepted twice, or may not be intercepted at all.
Sep 03 18:24:08 foreman-rocky server[189718]: 03-Sep-2022 18:24:08.182 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public void org.candlepin.model.EntitlementCertific
ateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@7760e931]. This could indicate a bug.  The meth
od may be intercepted twice, or may not be intercepted at all.
Sep 03 18:24:08 foreman-rocky server[189718]: 03-Sep-2022 18:24:08.255 WARNING [main] com.google.inject.internal.ProxyFactory.<init> Method [public java.lang.Iterable org.candlepin.resource.Ow
nerResource.createBatchContent(java.lang.String,java.util.List)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@7760e931]. This could indicate a
 bug.  The method may be intercepted twice, or may not be intercepted at all.
Sep 03 18:24:34 foreman-rocky server[189718]: 03-Sep-2022 18:24:34.996 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomc
at/webapps/candlepin] has finished in [73,125] ms
Sep 03 18:24:34 foreman-rocky server[189718]: 03-Sep-2022 18:24:35.014 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-127.0.0.1-23443"]
Sep 03 18:24:34 foreman-rocky server[189718]: 03-Sep-2022 18:24:35.052 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [73,291] milliseconds
Sep 03 18:43:42 foreman-rocky systemd[1]: tomcat.service: Main process exited, code=killed, status=9/KILL
Sep 03 18:43:42 foreman-rocky systemd[1]: tomcat.service: Failed with result 'signal'.
3

The immediate fix is to restart Candlepin:

foreman-maintain service restart --only tomcat

The underlying cause could be that the system doesn’t have enough RAM. How much does yours have?

4

"foreman-maintain service restart " would not be a permanent fix for the issue.
Machine Spec:
RAM-8GB
CPU-4
STORAGE-50GB

5

8 GB is not nearly enough RAM to run Katello. With only 8 GB you’ll have this issue often.

Please see the minimum requirements here: Installing Foreman nightly Server with Katello nightly Plugin on RHEL/CentOS

1 Like