Can not access foreman web after restart the server

Support
1

Hi,everyone:

I installed the foreman it works well, and I restart the server ,but can not access the web page any more.
It appears that :

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

I checked the server log, there are errors on httpd:

[Fri Dec 25 21:10:52.080732 2020] [proxy:error] [pid 13641] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed
[Fri Dec 25 21:10:52.081136 2020] [proxy:error] [pid 13641] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s
[Fri Dec 25 21:10:52.081180 2020] [proxy_http:error] [pid 13641] [client 192.168.10.104:57060] AH01114: HTTP: failed to make connection to backend: 127.0.0.1
[Fri Dec 25 21:10:52.119776 2020] [proxy:error] [pid 13638] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed
[Fri Dec 25 21:10:52.119929 2020] [proxy:error] [pid 13638] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s
[Fri Dec 25 21:10:52.119943 2020] [proxy_http:error] [pid 13638] [client 192.168.10.104:57062]

and the server of foreman are all well.

what should I do to resove the problem ?

Thanks!

2

How did you restart the server? is there anything in /var/log/foreman/production.log?

3

I used the foreman-maintain service stop ,and then restart the linux server.
the message in /var/log/foreman/production.log like this:

2020-12-25T20:46:33 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-12-25T20:46:33 [W|app|] Scoped order is ignored, it’s forced to be batch order.
2020-12-25T20:46:34 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-12-25T20:46:50 [I|app|] No new seed file updates found. Skipping
2020-12-25T20:47:00 [I|dyn|] Acquired orchestrator lock, entering active mode.
2020-12-25T20:47:00 [I|app|] Everything ready for world: d19e7792-08a8-4557-adf9-c58ffddfe42b
2020-12-25T20:47:32 [I|app|] Rails cache backend: File
2020-12-25T20:47:37 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-12-25T20:47:38 [W|app|] Scoped order is ignored, it’s forced to be batch order.
2020-12-25T20:47:38 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-12-25T20:47:49 [I|app|] No new seed file updates found. Skipping
2020-12-25T20:47:59 [I|app|] Everything ready for world: 1fcfebce-e7b8-4cfe-b58d-38d14d3f21a0
2020-12-25T20:47:59 [I|dyn|] Performing validity checks
2020-12-25T20:47:59 [I|dyn|] Finished performing validity checks
2020-12-25T20:48:04 [I|app|] Rails cache backend: File
2020-12-25T20:48:08 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-12-25T20:48:08 [W|app|] Scoped order is ignored, it’s forced to be batch order.
2020-12-25T20:48:08 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-12-25T20:48:20 [I|app|] No new seed file updates found. Skipping
2020-12-25T20:48:30 [I|app|] Everything ready for world: 8af8a727-acdf-46c2-b24a-23a030a1a173
2020-12-25T21:00:06 [I|app|] Rails cache backend: File
2020-12-25T21:00:10 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-12-25T21:00:11 [W|app|] Scoped order is ignored, it’s forced to be batch order.
2020-12-25T21:00:11 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-12-25T21:30:06 [I|app|] Rails cache backend: File
2020-12-25T21:30:10 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-12-25T21:30:11 [W|app|] Scoped order is ignored, it’s forced to be batch order.
2020-12-25T21:30:11 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.

4

there are some error message like this:

==> /var/log/messages <==
Dec 25 21:39:06 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 21:39:06 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.

other logs are right.

5

what is the output of foreman-maintain service status?

6

all service are ok:

All services are running [OK]

but,there are two error message like these:

Dec 25 20:47:03 foreman.tltd.com pulp[13632]: gofer.messaging.adapter.qpid.connection:INFO: open: URL: ssl://localhost|SSL: ca: /etc/pki/pulp/qpid/ca.crt|key: None|certificate: /etc/pki/pulp/qpid/client.crt|host-validation: False

Dec 25 20:46:15 foreman.tltd.com qdrouterd[13063]: 2020-12-25 20:46:15.481322 +0800 SERVER (info) [C1] Connection to localhost:5671 failed: proton:io Connection refused - on read from localhost:5671

7

[root@foreman foreman]# netstat -anp | grep 5671
tcp 0 0 127.0.0.1:5671 0.0.0.0:* LISTEN 13067/qpidd
tcp6 0 0 ::1:5671 :::* LISTEN 13067/qpidd

8

the apache httpd log is :

[Fri Dec 25 21:30:13.316626 2020] [proxy:error] [pid 13641] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed
[Fri Dec 25 21:30:13.316858 2020] [proxy:error] [pid 13641] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s
[Fri Dec 25 21:30:13.316888 2020] [proxy_http:error] [pid 13641] [client 192.168.10.104:57223] AH01114: HTTP: failed to make connection to backend: 127.0.0.1
[Fri Dec 25 21:30:13.350365 2020] [proxy:error] [pid 13638] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed
[Fri Dec 25 21:30:13.350407 2020] [proxy:error] [pid 13638] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s
[Fri Dec 25 21:30:13.350414 2020] [proxy_http:error] [pid 13638] [client 192.168.10.104:57225] AH01114: HTTP: failed to make connection to backend: 127.0.0.1, referer: https://foreman.tltd.com/

9

are there any selinux denials perhaps?

10

No, I closed the selinux, but still can not access.

11

[root@foreman ~]# getenforce
Disabled
[root@foreman ~]#
[root@foreman ~]#
[root@foreman ~]#
[root@foreman ~]# sestatus
SELinux status: disabled
[root@foreman ~]#

12

in /var/log/message , many such errors:

Dec 25 22:10:21 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:10:37 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:10:37 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:10:52 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:10:52 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:07 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:07 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:22 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:22 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:38 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:38 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:53 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:11:53 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:08 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:08 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:23 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:23 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:39 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:39 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:54 foreman pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Dec 25 22:12:54 foreman pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.

13

What version of Foreman/Katello are you on?

14

Please, how to show the version?
which command ?

15

rpm -qa |grep foreman

16

Also, are there any firewall rules perhaps blocking some connections?

17

I reinstalled the fireman and katello, it works now.
But I really want know why, :grinning:

Anyway,thanks a lot!