Problem:
I tried to migrate pulp2 to pulp3, but it seems that my pulp3 service is unavailable.
Hammer ping output:
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 27ms
candlepin_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
candlepin_auth:
Status: ok
Server Response: Duration: 25ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
pulp:
Status: ok
Server Response: Duration: 68ms
pulp_auth:
Status: ok
Server Response: Duration: 16ms
pulp3:
Status: FAIL
Server Response: Message: 503 Service Unavailable
foreman_tasks:
Status: ok
Server Response: Duration: 4ms
I have installed the latest updates as well. Any idea what would be the issue ?
Foreman and Katello versions:
Foreman 2.3.3
Katello 3.18
Is there any further debugging I can do on it. This is causing us some issues and I will have to revert to 2.2 and katello 3.17 as katello 3.18 has some serious issues for us with pulp3 not even being recognised as starting but all processes seem ok
Hi!
I will share my output as well, maybe i have the same issue.
● pulpcore-api.service - Pulp WSGI Server
Loaded: loaded (/etc/systemd/system/pulpcore-api.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-03-02 10:40:10 EET; 1 weeks 1 days ago
Main PID: 3841 (gunicorn)
Tasks: 2
CGroup: /system.slice/pulpcore-api.service
├─3841 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --bind 127.0.0.1:24817 --access-logfile -
└─3857 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --bind 127.0.0.1:24817 --access-logfile -
Mar 02 10:40:10 systemd[1]: Stopped Pulp WSGI Server.
Mar 02 10:40:10 systemd[1]: Started Pulp WSGI Server.
Mar 02 10:40:11 pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3841] [INFO] Starting gunicorn 20.0.4
Mar 02 10:40:11 pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3841] [INFO] Listening at: http://127.0.0.1:24817 (3841)
Mar 02 10:40:11 pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3841] [INFO] Using worker: sync
Mar 02 10:40:11 pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3857] [INFO] Booting worker with pid: 3857
● pulpcore-content.service - Pulp Content App
Loaded: loaded (/etc/systemd/system/pulpcore-content.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-03-02 10:40:11 EET; 1 weeks 1 days ago
Main PID: 3859 (gunicorn)
Tasks: 3
CGroup: /system.slice/pulpcore-content.service
├─ 3859 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -
├─14672 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -
└─14686 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -
Mar 02 14:38:32 pulpcore-content[3859]: [2021-03-02 14:38:30 +0200] [14635] [INFO] Booting worker with pid: 14635
Mar 02 14:38:34 pulpcore-content[3859]: [2021-03-02 14:38:30 +0200] [14633] [INFO] Booting worker with pid: 14633
Mar 02 14:38:53 pulpcore-content[3859]: [2021-03-02 14:38:53 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14633)
Mar 02 14:38:58 pulpcore-content[3859]: [2021-03-02 14:38:58 +0200] [14650] [INFO] Booting worker with pid: 14650
Mar 02 14:38:59 pulpcore-content[3859]: [2021-03-02 14:38:59 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14635)
Mar 02 14:39:05 pulpcore-content[3859]: [2021-03-02 14:39:03 +0200] [14653] [INFO] Booting worker with pid: 14653
Mar 02 14:39:25 pulpcore-content[3859]: [2021-03-02 14:39:25 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14650)
Mar 02 14:39:26 pulpcore-content[3859]: [2021-03-02 14:39:26 +0200] [14672] [INFO] Booting worker with pid: 14672
Mar 02 14:39:32 pulpcore-content[3859]: [2021-03-02 14:39:32 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14653)
Mar 02 14:39:33 pulpcore-content[3859]: [2021-03-02 14:39:33 +0200] [14686] [INFO] Booting worker with pid: 14686
I also noticed, that if i try to open repo “published at” link (//servername/pulp/repos/…), i will get white webpage with error message - ERR_BAD_SSL_CLIENT_AUTH_CERT
Thanks. So the services look happy, but Apache is reporting a 503 back to Katello when it tries to talk to the Pulp 3 services. Does anything in the Apache logs stand out?
After upgrading Katello from 3.16 to 3.18 the following error messages have occurred in log files:
foreman-ssl_error_ssl.log
[proxy:error] [pid 1630] (2)No such file or directory: AH02454: HTTP: attempt to connect to Unix domain socket /run/pulpcore-api.sock failed
[proxy:error] [pid 1630] AH00959: ap_proxy_connect_backend disabling worker for for 60s
[proxy_http:error] [pid 1630] [client] AH01114: HTTP: failed to make connection to backend: httpd-UDS
[ssl:warn] [pid 29076] [client] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’
I have 2 systems a dev and prod. The dev updated and works fine. The prod does not.
The dev system has
/etc/systemd/system/pulpcore-api.socket
/etc/systemd/system/sockets.target.wants/pulpcore-api.socket
The prod system doesn’t
OK it makes sense now looks like the puppet update is broken
/etc/httpd/conf.d/05.foreman.conf uses a reverse PROXY based on sockets but the pulpcore-api service is a network service