Pulp3 - 503 Service Unavailable

Support
#1

Problem:
I tried to migrate pulp2 to pulp3, but it seems that my pulp3 service is unavailable.
Hammer ping output:
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 27ms
candlepin_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
candlepin_auth:
Status: ok
Server Response: Duration: 25ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
pulp:
Status: ok
Server Response: Duration: 68ms
pulp_auth:
Status: ok
Server Response: Duration: 16ms
pulp3:
Status: FAIL
Server Response: Message: 503 Service Unavailable
foreman_tasks:
Status: ok
Server Response: Duration: 4ms

I have installed the latest updates as well. Any idea what would be the issue ?

Foreman and Katello versions:
Foreman 2.3.3
Katello 3.18

Distribution and version:
Centos 7.9

Br,
Kimmo

#2

Can you check the systemctl status of “pulpcore*”?

#3

The following services are active and in running status:
*pulpcore-content
*pulpcore-api
*pulpcore-worker@1/@2/@3/@4
*pulpcore-resource-manager

#4

I also found this:
image
maybe it will help somehow find a solution to my concern.

#5

Can you provide the output of

curl https://HOSTNAME/pulp/api/v3/status/
#6
503 Service Unavailable

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

#8

It seems that now my pulpcore does not support any content type…
image

#9

I have the same issue

#10

Is there any further debugging I can do on it. This is causing us some issues and I will have to revert to 2.2 and katello 3.17 as katello 3.18 has some serious issues for us with pulp3 not even being recognised as starting but all processes seem ok

#11

curl --cert /etc/pki/katello/certs/pulp-ci/katello/private/pulp-client.key https://hostname/api/v2/ping | /usr/libexec/platform-python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 583 0 583 0 0 1373 0 --:–:-- --:–:-- --:–:-- 1381
{
“results”: {
“foreman”: {
“database”: {
“active”: true,
“duration_ms”: “0”
}
},
“katello”: {
“services”: {
“candlepin”: {
“duration_ms”: “27”,
“status”: “ok”
},
“candlepin_auth”: {
“duration_ms”: “40”,
“status”: “ok”
},
“candlepin_events”: {
“duration_ms”: “0”,
“message”: “0 Processed, 0 Failed”,
“status”: “ok”
},
“foreman_tasks”: {
“duration_ms”: “5”,
“status”: “ok”
},
“katello_events”: {
“duration_ms”: “0”,
“message”: “0 Processed, 0 Failed”,
“status”: “ok”
},
“pulp”: {
“duration_ms”: “77”,
“status”: “ok”
},
“pulp3”: {
“message”: “503 Service Unavailable”,
“status”: “FAIL”
},
“pulp_auth”: {
“duration_ms”: “47”,
“status”: “ok”
}
},
“status”: “FAIL”
}
}
}

#12

What is systemd showing for:

systemctl status pulpcore-api
systemctl status pulpcore-content
#13

Hi!
I will share my output as well, maybe i have the same issue.

● pulpcore-api.service - Pulp WSGI Server
   Loaded: loaded (/etc/systemd/system/pulpcore-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-03-02 10:40:10 EET; 1 weeks 1 days ago
 Main PID: 3841 (gunicorn)
    Tasks: 2
   CGroup: /system.slice/pulpcore-api.service
           ├─3841 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --bind 127.0.0.1:24817 --access-logfile -
           └─3857 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --bind 127.0.0.1:24817 --access-logfile -

Mar 02 10:40:10  systemd[1]: Stopped Pulp WSGI Server.
Mar 02 10:40:10  systemd[1]: Started Pulp WSGI Server.
Mar 02 10:40:11  pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3841] [INFO] Starting gunicorn 20.0.4
Mar 02 10:40:11  pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3841] [INFO] Listening at: http://127.0.0.1:24817 (3841)
Mar 02 10:40:11  pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3841] [INFO] Using worker: sync
Mar 02 10:40:11  pulpcore-api[3841]: [2021-03-02 10:40:11 +0200] [3857] [INFO] Booting worker with pid: 3857

● pulpcore-content.service - Pulp Content App
   Loaded: loaded (/etc/systemd/system/pulpcore-content.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-03-02 10:40:11 EET; 1 weeks 1 days ago
 Main PID: 3859 (gunicorn)
    Tasks: 3
   CGroup: /system.slice/pulpcore-content.service
           ├─ 3859 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -
           ├─14672 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -
           └─14686 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -

Mar 02 14:38:32  pulpcore-content[3859]: [2021-03-02 14:38:30 +0200] [14635] [INFO] Booting worker with pid: 14635
Mar 02 14:38:34  pulpcore-content[3859]: [2021-03-02 14:38:30 +0200] [14633] [INFO] Booting worker with pid: 14633
Mar 02 14:38:53  pulpcore-content[3859]: [2021-03-02 14:38:53 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14633)
Mar 02 14:38:58  pulpcore-content[3859]: [2021-03-02 14:38:58 +0200] [14650] [INFO] Booting worker with pid: 14650
Mar 02 14:38:59  pulpcore-content[3859]: [2021-03-02 14:38:59 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14635)
Mar 02 14:39:05  pulpcore-content[3859]: [2021-03-02 14:39:03 +0200] [14653] [INFO] Booting worker with pid: 14653
Mar 02 14:39:25  pulpcore-content[3859]: [2021-03-02 14:39:25 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14650)
Mar 02 14:39:26  pulpcore-content[3859]: [2021-03-02 14:39:26 +0200] [14672] [INFO] Booting worker with pid: 14672
Mar 02 14:39:32  pulpcore-content[3859]: [2021-03-02 14:39:32 +0200] [3859] [CRITICAL] WORKER TIMEOUT (pid:14653)
Mar 02 14:39:33  pulpcore-content[3859]: [2021-03-02 14:39:33 +0200] [14686] [INFO] Booting worker with pid: 14686

I also noticed, that if i try to open repo “published at” link (//servername/pulp/repos/…), i will get white webpage with error message - ERR_BAD_SSL_CLIENT_AUTH_CERT

#14

Thanks. So the services look happy, but Apache is reporting a 503 back to Katello when it tries to talk to the Pulp 3 services. Does anything in the Apache logs stand out?

You can also check the health of Pulp 3 locally:

curl http://127.0.0.1:24817/pulp/api/v3/status/ | json_reformat
#15

The health of pulp3 seems fine:

"versions": [
    {
        "component": "pulpcore",
        "version": "3.7.3"
    },
    {
        "component": "pulp_2to3_migration",
        "version": "0.7.0"
    },
    {
        "component": "pulp_rpm",
        "version": "3.7.0"
    },
    {
        "component": "pulp_file",
        "version": "1.3.0"
    },
    {
        "component": "pulp_deb",
        "version": "2.7.0"
    },
    {
        "component": "pulp_container",
        "version": "2.1.0"
    },
    {
        "component": "pulp_certguard",
        "version": "1.0.3"
    }
],
"online_workers": [
    {
        "pulp_created": "2021-03-02T08:40:15.970005Z",
        "pulp_href": "/pulp/api/v3/workers/1321f66e-670f-412e-8b4a-6d7c8f00e4a4/",
        "name": "3845@",
        "last_heartbeat": "2021-03-10T14:52:48.999309Z"
    },
    {
        "pulp_created": "2021-03-02T08:40:16.193528Z",
        "pulp_href": "/pulp/api/v3/workers/89793b6e-3cb6-4fd4-a15c-65f49ae566d1/",
        "name": "3852@",
        "last_heartbeat": "2021-03-10T14:52:49.599504Z"
    },
    {
        "pulp_created": "2021-03-02T08:40:15.583648Z",
        "pulp_href": "/pulp/api/v3/workers/dba322cf-3541-4620-a6a4-e831c347f7f9/",
        "name": "3843@",
        "last_heartbeat": "2021-03-10T14:52:53.106479Z"
    },
    {
        "pulp_created": "2021-03-02T08:40:16.114760Z",
        "pulp_href": "/pulp/api/v3/workers/bb71e590-0f2d-442a-8f4d-f6e1866aa1c6/",
        "name": "3848@",
        "last_heartbeat": "2021-03-10T14:52:57.014572Z"
    },
    {
        "pulp_created": "2020-07-20T08:56:34.166749Z",
        "pulp_href": "/pulp/api/v3/workers/5dc0e420-eb85-4f3d-a252-62c2fcd0ce8b/",
        "name": "resource-manager",
        "last_heartbeat": "2021-03-10T14:52:59.021324Z"
    }
],
"online_content_apps": [
    {
        "name": "14686@",
        "last_heartbeat": "2021-03-10T14:53:01.321640Z"
    },
    {
        "name": "14672@",
        "last_heartbeat": "2021-03-10T14:52:59.461028Z"
    }
],
"database_connection": {
    "connected": true
},
"redis_connection": {
    "connected": true
},
"storage": {
    "total": 321965260800,
    "used": 158237736960,
    "free": 163727523840
}
}
#16

After upgrading Katello from 3.16 to 3.18 the following error messages have occurred in log files:

foreman-ssl_error_ssl.log

[proxy:error] [pid 1630] (2)No such file or directory: AH02454: HTTP: attempt to connect to Unix domain socket /run/pulpcore-api.sock failed
[proxy:error] [pid 1630] AH00959: ap_proxy_connect_backend disabling worker for for 60s
[proxy_http:error] [pid 1630] [client] AH01114: HTTP: failed to make connection to backend: httpd-UDS
[ssl:warn] [pid 29076] [client] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’

error_log

[:error] [pid 9558] [INFO] crane.data: loading metadata from /var/lib/pulp/published/docker/v2/app
[:error] [pid 9558] [INFO] crane.data: finished loading metadata

I don’t know if this information is helpful.

#17

Do you see any httpd or pulpcore related denials in SELinux?

#18

I changed my selinux to permissive several days ago

#19

I have 2 systems a dev and prod. The dev updated and works fine. The prod does not.
The dev system has
/etc/systemd/system/pulpcore-api.socket
/etc/systemd/system/sockets.target.wants/pulpcore-api.socket
The prod system doesn’t

#20

OK it makes sense now looks like the puppet update is broken
/etc/httpd/conf.d/05.foreman.conf uses a reverse PROXY based on sockets but the pulpcore-api service is a network service

#21

I’ve modified the systemd pulpcore service files to use sockets and it all now works