Pulp storage: Oops, we're sorry but something went wrong Pulp redis connection issue at https://capsule-server.eng.velocix.com/pulp/api/v3

Support
, ,
1

Problem:
Pulp storage: Oops, we’re sorry but something went wrong Pulp redis connection issue at https://capsule-server.eng.velocix.com/pulp/api/v3.
Facing above issue in after installing proxy server.Not able to sync the data from foreman server.
Expected outcome:
Porxy server sync successfully.
Foreman and Proxy versions:
Foreman 3.5 and katello 4.7
Distribution and version:
Rocky Linux release 8.10
Other relevant data:
There was no issues on proxy installation,

2024-10-17 11:19:13 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-10-17 11:19:17 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-10-17 11:19:17 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-10-17 11:19:22 [NOTICE] [configure] Starting system configuration.
2024-10-17 11:19:33 [NOTICE] [configure] 250 configuration steps out of 1112 steps complete.
2024-10-17 11:19:37 [NOTICE] [configure] 500 configuration steps out of 1117 steps complete.
2024-10-17 11:19:38 [NOTICE] [configure] 750 configuration steps out of 1120 steps complete.
2024-10-17 11:19:46 [NOTICE] [configure] 1000 configuration steps out of 1121 steps complete.
2024-10-17 11:19:55 [NOTICE] [configure] System configuration has finished.
  Success!
  * Foreman Proxy is running at https://capsule-server.eng.velocix.com:9090

  The full log is at /var/log/foreman-installer/foreman-proxy-content.log

image
image1121×537 47.3 KB

2

@evgeni Can you help on this issue and provide any solution?

3

No?

But I’d start by checking whether redis is running and if not fix that? :slight_smile:

4

Yes it is running,
On Foreman server:

[root@rhss-3 ~]# systemctl status redis
● redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis.service.d
           └─90-limits.conf
   Active: active (running) since Wed 2024-10-16 14:41:31 IST; 2 days ago
 Main PID: 1333 (redis-server)
   Status: "Ready to accept connections"
    Tasks: 5 (limit: 203804)
   Memory: 11.5M
   CGroup: /system.slice/redis.service
           └─1333 /usr/bin/redis-server 127.0.0.1:6379

Oct 16 14:41:31 rhss-3.eng.velocix.com systemd[1]: Starting Redis persistent key-value database...
Oct 16 14:41:31 rhss-3.eng.velocix.com systemd[1]: Started Redis persistent key-value database.

On proxy server:

[root@capsule-server ~]# systemctl status redis
● redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis.service.d
           └─90-limits.conf
   Active: active (running) since Wed 2024-10-16 20:11:18 IST; 1 day 20h ago
 Main PID: 1081 (redis-server)
   Status: "Ready to accept connections"
    Tasks: 5 (limit: 203832)
   Memory: 9.2M
   CGroup: /system.slice/redis.service
           └─1081 /usr/bin/redis-server 127.0.0.1:6379

Oct 16 20:11:18 capsule-server.eng.velocix.com systemd[1]: Starting Redis persistent key-value database...
Oct 16 20:11:18 capsule-server.eng.velocix.com systemd[1]: Started Redis persistent key-value database.
5

Issue still exists,redis service runs without issue but found this issue in /var/log/foreman/production.log file.Attaching the log details.
production.log (122.5 KB)

Able to find Completed 500 Internal Server Error in 270ms line in the logs.

6

Can you run curl https://capsule-server.eng.velocix.com/pulp/api/v3/status/ and show us the result?

The error you describe implies that there is no entry like

    "redis_connection": {
        "connected": true
    },

in that status output.

What’s the value of REDIS_URL in /etc/pulp/settings.py on that capsule? redis://localhost:6379/8? And CACHE_ENABLED?

7 
[root@capsule-server ~]# curl https://capsule-server.eng.velocix.com/pulp/api/v3/status/
{"versions":[{"component":"core","version":"3.21.17","package":"pulpcore"},{"component":"rpm","version":"3.18.14","package":"pulp-rpm"},{"component":"python","version":"3.7.2","package":"pulp-python"},{"component":"file","version":"1.11.1","package":"pulp-file"},{"component":"deb","version":"2.20.0","package":"pulp_deb"},{"component":"container","version":"2.14.3","package":"pulp-container"},{"component":"certguard","version":"1.5.5","package":"pulp-certguard"},{"component":"ansible","version":"0.15.0","package":"pulp-ansible"}],"online_workers":[{"pulp_href":"/pulp/api/v3/workers/4c03159c-0614-45d7-aea5-b72d498f61ab/","pulp_created":"2024-11-06T08:03:51.414323Z","name":"3345@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:16.843127Z","current_task":null},{"pulp_href":"/pulp/api/v3/workers/5b4a9b00-0861-4780-b1d4-b494c075aa2b/","pulp_created":"2024-11-06T08:03:52.446198Z","name":"3352@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:18.191635Z","current_task":null},{"pulp_href":"/pulp/api/v3/workers/037a2ae9-0377-4537-85c8-25fcac410f2e/","pulp_created":"2024-11-06T08:03:48.790400Z","name":"3298@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:14.659602Z","current_task":null},{"pulp_href":"/pulp/api/v3/workers/9ca9cadf-81f8-422e-a5f7-2fd990e47dad/","pulp_created":"2024-11-06T08:03:49.371948Z","name":"3316@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:14.825251Z","current_task":null},{"pulp_href":"/pulp/api/v3/workers/565cae1c-c636-40b4-8cf9-7a8d4682daef/","pulp_created":"2024-11-06T08:03:49.359018Z","name":"3307@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:14.888259Z","current_task":null},{"pulp_href":"/pulp/api/v3/workers/7c0e0265-5ad9-4b06-bc44-9508df423467/","pulp_created":"2024-11-06T08:03:50.043568Z","name":"3323@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:15.989955Z","current_task":null},{"pulp_href":"/pulp/api/v3/workers/0cc31a56-c1a6-4212-95bb-0b9a022407c0/","pulp_created":"2024-11-06T08:03:50.558358Z","name":"3331@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:16.460645Z","current_task":null},{"pulp_href":"/pulp/api/v3/workers/a6e0bc3a-d518-4fbc-b686-9ffac0792063/","pulp_created":"2024-11-06T08:03:50.718364Z","name":"3338@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:16.476589Z","current_task":null}],"online_content_apps":[{"name":"3527@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.193012Z"},{"name":"3528@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.200790Z"},{"name":"3517@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.215682Z"},{"name":"3540@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.453094Z"},{"name":"3526@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.619983Z"},{"name":"3533@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.559374Z"},{"name":"3519@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.605747Z"},{"name":"3538@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:21.642739Z"},{"name":"3530@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:14.735412Z"},{"name":"3543@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:14.808922Z"},{"name":"3522@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:14.875823Z"},{"name":"3537@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:14.960232Z"},{"name":"3546@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:15.056056Z"},{"name":"3542@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:15.411962Z"},{"name":"3548@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:15.424263Z"},{"name":"3534@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:15.389582Z"},{"name":"3545@capsule-server.eng.velocix.com","last_heartbeat":"2024-11-06T11:03:15.472791Z"}],"database_connection":{"connected":true},"redis_connection":{"connected":false},"storage":{"total":1092531261440,"used":11263459328,"free":1081267802112}}[root@capsule-server ~]#

In /etc/pulp/settings.py,

# File managed by Puppet module: pulpcore
################################################################################
# Not only will edits be overwritten later, there is also a strong
# possibility of breaking the system if changes are made here without making
# required corresponding changes elsewhere. Refer to the documentation used to
# install Pulpcore to determine the safe and persistent way to modify the
# configuration.
################################################################################

# Send anonymous usage data to https://analytics.pulpproject.org/ unless this is
# explicitly defined with value False. This data is used by the Pulp project
# to make informed, data-driven decisions about future feature development.
TELEMETRY = False

CONTENT_ORIGIN = "https://capsule-server.eng.velocix.com"
SECRET_KEY = "TeSgfTm3nK7QUeuKuQtM274ojFPfUnx9Hh7uVQfCskzM"
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'pulpcore',
        'USER': 'pulp',
        'PASSWORD': 'c7S2KbPYVBWmPkRn6sGF3TwGsBot9',
        'HOST': 'localhost',
        'PORT': '5432',
    },
}
REDIS_URL = "redis://localhost:6379/8"


MEDIA_ROOT = "/var/lib/pulp/media"
STATIC_ROOT = "/var/lib/pulp/assets"
STATIC_URL = "/pulp/assets/"
FILE_UPLOAD_TEMP_DIR = "/var/lib/pulp/tmp"
WORKING_DIRECTORY = "/var/lib/pulp/tmp"

REMOTE_USER_ENVIRON_NAME = 'HTTP_REMOTE_USER'
AUTHENTICATION_BACKENDS = ['pulpcore.app.authentication.PulpNoCreateRemoteUserBackend']

REST_FRAMEWORK__DEFAULT_AUTHENTICATION_CLASSES = (
    'rest_framework.authentication.SessionAuthentication',
    'pulpcore.app.authentication.PulpRemoteUserAuthentication'
)

ALLOWED_IMPORT_PATHS = ["/var/lib/pulp/sync_imports"]
ALLOWED_EXPORT_PATHS = []
ALLOWED_CONTENT_CHECKSUMS = ["sha1", "sha224", "sha256", "sha384", "sha512"]

# Derive HTTP/HTTPS via the X-Forwarded-Proto header set by Apache
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

CACHE_ENABLED = True
LOGGING = {
    "dynaconf_merge": True,
    "loggers": {
        '': {
            'handlers': ['console'],
            'level': 'INFO',
        },
        'pulpcore.deprecation': {
            'handlers': ['console'],
            'level': 'ERROR',
            'propagate': False,
        },
        'django_guid': {
            'handlers': ['console'],
            'level': 'WARNING',
            'propagate': False,
        },
    },
}

# ansible plugin settings
ANSIBLE_API_HOSTNAME = "https://capsule-server.eng.velocix.com"
ANSIBLE_CONTENT_HOSTNAME = "https://capsule-server.eng.velocix.com/pulp/content"

# container plugin settings
TOKEN_AUTH_DISABLED=True

# deb plugin settings
FORCE_IGNORE_MISSING_PACKAGE_INDICES = True
8

Right, that has:

    "redis_connection": {
        "connected": false
    },

Now, I wonder why…

(and you probably don’t want to share your passwords from the settings, even if the DB is local)

9

When you curl, do you see “Connection to Redis failed during status check!” in the logs (journalctl / /var/log/messages) on the proxy?

10

yes I can see the error in /var/log/messages,

Nov  6 16:56:08 capsule-server pulpcore-api[3447]: pulp [c6dc3e0e51d74c5ea9ab1788f5681575]: pulpcore.app.views.status:ERROR: Connection to Redis failed during status check!
Nov  6 16:56:08 capsule-server systemd[1]: Starting SSSD Kerberos Cache Manager...
Nov  6 16:56:08 capsule-server systemd[1]: Started SSSD Kerberos Cache Manager.
Nov  6 16:56:08 capsule-server sssd_kcm[4429]: Starting up
Nov  6 16:56:08 capsule-server pulpcore-api[3447]: pulp [c6dc3e0e51d74c5ea9ab1788f5681575]:  - - [06/Nov/2024:11:26:08 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4186 "-" "curl/7.61.1"
Nov  6 16:56:16 capsule-server pulpcore-api[3447]: pulp [8903bb6635a142859f8ddf73475c9972]: pulpcore.app.views.status:ERROR: Connection to Redis failed during status check!
Nov  6 16:56:16 capsule-server pulpcore-api[3447]: pulp [8903bb6635a142859f8ddf73475c9972]:  - - [06/Nov/2024:11:26:16 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4186 "-" "curl/7.61.1"

How to resolve this?

11

If I’d know…

Can you open a Redis shell (redis-cli -h localhost -p 6379 -n 8) and type ping there? does it PONG like this:

# redis-cli -h localhost -p 6379 -n 8
localhost:6379[8]> ping
PONG
12 
[root@capsule-server ~]# redis-cli -h localhost -p 6379 -n 8
localhost:6379[8]> ping
PONG
13

Can you run the following script with python3.11 (which is what Pulpcore uses):

from redis import Redis

REDIS_URL = "redis://localhost:6379/8"

r = Redis.from_url(REDIS_URL)

print(r.ping())
14 
[root@capsule-server ~]# python3.11 test_redis.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 357, in connect
    sock = self.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/retry.py", line 62, in call_with_retry
    return do()
           ^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 358, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
            ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 730, in _connect
    raise err
  File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 718, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/test_redis.py", line 7, in <module>
    print(r.ping())
          ^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 1212, in ping
    return self.execute_command("PING", **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 559, in execute_command
    return self._execute_command(*args, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 565, in _execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1422, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 363, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
15

and if you replace localhost in there with 127.0.0.1 it works?

16

yes its working

[root@capsule-server ~]# python3.11 test_redis.py
True
17

Great!

Now, questions! :slight_smile:

Why does the above stacktrace include paths in /usr/local? Did you ever install something using pip on that machine?

18

Yeah,actually I was trying to troubleshoot using the above commands you sent.
Between I found the cause of this issue mistake on my side,
In /etc/hosts file,

127.0.0.1 capsule.eng.velocix.com

Changed the above line to,

127.0.0.1 localhost localhost.localdomain localhost6 localhost6.localdomain6
[Server IP] capsule-server.eng.velocix.com

It is working now