New Foreman Install Fail: no smart proxy created Foreman 2.5/Katello 4.1

Support
21

O.K. The issue here seems to be that the access to https://…:9090/v2/features takes two minutes.

Can you check in /var/log/foreman-proxy/proxy.log That should contain the access log and the time it took, e.g.

2021-08-02T13:32:50 a0cdc839 [I] Started GET /v2/features 
2021-08-02T13:32:50 a0cdc839 [I] Finished GET /v2/features with 200 (27.45 ms)

Are there any other errors or messages?

I guess one of the katello developers must take a look why the request takes 2 minutes for you. How many CPUs does your VM have?

22

VM has 8 vCPUs

foreman-proxy/proxy.log:

2021-08-02T10:56:07  [I] WEBrick::HTTPServer#start: pid=58916 port=9090
2021-08-02T10:56:07  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2021-08-02T10:56:15  [W] Could not open DB for dynflow at '', will keep data in memory. Restart will drop all dynflow data.
2021-08-02T10:56:15  [I] Execution plan cleaner removing 0 execution plans.
2021-08-02T10:56:15 e77bbf02 [I] Started GET /v2/features
2021-08-02T10:57:15 e77bbf02 [I] Started GET /v2/features
2021-08-02T10:58:15 e77bbf02 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T10:58:15 e77bbf02 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T10:58:15 e77bbf02 [I] Finished GET /v2/features with 200 (120151.18 ms)
2021-08-02T10:58:15 e77bbf02 [I] Finished GET /v2/features with 200 (120151.18 ms)
2021-08-02T10:59:15 e77bbf02 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T10:59:15 e77bbf02 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T10:59:15 e77bbf02 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T10:59:15 e77bbf02 [I] Finished GET /v2/features with 200 (120146.34 ms)
2021-08-02T10:59:15 e77bbf02 [I] Finished GET /v2/features with 200 (120146.34 ms)
2021-08-02T10:59:15 e77bbf02 [I] Finished GET /v2/features with 200 (120146.34 ms)
2021-08-02T11:04:25 21f6e3ff [I] Started GET /features
2021-08-02T11:04:25 21f6e3ff [I] Started GET /features
2021-08-02T11:04:25 21f6e3ff [I] Started GET /features
2021-08-02T11:04:25 21f6e3ff [I] Finished GET /features with 200 (1.55 ms)
2021-08-02T11:04:25 21f6e3ff [I] Finished GET /features with 200 (1.55 ms)
2021-08-02T11:04:25 21f6e3ff [I] Finished GET /features with 200 (1.55 ms)
2021-08-02T11:05:36 2da6f283 [I] Started GET /features
2021-08-02T11:05:36 2da6f283 [I] Started GET /features
2021-08-02T11:05:36 2da6f283 [I] Started GET /features
2021-08-02T11:05:36 2da6f283 [I] Finished GET /features with 200 (0.58 ms)
2021-08-02T11:05:36 2da6f283 [I] Finished GET /features with 200 (0.58 ms)
2021-08-02T11:05:36 2da6f283 [I] Finished GET /features with 200 (0.58 ms)
2021-08-02T11:06:22 26a403c9 [I] Started GET /v2/features
2021-08-02T11:06:22 26a403c9 [I] Started GET /v2/features
2021-08-02T11:06:22 26a403c9 [I] Started GET /v2/features
2021-08-02T11:08:22 26a403c9 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:08:22 26a403c9 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:08:22 26a403c9 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:08:22 26a403c9 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:08:22 26a403c9 [I] Finished GET /v2/features with 200 (120138.52 ms)
2021-08-02T11:08:22 26a403c9 [I] Finished GET /v2/features with 200 (120138.52 ms)
2021-08-02T11:08:22 26a403c9 [I] Finished GET /v2/features with 200 (120138.52 ms)
2021-08-02T11:08:22 26a403c9 [I] Finished GET /v2/features with 200 (120138.52 ms)
2021-08-02T11:08:46 713f0375 [I] Started GET /v2/features
2021-08-02T11:08:46 713f0375 [I] Started GET /v2/features
2021-08-02T11:08:46 713f0375 [I] Started GET /v2/features
2021-08-02T11:08:46 713f0375 [I] Started GET /v2/features
2021-08-02T11:10:46 713f0375 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:10:46 713f0375 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:10:46 713f0375 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:10:46 713f0375 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:10:46 713f0375 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:10:46 713f0375 [I] Finished GET /v2/features with 200 (120109.62 ms)
2021-08-02T11:10:46 713f0375 [I] Finished GET /v2/features with 200 (120109.62 ms)
2021-08-02T11:10:46 713f0375 [I] Finished GET /v2/features with 200 (120109.62 ms)
2021-08-02T11:10:46 713f0375 [I] Finished GET /v2/features with 200 (120109.62 ms)
2021-08-02T11:10:46 713f0375 [I] Finished GET /v2/features with 200 (120109.62 ms)
2021-08-02T11:26:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:26:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:26:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:26:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:26:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:27:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:27:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:27:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:27:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:27:36 71832f75 [I] Started GET /v2/features
2021-08-02T11:28:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:28:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:28:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:28:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:28:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:28:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:28:36 71832f75 [I] Finished GET /v2/features with 200 (120135.35 ms)
2021-08-02T11:28:36 71832f75 [I] Finished GET /v2/features with 200 (120135.35 ms)
2021-08-02T11:28:36 71832f75 [I] Finished GET /v2/features with 200 (120135.35 ms)
2021-08-02T11:28:36 71832f75 [I] Finished GET /v2/features with 200 (120135.35 ms)
2021-08-02T11:28:36 71832f75 [I] Finished GET /v2/features with 200 (120135.35 ms)
2021-08-02T11:28:36 71832f75 [I] Finished GET /v2/features with 200 (120135.35 ms)
2021-08-02T11:29:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:29:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:29:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:29:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:29:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:29:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:29:36 71832f75 [E] Could not fetch capabilities: Net::ReadTimeout with #<TCPSocket:(closed)>
2021-08-02T11:29:36 71832f75 [I] Finished GET /v2/features with 200 (120110.38 ms)
2021-08-02T11:29:36 71832f75 [I] Finished GET /v2/features with 200 (120110.38 ms)
2021-08-02T11:29:36 71832f75 [I] Finished GET /v2/features with 200 (120110.38 ms)
2021-08-02T11:29:36 71832f75 [I] Finished GET /v2/features with 200 (120110.38 ms)
2021-08-02T11:29:36 71832f75 [I] Finished GET /v2/features with 200 (120110.38 ms)
2021-08-02T11:29:36 71832f75 [I] Finished GET /v2/features with 200 (120110.38 ms)
2021-08-02T11:29:36 71832f75 [I] Finished GET /v2/features with 200 (120110.38 ms)

I can see gunicorn pulpcore content process consuming 50% total CPU load intermittently, I can throw more vCPUs at it and take a look at pulpcore service if you like

UPDATE: pulp-core content is in failed systemd service state:

Aug 02 11:42:22 xv-sat001.uk2.scp.bskyb.com systemd[1]: Starting Pulp Content App...
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78183] [INFO] Starting gunicorn 20.1.0
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78183] [INFO] Listening at: unix:/run/pulpcore-content.sock (78183)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78183] [INFO] Using worker: aiohttp.GunicornWebWorker
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com systemd[1]: Started Pulp Content App.
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78188] [INFO] Booting worker with pid: 78188
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78191] [INFO] Booting worker with pid: 78191
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78192] [INFO] Booting worker with pid: 78192
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78194] [INFO] Booting worker with pid: 78194
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78196] [INFO] Booting worker with pid: 78196
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78197] [INFO] Booting worker with pid: 78197
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78188] [ERROR] Exception in worker process
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: Traceback (most recent call last):
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: worker.init_process()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/site-packages/aiohttp/worker.py", line 51, in init_process
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: super().init_process()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 134, in init_process
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.load_wsgi()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.wsgi = self.app.wsgi()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.callable = self.load()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return self.load_wsgiapp()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return util.import_app(self.app_uri)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/util.py", line 359, in import_app
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(module)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 18, in <module>
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: django.setup()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._setup(name)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._wrapped = Settings(settings_module)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(self.SETTINGS_MODULE)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 83, in <module>
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: plugin_app_config = entry_point.load()
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2449, in load
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.require(*args, **kwargs)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2472, in require
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: items = working_set.resolve(reqs, env, installer, extras=self.extras)
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 777, in resolve
Aug 02 11:42:23 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: raise VersionConflict(dist, req).with_context(dependent_req)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: pkg_resources.ContextualVersionConflict: (chardet 4.0.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('chardet<4.0,>=2.0'), {'aiohttp'})
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:23 +0000] [78188] [INFO] Worker exiting (pid: 78188)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78191] [ERROR] Exception in worker process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: Traceback (most recent call last):
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: worker.init_process()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/site-packages/aiohttp/worker.py", line 51, in init_process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: super().init_process()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 134, in init_process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.load_wsgi()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.wsgi = self.app.wsgi()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.callable = self.load()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return self.load_wsgiapp()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return util.import_app(self.app_uri)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/util.py", line 359, in import_app
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(module)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 18, in <module>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: django.setup()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._setup(name)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._wrapped = Settings(settings_module)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(self.SETTINGS_MODULE)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 83, in <module>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: plugin_app_config = entry_point.load()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2449, in load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.require(*args, **kwargs)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2472, in require
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: items = working_set.resolve(reqs, env, installer, extras=self.extras)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 777, in resolve
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: raise VersionConflict(dist, req).with_context(dependent_req)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: pkg_resources.ContextualVersionConflict: (chardet 4.0.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('chardet<4.0,>=2.0'), {'aiohttp'})
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78191] [INFO] Worker exiting (pid: 78191)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78198] [INFO] Booting worker with pid: 78198
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78192] [ERROR] Exception in worker process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: Traceback (most recent call last):
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: worker.init_process()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/site-packages/aiohttp/worker.py", line 51, in init_process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: super().init_process()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 134, in init_process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.load_wsgi()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.wsgi = self.app.wsgi()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.callable = self.load()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return self.load_wsgiapp()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return util.import_app(self.app_uri)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/util.py", line 359, in import_app
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(module)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 18, in <module>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: django.setup()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._setup(name)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._wrapped = Settings(settings_module)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(self.SETTINGS_MODULE)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 83, in <module>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: plugin_app_config = entry_point.load()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2449, in load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.require(*args, **kwargs)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2472, in require
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: items = working_set.resolve(reqs, env, installer, extras=self.extras)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 777, in resolve
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: raise VersionConflict(dist, req).with_context(dependent_req)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: pkg_resources.ContextualVersionConflict: (chardet 4.0.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('chardet<4.0,>=2.0'), {'aiohttp'})
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78192] [INFO] Worker exiting (pid: 78192)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78183] [WARNING] Worker with pid 78191 was terminated due to signal 15
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: Traceback (most recent call last):
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 202, in run
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.manage_workers()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.spawn_workers()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 623, in spawn_workers
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: time.sleep(0.1 * random.random())
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.reap_workers()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: During handling of the above exception, another exception occurred:
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: Traceback (most recent call last):
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/bin/gunicorn", line 11, in <module>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: load_entry_point('gunicorn==20.1.0', 'console_scripts', 'gunicorn')()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 231, in run
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: super().run()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: Arbiter(self).run()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 229, in run
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.halt(reason=inst.reason, exit_status=inst.exit_status)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 342, in halt
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.stop()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 393, in stop
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: time.sleep(0.1)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.reap_workers()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78194] [ERROR] Exception in worker process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: Traceback (most recent call last):
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: worker.init_process()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/site-packages/aiohttp/worker.py", line 51, in init_process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: super().init_process()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 134, in init_process
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.load_wsgi()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.wsgi = self.app.wsgi()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.callable = self.load()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return self.load_wsgiapp()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return util.import_app(self.app_uri)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/gunicorn/util.py", line 359, in import_app
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(module)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/content/__init__.py", line 18, in <module>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: django.setup()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._setup(name)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self._wrapped = Settings(settings_module)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: mod = importlib.import_module(self.SETTINGS_MODULE)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: return _bootstrap._gcd_import(name[level:], package, level)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 83, in <module>
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: plugin_app_config = entry_point.load()
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2449, in load
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: self.require(*args, **kwargs)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2472, in require
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: items = working_set.resolve(reqs, env, installer, extras=self.extras)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 777, in resolve
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: raise VersionConflict(dist, req).with_context(dependent_req)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: pkg_resources.ContextualVersionConflict: (chardet 4.0.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('chardet<4.0,>=2.0'), {'aiohttp'})
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com pulpcore-content[78183]: [2021-08-02 11:42:24 +0000] [78194] [INFO] Worker exiting (pid: 78194)
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com systemd[1]: pulpcore-content.service: main process exited, code=exited, status=1/FAILURE
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com systemd[1]: Unit pulpcore-content.service entered failed state.
Aug 02 11:42:24 xv-sat001.uk2.scp.bskyb.com systemd[1]: pulpcore-content.service failed.
Aug 02 11:42:27 xv-sat001.uk2.scp.bskyb.com systemd[1]: pulpcore-content.service holdoff time over, scheduling restart
23

I’ve seen this in the past (as above pulpcore-content service failing due to py package version conflicts) with chardet or other py packages being in conflict, I’ve resolved this manually in the past by downgrading to the last supported version, obviously this isn’t ideal and isn’t usually an issue on first install as we have here

24

To clarify, downgrading the conflicting python package is what I’ve done in the past to fix pulpcore-content service failures (which I haven’t had to do after the initial install before)
Now I have an account I plan to more actively raise these issues (we’ve seen some others) in future to try to help the project :+1:

25

I guess, that’s something for the katello developers.

I don’t quite understand where you have got the chardet version 4.0.0 from. Did you install that with pip?

The only version I can find in the repositories is 3.0.4-2.el7 from pulpcore and that’s the version which is installed on my foreman server.

26

Nothing manually installed with pip, let me find the yum transaction for this package to see where it is pulled in from

27

If I understand the error message correctly, it must be installed in /usr/local/lib/python3.6/site-packages

On my foreman server /usr/local/lib/python3.6 doesn’t even exist.

28

Thanks again @gvde

Leave it with me, I can see from the yum transaction that chardet 3.0.4 should be installed:

$ sudo yum history info python3-pulpcore
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Transaction ID : 38
Begin time     : Mon Aug  2 10:37:38 2021
Begin rpmdb    : 793:054a32af9e191f1c52e1fa3fe2616aa22626380c
End time       :            10:37:54 2021 (16 seconds)
End rpmdb      : 858:4b20a23991fced7dedffeca73d14028fa4b68c17
User           : omitted
Return-Code    : Success
Command Line   : -d 0 -e 0 -y install python3-pulpcore
Transaction performed with:
    Installed     rpm-4.11.3-45.el7.x86_64                         @anaconda
    Installed     subscription-manager-1.24.48-1.el7.centos.x86_64 @updates
    Installed     yum-3.4.3-168.el7.centos.noarch                  @anaconda
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64          @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch  @anaconda
Packages Altered:
    Dep-Install python3-aiodns-3.0.0-1.el7.noarch                          @pulpcore
    Dep-Install python3-aiofiles-0.7.0-1.el7.noarch                        @pulpcore
    Dep-Install python3-aiohttp-3.7.4-1.el7.x86_64                         @pulpcore
    Dep-Install python3-aioredis-2.0.0-0.1.b1.el7.noarch                   @pulpcore
    Dep-Install python3-async-timeout-3.0.1-2.el7.noarch                   @pulpcore
    Dep-Install python3-asyncio-throttle-1.0.2-1.el7.noarch                @pulpcore
    Dep-Install python3-attrs-21.2.0-1.el7.noarch                          @pulpcore
    Dep-Install python3-backoff-1.10.0-3.el7.noarch                        @pulpcore
    Dep-Install python3-cffi-1.14.5-1.el7.x86_64                           @pulpcore
    Dep-Install python3-chardet-3.0.4-2.el7.noarch                         @pulpcore
    Dep-Install python3-click-7.1.2-1.el7.noarch                           @pulpcore
    Dep-Install python3-defusedxml-0.7.1-1.el7.noarch                      @pulpcore
    Dep-Install python3-diff-match-patch-20200713-1.el7.noarch             @pulpcore
    Dep-Install python3-django-2.2.24-1.el7.noarch                         @pulpcore
    Dep-Install python3-django-currentuser-0.5.3-1.el7.noarch              @pulpcore
    Dep-Install python3-django-filter-2.4.0-1.el7.noarch                   @pulpcore
    Dep-Install python3-django-guardian-2.4.0-1.el7.noarch                 @pulpcore
    Dep-Install python3-django-guid-2.2.1-1.el7.noarch                     @pulpcore
    Dep-Install python3-django-import-export-2.5.0-1.el7.noarch            @pulpcore
    Dep-Install python3-django-lifecycle-0.9.1-1.el7.noarch                @pulpcore
    Dep-Install python3-django-prometheus-2.1.0-1.el7.noarch               @pulpcore
    Dep-Install python3-djangorestframework-3.12.4-1.el7.noarch            @pulpcore
    Dep-Install python3-djangorestframework-queryfields-1.0.0-3.el7.noarch @pulpcore
    Dep-Install python3-drf-access-policy-0.9.0-1.el7.noarch               @pulpcore
    Dep-Install python3-drf-nested-routers-0.93.3-1.el7.noarch             @pulpcore
    Dep-Install python3-drf-spectacular-0.17.2-1.el7.noarch                @pulpcore
    Dep-Install python3-dynaconf-3.1.4-1.el7.noarch                        @pulpcore
    Dep-Install python3-et-xmlfile-1.0.1-1.el7.noarch                      @pulpcore
    Dep-Install python3-gnupg-0.4.7-1.el7.noarch                           @pulpcore
    Dep-Install python3-gunicorn-20.1.0-1.el7.noarch                       @pulpcore
    Dep-Install python3-idna-2.10-1.el7.noarch                             @pulpcore
    Dep-Install python3-idna-ssl-1.1.0-3.el7.noarch                        @pulpcore
    Dep-Install python3-importlib-metadata-1.7.0-1.el7.noarch              @pulpcore
    Dep-Install python3-inflection-0.5.1-1.el7.noarch                      @pulpcore
    Dep-Install python3-jinja2-2.11.3-1.el7.noarch                         @pulpcore
    Dep-Install python3-jsonschema-3.2.0-4.el7.noarch                      @pulpcore
    Dep-Install python3-markuppy-1.14-1.el7.noarch                         @pulpcore
    Dep-Install python3-markupsafe-1.1.1-2.el7.x86_64                      @pulpcore
    Dep-Install python3-multidict-5.1.0-1.el7.x86_64                       @pulpcore
    Dep-Install python3-odfpy-1.4.1-2.el7.noarch                           @pulpcore
    Dep-Install python3-openpyxl-3.0.7-1.el7.noarch                        @pulpcore
    Dep-Install python3-prometheus-client-0.8.0-1.el7.noarch               @pulpcore
    Dep-Install python3-psycopg2-2.8.6-1.el7.x86_64                        @pulpcore
    Install     python3-pulpcore-3.14.3-1.el7.noarch                       @pulpcore
    Dep-Install python3-pycares-4.0.0-1.el7.x86_64                         @pulpcore
    Dep-Install python3-pycparser-2.20-1.el7.noarch                        @pulpcore
    Dep-Install python3-pygtrie-2.4.2-1.el7.noarch                         @pulpcore
    Dep-Install python3-pyparsing-2.4.7-1.el7.noarch                       @pulpcore
    Dep-Install python3-pyrsistent-0.17.3-1.el7.x86_64                     @pulpcore
    Dep-Install python3-pytz-2021.1-1.el7.noarch                           @pulpcore
    Dep-Install python3-pyyaml-5.4.1-1.el7.x86_64                          @pulpcore
    Dep-Install python3-redis-3.5.3-1.el7.noarch                           @pulpcore
    Dep-Install python3-rq-1.8.1-1.el7.noarch                              @pulpcore
    Dep-Install python3-six-1.15.0-1.el7.noarch                            @pulpcore
    Dep-Install python3-sqlparse-0.4.1-1.el7.noarch                        @pulpcore
    Dep-Install python3-tablib-3.0.0-1.el7.noarch                          @pulpcore
    Dep-Install python3-typing-3.7.4.3-1.el7.noarch                        @pulpcore
    Dep-Install python3-typing-extensions-3.7.4.3-1.el7.noarch             @pulpcore
    Dep-Install python3-uritemplate-3.0.1-2.el7.noarch                     @pulpcore
    Dep-Install python3-urlman-1.4.0-1.el7.noarch                          @pulpcore
    Dep-Install python3-whitenoise-5.2.0-1.el7.noarch                      @pulpcore
    Dep-Install python3-xlrd-2.0.1-1.el7.noarch                            @pulpcore
    Dep-Install python3-xlwt-1.3.0-1.el7.noarch                            @pulpcore
    Dep-Install python3-yarl-1.6.3-1.el7.x86_64                            @pulpcore
    Dep-Install python3-zipp-3.4.0-2.el7.noarch                            @pulpcore
history info

However pip freeze shows chardet==4.0.0

I’ll revert this snapshot and take a look at repos enabled and chardet version before and after another attempt and report back, appreciate your time, many thanks

29

Ok, so chardet 4.0.0 exists on the VM snapshot (py3 is expected), we have an automated build which produces the VM, I’m fairly familiar with this automated build process and nothing explicitly brings this in, however ripping that apart to determine where it came from, is one for another day.

I’ve removed chardet 4 and reran the installer and this is working reliably now (producing zero return code and smart proxy created, no installer errors), I’ll run it a few times to confirm

Sincere apologies for your time wasted on this :frowning: and many thanks for your time and input @gvde and @rainer_d. You’ve both been very kind to offer your support, thank you.

30

Just to confirm, I’ve seen no issues with chardet 4 removed prior to foreman-installer being run :man_facepalming: (sorry)

I’ve completed several install attempts now with additional configuration management running on top, using the foreman ansible collection.

Thanks again all