Smartproxy Foreman 3.0.1 / Katello 4.2.0.1 not all pulp workers starting

Support
1

Problem:

A freshly installed Smart Proxy is unable to sync because it reports the following error:

Oops, we're sorry but something went wrong Not all necessary pulp workers running at https://smartproxy.lont.htm.lan/pulp/api/v3.

When running foreman-maintain service status on the system, it reports everything is running.

From what I was able to find on the forum in other posts is that it could have something to do the pulpcore-worker service.

When looking on the system I can only see 1 of these services running:

[root@smartproxy ~]#  systemctl status pulpcore-worker@* --all
● pulpcore-worker@1.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-11-04 12:31:19 UTC; 7min ago
 Main PID: 1016 (pulpcore-worker)
    Tasks: 1 (limit: 36569)
   Memory: 99.4M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
           └─1016 /usr/libexec/platform-python /usr/bin/pulpcore-worker

Nov 04 12:31:19 smartproxy.lont.htm.lan systemd[1]: Started Pulp Worker.
Nov 04 12:31:38 smartproxy.lont.htm.lan pulpcore-worker-1[1016]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Start>
Nov 04 12:31:38 smartproxy.lont.htm.lan pulpcore-worker-1[1016]: pulp [None]: pulpcore.tasking.worker_watcher:INFO:

And when I manually start pulpcore-worker@2 the problem goes away until the next reboot.

[root@smartproxy ~]#  systemctl status pulpcore-worker@* --all
● pulpcore-worker@2.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-11-04 12:40:05 UTC; 1s ago
 Main PID: 1826 (pulpcore-worker)
    Tasks: 1 (limit: 36569)
   Memory: 44.0M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
           └─1826 /usr/libexec/platform-python /usr/bin/pulpcore-worker

Nov 04 12:40:05 smartproxy.lont.htm.lan systemd[1]: Started Pulp Worker.

● pulpcore-worker@1.service - Pulp Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-11-04 12:31:19 UTC; 8min ago
 Main PID: 1016 (pulpcore-worker)
    Tasks: 1 (limit: 36569)
   Memory: 99.4M
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
           └─1016 /usr/libexec/platform-python /usr/bin/pulpcore-worker

Nov 04 12:31:19 smartproxy.lont.htm.lan systemd[1]: Started Pulp Worker.
Nov 04 12:31:38 smartproxy.lont.htm.lan pulpcore-worker-1[1016]: pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
Nov 04 12:31:38 smartproxy.lont.htm.lan pulpcore-worker-1[1016]: pulp [None]: pulpcore.tasking.worker_watcher:INFO: New worker '1016@smartproxy.lont.htm.lan' discovered

What’s up with that? :slight_smile:

Expected outcome:

Just work™

Foreman and Proxy versions: 3.0.1

Foreman and Proxy plugin versions: Katello 4.2.0.1

Distribution and version: Rocky 8.4

Other relevant data:

2

@katello Ping! Anyone have a clue? :slight_smile:

3

Interesting.

This is not so much Katello, but @installer territory.

Does your machine have 2 CPU cores? And did the installer write down “worker_count: 2” in /etc/foreman-installer/scenarios.d/foreman_proxy_content-answers.yaml?
How does the content of /etc/systemd/system/multi-user.target.wants look like, one or two pulpcore-worker@X.service symlinks?

4

You might be on to something, this is running in a lab setup on my laptop, on which I have limited resources. So some of the the VMs are running on lower then recommended specs.

But yes, indeed, the answer file mentions worker_count: 1 and there is only 1 symlink in the multi-user.target.wants folder.

So this is ‘just’ a case of too little resources allocated to the VM? :slight_smile:

On a side node, even though my laptop tries to take off when the whole lab is running, it does still work, be it a bit slow sometime. So thank you for that! :smiley:

5

Well, I would have expected it to work with just one worker, but if it doesn’t it might be a bug, will try to understand later :wink:

6

Heh, seems it is Katello after all… There was a check that it wanted more than one worker… But that was removed for 4.3:

https://github.com/katello/katello/commit/76eef8331694c66c073e9b03357d88b91c6ced8d

@Justin_Sherrill should this be backported to 4.2?

1 Like
7

I didn’t actually expect anyone to hit that! Yes we can put it on 4.2.2, but in the meantime you might want to just start another pulpcore worker:

foreman-installer --foreman-proxy-content-pulpcore-worker-count 2

1 Like
8

Thanks Justin, well as mentioned, this is very far from a production setup, my poor laptop already has a hard time running the whole solution we’re developing for our client :joy:

But thanks for looking into this! It might be a nice fix, but only for nice cases where someone is running a Smart Proxy on a undersized VM :innocent:

But as mentioned, it does work quite well :slight_smile: