Problem:
Trying to sync content to capsules does not start at all, dynflow only shows ‘waiting for puppet to start the task’
pulp task list --state=waiting or running shows nothing (0), state=completed does show local tasks (like orphan cleanup, delete_version ), but no sync tasks
Expected outcome:
To sync or not to sync, that’s the question…
Foreman and Proxy versions:
2.4.1.1 (central and capsule)
Foreman and Proxy plugin versions:
katello 4.0.3-1
Distribution and version:
CentOS Linux release 7.9.2009
stopped all running (but waiting for pulp) tasks via foreman-rake console find_task=‘id…’.destroy
canceled all (very old) waiting pulp tasks (2 months old)
foreman-rake cleanup
foreman-installer
pulp worker list shows 22667 entries, some dating back a very long time, (with a heartbeat that’s also very old) pulp worker list --missing shows 22621 entries… yikes pulp worker list --online shows only 5 workers, which should be ok.
Is your stuck smart proxy also running Pulp 3? Is “Pulpcore” in the “Active features”? Are there Pulp3-related tasks in the smart proxy sync Dynflow console?
It would be good to see the following on the smart proxy:
sudo pip3 list | grep pulp
If it is running Pulp 3, then perhaps your stuck workers are on the smart proxy instead of the main Katello server. Try running this from the other thread on the smart proxy:
[root@formanserver ~]# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell <<EOF
> from pulpcore.app.models import ReservedResource, Worker
> worker_to_res = {}
> for rr in ReservedResource.objects.all():
> worker_to_res[rr.worker_id] = rr.pulp_id
> workers = [w.pulp_id for w in Worker.objects.online_workers()]
> for rwork in worker_to_res:
> if rwork not in workers:
> print(f'Worker {rwork} owns ReservedResource {worker_to_res[rwork]} and is not in online_workers!!')
> EOF
If Pulpcore is older than 3.14, then the old tasking system is in use. That one was susceptible to having stalled workers.
Did you also upgrade Pulp 3 on your smart proxies? If you did, do you see USE_NEW_WORKER_TYPE = True in /etc/pulp/settings.py?
If you want to use the pulp-cli on your proxy, you have a couple options. You can copy the Katello certs from your Katello machine over to your smart proxy and use pulp-cli from their. Or, you can configure a profile from pulp-cli on your Katello machine to talk to your smart proxy. That was you wouldn’t need to copy certs.
The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you.
The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you.
The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you.
The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you.
