Waiting for Pulp to start the task (RefreshDistribution)

Problem: Sync jobs hanging, waiting for Pulp to start the task

Expected outcome: Synced repo’s

Foreman and Proxy versions: 2.3.5

Foreman and Proxy plugin versions: pulp-deb-plugins-1.10.2-1

Distribution and version: Foreman / Katello is installed on CentOS 7.9.2009, fully patched

Other relevant data: (continuing from this issue

Yesterday, I tried to change the settings in
/usr/share/foreman/config/settings.plugins.d/katello.yaml
to force the deb content type to use pulp2.

That rendered the deb-repo’s unusable, so I reverted that change.

Don’t know if it’s related, but all my daily-sync jobs are hanging. Dynflow tells me:

14: Actions::Pulp3::Repository::RefreshDistribution (waiting for Pulp to start the task)

I’m not sure how to fix this.

Can you check journalctl for any ERROR coming from Pulp services?

Everything is Loaded/Active, no errors or fails.

pulp-admin status looks healty too as far as I can tell. I could manually stop or force cancel all running tasks and try to sync one repo manually to see if that works?

When I have seen it “hang” it’s been due to needing to restart the workers or resource manager or from a recent issue we’ve seen Issue #8708: Tasking: Uploading many advisories in a row can have intermittent failures - Pulp

If you are not seeing evidence of this error in journalctl / /var/log/messages then you could try bouncing the Pulp services.

I did a foreman-maintain service restart and checked with hammer ping if everything looked okay. After a while it did, but the jobs still were stuck with the “waiting for pulp” message in dynflow. I decided to cancel and even force cancel the jobs en cleaned them with foreman-rake foreman_tasks:cleanup (with the appropriate filters.

After this I was able to do a new sync of a repo, so I guess it’s all fine now.

@voorman Can you please share the foreman-rake foreman_tasks:cleanup command that you ran to clear the tasks?
I’m having a hard time coming up with the proper filters.

Thanks!
Raj