Task Pending, How i can debug what is looking the task to start

Scenario:

I tried to sync 3 repos just 1 still pending for more of 34 hours now i
cancel that one and start again a new sync but still show me is pending.

My question here is how i can diagnostic or debug what the task is waiting
for to starting?

Here is the task

3: Actions::Pulp::Repository::Sync (suspended) [ 1394.34s / 5.24s ] Cancel
<https://10.66.208.228/foreman_tasks/dynflow/43da3364-de06-43c0-be34-72f6cf0fcc0a/cancel/3>

Started at: 2015-01-09 06:34:05 UTC

Ended at: 2015-01-09 06:57:20 UTC

Real time: 1394.34s

Execution time (excluding suspended state): 5.24s

Input:

··· --- pulp_id: Red_Hat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server task_id: remote_user: admin-22e88571 locale: en

Output:


pulp_tasks:- exception:
task_type: pulp.server.tasks.repository.sync_with_auto_publish
_href: /pulp/api/v2/tasks/2f522af2-ec94-4c7a-81c0-10e1c60e68a3/
task_id: 2f522af2-ec94-4c7a-81c0-10e1c60e68a3
tags:

  • pulp:repository:Red_Hat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server

  • pulp:action:sync
    finish_time:
    start_time:
    traceback:
    spawned_tasks: []
    progress_report: {}
    queue: reserved_resource_worker-0@satellite7.example.com.dq
    state: waiting
    result:
    error:
    _id:
    $oid: 54af765ebcbc625ea0a77cd2
    id: 54af765e21a08f0d39495fcb
    poll_attempts:
    total: 107
    failed: 0

    6: Actions::Pulp::Repository::DistributorPublish (pending) 
    

Started at:

Ended at:

Real time: 0.00s

Execution time (excluding suspended state): 0.00s

Input:


pulp_id: Red_Hat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server
dependency: Step(3).output[:pulp_tasks]
distributor_type_id: nodes_http_distributor
remote_user: admin-22e88571
locale: en

Output:

— {}

  8: Actions::ElasticSearch::Repository::IndexContent (pending) 

Started at:

Ended at:

Real time: 0.00s

Execution time (excluding suspended state): 0.00s

Input:


dependency: Step(3).output[:pulp_tasks]
id: 5
remote_user: admin-22e88571
locale: en

Output:

— {}

    17: Actions::Katello::Repository::Sync (pending) 

Started at:

Ended at:

Real time: 0.00s

Execution time (excluding suspended state): 0.00s

Input:


repository:
id: 5
name: Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server
label: Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server
product:
id: 6
name: Red Hat Enterprise Linux Server
label: Red_Hat_Enterprise_Linux_Server
cp_id: '69’
provider:
id: 6
name: Red Hat
organization:
id: 4
name: Red_Hat
label: Red_Hat
sync_result: Step(3).output
locale: en

Output:

Hi,

This case, where pulp is not picking up the task can have these causes:

  1. other concurrent tasks are making all workers busy: after some of the tasks finishes, pulp should pick up another one: probably not your case

  2. https://bugzilla.redhat.com/show_bug.cgi?id=1124570 - which can cause that although some workers are free, this task was assigned to some worker that has some work to do still. It's being addressed in latest pulp, but we don't ship that one so far. After all the tasks on the particular worker finish, the pending one should get picked up

  3. occasinaly, pulp decides not pick up some task (for me unknown reason): in that case restarting the pulp services usually helps:

    for s in {qpidd,pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do sudo service $s restart; done

after doning this, you should start seeing the task getting started.

So my suggestion is trying to restart the pulp services…

– Ivan

··· ----- Original Message ----- > Scenario: > > I tried to sync 3 repos just 1 still pending for more of 34 hours now i > cancel that one and start again a new sync but still show me is pending. > > My question here is how i can diagnostic or debug what the task is waiting > for to starting? > > > Here is the task > > 3: Actions::Pulp::Repository::Sync (suspended) [ 1394.34s / 5.24s ] Cancel > > > *Started at:* 2015-01-09 06:34:05 UTC > > *Ended at:* 2015-01-09 06:57:20 UTC > > *Real time:* 1394.34s > > *Execution time (excluding suspended state):* 5.24s > > *Input:* > > --- > pulp_id: > Red_Hat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server > task_id: > remote_user: admin-22e88571 > locale: en > > *Output:* > > --- > pulp_tasks:- exception: > task_type: pulp.server.tasks.repository.sync_with_auto_publish > _href: /pulp/api/v2/tasks/2f522af2-ec94-4c7a-81c0-10e1c60e68a3/ > task_id: 2f522af2-ec94-4c7a-81c0-10e1c60e68a3 > tags: > - > pulp:repository:Red_Hat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server > - pulp:action:sync > finish_time: > start_time: > traceback: > spawned_tasks: [] > progress_report: {} > queue: reserved_resource_worker-0@satellite7.example.com.dq > state: waiting > result: > error: > _id: > $oid: 54af765ebcbc625ea0a77cd2 > id: 54af765e21a08f0d39495fcb > poll_attempts: > total: 107 > failed: 0 > > 6: Actions::Pulp::Repository::DistributorPublish (pending) > > *Started at:* > > *Ended at:* > > *Real time:* 0.00s > > *Execution time (excluding suspended state):* 0.00s > > *Input:* > > --- > pulp_id: > Red_Hat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server > dependency: Step(3).output[:pulp_tasks] > distributor_type_id: nodes_http_distributor > remote_user: admin-22e88571 > locale: en > > *Output:* > > --- {} > > 8: Actions::ElasticSearch::Repository::IndexContent (pending) > > *Started at:* > > *Ended at:* > > *Real time:* 0.00s > > *Execution time (excluding suspended state):* 0.00s > > *Input:* > > --- > dependency: Step(3).output[:pulp_tasks] > id: 5 > remote_user: admin-22e88571 > locale: en > > *Output:* > > --- {} > > 17: Actions::Katello::Repository::Sync (pending) > > *Started at:* > > *Ended at:* > > *Real time:* 0.00s > > *Execution time (excluding suspended state):* 0.00s > > *Input:* > > --- > repository: > id: 5 > name: Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server > label: Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server > product: > id: 6 > name: Red Hat Enterprise Linux Server > label: Red_Hat_Enterprise_Linux_Server > cp_id: '69' > provider: > id: 6 > name: Red Hat > organization: > id: 4 > name: Red_Hat > label: Red_Hat > sync_result: Step(3).output > locale: en > > *Output:* > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. >